No title

Communicated by Lawrence Jackel VIEW Neural Networks and the BiadVariance Dilemma Stuart Geman Division of Applied Ma...

Author: MIT Press

5 downloads 800 Views 26MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Communicated by Lawrence Jackel

VIEW

Neural Networks and the BiadVariance Dilemma Stuart Geman Division of Applied Mathematics, Brown University, Providence, XI 02912 USA

Elie Bienenstock RenC Doursat ESPCI, 10 rue Vuuquelin, 75005 Paris, France

Feedforward neural networks trained by error backpropagation are examples of nonparametric regression estimators. We present a tutorial on nonparametric inference and its relation to neural networks, and we use the statistical viewpoint to highlight strengths and weaknesses of neural models. We illustrate the main points with some recognition experiments involving artificial data as well as handwritten numerals. In way of conclusion, we suggest that current-generation feedforward neural networks are largely inadequate for difficult problems in machine perception and machine learning, regardless of parallelversus-serial hardware or other implementation issues. Furthermore, we suggest that the fundamental challenges in neural modeling are about representation rather than learning per se. This last point is supported by additional experiments with handwritten numerals. 1 Introduction

Much of the recent work on feedforward artificial neural networks brings to mind research in nonparametric statistical inference. This is a branch of statistics concerned with model-free estimation, or, from the biological viewpoint, tabula rasa learning. A typical nonparametric inference problem is the learning (or “estimating,” in statistical jargon) of arbitrary decision boundaries for a classification task, based on a collection of labeled (pre-classified) training samples. The boundaries are arbitrary in the sense that no particular structure, or class of boundaries, is assumed a priori. In particular, there is no parametric model, as there would be with a presumption of, say, linear or quadratic decision surfaces. A similar point of view is implicit in many recent neural network formulations, suggesting a close analogy to nonparametric inference. Of course statisticians who work on nonparametric inference rarely concern themselves with the plausibility of their inference algorithms Neural Computation 4,l-58 (1992)

@ 1992 Massachusetts Institute of Technology

2

S. Geman, E. Bienenstock, and R. Doursat

as brain models, much less with the prospects for implementation in “neural-like” parallel hardware, but nevertheless certain generic issues are unavoidable and therefore of common interest to both communities. What sorts of tasks for instance can be learned, given unlimited time and training data? Also, can we identify “speed limits,” that is, bounds on how fast, in terms of the number of training samples used, something can be learned? Nonparametric inference has matured in the past 10 years. There have been new theoretical and practical developments, and there is now a large literature from which some themes emerge that bear on neural modeling. In Section 2 we will show that learning, as it is represented in some current neural networks, can be formulated as a (nonlinear) regression problem, thereby making the connection to the statistical framework. Concerning nonparametric inference, we will draw some general conclusions and briefly discuss some examples to illustrate the evident utility of nonparametric methods in practical problems. But mainly we will focus on the limitations of these methods, at least as they apply to nontrivial problems in pattern recognition, speech recognition, and other areas of machine perception. These limitations are well known, and well understood in terms of what we will call the bias/variance dilemma. The essence of the dilemma lies in the fact that estimation error can be decomposed into two components, known as bias and variance; whereas incorrect models lead to high bias, truly model-free inference suffers from high variance. Thus, model-free (tabula vasa) approaches to complex inference tasks are slow to “converge,” in the sense that large training samples are required to achieve acceptable performance. This is the effect of high variance, and is a consequence of the large number of parameters, indeed infinite number in truly model-free inference, that need to be estimated. Prohibitively large training sets are then required to reduce the variance contribution to estimation error. Parallel architectures and fast hardware do not help here: this ”convergence problem” has to do with training set size rather than implementation. The only way to control the variance in complex inference problems is to use model-based estimation. However, and this is the other face of the dilemma, model-based inference is biasprone: proper models are hard to identify for these more complex (and interesting) inference problems, and any model-based scheme is likely to be incorrect for the task at hand, that is, highly biased. The issues of bias and variance will be laid out in Section 3, and the “dilemma” will be illustrated by experiments with artificial data as well as on a task of handwritten numeral recognition. Efforts by statisticians to control the tradeoff between bias and variance will be reviewed in Section 4. Also in Section 4, we will briefly discuss the technical issue of consistency, which has to do with the asymptotic (infinite-training-sample) correctness of an inference algorithm. This is of some recent interest in the neural network literature. In Section 5, we will discuss further the bias/variance dilemma, and

Neural Networks and the Bias/Variance Dilemma

3

relate it to the more familiar notions of interpolation and extrapolation. We will then argue that the dilemma and the limitations it implies are relevant to the performance of neural network models, especially as concerns difficult machine learning tasks. Such tasks, due to the high dimension of the “input space,” are problems of extrapolation rather than interpolation, and nonparametric schemes yield essentially unpredictable results when asked to extrapolate. We shall argue that consistency does not mitigate the dilemma, as it concerns asymptotic as opposed to finitesample performance. These discussions will lead us to conclude, in Section 6, that learning complex tasks is essentially impossible without the a priori introduction of carefully designed biases into the machine’s architecture. Furthermore, we will argue that, despite a long-standing preoccupation with learning per se, the identification and exploitation of the ”right” biases are the more fundamental and difficult research issues in neural modeling. We will suggest that some of these important biases can be achieved through proper data representations, and we will illustrate this point by some further experiments with handwritten numeral recognition.

2 Neural Models and Nonparametric Inference

2.1 Least-Squares Learning and Regression. A typical learning problem might involve a feature or input vector x, a response vector y, and the goal of learning to predict y from x, where the pair (x,y) obeys some unknown joint probability distribution, P. A training set (xl, y,), . . . , (XN,y ~ ) is a collection of observed (x, y) pairs containing the desired response y for each input x. Usually these samples are independently drawn from P, though many variations are possible. In a simple binary classification problem, y is actually a scalar y E (0, l}, which may, for example, represent the parity of a binary input string x E (0, l}’,or the voiced/unvoiced classification of a phoneme suitably coded by x as a second example. The former is ”degenerate” in the sense that y is uniquely determined by x, whereas the classification of a phoneme might be ambiguous. For clearer exposition, we will take y = y to be one-dimensional, although our remarks apply more generally. The learning problem is to construct a function (or “machine”) f(x) based on the data ( x I , y l ) ,. . . (XN, y ~ )so, that f(x) approximates the desired response y. Typically, f is chosen to minimize some cost functional. For example, in feedforward networks (Rumelhart et al. 1986a,b), one usually forms the sum of observed squared errors,


4

and f is chosen to make this sum as small as possible. Of course f is really parameterized, usually by idealized “synaptic weights,” and the minimization of equation 2.1 is not over all possible functions f, but over the class generated by all allowed values of these parameters. Such minimizations are much studied in statistics, since, as we shall later see, they are one way to estimate a regression. The regression of y on x is E[y I XI,that is, that (deterministic) function of x that gives the mean value of y conditioned on x. In the degenerate case, that is, if the probability distribution P allows only one value of y for each x (as in the parity problem for instance), E[y I x] is not really an average: it is just the allowed value. Yet the situation is often ambiguous, as in the phoneme classification problem. Consider the classification example with just two classes: “Class A“ and its complement. Let y be 1 if a sample x is in Class A, and 0 otherwise. The regression is then simply

E [y I x] = P (y = 1 I x)

=P

(Class A I x)

the probability of being in Class A as a function of the feature vector x. It may or may not be the case that x unambiguously determines class membership, y. If it does, then for each x, Ely I x] is either 0 or 1: the regression is a binary-valued function. Binary classification will be illustrated numerically in Section 3, in a degenerate as well as in an ambiguous case. More generally, we are out to “fit the data,” or, more accurately, fit the ensemble from which the data were drawn. The regression is an excellent solution, by the following reasoning. For any function f(x), and any fixed x,’

E [(Y

- f W 21x1

[((Y - Ek I XI) + ( E b I XI -f(X)))2 I XI E [(Y - ElY I I x] + (El!/ I XI -f(x))’ + 2E [(Y - E k I XI) I xl . (Eb I XI - f ( x ) ) E [ (Y - El!/ I I x] + (ELy I XI - fW2 + 2 Fly I XI - E b I XI) . ( E [ y I XI -f(x)) E [(Y - E b I XI)* I x] + (Eb I XI -f(x))’

= E =

=

=

(2.2)

2 E [(Y - ELy I xu2 I x] In other words, among all functions of x, the regression is the best predictor of y given x, in the mean-squared-error sense. Similar remarks apply to likelihood-based (instead of least-squaresbased) approaches, such as the Boltzmann Machine (Ackley et al. 1985; Hinton and Sejnowski 1986). Instead of decreasing squared error, the ‘For any function 4(x,y), and any fixed x, E[$(x,y) I x] is the conditional expectation of 4(x,y) given x, that is, the average of d(x,y) taken with respect to the conditional probability distribution P(y I x).


5

Boltzmann Machine implements a Monte Carlo computational algorithm for increasing likelihood. This leads to the maximum-likelihood estimator of a probability distribution, at least if we disregard local maxima and other confounding computational issues. The maximum-likelihood estimator of a distribution is certainly well studied in statistics, primarily because of its many optimality properties. Of course, there are many other examples of neural networks that realize well-defined statistical estimators (see Section 5.1). The most extensively studied neural network in recent years is probably the backpropagation network, that is, a multilayer feedforward network with the associated error-backpropagation algorithm for minimizing the observed sum of squared errors (Rumelhart et al. 1986a,b). With this in mind, we will focus our discussion by addressing least-squares estimators almost exclusively. But the issues that we will raise are ubiquitous in the theory of estimation, and our main conclusions apply to a broader class of neural networks.

2.2 Nonparametric Estimation and Consistency. If the response variable is binary, y E {O,l}, and if y = 1 indicates membership in "Class A," then the regression is just P(C1ass A I x ) , as we have already observed. A decision rule, such as "choose Class A if P(C1ass A I x ) > 1/2," then generates a partition of the range of x (call this range H ) into H A = { x : P(C1ass A I x) > 1/2} and its complement H - HA = HA. Thus, x E HA is classified as "A," x E H A is classified as "not A." It may be the case that HA and H A are separated by a regular surface (or "decision boundary"), planar or quadratic for example, or the separation may be highly irregular. Given a sequence of observations ( x l , y l ) (x2, , y2), . . . we can proceed to estimate P(C1ass A I x ) (= E[y I XI), and hence the decision boundary, from two rather different philosophies. On the one hand we can assume a priori that H A is known up to a finite, and preferably small, number of parameters, as would be the case if H A and H A were linearly or quadratically separated, or, on the other hand, we can forgo such assumptions and "let the data speak for itself." The chief advantage of the former, parametric, approach is of course efficiency: if the separation really is planar or quadratic, then many fewer data are needed for accurate estimation than if we were to proceed without parametric specifications. But if the true separation departs substantially from the assumed form, then the parametric approach is destined to converge to an incorrect, and hence suboptimal solution, typically (but depending on details of the estimation algorithm) to a "best" approximation within the allowed class of decision boundaries. The latter, nonparamefric, approach makes no such a priori commitments. The asymptotic (large sample) convergence of an estimator to the object of estimation is called consistency. Most nonparametric regression

6


algorithms are consistent, for essentially any regression function E[y I XI.' This is indeed a reassuring property, but it comes with a high price: depending on the particular algorithm and the particular regression, nonparametric methods can be extremely slow to converge. That is, they may require very large numbers of examples to make relatively crude approximations of the target regression function. Indeed, with small samples the estimator may be too dependent on the particular samples observed, that is, on the particular realizations of (x, y) (we say that the variance of the estimator is high). Thus, for a fixed and finite training set, a parametric estimator may actually outperform a nonparametric estimator, even when the true regression is outside of the parameterized class. These issues of bias and variance will be further discussed in Section 3. For now, the important point is that there exist many consistent nonparametric estimators, for regressions as well as probability distributions. This means that, given enough training samples, optimal decision rules can be arbitrarily well approximated. These estimators are extensively studied in the modern statistics literature. Parzen windows and nearestneighbor rules (see, e.g., Duda and Hart 1973; Hardle 1990), regularization methods (see, e.g., Wahba 1982) and the closely related method of sieves (Grenander 1981; Geman and Hwang 1982), projection pursuit (Friedman and Stuetzle 1981; Huber 19851, recursive partitioning methods such as "CART," which stands for "Classification and Regression Trees" (Breiman et al. 1984),Alternating Conditional Expectations, or "ACE" (Breiman and Friedman 1985), and Multivariate Adaptive Regression Splines, or "MARS" (Friedman 1991), as well as feedforward neural networks (Rumelhart et al. 1986a,b) and Boltzmann Machines (Ackley et al. 1985; Hinton and Sejnowski 1986), are a few examples of techniques that can be used to construct consistent nonparametric estimators. 2.3 Some Applications of Nonparametric Inference. In this paper, we shall be mostly concerned with limitations of nonparametric methods, and with the relevance of these limitations to neural network models. But there is also much practical promise in these methods, and there have been some important successes. An interesting and difficult problem in industrial "process specification" was recently solved at the General Motors Research Labs (Lorenzen 1988) with the help of the already mentioned CART method (Breiman et al. 1984). The essence of CART is the following. Suppose that there are m classes, y E (1.2,. . . , m}, and an input, or feature, vector x. Based on a training sample (XI, yl), . . . , (xN,yN) the CART algorithm constructs a partitioning of the (usually high-dimensional) domain of x into rectan*One has to specify the mode of convergence: the estimator is itself a function, and furthermore depends on the realization of a random training set (see Section 4.2). One also has to require certain technical conditions, such as measurability of the regression function.

Neural Networks and the BiasNariance Dilemma

7

gular cells, and estimates the class-probabilities { P ( y = k) : k = 1,. . . , m } within each cell. Criteria are defined that promote cells in which the estimated class probabilities are well-peaked around a single class, and at the same time discourage partitions into large numbers of cells, relative to N. CART provides a family of recursive partitioning algorithms for approximately optimizing a combination of these competing criteria. The GM problem solved by CART concerned the casting of certain engine-block components. A new technology known as lost-foam casting promises to alleviate the high scrap rate associated with conventional casting methods. A Styrofoam ”model” of the desired part is made, and then surrounded by packed sand. Molten metal is poured onto the Styrofoam, which vaporizes and escapes through the sand. The metal then solidifies into a replica of the styrofoam model. Many “process variables” enter into the procedure, involving the settings of various temperatures, pressures, and other parameters, as well as the detailed composition of the various materials, such as sand. Engineers identified 80 such variables that were expected to be of particular importance, and data were collected to study the relationship between these variables and the likelihood of success of the lost-foam casting procedure. (These variables are proprietary.) Straightforward data analysis on a training set of 470 examples revealed no good “first-order” predictors of success of casts (a binary variable) among the 80 process variables. Figure 1 (from Lorenzen 1988) shows a histogram comparison for that variable that was judged to have the most visually disparate histograms among the 80 variables: the left histogram is from a population of scrapped casts, and the right is from a population of accepted casts. Evidently, this variable has no important prediction power in isolation from other variables. Other data analyses indicated similarly that no obvious low-order multiple relations could reliably predict success versus failure. Nevertheless, the CART procedure identified achievable regions in the space of process variables that reduced the scrap rate in this production facility by over 75%. As might be expected, this success was achieved by a useful mix of the nonparametric algorithm, which in principal is fully automatic, and the statistician’s need to bring to bear the realities and limitations of the production process. In this regard, several important modifications were made to the standard CART algorithm. Nevertheless, the result is a striking affirmation of the potential utility of nonparametric methods. There have been many success stories for nonparametric methods. An intriguing application of CART to medical diagnosis is reported in Goldman et al. (19821, and further examples with CART can be found in Breiman et al. (1984). The recent statistics and neural network literatures contain examples of the application of other nonparametric methods as well. A much-advertised neural network example is the evaluation of loan applications (cf. Collins et al. 1989). The basic problem is to classify a loan candidate as acceptable or not acceptable based on 20 or so

8


Figure 1: Left histogram: distribution of process variable for unsuccessful castings. Right histogram: distribution of same process variable for successful castings. Among all 80 process variables, this variable was judged to have the most dissimilar success/failure histograms. (Lorenzen 1988)

variabIes summarizing an applicant’s financial status, These include, for example, measures of income and income stability, debt and other financial obligations, credit history, and possibly appraised values in the case of mortgages and other secured loans. A conventional parametric statistical approach is the so-called logit model (see, for example, Cox 1970), which posits a linear relationship between the logistic transformation of the desired variable (here the probability of a successful return to the lender) and the relevant independent variables (defining financial s t a t u ~ ) .Of ~ course, a linear model may not be suitable, in which case the logit estimator would perform poorly; it would be too biased. On the other hand, very large training sets are available, and it makes good sense to try less parametric methods, such as the backpropagation algorithm, the nearest-neighbor algorithm, or the ”Multiple-Neural-Network Learning System” advocated for this problem by Collins et al. (1989).

3The logistic transformation of a probability p is log,[p/(l

-

p)].


9

These examples will be further discussed in Section 5, where we shall draw a sharp contrast between these relatively easy tasks and problems arising in perception and in other areas of machine intelligence. 3 Bias and Variance

3.1 The BiasIVariance Decomposition of Mean-Squared Error. The regression problem is to construct a functionf(x) based on a ”training set” ( x ~ , y ~. .). , (XN,YN), for the purpose of approximating y at future observations of x. This is sometimes called “generalization,” a term borrowed from psychology. To be explicit about the dependence o f f on the data V = {(XI,yl), . . . , (XN, YN)},we will write f(x; V )instead of simply f(x). Given D,and given a particular x, a natural measure of the effectiveness off as a predictor of y is

the mean-squared error (where €[.I means expectation with respect to the probability distribution P, see Section 2). In our new notation emphasizing the dependency of f on V (which is fixed for the moment), equation 2.2 reads

E [(Y - f ( x ; m 2

IX>D]

= E

[(Y - E [ y

+ (f(x; 2))

I XI)’ -

I X > 4

E[y I XI)’

E[(y-Ely I x ] ) I~x, V ]does not depend on the data, V,or on the estimator, f; it is simply the variance of y given x. Hence the squared distance to the regression function,

measures, in a natural way, the effectiveness off as a predictor of y. The mean-squared error off as an estimator of the regression E [ y I x] is €27

[(f(x;D)- Ely

I XI)’]

(3.1)

where E D represents expectation with respect to the training set, V,that is, the average over the ensemble of possible V (for fixed sample size N ) . It may be that for a particular training set, V,f(x;V) is an excellent approximation of Ely I x], hence a near-optimal predictor of y. At the same time, however, it may also be the case that f(x; D)is quite different for other realizations of D,and in general varies substantially with V,or it may be that the average (over all possible V)of f(x; V )is rather far from the regression E[y I XI. These circumstances will contribute large values in 3.1, making f(x; V)an unreliable predictor of y. A useful way to assess


10

these sources of estimation error is via the bias/ variance decomposition, which we derive in a way similar to 2.2 for any x, ED

[ ( f2)) h- E l y I XI)’] = ED

[ ( ( f ( x ; D-) ED V(x;DO)l)+ ( E D [ f ( x ; D )] Ely

I XI))‘]

[ ( f ( x ; D) E D [f(x;D)I)’]+ E D [ ( E D [ f ( x ; D )] E[y I XI)’] + ED [ ( f ( x ;2)) - ED [ f ( D~ ) ] )( E D [ f ( D)] ~ - E[y I X I ) ] = E D [ ( f ( x ;D )- E D [ f ( D ~ ) ] + ( E D V(X;D)]- E[y I XI)’ + ED [ f ( x ; D -) E D [ f ( x ; D ) ].]( E D [f(x;D)] - E[y I x ] ) “bias” = ( E D [ f ( x ; D )] Ely I ~ 1 ) ~ ”variance” E~ [ ~ ( xD) ; - E= ~ ( xD)] ;

= ED

I’)

+

)’I

If, on the average, f ( x ;D)is different from E[y I x], then f ( x ;D)is said to be biased as an estimator of E[y I XI. In general, this depends on P; the same f may be biased in some cases and unbiased in others. As said above, an unbiased estimator may still have a large meansquared error if the variance is large: even with EDV(X;D)]= E l y I x], f ( x ; D ) may be highly sensitive to the data, and, typically, far from the regression E l y I XI. Thus either bias or variance can contribute to poor performance. There is often a tradeoff between the bias and variance contributions to the estimation error, which makes for a kind of “uncertainty principle” (Grenander 1951). Typically, variance is reduced through “smoothing,” via a combining, for example, of the influences of samples that are nearby in the input ( x ) space. This, however, will introduce bias, as details of the regression function will be lost; for example, sharp peaks and valleys will be blurred. 3.2 Examples. The issue of balancing bias and variance is much studied in estimation theory. The tradeoff is already well illustrated in the one-dimensional regression problem: x = x E [0,1]. In an elementary version of this problem, y is related to x by

Y = g ( x ) + 77

(3.2)

where g is an unknown function, and 77 is.zero-mean “noise” with distribution independent of x. The regression is then g ( x ) , and this is the best (mean-squared-error) predictor of y. To make our points more clearly, we will suppose, for this example, that only y is random - x can be chosen as we please. If we are to collect N observations, then a natural ”design” for the inputs is xi = i/N, 1 5 i 5 N, and the data are then the corresponding N values of y, 2, = {yl, . . . ,yN}. An example (from Wahba and Wold 19751, with N = 100, g(x) = 4.26(ecX- 4e-& 3ec3’), and 77 gaussian with standard deviation 0.2, is shown in Figure 2. The squares are the data points, and the broken curve, in each panel, is the

+

Neural Networks and the Bias/Variance DiIemma

11

a

b

t

C

Figure 2: One hundred observations (squares) generated according to equation 4, with g ( x ) = 4.26(eCX - 4eCZX+ 3 ~ ~The ~ noise ) . is zero-mean gaussian with standard error 0.2. In each panel, the broken curve is g and the solid curve is a spline fit. (a) Smoothing parameter chosen to control variance. (b) Smoothing parameter chosen to control bias. (c) A compromising value of the smoothing parameter, chosen automatically by cross-validation. (From Wahba and Wold 1975)

regression, g(X). (The solid curves are estimates of the regression, as will be explained shortly.) The object is to make a guess at g(x), using the noisy observations, yi = g(xi) yi, 1 5 i 5 N. At one extreme, f ( x ; D ) could be defined as the linear (or some other) interpolant of the data. This estimator is truly unbiased at x = xi, 1 5 i 5 N, since

+

+~

E D v ( X i ; D)]== E [g(Xi)

i ] g(Xi) = E

[y

I xi]

Furthermore, if g is continuous there is also very little bias in the vicinity of the observation points, X i , 1 5 i 5 N. But if the variance of y is large, then there will be a large variance component to the mean-squared error (3.11, since ED

[V(xi;D) - E D v ( x i ; ~ ) 1 ) ~=] E D [k(xi)+ vi - g(xi))’] = E

12


which, since qi has zero mean, is the variance of 77,. This estimator is indeed very sensitive to the data. At the other extreme, we may take f ( x ; V ) = k ( x ) for some wellchosen function h(x), independent of V . This certainly solves the variance problem! Needless to say, there is likely to be a substantial bias, for this estimator does not pay any attention to the data. A better choice would be an intermediate one, balancing some reasonable prior expectation, such as smoothness, with faithfulness to the observed data. One example is a feedforward neural network trained by error backpropagation. The output of such a network isf(x; w) = f [ x ; w(V)], where w(D)is a collection of weights determined by (approximately) minimizing the sum of squared errors:

(3.3) How big a network should we employ? A small network, with say one hidden unit, is likely to be biased, since the repertoire of available functions spanned by f ( x ; w) over allowable weights will in this case be quite limited. If the true regression is poorly approximated within this class, there will necessarily be a substantial bias. On the other hand, if we overparameterize, via a large number of hidden units and associated weights, then the bias will be reduced (indeed, with enough weights and hidden units, the network will interpolate the data), but there is then the danger of a significant variance contribution to the mean-squared error. (This may actually be mitigated by incomplete convergence of the minimization algorithm, as we shall see in Section 3.5.5.) Many other solutions have been invented, for this simple regression problem as well as its extensions to multivariate settings (y -+ y E Rd, x + x E R‘, for some d > 1 and I > 1). Often splines are used, for example. These arise by first restricting f via a “smoothing criterion” such as

for some fixed integer m 2 1 and fixed A. (Partial and mixed partial derivatives enter when x + x E R’; see, for example, Wahba 1979.) One then solves for the minimum of

among all f satisfying equation 3.4. This minimization turns out to be tractable and yields f ( x ) = f ( x ; D ) , a concatenation of polynomials of degree 2m-1 on the intervals (xi, x i + l ) ; the derivatives of the polynomials, up to order 2m -2, match at the “knots” {xi}El. With m = 1, for example, the solution is continuous and piecewise linear, with discontinuities in


13

the derivative at the knots. When m = 2 the polynomials are cubic, the first two derivatives are continuous at the knots, and the curve appears globally “smooth.” Poggio and Girosi (1990) have shown how splines and related estimators can be computed with multilayer networks. The “regularization” or ”smoothing” parameter X plays a role similar to the number of weights in a feedforward neural network. Small X produce small-variance high-bias estimators; the data are essentially ignored in favor of the constraint (“oversmoothing”). Large values of X produce interpolating splines: f ( x ; ; 2)) = y,, 1 5 i 5 N, which, as we have seen, may be subject to high variance. Examples of both oversmoothing and undersmoothing are shown in Figure 2a and b, respectively. The solid lines are cubic-spline (rn = 2) estimators of the regression. There are many recipes for choosing A, and other smoothing parameters, from the dnfa, a procedure known as ”automatic smoothing” (see Section 4.1). A popular example is called cross-validation (again, see Section 4,1), a version of which was used in Figure 2c. There are of course many other approaches to the regression problem. Two in particular are the nearest-neighbor estimators and the kernel estimators, which we have used in some experiments both on artificial data and on handwritten numeral recognition. The results of these experiments will be reviewed in Section 3.5. 3.3 Nonparametric Estimation. Nonparametric regression estimators are characterized by their being consistent for all regression problems. Consistency requires a somewhat arbitrary specification: in what sense does the estimator f(x;V)converge to the regression E [ y I Let us be explicit about the dependence off on sample size, N, by writing V = V N and then f ( x ; D ~for ) the estimator, given the N observations DN. One version of consistency is ”pointwise mean-squared error”:

XI?

for each x. A more gIobal specification is in terms of integrated meansquared error: (3.5) There are many variations, involving, for example, almost sure convergence, instead of the mean convergence that is defined by the expectation operator ED. Regardless of the details, any reasonable specification will require that both bias and variance go to zero as the size of the training sample increases. In particular, the class of possible functions f(x; V N ) must approach Ely I x] in some suitable sense? or there will necessarily 4The appropriate metric is the one used to define consistency. Lz, for example, with 3.5.

14


be some residual bias. This class of functions will therefore, in general, have to grow with N. For feedforward neural networks, the possible functions are those spanned by all allowed weight values. For any fixed architecture there will be regressions outside of the class, and hence the network cannot realize a consistent nonparametric algorithm. By the same token, the spline estimator is not consistent (in any of the usual senses) whenever the regression satisfies

since the estimator itself is constrained to violate this condition (see equation 3.4). It is by now well-known (see, e.g., White 1990) that a feedforward neural network (with some mild conditions on Ely I x] and network structure, and some optimistic assumptions about minimizing 3.3) can be made consistent by suitably letting the network size grow with the size of the training set, in other words by gradually diminishing bias. Analogously, splines are made consistent by taking X = AN T 03 sufficiently slowly. This is indeed the general recipe for obtaining consistency in nonparametric estimation: slowly remove bias. This procedure is somewhat delicate, since the variance must also go to zero, which dictates a gradual reduction of bias (see discussion below, Section 5.1). The main mathematical issue concerns this control of the variance, and it is here that tools such as the Vapnikxervonenkis dimension come into play. We will be more specific in our brief introduction to the mathematics of consistency below (Section 4.2). As the examples illustrate, the distinction between parametric and nonparametric methods is somewhat artificial, especially with regards to fixed and finite training sets. Indeed, most nonparametric estimators, such as feedforward neural networks, are in fact a sequence of parametric estimators indexed by sample size. 3.4 The Dilemma. Much of the excitement about artificial neural networks revolves around the promise to avoid the tedious, difficult, and generally expensive process of articulating heuristics and rules for machines that are to perform nontrivial perceptual and cognitive tasks, such as for vision systems and expert systems. We would naturally prefer to "teach our machines by example, and would hope that a good learning algorithm would "discover" the various heuristics and rules that apply to the task at hand. It would appear, then, that consistency is relevant: a consistent learning algorithm will, in fact, approach optimal performance, whatever the task. Such a system might be said to be unbiased, as it is not a priori dedicated to a particular solution or class of solutions. But the price to pay for achieving low bias is high variance. A machine sufficiently versatile to reasonably approximate a broad range of


15

input/ output mappings is necessarily sensitive to the idiosyncrasies of the particular data used for its training, and therefore requires a very large training set. Simply put, dedicated machines are harder to build but easier to train. Of course there is a quantitative tradeoff, and one can argue that for many problems acceptable performance is achievable from a more or less tabula rasa architecture, and without unrealistic numbers of training examples. Or that specific problems may suggest easy and natural specific structures, which introduce the "right" biases for the problem at hand, and thereby mitigate the issue of sample size. We will discuss these matters further in Section 5. 3.5 Experiments in Nonparametric Estimation. In this section, we shall report on two kinds of experiments, both concerning classification, but some using artificial data and others using handwritten numerals. The experiments with artificial data are illustrative since they involve only two dimensions, making it possible to display estimated regressions as well as bias and variance contributions to mean-squared error. Experiments were performed with nearest-neighbor and Parzen-window estimators, and with feedforward neural networks trained via error backpropagation. Results are reported following brief discussions of each of these estimation methods.

3.5.1 Nearest-Neighbor Regression. This simple and time-honored approach provides a good performance benchmark. The "memory" of the . any machine is exactly the training set V = {(xI,yI), , . .,(XN, y ~ ) ) For input vector x, a response vector y is derived from the training set by averaging the responses to those inputs from the training set which happen to lie close to x. Actually, there is here a collection of algorithms indexed by an integer, "k," which determines the number of neighbors of x that enter into the average. Thus, the k-nearest-neighbor estimator is just

where Nk(x) is the collection of indices of the k nearest neighbors to x among the input vectors in the training set {xi}:,. (There is also a k-nearest-neighbor procedure for classification: If y = y E { 1,2, . . . , C}, representing C classes, then we assign to x the class y E { 1,2, . . . , C) most frequent among the set { ~ i } i ~ N ~ ( where ~), y i is the class of the training input xi.) If k is "large" (e.g., k is almost N)then the response f(x; V)is a relatively smooth function of x, but has little to do with the actual positions of the x[s in the training set. In fact, when k = N , f(x;D) is independent of x, and of {xi}El; the output is just the average observed output 1 / N ELl yi. When N is large, 1 / N ELl yi is likely to be nearly unchanged

16


from one training set to another. Evidently, the variance contribution to mean-squared error is then small. On the other hand, the response to a particular x is systematically biased toward the population response, regardless of any evidence for local variation in the neighborhood of x. For most problems, this is of course a bad estimation policy. The other extreme is the first-nearest-neighbor estimator; we can expect less bias. Indeed, under reasonable conditions, the bias of the firstnearest-neighbor estimator goes to zero as N goes to infinity. On the other hand, the response at each x is rather sensitive to the idiosyncrasies of the particular training examples in D. Thus the variance contribution to mean-squared error is typically large. From these considerations it is perhaps not surprising that the best solution in many cases is a compromise between the two extremes k = 1 and k = N. By choosing an intermediate k, thereby implementing a reasonable amount of smoothing, one may hope to achieve a significant reduction of the variance, without introducing too much bias. If we now consider the case N + 00, the k-nearest-neighbor estimator can be made consistent by choosing k = k~ T 00 sufficiently slowly. The idea is that the variance is controlled (forced to zero) by k~ T m, whereas the bias is controlled by ensuring that the kNth nearest neighbor of x is actually getting closer to x as N -+ m. 3.5.2 Parzen- Window Regression. The "memory" of the machine is again the entire training set D, but estimation is now done by combining "kernels," or "Parzen windows," placed around each observed input point xi, 1 5 i 5 N.The form of the kernel is somewhat arbitrary, but it is usually chosen to be a nonnegative function of x that is maximum at x = 0 and decreasing away from x = 0. A common choice is

the gaussian kernel, for x E Rd. The scale of the kernel is adjusted by a "bandwidth CJ: W ( x )-+ (1/0)~W(x/o).The effect is to govern the extent to which the window is concentrated at x = 0 (small CJ),or is spread out over a significant region around x = 0 (large (T). Having fixed a kernel W(.), and a bandwidth CJ, the Parzen regression estimator at x is formed from a weighted average of the observed responses {yi}El:

Clearly, observations with inputs closer to x are weighted more heavily. There is a close connection between nearest-neighbor and Parzenwindow estimation. In fact, when the bandwidth o is small, only close neighbors of x contribute to the response at this point, and the procedure is akin to k-nearest-neighbor methods with small k. On the other hand,


17

when o is large, many neighbors contribute significantly to the response, a situation analogous to the use of large values of k in the k-nearestneighbor method. In this way, D governs bias and variance much as k does for the nearest-neighbor procedure: small bandwidths generally offer high-variance/low-bias estimation, whereas large bandwidths incur relatively high bias but low variance. There is also a Parzen-window procedure for classification: we assign to x the class y = y E {1,2, . . . C} which maximizes ~

where Ny is the number of times that the classification y is seen in the training set, N,,= #{i : yi = y}. If W ( x )is normalized, so as to integrate to one, then fu(x;23) estimates the density of inputs associated with the class y (known as the ”class-conditional density”). Choosing the class with maximum density at x results in minimizing the probability of error, at least when the classes are a priori equally likely. (If the a priori probabilities of the C classes, p(y) y E (1.2,. . . , C}, are unequal, but known, then the minimum probability of error is obtained by choosing y to maximize P(Y) . f y ( x ; W 3.5.3 Feedforward Network Trained by Error Backpropagation. Most readers are already familiar with this estimation technique. We used twolayer networks, that is, networks with one hidden layer, with full connections between layers. The number of inputs and outputs depended on the experiment and on a coding convention; it will be laid out with the results of the different experiments in the ensuing paragraphs. In the usual manner, all hidden and output units receive one special input that is nonzero and constant, allowing each unit to learn a “threshold.” Each unit outputs a value determined by the sigmoid function (3.8)

given the input

Here, {_ sin((n/2)x~), y = 0.91 and "bottom" [xq < sin((7r/2)xl),y= -0.91 pieces. The regression is then the binary-valued function E[y 1 x] = .9 above the sinusoid and -0.9 below (see Fig. 3a). The training set, 2, = {(x,,yl), .. . , (xN,YN)}, is constructed to have 50 examples from each class. For y = 0.9, the 50 inputs are chosen from the uniform distribution on the region above the sinusoid; the y = -0.9 inputs are chosen uniformly from the region below the sinusoid. Classification can be made ambiguous within the same basic setup, by randomly perturbing the input vector before determining its class. To describe precisely the random mechanism, let us denote by B1 (x) the


19

a

b

Figure 3: Two regression surfaces for experimentswith artificial data. (a)Output is deterministic function of input, f0.9 above sinusoid, and -0.9 below sinusoid. (b) Output is perturbed randomly. Mean value of zero is coded with white, mean value of $0.9 is coded with gray, and mean value of -0.9 is coded with black.

disk of unit radius centered at x. For a given x, the classification y is chosen randomly as follows: x is "perturbed" by choosing a point z from the uniform distribution on Bl(x), and y is then assigned value 0.9 if z2 2 sin((7r/2)zl), and -0.9 otherwise. The resulting regression, E[y I XI], is depicted in Figure 3b, where white codes the value zero, gray codes the value f0.9, and black codes the value -0.9. Other values are coded by interpolation. (This color code has some ambiguity to it: a given gray level does not uniquely determine a value between -0.9 and 0.9. This code was chosen to emphasize the transition region, where y x 0.) The effect of the classification ambiguity is, of course, most pronounced near the "boundary" x2 = sin((7r/Z)xI). If the goal is to minimize mean-squared error, then the best response to a given x is E [ y I XI. On the other hand, the minimum error classifier will assign class "$0.9" or "-0.9" to a given x, depending on whether E[y I x] 2 0 or not: this is the decision function that minimizes the probability of misclassifying x. The decision boundary of the optimal classifier ({x : E[y I x] = 0)) is very nearly the original sinusoid x2 = sin((a/2)xl);it is depicted by the whitest values in Figure 3b. The training set for the ambiguous classification task was also constructed to have 50 examples from each class. This was done by repeated Monte Carlo choice of pairs (x,y), with x chosen uniformly from the rectangle [-6,6] x [-1.5,1.5] and y chosen by the above-described random


20

mechanism. The first 50 examples for which y = 0.9 and the first 50 examples for which y = -0.9 constituted the training set. In each experiment, bias, variance, and mean-squared error were evaluated by a simple Monte Carlo procedure, which we now describe. Denote by f(x;V) the regression estimator for any given training set V. Recall that the (squared) bias, at x, is just (E.DV(x;.O)l - Ely I xll2 and that the variance is

ED [(f(x;V)- E D ~ ( X ; ~ ) ] ) ~ ] These were assessed by choosing, independently, 100 training sets V', .O2# . .., Dl00 , and by forming the corresponding estimators f(x; Dl), ..., f(x;D,'"). Denote by f(x) the average response at x: f(x) = ( l / l O O ) EkEl f(x;@). Bias and variance were estimated via the formulas: Bias(x)

=

(f(x) - Ely

I XI)'

(Recall that E[y I x] is known exactly - see Fig. 3.) The sum, Bias(x) Variance(x) is the (estimated) mean-squared error, and is equal to

+

In several examples we display Bias(x) and Variance(x) via gray-level pictures on the domain [-6,6] x [-1.5,1.5]. We also report on integrated bias, variance, and mean-squared error, obtained by simply integrating these functions over the rectangular domain of x (with respect to the uniform distribution). The experiments with artificial data included both nearest-neighbor estimation and estimation by a feedforward neural network. Results of the experiments with the nearest-neighbor procedure are summarized in Figure 4. In both the deterministic and the ambiguous case, bias increased while variance decreased with the number of neighbors, as should be expected from our earlier discussion. In the deterministic case, the least meansquared error is achieved using a small number of neighbors, two or three; there is apparently, and perhaps not surprisingly, little need to control the variance. In contrast, the more biased eight- or nine-nearestneighbor estimator is best for the ambiguous task. Figures 5 through 7 demonstrate various additional features of the results from experiments with the ambiguous classification problem. Figure 5 shows the actual output, to each possible input, of two machines


21

Deterministic Classification

0.4,

I

I 1

2

3

4

2

3

4

5

6

I

8

9

10

5

6

I

8

9

10

0.5 0.4

0.3 0.2 0.1

0

1

# Neighbors

Figure 4: Integrated bias (os), variance (xs), and total error (+s) as functions of the number of neighbors in the nearest-neighborregression. trained on a typical sample of the data: Figure 5a is the first-nearestneighbor solution and Figure 5b is the two-nearest-neighbor solution. The actual training data are also displayed - see figure legend for interpretation. Average output of the five-nearest-neighbor machine (averaged over 100 independently chosen training sets - see earlier discussion) is depicted in Figure 6 (using the same color convention as for the regression). Compare this with the regression (Fig. 3b): there apparently is very little bias. Finally, in Figure 7, bias and variance are displayed as functions of the input x, both for the first-nearest-neighbor and the 10nearest-neighbor machines. Notice again the tradeoff. An analogous pattern emerged from the experiments with the feedforward neural network. In these experiments, the error-backpropagation algorithm (see equation 3.9) was run for 3,000 iterations, using t = 0.05, and initializing the weights as independent random variables chosen from the uniform distribution on [-0.2,0.2]. The results are summarized in Figure 8. The relatively unbiased, high-variance, 15-hidden-unit machine is best for the simpler deterministic classification task. For the ambiguous task, the more biased, single-hidden-unit machine is favored. Figure 9 shows

22


Figure 5: Nearest-neighbor estimates of regression surface shown in Figure 3b. Gray-level code is the same as in Figure 3. The training set comprised 50 examples with values +0.9 (circles with white centers) and 50 examples with values -0.9 (circles with black centers). (a) First-nearest-neighbor estimator. (b) Two-nearest-neighbor estimator.

Figure 6: Average output of 100 five-nearest-neighbor machines, trained on independent data sets. Compare with the regression surface shown in Figure 3b (gray-level code is the same) - there is little bias. the output of two typical machines, with five hidden units each. Both were trained for the ambiguous classification task, but using statistically independent training sets. The contribution of each hidden unit is partially revealed by plotting the line ~ 1 x t1 ~ 2 x 2 w3 = 0, where x = (xI,x2)is the input vector, w1and w2are the associated weights, and w3 is the threshold. On either side of this line, the unit’s output is a function solely of distance to the line. The differences between these two machines hint at the variance contribution to mean-squared error (roughly 0.13 - see Fig. 8). For the same task and number of hidden

+


23

Figure 7: Bias and variance of first-nearest-neighbor and 10-nearest-neighbor estimators, as functions of input vector, for regression surface depicted in Figure 3b. Scale is by gray levels, running from largest values, coded in black, to zero, coded in white. (a) Bias of first-nearest-neighbor estimator. (b) Variance of first-nearest neighbor estimator. (c) Bias of 10-nearest-neighbor estimator. (d) Variance of 10-nearest-neighborestimator. Overall, the effect of additional neighbors is to increase bias and decrease variance.

units, the bias contribution to error is relatively small (0.05, again from Fig. 8). This is clear from Figure 10, which shows the average output of the five-hidden-unit machine to the ambiguous classification task. The fit to the regression (Fig. 3b) is good, except for some systematic bias at the left and right extremes, and at the peaks and valleys of the sinusoid. Finally, with reference again to the ambiguous classification task,


24

Determiniistic Classification

0.5 I

"0

2

4

n

6

10

12

14

16

12

14

16

Ambiguous Classification

02

0 0s ---w

00

2

4

6

x

8

10

# Hidden Units

Figure 8: Integrated bias (os), variance (xs), and total error (+s) as functions of the number of hidden units in a feedforward neural network. Figure 11 shows bias and variance contributions to error for the onehidden-unit and the 15-hidden-unit machines. The pattern is similar to Figure 7 (nearest-neighbor machines), and reflects the tradeoff already apparent in Figure 8.

3.5.5 Experiments with Handwritten Numerals. The data base in these experiments consisted of 1200 isolated handwritten numerals, collected from 12 individuals by I. Guyon at the AT&T Bell Laboratories (Guyon 1988). Each individual provided 10 samples of each of the 10 digits, 0 , 1 , .. . ,9. Each sample was digitized and normalized to fit exactly within a 16 x 16 pixel array; it was then thresholded to produce a binary picture. A sampling of characters from this data base is displayed in the top four rows of Figure 12. The problem of recognizing characters in this set is rather easy, at least when compared to other widely available data sets, involving for example postal zip codes (see Le Cun et al. 1989)or courtesy numbers from checks. In fact, the data were collected with the intention of producing a more or less canonical set of examples: a standard "model" was chosen for each digit and the 12 individuals were asked to follow the model. However, our interest here was to demonstrate generic features of nonparametric estimation, and this turned out to be more easily done


25

Figure 9: Output of feedfonvard neural networks trained on two independent samples of size 100. Actual regression is depicted in Figure 3b, with the same gray-level code. The training set comprised 50 examples with values +0.9 (circles with white centers) and 50 examples with values -0.9 (circles with black centers). Straight lines indicate points of zero output for each of the five hidden units - outputs are functions of distance to these lines. Note the large variation between these machines. This indicates a high variance contribution to mean-squared error.

Figure 1 0 Average output of 100 feedforward neural networks with five hidden units each, trained on independent data sets. The regression surface is shown in Figure 3b, with the same gray-level code. with a somewhat harder problem; we therefore replaced the digits by a new, “corrupted,” training set, derived by flipping each pixel (black to white or white to black), independently, with probability 0.2. See the bottom four rows of Figure 12 for some examples. Note that this corruption does not in any sense mimic the difficulties encountered in

26


a

b

C

Figure 11: Bias and variance of single-hidden-unit and 15-hidden-unit feedforward neural networks, as functions of input vector. Regression surface is depicted in Figure 3b. Scale is by gray levels, running from largest values, coded in black, to zero, coded in white. (a) Bias of single-hidden-unit machine. (b)Variance of single-hidden-unit machine. (c) Bias of 15-hidden-unit machine. (d) Variance of 15-hidden-unit machine. Bias decreases and variance increases with the addition of hidden units. real problems of handwritten numeral recognition; the latter are linked to variability of shape, style, stroke widths, etc. The input x is a binary 16 x 16 array. We perform no ”feature extraction” or other preprocessing. The classification, or output, is coded via a 10-dimensional vector y = (yo,. . . ,y ~ ) where , yi = +0.9 indicates the digit ’5,” and yi = -0.9 indicates “not i.” Each example in the (noisy) data set is paired with the correct classification vector, which has one component with value +0.9 and nine components with values -0.9.


27

Figure 12: Top four rows: examples of handwritten numerals. Bottom four rows: same examples, corrupted by 20% flip rate (black to white or white to black).

To assess bias and variance, we set aside half of the data set (600 digits), thereby excluding these examples from training. Let us de, the renote these excluded examples by (x601, y601), . . . , (xI~M),~ I ~ o o ) and maining examples by (xl, yl), . . . , (x600,y6M)). The partition was such that each group contained 60 examples of each digit; it was otherwise random. Algorithms were trained on subsets of {(xl,yl)}fz, and assessed on {(xi,Yi)};2%. Each training set V consisted of 200 examples, 20 of each digit, chosen randomly from the set {(xi,yi)}z:. As with the previous data set, performance statistics were collected by choosing independent training sets, V1, . . . ,P , and forming the associated (vector-valued) estimators f(x; V'), . . ., f(x; p).The performances of the nearest-neighbor and Parzen-window methods were assessed by using M = 100 independent training sets. For the error-backpropagation procedure, which is much more computationally intensive, M = 50 training sets were generally used. Let us again denote by f(x) the average response at x over all training sets. For the calculation of bias, this average is to be compared with the

28


regression E[y I XI. Unlike the previous example (“artificial data”), the regression in this case is not explicitly available. Consider, however, the 600 noisy digits in the excluded set: {x1};2’&. Even with 20% corruption, the classification of these numerals is in most cases unambiguous, as judged from the bottom rows of Figure 12. Thus, although this is not quite what we called a “degenerate“ case in Section 2.1, we can approximate the regression at XI, 601 5 I 5 1200, to be the actual classification, y~: E[y I XI] x yl. Of course there is no way to display visually bias and variance as functions of x, as we did with the previous data, but we can still calculate approximate integrated bias, variance, and mean-squared error, using the entire excluded set, XbOl, . . . ,xl20o, and the associated (approximate) “regressions” y 6 0 l . . . . , ~ 1 2 0 0: Integrated Bias

Integrated Mean-Squared Error

M

=

1

1200

‘O0

f=601

1

1200

C

~

-

C

6oo /=60]

1f(xI)

-

yII2

1 M -

C If(xr; Dk) yl 12 -

k=l

The last estimate (for integrated mean-squared error) is exactly the sum of the first two (for integrated bias and integrated variance). Notice that the nearest-neighbor and Parzen-window estimators are both predicated on the assignment of a distance in the input (x) space, which is here the space of 16 x 16 binary images, or, ignoring the lattice structure, simply (0. 1}256. We used Hamming distance for both estimators. (In Section 6 , we shall report on experiments using a different metric for this numeral recognition task.) The kernel for the Parzenwindow experiments was the exponential: W ( x ) = exp{-lxl}, where 1x1 is the Hamming distance to the zero vector. We. have already remarked on the close relation between the kernel and nearest-neighbor methods. It is, then, not surprising that the experimental results for these two methods were similar in every regard. Figures 13 and 14 show the bias and variance contributions to error, as well as the total (mean-squared) error, as functions of the respective “smoothing parameters” - the number of nearest neighbors and the kernel ”bandwidth.” The bias/variance tradeoff is well illustrated in both figures. As was already noted in Sections 3.5.1 and 3.5.2, either machine can be used as a classifier. In both cases, the decision rule is actually asymptotically equivalent to implementing the obvious decision function: choose that classification whose 10-component coding is closest to the machine‘s output. To help the reader calibrate the mean-squared-error scale in these figures, we note that the values 1 and 2 in mean-squared error correspond, roughly, to 20 and 40% error rates, respectively.


1

TotalError

r:4,

29

A

Bias

Variance

k

Figure 13: Nearest-neighbor regression for handwritten numeral recognition. Bias, variance, and total error as a function of number of neighbors. The results of experiments with the backpropagation network are more complex. Indeed, the networks output to a given input x is not uniquely defined by the sole choice of the training set V and of a “smoothing parameter,” as it is in the nearest-neighbor or the Parzen-window case. As we shall now see, convergence issues are important, and may introduce considerable variation in the behavior of the network. In the following experiments, the learning algorithm (equation 3.9) was initialized with independent and uniformly distributed weights, chosen from the interval [-0.1,0.1]; the gain parameter, E, was 0.1. Figure 15 shows bias, variance, and total error, for a four-hiddenunit network, as a function of iteration trial (on a logarithmic scale). We observe that minimum total error is achieved by stopping the training after about 100 iterations, despite the fact that the fit to the training data


30

Variance

10

u)

25

Figure 14: Kernel regression for handwritten numeral recognition. Bias, variance, and total error as a function of kernel bandwidth.

continues to improve, as depicted by the curve labeled “learning.“ Thus, even with just four hidden units, there is a danger of “overfitting,“ with consequent high variance. Notice indeed the steady decline in bias and increase in variance as a function of training time. This phenomenon is strikingly similar to one observed in several other applications of nonparametric statistics, such as maximum-likelihood reconstruction for emission tomography (cf. Vardi et al. 1985; Veklerov and Llacer 1987). In that application, the natural “cost functional” is the (negative) log likelihood, rather than the observed mean-squared error. Somewhat analogous to gradient descent is the “ E - M algorithm (Dempster et al. 1976, but see also Baum 1972) for iteratively increasing likelihood. The reconstruction is defined on a pixel grid whose resolution plays a role similar to the number of hidden units. For sufficiently fine grids,


31

log(time)

Figure 15: Neural network with four hidden units trained by error backpropagation. The curve marked “Learning” shows the mean-squared error, on the training set, as a function of the number of iterations of backpropagation [denoted ”log(time)”]. The best machine (minimum total error) is obtained after about 100 iterations; performance degrades with further training.

the E-M algorithm produces progressively better reconstructions up to a point, and then decisively degrades. In both applications there are many solutions that are essentially consistent with the data, and this, in itself, contributes importantly to variance. A manifestation of the same phenomenon occurs in a simpler setting, when fitting data with a polynomial whose degree is higher than the number of points in the data set. Many different polynomials are consistent with the data, and the actual solution reached may depend critically on the algorithm used as well as on the initialization. Returning to backpropagation, we consistently found in the experi-


32

3

5

10

IS

m

# Hidden Units

Figure 16: Total error, bias, and variance of feedforward neural network as a function of the number of hidden units. Training is by error backpropagation. For a fixed number of hidden units, the number of iterations of the backpropagation algorithm is chosen to minimize total error.

ments with handwritten numerals that better results could be achieved by stopping the gradient descent well short of convergence; see, for example, Chauvin (1990) and Morgan and Bourlard (1990) who report on similar findings. Keeping in mind these observations, we have plotted, in Figure 16, bias, variance, and total mean-squared error as a function of the number of hidden units, where for each number of hidden units we chose the optimal number of learning steps (in terms of minimizing total error). Each entry is the result of 50 trials, as explained previously, with the sole exception of the last experiment. In this experiment, involving 24 hidden units, only 10 trials were used, but there was very little fluctuation around the point depicting (averaged) total error.


33

The basic trend is what we expect: bias falls and variance increases with the number of hidden units. The effects are not perfectly demonstrated (notice, for example, the dip in variance in the experiments with the largest numbers of hidden units), presumably because the phenomenon of overfitting is complicated by convergence issues and perhaps also by our decision to stop the training prematurely. The lowest achievable mean-squared error appears to be about 2. 4 Balancing Bias and Variance

This section is a brief overview of some techniques used for obtaining optimal nonparametric estimators. It divides naturally into two parts: the first deals with the finite-sample case, where the problem is to do one’s best with a given training set of fixed size; the second deals with the asymptotic infinite-sample case. Not surprisingly, the first part is a review of relatively informal “recipes,” whereas the second is essentially mathematical. 4.1 Automatic Smoothing. As we have seen in the previous section, nonparametric estimators are generally indexed by one or more parameters which control bias and variance; these parameters must be properly adjusted, as functions of sample size, to ensure consistency, that is, convergence of mean-squared error to zero in the large-sample-size limit. The number of neighbors k, the kernel bandwidth GT,and the number of hidden units play these roles, respectively, in nearest-neighbor, Parzenwindow, and feedforward-neural-network estimators. These “smoothing parameters” typically enforce a degree of regularity (hence bias), thereby “controlling” the variance. As we shall see in Section 4.2, consistency theQrems specify asymptotic rates of growth or decay of these parameters to guarantee convergence to the unknown regression, or, more generally, to the object of estimation. Thus, for example, a rate of growth of the number of neighbors or of the number of hidden units, or a rate of decay of the bandwidth, is specified as a function of the sample size N . Unfortunately, these results are of a strictly asymptotic nature, and generally provide little in the way of useful guidelines for smoothingparameter selection when faced with a fixed and finite training set V. It is, however, usually the case that the performance of the estimator is sensitive to the degree of smoothing. This was demonstrated previously in the estimation experiments, and it is a consistent observation of practitioners of nonparametric methods. This has led to a search for ”automatic” or “data-driven” smoothing: the selection of a smoothing parameter based on some function of the data itself. The most widely studied approach to automatic smoothing is **crossvalidation.” The idea of this technique, usually attributed to Stone (19741, is as follows. Given a training set V N = { ( x ~ , y ~.).., , (XN,YN)} and a

34

S . Geman, E. Bienenstock, and R. Doursat

"smoothing parameter" X, we denote, generically, an estimator by f(x;N,A, D N ) [see, for example, (3.6) with X = k or (3.7) with X = r r ] . Cross-validation is based on a "leave-one-out" assessment of estimation performance. Denote by V ( ' ) N1, 5 i 2 N, the data set excluding the ith observation ( x , , y ; ) : D(')N= { ( x l , y l ) , . . . , ( x i - l , y i - l ) , ( x i + ! , y i + l ) , . . ., ( x N , Y N ) } . Now fix A and form the estimator f ( x ; N - 1,X , D ( ' ) N ) , which is independent of the excluded observation ( x ; , y i ) . We can "test," or "cross-validate," on the excluded data: if f ( x , ; N -- 1,A, D$)) is close to yi, then there is some evidence in favor of f(x; N, A, D N )as an estimator of E[y I XI, at least for large N, wherein we do not expect f ( x ; N - 1,A , D i ' ) and f(x; N,A, DN) to be very different. Better still is the pooled assessment

The cross-validated smoothing parameter is the minimizer of A( A), which we will denote by A*. The cross-validated estimator is thenf(x;N, A', D N ) . Cross-validation is computation-intensive. In the worst case, we need to form N estimators at each value of A, to generate A(A), and then to find a global minimum with respect to A. Actually, the computation can often be very much reduced by introducing closely related (sometimes even better) assessment functions, or by taking advantage of special structure in the particular function f ( x ; N,A, D N )at hand. The reader is referred to Wahba (1984,1985) and OSullivan and Wahba (1985) for various generalizations of cross-validation, as well as for support of the method in the way of analytic arguments and experiments with numerous applications. In fact, there is now a large statistical literature on cross-validation and related methods (Scott and Terrell 1987; Hardle et al. 1988; Marron 1988; Faraway and Jhun 1990; Hardle 1990 are some recent examples), and there have been several papers in the neural network literature as well - see White (1988,19901, Hansen and Salamon (19901, and Morgan and Bourlard (1990). Computational issues aside, the resulting estimator, f ( x ; N,A*, D N ) ,is often strikingly effective, although some "pathological" behaviors have been pointed out (see, tor example, Schuster and Gregory 1981). In general, theoretical underpinnings of cross-validation remain weak, at least as compared to the rather complete existing theory of consistency for the original (not cross-validated) estimators. Other mechanisms have been introduced with the same basic design goal: prevent overfitting and the consequent high contribution of variance to mean-squared error (see, for example, Mozer and Smolensky 1989 and Karnin 1990 for some "pruning" methods for neural networks). Most of these other methods fall into the Bayesian paradigm, or the closely related method of regularization. In the Bayesian approach, likely regularities are specified analytically, and a priori. These are captured in a prior probability distribution, placed on a space of allowable input-to-output mappings ("machines"). It is reasonable to hope that estimators then


35

derived through a proper Bayesian analysis would be consistent; there should be no further need to control variance, since the smoothing, as it were, has been imposed a priori. Unfortunately, in the nonparametric case it is necessary to introduce a distribution on the infinite-dimensional space of allowable mappings, and this often involves serious analytical, not to mention computational, problems. In fact, analytical studies have led to somewhat surprising findings about consistency or the lack thereof (see Diaconis and Freedman 1986). Regularization methods rely on a “penalty function,” which is added to the observed sum of squared errors and promotes (is minimum at) ”smooth,” or “parsimonious,” or otherwise “regular” mappings. Minimization can then be performed over a broad, possibly even infinitedimensional, class of machines; a properly chosen and properly scaled penalty function should prevent overfitting. Regularization is very similar to, and sometimes equivalent to, Bayesian estimation under a prior distribution that is essentially the exponential of the (negative) penalty function. There has been much said about choosing the ”right” penalty function, and attempts have been made to derive, logically, informationbased measures of machine complexity from “first principles” (see Akaike 1973; Rissanen 1986). Regularization methods, complexity-based as well as otherwise, have been introduced for neural networks, and both analytical and experimental studies have been reported (see, for example, Barron 1991; Chauvin 1990). 4.2 Consistency and Vapnik-cervonenkis Dimensionality. The study of neural networks in recent years has involved increasingly sophisticated mathematics (cf. Barron and Barron 1988; Barron 1989; Baum and Haussler 1989; Haussler 1989b; White 1989, 1990; Amari 1990; Amari et al. 1990; Azencott 1990; Baum 1990a1, often directly connected with the statistical-inferenceissues discussed in the previous sections. In particular, machinery developed for the analysis of nonparametric estimation in statistics has been heavily exploited (and sometimes improved on) for the study of certain neural network algorithms, especially least-squares algorithms for feedforward neural networks. A reader unfamiliar with the mathematical tools may find this more technical literature unapproachable. He may, however, benefit from a somewhat heuristic derivation of a typical (and, in fact, much-studied) result: the consistency of leastsquares feedforward networks for arbitrary regression functions. This is the purpose of the present section: rather than a completely rigorous account of the consistency result, the steps below provide an outline, or plan of attack, for a proper proof. It is in fact quite easy, if somewhat laborious, to fill in the details and arrive at a rigorous result. The nontechnically oriented reader may skip this section without missing much of the more general points that we shall make in Sections 5 and 6. Previously, we have ignored the distinction between a random variable on the one hand, and an actual value that might be obtained on


36

making an observation of the random variable on the other hand. In this discussion of consistency, we will be more careful and adopt the usual convention of denoting random variables by upper-case letters, and their values by the corresponding lower-case letters. In the general regression problem, there are two random vectors, X and Y,which we might think of as the argument and corresponding value of some unknown, and possibly random, function. We observe N independent samples with values DN = {(x,.y,), .... (xN,YN)}. Based on this "training set," we wish to learn to accurately predict Y from the "input" X. Because there is nothing special about the mathematics of learning a uector relation per se, we will simplify our discussion by treating X and Y as scalars X and Y. We will continue to use mean-squared error,

to evaluate the accuracy of a function f as a predictor of Y. We recall (see 2.2) that

E [(Y - f ( X ) ) 2 ] = E [ ( f ( X )- E[Y I XI)']

+ E [(Y - E[Y I XI)']

(4.1)

where E [ . ] means expectation (averaging) with respect to the joint distribution on X and Y. Since the second term of the right-hand side does not involve f , we will, as usual, adopt E [ c f ( X ) - E[Y I in evaluating f as a predictor of Y from X . The actual estimator is drawn from some class of functions that we will denote by FM.The primary example that we have in mind is a class of feedforward networks with M hidden units. Depending on details about the distribution of X and Y, and about the architecture of machines in 3 M , it may be necessary to restrict the magnitudes of the "synaptic weights," for example, IW,~I 5 & for all weights {wij}, where [jM 1 00 as the number of hidden units, M, is increased to infinity. Given M and a training set DN of size N , we define now our estimator f ( x ; N , M, DN)to be the best fit to the data within the class FM: N

(4.2)

Of course, actually getting this solution may be a very difficult, perhaps even intractable, computational problem. Our discussion of consistency necessarily concerns the true minimizer of 4.2, rather than the actual output of an algorithm designed to minimize 4.2, such as error backpropagation. In practice, there are serious convergence issues for such algorithms, with, unfortunately, very few analytical tools to address them. Also, it may be that 4.2 has multiple global minima. This is only a technical complication, as we could replace f ( x ; N,M, D N ) ,in what follows, by the set of minimizers. We shall therefore assume that the minimization is unique.

NeuraI Networks and the Bias/Variance Dilemma

37

For large N , we would hope to find f ( x ;N , M , D N ) ”close” to E[Y I X I . Let us denote by f M the best that can be done within the family FM:

j , = arg min E f€FM

[v(x) E[Y I XI)’] -

(4.3)

Of course, if FM is ”too small,” or at least if E[Y 1 x] is not well approximated by any element in FM,then f ( x ; N , M , D N ) cannot be a very good estimator of the regression. But later we will make FM ”large” by taking M 1‘ 00. For now, let us compare f ( x ;N , M, D N )to f M , the best solution available in .%. We will argue that, under appropriate conditions, f ( x ;N , M, DN) is essentially as good as ~ M ( x ) . Notice that for any fixed f E FM,the N numbers bi -f(xi)]’, 1 5 i 5 N, are independent and identically distributed (”i.i.d.”) observations of the random variable [Y - f(X)]’. Therefore, we expect that ( l / N )&bi f(xi)]’ is well approximated by E[(Y-f(X))’] (the “law of large numbers,” or “LLN). With this in mind, we can proceed to bound the meansquared error of f ( X ; N , M , V N ) :

=

E [ ( f ( X ; N , M , D N-) E[Y I XI)‘] (as before - see 4.1) E [ (Y - f ( X ; N , M , a,))’]

-E [ ( Y - E[Y I XI)’] N N

(LLN)

< (by defn. - see 4.2) N

(LLN)

=

(again, see 4.1)

-

l N -C [yi -f(Xi;N,M,DN)]’ N i=l

-E [(Y - E[Y t XI)’] l N

[ y j -fM(Xi)]’ - E [ ( Y - E[Y I XI)’] N j=l E [(Y - f m ) ’ ] - E [(Y - E[Y I XI)’] E [ ( ~ M (X E[Y ) I XI)’]

-

minE [ ( f ( X )- E[Y I XI)’]

fE3M

We thus obtained the desired result: f ( x ;N , M, DN)is asymptotically optimal in the class FM. Although this reasoning is essentially correct, we are still two fairly big steps away from a rigorous consistency argument. The first gap to be filled in has to do with the bias of the function fM: E[(f,,(X)- E[Y 1 XI)’] is not likely to be zero, hence we have no reason to believe that, even as N + 00, E [ V ( X ; N , M , D N) E[Y I XI)’] -+ 0. In other words, since F M may not (probably does not) contain E[Y 1 x ] ,there is a residual bias. This is remedied by adding more hidden units: we take M = M N 1‘ co as N -+ co. The reason why this indeed eliminates residual bias is that the class FM is asymptotically (M -+ 00) dense in the space of all “reasonable” functions. (This is an often-quoted result in the neuralmodeling literature. Of course it depends on details about architecture


38

and the particular “neuronal response function” used - see Barron 1989; Cybenko 1989; Funahashi 1989; Hornik et al. 1989; Hartman et al. 1990.) That is to say, for any (measurable) E[Y I X I , there exists a sequence gM E F~ such that E[(gM(X)- E[Y I XI)’] + o as M + m. In particular, the sequence f~ defined in 4.3 will have this property. The second problem is more difficult to solve, and is moreover confounded by the evident necessity of taking M = MN T 00. The difficulty lies in our (first) use of the law of large numbers. It is true that

f(xi)]’ -+ E [ (Yas N + 00, for fixed functions f E FM.This is so because, as we have noted, [yi - f(xi)]’ are i.i.d. realizations of a random variable. However, the function f ( x ;N, M , D N ) depends on all of the xi’s and yi’s; therefore, the numbers { (yi - f(xi;N , M , DN))2}are tzof i.i.d. observations. They are coupled by the minimization procedure that defines f ( x ; N , M , DN), and this coupling introduces dependence. One rather crude solution consists in proving a uniform law of large numbers: (4.4) Then, in particular, lim N-m

l N -

N i=l

[yi

-

f ( x i ;N , M , .ON)]’ = E [(Y- f(X))’]

Recall however that we must take M = M N T 03 to eliminate any residual bias. Therefore, what we actually need is a result like (4.5)

, be increasing with N , so In most cases, the class of machines, F M ~will 4.5 is stronger than 4.4. But it is not much more difficult to prove. In fact, it actually follows from 4.4, provided that 4.4 can be established along with bounds on the rate of convergence. One then specifies a sequence M = MN, increasing to infinity sufficiently slowly, so that 4.5 is true as well. We will forgo the details of making this extension (from 4.4 to 4.5), and concentrate instead on the fundamental problem of establishing a uniform LLN, as in 4.4. Recall that for every element f E FM we have 1 / N CEIIyi - f(xi)]’ + E[(Y-f(X))’] as N + 00, by the ordinary LLN. In this case, moreover, we know essentially everything there is to know about the rate of convergence. One way to prove 4.4 is then to cover FM with ”small balls” (see, for example, Grenander 1981; Geman and Hwang 1982). We judiciously


39

choose fl,f*, . . . , f ~E 3 ’ such that every other f € F ,is close to one of these (inside one of L small balls centered at the fi’s). Since convergence for each fi, 1 5 i 5 L, always guarantees uniform convergence for the finite set (fi}f.=l, and since all other f are close to one of these fi’s, 4.4 is shown to be ”nearly true.” Finally, taking L cc (so that the balls can get smaller), 4.4 can be rigorously established. The modern approach to the problem of proving 4.4 and 4.5 is to use the Vapnik2ervonenkis dimension. Although this approach is technically different, it proceeds with the same spirit as the method outlined above. Evidently, the smaller the set 3 M , the easier it is to establish 4.4, and in fact, the faster the convergence. This is a direct consequence of the argument put forward in the previous paragraph. The Vapnikeervonenkis approach “automates” this statement by assigning a size, or dimension, to a class of functions. In this case we would calculate, or at least bound, the size or Vapnikzervonenkis dimension of the class of functions of (x,y) given by --$

((Y -

(4.6)

For a precise technical definition of Vapnikzervonenkis dimension (and some generalizations), as well as for demonstrations of its utility in establishing uniform convergence results, the reader is referred to Vapnik (1982), Pollard (1984), Dudley (1987), and Haussler (1989a). Putting aside the details, the important point here is that the definition can be used constructiuely to measure the size of a class of functions, such as the one defined in 4.6. The power of this approach stems from generic results about the rate of uniform convergence (see, e.g., 4.41, as a function of the Vapnik-eervonenkis dimension of the corresponding function class, see, e.g., 4.6. One thereby obtains the desired bounds for 4.4, and, as discussed above, these are rather easily extended to 4.5 by judicious choice of M = M N T ~ 0 . Unfortunately, the actual numbers that come out of analytical arguments such as these are generally discouraging: the numbers of samples needed to guarantee accurate estimation are prohibitively large. (See, for example, the excellent paper by Haussler 1989b, in which explicit upper bounds on sample sizes are derived for a variety of feedforward networks.) Of course these analytical arguments are of a general nature. They are not dedicated to particular estimators or estimation problems, and therefore are not “tight”; there may be some room for improvement. But the need for large sample sizes is already dictated by the fact that we assume essentially no a priori information about the regression E[Y I It is because of this assumption that we require uniform convergence results, making this a kind of ”worst case analysis.” This is just another view of the dilemma: if we have a priori information about E[Y I x] then we can employ small sets FM and achieve fast convergence, albeit at the risk of large bias, should E[Y I x ] in fact be far from F’.

XI.


40

We end this section with a summary of the consistency argument: Step 1. Check that the class FM is rich enough, that is, show that the - E[Y 1 -+ 0 as M + 00. sequence f~ (see 4.3) is such that E [ ( ~ M Step 2. Establish a uniform LLN:

together with a (probabilistic) rate of convergence (eg., with the help of Vapnikxervonenkis dimensionality). Step 3. Choose M = MN T M replaced by MN.

00

sufficiently slowly that 4.7 is still true with

Step 4. Put together the pieces:

5 interpretation and Relevance to Neural Networks Let us briefly summarize the points made thus far. We first remarked that the goals one is trying to achieve with artificial neural networks, particularly of the feedforward layered type (multilayer perceptrons), generally match those of statistical inference: the training algorithms used to adjust the weights of such networks - for example, the error-backpropagation algorithm - can be interpreted as algorithms for statistical inference. We further mentioned that although learning precisely consists of adjusting a collection of parameters, namely the “synaptic weights” of the network,

Neural Networks and the BiasNariance DiIemma

41

such networks with their associated learning algorithms really belong to the class of nonparametric inference schemes, also called model-free methods. Nonparametric methods may be characterized by the property of consistency: in the appropriate asymptotic limit they achieve the best possible performance for any learning task given to them, however difficult this task may be. We saw that in many tasks performance is adequately measured by mean-squared error, and that optimal performance is achieved by the conditional mean, or "regression" of the output on the input: this is, among all possible functions, the one that minimizes mean-squared error. We also saw that mean-squared error can be decomposed into a bias term and a variance term. Both have to be made small if we want to achieve good performance. The practical issue is then the following: Can we hope to make both bias and variance "small," with "reasonably" sized training sets, in "interesting" problems, using nonparametric inference algorithms such as nearest-neighbor, CART, feedforward networks, etc.? It is one of the main goals of the present paper to provide an answer to this question, and we shall return to it shortly. Let us, however, immediately emphasize that the issue is purely about sample size, and quite distinct from implementation considerations such as the speed of the hardware, the parallel versus serial or analog versus digital type of machine, or the number of iterations required to achieve convergence in the case of the backpropagation algorithm. 5.1 Neural Nets and Nonparametric Inference for Machine Learning and Machine Perception Tasks. We mentioned that the focus of most connectionist models is the problem of inferring relationships among a set of observable variables, from a collection of examples called a training set. This is also the focus of the statistical sciences, so it is not surprising that statistical tools are increasingly exploited in the development and analysis of these kinds of neural models (Lippmann 1987; Barron and Barron 1988; Gallinari et al. 1988; Barron 1989; Haussler 1989a; Tishby et al. 1989; White 1989; Amari ef al. 1990; Baum 1990b; Hinton and Nowlan 1990). Thus, the perceptron (Rosenblatt 1962) and other adaptive pattern classifiers (e.g., Amari 1967) are machines for computing linear decision boundaries; the "Brain State in a Box" model of categorical perception (Anderson ef al. 1977) is related to factor analysis; Boltzmann Machines (Ackley ef al. 1985; Hinton and Sejnowski 1986) compute (approximate) maximum-likelihood density estimators; and backpropagation networks realize an algorithm for nonparametric least-squares regression. Backpropagation networks can also be trained to achieve transformations related to principal-component analysis (Bourlard and Kamp 1988, Baldi and Hornik 1989). A good state-of-the-art statistical method for highdimensional data analysis is projection pursuit (Friedman and Stuetzle 1981, Huber 1985). It may then be a good strategy to start from a statistical method such as this and to look for neural-like realizations of it,

42


thereby suggesting efficient parallel, and possibly even ultrafast, analog, implementations. Examples of networks based upon projection pursuit can be found in Intrator (1990) and Maechler ef ul. (1990). Modern nonparametric statistical methods, and hence many recent neural models, are important tools for wide-ranging applications. Two rather natural applications were discussed in Section 2: the General Motors foam casting problem and the problem of evaluating loan applications. There are no doubt many other applications in economics, medicine, and more generally in modern data analysis. Nevertheless, the enthusiasm over neural modeling is mostly about different kinds of problems. Indeed, anybody remotely connected to the field knows, if only from his or her mail, that much more is expected from neural networks than making additions to the statistician’s toolbox. The industrial and military scientists, and to some extent academia, are poised for the advances in machine intelligence that were once anticipated from artificial intelligence. There is, for example, much enthusiasm anticipating important advances in automatic target recognition, and more generally in invariant object recognition. In speech processing there have been successes in isolated phoneme recognition (Waibel et al. 1988; Lippmann 1989) and there is a suggestion of neural networks (or other nonparametric methods) as good “front-ends” for hidden Markov models (Bourlard and Wellekens 1990), and, beyond this, of advances in continuous speech recognition via trained neural networks, avoiding the difficult task of estimating parameters in complex hidden Markov models. Further, there is the hope of building expert systems without ”engineering knowledge”: neural networks can learn by example. In this regard, evaluating loans is indeed a modest start. Typical applications of expert systems would involve many more variables, and would need to predict more than just a small number of possible outcomes. Finally, it should not be forgotten that from their inception neural networks were also, if not chiefly, meant to contribute to the understanding of real brains. The debate about their adequacy as models of cognition is probably more intense now than ever (Fodor and Pylyshyn 1988; Smolensky 1988). From at least one point of view the optimism about neural networks is well-founded. The consistency theorems mentioned in Sections 2 and 4 guarantee a (suitably formulated) optimal asymptotic performance. Layered networks, Boltzmann Machines, and older methods like nearestneighbor or window estimators, can indeed form the basis of a trainable, “from scratch,” speech recognition system, or a device for invariant object recognition. With enough examples and enough computing power, the performance of these machines will necessarily approximate the best possible for the task at hand. There would be no need for preprocessing or devising special representations: the ”raw data” wouId do. Is this hope indeed realistic? Also, is there any reason to believe that neural networks will show better performances than other nonparametric


43

methods with regard to difficult problems that are deemed to require some form of "intelligence"? As we have seen, the question really boils down to the following: Can training samples be large enough to make both bias and variance small? To get a feeling about this issue, consider for a moment the problem of recognizing all nonambiguous handwritten characters. This is somewhat ill-defined, but we mean, roughly, the following. The input X is a digitized raw image of a single segmented character, handwritten, drawn, or etched, using any kind of tool or process, on any kind of material. The distribution of inputs includes various styles of writing, various sizes, positions, and orientations of the character in the image, various widths of stroke, various lighting conditions, various textures, shadings, etc. Images may moreover include substantial "noise" and degradations of various sorts. It is assumed that in spite of the variability of the data, the conditional distribution P(Y I X) is degenerate for all X: a human observer provides a perfectly nonambiguous labeling Y for any X drawn from this distribution. By definition, an optimal classifier for this task achieves zero mean-squared error, since the labeling Y is a deterministic function of the input X. This general problem is certainly more difficult than the hardest of character recognition problems actually solved today, for example, by neural methods (cf. Le Cun et al. 1989; Martin and Pittman 1991). On the other hand, insofar as this problem is well defined, consistency theorems apply to it and guarantee optimal performance, in this case zero-error classification. One should thus be able to devise a sequence of machines that would, in the limit, when trained on enough data drawn from the given distribution of (X,Y), perform the task just as accurately as the human observer, that is, never fail, since we assumed nonambiguous labeling. In reality, the reason why we are still quite far from building optimal performance machines for this problem is the wide gap between the theoretical notion of consistency - an asymptotic property - and conditions realized in practice. As we have seen in Section 4, consistency requires that the size of the training set grows to infinity, and that the algorithm S i ~ ~ Z ~ ~ adapts n e ~ ~itself s Zto~ larger and larger training sets. In essence, the machine shouId become more and more versatile, that is, eliminate all biases. On the other hand, elimination of bias should not happen too fast, lest the machine become dedicated to the idiosyncrasies of the training examples. Indeed, we have seen that for any finite-size training set the price to pay for low bias is high variance. In most cases, as we have seen in Section 3, there is a "smoothing parameter" whose value can be adjusted to achieve the very best bias/variance tradeoff for any fixed size of the training set. However, even with the best compromise, an estimator can still be quite far from optimal. Only when the size of the training set literally grows to infinity, can one eliminate at the same

44


time both bias and variance. This justifies the term "dilemma," and the consequence is prohibitively slow convergence. In practice, the size of the training set for our "general" character recognition problem will always be considerably smaller than would be required for any nonparametric classification scheme to meaningfully approximate optimality. In other words, for complex perceptual tasks such as this, a "sufficiently large training set" exists only in theory. 5.2 Interpolation and Extrapolation. The reader will have guessed by now that if we were pressed to give a yes/no answer to the question posed at the beginning of this chapter, namely: "Can we hope to make both bias and variance 'small,' with 'reasonably' sized training sets, in 'interesting' problems, using nonparametric inference algorithms?" the answer would be no rather than yes. This is a straightforward consequence of the bias/variance "dilemma." Another way to look at this stringent limitation is that if a difficult classification task is indeed to be learned from examples by a general-purpose machine that takes as inputs raw unprocessed data, then this machine will have to "extrapolate," that is, generalize in a v e y nontrivial sense, since the training data will never "cover" the space of all possible inputs. The question then becomes: What sorts of rules do conventional algorithms follow when faced with examples not suitably represented in the training set? Although this is dependent on the machine or algorithm, one may expect that, in general, extrapolation will be made by "continuity," or "parsimony." This is, in most cases of interest, not enough to guarantee the desired behavior. For instance, consider again the sinusoid-within-rectangle problem discussed in Section 3. Suppose that after training the machine with examples drawn within the rectangle, we ask it to extrapolate its "knowledge'' to other regions of the plane. In particular, we may be interested in points of the plane lying far to the left or to the right of the rectangle. If, for instance, the k-nearest-neighbor scheme i s used, and if both k and the size of the training set are small, then it is fairly easy to see that the extrapolated decision boundary will be very sensitive to the location of training points at the far extremes of the rectangle. This high variance will decrease as k and the sample size increase: eventually, the extrapolated decision boundary will stabilize around the horizontal axis. Other schemes such as Parzen windows and layered networks will show similar behavior, although the details will differ somewhat. At any rate, it can be seen from this example that extrapolation may be to a large extent arbitrary. Using still the same example, it may be the case that the number of training data is too small for even a good interpolation to take place. This will happen inevitably if the size of the training sample is kept constant while the number of periods of the sinusoid in the rectangle, that is, the complexity of the task, is increased. Such a learning task will defeat general nonparametric schemes. In reality, the problem has now become

Neural Networks and the Bias /Variance Dilemma

45

extrapolation rather than interpolation, and there is no a priori reason to expect the right type of extrapolation. One recourse is to build in expectations: in this particular case, one may favor periodic-type solutions, for.instance by using estimators based on Fourier series along the xaxis. Evidently, we are once more facing the bias/variance dilemma: without anticipating structure and thereby introducing bias, one should be prepared to observe substantial dependency on the training data. If the problem at hand is a complex one, such as the high-frequency twodimensional sinusoid, training samples of reasonable size will never adequately cover the space, and, in fact, which parts are actually covered will be highly dependent on the particular training sample. The situation is similar in many real-world vision problems, due to the high dimensionality of the input space. This may be viewed as a manifestation of what has been termed the ”curse of dimensionality” by Bellman (1961). The fundamental limitations resulting from the bias-variance dilemma apply to all nonparametric inference methods, including neural networks. This is worth emphasizing, as neural networks have given rise in the last decade to high expectations and some surprising claims. Historically, the enthusiasm about backpropagation networks stemmed from the claim that the discovery of this technique allowed one, at long last, to overcome the fundamental limitation of its ancestor the perceptron, namely the inability to solve the ”credit (or blame) assignment problem.” The hope that neural networks, and in particular backpropagation networks, will show better generalization abilities than other inference methods, by being able to develop clever “internal representations,” is implicit in much of the work about neural networks. It is indeed felt by many investigators that hidden layers, being able to implement any nonlinear transformation of the input space, will use this ability to “abstract the regularities” from the environment, thereby solving problems otherwise impossible or very difficult to solve. In reality, the hidden units in a layered network are a nonlinear device that can be used to achieve consistency like many others. There would seem to be no reason to expect sigmoid functions with adaptive weights to do a significantly better job than other nonlinear devices, such as, for example, gaussian kernels or the radial basis functions discussed by Poggio and Girosi (1990). Consistency is an asymptotic property shared by all nonparametric methods, and it teaches us all too little about how to solve difficult practical problems. It does not help us out of the bias/variance dilemma for finite-size training sets. Equivalently, it becomes irrelevant as soon as we deal with extrapolation rather than interpolation. Unfortunately, the most interesting problems tend to be problems of extrapolation, that is, nontrivial generalization. It would appear, then, that the only way to avoid having to densely cover the input space with training examples - which is unfeasible in practice - is to prewire the important generalizations.

46


In light of these rather pessimistic remarks, one is reminded of our earlier discussion (see Section 2.3) of some successful applications of nonparametric methods. Recall that General Motors, for example, made an important reduction in the scrap rate of Styrofoam castings after building a nonparametric classifier based on the CART procedure. The input, or feature, space comprised 80 process variables. It was not reasonable to hope that the 470 training examples would meaningfully cover the potentially achievable settings of these variables. Certainly, extrapolation to regions not represented would have been hazardous, at least without a believable model for the relationship between castings and process variables. But this was not a problem of extrapolation; the goal was not to learn the relationship between casting success and process variables per se, but rather to identify an achievable range of process variables that would ensure a high likelihood of good castings. With this more modest goal in mind, it was not unreasonable to anticipate that a data set with substantial variation in the settings of the process variables would help locate regions of high likelihood of success. Also discussed in Section 2.3 was the application of a neural network learning system to risk evaluation for loans. In contrast to the Styrofoam casting problem, there is here the luxury of a favorable ratio of trainingset size to dimensionality. Records of many thousands of successful and defaulted loans can be used to estimate the relation between the 20 or so variables characterizing the applicant and the probability of his or her repaying a loan. This rather uncommon circumstance favors a nonparametric method, especially given the absence of a well-founded theoretical model for the likelihood of a defaulted loan. 6 Designing Bias If, as we have seen in the previous chapter, the asymptotic property of consistency does not help us much in devising practical solutions to the more substantial’problems of machine learning and machine perception, how could one improve on the capabilities of existing algorithms? It is sometimes argued that the brain is a proof of existence of near-optimal methods that do not require prohibitively large training samples. Indeed, in many cases, we do learn with remarkable speed and reliability. Language acquisition offers a striking example: children often memorize new words after hearing them just once or twice. Such ”one-shot” learning has apparently little to do with statistical inference. Without going to such extremes, does not the simple observation that quick and reliable perceptual learning exists in living brains contradict the conclusions of the previous chapter? The answer is that the bias/variance dilemma can be circumvented if one is willing to give up generality, that is, purposefully introduce bias. In this way, variance can be eliminated, or significantly reduced. Of


47

course, one must ensure that the bias is in fact harmless for the problem a t hand: the particular class of functions from which the estimator is to be drawn should still contain the regression function, or at least a good approximation thereof. The bias will then be harmless in the sense that it will contribute significantly to mean-squared error only if we should attempt to infer regressions that are not in the anticipated class. In essence, bias needs to be designed for each particular problem. For a discussion of this point in a psychological perspective, and some proposals for specific regularities that living brains may be exploiting when making nontrivial generalizations, see Shepard (1989). Similar suggestions have been made by several other authors in the specific context of neural networks (cf. Anderson et al. 1990). Indeed, it has been found that for many problems a constrained architecture can do better than a general-purpose one (Denker et al. 1987; Waibel et al. 1988; Le Cun et al. 1989; Solla 1989). This observation has a natural explanation in terms of bias and variance: in principle, the synaptic weights in a "generalist" neural network should converge to a satisfactory solution if such a solution exists, yet in practice this may be unfeasible, as a prohibitive number of examples are required to control the variance (not to mention the computational problem of identifying good minima in "weight space"). In some cases, a set of simple constraints on the architecture, that is, a bias, will essentially eliminate the variance, without at the same time eliminating the solution from the family of functions generated by the network. A simple example of such a situation is the socalled contiguity problem (Denker et ul. 1987; Solla 1989). In a statistical physics perspective, introducing bias may also be viewed as a means of decreasing an appropriately defined measure of entropy of the machine (Carnevali et al. 1987; Denker et al. 1987; Tishby et al. 1989; Schwartz et al. 1990). In many cases of interest, one could go so far as to say that designing the right biases amounts to solving the problem. If, for example, one could prewire an invariant representation of objects, then the burden of learning complex decision boundaries would be reduced to one of merely storing a label. Such a machine would indeed by very biased; it would, in fact, be incapable of distinguishing among the various possible presentations of an object, up-side-up versus up-side-down, for example. This is, then, perhaps somewhat extreme, but the bias/variance dilemma suggests to us that strong a priori representations are unavoidable. Needless to say, the design of such representations, and other biases that may be essential, for example, to auditory perception or to other cognitive tasks, is a formidable problem. Unfortunately, such designs would appear to be much more to the point, in their relevance to real brains, than the study of nonparametric inference, whether neurally inspired or not. This suggests that the paradigm of near tabula rasa learning (i.e., essentially unbiased learning), which has been so much emphasized in

48


the neural-computing literature of the last decade, may be of relatively minor biological importance. It may still be a good idea, for example, for the engineer who wants to solve a task in machine perception, to look for inspiration in living brains. In the best of all cases, this could allow him or her to discover the nature of the biases ”internalized” during the course of phylogenetic and ontogenetic evolution. However, the hope that current connectionist networks already inherit the biases of real brains from the mere fact that they are built from “brain-like” processing elements seems farfetched, to say the least. Indeed, one could reasonably question the extent to which connectionist networks adequately reflect the basic principles of anatomy and physiology of living brains (see, e.g., Crick 1989). 6.1 Further Experiments with Handwritten Numerals. We have performed some further experiments on handwritten-numeral recognition for the purpose of illustrating the possible advantages of forgoing generality in a challenging inference problem, and concentrating, instead, on the design of appropriate bias. These experiments were inspired from a theory of neural coding (von der Malsburg 1981,1986; von der Malsburg and Bienenstock 1986) that emphasizes the role of accurate temporal correlations across neuronal populations. This theory leads to an alternative notion of distance between patterns (sensory stimuli) that accommodates, a priori, much of the invariance that would otherwise need to be learned from examples (Bienenstock and Doursat 1991). In brief, von der Malsburg and Bienenstock argue that living brains, exploiting the fine temporal structure of neural activity, are well-suited to the task of finding near-optimal, that is, topology-preserving, maps between pairs of labeled graphs. In a simple example, such graphs could be the nearest-neighbor, black-white, graphs defined by the 16 x 16 binary character arrays used in our handwritten-numeral experiments. These map computations give rise to a natural metric, which measures the extent to which one pattern needs to be distorted to match a second pattern. Shifts, for example, cost nothing, and two patterns that differ only by a shift are therefore deemed to be zero-distance apart. Small distortions, such as a stroke extending at a slightly different angle in one character than in another, incur only small costs: the distance between two such patterns is small. A very similar notion of distance has arisen in computer vision, via so-called deformable templates, for the purpose of object recognition (see Fischler and Elschlager 1973; Widrow 1973; Burr 1980; Bajcsy and Kovacic 1989; Yuille 1990). For applications to image restoration and pattern synthesis, as well as to recognition, see Grenander et al. (1990), Knoerr (1988), and Amit ef al. (1991). Given a pair of 16 x 16 binary images, x and x’, let us denote by m ( x ,x’) the “deformation metric” suggested by the von der MalsburgBienenstock theory. A formal definition of the metric m, as well as details behind the biological motivation and a more extensive discussion of


49

experiments, can be found in Bienenstock and Doursat (1989, 1991) and in Buhmann et al. (1989). For our purposes, the important point is that m ( x ,x’) measures the degree of deformation necessary to best match the patterns represented by x and x’. Recall that the Parzen-window and nearest-neighbor methods require, for their full specification, some metric on the range of the input, x, and recall that we used the Hamming distance in our earlier experiments (see Section 3.5.5). Here, we experiment with k-nearest-neighbor estimation using the graph-matching metric, m, in place of Hamming distance. Of course by so doing we introduce a particular bias: small changes in scale, for example, are given less importance than when using Hamming distance, but this would seem to be highly appropriate for the task at hand. Figure 17 summarizes the results of these experiments. The task was the same as in Section 3.5.5, except that this time we added no noise to the discretized images of handwritten numerals. Examples of the uncorrupted numerals used for these experiments are shown in the top four rows of Figure 12. As in our previous experiments, the y-axis indicates mean-squared error (see Section 3.5.5), which can be used to approximate the percent misclassification by the rough rule of thumb: percentage misclassification = 20 x mean-squared error. There are three curves in Figure 17. Two of these give results from experiments with the k-nearestneighbor estimator: one using the graph-matching metric and the other, for comparison, using the Hamming distance. Also for comparison, a third curve gives results from experiments with the backpropagation network described in Section 3.5.3. As we have noted earlier, the neural net performance does not necessarily improve with each learning iteration. To make the most of the feedforward neural network, we have again used, for each number of hidden units, the optimal number of iterations (see Section 3.5.5). Note that the x-axis now serves two purposes: it indicates the value of k for the two k-nearest-neighbor curves, but also the number of hidden units for the neural network estimator; there is no correspondence between the two scales, the only purpose of this simultaneous display being the comparison of the performances in each of the three methods. We first observe that the best performances of the two nonparametric estimators (k-nearest-neighbor with Hamming distance and backpropagation network) are almost identical. This comes as no surprise since we observed similar results in our experiments with the noisy data in Section 3. The result of interest here is that when the image space is equipped with the graph-matching distance m, rather than the usual Hamming distance, the performance of the k-nearest-neighbor classifier improves significantly. More experiments, including other nonparametric schemes (Parzen windows as well as various neural networks) applied either to the raw image or to an image preprocessed by extraction of local features, confirm that the use of the graph-matching distance yields significantly better results on this task (Bienenstock and Doursat 1991).


50

backpropgation --%

H : : ’ 5

IS

I0

20

k, H.U.

Figure 1 7 Classification of handwritten numerals: performance as a function of the number k of neighbors in a k-nearest-neighbor estimator (curves marked “Hamming” and “elastic matching”) and the number of hidden units in a feedforward neural network trained by error backpropagation (curve marked “backpropagation”). The two curves representing k-nearest-neighbor estimation are the results of experiments with two different measures of distance, and hence two notions of “neighbor.” The first is ordinary Hamming distance (the patterns are binary); the second is a measure of the deformation necessary to bring one pattern into another (the “elastic matching” metric).

Evidently then, the metric arising from graph matching is more suitable for the problem at hand than the straightforward Hamming distance, arising from the pixel-array representation. By adopting a different representation, we have introduced a very significant bias, thereby achieving a better control of the variance. We believe, more generally, that adopting


51

an appropriate data representation is an efficient means for designing the bias required for solving many hard problems in machine perception. This view is of course shared by many authors. As Anderson and Rosenfeld (1988) put it: “A good representation does most of the work.”

7 Summary To mimic substantial human behavior such as generic object recognition in real scenes - with confounding variations in orientation, lighting, texturing, figure-to-ground separation, and so on - will require complex machinery. Inferring this complexity from examples, that is, learning it, although theoretically achievable, is, for all practical matters, not feasible: too many examples would be needed. Important properties must be built-in or “hard-wired,” perhaps to be tuned later by experience, but not learned in any statistically meaningful way. These conclusions and criticisms are probably shared by many authors. They can perhaps be argued most convincingly from the point of view of modern statistical theory, especially the theory of nonparametric inference. We have therefore presented a tutorial on the connection between nonparametric inference and neural modeling as it stands today, and we have used the statistical framework, together with some simple numerical experiments, to argue for the limitations of learning in neural modeling. Of course most neural modelers do not take tabula rasa architectures as serious models of the nervous system; these are viewed as providing a mere starting point for the study of adaptation and self-organization. Such an approach is probably meant as a convenience, a way of concentrating on the essential issue of finding neurally plausible and effective learning algorithms. It strikes us, however, that identifying the right “preconditions” is the substantial problem in neural modeling. More specifically, it is our opinion that categorization must be largely built in, for example, by the use of generic mechanisms for representation, and that identifying these mechanisms is at the same time more difficult and more fundamental than understanding learning per se.

Acknowledgments We are indebted to Isabelle Guyon and the AT&T Bell Laboratories for providing the data set of handwritten numerals used in our experiments, and to Nathan Intrator for a careful reading of the manuscript and many useful comments. S. G. was supported by Army Research Office Contract DAAL03-86-K-0171 to the Center for Intelligent Control Systems, National Science Foundation Grant DMS-8813699, Office of Naval Research

52


Contract N00014-88-K-0289, and the General Motors Research Laboratories. E. B. was supported by grants from the Commission of European Communities (B.R.A.I.N. ST2J-0416) and the French MinistGre de la Recherche (87C0187).

References Ackley, D. H., Hinton, G. E., and Sejnowski, T. J. 1985. A learning algorithm for Boltzmann machines. Cog. Sci. 9, 147-169. Akaike, H. 1973. Information theory and an extension of the maximum likelihood principle. In Proceedings of the 2nd International Symposium on Information Theory, B. N. Petrov and F. Csaki, eds., pp. 267-281. Akademia Kiado, Budapest. Amari, S. I. 1967. A theory of adaptive pattern classifiers. I E E E Transact. Elect. Computers EC-16, 299-307. Amari, S. I. 1990. Dualistic geometry of the manifold of higher-order neurons. Tech. Rep. METR 90-17, Department of Mathematical Engineering and Instrumentation Physics, University of Tokyo, Bunkyo-Ku, Tokyo. Amari, S. I., Kurata, K., and Nagaoka, H. 1990. Differential geometry of Boltzmann machines. Tech. Rep., Department of Mathematical Engineering and Instrumentation Physics, University of Tokyo, Bunkyo-Ku, Tokyo. Amit, Y., Grenander, U., and Piccioni, M. 1991. Structural image restoration through deformable templates. I. A m . Statist. Assoc. 86, 376-387. Anderson, J. A., and Rosenfeld, E. 1988. Neurocomputing, Foundations of Research, p. 587. MIT Press, Cambridge MA. Anderson, J. A., Silverstein, J. W., Ritz, S. A., and Jones, R. S. 1977. Distinctive features, categorical perception, and probability learning: Some applications of a neural model. Psychol. Rev. 84, 413451. Anderson, J. A., Rossen, M. L., Viscuso, S. R., and Sereno, M. E. 1990. Experiments with representation in neural networks: Object motion, speech, and arithmetic. In Synergetics of Cognition, H. Haken and M. Stadler, eds. Springer-Verlag, Berlin. Azencott, R. 1990. Synchronous Boltzmann machines and Gibbs fields: Learning algorithms. In Neurocomputing, Algorithms, Architectures and Applications, F. Fogelman-Soulie and J. Herault, eds., pp. 51-63. NATO AS1 Series, Vol. F68. Springer-Verlag, Berlin. Bajcsy, R., and Kovacic, S. 1989. Multiresolution elastic matching. Comput. Vision, Graphics, Image Process. 46, 1-21. Baldi, P., and Hornik, K. 1989. Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks 2, 53-58. Barron, A. R. 1989. Statistical properties of artificial neural networks. Proc. of the 28th Conf. Decision Control, Tampa, Florida, 280-285.


53

Barron, A. R. 1991. Complexity regularization with application to artificial neural networks. In Nonparametric Functional Estimation and Related Topics, G. Roussas, ed., pp. 561-576. Kluwer, Dordrecht. Barron, A. R., and Barron, R. L. 1988. Statistical learning networks: A unifying view. In Computing Science and Statistics, Proceedings of the 20th Symposium Interface, E. Wegman, ed., pp. 192-203. American Statistical Association, Washington, DC. Baum, E. B. 1990a. The perceptron algorithm is fast for nonmalicious distributions. Neural Comp. 2, 248-260. Baum, E. B. 1990b. When are k-nearest-neighbor and backpropagation accurate for feasible-sized sets of examples? In Proceedings Eurasip Workshop on Neural Networks, L. B. Almeida and C. J. Wellekens, eds., pp. 2-25. Springer-Verlag, Berlin. Baum, E. B., and Haussler, D. 1989. What size net gives vaIid generalization? Neural Comp. 1, 151-160. Baum, L. E. 1972. An inequality and associated maximization technique in statistical estimation for probabilistic functions of Markov processes. Inequalities 3, 1-8. Bellman, R. E. 1961. Adaptive Control Processes. Princeton University Press, Princeton, NJ. Bienenstock, E., and Doursat, R. 1989. Elastic matching and pattern recognition in neural networks. In Neural Networks: From Models to Applications, L. Personnaz and G. Dreyfus, eds., pp. 472-482. IDSET, Paris. Bienenstock, E., and Doursat, R. 1991. Issues of representation in neural networks, In Representations of Vision: Trends and Tacit Assumptions in Vision Reseauch, A. Gorea, ed., pp. 47-67. Cambridge University Press, Cambridge. Bourlard, H., and Kamp, Y. 1988. Auto-association by multi-layer perceptrons and singular value decomposition. Biol. Cybernet. 59, 291-294. Bourlard, H., and Wellekens, C. J. 1990. Links between Markov models and multilayer perceptrons. I E E E Transact. Pattern Anal. Machine Intell. 12, 11671178. Breiman, L., and Friedman, J. H. 1985. Estimating optimal transformations for multiple regression and correlation. 1. Am. Stat. Assoc. 80, 580-619. Breiman, L., Friedman, J. H., Olshen, R. A., and Stone, C. J. 1984. Classification and Regression Trees. Wadsworth, Belmont, CA. Buhmann, J., Lange, J., von der Malsburg, Ch., Vorbruggen, J. C., and Wiirtz, R. P. 1989. Object recognition in the dynamic link architecture: Parallel implementation on a transputer network. In Neural Networks: A Dynamic Systems Approach f o Machine Intelligence, B. Kosko, ed. Prentice Hall, New York. Burr, D. J. 1980. Elastic matching of line drawings. IEEE Transact. Pattern Anal. Machine Intell. PAMI-3 6, 708-713. Carnevali, P., and Patarnello, S. 1987. Exhaustive thermodynamic analysis of Boolean learning networks. Europhys. Lett. 4, 1199-1204. Chauvin, Y. 1990. Dynamic behavior of constrained back-propagation networks. In Neural Information Processing Systems 11, D. S. Touretzky, ed., pp. 642-649.

54


Morgan Kaufmann, San Mateo, CA. Collins, E., Ghosh, S., and Scofield, C. 1989. An application of a multiple neural network learning system to emulation of mortgage underwriting judgements. Nestor Inc., Providence, RI. Cox, D. R. 1970. The Analysis of Binary Data. Methuen, London. Crick, F. 1989. The recent excitement about neural networks. Nature (London) 337, 129-132. Cybenko, G. 1989. Approximations by superpositions of a sigmoidal function. Math. Control, Signals Syst. 2, 303-314. Dempster, A. I?, Laird, N. M., and Rubin, D. 8. 1976. Maximum likelihood from incomplete data via the EM algorithm. J. R. Statist. SOC. B 39, 1-38. Denker, J., Schwartz, D., Wittner, B., Solla, S., Howard, R., Jackel, L., and Hopfield, J. 1987. Automatic learning, rule extraction and generalization. Complex Syst. 1, 877-922. Diaconis, P., and Freedman, D. 1986. On the consistency of Bayes estimates. Ann. Statist. 14, 1-26. Duda, I Vplr = 120, VL = -60, and Vsyn = -80. With C = 1 pF/cm2 and gL = 0.1 mS/cm2, the passive membrane time constant 70 is 10 msec. The factor $ scales the kinetics of k; $I = 3 unless stated otherwise. The voltage dependent gating functions are rnco(V)= 1/{1+ exp[-(V 65)/7.8]}, h,(V) = l/{l+ exp[(V 81)/11]}, and q ( V ) = h,(V) exp[(V+162.3)/17.8].The time constant fork (TJJ has a maximal value of 65 msec at about -70 mV. We note in passing that a sag current would have an expression similar to I,,, in equation 2.1, but with the factor m k ( V ) omitted and with some quantitative changes (e.g., much slower time constant for k and different reversal potential). Below we describe the dynamic behavior of the two-cell system and its dependence on the two remaining parameters: the maximum conductance of the PIR current, gplr,which specifies an intrinsic membrane property, and the synaptic threshold, Osynr a coupling parameter. Numerical integrations were carried out using the software package “PHASEPLANE (Ermentrout 1990). The Gear method was used, with the tolerance parameter set to 0.001 or 0.01. The “ A U T O program (Doedel 1981) was used for generating the bifurcation diagrams in Figure 2.

+

+

3 Alternating Oscillation by the Release Mechanism

Figure l a illustrates the classical alternating pattern obtained with our model in the parameter regime for ”release.” This out-of-phase oscillation proceeds as follows: an excited cell (#1)sends inhibitory input to its partner (cell #2), so that a PIR current is deinactivated in the latter. Following the excitation of cell #1, cell #2 is released from hyperpolarization and executes a rebound excitation; this PIR leads to the inhibition of cell #1 in turn. The process repeats itself periodically.

Rhythms and Synchrony of Inhibitory Neurons

a7

V=O (inhibited)

-1'20

'

v -;3m"vv,

2b

Figure 1: Two reciprocally inhibitory neurons establish an alternating rhythmic oscillation by the "release" mechanism. (a) Membrane potential time courses for the out-of-phase oscillation between the two cells. Numerical solution of equations 2.1-2.2 with gpi, = 0.3 mS/cm2, Bsyn = -44 mV. A cell is released from inhibition when the membrane potential of its partner falls below the synaptic threshold, Bsyn (dashed horizontal line). (b) The V - h phase plane of a single neuron; V- and h-nullclines shown both in the absence and presence of an inhibitory synaptic current gsyn(V-Vsyn).The steady state (filled circles) in each case is denoted by V Fand VI, respectively. The arrow on the V-axis indicates the value of Bsyn. Trajectory of a neuron (closed curve with arrowheads) from the two-cell simulation shows that the alternating oscillation consists of repetitive switchings between the "free" single cell configuration (labeled a and b) and the "inhibited" one (labeled c and d).

88

Xiao-Jing Wang and John Rinzel

To dissect this oscillatory behavior mathematically we consider a limiting situation with the parameter ksyn very small, so that the synaptic function S,(V) is either zero or one except at the abrupt transition level V = Bsyn. Then, a neuron is either "free" or "inhibited" when the membrane potential of its partner is, respectively, below or above the synaptic threshold Bsyn. The oscillatory pattern in Figure l a then consists of repetitive switchings between these two states that occur in precise alternation in the two cells. Each state is described by the equations for a single neuron, one with and the other without, a synaptic current, gsyn(V- Vsyn), of constant conductance. Since the single-neuron model has two variabl.es, V and h, we may analyze it by phase-plane methods (Edelstein-Keshet 1988). The nullclines of V and h are the curves obtained by setting dV/dt = 0 and dh/dt = 0, respectively (in equations 2.1-2.2), which yield (3.1) In the free state, the synaptic term is absent in the V-nullcline in equation 3.1. The two V-nullclines, with and without the synaptic term, are displayed in Figure lb, together with the h-nullcline. An intersection point (filled circle) of the V- and h-nullclines corresponds to a timeindependent steady state. We denote the steady-state membrane potential as VF and VI, for the free case and the inhibited one, respectively. In Figure la, with gpir= 0.3 mS/cm2, we have VF = -45 mV, and VI = -74 mV. Also plotted in Figure l b is the V - k trajectory of one neuron, obtained by numerically integrating the full four-variable system equations 2.1-2.2. From this phase-plane portrait we reinterpret the oscillation as follows. Referring to the labels on the trajectory and on the time course in Figure l a we see that phases a and b correspond to the free state while c and d correspond to the inhibited state. During phase a the neuron undergoes its rebound depolarization after release from inhibition. The cell reaches maximum depolarization when its trajectory crosses the "free" Vnullcline. Then V decreases toward its resting value VF (phase b); a slight undershoot of h is seen late in b as the trajectory crosses the k-nullcline. During phase b, as V passes downward through OSyn its partner is released from inhibition. This phase ends forcibly when the cell becomes postsynaptic to its partner (whose membrane potential rises above Bsyn during its rebound excitation). At the time of this transition, one should imagine that the free V-nullcline disappears and is replaced by the inhibited V-nullcline. The current position of (V. h ) (previously, resting) now lies in the region where dV/dt < 0 so the trajectory moves leftward toward the now-applicable, inhibited V-nullcline (phase c). Deinactivation occurs as the trajectory moves upward along the V-nullcline (phase d). Finally, upon being released by its partner, the cell becomes free, the free V-nuIlcline reappears, and the trajectory makes a sharp rightward turn.


1

n

89

501

-1

0’ 4 0

I

-30 Synaptic Threshold 0, (mV)

-50

Figure 2: Period of alternating oscillation versus the synaptic threshold Osyn for gpir = 0.3 and 1.0 mS/cm2. Periods for Figures 1 and 3 correspond, respectively, to the values of the curves at esyn= -44 mV. Solid curve: stable; dotted curve: unstable. Beyond a turning point (where solid and dotted curves coalesce), stable oscillation ceases to exist. Periodic solutions emerge via Hopf bifurcation (Edelstein-Keshet1988) at endpoints of curves (filled circles). In the case gpir = 1.0 mS/cm2, we have ”release” for Osyn > -35 mV (= V F )and “escape,” with full inhibition, to the left of the steep region. Notice that, in the release case, the period depends strongly on Bsyn, which controls duration of the synaptic current while in the escape case the period is largely constant. [Note, if the cell were not released, ( V , h ) would continue u p the left branch toward the stable steady state at V,.] The synaptic threshold Osyn is an important control parameter in this release scenario, for two reasons. First, the release event and the oscillatory pattern of Figure 1 are possible only if Qsyn exceeds the resting membrane potential V , of a free cell. The inequality Bsyn > VF means that a neuron at rest is not depolarized enough to inhibit its partner. Moreover, since both cells can thus sit stably at the free resting state, this system is bistable, with the coexistence of a stable steady state and a n oscillatory one. Second, Osyn critically determines the fraction of time of a PIR excitation during which the depolarized neuron effectively sends inhibition to its partner and, thereby, it determines the period of the alternating oscillation. The oscillation period as a function of Bsyn is plotted in Figure 2. Consider first the solid portion of the curve. If Bsyn is too high, no oscillation occurs since the synaptic inhibition generated by a rebound excitation would be too brief to deinactivate the PIR current. The only maintained behavior for the two cells is to be both at rest: V , = V , = V,. Moreover, as long as Osyn > VF, this resting state is expected to be stable. As Osyn is

90


decreased, sustained pacemaking emerges with minimum period. This period is determined primarily by the time constant T ~ ( Vof) the inactivation gate, which sets the minimum duration of inhibition for a PIR response (Wang et al. 1991, 1992). The period then increases with decreasing Bsyn, and the oscillation disappears as esynapproaches V,. This disappearance occurs either because the period diverges to infinity (Wang et al. 1992) or, as in Figure 2, because the stable periodic orbit coalesces with another coexistent (unstable) periodic orbit. This latter mechanism leads to a ”turning point” (or tangent bifurcation) in a plot of oscillation period (or amplitude) versus parameter such as in Figure 2. It is also how the stable oscillation first appeared for Bsyn equal to about -37 mV. For Osyn below VF, the membrane potential of a free cell would return to VF after excitation but without passing below Osyn. Hence, V1 would remain at VF and the free cell would permanently “enslave” the other cell (at V2 = Vr). Obviously, there are two such asymmetric time-independent solutions, with the role interchanged in the pair. 4 Alternating Oscillation by the Escape Mechanism

In the preceding case, a cell that is inhibited will remain so because, at V,, the deinactivated PIR current (together with I L ) is offset by the inhibitory synaptic current. However, if g,,: is larger the slowly deinactivating Ipir can overcome the hyperpolarizing synaptic current, and we call this escape in the presence of maintained inhibition by a free cell. An example of escape is obtained with the same parameter values as in the release case, except for gpir which is increased from 0.3 to 1.0 mS/cm2 (Fig. 3a). We may distinguish the two cases by comparing their phase plane profiles (Fig. 3b and Fig. lb). The increase of gpir brings about important changes both for the free V-nullcline and the inhibited one. The resting membrane potential of a free neuron is now V, = -35 mV, more positive than Bsyn (-44 mV), so that release becomes impossible. On the other hand, the V-nullcline of an inhibited neuron is lowered by larger gP” (cf. equation 3.1). As a result, the steady state V, is shifted onto the middle branch and is destabilized. The trajectory of an inhibited neuron now reaches larger values of h, along the left branch of the V-nullcline, thereby further deinactivating Ipir.The trajectory is constrained to remain leftward of this branch until it reaches the top of the hump, when it moves rapidly rightward, and the neuron escapes from the inhibition. Unlike the release case, here the switching event is controlled by the inhibited neuron rather than the free one. If switching happens rapidly, then the oscillation period is about twice the time needed for an inhibited neuron to ascend fully the left branch of its V-nullcline. Therefore, the period of oscillation is expected to be insensitive to the synaptic parameter Bsyn in the escape case.


0.5

1

91

\h=o

v=o

h

v-gv)

2b

Figure 3: Alternating oscillation by the "escape" mechanism, with gpir = 1.0 mS/cm2 in (a and b) and gpir = 1.5 mS/cm2 in (c and d); Bsyn = -44 mV in both cases. (a) membrane potential versus time reveals higher peaks here, compared to Figure la, due to the increased gpir. From t = 380 to 580 msec, cell #1 is voltage clamped at V1 > Bsyn. Cell #2 receives constant inhibition and executes a self-sustained oscillation [cf. dashed curve in (b)]. (b) Phase-plane portrait with nullclines for "free" and "inhibited" cell. An inhibited neuron can escape from hyperpolarized state because the left "hump" of its V-nullcline has no stable steady state; here, steady state is unstable and surrounded by a stable periodic orbit (dashed closed curve). Continued next page.

For Osyn values higher than VF, release becomes possible again. Either release or escape may occur depending on which of the following two processes is faster: the fall of V from its peak to OSp for the free neuron o r the ascension along the left branch for the inhibited neuron. The period of oscillation versus Bsyn is plotted in Figure 2. The significant increase of the period for Osyn near -45 is reminiscent of the release case. Note that, for Osyn just lower than VF = -36, due to a not so small value of


92

0'5 h

1I \"-"\

v=o

(inhibited)

Figure 3: Continued from previous page. (c,d) In this escape case, different from (a,b), the inhibited nullcline has a stable steady state on right branch of V nullcline. This leads to bistability for the neuron pair with a stable alternating oscillation and an asymmetric steady state (Vl = - 34.3 mV, hl=0.0141, Vz = -50.5 mV, k2 =0.0587).

ksyn the hyperpolarization from the free cell does not achieve its maximal strength, so that the escape for the inhibited cell is easier and quicker. The idealized escape situation applies only when Qsyn 5 -45 mV, where the period remains virtually constant. Of further interest in this escape case is tha.t an inhibited neuron has a unique steady state, at V,, and it is an unstable spiral. There exists a limit cycle around it (dashed curve in Fig. 3b), that is, a single neuron can be a pacemaker under a constant synaptic inhibitory drive. Thus, if a cell is transiently voltage-clamped to a depolarized level above Osyn, its partner would undergo this self-sustained oscillation (Fig. 3a). Such


93

a protocol might be used experimentally to distinguish the release and escape mechanisms. For even higher values of gpir,the inhibited V-nullcline is further lowered, the steady state at VI is shifted near or onto the right branch and it may become stable again (Fig. 3d). Nevertheless, a transient hyperpolarization could force the trajectory to cross the left branch of the V-nullcline. The succeeding large amplitude PIR response could lead to inhibiting the partner, thereby initiating an escape so that an alternating oscillation might be established. In contrast to the previous escape case, here a single inhibited neuron could not oscillate. It is also readily seen from Fig. 3d that the oscillation usually coexists with a stable asymmetric steady state with V1 = VF and V, = VI; for this, we have Vl < Osyn < VF. Figure 3c shows a protocol to detect such a bistability: a transient pulse of hyperpolarization of 50 msec leads to a transition from an asymmetric steady state to a sustained oscillation. Such bistability has been discovered using a similar protocol in laboratory experiments on a subnetwork of the CPG for the leech heartbeat (Arbas and Calabrese 1987; compare their Fig. 7 to our Fig. 3c). This demonstration is a striking indication that the escape mechanism described here may be relevant to that system, which contains pairs of inhibitorily coupled cells possessing a sag current. To draw a closer correspondence between the leech heartbeat and our theoretical model, it should be investigated experimentally whether the oscillatory period in the leech heartbeat interneurons is sensitive to the synaptic parameters that control the duration of postsynaptic inhibitory potentials. One may question how the important prerequiste for the escape mechanism, namely that the resting potential of a free neuron is higher than the synaptic threshold, could be realized in real neuronal systems. The leech heartbeat interneurons exhibit a very slowly decaying plateau potential (Arbas and Calabrese 1987). Therefore, this plateau depolarization may contribute to maintaining a quasistationary potential level in a free cell that is higher than the synaptic threshold, at least during the phase just prior to the escape event of a contralateral partner.

5 Synchronization by a Slowly Decaying Synaptic Activation

~

In accord with common wisdom, we have shown that neurons of an inhibitory pair tend to oscillate out-of-phase. Indeed, when s,, depends instantaneously on V it would be impossible to imagine a pattern in which two cells were simultaneously inhibited. However, another possibility arises if the synaptic action displays a slow time course, so that inhibition can outlast the PIR event. We report here an example in which our two model cells can be brought into perfect synchrony by the effects of a slow synaptic decay. Assume now that the synaptic variables s,j obey

94


first-order kinetics, described by

Then, if k, is sufficiently small, both cells would "feel" perpetually some average synaptic input. If, in addition, the PIR is strong, oscillation in two cells is possible. Oscillatory synaptic inputs around the average communicate phasic information between the two cells which may allow them to synchronize. Such an in-phase oscillation is shown in Figure 4, together with two other coexisting attractors: an out-of-phase oscillatory state and an asymmetric steady state. A depolarizing pulse applied simultaneously to both cells induces a transition from the asymmetric steady state to the synchronized oscillatory state; whereas another, asymmetric perturbation (current pulse of same duration and same absolute intensity, but depolarizing to one and hyperpolarizing to the other) would lead to the out-of-phase oscillatory state, thus desynchronizing the system. The in-phase oscillation uncovered here seems suggestive for the reticular thalamic nucleus, where inhibitory cells interact with each other via GABAergic synapses that usually possess a slow component and where spindling oscillations are marked by a high degree of synchronization in the thalamocortical system (Steriade et al. 1990). 6 Discussion

We have explored, via simulation and analysis, the, activity patterns supported by biophysically meaningful model cells which exhibit PIR and which are coupled by reciprocal inhibition. Two cells (each nonautorhythmic) can generate oscillatory patterns with the cells either outof-phase or, surprisingly, in-phase. The former, the classical alternating pattern, arises ubiquitously when post-synaptic variable, sji, depends instantaneously on pre-synaptic potential, Vpre.In-phase rhythms can occur when sji decays slowly after a transient depolarization by V,,,. In either case, these behaviors are not unique; in some parameter regimes two, or more, stable activity patterns coexist. Our simplified ionic model for a cell has only two membrane variables. By applying phase plane techniques, when sli is an instantaneous and steep sigmoidal function of Vpre,we find that two distinct mechanisms, "release" or "escape," underlie the alternating oscillation. In the first case, but not the second, the oscillation period depends sensitively on the duration of synaptic hyperpolarization. In "escape," a cell's intrinsic properties allow rebound excitation [and, in some parameter regimes (cf. Fig. 3a,b), sustained oscillation] to proceed even under maintained synaptic inhibition. In either case bistability may occur, where the oscillation coexists with a stationary pattern of both cells resting, or one


95

-34 -6

-9d

-1l 1

n

n

11

3 -! -6 -9d

I

0

500

1000 t (ms)

1500

1

2000

Figure 4: The two-cell system, equations 2.1-2.2, can oscillate in-phase when the post-synaptic conductance obeys first order kinetics (equation 5.1)with slow decay rate. Cells are started in a stable asymmetric steady state with V1 = -37 mV, V2 = -72 mV. Compare the membrane potential and synaptic activation time courses to see that cell #1 is free (s21 = 0) and cell #2 is inhibited (s12 = 1). The cells are switched into a synchronous oscillation when, at t = 300 msec, a depolarizing current pulse of intensity 1.0 pA/cm2 and of duration 50 msec is delivered to each cell. The time courses of s,j during this phase exhibit rapid onset but slow decay of synaptic activation. Then, at t = 1100 msec, a depolarizing current pulse to cell #1 (intensity, 1.0 ,uA/cm2; duration, 50 msec) and a hyperpolarizing one to cell #2 (intensity, -1.0 ,uA/cm2; duration, 50 msec) send the neuronal pair into an out-of-phase periodic pattern. This system thus has at least three coexistent maintained activity patterns for these parameter values, which differ from those of previous figures as follows: gpir= 0.5 mS/cm2, gsyn= 0.2 mS/cm2, gL = 0.05 mS/cm2, Q, = 2, Bsyn = -35 mV, and k, = 0.005.

96


resting and one inhibited, respectively. Our results suggest that the "release" and "escape" cases may perhaps be distinguished experimentally by selectively modulating parameters that control synaptic activation, particularly synaptic duration; or those that control the inward current that is unmasked by hyperpolarization and that underlies PIR. In an early modeling study (Reiss 1962), stable generation of an alternating rhythm relied on the fatigue of synapses. Later (Perkel and Mulloney 1974), PIR was proposed as an alternative mechanism, originating as an intrinsic cellular rather than a coupling property. The ionic basis of the PIR, however, was not identified and modeled in their study. Here, we have shown explicitly that inactivation of Zpir can play such a role. Either release occurs naturally as Vpre falls toward rest after its rebound excitation, as lpir inactivates; or the inhibited cell escapes on its own, as deinactivation allows Ipirto overcome the synaptic current. We note that rebound excitation could also result from deactivation of an outward current. Although "release" would still be possible, it appears that for "escape" to occur an additional factor, beyond such a n outward current, would be necessary. We have presented results only for two coupled inhibitory cells, but our interest extends to larger ensembles, for example, in connection with the reticular thalamic nucleus as a pacemaker for thalamocortical bursting oscillations. Our preliminary simulations have shown that slow decay of synaptic actions can lead to total synchronization in larger networks where inhibition is widespread (all-to-all coupling).

Acknowledgments We thank Dr. Arthur Sherman for a careful reading of our manuscript.

References Angstadt, J. D., and Calabrese, R. L. 1989. A hyperpolarization-activatedinward current in heart interneurons of the medicinal leech. J. Neurophysiol. 9,28462857. Arbas, E. A,, and Calabrese, R. L. 1987. Slow oscillations of membrane potential in interneurons that control heartbeat in the medicinal leech. J. Neurosci. 7, 3953-3960. Doedel, E. 1981. AUTO: A program for the automatic bifurcation analysis of autonomous systems. Cong. Num. 30,265-284. Edelstein-Keshet,L. 1988. Mathematical Models in Biology. Random House, New York. Ermentrout, G. B. 1990. PHASEPLANE: The Dynamzcal Systems Tool, Version 3.0. Brooks/Cole Publishing Co., Pacific Grove, CA. Perkel, D. H., and Mulloney, B. 1974. Motor pattern production in reciprocally inhibitory neurons exhibiting postinhibitory rebound. Science 185, 181-183.


97

Reiss, R. F. 1962. A theory and simulation of rhythmic behavior due to reciprocal inhibition in small nerve nets. Proc. M I P S Spring Joint Comput. Conf. 21, 171-1 94. Satterlie, R. A. 1985. Reciprocal inhibition and postinhibitory rebound produce reverbation in a locomotor pattern generator. Science 229, 402-404. Selverston, A. I., and Moulins, M. 1985. Oscillatory neural networks. Annu. Rev. Physiol. 47, 2948. Steriade, M., Jones, E. G., and Llinb, R. R. 1990. Thalamic Oscillations and Signaling. John Wiley, New York. Wang, X.-J., Rinzel, J., and Rogawski, M. A. 1991. A model of the T-type calcium current and the low-threshold spikes in thalamic neurons. J, Neurophysiol. 66,839450. Wang, X.-J., Rinzel, J., and Rogawski, M. A. 1992. Low threshold spikes and rhythmic oscillations in thalamic neurons. In Analysis and Modeling of Neural Systems, F.Eeckman, ed., pp. 85-92. Kluwer, Boston.

Received 8 July 1991; accepted 12 August 1991.

This article has been cited by: 2. I. Belykh, S. Jalil, A. Shilnikov. 2010. Burst-duration mechanism of in-phase bursting in inhibitory networks. Regular and Chaotic Dynamics 15:2-3, 146-158. [CrossRef] 3. Sachin S. Talathi, Dong-Uk Hwang, Paul R. Carney, William L. Ditto. 2010. Synchrony with shunting inhibition in a feedforward inhibitory network. Journal of Computational Neuroscience 28:2, 305-321. [CrossRef] 4. Sajiya Jalil, Igor Belykh, Andrey Shilnikov. 2010. Fast reciprocal inhibition can synchronize bursting neurons. Physical Review E 81:4. . [CrossRef] 5. Asya Shpiro, Ruben Moreno-Bote, Nava Rubin, John Rinzel. 2009. Balance between noise and adaptation in competition models of perceptual bistability. Journal of Computational Neuroscience 27:1, 37-54. [CrossRef] 6. Silvia Daun, Jonathan E. Rubin, Ilya A. Rybak. 2009. Control of oscillation periods and phase durations in half-center central pattern generators: a comparative mechanistic analysis. Journal of Computational Neuroscience 27:1, 3-36. [CrossRef] 7. Ramana Dodla, Charles Wilson. 2009. Asynchronous Response of Coupled Pacemaker Neurons. Physical Review Letters 102:6. . [CrossRef] 8. Yuzhu Xiao, Wei Xu, Xiuchun Li, Sufang Tang. 2009. The effect of noise on the complete synchronization of two bidirectionally coupled piecewise linear chaotic systems. Chaos: An Interdisciplinary Journal of Nonlinear Science 19:1, 013131. [CrossRef] 9. John Guckenheimer, Christian Kuehn. 2009. Computing Slow Manifolds of Saddle Type. SIAM Journal on Applied Dynamical Systems 8:3, 854. [CrossRef] 10. Sachin S. Talathi, Dong-Uk Hwang, William L. Ditto. 2008. Spike timing dependent plasticity promotes synchrony of inhibitory networks in the presence of heterogeneity. Journal of Computational Neuroscience 25:2, 262-281. [CrossRef] 11. Igor Belykh, Andrey Shilnikov. 2008. When Weak Inhibition Synchronizes Strongly Desynchronizing Networks of Bursting Neurons. Physical Review Letters 101:7. . [CrossRef] 12. K.L. Briggman, W.B. Kristan. 2008. Multifunctional Pattern-Generating Circuits. Annual Review of Neuroscience 31:1, 271-294. [CrossRef] 13. Xiao Yu-Zhu, Xu Wei, Li Xiu-Chun, Tang Su-Fang. 2008. Complete synchronization of uncertain chaotic dynamical network via a simple adaptive control. Chinese Physics B 17:1, 80-86. [CrossRef] 14. Andrey Shilnikov, René Gordon, Igor Belykh. 2008. Polyrhythmic synchronization in bursting networking motifs. Chaos: An Interdisciplinary Journal of Nonlinear Science 18:3, 037120. [CrossRef]

15. Rodica Curtu, Asya Shpiro, Nava Rubin, John Rinzel. 2008. Mechanisms for Frequency Control in Neuronal Competition Models. SIAM Journal on Applied Dynamical Systems 7:2, 609. [CrossRef] 16. Victor Matveev, Amitabha Bose, Farzan Nadim. 2007. Capturing the bursting dynamics of a two-cell inhibitory network using a one-dimensional map. Journal of Computational Neuroscience 23:2, 169-187. [CrossRef] 17. Janet Best, Choongseok Park, David Terman, Charles Wilson. 2007. Transitions between irregular and rhythmic firing patterns in excitatory-inhibitory neuronal networks. Journal of Computational Neuroscience 23:2, 217-235. [CrossRef] 18. Yuzhu Xiao, Wei Xu, Xiuchun Li, Sufang Tang. 2007. Adaptive complete synchronization of chaotic dynamical network with unknown and mismatched parameters. Chaos: An Interdisciplinary Journal of Nonlinear Science 17:3, 033118. [CrossRef] 19. Marlene Bartos, Imre Vida, Peter Jonas. 2007. Synaptic mechanisms of synchronized gamma oscillations in inhibitory interneuron networks. Nature Reviews Neuroscience 8:1, 45-56. [CrossRef] 20. Allen I. Selverston, Joseph Ayers. 2006. Oscillations and oscillatory behavior in small neural circuits. Biological Cybernetics 95:6, 537-554. [CrossRef] 21. Murat Sekerli, Robert J. Butera. 2005. Oscillations in a Simple Neuromechanical System: Underlying Mechanisms. Journal of Computational Neuroscience 19:2, 181-197. [CrossRef] 22. Antonio Palacios, Ricardo Carretero-González, Patrick Longhini, Norbert Renz, Visarath In, Andy Kho, Joseph Neff, Brian Meadows, Adi Bulsara. 2005. Multifrequency synthesis using two coupled nonlinear oscillator arrays. Physical Review E 72:2. . [CrossRef] 23. Benjamin Pfeuty , Germán Mato , David Golomb , David Hansel . 2005. The Combined Effects of Inhibitory and Electrical Synapses in SynchronyThe Combined Effects of Inhibitory and Electrical Synapses in Synchrony. Neural Computation 17:3, 633-670. [Abstract] [PDF] [PDF Plus] 24. Christoph Börgers , Nancy Kopell . 2005. Effects of Noisy Drive on Rhythms in Networks of Excitatory and Inhibitory NeuronsEffects of Noisy Drive on Rhythms in Networks of Excitatory and Inhibitory Neurons. Neural Computation 17:3, 557-608. [Abstract] [PDF] [PDF Plus] 25. E. I. Volkov. 2005. Limit Cycles Arising in a Chain of Inhibitorily Coupled Identical Relaxation Oscillators Near the Self-Oscillation Threshold. Radiophysics and Quantum Electronics 48:3, 212-221. [CrossRef] 26. Alain Destexhe, Eve Marder. 2004. Plasticity in single neuron and circuit computations. Nature 431:7010, 789-795. [CrossRef] 27. Brent Doiron, Benjamin Lindner, André Longtin, Leonard Maler, Joseph Bastian. 2004. Oscillatory Activity in Electrosensory Neurons Increases with the

Spatial Correlation of the Stochastic Input Stimulus. Physical Review Letters 93:4. . [CrossRef] 28. F.A. Mussa-Ivaldi, S.A. Solla. 2004. Neural Primitives for Motion Control. IEEE Journal of Oceanic Engineering 29:3, 640-650. [CrossRef] 29. Jose L. Perez Velazquez. 2003. Bicarbonate-dependent depolarizing potentials in pyramidal cells and interneurons during epileptiform activity. European Journal of Neuroscience 18:5, 1337-1342. [CrossRef] 30. David Chik, Z. Wang. 2003. Postinhibitory rebound delay and weak synchronization in Hodgkin-Huxley neuronal networks. Physical Review E 68:3. . [CrossRef] 31. D. Hansel , G. Mato . 2003. Asynchronous States and the Emergence of Synchrony in Large Networks of Interacting Excitatory and Inhibitory NeuronsAsynchronous States and the Emergence of Synchrony in Large Networks of Interacting Excitatory and Inhibitory Neurons. Neural Computation 15:1, 1-56. [Abstract] [PDF] [PDF Plus] 32. Yasuomi Sato, Masatoshi Shiino. 2002. Spiking neuron models with excitatory or inhibitory synaptic couplings and synchronization phenomena. Physical Review E 66:4. . [CrossRef] 33. Antonis Karantonis, Yasuyuki Miyakita, Seiichiro Nakabayashi. 2002. Synchronization of coupled assemblies of relaxation oscillatory electrode pairs. Physical Review E 65:4. . [CrossRef] 34. Adam L. Taylor , Garrison W. Cottrell , William B. Kristan, Jr. . 2002. Analysis of Oscillations in a Reciprocally Inhibitory Network with Synaptic DepressionAnalysis of Oscillations in a Reciprocally Inhibitory Network with Synaptic Depression. Neural Computation 14:3, 561-581. [Abstract] [PDF] [PDF Plus] 35. Jan Karbowski, G. Ermentrout. 2002. Synchrony arising from a balanced synaptic plasticity in a network of heterogeneous neural oscillators. Physical Review E 65:3. . [CrossRef] 36. G.N. Borisyuk, R.M. Borisyuk, Yakov B. Kazanovich, Genrikh R. Ivanitskii. 2002. Models of neural dynamics in brain information processing the developments of 'the decade'. Uspekhi Fizicheskih Nauk 172:10, 1189. [CrossRef] 37. S. Coombes, M Owen, G. Smith. 2001. Mode locking in a periodically forced integrate-and-fire-or-burst neuron model. Physical Review E 64:4. . [CrossRef] 38. F. K. Skinner, C. Wu, L. Zhang. 2001. Phase-coupled oscillator models can predict hippocampal inhibitory synaptic connections. European Journal of Neuroscience 13:12, 2183-2194. [CrossRef] 39. Amitabha Bose, Michael Recce. 2001. Phase precession and phase-locking of hippocampal pyramidal cells. Hippocampus 11:3, 204-215. [CrossRef]

40. I. Susa, T. Erneux, A. Barsella, C. Lepers, D. Dangoisse, P. Glorieux. 2000. Synchronization through bursting oscillations for two coupled lasers. Physical Review A 63:1. . [CrossRef] 41. Gennady S. Cymbalyuk , Girish N. Patel , Ronald L. Calabrese , Stephen P. DeWeerth , Avis H. Cohen . 2000. Modeling Alternation to Synchrony with Inhibitory Coupling: A Neuromorphic VLSI ApproachModeling Alternation to Synchrony with Inhibitory Coupling: A Neuromorphic VLSI Approach. Neural Computation 12:10, 2259-2278. [Abstract] [PDF] [PDF Plus] 42. Jan Karbowski , Nancy Kopell . 2000. Multispikes and Synchronization in a Large Neural Network with Temporal DelaysMultispikes and Synchronization in a Large Neural Network with Temporal Delays. Neural Computation 12:7, 1573-1606. [Abstract] [PDF] [PDF Plus] 43. J. Rubin , D. Terman . 2000. Geometric Analysis of Population Rhythms in Synaptically Coupled Neuronal NetworksGeometric Analysis of Population Rhythms in Synaptically Coupled Neuronal Networks. Neural Computation 12:3, 597-645. [Abstract] [PDF] [PDF Plus] 44. Paul C. Bressloff , S. Coombes . 2000. Dynamics of Strongly Coupled Spiking NeuronsDynamics of Strongly Coupled Spiking Neurons. Neural Computation 12:1, 91-129. [Abstract] [PDF] [PDF Plus] 45. P. C. Bressloff, S. Coombes. 2000. A Dynamical Theory of Spike Train Transitions in Networks of Integrate-and-Fire Oscillators. SIAM Journal on Applied Mathematics 60:3, 820. [CrossRef] 46. Jose L. Perez Velazquez, Peter L. Carlen. 1999. Synchronization of GABAergic interneuronal networks during seizure-like activity in the rat horizontal hippocampal slice. European Journal of Neuroscience 11:11, 4110-4118. [CrossRef] 47. Seon Park, Seunghwan Kim, Hyeon-Bong Pyo, Sooyeul Lee. 1999. Multistability analysis of phase locking patterns in an excitatory coupled neural system. Physical Review E 60:2, 2177-2181. [CrossRef] 48. A. SELVERSTON, R. ELSON, M. RABINOVICH, R. HUERTA, H. ABARBANEL. 1998. Basic Principles for Generating Motor Output in the Stomatogastric Ganglion. Annals of the New York Academy of Sciences 860:1 NEURONAL MECH, 35-50. [CrossRef] 49. P. Bressloff, S. Coombes. 1998. Desynchronization, Mode Locking, and Bursting in Strongly Coupled Integrate-and-Fire Oscillators. Physical Review Letters 81:10, 2168-2171. [CrossRef] 50. Eve Marder. 1998. FROM BIOPHYSICS TO MODELS OF NETWORK FUNCTION. Annual Review of Neuroscience 21:1, 25-45. [CrossRef] 51. S. Coombes, G. Lord. 1997. Intrinsic modulation of pulse-coupled integrate-and-fire neurons. Physical Review E 56:5, 5809-5818. [CrossRef]

52. David Terman, Euiwoo Lee. 1997. Partial Synchronization in a Network of Neural Oscillators. SIAM Journal on Applied Mathematics 57:1, 252. [CrossRef] 53. C. Vreeswijk. 1996. Partial synchronization in populations of pulse-coupled oscillators. Physical Review E 54:5, 5522-5537. [CrossRef] 54. S. Coombes, S. Doole. 1996. Neuronal populations with reciprocal inhibition and rebound currents: Effects of synaptic and threshold noise. Physical Review E 54:4, 4054-4065. [CrossRef] 55. William W. Lytton . 1996. Optimizing Synaptic Conductance Calculation for Network SimulationsOptimizing Synaptic Conductance Calculation for Network Simulations. Neural Computation 8:3, 501-509. [Abstract] [PDF] [PDF Plus] 56. Wei-Ping Wang . 1996. Binary-Oscillator Networks: Bridging a Gap between Experimental and Abstract Modeling of Neural NetworksBinary-Oscillator Networks: Bridging a Gap between Experimental and Abstract Modeling of Neural Networks. Neural Computation 8:2, 319-339. [Abstract] [PDF] [PDF Plus] 57. Galina N. Borisyuk, Roman M. Borisyuk, Alexander I. Khibnik, Dirk Roose. 1995. Dynamics and bifurcations of two coupled neural oscillators with different connection types. Bulletin of Mathematical Biology 57:6, 809-840. [CrossRef] 58. Seung Han, Christian Kurrer, Yoshiki Kuramoto. 1995. Dephasing and Bursting in Coupled Neural Oscillators. Physical Review Letters 75:17, 3190-3193. [CrossRef] 59. Rolf K�tter, Jeff Wickens. 1995. Interactions of glutamate and dopamine in a computational model of the striatum. Journal of Computational Neuroscience 2:3, 195-214. [CrossRef] 60. Farzan Nadim, �ystein H. Olsen, Erik Schutter, Ronald L. Calabrese. 1995. Modeling the leech heartbeat elemental oscillator I. Interactions of intrinsic and synaptic currents. Journal of Computational Neuroscience 2:3, 215-235. [CrossRef] 61. Ronald L. Calabrese, Farzan Nadim, �ystein H. Olsen. 1995. Heartbeat control in the medicinal leech: A model system for understanding the origin, coordination, and modulation of rhythmic motor patterns. Journal of Neurobiology 27:3, 390-402. [CrossRef] 62. Maureen E. Rush, John Rinzel. 1994. Analysis of bursting in a thalamic neuron model. Biological Cybernetics 71:4, 281-291. [CrossRef] 63. Alain Destexhe, Zachary F. Mainen, Terrence J. Sejnowski. 1994. Synthesis of models for excitable membranes, synaptic transmission and neuromodulation using a common kinetic formalism. Journal of Computational Neuroscience 1:3, 195-230. [CrossRef] 64. Alain Destexhe. 1994. Oscillations, complex spatiotemporal behavior, and information transport in networks of excitatory and inhibitory neurons. Physical Review E 50:2, 1594-1606. [CrossRef]

65. Thomas LoFaro , Nancy Kopell , Eve Marder , Scott L. Hooper . 1994. Subharmonic Coordination in Networks of Neurons with Slow ConductancesSubharmonic Coordination in Networks of Neurons with Slow Conductances. Neural Computation 6:1, 69-84. [Abstract] [PDF] [PDF Plus] 66. David Golomb, John Rinzel. 1993. Dynamics of globally coupled inhibitory neurons with heterogeneity. Physical Review E 48:6, 4810-4814. [CrossRef] 67. Frances K. Skinner, Gina G. Turrigiano, Eve Marder. 1993. Frequency and burst duration in oscillating neurons and two-cell networks. Biological Cybernetics 69:5-6, 375-383. [CrossRef] 68. CC Canavier, DA Baxter, JH ByrneRepetitive Action Potential Firing . [CrossRef] 69. Xiao-Jing WangNeural Oscillations . [CrossRef]

Communicated by Geoffrey Hinton

Feature Extraction Using an Unsupervised Neural Network Nathan Intrator Center for Neural Science, Brown University Providence, NO2912 USA

A novel unsupervised neural network for dimensionality reduction that seeks directions emphasizing multimodality is presented, and its connection to exploratory projection pursuit methods is discussed. This leads to a new statistical insight into the synaptic modification equations governing learning in Bienenstock, Cooper, and Munro (BCM) neurons (1982). The importance of a dimensionality reduction principle based solely on distinguishing features is demonstrated using a phoneme recognition experiment. The extracted features are compared with features extracted using a backpropagation network.

1 Introduction When a classification of high-dimensional vectors is sought, the curse of dimensionality (Bellman 1961) becomes the main factor affecting the classification performance. The curse of dimensionality is due to the inherent sparsity of high-dimensional spaces, implying that, in the absence of simplifying assumptions, the amount of training data needed to get reasonably low variance estimators is ridiculously high. This has led many researchers in recent years to construct methods that specifically avoid this problem (see Geman et al. 1991 for review in the context of neural networks). One approach is to assume that important structure in the data actually lies in a much smaller dimensional space, and therefore try to reduce the dimensionality before attempting the classification. This approach can be successful if the dimensionality reduction/ feature extraction method loses as little relevant information as possible in the transformation from the high-dimensional space to the low-dimensional one. Performing supervised feature extraction using the class labels is sensitive to the dimensionality in a similar manner to a high-dimensional classifier, and may result in a strong bias to the training data leading to poor generalization properties of the resulting classifier (Barron and Barron 1988). A general class of unsupervised dimensionality reduction methods, called exploratory projection pursuit, is based on seeking interesting projections of high-dimensional data points (Kruskal 1972; Friedman and Neural Computation 4, 98-107 (1992)


Feature Extraction Using an Unsupervised Neural Network

99

Tukey 1974; Friedman 1987; Huber 1985, for review). The notion of interesting projections is motivated by an observation made by Diaconis and Freedman (1984) that for most high-dimensional clouds, most lowdimensional projections are approximately normal. This finding suggests that important information in the data is conveyed in those directions whose single-dimensional projected distribution is far from gaussian. Various projection indices differ on the assumptions about the nature of deviation from normality, and in their computational efficiency. Friedman (1987) argues that the most computationally efficient measures are based on polynomial moments. However, polynomial moments heavily emphasize departure from normality in the tails of the distribution (Huber 1985). Moreover, although many synaptic plasticity models are based on second-order statistics and lead to extraction of the principal components (Sejnowski 1977; von der Malsburg 1973; q a 1982; Miller 1988; Linsker 1988), second-order polynomials are not sufficient to characterize the important features of a distribution (see examples in Duda and Hart 1973, p. 212). This suggests that in order to use polynomials for measuring deviation from normality, higher order polynomials are required, and care should be taken to avoid their oversensitivity to outliers. In this paper, the observation that high-dimensional clusters translate to multimodal low-dimensional projections is used for defining a measure of multimodality for seeking interesting projections. In some special cases, where the data are known in advance to be bimodal, it is relatively straightforward to define a good projection index (Hinton and Nowlan 1990), however, when the structure is not known in advance, defining a general multimodal measure of the projected data is not straightforward, and will be discussed in this paper. There are cases in which it is desirable to make the projection index invariant under certain transformations, and maybe even remove second-order structure (see Huber 1985 for desirable invariant properties of projection indices). In those cases it is possible to make such transformations beforehand (Friedman 1987), and then assume that the data possess these invariant properties. 2 Feature Extraction Using ANN

In this section, the intuitive idea presented above is used to form a statistically plausible objective function whose minimization will find those projections having a single-dimensional projected distribution that is far from gaussian. This is done using'a loss function that has an expected value that leads to the desired projection index. Mathematical details are given in Intrator (1990). Before presenting our version of the loss function, we review some necessary notation and assumptions. Consider a neuron with input vector x = ( X I ,..., xN), synaptic weight vector m = (ml,... , m ~ )both , in

Nathan Intrator

100

RN,and activity (in the linear region) c

-

=x

. m. Define the threshold

0, = E [ ( x m)’], and the functions $(c, 0,) = c2 - (2/3)~0m,4(c, 0,) =

c2 - (4/3)c0,. The 4 function has been suggested as a biologically plausible synaptic modification function to explain visual cortical plasticity (Bienenstock et al. 1982). 0, is a dynamic threshold that will be shown later to have an effect on the sign of the synaptic modification. The input x , which is a stochastic process, is assumed to be of Type I1 p mixing,’ bounded, and piecewise constant. These assumptions are plausible, since they represent the closest continuous approximation to the usual training algorithms, in which training patterns are presented at random. The p mixing property allows for some time dependency in the presentation of the training patterns. These assumptions are needed for the approximation of the resulting deterministic gradient descent by a stochastic one (Intrator and Cooper 1991). For this reason we use a learning rate p that has to decay in time so that this approximation is valid. We want to base the projection index on polynomial moments of low order, and to use the fact that a projection that leads to a bimodal distribution is already interesting, and any additional mode in the projected distribution should make the projection even more interesting. With this in mind, consider the following family of loss functions that depends on the synaptic weight vector m and on the input x ;

The motivation for this loss function can be seen in Figure I, which represents the $ function and the associated loss function Lm(c). For simplicity the loss for a fixed threshold 0, and synaptic vector m can be written as Lm(c)= - ( p / 3 ) c 2 ( c - em), where c = ( x . m). The graph of the loss function shows that for any fixed m and Om,the loss is small for a given input x , when either c = x . m is close to zero, or when x . m is larger than 0,. Moreover, the loss function remains negative for ( x .m) > Om,therefore any kind of distribution at the right-hand side of 0, is possible, and the preferred ones are those that are concentrated further from 0,. It remains to be shown why it is not possible that a minimizer of the average loss will be such that all the mass of the distribution will be concentrated on one side of 0,. This can not happen because the threshold 0, is dynamic and depends on the projections in a nonlinear way, namely, 0, = E ( x . ~ )This ~ . implies that 0, will always move itself to a position such that the distribution will never be concentrated at only one of its sides. The risk (expected value of the loss) is given by Rm

=

P

--

3

{ E [ ( x * W I ) ~] E ’ [ ( x . m)’]}

‘The (p mixing property specifies the dependency of the future of the process on its past.


101

Figure 1: The function (b and the loss functions for a fixed rn and 0,. Since the risk is continuously differentiable, its minimization can be achieved via a gradient descent method with respect to m, namely

The resulting differential equations give a modified version of the law governing synaptic weight modification in the BCM theory for learning and memory (Bienenstock et al. 1982). This theory was presented to account for various experimental results in visual cortical plasticity. The modification lies in the way the threshold 0, is calculated. In the original form this threshold was 0, = EP(c) for p > 1, while in the current form 0, = E(cP) for p > 1. The latter takes into account the variance of the activity (for p = 2) and therefore is always positive; this ensures stability even when the average of the inputs is zero. The biological relevance of the theory has been extensively studied (Bear et al. 1987; Bear and Cooper 1988) and it was shown that the theory is in agreement with the classical deprivation experiments (Clothiaux et al. 1991). The fact that the distribution has part of its mass on both sides of 0, makes this loss a plausible projection index that seeks multimodalities. However, we still need to reduce the sensitivity of the projection index to outliers, and for full generality, allow any projected distribution to be shifted so that the part of the distribution that satisfies c < 0, will have its mode at zero. The oversensitivity to outliers is addressed by considering a nonlinear neuron in which the neuron’s activity is defined to be c = a(x.m),where a usually represents a smooth sigmoidal function. A more general definition that would allow symmetry breaking of the projected distributions, as well as provide a solution to the second problem raised above, and will still be consistent with the statistical formulation, is c = o ( x . m - a ) , for an arbitrary threshold a. The threshold a can be found by using gradient descent as well. For the nonlinear neuron, 0, is defined

Nathan Intrator

102

to be 0,

= E[a*(x. m ) ] .The

loss function is given by

The gradient of the risk becomes -V,Rm

=p

E[d (u(x m ) ,0,)

u’x]

where u’ represents the derivative of u at the point ( x . m). Note that the multiplication by u’ reduces sensitivity to outliers of the differential equation since for outliers u‘ is close to zero. The gradient descent is valid, provided that the risk is bounded from below. Based on this formulation, a network of Q identical nodes may be constructed. All the neurons in this network receive the same input and inhibit each other, so as to extract several features in parallel. The relation between this network and the network studied by Cooper and Scofield (1988) is discussed in Intrator and Cooper (1991). The activity of neuron k in the network is defined as ck = u ( x .?nk - nk),where mk is the synaptic weight vector of neuron k, and Nk is its threshold. The inhibited activity and threshold of the kth neuron are given by ck = ck - 77 &#k Cj, @, = E[ci].A more general inhibitory pattern such as a Mexican hat is possible with minor changes in the mathematical details. We omit the derivation of the synaptic modification equations, and present only the resulting stochastic modification equations for a synaptic vector mk in a lateral inhibition network of nonlinear neurons:

hk

= /L [$(ck, bk,)d(sk) - 7

$(zj,

@,i)d(?f)]x

i#k

The lateral inhibition network performs a direct search of Q-dimensional projections in parallel, and therefore may find a richer structure that a step wise approach may miss (see example 14.1 in Huber 1985). 3 Comparison with Other Feature Extraction Methods

The above feature extraction method has been applied so far to various high-dimensional classification problems: extracting rotation invariant features from 3D wire-like objects (Intrator and Gold 1991) based on a set of sophisticated psychophysical experiments (Edelman and Bulthoff 1991); feature extraction from the TIMIT speech data base using Lyon’s Cochlea model (Intrator and Tajchman 1991). The dimensionality of the feature extraction problem for these experiments was 3969 and 5500 dimensions, respectively. It is surprising that a very moderate amount of training data was needed for extracting robust features as will be shown below. In this section we briefly describe a linguistically motivated feature extraction experiment from stop consonants. We compare classification performance of the proposed method to a network that performs


i

Unsupervised Feature Extractlon

U

01

1

20

r

Labeis

103

LowDim Classifier

-

Figure 2: Low-dimensional classifier is trained on features extracted from the high-dimensional data. Training of the feature extraction network stops when the misclassification rate drops below a predetermined threshold on either the same training data (cross-validatorytest) or on different testing data. dimensionality reduction based on minimization of misclassification error (using backpropagation with MSE criterion). In the latter we regard the hidden unit representation as a new reduced feature representation of the input space. Classification on the new feature space was done using backpropagation.2 The unsupervised feature extraction/classification method is presented in Figure 2. The pixel images corresponding to speech data, are shown in Figure 3. Similar approaches using the RCE and backpropagation network have been carried out by Reilly et al. (1988). The following describes the linguistic motivation of the experiment. Consider the six stop consonants [p,k,t,b,g,dl, which have been a subject of recent research in evaluating neural networks for phoneme recognition (see review in Lippmann 1989). According to phonetic feature theory, these stops possess several common features, but only two distinguishing phonetic features, place of articulation and voicing (see Lieberman and Blumstein 1988, for a review and related references on phonetic feature theory). This theory suggests an experiment in which features extracted from unvoiced stops can be used to distinguish place of articulation in voiced stops as well. It is of interest if these features can be found from a single speaker, how sensitive they are to voicing and whether they are speaker invariant. The speech data consists of 20 consecutive time windows of 32 msec with 30 msec overlap, aligned to the beginning of the burst. In each time window, a set of 22 energy levels is computed. These energy levels cor%e Intrator (1990) for comparisonwith principal components feature extraction and with k-NN as a classifier.

104

Nathan Intrator

Figure 3: An average of the six stop consonants followed by the vowel [a]. Their order from left to right [pa] [ba] [ka] [gal ita] [da]. Time increases from the burst release on the X axis, and frequency increases on the Y axis. Brighter areas correspond to stronger energy.

respond to Zwicker critical band filters (Zwicker 1961). The consonantvowel (CV) pairs were pronounced in isolation by native American speakers (two male BSS and LTN, and one female JES.) Additional details on biological motivation for the preprocessing, and linguistic motivation related to child language acquisition can be found in Seebach (1990). An average (over 25 tokens) of the six stop consonants followed by the vowel [a] is presented in Figure 3. All the images are smoothed using a moving average. One can see some similarities between the voiced and unvoiced stops especially in the upper left corner of the image (high frequencies beginning of the burst) and the radical difference between them in the low frequencies. In the experiments reported here, five features were extracted from the 440 dimension original space. Although the dimensionality reduction methods were trained only with the unvoiced tokens of a single speaker, the classifier was trained on (five-dimensional)voiced and unvoiced data from the other speakers as well. The classification results, which are summarized in Table 1, show that the backpropagation network does well in finding structure useful for classification of the trained data, but this structure is more sensitive to voicing. Classification results using a BCM network suggest that for this specific task structure that is less sensitive to voicing can be extracted, even though voicing has significant effects on the speech signal itself. The results also suggest that these features are more speaker invariant. The difference in performance between the two feature extractors may be partially explained by looking at the synaptic weight vectors (images) extracted by both methods (Fig. 4): For the backpropagation feature extraction it can be seen that although five units were used, less features were extracted. One of the main distinctions between the unvoiced stops in the training set is the high frequency burst at the beginning of the consonant (the upper left corner). The backpropagation method concentrated mainly on this feature, probably because it is sufficient to base the recognition of the training set on this feature, and the fact that training


105

Table 1: Percentage of Correct Classificationof Place of Articulation in Voiced and Unvoiced Stops. Place of articulation classification (B-P)

BSS /p,k,t/ BSS /b,g,d/ LTN /p,k,t/ LTN /b,g,d/ JES (both)

B-P (%)

BCM (%)

100 83.4 95.6 78.3 88.0

100 94.7 97.7 93.2 99.4

Figure 4: Synaptic weight images of the five hidden units of backpropagation (top), and the five BCM neurons (bottom).

stops when misclassification error falls to zero. On the other hand, the BCM method does not try to reduce the misclassification error and is able to find a richer, linguistically meaningful structure, containing burst locations and format tracking of the three different stops that allowed a better generalization to other speakers and to voiced stops. The network and its training paradigm present a different approach to speaker independent speech recognition. In this approach the speaker variability problem is addressed by training a network that concentrates mainly on the distinguishing features of a single speaker, as opposed to training a network that concentrates on both the distinguishing and common features, on multispeaker data.

106

Nathan Intrator

Acknowledgments

I wish to thank Leon N. Cooper for suggesting the problem and for providing many helpful hints and insights. Geoff Hinton made invaluable comments. The application of BCM to speech is discussed in more detail in Seebach (1991) and in a forthcoming article (Seebach and Intrator, in press). Charles Bachmann assisted in running the backpropagation experiments. Research was supported by the National Science Foundation, the Army Research Office, and the Office of Naval Research.

References Barron, A. R., and Barron, R. L. 1988. Statistical learning networks: A unifying view. In Computing Science and Statistics: Proc. 20th Symp. Interface, E. Wegman, ed., pp. 192-203. American Statistical Association, Washington,

Dc. Bear, M. F., and Cooper, L. N 1988. Molecular mechanisms for synaptic modification in the visual cortex: Interaction between theory and experiment. In Neuroscience and Connectionist Theory, M. Gluck and D. Rumelhart, eds., pp. 65-94. Lawrence Erlbaum, Hillsdale, NJ. Bear, M. F., Cooper, L. N, and Ebner, F. F. 1987. A physiological basis for a theory of synapse modification. Science 237, 42-48. Bellman, R. E. 1961. Adaptive Control Processes. Princeton University Press, Princeton, NJ. Bienenstock, E. L., Cooper, L. N, and Munro, P. W. 1982. Theory for the development of neuron selectivity: Orientation specificity and binocular interaction in visual cortex. I. Neurosci. 2, 3248. Clothiaux, E. E., Cooper, L. N, and Bear, M. F. 1991. Synaptic plasticity in visual cortex: Comparison of theory with experiment. I. Neurophysiol. To appear. Cooper, L. N, and Scofield, C. L. 1988. Mean-field theory of a neural network. Proc. Natt. Acad. Sci. U S A . 85, 1973-1977. Diaconis, P., and Freedman, D. 1984. Asymptotics of graphical projection pursuit. Ann. Statist. 12, 793-815. Duda, R. O., and Hart, l? E. 1973. Pattern Classification and Scene Analysis. John Wiley, New York. Edelman, S., and Biilthoff, H. H. 1991. Canonical views and the representation of novel three-dimensional objects. To appear. Friedman, J. H. 1987. Exploratory projection pursuit. J. Arner. Statist. Assoc. 82, 249-266. Friedman, J. H., and Tukey, J. W. 1974. A projection pursuit algorithm for exploratory data analysis. IEEE Trans. Comput. C(23), 881-889. Geman, S., Bienenstock, E., and Doursat, R. 1991. Neural networks and the bias-variance dilemma. To appear. Hinton, G. E., and Nowlan, S. J. 1990. The bootstrap Widrow-Hoff rule as a cluster-formation algorithm. Neural Comp. 2(3), 355-362.


107

Huber, P. J. 1985. Projection pursuit (with discussion). Ann. Statist. 13, 435-475. Intrator, N., 1990. Feature extraction using an unsupervised neural network. In Proceedings of the 1990 Connectionist Models Summer School, D. S. Touretzky, J. L. Ellman, T. J. Sejnowski, and G. E. Hinton, eds., pp. 310-318. Morgan Kaufmann, San Mateo, CA. Intrator, N., and Cooper, L. N 1991. Objective function formulation of the BCM theory of visual cortical plasticity: Statistical connections, stability conditions. Neural Networks. To appear. Intrator, N., and Gold, J. I. 1991. Three-dimensional object recognition of gray level images: The usefulness of distinguishing features. To appear. Intrator, N., and Tajchman, G. 1991. Supervised and unsupervised feature extraction from a cochlear model for speech recognition. In Neural Networks for Signal Processing - Proceedings of the 1992 IEEE Workshop, B. H. Juang, S. Y. Kung, and C. A. Kamm, eds., pp. 460-469. Kruskal, J. B. 1972. Linear transformation of multivariate data to reveal clustering. In Multidimensional Scaling: The0ry and Application in the Behavioral Sciences, I , Theory, R. N. Shepard, A. K. Romney, and S. 8.Nerlove, eds., pp. 179-191. Seminar Press, New York and London. Lieberman, P., and Blumstein, S. E. 1988. Speech Physiology, Speech Perception, and Acoustic Phonetics. Cambridge University Press, Cambridge. Linsker, R. 1988. Self-organization in a perceptual network. IEEE. Comp. 88, 105-117. Lippmann, R. P. 1989. Review of neural networks for speech recognition. Neural Comp. 1, 1-38. Miller, K. D. 1988. Correlation-based models of neural development. In Neuroscience and Connectionist Theory, M. Gluck and D. Rumelhart, eds., pp. 267353. Lawrence Erlbaum, Hillsdale, NJ. Oja, E. 1982. A simplified neuron model as a principal component analyzer. Math. Biol. 15, 267-273. Reilly, D. L., Scofield, C. L., Cooper, L. N, and Elbaum, C. 1988. Gensep: A multiple neural network with modifiable network topology. INNS Conf. Neural Networks. Seebach, 8. S. 1991. Evidence for the development of phonetic property detectors in a neural net without innate knowledge of linguistic structure. Ph.D. dissertation, Brown University. Seebach, B. S., and Intrator, N. A neural net model of perinatal inductive acquisition of phonetic features. Sejnowski, T. J. 1977. Storing covariance with nonlinearly interacting neurons. J. Math. Biol. 4, 303-321. von der Malsburg, C. 1973. Self-organization of orientation sensitivity cells in the striate cortex. Kybernetik 14,85-100. Zwicker, E. 1961. Subdivision of the audible frequency range into critical bands (frequenzgruppen). J. Acoust. SOC. Am. 33(2): 248.

Received 18 March 1991; accepted 20 May 1991.

This article has been cited by: 2. Q.Q. Huynh, L.N. Cooper, N. Intrator, H. Shouval. 1998. Classification of underwater mammals using feature extraction based on time-frequency analysis and BCM theory. IEEE Transactions on Signal Processing 46:5, 1202-1207. [CrossRef] 3. Y. Dotan, N. Intrator. 1998. Multimodality exploration by an unsupervised projection pursuit neural network. IEEE Transactions on Neural Networks 9:3, 464-472. [CrossRef] 4. Robert L. Goldstone. 1998. PERCEPTUAL LEARNING. Annual Review of Psychology 49:1, 585-612. [CrossRef] 5. Harel Shouval, Nathan Intrator, C. Charles Law, Leon N Cooper. 1996. Effect of Binocular Cortical Misalignment on Ocular Dominance and Orientation SelectivityEffect of Binocular Cortical Misalignment on Ocular Dominance and Orientation Selectivity. Neural Computation 8:5, 1021-1040. [Abstract] [PDF] [PDF Plus] 6. Suzanna Becker, Mark Plumbley. 1996. Unsupervised neural network learning procedures for feature extraction and classification. Applied Intelligence 6:3, 185-203. [CrossRef] 7. Ying Zhao, C.G. Atkeson. 1996. Implementing projection pursuit learning. IEEE Transactions on Neural Networks 7:2, 362-373. [CrossRef] 8. David J. Field . 1994. What Is the Goal of Sensory Coding?What Is the Goal of Sensory Coding?. Neural Computation 6:4, 559-601. [Abstract] [PDF] [PDF Plus] 9. Shimon Edelman. 1993. Representing three-dimensional objects by sets of activities of receptive fields. Biological Cybernetics 70:1, 37-45. [CrossRef] 10. Nathan Intrator . 1993. Combining Exploratory Projection Pursuit and Projection Pursuit Regression with Application to Neural NetworksCombining Exploratory Projection Pursuit and Projection Pursuit Regression with Application to Neural Networks. Neural Computation 5:3, 443-455. [Abstract] [PDF] [PDF Plus] 11. Nathan Intrator , Joshua I. Gold . 1993. Three-Dimensional Object Recognition Using an Unsupervised BCM Network: The Usefulness of Distinguishing FeaturesThree-Dimensional Object Recognition Using an Unsupervised BCM Network: The Usefulness of Distinguishing Features. Neural Computation 5:1, 61-74. [Abstract] [PDF] [PDF Plus]

Communicated by Alex Waibel

Speaker-Independent Digit Recognition Using a Neural Network with Time-Delayed Connections K. P. Unnikrishnan" Molecular Biophysics Research Department, AT&T Bell Laboratories, Murray Hill, N] 07974 USA

J. J. Hopfield Molecular Biophysics Research Department, AT&T Bell Laboratories, Murray Hill, N] 07974 U S A and Divisions of Chemistry and Biology, California Institute of Technology, Pasadena, C A 91125 U S A D. W. Tank Molecular Biophysics Research Department, AT&T Bell Laboratories, Murray Hill, N ] 07974 USA

The capability of a small neural network to perform speaker-independent recognition of spoken digits in connected speech has been investigated. The network uses time delays to organize rapidly changing outputs of symbol detectors over the time scale of a word. The network is data driven and unclocked. To achieve useful accuracy in a speakerindependent setting, many new ideas and procedures were developed. These include improving the feature detectors, self-recognition of word ends, reduction in network size, and dividing speakers into natural classes. Quantitative experiments based on Texas Instruments (TI) digit data bases are described.

1 Introduction Accurate recognition of spoken words in connected speech is difficult to achieve with limited computational resources. A "neural network approach using time delays to organize the incoming signal into a recognizable form was constructed in earlier work, and studied in detail for *Present address: Computer Science Department, GM Research Laboratories, Warren, MI 48090-9055 USA. Neural Computation 4,108-119 (1992) @ 1992 Massachusetts Institute of Technology

Speaker-IndependentDigit Recognition

109

the case of a single speaker (Unnikrishnan et al. 1988,1991). The case of a single speaker is, however, notoriously easier than speaker-independent word recognition, and is of rather limited utility in the world of engineering compared to the case of speaker independence. The present paper studies time-delay networks for speaker-independent connected speech. The problem of identifying the spoken digits 0-9 in connected speech was chosen because it is well defined, small enough to study in detail, and has an established data base used as a standard for intercomparisons of results (Leonard 1984). This data base is sufficiently diverse that adequate performance on it is believed to be sufficient for field use in the United States. In addition, this particular problem is sufficiently important that a compact, low cost, and low power-consumption solution to it would be commercially useful. The multiple-speaker problem is much more difficult than the singlespeaker case, and its adequate solution demands many additional ideas and methods not present in our previous studies. Based on how well the original network performed on a speaker-dependent data base, we set out to examine whether a small number of networks could be used in parallel to solve the more difficult speaker-independent problem. Each subnetwork would be optimized on a separate cluster of data, for example, males or females. Because it is simple to train networks that make few mistakes of erroneous recognition, parallel use of multiple networks is a feasible approach to the general problem. In the course of these studies we found that even when the data were clustered into a few simpler problems, recognition accuracy was inadequate. Changes were therefore made to improve network performance. The most important of these changes are improved front-end signal processing for more reliable generation of invariant features from the input speech, reduction in the size of the network to favor generalization over memorization in the learning process, using the network itself to recognize what to learn, automatic segmentation of spoken digits from multiword strings, and explorations of dividing speakers into natural classes to simplify the problem faced by a single network. In this paper we describe the performance of the various networks and approaches, presenting critical experiments for deciding to incorporate or abandon particular ideas and structures in the overall scheme. These results are described approximately in the order in which they were obtained. They begin with the obvious: using the same network that had proved successful for the single-speaker problem on the multiple-speaker data base. They conclude with experiments on a much-improved network and a data base of male speakers only (having found along the way that a single network shares with the simple hidden Markov model (HMM) performance at only a moderate level when men and women are placed together in the data base). The size and complexity of the networks simulated are such that an analog CMOS implementation would require less than a square centimeter.

K. P. Unnikrishnan, J. J. Hopfield, and D. W. Tank

110

2 Network Architecture and Learning Algorithm The conceptual design of the word-recognition neural network is sketched in Figure 1. Details of the architecture and the learning algorithm have been described elsewhere (Unnikrishnan et al. 1991) and here we give only a very brief summary of it.

w

v

*

BANDPASS FILTERS FEATURE M s\;o; DETECTORS

$oqi"->

z 2 !=w zk az

03

B a

" V

"

Figure 1: Block diagram of the speech recognition circuit with 32 bandpass filters. The feature detectors use a "center-surround" mechanism and the responses are generated by comparing outputs of neighboring filters. In some of the experiments, a different front end was used. It consisted of 15 bandpass filters and a zero crossing detector. The zero crossing detector uses raw speech waveform. The time delays used in the network have gaussian profiles with different widths. The connection matrix is learned using a learning algorithm and the rest of the circuit is fixed.


111

The analog speech waveform first goes to a bank of parallel analog frequency band filters. The rectified output from this filter bank is sent to a feature detector network that identifies the presence of short time scale features by a competitive comparison of the outputs of the filter bank. This procedure converts the original single-channel analog signal to multiple channel binary outputs crudely describing the frequency location of three significant peaks in the power spectrum. These outputs change much more slowly in time than the input signal does, and provide a suitable pattern to be recognized by neural network approaches. The multiple channel signal is sent down parallel tapped dispersive delay lines. The recognition unit for a particular word is connected by analog "weights" to these delay lines and the outputs of these nonlinear units signify recognition of the completion of particular words. The weights in the network are learned using an algorithm that minimizes a mutual discrimination error measure (see Hopfield 1987 and Unnikrishnan et al. 1991 for details of the learning algorithm). By storing the delay information at discrete time intervals, the learning problem for a particular recognition unit can be reduced to a single layer of weights in an analog network with many inputs and a single output. The learning rule uses the analog response of the output units as a representation of the probability of a word having been said. For a given set of data, gradient descent leads to a unique set of weights for this learning rule and network structure. To compensate for the temporal distortions, we have used dispersive time delays in the network (see also Tank and Hopfield 1987). These delays have gaussian profiles with different widths. In addition, each recognition unit also has an integration time constant. The summed signal from all delays is filtered with this time constant at the input of each recognition unit. Hence there are two parameters that determine the temporal tolerance of the network: (1) the width of the time delays ([T) and (2) the integration time constant of recognition units ( ~ ~ ~ 1 . 3 Data Base and Scoring Protocols

All the results reported here are based on two spoken digit data bases from TI. The TI isolated digit data base used consists of two utterances of each digit by 112 speakers, a regionally balanced mixture containing both men and women. These files were divided into a training set containing 50 speakers and a test set of 62 speakers. There is an appreciable variance in the distribution of utterance lengths. The average fractional time distortion [(longest - shortest)/average] for the training data was about 92%, but individual cases were as high as 157%. The TI connected digit data base contains 330 speakers, a balanced mixture including children as well as men and women. There are two examples from each speaker of each individual digit, 11 examples from

112


each speaker of spoken digit pairs, and strings of up to 7 digits (Leonard 1984). The experimental results for any particular set of data, network structure, and connections can be described by the percentage of correctly identified digits. Two measures were used to evaluate performance and provide recognition accuracy scores. The first is the threshold score: according to this measure, a recognition is scored as correct only if the output of the correct recognition unit was above a threshold value near the end of the utterance, with all the incorrect recognition units remaining below this threshold throughout the utterance. The second performance measure is the area score: according to this measure, a recognition is scored as correct if the time integrated output of the correct recognition unit over the period of the utterance is larger than the integrated output of any of the incorrect units. The threshold criterion for recognition is required for word spotting (recognition of individual words independent of the context in which they occur). The area criterion requires segmentation of data for recognition and is analogous to the scoring procedure used in HMM models (Levinson et al. 1983). We include recognition accuracies according to the area criterion for comparison with the results of other groups. Real time recognition of words will be impossible with the area criterion without cumbersome algorithmic additions to an otherwise simple network structure. In the models that use such a criterion, the recognition is usually done by waiting until the end of the sequence and then doing multiple matches with respect to the number of words spoken and possible candidates. The threshold criterion is a more strict measure of performance of the network than the area criterion and hence in all cases the threshold score is lower. In the following text and table we give the recognition accuracy with the area criterion outside the parentheses and the accuracy with the threshold criterion within parentheses. 4 Results

It was previously demonstrated (Unnikrishnan et al. 1991) that on a single-speaker connected-digit data base, a network with learned timedelayed connectionshad a performance similar to that provided by HMM digit recognition systems. When trained on 288 utterances of the digits, the network was able to learn all of the training data (recognition accuracy - 100% with threshold and area criteria). It could recognize a test set of 144 utterances with an accuracy of 100% (99.3% with threshold criterion). To evaluate the extent to which this same network and training algorithm could solve the speaker-independent isolated digit recognition problem, it was trained on 500 utterances from the TI isolated digit data base. This data base contains a mixture of males, females, and children. The recognition accuracy on the training set was 98.6% [(91.4%

Speaker-Independent Digit Recognition

113

Table 1: Recognition Accuracy of Training a n d Testing Data with Various Network Configurations and Data Sets.'

Recognition accuracy ( W ) Row

Training data

Testing data

a

98.6 (91.4)

81.5 (61.8)

b

99.8 (98.4)

92.0 (78.0)

C

99.6 (96.1)

d

(98.5)

e

97.6 (90.0)

f

100 (99.5)

g

99.8 (93.7)

h

99.9 (95.2)

95.6 (81.1)

i

99.6 (93.5)

97.5 (84.9)

j

98.0 (83.3)

95.5 (75.4)

k

(92.6)

(82.4)

98.3 (92.6)

Comments 32 input channels, isolated digit data base, 50-speaker training set, 62-speaker test set As in case (a) but learning with selfjustification Learning on the combined sets of (a), 112 speakers As in (c) but with 15 frequency channels and the "unvoiced" channel Trained on 309 speakers, one example of each digit from each speaker Train on one utterance of 110 males, test on other utterances of same males Trained on 2090 one-word segments from two-digit strings of 110 males Train on 1056 segments from two-digit strings of 55 males marked as training set, tested on 1037 segments from other 55 males As in (h), but adding an additional 544 segments from three-digit strings of same speakers to training data; test data are same as in (h) As in (i) but adding 1100 isolated word files of same speakers to training data Recognition accuracy for male connected speech; trained on segments from connected speech set, tested on strings from the test set

"Recognition accuracy using area criteria is given outside the parentheses and the accuracy using threshold criteria is given within the parentheses. Rows a 4 contain recognition results on the TI isolated-digit data base and rows e-k contain results on the TI connected-digit data base. All results in rows e-k are using a front end with 15 frequency channels and an "unvoiced channel. See text for more details.

with threshold criterion); row a, Table I]. It recognized an independent test set (different speakers) of 620 utterances with an accuracy of 81.5% [(61.8%); row a, Table 11. These scores indicate that the circuitry and the learning paradigm as was used in the single speaker case was not sufficient for reliable recognition using the multiple-speaker data base. 4.1 Time-Duration Clustering. A series of experiments were done to determine the effects of temporal distortions on the network performance. In the first set of experiments, the data base was split into two clusters

114


(one containing the shorter utterances and the other containing the longer utterances) and separate networks trained on each one of them. In the next set of experiments, the time delays were made rigid. These did not change the network performance drastically, suggesting that for this data base, most of the difficulty may be due to variance in the frequency domain. 4.2 Frequency Clustering. Using a network trained on the entire data base, the files were split into two clusters: one containing high frequency utterances and the other containing low frequency utterances. Networks were able to learn the utterances in these clusters to better accuracies than those from an unbiased group taken from the same total data set. Also, a network trained on one cluster recognized test sets from the other cluster very poorly. These results demonstrate that spectral variance in acoustic features contribute substantially to the limited performance of the speech recognition network. We therefore adopted the premise that any complete recognition system would have two separate networks devoted to different frequency clusters and focused on improving the accuracy of a network for the male speaker data base subset. 4.3 Self-Justification. The speech examples were end-pointed by hand for use in the supervised learning procedure. But an analysis of the network outputs after learning showed that for many of the examples, the maximum response of the correct recognition unit was not at the assigned end point. This suggested that to generate optimal networks, the output of the network itself should be used for determining the end point of words. To accomplish this, the network was partially trained with the hand-justified end points. The time point of maximum response for the correct recognition unit, with these partially trained connections, was taken as the new end point and the training continued. This procedure implements self-justification of a11 the examples. This led to much better recognition of the training and testing data (compare row b with row a in Table l), decreasing the error rate by about a factor of two on the independent test set.

4.4 Data Limitations. The TI isolated word multiple speaker data base was studied with the 32-channel front-end described in Unnikrishnan et al. (1991). Row b in Table 1 shows the results for training sets and test sets of approximately equal size. The excellent recognition score of (98.7%)according to the stringent threshold criterion on the training set was not matched by the score of (78%) on the test set. This discrepancy indicates that the system is to some extent memorizing the voices of the particular 50 speakers in the training set, and that the ensemble used for training is not large enough.


115

To examine whether the network is in principle capable of correctly classifying all the data at once, it was trained on all the data, comprising all 112 speakers and 1120 speech files. Row c shows the results. By the area criterion, the classification was near perfect, and with the threshold criterion, the performance was at the 96%level. This result suggested that to be able to both train and test the system, more speakers and more data per connection would be necessary. It also suggested that the network was near the end of its capability, and that some improvements would prove necessary to reach high levels of performance on an adequately broad data set. The system requires too much data because it has too many connections to train. A typical word detector requires about 7 x 32 + 1 = 225 connections (on the average 7 time delays for each one of the 32 input channels and a bias value). While many more speech files than this are available, the similarity between different speakers or files is such that the amount of training data available is not adequately large. To alleviate this problem, we reduced the network size to 16 channels, with typically 7 x 16 + 1 = 113 connections per digit. 4.5 Zero-Crossing Detectors. The original 32 frequency band "front end" followed by a feature detector network was designed to locate peaks in the power spectrum and does not distinguish very well between vowel and consonant sounds or between voiced and unvoiced speech. A detector was designed to distinguish between voiced and unvoiced speech and used as one of the channels in a reduced 16 channel front end. A variety of methods can do this with high reliability. We chose a method based on zero crossings of the raw wave form as a method that would be easy to implement in analog hardware, and relatively independent of the intensity normalization of the sound. An impulse was generated at the time of each upward zero crossing of the raw speech signal. These impulses were filtered with an integration time constant of 0.005 sec. The unvoicing channel was turned on if the output of this filter corresponded to a zero crossing rate which was above 2000 crossings per second, and if the total power in the speech was above a threshold level. The output of this channel located unvoiced consonants x, s, f, v, and t in the data set with excellent reliability. Further explorations in this paper have all been based on a 16-channel system containing the zero crossing detector. The other 15 channels are frequency channels of the previous type (see Unnikrishnan ef al. 1991 for details), but having twice the frequency bandwidth. These channels were centered at the locations of the previous even-numbered channels 2-30. The feature detector network was modified slightly to prevent the identification of a peak in two adjacent frequency channels. The replacement of the 32-channel front end by the 16-channel system described above resulted in better performance on the entire 112 speaker data base (compare rows c and d in Table 1). Confronted with the necessity of

116

K. P.Unnikrishnan, J. J. Hopfield, and D. W. Tank

obtaining more data, and desiring to move toward connected speech, we began working with the much larger TI data base for connected speech. 4.6 Separating Males and Females. The TI connected-digit data base available to us contains a regionally balanced set of 309 speakers, including men, women, and children. When trained on one utterance of each speaker on each of the 10 digits spoken as an isolated word (3090 files), a relatively poor performance level [97.6%(90.0%);row e, Table 11 on the training set was achieved. Clearly the speech variation is now greater than the network can encompass. One major difference between the present data base and the previous one is the inclusion of children. Following the partitioning idea described earlier, we split the data base into two portions, males and nonmales. For the male training set of isolated words from the connected digit data base (consisting of one example of each digit spoken by all 110 speakers) the network could be trained to a high level of performance on the training set [loo% (99.5%);row f, Table 11. The poorer performance on the test set from the same speakers (row f, Table 1)indicates that there is still an inadequate number of speech files in the training set. However, more data were now available from strings of two and three digits.

4.7 Automatic Segmentation of Training Data. To obtain individual words necessary for training from digit strings without a large amount of hand segmentation, a bootstrap procedure was employed. To begin, the recognition network with the connections learned from one utterance of each digit from the 110 male speakers was used to label the ends of words in the connected speech files. The recognition score was 100% [(99.5%); row f, Table 11 on the training set and 98.3% [(92.6%);row f, Table 11 on the test set. These connections were then used to segment individual digits from two-digit strings. The system could now be trained on this larger data base. By iteration, the total training set size was ultimately increased to 2090 utterances. This training set could be recognized with an accuracy of 99.8% [(93.7%);row g, Table 11. Since the performance is lower by the threshold criteria than that obtained with the isolated digits database (row f, Table 11, we surmise that the recognition of segmented digits from strings is a harder problem than working with isolated words. The two obvious differences between this data base and the isolated word data base are the larger variation in the lengths of utterances and wordword coarticulation effects. This enlarged data set was split into training and test sets (55 speakers for training and 55 speakers for testing) yielding 1056 segmented words for training and 1037 words for testing. The network could be trained to recognize the test set with an accuracy of 99.9% [(95.2%);row h, Table 11. The test set was recognized with an accuracy of 95.6% [(81.1%);row h, Table 11. The fact that training set could be learned very well and not


117

the test set shows that the total number of files in the training set is still small. We proceeded to increase the size of the training set by segmenting digits from three-digit strings. Adding segments from three-digit strings yielded a total of 2600 training words. The network could be trained to recognize this training set with an accuracy of 99.6% [(93.5%);row i, Table 11 and to recognize the test data (same data set as in the previous case) with an accuracy of 97.5% 1(84.9%); row i, Table 11. But while the addition of new words to the training data increases the recognition accuracy for a test set, the continued poorer performance on the test set compared to the training set shows that there is still inadequate training data. An experiment was tried in which the isolated digits were added to the segmented connected digits training data. The recognition score on the training set was reduced to 98% [(83.3%); row j, Table 11 and the score on test set was reduced to 95.5% [(75.4%);row j, Table 11. The isolated digits typically have much longer duration than segmented digits from strings. The resultant additional variance in length is probably the cause of the reduced recognition accuracy. 4.8 Recognition of Strings. The experiments described above were done on isolated digits or digits segmented out from strings. We tested the performance of the network mentioned in row i of Table 1 on unsegmented connected digit strings from which the segmented test set had been previously produced. Some readjustment of delay-line parameters and integration time constants was necessary to eliminate the inhibitory signals from previous digits preventing recognition of a current digit. Such a network was able to recognize the training data with an threshold criteria accuracy of (92.4%) (row k, Table 1) and test data with a threshold accuracy of (82.4%) (row k, Table 1). We did not write the more complex software to do scoring for continuous speech by the area criterion, since this is not the desired ultimate use of the network. But by analogy with other experiments (cf, row i, Table 1) we would anticipate a recognition accuracy by the area criteria of approximately 99% on the training data set and 97% on the test data set. More conventional approaches to this problem, involving extensive high-speed digital processing and a combinatorial examination of locations for word boundaries, have been carried out on this data base by many others. Using a network that uses acoustic-phonetic features, Bush and Kopec (1987) achieved an accuracy of 96%. Rabiner et al. (1988) achieved an accuracy of 97.1% using a hidden Markov model. Our network can easily be implemented using low precision and low power analog circuitry. The connections would require only two bits of precision and an algebraic sign and the network has been shown to tolerate a considerable amount of noise (Unnikrishnan et al. 1991). While the experiments are not strictly comparable (the earlier work is on string recognition, and we have not made a complete study of all strings of all

118


nonchildren), the difference between them is comparable to that expected between threshold and area criteria within our studies. This indicates that the two approaches are extracting similar amounts of information from the sound signal (though not necessarily the same information), and that the major addition of the HMM procedure is to be able to work somewhat better with words of great length variability through massive computation. The direct neural network approaches to this problem are to use multiple or hierarchical time-scale networks (see also Waibel et al. 1989).

5 Conclusions We believe that the time-delay networks of this style we have studied are likely to be able to solve the speaker-independent digit recognition problem at a useful engineering level, with a neural network small enough to fit onto a single very low power analog VLSI chip. Even if four sets of connections (two sets of time delays and two sets of voice qualities) are needed, the total number of connections required is less than 6000. The four networks would share a common front end and delay network. The Intel 80170NW electrically trainable analog neural network chip based on eeprom technology already has 10,000 adjustable analog connections. The experiments we have described permit us to delineate the remaining problems and possible ways to solve them. First, the front-end needs some improvement. The very large increase in performance produced by the inclusion of a voicing detector is an indication that the substitution of one or two frequency filters by more clever feature detectors would be of enormous help. Even the frequency filters themselves are not optimal. The output of the filter bank often lacks a formant trajectory when that trajectory is clearly visible in a windowed FFT power spectrum. The variability of our front-end output compared to that of the WAVES program (Entropic Speech Inc.) suggests that better filters alone would be of considerable help. Second, the amount of available data in the TI data set is inadequate for the learning procedure used in the present study. It is, for example, responsible for the large difference in recognition accuracy between the test and the training set illustrated in row k of Table 1. A modified learning procedure that can capture outliers and generalizes better could be adopted, or alternatively a brute force approach of using a larger data set. For example, a variance model could be used in conjunction with the training set to effectively enlarge it. Third, speaker clusters should be produced by the networks directly. In the experiments described here, training data were clustered using males and nonmales as predefined categories. This would make each cluster more compact and simplify the problem.

Speaker-Independent Digit Recognition

119

Fourth, when connected speech and isolated words are combined in a single data base, the difference in the duration of a given word within the data now begins to matter. This problem can be circumvented by dividing the data into fast and slow categories by clustering as illustrated in the text, and training a network for each cluster. These networks could be run in parallel, since false recognitions are generally not a problem. The output of the best network can then be used for recognition. The alternative use of a hierarchy of two delay time scales is also attractive. Acknowledgments The TI connected digit data base was provided by the National Bureau of Standards. We wish to thank David Talkin for providing us the WAVES program and the Speech Research Department at Bell Labs for computer support. The work of J. J. H. at Caltech was supported in part by Office of Naval Research (Contract No. N00014-87-K-0377). References Bush, M. A., and Kopec, G. E. 1987. Network-based connected digit recognition. IEEE Trans. ASSP 35, 1401-1413. Hopfield, J. J. 1987. Learning algorithms and probability distributions in feedforward and feed-back networks. Proc. Natl. Acad. Sci. U S A . 84, 8429-8433. Leonard, G. E. 1984. A database for speaker-independent digit recognition. Proc. lntl. Conf. Acoustics Speech Signal Process. 3, 42.11.1-42.11.4. Levinson, S. E., Rabiner, L. R., and Sondhi, M. M. 1983. An introduction to the application of the theory of probabilistic functions of a Markov process to automatic speech recognition. Bell System Tech. J. 62, 1035-1074. Rabiner, L. R., Wilpon, J. G., and Soong, F. K. 1988. High performance connected digit recognition using hidden Markov models. Proc. lntl. Conf. Acoustics Speech Signal Process. S3.6, 119-122. Tank, D. W., and Hopfield, J. J. 1987. Concentrating information in time: Analog neural networks with applications to speech recognition problems. Proc. I E E E First lntl. Conf. Neural Networks, San Diego, CA. Unnikrishnan, K. P., Hopfield, J. J., and Tank, D. W. 1988. Learning Timedelayed connections in a speech recognition circuit. Abstr. Neural Networks Compuf. Conf., Snowbird, UT. Unnikrishnan, K. I?, Hopfield, J. J., and Tank, D. W. 1991. Connected-digit speaker-dependent speech recognition using a neural network with timedelayed connections. IEEE Transact. Signal Proc. 39, 698-713. Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., and Lang, K. 1989. Phoneme recognition using time-delay neural networks. IEEE Trans. ASSP 37, 328339. Received 11 February 1991; accepted 15 July 1991.

This article has been cited by: 2. B. K. Szymanski, G. G. Chen. 2007. Computing with Time: From Neural Networks to Sensor Networks. The Computer Journal 51:4, 511-522. [CrossRef]

Communicated by David Zipser

Local Feedback Multilayered Networks Paolo Frasconi Marco Gori Giovanni Soda Dipartimento di Sistemi e lnformatica, Via di S . Marta, 3-50139 Firenze, Italy In this paper, we investigate the capabilities of local feedback multilayered networks, a particular class of recurrent networks, in which feedback connections are only allowed from neurons to themselves. In this class, learning can be accomplished by an algorithm that is local in both space and time. We describe the limits and properties of these networks and give some insights on their use for solving practical problems. 1 Introduction

Recurrent networks have recently been investigated by several researchers (Bourlard 1989; Cleeremans et al. 1989; Elman 1988; Gherrity 1989; Gori et al. 1989; Pearlmutter 1989; Pineda 1987; Watrous 1988; Williams and Zipser 1989) because of these networks’ potential capability of coping with sequential tasks. The problem of learning has received particular attention. A review on this subject has recently been proposed by Pearlmutter (1990). He presents the basic contributions on the subject by distinguishing algorithms for full recurrent networks from those used for local feedback networks. In spite of the efforts made for discovering learning algorithms, insufficient attention has been paid to finding out which networks are suited for which problems. Such an investigation should provide design criteria for selecting architectures that are welltailored for solving different classes of problems. Instead of learning with huge full recurrent architectures, it is better to get rid of unnecessary connections because the network‘s generalization to new examples is likely to increase significantly. This paper is a first attempt to gain some knowledge concerning the behavior of particular recurrent networks, referred to as local feedback multilayered neural networks (Gori 1990). Only local feedback connections (self-loops) are assumed, and, therefore, the resulting architecture is very similar to Mozer’s focused backpropagation (Mozer 1988). These networks were mainly conceived for phoneme recognition problems in the attempt of capturing the dynamic nature of speech (Gori et al. 1989). Neural Computation 4, 120-130 (1992)


Local Feedback Multilayered Networks

121

The dramatic constraint on feedback connections make learning possible by means of an algorithm that is local in both space and time, and that has the same complexity as backpropagation (BE'). It is proven that these networks are particularly suited for exhibiting a forgetting behavior, which is a very desirable feature for a task like phoneme recognition. They can latch events that occur arbitrarily far in time, but they are very poor for dealing with arbitrarily Iong sequences. For example, a problem as easy as implementing a counter cannot be learned by a local feedback MLN. 2 Local Feedback MLNs and BPS'

The network class we consider relies on an MLN architecture in which some neurons exhibit dynamic behavior. The following definition sketches the hypotheses we assume. Definition 1. (Local Feedback MLNs) 1. The network has an MLN architecture. Static and dynamic neurons are considered. For convenience, we group neurons in the following sets: Z

1 1 1=I

input neuron set;

1x1= m

Ft = hidden neuron set; U = output neuron set; 2) = dynamic

IU1 = n

neuron set; D

C

7l U 0

where I . I denotes cardinality; 2. Static neurons perform a weighted sum of their inputs

a;(t)= C W j j X j ( t ) i

(2.1)

3. The activation of the dynamic neurons follows one of the equations Uj(t)

=

w;;a;(t - 1)

+ CWjjXj(f) i

a;(t) =

(local activation feedback MLN) WjjXj(t - 1) X W j j X j ( t )

+

(2.2)

i

(local output feedback MLN)

(2.3)

ieD only connections from network inputs can be accepted (jE 1) 'BPS was first derived in a slightly different form in Gori et al. 1989.

(2.4)

P. Frasconi, M. Gori, and G. Soda

122

Output layer

n

n

S e l f - 1oop c o n n e c t i o n s

Dynamic layer

Input layer

Figure 1: An example of local feedback MLN with dynamic neurons in the first hidden layer.

4. Each neuron has a sigmoidal output function f(a) A tanh(a/2). 5. The learning environment is composed of a sequence of frames and the supervision can be arbitrarily placed along the sequence. The error C with respect to the targets is evaluated according to (2.5) where 7 is the set of frame indexes, di(t) represents the desired output for unit i E 0, when feeding the network with frame t, and of is a binary switch that defines at which time t E 7 supervision takes place on output neurons. Figure 1 shows a typical example of local feedback MLN, in which all dynamic neurons are placed in the hidden layer (i.e., D = 7-0. Such an arrangement of dynamic neurons has been successfully employed for phoneme recognition tasks (Gori et al. 1989). Local feedback MLN can be derived from continuous models similar to that of Pearlmutter. A detailed analysis on discrete approximation of continuous recurrent nets, which enlightens the role of the model's parameters, can be found in Tsung (1990). It is worth mentioning that in our model, the term w;; of equations 2.2 and 2.3 provides the unit with a time constant which is responsible for dynamic behaviors. Learning is accomplished by means of a gradient descent algorithm. The previous hypotheses, particularly those concerning the architecture, make it possible to compute the gradient by means of an algorithm,


123

called backpropagation for sequences ( B E ) (Gori et al. 1989), which is local in both space and in time. This represents a significant advantage in using this kind of architecture. A similar algorithm was independently discovered by Mozer (1988) for his focused architecture.

Theorem 1. (on Learning in Local Feedback MLNs). In local feedback MLNs the following facts are true:

The gradient of cost (2.5) can be computed by (2.6)

where y , ( t ) needs to be computed only if superuision takes place at time t. The backward factor yi(t) can be computed by using the ordinary backpropagation backward step. If wij is a static weight, then the fonuard factor zi,(t) can be computed in exactly the same way as BP. For all weights connected to dynamic neurons, one of the following relationships holds:

+ Xj(t)(l- 6ij) + 6,Xj(t - 1) (output feedback) (2.7) wiizij(t - I ) + xj(t)(I - Sij) + 6ijaj(t - I ) (activation feedback) (2.8)

Zij(t)

= Wiif"Ui(t - l)]Z,j(t- 1)

zij(t)

=

with zjj(t = 0 ) = 0 6 , = 1 iff i = j

i E D , j E 1.

(2.9)

Proof. First, let us consider static weights. The architectural hypotheses imply that static neurons cannot feed dynamic ones and thus the path from each input to each output is entirely static, exactly as in BP. As a result, the forward term q ( t ) is simply reduced to xj(t). Second, for dynamic weights, 2.6 holds because of hypothesis 2.4. When a dynamic weight changes, the activation of the corresponding neuron also changes according to the forward factor of 2.6, whereas the change in the cost is related to the path connecting this neuron to its outputs; this path, according to 2.4, is a static one. It follows that yi(t) can be computed by using BP's ordinary backward step. Finally, by taking the weight derivatives of 2.2 and 2.3, we obtain 2.7 and 2.8. 0 It is worth mentioning that a learning procedure based on similar results can be also derived by assuming that the forward connections carry out multiple delays (Gori 1989).


124

3 Forgetting Behavior

Definition 2 (Forgetting Behavior). We say a recurrent network exhibits forgetting behavior if the following relationship holds:

When adopting local activation feedback MLNs, quite a simple result is that forgetting is strictly related to the stability of each neuron (Gori 1990). Moreover, this architecture turns out to have a canonical form, because it is equivalent to all possible networks having full connection in a given context layer (Frasconi et al. 1990). If output feedback is assumed, then a similar conclusion can be drawn concerning forgetting. Theorem 2 (Forgetting Behavior). Local output feedbuck MLNs exhibit forgetting behavior, provided that

lwiij < l / d where d

V i E 2,

= maxlf‘(a)]=

(3.2)

1/2.

Proof? Let us define (3.3) i E O,kE 3-1,j E Z

By using the chain rule, it is quite simple to prove the following: (3.4) k€U

Now let us define the static contribution h(t)

CWk,Xj(t)

(3.5)

1EZ

for each neuron k E 3-1. By using the chain rule again, we can compute $ i j ( t ) for each dynamic neuron as follows: (3.6)

Since f ’ ( . ) E (O,d], it can be proved by induction on 9 that the following inequality holds: (3.7) ’For the sake of simplicity, the proof is limited to networks having just one hidden layer, but it can be easily extended to general local feedback MLNs.


125

and thus, by using 3.4: (3.8)

lim$(f)

4-00

=0

(3.9)

These results suggest that the class of network we consider is very well suited for all applications in which the network output has to take relatively recent information into account. Successful results have been obtained by using this model for phoneme recognition, in which a forgetting behavior turns out to be very useful (Gori ef d . 1989). 4 Learning Sequences

There are several situations in which we want to recognize patterns composed of sequences. Tasks of this kind occur, for example, in automatic speech recognition for recognizing words. Let us distinguish two cases, depending on whether or not the maximum sequence length is fixed in advance. Moreover, let us assume that the sequences are supervised only at the end. In other words, the class is established only when the whole sequence has been scanned by the network. 4.1 Fixed Length Sequences. Let us assume a local activation feedback architecture with two hidden layers. The first one is composed of dynamic neurons that extract a distinctive feature from each sequence. The second one is an ordinary static layer that is necessary to perform universal interpolation from the previous feature vector and the output. The learning environment is composed of sequences of fixed length (maximum length = TI. It can be proved (Frasconi et al. 1991) that these sequences can be learned with arbitrary null cost by a network having at least IT dynamic neurons in the first hidden layer and enough static neurons in the second hidden layer. We can observe that the required number of dynamic hidden neurons is the same number of inputs that would be necessary to model the sequence as a static pattern. This represents a dramatic practical limitation that cuts down the advantages with respect to feedforward architectures. However, since local feedback networks are likely to achieve better generalization in these cases, they may still be more advantageous.

4.2 Information Latching. The limitation on the maximum sequence length may sometimes represent a serious restriction. A sequence can be arbitrarily long, thus requiring a discrimination rule depending on arbitrarily past information. In these cases, local activation feedback MLNs

I? Frasconi, M. Gori, and G. Soda

126

are surely to fail because of their relaxation at the unique equilibrium state. Moreover, with finite-length sequences too, it may turn out to be better to use other recurrent architectures. The result outlined in the previous paragraph also suggests investigating the effect of adopting output instead of activation feedback. This kind of limitation derives from the network's linear dynamic behavior. It is interesting to see if further improvement could be attained by exploiting the non-linear dynamic behavior typical of output feedback. Definition 3 (Information Latching). We say that a given dynamic hidden neuron latches information if its free evolution (null inputs) satisfies the following relationship: sign[a(t)]= sign[a(to)]

Vt > t o

(4.1)

This definition suggests the interpretation of the neuron output as a boolean status, in that only the sign of xi(t) is relevant and not its absolute value.

Theorem 3. Given a generic dynamic hidden neuron, information latching (for null inputs) occurs provided that (wI > 2. The latching condition also holds if the forcing term of 2.3 is bounded in module by a constant B = w f ( & )- &, where 0

(4.4)

Let us define the Lyapunov function V(a)as

V ( a )= ( a - u$)2

(4.5)

We have AV

'2(,

=

V[g(a)]- V(a)= [g(a) - ~

=

[g(a)

+ a - &$][g(a)

$ -1 (a~ - a t ) *

- a]

is the weight of the local feedback connection of the neuron.

(4.6)


If a 2 g(a) then a 2 a$ and positive and consequently

g(6)

127

2 a:. As a result the first factor is

AVlO (4.7) Conversely, if 0 < a 5 g(a) then a 5 a: and g(a) 5 a:, and therefore, the first factor is negative and we have AV 5 0 again. It follows the stability of a: for each a E (0,m). A similar proof can be provided for the stability of the other point a;. It can be easily proved that the inequality 4.7 is also valid if g(u) is translated by a constant b such that (bl < B, where B is defined by equation 4.2. Now let us consider the effect of adding a time-variant forcing term b(t) bounded in module by a constant bo such that 0 < bo 5 B. As previously done, let us limit the analysis to the positive solution. From the previous discussion, it follows that the system a(t

+ I ) = wa(t)

bo

(4.8) has a stable equilibrium point a:. We can easily prove that the activation a ( t ) of the system a(t

+ 1) =

-

+ b(t)

ZUU(~)

(4.9)

satisfies the inequality 4t)24 t )

(4.10)

By assuming null initial state, equation 4.10 is obviously valid for t = 0. Let us suppose that 4.10 is valid at t; then: a(t

+ 1)

wa(t)+ b ( t ) 2 W a ( t ) + b ( t ) = (Y(t + 1)+ bo + b(f) 2 a(t + I ) =

(4.11)

Because of the previous considerations on the stability of a: (see equation 4.81, the activation a(t),and therefore a(t), cannot change their sign, and, 0 by definition 3, information latching occurs.

Example 1. Let us consider the problem of recognizing arbitrarily long strings of characters. In particular, let us assume that the alphabet is composed of five symbols (A, B, C, D, E), and that the class is decided just by taking the first string symbol. We want to see if a local feedback MLN is able to learn this rule. We adopted a network with five inputs and exclusive symbol coding, three dynamic hidden neurons with output feedback, and five outputs, with exclusive symbol coding. The learning environment was composed of 500 sequences (100 for each class). The network learned the task quickly by discovering a coding of the five classes by means of the three dynamic hidden neurons. In particular, for each sequence, the network was able to discover automatically the only significant symbol - the first one - and to latch all the related information until the end of the occurred sequence. All patterns of the learning environment were recognized.

128


4.3 General Sequences. At this point, we must address an important question related to the behavior of a local output feedback MLN in tasks such as classification of arbitrary sequences. Information latching is an indispensable property for dealing with this class of problems, but it is not sufficient. The real weakness of local feedback architectures regards problems in which keeping track of the order inside a sequence is required. For the sake of simplicity, we restrict our discussion to a particular class of sequences, in which the problem of recognition is very similar to the problem of parsing a string of symbols.

Definition 4 (Event Sequences). We say that a sequence of symbols4 { S ( T ) , T E 7, S ( T ) E A } ( Ais a given finite alphabet) is an event sequence (ES) if the following conditions hold: 1. A can be partitioned into two sets R (relevant set) and N (nonrelevant set), according to a given criterion; symbols belonging to the relevant set are referred to as events.

2. An arbitrary number of symbols belonging to N can be interposed between two consecutive symbols belonging to R. In the following, if a given dynamic hidden neuron performs information latching, we assume that symbols of R (N) can (cannot) modify the boolean status of the neuron. This can be accomplished in local feedback architectures under some simple hypotheses on ”separability” of symbols. The previous definition refers to network configurations in which the dynamic hidden neurons perform information latching. On the other hand, it is clear that, without this assumption, local feedback MLNs cannot deal with sequences of any length. When using ES the nonrelevant symbols do not affect net evolution. Conversely, the relevant symbols draw the status of each dynamic hidden neuron. Local feedback MLNs are not suited for dealing with ES, as shown in the following simple example.

Example 2. Let us consider the problem of counting the number of occurrences of a given event e in an ES for which R = {e}. It is quite simple to show that this problem cannot be solved by any local feedback MLN. Because of condition 2 of ES, we begin by pointing out that a network that does not perform information latching obviously cannot solve a problem like this. Let us assume that one or more dynamic hidden neurons latch event e. The first occurrence of the event changes the boolean state of these neurons. However, whenever the event e comes out again, this state cannot be changed. As a result, our net cannot deal with multiple occurrences of the same event. 4T0 feed the networks, these symbols must be represented as vectors of R’.


129

The example proposed is not a particular case in which local feedback MLNs fail. It can be shown that they also fail in situations in which the ordering of the sequence must be taken into account.

5 Conclusions The analyses reported in this paper indicate that local feedback MLNs are particularly suited for problems in which classification depends on relatively recent information because of their capability of exhibiting forgetting effect. Our networks also behave quite well in problems of sequence recognition, provided that the sequences involved are relatively short. When dealing with long sequences, and particularly with sequences in which no limitation can be assumed concerning their length, local feedback MLNs exhibit serious limitations. A problem as easy as counting the occurrences of a given event cannot be accomplished in sequences with arbitrary length. As a matter of fact, for all these situations different recurrent architecture must be used. In these cases, learning algorithms like the ones suggested by Williams and Zipser (1989) and Pearlmutter (1989) must be adopted for learning. Although they are not local in both space and in time, they allow us to learn in full recurrent networks.

Acknowledgments This research was partially supported by M U S T 40% and CNR Grant 90.01530.CT01. We thank Renato De Mori and Yoshua Bengio of the School of Computer Science, McGill University, Montreal, Canada for their contribution on some ideas reported in this paper.

References Bourlard, H., and Wellekens, C. 1990. Links between hidden Markov models and multilayer perceptrons. IEEE Transact. Pattern Anal. Machine Intelligence PAMI-12, 1167-1178. Cleeremans, A., Servan-Schreiber, D., and McClelland, J. L. 1989. Finite state automata and simple recurrent networks. Neural Cornp. 3, 372-381. Elman, J. L. 1988. Finding structure in time. CRL Tech. Rep. 8801. La Jolla: University of California, San Diego, Center for Research in Language. Frasconi, P., Gori, M., and Soda, G. 1990. Recurrent networks with activation feedback. Proc. 3rd ltalian Workshop Parallel Architectures Neural Networks, Vietri sul Mare, Salerno, 15-18 May 1990, pp. 329-336. Frasconi, P., Gori, M., and Soda, G. 1991. Local feedback rnultilayered networks. Tech. Rep. RT2/91, Dipartimento di Sistemi e Informatica, Universith di Firenze. Gherrity, M. 1989. A learning algorithm for analog, fully recurrent neural networks. Proc. IEEE-lJCNN89 I, 643-644, Washington DC,June 18-22, 1989.

130


Gori, M. 1989. An extension of BPS. Proc. Neuro-Nimes '89,83-93, Nimes, 13-16 November 1989, France. Gori, M. 1990. Apprendimento con supervisione in reti neuronali. Ph.D. Thesis, Universiti di Bologna, Italy. Gori, M., Bengio, Y., and De Mori, R. 1989. BPS: A learning algorithm for capturing the dynamical nature of speech. Proc. IEEE-IJCNN89 11, 417-423, Washington DC,June 18-22, 1989. Mozer, M. C . 1988. A focused back-propagation algorithm for temporal pattern recognition. Tech. Rep. CRG-TR-88-3, University of Toronto. Pearlmutter, B. A. 1989. Learning state space trajectories in recurrent neural networks. Neural Comp. 2,263-269. Pearlmutter, B. A. 1990. Two new learning procedures for recurrent networks. Neural Networks Rev. 3, 99-101. Pineda, F. J. 1987. Generalization of back-propagation in recurrent networks. Physical Rtu. Lett. 29, 2229-2232. Tsung, F. 5. 1990. Learning in recurrent finite difference networks. In Proceedings of the 2990 Summer School, D. s. Touretzky, J. L. Elman, T. J. Sejnowski, and G. E. Hinton, eds., pp. 123-130. Morgan Kaufmann, San Mateo, CA. Watrous, R. L., Ladendorf, B., and Kuhn, G. 1988. Complete gradient optimization of a recurrent network applied to /b/, / d / , /g/ discrimination. 1. Acoust. SOC.Am. 87, 1301-1309. Williams R. J., and Zipser D. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural Comp. 1, 270-280.

Received 4 March 1991; accepted 25 April 1991.

This article has been cited by: 2. Krzysztof Patan. 2010. Local stability conditions for discrete-time cascade locally recurrent neural networks. International Journal of Applied Mathematics and Computer Science 20:1, 23-34. [CrossRef] 3. Khurshid M. Kiani, Terry L. Kastens. 2008. Testing Forecast Accuracy of Foreign Exchange Rates: Predictions from Feed Forward and Various Recurrent Neural Network Architectures. Computational Economics 32:4, 383-406. [CrossRef] 4. Mohammad Karamouz, Saman Razavi, Shahab Araghinejad. 2008. Long-lead seasonal rainfall forecasting using time-delay recurrent neural networks: a case study. Hydrological Processes 22:2, 229-241. [CrossRef] 5. Paris Mastorocostas, Dimitris Varsamis, Constantinos Hilas, Constantinos Mastorocostas. 2007. A generalized Takagi–Sugeno–Kang recurrent fuzzy-neural filter for adaptive noise cancelation. Neural Computing and Applications . [CrossRef] 6. Aydogan Savran. 2007. Multifeedback-Layer Neural Network. IEEE Transactions on Neural Networks 18:2, 373-384. [CrossRef] 7. T.G. Barbounis, J.B. Theocharis, M.C. Alexiadis, P.S. Dokopoulos. 2006. Long-Term Wind Speed and Power Forecasting Using Local Recurrent Neural Network Models. IEEE Transactions on Energy Conversion 21:1, 273-284. [CrossRef] 8. Alex Aussem . 2002. Sufficient Conditions for Error Backflow Convergence in Dynamical Recurrent Neural NetworksSufficient Conditions for Error Backflow Convergence in Dynamical Recurrent Neural Networks. Neural Computation 14:8, 1907-1927. [Abstract] [PDF] [PDF Plus] 9. P.A. Mastorocostas, J.B. Theocharis. 2002. A recurrent fuzzy-neural model for dynamic system identification. IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) 32:2, 176-190. [CrossRef] 10. Stefan C. Kremer . 2001. Spatiotemporal Connectionist Networks: A Taxonomy and ReviewSpatiotemporal Connectionist Networks: A Taxonomy and Review. Neural Computation 13:2, 249-306. [Abstract] [PDF] [PDF Plus] 11. P. Campolucci, F. Piazza. 2000. Intrinsic stability-control method for recursive filters and neural networks. IEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing 47:8, 797-802. [CrossRef] 12. S. Lawrence, C.L. Giles, S. Fong. 2000. Natural language grammatical inference with recurrent neural networks. IEEE Transactions on Knowledge and Data Engineering 12:1, 126-140. [CrossRef] 13. P. Campolucci, A. Uncini, F. Piazza, B.D. Rao. 1999. On-line learning algorithms for locally recurrent neural networks. IEEE Transactions on Neural Networks 10:2, 253-271. [CrossRef]

14. S.C. Kremer. 1999. Identification of a specific limitation on local-feedback recurrent networks acting as Mealy-Moore machines. IEEE Transactions on Neural Networks 10:2, 433-438. [CrossRef] 15. Jie Zhang, A.J. Morris. 1999. Recurrent neuro-fuzzy networks for nonlinear process modeling. IEEE Transactions on Neural Networks 10:2, 313-326. [CrossRef] 16. H.T. Siegelmann, B.G. Horne, C.L. Giles. 1997. Computational capabilities of recurrent NARX neural networks. IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) 27:2, 208-215. [CrossRef] 17. D. Bonnet, V. Labouisse, A. Grumbach. 1997. δ-NARMA neural networks: a new approach to signal prediction. IEEE Transactions on Signal Processing 45:11, 2799-2810. [CrossRef] 18. S.C. Kremer. 1996. Comments on "Constructive learning of recurrent neural networks: limitations of recurrent cascade correlation and a simple solution". IEEE Transactions on Neural Networks 7:4, 1047-1051. [CrossRef] 19. Y. Bengio, P. Frasconi. 1996. Input-output HMMs for sequence processing. IEEE Transactions on Neural Networks 7:5, 1231-1249. [CrossRef] 20. P. Frasconi, M. Gori. 1996. Computational capabilities of local-feedback recurrent networks acting as finite-state machines. IEEE Transactions on Neural Networks 7:6, 1521-1525. [CrossRef] 21. Tsungnan Lin, B.G. Horne, P. Tino, C.L. Giles. 1996. Learning long-term dependencies in NARX recurrent neural networks. IEEE Transactions on Neural Networks 7:6, 1329-1338. [CrossRef] 22. P. Frasconi, M. Gori, M. Maggini, G. Soda. 1995. Unified integration of explicit knowledge and learning by example in recurrent networks. IEEE Transactions on Knowledge and Data Engineering 7:2, 340-346. [CrossRef] 23. Paul Bressloff, John Taylor. 1993. Spatiotemporal pattern processing in a compartmental-model neuron. Physical Review E 47:4, 2899-2912. [CrossRef]

Communicated by Fernando Pineda

Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks Jiirgen Schmidhuber' lnstitut f i r Informatik, Technische Universitat Miinchen, Arcisstr. 21, 8000 Miinchen 2, Germany

Previous algorithms for supervised sequence Iearning are based on dynamic recurrent networks. This paper describes an alternative class of gradient-based systems consisting of two feedforward nets that learn to deal with temporal sequences using fast weights: The first net learns to produce context-dependent weight changes for the second net whose weights may vary very quickly. The method offers the potential for STM storage efficiency: A single weight (instead of a full-fledged unit) may be sufficient for storing temporal information. Various learning methods are derived. Two experiments with unknown time delays illustrate the approach. One experiment shows how the system can be used for adaptive temporary variable binding. 1 The Task A training sequence p with np discrete time steps (called an episode) consists of np ordered pairs [xf'(t),dP(t)] E R" x R" ,0 < t 5 np. At time t of episode p a learning system receives x P ( t ) as an input and produces the output yP(t). The goal of the learning system is to minimize

where d;(t) is the ith of the rn components of dP(t), and $ ( t ) is the ith of the m components of yP(t). In general, this task requires storage of input events in a short-term memory. Previous solutions to this problem have employed gradientbased dynamic recurrent nets (e.g., Robinson and Fallside 1987; Pearlmutter 1989; Williams and Zipser 1989). In the next section an alternative gradient-based approach is described. For convenience, we drop the indices p that stand for the various episodes. The gradient of the error over all episodes is equal to the sum of the gradients for each episode. Thus we only require a method for minimiz*Current address: Department of Computer Science, University of Colorado, Campus Box 430, Boulder, CO 80309 USA.

Neural Computation 4,131-139 (1992) @ 1992 Massachusetts Institute of Technology

Jiirgen Schmidhuber

132

ing the error observed during one particular episode:

E

=

CE(t) t

where E ( t ) = ci[di(t) - yi(t)12. [In the practical on-line version of the algorithm below there will be no episode boundaries; one episode will 'Mend into the next (Williams and Zipser 1989).] 2 The Architecture and the Algorithm

The basic idea is to use a slowly learning feedforward network S (with a set of randomly initialized weights WS) whose input at time t is the vector x ( t ) and whose output is transformed into immediate weight changes for a second "fast-weight" network F. The input to F at time t is also x ( t ) , its m-dimensional output is yff), and its set of weight variables is W,. F serves as a short-term memory: At different time steps, the same input event may be processed in different ways depending on the time-varying state of W,. The standard method for processing temporal sequences is to employ a recurrent net with feedback connections. The feedback connections allow for a short-term memory of information earlier in a sequence. The present work suggests a novel approach to building a short-term memory by employing fast weights that can be set and reset by the "memory controller" S. Fast weights can hold on to information over time because they remain essentially invariant unless they are explicitly modified. One potential advantage of the method over the more conventional recurrent net algorithms is that it does not necessarily require full-fledged units - experiencing some sort of feedback - for storing temporal information. A single weight may be sufficient. Because there are many more weights than units in most networks, this property represents a potential for storage efficiency. For related reasons, the novel representation of past inputs is well-suited for solving certain problems involving temporary variable binding in a natural manner: F's current input may be viewed as a representation of the addresses of a set of variables; F's current output may be viewed as the representation of the current contents of this set of variables. In contrast with recurrent nets, temporary bindings can be established very naturally by temporary connectivity pat terns instead of temporary activation patterns (see Section 3.2 for an illustrative experiment). For initialization reasons we introduce an additional time step 0 at the beginning of an episode. At time step 0 each weight variable w,b E W, of a directed connection from unit a to unit b is set to Ow,b(O) (a function of S's outputs as described below). At time step t > 0, the w,b(t - 1) are used to compute the output of F according to the usual activation spreading rules for backpropagation networks (e.g., Werbos 1974). After

133

Fast-Weight Memories

this, each weight variable Wab E W, is altered according to W u b ( f ) = g[Wub(t-

(2.1)

OWub(t)]

where ~7 (e.g., a sum-and-squash function) is differentiable with respect to all its parameters and where the activations of S’s output units (again computed according to the usual activation spreading rules for backpropagation networks) serve to compute OWub(t) by a mechanism specified below. O W & ( f ) is s’s contribution to the modification of wab at time step t. Equation 2.1 is essentially identical to Moller and Thrun’s equation 1 in Moller and Thrun (1990). Unlike Moller and Thrun (19901, however, the current paper derives an exact gradient descent algorithm for timevarying inputs and outputs for this kind of architecture. For all weights w, E WS (from unit i to unit j ) we are interested in the increment

Here 77 is a constant learning rate. At each time step t > 0, the factor

can be computed by conventional backpropagation (e.g., Werbos 1974). For t > 0 we obtain the recursion aWub(t)

- -

h i j

- a c [ W u b ( t - 11, OWub(t)]awub(t - 1) awub(t - 1) aw, - I), nWub(t)] aowab(t) + ag[Wub(t a‘wab(t) h i j

We can employ a method similar to the one described in Robinson and Fallside (1987) and Williams and Zipser (1989): For each wub E WF and each wij E W, we introduce a variable $’ (initialized to zero at the beginning of an episode) that can be updated at each time step t > 0:

(2.3)

aOwub(t)/dwijdepends on the interface between S and F. With a given interface (two possibilities are given below) an appropriate backpropagation procedure for each wab E WF gives us dOw,b(t)/aw, for all wv E Ws.

Jiirgen Schmidhuber

134

After having updated the f i b variables, (2.2) can be computed using the formula

A simple interface between S and F would provide one output unit sob

E S for each weight variable Wab E W,, where

being the output unit’s activation at time t 2 0. A disadvantage of 2.4 is that the number of output units in S grows in proportion to the number of weights in F. An alternative is the following: Provide an output unit in S for each unit in F from which at least one fast weight originates. Call the set of these output units FROM. Provide an output unit in S for each unit in F to which at least one fast weight leads. Call the set of these output units TO. For each weight variable wab E WF we now have a unit s, E FROM and a unit sb E TO. At time t, define OW,b(t) := g[s,(t), S b ( t ) ] , where g is differentiable with respect to all its parameters. As a representative example we will focus on the special case of g being the multiplication operator:

sab(t)

Owab(t) := % ( f ) S b ( f )

(2.5)

Here the fast weights in F are manipulated by the outputs of S in a Hebb-like manner, assuming that u is just a sum-and-squash function as employed in the experiments described below. One way to interpret the FROMITO architecture is to view S as a device for creating temorary associations by giving two parameters to the short-term memory: The first parameter is an activation pattern over FROM representing a key to a temporary association pair; the second parameter is an activation pattern over TO representing the corresponding entry. Note that both key and entry may involve hidden units. Equations 2.4 and 2.5 differ in the way that error signals are obtained at S’s output units: If 2.4 is employed, then we use conventional backpropagation to compute a s & ( t ) / h q in 2.3. If 2.5 is employed, note that

Conventional backpropagation can be used to compute as, ( t ) / a W i j for each output unit a and for all wij. The results can be kept in I WS I * I FROMU TO I variables. This makes it easy to solve 2.6 in a second pass. The algorithm is local in time, its update-complexity per time step is O(l WF 1) W, I). However, it is not local in space (see Schmidhuber 1990b for a definition of locality in space and time).


135

2.1 On-Line versus Off-Line Learning. The off-line version of the algorithm would wait for the end of an episode to compute the final change of W, as the sum of all changes computed at each time step. The on-line version changes W Sat every time step, assuming that 7 is small enough to avoid instabilities (Williams and Zipser 1989). An interesting property of the on-line version is that we do not have to specify episode boundaries [“all episodes blend into each other” (Williams and Zipser 1989)l. 2.2 Unfolding in Time. An alternative of the method above would be to employ a method similar to the ”unfolding in time” algorithm for recurrent nets (e.g., Rumelhart et al. 1986). It is convenient to keep an activation stack for each unit in S. At each time step of an episode, some unit’s new activation should be pushed onto its stack. S’s output units should have an additional stack for storing sums of error signals received over time. With both 2.4 and 2.5, at each time step we essentially propagate the error signals obtained at S’s output units down to the input units. The final weight change of W Sis proportional to the sum of all contributions of all errors observed during one episode. The complete gradient for S is computed at the end of each episode by successively popping off the stacks of error signals and activations analogously to the ”unfolding in time” algorithm for recurrent networks. A disadvantage of the method is that it is not local in time. 2.3 Limitations and Extensions. When both F and S are feedforward networks, the technique proposed above is limited to only certain types of time-varying behavior. With CJ being a sum-and-squash function, the only kind of interesting time-varying output that can be produced is in response to variations in the input; in particular, autonomous dynamic behavior like oscillations (e.g., Williams and Zipser 1989) cannot be performed while the input is held fixed. It is straightforward to extend the system above to the case where both S and F are recurrent. In the experiment below S and F are nonrecurrent, mainly to demonstrate that even a feedforward system employing the principles above can solve certain tasks that only recurrent nets were supposed to solve. The method can be accelerated by a procedure analogous to the one presented in Schmidhuber (1992). 3 Experiments The following experiments were conducted in collaboration with Klaus Bermer, a student at Technische Universitat Miinchen.

136

Jiirgen Schmidhuber

3.1 An Experiment with Unknown Time Delays. In this experiment, the system was presented with a continuous stream of input events and F's task was to switch on the single output unit the first time an event " B occurred following an event "A." At all other times, the output unit was to be switched off. This is the flip-flop task described in Williams and Zipser (1989). One difficulty with this task is that there can be arbitrary time lags between relevant events. An additional difficulty is that no information about "episode boundaries" is given. The on-line method was employed: The activations of the networks were never reset. Thus, activations caused by events from past "episodes" could have a harmful effect on activations and weights in later episodes. Both F and S had the topology of standard feedforward perceptrons. F had 3 input units for 3 possible events "A," "B," and "C." Events were represented in a local manner: At a given time, a randomly chosen input unit was activated with a value of 1.0; the others were deactivated. F's output was one-dimensional. S also had 3 input units for the possible events "A," "B," and "C," as well as 3 output units, one for each fast weight of F. Neither of the networks needed hidden units for this task. The activation function of all output units was the identity function. The weight-modification function (1) for the fast weights was given by

Here T determines the maximal steepness of the logistic function used to bound the fast weights between 0 and 1. The weights of S were randomly initialized between -0.1 and 0.1. The task was considered to be solved if for 100 time steps in a row F's error did not exceed 0.05. With fast-weight changes based on 2.4, T = 10 and q = 1.0 the system learned to solve the task within 300 time steps. With fast-weight changes based on the FROM/TO architecture and 2.5, T = 10 and 17 = 0.5 the system learned to solve the task within 800 time steps. The typical solution to this problem has the following properties: When an A-signal occurs, S responds by producing a large weight on the B input line of F (which is otherwise small), thus enabling the F network as a B detector. When a B signal occurs, S "resets" F by causing the weight on the B line in F to become small again, thereby making F unresponsive to further B signals until the next A is received.

3.2 Learning Temporary Variable Binding. Some researchers have claimed that neural nets are incapable of performing variable binding. Others, however, have argued for the potential usefulness of "dynamic links" (e.g., von der Malsburg 19811, which may be useful for variable


137

binding. With the fast-weight method, it is possible to train a system to use fast weights as dynamic links in order to temporarily bind variable contents to variable names (or “fillers” to ”slots”)as long as it is necessary for solving a particular task. In the simple experiment described next, the system learns to remember where in a parking lot a car has been left. This involves binding a value in a variable that represents the car’s location. Neither F nor S needed hidden units for this task. The activation function of all output units was the identity function. All inputs to the system were binary, as were F‘s desired outputs. F had one input unit which stood for the name of the variable WHERE-IS-MY-CAR? In addition, F had three output units for the names of three possible parking slots PI, P2, and P3 (the possible answers to WHERE-IS-MY-CAR?). S had three output units, one for each fast weight, and six input units. (Note that S need not always have the same input as F.) Three of the six input units were called the parking-slot detectors - I], 12, 4.These detectors were activated for one time step when the car was parked in a given slot (while the other slot-detectors remained switched off). The three additional input units were randomly activated with binary values at each time step. These random activations served as distracting time varying inputs from the environment of a car owner whose life looks like this: He drives his car around for zero or more time steps (at each time step the probability that he stops driving is 0.25). Then he parks his car in one of three possible slots. Then he conducts business outside the car for zero or more time steps during which all parking-slot-detectors are switched off again (at each time step the probability that he finishes business is 0.25). Then he remembers where he has parked his car, goes to the corresponding slot, enters his car, and starts driving again, etc. Our system focused on the problem of remembering the position of the car. It was trained by activating the WHERE-IS-MY-CAR? unit at randomly chosen time steps and by providing the desired output for F, which was the activation of the unit corresponding to the current slot Pi, as long as the car was parked in one of the three slots. The weights of S were randomly initialized between -0.1 and 0.1. The task was considered to be solved if for 100 time steps in a row F‘s error did not exceed 0.05. The on-line version (without episode boundaries) was employed. With the weight-modification function 3.1, fastweight changes based on 2.4, T = 10 and q = 0.02 the system learned to solve the task within 6000 time steps. As it was expected, S learned to “bind” parking slot units to the WHERE-IS-MY-CAR? unit by means of strong temporary fast-weight connections. Due to the local output representation, the binding patterns were easy to understand: At a given time there was a large fast weight on the connection leading from the WHERE-IS-MY-CAR? unit to the appropriate parking slot unit (given the car was currently parked). The other fast weights remained temporarily suppressed.

138

Jurgen Schmidhuber

4 Concluding Remarks

The system described above is a special case of a more general class of adaptive systems (which also includes conventional recurrent nets) that employ some parameterized memory function for changing a vectorvalued memory structure and that employ some parameterized retrieval function for processing the contents of the memory structure and the current input. The only requirement is that the memory and retrieval functions be differentiable with respect to their internal parameters. Such systems work because of the existence of the chain rule. Results as above [as well as other novel applications of the chain rule (Schmidhuber 1990a, 1991)l indicate that there may be additional interesting (yet undiscovered) ways of applying the chain rule for temporal credit assignment in adaptive systems. Acknowledgments

I wish to thank Klaus Bergner for conducting the experiments. Furthermore I wish to thank Mike Mozer, Bernhard Schatz, and Jost Bernasch for providing comments on a draft of this paper. References Moller, K., and Thrun, S. 1990. Task modularization by network modulation. In Proceedings of Neuro-Nimes ‘90, J. Rault, ed.,pp. 419432. Pearlmutter, B. A. 1989. Learning state space trajectories in recurrent neural networks. Neural Comp. 1,263-269. Robinson, A. J., and Fallside, F. 1987. The utility driven dynamic error propagation network. Tech. Rep. CUED/F-INFENG/TR.l, Cambridge University Engineering Department. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing, D. E. Rumelhart and J. L. McClelland, eds., Vol. 1, pp. 318-362. MIT Press, Cambridge, MA. Schmidhuber,J. H. 1990a. Dynamische neuronale Netze und das fundamentale raumzeitliche Lernproblem. Dissertation, Institut fur Informatik, Technische Universitat Miinchen. Schmidhuber, J. H. 1990b. Learning algorithms for networks with internal and external feedback. In Proceedings of the 1990 Connectionist Models Summer School, D. S. Touretzky, J. L. Elman, T. J. Sejnowski, and G. E. Hinton, eds., pp. 52-61. Morgan Kaufmann, San Mateo, CA. Schmidhuber,J. H. 1991. Learning to generate sub-goals for action sequences. In Proceedings of the International Conference on Artificial Neural Networks K A N N 91, T. Kohonen, K. Makisara, 0.Simula, and J. Kangas, eds., pp. 967972. Elsevier Science Publishers B.V., Amsterdam.


139

Schmidhuber, J. H. 1992. A fixed size storage O(n3)time complexity learning algorithm for fully recurrent continually running networks. Neural Cornp., in press. von der Malsburg, C. 1981. Internal Report 81-2, Abteilung fiir Neurobiologie, Max-Planck Institut fiir Biophysik und Chemie, Gottingen. Werbos, I? J. 1974. Beyond regression: New tools for prediction and analysis in the behavioral sciences. Ph.D. thesis, Harvard University. Williams, R. J., and Zipser, D. 1989. Experimental analysis of the real-time recurrent learning algorithm. Connect. Sci. 1(1), 87-111.

Received 4 April 1991; accepted 18 July 1991.

This article has been cited by: 2. Andrew D. Back , Tianping Chen . 2002. Universal Approximation of Multiple Nonlinear Operators by Neural NetworksUniversal Approximation of Multiple Nonlinear Operators by Neural Networks. Neural Computation 14:11, 2561-2566. [Abstract] [PDF] [PDF Plus] 3. Hiroyuki Nakahara* , Kenji Doya . 1998. Near-Saddle-Node Bifurcation Behavior as Dynamics in Working Memory for Goal-Directed BehaviorNear-Saddle-Node Bifurcation Behavior as Dynamics in Working Memory for Goal-Directed Behavior. Neural Computation 10:1, 113-132. [Abstract] [PDF] [PDF Plus]

Communicated by Yann LeCun

REVIEW

First- and Second-Order Methods for Learning: Between Steepest Descent and Newton’s Method Roberto Battiti Dipartimento di Matematica, Universita di Trento, 38050 Povo (Trento),Italy On-line first-order backpropagation is sufficiently fast and effective for many large-scale classification problems but for very high precision mappings, batch processing may be the method of choice. This paper reviews first- and second-order optimization methods for learning in feedfomard neural networks. The viewpoint is that of optimization: many methods can be cast in the language of optimization techniques, allowing the transfer to neural nets of detailed results about computational complexity and safety procedures to ensure convergence and to avoid numerical problems. The review is not intended to deliver detailed prescriptions for the most appropriate methods in specific applications, but to illustrate the main characteristics of the different methods and their mutual relations. 1 Introduction

There are cases in which learning speed is a limiting factor in the practical application of multilayer perceptrons to problems that require high accuracy in the network mapping function. In this class are applications related to system identification and nonlinear modeling, time-series prediction, navigation, manipulation, and robotics. In addition, the standard batch backpropagation (BP) method ( e g , Rumelhart and McClelland 1986) requires a selection of appropriate parameters by the user, that is mainly executed with a trial-and-error process. Since one of the competitive advantages of neural networks is the ease with which they may be applied to novel or poorly understood problems, it is essential to consider automated and robust learning methods with a good average performance on many classes of problems. This review describes some methods that have been shown to accelerate the convergence of the learning phase on a variety of problems, and suggests a possible ”taxonomy” of the different techniques based on their order (i.e., their use of first or second derivatives), space and computational requirements, and convergence properties. Some of these methods, while requiring only limited modifications of the standard BP algorithm, Neural Computation 4,141-166 (1992)


142

Roberto Battiti

yield a speed-up of very large factors’ and, furthermore, are easier to apply because they do not require the choice of critical parameters (like the learning rate) by the neural network practitioners. The presentation attempts a classification of methods derived from the literature based on their underlying theoretical frameworks, with particular emphasis on techniques that are appropriate for the supervised learning of multilayer perceptrons (MLPs). The general strategy for the supervised learning of an input-output mapping is based on combining a quickly convergent local method with a globally convergent one. Local methods are based on appropriate local models of the function to be minimized. In the following sections, first we consider the properties of methods based on a linear model (steepest descent and variations), then we consider methods based on a quadratic model (Newton’s method and approximations). 2 Backpropagation and Steepest Descent

The problem of learning an input-output mapping from a set of P examples can be transformed into the minimization of a suitably defined error function. Although different definitions of the error have been used, for concreteness we consider the “traditional” sum-of-squared-differences error function defined as (2.1)

where t,, and oprare the target and the current output values for pattern

p , respectively, and no is the number of output units. The learning procedure known as backpropagation (Rumelhart and McClelland 1986) is composed of two stages. In the first, the contributions to the gradient coming from the different patterns (dEp/8wij) are calculated ”backpropagating” the error signal. The partial contributions are then used to correct the weights, either after every pattern presentation (on-line BP), or after they are summed in order to obtain the total gradient (batch BP). Let us define as gk the gradient of the error function kk = VE(wk)]. The batch backpropagation update is a form of gradient descent defined as

while the on-line update is

‘It is not unusual to observe speed-ups of 100-1000 with respect to BP with fixed learning and momentum rates.

First- and Second-Order Methods for Learning

143

If the learning rate E tends to zero, the difference between the weight vectors Wk+,,during one epoch of the on-line method tends to be small and the step EVE~(W~+,,) induced by a particular pattern p can be approximated by cVEp(wk)(by calculating the gradient at the initial weight vector). Summing the contributions for all patterns, the movement in weight space during one epoch will be similar to the one obtained with a single batch update. However, in general the learning rate has to be large to accelerate convergence, so that the paths in weight space of the two methods differ. The on-line procedure has to be used if the patterns are not available before learning starts [see, for example, the perceptron used for adaptive equalization in Widrow and Stearns (1985)1, and a continuous adaptation to a stream of input-output signals is desired. On the contrary, if all patterns are available, collecting the total gradient information before deciding the next step can be useful in order to avoid a mutual interference of the weight changes (caused by the different patterns) that could occur for large learning rates (this effect is equivalent to a sort of noise in the true gradient direction). One of the reasons in favor of the on-line approach is that it possesses some randomness that may help in escaping from a local minimum. The objection to this is that the method may, for the same reason, miss a good local minimum, while there is the alternative method of converging to the "nearest" local minimum and using randomness2 to escape only after convergence. In addition, the on-line update may be useful when the number of patterns is so large that the errors involved in the finite precision computation of the total gradient may be comparable with the gradient itself. This effect is particularly present for analog implementations of backpropagation, but it can be controlled in digital implementations by increasing the number of bits during the gradient accumulation. The fact that many patterns possess redundant information [see, for example, the case of hand-written character recognition in LeCun (1986)l has been cited as an argument in favor of on-line BP, because many of the contributions to the gradient are similar, so that waiting for all contributions before updating can be wasteful. In other words, averaging over more examples to obtain a better estimate of the gradient does not improve the learning speed sufficiently to compensate for the additional computationaI cost of taking these patterns into account. Nonetheless, the redundancy can also be limited using batch BP, provided that learning is started with a subset of "relevant" patterns and continued after convergence by progressively increasing the example set. This method has for example been used in Kramer and SangiovanniVicentelli (1988) for the digit recognition p r ~ b l e m .Even ~ if the training 21n addition, there are good reasons why random noise may not be the best available alternative to escape from a local minimum and avoid returning to it. See, for example, the recently introduced TABU methods (Glover 1987). 31f the redundancy is clear (when for example many copies of the sume example are present) one may preprocess the example set in order to eliminate the duplication. On

144

Roberto Battiti

set is redundant, on-line BP may be slow relative to second-order methods for badly conditioned problems. The convergence of methods based on gradient descent (or approximations thereof) depends critically on the relative size of the maximum and minimum eigenvalues of the Hessian matrix [see LeCun et al. (1991) and equation 2.6 for the case of steepest descent]. This is related to the "narrow valley" effect described in Rumelhart and McClelland (1986). In addition, the batch approach lends itself to straightforward modifications using second-order information, as it will be shown in the following sections. At this point, in order not to mislead the reader in the choice of the most appropriate method for a specific application, it is useful to remember that many large-scale experiments (mainly in pattern recognitionclassification) have used the simple on-line version of BP with full satisfaction, considering both the final result and the number of iterations. In some cases, with a careful tuning of the on-line procedure the solution is reached in a very small number of epochs, that is, in a few presentations of the complete example set [see, for example, Rumelhart and McClelland (198611 and it is difficult to reach a comparable learning speed with batch techniques (Cardon ef al. 1991). Tuning operations are, for example, the choice of appropriate parameters like the learning and momentum rate (Fahlman 1988), "annealing" schedules for the learning rate (that is progressively reduced) (Malferrari et al. 19901, updating schemes based on summing the contributions of related patterns (Sejnowski and Rosenberg 1986), "small batches," "selective" corrections only if the error is larger than a threshold (that may be progressively reduced) (Vincent 1991; Allred and Kelly 19901, randomization of the sequence of pattern presentation, etc. The given references are only some examples of significant applications, out of an abundant literat~re.~ Because in these cases only approximated output values are required and the example set often is characterized by a large degree of redundancy, these two attributes could be considered as votes in favor of on-line BP, again provided that the trial-and-error phase is not too expensive. In the original formulation, the learning rate 6 was taken as a fixed parameter. Unfortunately, if the learning rate is fixed in an arbitrary way, there is no guarantee that the net will converge to a point with vanishing gradient. Nonetheless, convergence in the on-line approach can be obtained by appropriately choosing a fixed and sufficiently small learning rate. The issue of an appropriate fixed learning rate for on-line LMS learning has been investigated in the adaptive signal processing literature the contrary, if redundancy is only partial, the redundant patterns have to be presented to and learned by the network in both versions. 4The availability of many variations of the on-line technique is one of the reasons why "fafi" comparisons with the batch and second-order versions are complex. Which version has to be chosen for the comparison? If the final convergence results have been obtained after a tuning process, should the tuning times be included in the comparison?


145

[see, for example, Bingham (1988)l. The result is that the convergence of stochastic LMS is guaranteed if E < 1 / ( N G), where N is the number of parameters being optimized and A, is the largest eigenvalue of the autocorrelation function of the input.5 A detailed study of adaptive filters is presented in Widrow and Stearns (1985). The effects of the autocorrelation matrix of the inputs on the learning process (for a single linear unit) are discussed in LeCun et al. (1991). In this framework the appropriate learning rate for gradient descent is 1/Amax. These results cannot be extended to multilayer networks (with nonlinear transfer functions) in a straightforward manner, but they can be used as a starting point for useful heuristics. The convergence properties of the LMS algorithm with adaptive learning rate are presented in Luo (1991), together with a clear comparison of the LMS algorithm with stochastic gradient descent and adaptive filtering algorithms. The main result is that if the learning rate E , for the nth training cycle satisfies the two conditions: (2.4) n=l

n=l

then the sequence of weight matrices generated by the LMS algorithm (with a cyclic pattern presentation) will converge to the optimal solution (minimizing the mean-square error). But even if E is appropriately chosen so that the error decreases with a reasonable speed and oscillations are avoided, gradient descent is not always the fastest method to employ. This is not an intuitive result, because the negative gradient is the direction of fastest decrease in the error. Unfortunately, the “greed” in trying to reach the minimum along this one-dimensional direction is paid at the price that subsequent gradient directions tend to interfere, so that in a weight space with dimension N, the one-dimensional minimization process has to be repeated for a number of times that is normally much higher than N (even for simple quadratic functions).6 In the steepest descent method, the process of minimizing along successive negative gradients is described as:

where f k minimizes E ( w ~- E g k ) . If steepest descent is used to minimize a quadratic function Q(w) = cTw ;wTGw (G symmetric and positive

+

5l owe this remark to the referee. 61t is easy to show that, if exact one-dimensional optimization along the negative gradient is used, the gradient at the next step is perpendicular to the previous one. If, considering an example in two dimensions, the lines at equal error are given by elongated ellipsoids, the system, for a “general” starting point, goes from a point to the one that is tangent to the equal-error lines along the gradient, and then repeats along a perpendicular direction.

Roberto Battiti

146

definite), it can be shown that IQ(wk+i)

- Q(w*)l M

(

7)max

- rlmin

Vmax

+ Vmin ) 2

lQ(wk)

-

Q(w*)l

(2.6)

where rimax and qmln are the maximum and minimum eigenvalues of the matrix G, and wl; is the minimizer [see Luenberger (197311. If these two eigenvalues are very different, the distance from the minimum ~ a l u e is multiplied each time by a number that is close to one. The type of convergence in equation 2.6 is termed q-linear convergence. One has 9superlinear convergence if, for some sequence ck that converges to 0, one has IQ(W+l)

- Q(w*)I 5 C k I Q ( W )

-

Q(w*)I

(2.7)

Finally, wk is said to converge with q-order at least p if, for some constant c, one has

I Q ( w +I )

- Q(w*)l 5 clQ(wk) - Q(w*)lP

(2.8)

In practice q-linear convergence tends to be very slow, while q-superlinear or q-quadratic ( p = 2) convergence is eventually much faster7 In the following sections, we illustrate some techniques that can be used to ensure convergence, to avoid numerical problems related to finite-precision computation, and to accelerate the minimization process with respect to standard batch BP. 3 Conjugate Gradient Methods

Let us consider a quadratic function Q ( w ) of the type described in the previous section. We have seen that one of the difficulties in using the steepest descent method is that a one-dimensional minimization in direction a followed by a minimization in direction b does nut imply that the function is minimized on the subspace generated by a and b. Minimization along direction b may in general spoil a previous minimization along direction a (this is why the one-dimensional minimization in general has to be repeated a number of times much larger than the number of variables). On the contrary, if the directions were noninterfering and linearly independent, at the end of N steps the process would converge to the minimum of the quadratic function. The concept of noninterfering directions is at the basis of the conjugate gradient method (CG) for minimization. Two directions are mutually conjugate with respect to the matrix G if

pTGpj = O

when

i#J

(3.1)

7For example, for a q-quadratically convergent method, the number of significant digits in the solution is doubled after each iteration.


147

After minimizing in direction pi, the gradient at the minimizer will be perpendicular to p i . If a second minimization is in direction p i + l , the change of the gradient along this direction is gi+l -gi = aGp,+l (for some constant a). The matrix G is indeed the Hessian, the matrix containing the second derivatives, and in the quadratic case the model coincides with the original function. Now, if equation 3.1 is valid, this change is perpendicular to the previous direction [pT(gitl - gi) = 01, therefore the gradient at the new point remains perpendicular to pi and the previous minimization is not spoiled. While for a quadratic function the conjugate gradient method is guaranteed to converge to the minimizer in at most (N + 1) function and gradient evaluations (apart from problems caused by the finite precision), for a general function it is necessary to iterate the method until a suitable approximation to the minimizer is obtained. Let us introduce the vector Y k = &+I - gk. The first search direction p l is given by the negative gradient -gl. Then the sequence wk of approximations to the minimizer is defined in the following way: (3.2) (3.3)

where g k is the gradient, (Yk is chosen to minimize E along the search direction p k and /3k is defined by (3.4)

There are different versions of the above equation. In particular, the Polak-Ribiere choice is A = ylgk+l the Fletcher-Reeves choice is p k = g[+‘+1gk+l/gkTgk* They all coincide for a quadratic function (Shanno 1978). A major difficulty with all the above iorms is that, for a general function, the obtained directions are not necessarily descent directions and numerical instability can result. Although for a wide class of functions the traditional CG method with exact searches and exact arithmetic is superlinearly convergent, implementations of the conjugate-gradient method with finite precision computation are “nearly always linearly convergent” (Gill et al. 1981), but the number of steps is in practice much smaller than that required by steepest descent. The use of a momentum term to avoid oscillations in Rumelhart and McClelland (1986) can be considered as an approximated form of conjugate gradient. In both cases, the gradient direction is modified with a term that takes the previous direction into account, the important difference being that the parameter p in conjugate gradient is automatically defined by the algorithm, while the momentum rate has to be ”guessed” by the user. Another difficulty related to the use of a momentum term is due to the fact that there is an upper bound on the adjustment caused by the momentum. For example, if all partial derivatives are equal to 1, then the

148

Roberto Battiti

exponentially weighted sum caused by the momentum rate a converges to 1/(1 - a ) [see Jacobs (1988) for details].' Furthermore, summing the momentum term to the one proportional to the negative gradient may produce an ascent direction, so that the error increases after the weight update. Among the researchers using conjugate gradient methods for the MLP are Barnard and Cole (1988), Johansson et al. (19901, Bengio and Moore (19891, Drago and Ridella (1991), Hinton's group in Toronto, the groups at CMU, Bell Labs, etc. A version in which the one-dimensional minimization is substituted by a scaling of the step that depends on success in error reduction and goodness of a one-dimensional quadratic approximation is presented in Moller (1990) (SCG).This O ( N )scheme incorporates ideas from the "model-trust region" methods (see Section 4.3) and "safety" procedures that are absent in the CG schemes, yielding convergence results that are comparable with the OSS method described in Section 6. Some modifications of the method are presented in Williams (1991). It is worth stressing that expensive one-dimensional searches are also discouraged by current results in optimization (see Section 4.1): the search can be executed using only a couple of function and gradient evaluations. 4 Newton's Method

Newton's method can be considered as the basic local method using second-order information. It is important to stress that its practical applicability to multilayer perceptrons is hampered by the fact that it requires a calculation of the Hessian matrix (a complex and expensive task'). Nonetheless, the method is briefly illustrated because most of the "useful" second-order methods originate from it as approximations or variations. It is based on modeling the function with the first three terms of the Taylor-series expansion about the current point wc:

and solving for the step sN that brings to a point where the gradient of the model is zero: Vmc(w, + s N ) = 0. This corresponds to solving the addition, B. Pearlmutter has recently shown that momentum even if chosen "optimally" can do no better than q-linear convergence (see his poster at the NIPS 1991 conference). 9A "brute force" method to calculate H is that of using a finite-difference formula. If the gradient is available (as is the case for feedforward nets), one may use: H , = [VE(w,+ h e ) - VE(w,)]/h,, with suitable h, steps [see Dennis et al. (1981)l. Note that N + 1 grad& computations are needed, so that the method is not suggested for large networks!

First- and Second-OrderMethods for Learning

149

following linear system: V2E(w,)sN= -VE(w,) (4.2) sN is, by definition, Newton’s step (and direction). If the Hessian matrix (V2E or H, for short) is positive definite and the quadratic model is correct, one iteration is sufficient to reach the minimum. Because one iteration consists in solving the linear system in equation 4.2, the complexity of one step is O(N3),using standard methods.’O In general, if the initial point wo is sufficiently close to the minimizer w,, and V2E(w,)is positive definite, the sequence generated by repeating Newton’s algorithm converges q-quudruticully to w* [see Dennis and Schnabel (1983) for details]. Assuming that the Hessian matrix can be obtained in reasonable computing times, the main practical difficulties in applying the ”pure” Newton’s method of equation 4.2 arise when the Hessian is not positive definite, or when it is singular or ill-conditioned. If the Hessian is not positive definite (this may be the case in multilayer perceptron learning!), there is no ”natural” scaling in the problem: there are directions p k of negative curvature (i.e., such that $Hpk 5 0) that would suggest ”infinite” steps in order to minimize the model. Unfortunately, long steps increase the probability of leaving the region where the model is appropriate, producing nonsense. This behavior is not uncommon for multilayer perceptron learning: in some cases a local minimization step increases some weights by large amounts, pushing the output of the sigmoidal transfer function into the saturated region. When this happens, some second derivatives are very small and, given the finite machine precision or the approximations, the calculated Hessian will not be positive definite. Even if it is, the linear system of equation 4.2 may be seriously ill-conditioned. Modified Newton‘s methods incorporate techniques for dealing with the above problems, changing the model Hessian in order to obtain a sufficiently positive definite and non-singular matrix. It is worth observing that, although troublesome for the above reasons, the existence of directions of negative curvature may be used to continue from a saddle point where the gradient is close to zero.” While calculating the analytic gradient for multilayer perceptrons can be efficiently executed ”backpropagating” the error, calculating the Hessian is computationally complex, so that practical methods have to rely on suitable approximations. In the following sections we illustrate some modifications of Newton’s method to deal with global convergence, indefinite Hessian, and iterative approximations to the Hessian itself. In the review by White (1989) the use of appropriate modifications of Newton’s methods for learning is considered starting from a statistical perspective. ‘OA smaller upper bound f o r matrix inversion is actually O ( T I ‘ O ~see ~ ~ )Press ; et a!. (1988) for details. “Only second-order methods provide this possibility, while methods based on steepest descent are condemned to failure in this case: how many “local minima” are in reality saddle points!

Roberto Battiti

150

4.1 Globally Convergent Modifications: Line Searches. Considering second-order methods, their fast local convergence property suggests that they are used when the approximation is close to the minimizer. On the other hand, getting from an initial configuration (that may be very different from the minimizer) to a point where the local model is accurate requires some additional effort. The key idea to obtain a general-purpose successful learning algorithm is tbat of combining a fast tactical local method with a robust strategic method that assures global convergence. Because Newton's method (or its modifications when the analytic Hessian is not available) has to be used near the solution, the suggested method is that of trying the Newton step first, deciding whether the obtained point is acceptable or not, and backtracking in the last case (i.e., selecting a shorter step in the Newton direction). One of the reasons for searching the next point along the Newton direction is that this is a descent direction, that is, the value of the error is guaranteed to decrease for sufficiently small steps along that direction. It is easy to see why: because the Hessian (and therefore its inverse) is symmetric and positive definite, the directional derivative of the error is negative:

dE -(wc dX

+ AsN)= V E ( W , ) ~ =S ~- V E ( W , ) ~ H ~ ' V E ( W ,O

(3.4)

Also in terms of this notation we may express the gradient of 3.2 as (3.5)

Assume p satisfies 3.4 and is fixed except for player i, whose payoff is given by 3.2. Since this function is concave, and the constraint set (a simplex) is convex, a given mixed strategy for i will have a maximum payoff if and only if i’s gradient 3.5 has a vanishing projection onto the constraint set 3.4. Equivalently, pi is an optimal strategy for player i if and only if there exists no directional vector d such that dj = 0 for j # i, and

Aj.d = 0 for all Xi, pi(&) = 0 implies dj(Xj) 2 0

(3.6)

Now let all players be free to change their strategies. For p to be a Nash equilibrium, 3.6 must be simultaneously nonsatisfiable for each player i. It can be shown (e.g., Miller and Zucker 1991) that this set of simultaneous conditions is equivalent to there being no d satisfying the system

d T ( R p+c) > 0 Ad = 0 for all i, Xi, pi(Xi) = 0 implies di(Xi) 1 0 Ap=q, p 1 0

(3.7)

In view of 3.7 we now have an alternative characterization of the Nash equilibria of the polymatrix game 3.3 in terms of the equilibria of the dynamical system

p’=Rp+c Ap=q, p 2 0

(3.8)

In other words, these equilibria are precisely the points at which the vector field of 3.8 vanishes. Notice if R is symmetric then p’ is the gradient of 1 -pTR p cTp (3.9) 2 The first term in 3.9 corresponds to the average local potential in relaxation labeling (Hummel and Zucker 1983).

+

Douglas A. Miller and Steven W. Zucker

174

4 Analog Networks as Polymatrix Games

We take as our point of departure the class of analog networks defined by Hopfield (1984). These are dynamical systems defined by the equations

u; = g;'(VJ

for i = 1,.. . , n. Here u, and V , are interpreted as the input and output voltages of an instantaneous amplifier described by a continuous monotonic sigmoid function g,(u,). In addition we define ITIII as the conductance between the output of amplifier j and the input of amplifier i, we let C, be the input capacitance of i, we let I, be a fixed input bias current for i, and we define R, by (4.2) where p l is the resistance across C,. If TI]is negative, then the input to amplifier i comes from an inverting amplifier -gJ(ul). Such a network is illustrated in Figure 1. Suppose now g, is a linear function on the real interval [ a l l P l ] , (r, < 0 < PI, such that g,(al) = 0, g I ( p 1 )= 1, and that u,, 0, are also upper and lower bounds on the voltage that the input capacitor to amplifier i can attain. Thus a further input current to a saturated capacitor would produce no effect. (We shall show in the next section that this model actually includes piecewise-linear voltage amplifiers as well.) Letting 6, = (PI - a , ) be a unitless scalar these assumptions give us a new version of 4.1:

(4.3)

Rewriting this in terms of the output voltages Vi and dividing through by Sic/we obtain

O S/w with C(0) = CO= (p' + 1)/3 - (p 1)z/9,C ( 1 ) = p/3 - ( p + 1)2/9,C(2) = C(1). C(3) = C(O), that is, an oscillati_onwith period 3. 3Themean squared fluctuation is (s2) - ( s ) = ~ J d
Ji,u

scales.

mean value of s, which coincides for n -+. 0;) with the maximum of P(s). If q = 0 the correlation function is a constant according to equation 3.11, while the system will exhibit undamped oscillations with period 3 for q = 1. An earlier example where the mean activity of a large popuIation of neurons converges either to a fixed point or to a limit cycle has been discussed by Sompolinsky (1988). Therefore, the irregularity of the burst intervals, as shown, for instance, in Figure 2, is for independent a finite

Simple Network Showing Burst Synchronization

221

size effect. Such synchronized dephasing due to finite size has already been reported by Sompolinsky ef al. (1989). the width of However, for biologically realistic correlated inputs P(s) can remain finite for n >> 1. For example, if the inputs ti... . .tt, can be grouped into q correlated sets ti.. .ti.(:. . .E:, . . . . t i .. .ti,with finite q, then the width of P(s) scales like l/&. Our model, which now effectively corresponds to a situation with a finite number q of inputs, leads in this case to irregular bursts that mirror and amplify the correlations present in the input signals, with an oscillatory component superimposed due to the dynamic threshold.

> 1

(3.2)

k 1

where the sum extends over all the levels in the coupling tree. Two typical profiles are found to satisfy the above constraint. The first one, which we call linear, is defined as 1

d(1) =

I

L(b - 1)bI-l

(3.3)

and amounts to an equal contribution to the coupling strength from the clusters at all scales in the population. In other words, the fall off of the coupling strength with ultrametric distance is exactly balanced by the increase in the number of interacting oscillators at such distance. Using a renormalization technique similar to the one developed by Daido (1988) for lattice structures, we have shown that the evolution of the average phase within a large cluster is driven by interaction terms with other clusters proportional to their average phase difference. Thus our results apply to more complex hierarchies than just regular ones, since the net interaction between any two large clusters of oscillators is independent of the detailed connections between them. We can further demonstrate that if the distribution of intrinsic frequencies drops fast enough away from the average value (i.e., if a > 11, a single phase transition to a collective state of synchrony among all the oscillators will arise as soon as the coupling constant K exceeds a threshold value 1

(3.4)

Binding Hierarchies

347

which implies that a larger dispersion in frequency moves the transition threshold to a higher value. A similar relation exists between the critical coupling strength and the width of dynamic noise added to a uniform firing rate among the population of oscillators (Sompolinski ef al. 1990). A collective state of synchrony is defined by the presence of a macroscopic number of cells (i.e., of order N) oscillating with a common frequency dO/df. Notice that because the frequency-locked neurons have different intrinsic firing rates, they cannot be strictly in phase. The phase dispersion among these cells is an increasing function of O w / K , with Aw measuring the intrinsic frequency spread. Thus, if the coupling constant is large compared to the dispersion of firing rates, collective synchrony will also imply near phase-locking among the oscillators. Therefore, frequency-locking in assemblies of oscillators is both a precursor and a strong indicator of phase-locked states that are relevant to the binding problem. Since oscillators with a linear profile of couplings exhibit a single nonlocal state of synchrony, they cannot separate percepts according to the Gestalt cues of spatial proximity. Of greater interest is a structure that allows for the perceptual grouping of elements close enough in spatial or feature space via synchronization, while segregating sharply more distant ones. We therefore consider a second class of profiles, which we shall call exponential, defined as d(1) =

1

( b - 1)b'-1

a-1 x-

af '

u > l

(3.5)

Notice that within this profile class a unit increment of the ultrametric separation corresponds to a reduction by a geometric factor u of the total coupling with oscillators at the new distance. Such profiles will lead to a discrete transition to nonlocal synchrony in the limit of a population of infinite size if the following inequalities are satisfied:

b> - aa/(O-*),

cy

>1

(3.6)

where the parameter cy measures the frequency dispersion according to equation 2.3. Provided that the asymptotic distribution of frequency drops faster than a power-law with exponent a=1, the above relations define a critical connectivity b, = aa/(o-l) in the network of interactions that increases with the profile factor a and the dispersion of frequency. As soon as the branching ratio drops below b,, the collective dynamics in the cell assembly changes dramatically. No global synchrony is possible in a system of infinite size; instead, increasing K drives the system through an infinite cascade of abrupt transitions to synchrony within clusters of increasingly larger dimensions, which reach global synchrony only in the unphysical limit of an infinite coupling constant. Also, a larger profile factor a results in broader windows of coupling constant values for which the maximum size of synchronized clusters remains unchanged.

Erik D. Lumer and Bernard0 A. Huberman

348

It is possible to show in a rigorous manner that the system described by equation 2.1 exhibits these distinct regimes, depending on whether b is smaller or larger than b,. The proof consists in renormalizing equation 2.1 so as to derive the equations of evolution for aggregated cluster oscillators at a given level I in the hierarchy. These equations are formally identical to equation 2.1, with the individual phases and frequencies being replaced by cluster averages and the interaction terms properly rescaled. When b is smaller than b,, the rescaling factor is such that the interactions terms converge to zero as the level I considered increases, a fact that indicates the absence of a single state of global synchrony for any finite value of the coupling constant (Daido 1988; Lumer and Huberman 1991). In the case of the branching ratio smaller than b,, the positive value of an order parameter for K larger than a critical threshold similar to that in equation 3.4 is the signature of a single phase transition to global synchrony. In place of a formal proof, we can obtain a qualitative understanding of these properties when the dispersion of firing rates is gaussian. Suppose that K is driven through the threshold value K, at which clusters, internally synchronized up to level 1 - 1, lock their frequencies. Then, each newly synchronized cluster at level I becomes equivalent to a single giant phase oscillator as far as its interactions with other such assemblies is concerned. Its overall intrinsic frequency is distributed according to a gaussian, which is now narrower by a factor & than the distribution of aggregated frequencies one level below. At the same time, the effective coupling between clusters at level I is reduced by the profile factor a. If &>a

(3.7)

Equation 3.4 shows that clusters at level I should also synchronize as K exceeds &. Since the same argument applies recursively to any two consecutive levels larger than I, one deduces that a single transition will lead to the bulk synchronization of all clusters larger than a certain size. (Notice that condition 3.7 is exactly the one predicted by equation 3.6 when a is set to 2.) If the effective coupling strength drops faster than the frequency dispersion, that is, a > 6, successive threshold values of K that correspond to a unit increment of the maximum ultrametric distance separating two synchronized oscillators are spaced by a multiplicative factor, which should converge to u/v% as one moves up in the coupling tree. We have simulated a discrete version of the system 2.1 for a population of 1024 oscillators. Equation 2.1 is integrated forward in time for 9000 iterations, with a temporal step dt = 0.05. We refer the reader to the caption of Figure 2 for a description of the simulations. To summarize our findings, we notice that when the coupling profile drops exponentially, a cascade of phase transitions is observed, and one can control the maximum size of synchronized clusters by an appropriate choice of the coupling constant. Notice, however, that the value of this coupling con-

Binding Hierarchies

349

stant increases roughly as the square root of the size of the synchronized population, a feature that is neither desirable nor implemented in real systems. Nevertheless, the exponential profile of coupling does provide a critical shape to which others might be easily compared. Consider, for example, a coupling profile that decreases slower than exponentially below a certain distance d and faster beyond, possibly becoming null at finite range. Our analysis indicates that in this case collective synchrony might be reached in clusters of size d or smaller for finite values of the coupling strengths, but with clusters of size d or larger being mutually incoherent. Thus, the properties of perceptual grouping, that is, the strong linking of related activities in spatial or feature space, along with the sharp desynchronization of separated groups, can be achieved. We point out that elaborate computer-based models of cortical systems implement a configuration of synchronizing connections whose density must fall off exponentially with distance in order to reproduce experimental observations (Sporns et al. 1991). Our study might be a theoretical justification for this empirical design.

4 A Binding Architecture Under suitable conditions, a population of oscillatorswith reciprocal coupling links is capable of grouping neighbor elements and segregating distant ones. However, this architecture presents several limitations in its grouping abilities. First, it was shown in a special case that synchronization in such assemblies is a very slow process (Kammen et al. 1989). Our numerical experiments confirm this point. Also, when the intrinsic firing rates suffer from a steady dispersion, the interactions have to be further increased for the synchronization (i.e., frequency-locking) to occur with a small phase spread. This entails a very high degree of connectivity among the cells. Kammen et al. (1989)recently demonstrated that special feedback units called comparators are very effective in inducing fast phase-locking between the neural oscillators they connect to in a fashion immune to high levels of external noise. In a simple comparator model, cells project to a unit that couples them via a common feedback of their average phase 6. Their dynamics is thus given by do. - = wj wl(6 - 0,) (4.1) dt Notice that equation 4.1 does not contain any spatial dimension. Therefore, the comparator will group similar stimuli regardless of their geometrical arrangement or separation, in contradiction with known Gestalt laws. We propose here an alternative architecture to the one presented in last section that combines the advantages of the comparator model with the segregation capabilities of the networks discussed earlier. To do so,

+


350

N. N

0 s

‘I

1

1.5

2

0.8

2.5

r“ I

3

j’

c::[ 0.2

f

0.5

1

1.5

2

2.5

3

Figure 2: Phase transitions to synchrony. The largest fraction of frequencylocked cells, N s / N , vs coupling constant in a population of 1024 oscillators is shown. Averages of the frequencies dO;/dt, a;, were computed to divide the population in synchronized clusters, where Is2j - Oil < 1/2T was used as the criterion of frequency locking between i and j, following Daido and others. T designates the integration time. (a) A single transition to global order. The parameters are choosen to satisfy eq. (3.6). Specifically, the profile function is exponential with a = 1.2, the distribution of frequencies gaussian (with variance D = 1.2) and the tree binary (b = 2). The coupling function has the form h(A0) = sin(2rAB)/Zr. (b) The profile is exponential with a = 2 and the same parameters as above. A cascade of transitions to synchrony up to a certain ultrametric distance is observed, as indicated by the heights of the plateaus equal to fractions of power of 2. Notice the different scale along the K-axis. (c) A discrete transition is recovered in (c) by doubling the branching ratio of the tree ( b = 4). The profile factor is still 2, which leads to the equality of both sides in relation 3.6. (d) The profile factor is set to its value in (a) but the frequency distribution here is Lorentzian (i.e., Q = 1). The simulation in this case seems to indicate the absence of global synchrony for any finite value of K . Indeed, although about 80%of the population is synchronized for K = 3, we verified that only 90% of the cells were synchronized for K = 27. This behavior contrasts with that in simulations (a-c), where a 100% synchronization is obtained shortly after more than half of the cells have locked their frequencies. Further simulations with larger populations are needed to validate the theoretical predictions in this case, which are derived in the limit of an infinite number of coupled cells.

Binding Hierarchies

351

we embed the feedback units in a tree in a way such that the coupling strength between successive levels decreases exponentially. Such a structure of minimal connectivity seems particularly relevant to support possible neurobiological binding mechanism (Damasio 1990; Crick and Koch 1991). We constructed a hierarchy of modified comparators by assuming that each node at any level receives as input the phases from its descendant nodes, computes their average value, and increments it with the coupling term fed back by its ancestor. The resulting value represents the node updated phase placed at its output connections. It is propagated up to its ancestor unit and fed back to its own children. We made the additional assumption that coupling is effective only between excited cells, that is, the ones receiving detectable activities at their inputs. This is the case, for instance, if the coupling links between real neurons (as opposed to the reduced phase oscillators) are modulatory (Eckhorn et al. 1989). A discrete dynamics of the system can be expressed as follows. The phases of the neural oscillators are updated according to Bj(t

+ 6t) = B j ( t ) + St x {wj + T

(z)

Kh [Ba,](t) - B j ( t ) ]

}

(4.2)

where Oak]. is the average phase of j’s ancestor feedback unit. The threshold of activity T is defined by

($)-{0

T - -

ifz>w,

1 otherwise

(4.3)

where GBjlSt represents j’s instantaneous frequency. The phases of the units at level 2 larger than 1 and less than L - 1 evolve according to

The sum on the right side of equation 4.4 extends over the Ncbl active children of j. d(2) gives the profile decrement at level 2. Finally, the updated activity of the root unit is given by (4.5) As for models of networks with horizontal connections, the proposed hierarchy of feedback units possesses a number of free parameters, in particular its profile function, d(2). The analysis of the previous section suggests that exponential profiles of coupling might again play a unique role in defining the range of macroscopic behaviors exhibited as the con-

352


Figure 3: A cascade of phase transitions to phase-locking of progressively larger clusters is observed in a 16 x 16 array as the coupling constant is increased. Successive values of K are 1, 15, 25, and 40. All the oscillators are excited. The instantaneous phase of each oscillator is plotted as a function of its position in the array. The coupling profile is exponential with a decrement factor of 2 between successive levels in the hierarchy. The coupling function is sinusoidal. The results are obtained after only 50 iterations of the system dynamics. The distribution of firing rates is gaussian with variance u = 0.12 and external noise is added. The range of fluctuations of the noise is equal to the variance of the frequency distribution.

trol parameter K is driven across its critical values. Simulations confirm this conjecture. Figure 3a-d illustrates the clustered activities of oscillators arranged in a 16 x 16 array at the bottom of the coupling tree. The two-dimensional tree has a branching ratio of 4. A cascade of transitions to collective phase-locking is observed when the coupling strengths drop fast enough (i.e., exponentially with a geometric factor of 2) with ultrametric distance. The effect of external noise on the short-range phase coherence is canceled by the averaging process in the comparators. Notice also the fast synchronization of clusters achieved after only a few iterations of the binding mechanism. Figure 4 demonstrates the progressive loss of synchrony between the representations of two localized stimuli inputting on a 32 x 32 array of model neurons, as they drift apart. The profile of coupling strengths is again exponential.

Binding Hierarchies

353

‘T

Figure 4: Perceptual grouping and segmentation. Two stimuli are presented to a 32 x 32 array of model neurons. Each stimulus covers a 4 x 4 array of cells. The coupling constant K is set to a value of 30. The other parameters are as in the previous figure. Notice the progressive loss of correlation between the excited regions as the stimuli are moved farther apart.

5 Conclusion Recent experiments seem to indicate the presence of extended coherence within assemblies of cells coding similar features in visual scenes. As a possible explanation of these observations, several authors have conjectured that the brain binds fragmentary representations of perceptual events via synchronization of oscillating groups of neurons. If such is the case, it is important to determine how extended synchronization can occur in a clustered organization of cells oscillating with a distribution of firing rates. To answer that question we established the basic conditions for the feasibility of a binding mechanism based on synchronized oscillations. Constraints were placed on the connectivity between cells, their various coupling strengths, and the distribution of their intrinsic firing rates. We showed that the coupling strength has to drop fast enough with distance to avoid bulk synchronization among all the oscillators. On the other hand, a large dispersion of firing rates precludes the grouping of percepts via neural synchronization for finite values of the coupling strengths. We also designed an architecture that could be regarded as a direct model of neurological mechanisms akin to those described qualitatively by Damasio. Such a structure can be used as an artificial perception network, thus providing both a possibly useful machine and an experimental tool for the study of perceptual binding mechanisms.

354


We are currently exploring two possible extensions of the model. The first one imposes adaptive coupling strengths in the hierarchy, so that perceptual grouping and segregation can be learned dynamically. The second allows for the feedback units at various levels to remember the patterns of activities received from their children nodes as well as the corresponding coupling strengths after convergence. This should enable us to study how perceptual memories can be distributed at various scales throughout the hierarchy and retrieved at a later time.

Acknowledgments We thank a referee for constructive remarks. This work was partially supported by the U. S. Office of Naval Research, Contract No. N0001482-0699.

References Atiya, A., and Baldi, P. 1989. Oscillations and synchronizations in neural networks: An exploration of the labeling hypothesis. Int. J. Neural Sysf. 1(2), 103-124. Baldi, P., and Meir, R. 1990. Computing with arrays of coupled oscillators: An application to preattentive texture discrimination. Neural Cornp. 2(4), 459471. Cohen, A. H., Holmes, I? J., and Rand, R. H. 1982. The nature of the coupling between segmental oscillators of the lamprey spinal generator for locomotion: A mathematical model. J. Math. Biol. 13, 345-369. Crick, F. H. C., and Koch, C. 1990. Towards a neurobiological theory of consciousness. Sem. Neurosci. 2, 263-275. Daido, H. 1988. Lower critical dimension for population of oscillators with randomly distributed frequencies: A renormalization-group analysis. Phys. Rev. Lett. 61(2), 231-234. Damasio, A. R. 1989. The brain binds entities and events by multiregional activation from convergence zones. Neural Comp. 1,123-132. Duncan, J. 1984. Selective attention and the organization of visual information. J. Exp. Psych.: General 113(4), 501-517. Eckhorn, R., Bauer, R., Jordan, W., Brosh, M., Kruse, W., Munk, M., and Reitboeck, H. J. 1988. Coherent oscillation%:A mechanism of feature linking in the visual cortex? Multiple electrode and correlation analysis in the cat. Biol. Cybernet. 60, 121-130. Eckhorn, R., Reitboeck, H. J., Arndt, M., and Dicke, P. 1989. A neural network for feature linking via synchronous activity: Results from cat visual cortex and from simulations. In Models of Brain Function, R. M. J. Cotterill, ed., pp. 255-272. Cambridge Univ. Press, Cambridge. Gilbert, C. D., and Wiesel, T. N. 1983. Clustered intrinsic connections in cat visual cortex. J. Neurosci. 3, 1116-1133.

Binding Hierarchies

355

Gray, C. M., Konig, P.,Engel, A. K., and Singer, W. 1989. Oscillatory responses in cat visual cortex exhibit inter-columnar synchronization which reflects global stimulus properties. Nature (London) 338, 334-337. Hummel, J. E., and Biederman, I. 1990. Dynamic binding: A basis for the representation of shape by neural networks. In Proceedings of the 12th Annual Conference of the Cognitive Science Society, pp. 614-621. Lawrence Erlbaum, Hillsdale, NJ. Kammen, D., Koch, C., and Holmes, P. J. 1989. Collective oscillations in the visual cortex. In Advances in Neural information Processing Systems 2 ,D. Z. Anderson, ed. Morgan Kaufmann, San Mateo, CA. Kuramoto, Y. 1984. Progr. Theor. Phys. (Suppl.). 79, 223-241. Kuramoto, Y. 1990. Collective synchronization of pulse-coupled oscillators and excitable units. Physica D,submitted. Lumer, E., and Huberman, B. A. 1991. Hierarchical dynamics in large assemblies of interacting oscillators. Phys. Lett. A 160(3), 1236-1244. Mountcastle, V. B. 1977. An organizing principle for cerebral function: The unit module and the distributed system. In Neuroscience4th Study Program, M.I.T. Press, Cambridge, MA. Pabst, M., Reitboeck, H. J., and Eckhorn, R. 1989. A model of preattentive region definition based on texture analysis. In Models of Brain Function, R. M. J. Cotterill, ed., pp. 137-150. Cambridge Univ. Press, Cambridge. Shepard, R. N. 1980. Multidimensional scaling, tree-fitting, and clustering. Sczence 210, 390-398. Sompolinsky, H., Golomb, D., and Kleinfeld, D. 1991. Cooperative dynamics in visual processing. Phys. Rev.A 43, 6990-7011. Sporns, O., Gally, J. A., Reeke, G. N., and Edelman, G. M. 1989. Reentrant signaling among simulated neuronal groups leads to coherency in their oscillatory activity. Proc. Natl. Acad. Sci. U S A . 86, 7265-7269. Sporns, O., Gally, J. A., Reeke, G. N., and Edelman, G. M. 1991. Modeling perceptual grouping and figure-ground segregation by means of active reentrant connections. Proc. Natl. Acad. Sci. U.S.A. 88, 129-133. Von der Malsburg, Ch. 1981. The correlation theory of brain function. Internal Report 81-2, Dept. of Neurobiology, Max Planck Institute for Biophysical Chemistry, Gottigen. Von der Malsburg, Ch., and Schneider, W. 1986. A neural cocktail-party processor. Biol. Cybernet. 54,29-40. Winfree, A. T. 1980. The Geometry of Biological Time. Springer-Verlag, Berlin.

Received 4 April 1991; accepted 8 November 1991.

This article has been cited by: 1. Juan Acebrón, L. Bonilla, Conrad Pérez Vicente, Félix Ritort, Renato Spigler. 2005. The Kuramoto model: A simple paradigm for synchronization phenomena. Reviews of Modern Physics 77:1, 137-185. [CrossRef] 2. Karl J. Friston, G. Tononi, O. Sporns, G. M. Edelman. 1995. Characterising the complexity of neuronal interactions. Human Brain Mapping 3:4, 302-314. [CrossRef] 3. J. Heagy, T. Carroll, L. Pecora. 1994. Synchronous chaos in coupled oscillator systems. Physical Review E 50:3, 1874-1885. [CrossRef] 4. Michael C. Mozer , Richard S. Zemel , Marlene Behrmann , Christopher K. I. Williams . 1992. Learning to Segment Images Using Dynamic Feature BindingLearning to Segment Images Using Dynamic Feature Binding. Neural Computation 4:5, 650-665. [Abstract] [PDF] [PDF Plus]

Communicated by Rodney Brooks

A Distributed Neural Network Architecture for Hexapod Robot Locomotion Randall D. Beer Departments of Computer Engineering and Science and Biology, Case Western Reserve University, Cleveland, OH 44106 U S A

Hillel J. Chiel Departments of Biology and Neuroscience, Case Western Reserve University, Cleveland, OH 44106 U S A

Roger D. Quinn Kenneth S. Espenschied Department of Mechanical and Aerospace Engineering, Case Western Reserve University, Cleveland, OH 44106 U S A

Patrik Larsson Department of Electrical Engineering and Applied Physics, Case Western Reserve University, Cleveland, OH 44106 U S A

We present a fully distributed neural network architecture for controlling the locomotion of a hexapod robot. The design of this network is directly based on work on the neuroethology of insect locomotion. Previously, we demonstrated in simulation that this controller could generate a continuous range of statically stable insect-like gaits as the activity of a single command neuron was varied and that it was robust to a variety of lesions. We now report that the controller can be utilized to direct the locomotion of an actual six-legged robot, and that it exhibits a range of gaits and degree of robustness in the real world that is quite similar to that observed in simulation. 1 Introduction

Even simpler animals are capable of feats of sensorimotor control that exceed those of our most sophisticated robots. Insects, for example, can walk rapidly over rough terrain with a variety of gaits and can immediately adapt to changes in load and leg damage, as well as developmental changes (Graham 1985). Even on flat horizontal surfaces, insects walk with a variety of different gaits at different speeds (Wilson 1966). These gaits range from the wave gait, in which only one leg steps at a time in a back-to-front sequence on each side of the body (this sequence is called Neural Computation 4,356-365 (1992) @ 1992 Massachusetts Institute of Technology

Neural Network Architecture for Hexapod Robot Locomotion

357

Figure 1: A comparison of simulated and robot gaits. Black bars represent the swing phase of a leg and the space between bars represents its stance phase. (Top) Leg labeling conventions. (Left) Selected gaits observed in simulation as the activity of the command neuron is varied from lowest (top) to highest (bottom) (Beer 1990). (Right) Gaits generated by the robot under corresponding conditions. Here the duration of a swing bar is 0.5 seconds. a metachronal wave), to the tripod gait, in which the front and back legs on each side of the body step in unison with the middle leg on the opposite side (see left side of Fig. 1). While most current research in legged robot locomotion utilizes centralized control approaches that are computationally expensive and brittle, insect nervous systems are distributed and robust. What can we learn from biology? In previous work (Beer et at. 1989), w e described a neural network architecture for hexapod locomotion. The design of this network was based on work on the neuroethology of insect locomotion, especially Pearson's flexor burst-generator model for walking in the American cockroach (Periplaneta americana) (Pearson et al. 1973; Pearson 1976). Through simulation, we demonstrated that this network was capable of generating a continuous range of statically stable gaits similar to those observed

358

Randall D. Beer et al.

in insects (see left side of Fig. 11, as well as smooth transitions between these gaits. The different gaits were produced simply by varying the tonic level of activity of a single command neuron. In addition, a lesion study of this network demonstrated both its surprising robustness and the subtlety of the interaction between its central and peripheral components (Chiel and Beer 1989). A natural question to ask is whether these results were just artifacts of the many physical simplifications of the simulation or whether they are robust properties of the network that persist in the presence of such physical realities as delay, friction, inertia, and noise. This is a difficult question to resolve given the subtle dependencies of this controller on sensory feedback (Chiel and Beer 1989). The only way to determine whether this distributed controller had any practical utility was to design and build a six-legged robot and interface it to the locomotion network. 2 Locomotion Controller

The circuit responsible for controlling each leg is shown in Figure 2. Each leg controller operates in the following manner: Normally, the foot motor neuron is active (i.e., the leg is down and supporting weight) and excitation from the command neuron causes the backward swing motor neuron to move the leg back, resulting in a stance phase. Periodically, this stance phase is interrupted by a burst from the pacemaker, which inhibits the backward swing and foot motor neurons and excites the forward swing motor neuron, resulting in a swing phase. The time between bursts in the pacemaker, as well as the velocity output of the backward swing motor neuron during a stance phase, depend on the level of excitation provided by the command neuron. In addition, sensory feedback is capable of resetting the pacemaker neuron, with the forward angle sensor encouraging the pacemaker to terminate a burst when the leg is at an extreme forward position and the backward angle sensor encouraging the pacemaker to begin a burst when the leg is at an extreme backward position. There are six copies of the leg controIler circuit, one for each leg, except that the single command neuron makes the same two connections on each of them. Following Pearson’s model, the pacemakers of all adjacent leg controllers mutually inhibit one another, discouraging adjacent legs from swinging at the same time (Fig. 3). At high speeds of walking, this architecture is sufficient to reliably generate a tripod gait. However, at lower speeds of walking, the network is underconstrained, and there is no guarantee that the resulting gaits will be statically stable. To enforce the generation of metachronal waves, we added the additional constraint that the natural periods of the pacemakers are arranged in a gradient, with longer periods in the back than in the front (Graham 1977). Under these conditions, the pacemakers phase-lock into a stable metachronal


359

Command Backward Angle Sensor

r

Swing

-t

Pacemaker

uw Excitatory Connection

ForwardAngle Sensor

c Inhibitory Connection

Figure 2: The leg control circuit. Each leg is monitored by two sensory neurons that signal when it has reached an extreme forward or backward position. Each leg is controlled by three motor neurons responsible for the state of the foot, the velocity with which the leg swings forward, and the velocity with which the leg swings backward, respectively. The motor neurons are driven by a pacemaker neuron whose output rhythmically oscillates. A single command neuron makes the same two connections on every leg controller. The architecture also includes direct connections from the forward angle sensor to the motor neurons, duplicating a leg reflex known to exist in the cockroach. The state of each neuron c j ~ j i f j ( V j ) INTi EXTi, is governed by the equation CidVildt = -Vi/Ri where Vi, Ri, and Ci, respectively, represent the voltage, membrane resistance, and membrane capacitance of the ith neuron, wji is the strength of the connection from the jth to the ith neuron, f is a saturating linear threshold activation function, and EXTi is the external current injected into the neuron. INTi is an intrinsic current present only in the pacemaker neurons that causes them to oscillate. This current switches between a high state of fixed duration and a low state whose duration depends linearly on the tonic level of synaptic input, with excitation decreasing this duration and inhibition increasing it. In addition, a brief inhibitory pulse occurring during a burst or a brief excitatory pulse occurring between bursts can reset the bursting rhythm of the pacemaker.

+

+

+

360


Figure 3: The pacemaker neurons of adjacent leg controllers are coupled by mutual inhibition. relationship. We chose to enforce this constraint by making the range of motion of the rear legs slightly larger than that of the middle legs, whose range of motion in turn is slightly larger than that of the front legs. A complete discussion of the design of this network and its relationship to Pearson’s model can be found in Beer (1990). 3 Robot

To examine the practical utility of this locomotion controller, we designed and built a six-legged robot (Fig. 4, top). The network was simulated on a personal computer using the C programming language and interfaced with the robot via A / D and D/A boards. Because the controller was originally designed for a simpler simulated body (see top of Fig. l),two main issues had to be addressed in order to connect this controller to the robot. First, the locomotion controller assumes that the swing and lift motions of the leg are independent, whereas in the robot these two degrees of freedom are coupled (Fig. 4, bottom). In simulation, this problem was dealt with by having a stancing leg passively stretch between its joint and foot. For the robot to maintain a constant height (h) above the ground, the radial length (Y) of a stancing leg must be adjusted by the simple kinematic transformation Y = h/ cos 0.


361

Figure 4: (Top) The hexapod robot. Its dimensions are 50 cm long by 30 cm wide and it weighs approximately 1 kg. (Bottom) Each leg has two degrees of freedom: an angular motion responsible for swing and stance movements and a radial motion involved in raising and lowering the leg. The swing motion has a range of over 45" from vertical in either direction. The radial motion is accomplished by means of a rack-and-pinion transmission. Both degrees of freedom are driven by 2 W DC motors with associated integral transmissions. Position sensing for each degree of freedom is provided by potentiometers mounted in parallel with the motors.

362


A second compatibility issue derives from the simplified physics utilized in the original simulation, in which the activity of the forward and backward swing motor neurons was translated directly into velocities (Beer 1990). To interface the output of the neural network controller to the physical dynamics of the robot, we made use of the equilibrium point hypothesis for motor control (for review see Bizzi and Mussa-Ivaldi 1990). This hypothesis states that the nervous system generates limb trajectories not by directly specifying joint torques but rather by specifying a sequence of equilibrium positions known as a virtual trajectory. This hypothesis is based on the following two facts: (1) muscles have springlike properties that are in equilibrium when the torques generated by opposing muscles exactly cancel and (2) neural input to a muscle has the effect of selecting a length/ tension curve and therefore an equilibrium position for the limb as a whole. The equilibrium point hypothesis suggests the following approach. The velocity output of the network is integrated to obtain a sequence of virtual positions. These virtual positions are then translated (via the trigonometric transformations described above) into sequences of positions of the swing and lift motors. Finally, these swing and lift positions are fed into position controllers that drive the corresponding motors with voltages proportional to the deviations between the actual positions and the desired positions. These position controllers are implemented in analog circuitry for speed. Because the motors are backdrivable, this scheme also gives a spring-like property to the legs that lends stability to the robot.

4 Results and Discussion

Under the control of the locomotion network, the robot exhibits a range of gaits similar to those observed in simulation as the command neuron activity is changed (see right side of Fig. 1). These gaits range from ones in which distinct metachronal waves are readily distinguished at low speeds of walking to the tripod gait at high speeds of walking. Within this range of gaits, the robot's speed of progression varies from 4.5 to 8.3 cm/sec. In addition, we studied the effects on the robot's walking of a number of lesions that were previously performed in simulation (Chiel and Beer 1989). In all cases, the response of the physical robot was quite similar to what we had previously observed in simulation (Chiel et al., in press). The controller was able to cope with the removal of such components as single sensors, a small fraction of the coupling connections between pacemakers, or the command neuron to pacemaker connections. In addition, we found that the robot was capable of reflex stepping. If the command neuron is disabled but the robot is steadily pushed forward at different speeds by externally updating the position


363

controllers, then it still exhibits the full range of gaits. Thus it appears that the continuous range of gaits is a robust property of the locomotion network and not simply an accident of simulation. Interestingly, this robotic implementation did reveal one weakness of the locomotion controller that we did not think to examine in simulation. While we have found the controller to be quite robust in general to the delays inherent in the physical robot, it is sensitive to asymmetic delays that cause the legs on one side of the body to consistently lag behind those on the opposite side. These asymmetric delays are due to the inevitable variations in the response characteristics of the electrical and mechanical components of the robot. In the presence of such asymmetric delays, the crossbody phasing of the legs is disturbed. Once we identified this problem, however, a simple adjustment to the stiffnesses of the position controllers, which affects the amount that a leg lags behind its prescribed position, restored the proper phasing. Nevertheless, discoveries such as this justify the effort involved in undertaking a robotic implementation. Brooks has described a partially distributed locomotion controller for a six-legged robot known as Genghis (Brooks 1989). This robot is controlled by a network of finite state machines augmented with registers and timers. In Brooks’ locomotion controller, the basic swing/stance cycle of each leg is driven by a chain of reflexes involving only coarse local information about leg position and load. For example, whenever a leg is lifted for some reason, it reflexively swings forward and whenever one leg swings forward, all other legs move backward slightly. With elaborations of this basic controller, the robot not only sqcessfully walked, but could also negotiate small obstacles and follow slowly moving objects with infrared sensors. However, Brooks’ controller is not as fully distributed as the network architecture described in this paper. In Genghis, the movements of the individual leg controllers are coordinated by a single, centralized finite state machine which tells each leg when to lift. Different gaits are generated by moddying this machine. While Maes and Brooks (1990) have recently described a version of this controller that does not require a centralized gait sequencer, their new controller is capable of generating only the tripod gait. In contrast, our neural network controller generates a continuous range of gaits without any centralized gait sequencer. Instead, the different gaits result from the dynamics of interaction between the pacemaker neurons controlling each leg and the sensory feedback that they receive. The architecture described in this paper focuses solely on the problem of sequencing leg movements, so as to maintain static stability at a variety of walking speeds during straight-line locomotion across flat, horizontal surfaces. Of course, gait control is only one aspect of legged locomotion. Other important issues include postural control, turning, negotiation of complex terrain, and compensation for leg damage or loss. In future work, we plan to draw on further studies of insect locomotion and its

364


neural basis to address these additional issues (Burrows 1989; Pearson and Franklin 1984; Cruse 1990). If we had taken a classical control approach to legged locomotion, it is unlikely that a distributed architecture such as the one we have presented here would have resulted. We believe that our results represent a simple example of a very powerful idea: neural network architectures abstracted from biological systems can be directly applied to the control of autonomous agents (Beer 1990). Because they have evolved over a significant period of time, biological control systems are much more flexible and robust than their engineered counterparts. However, they are also much more difficult to understand. Simulation can serve as a n important intermediate step in the process of abstracting a biological control principle. On the other hand, only a physical implementation in an actual robot can prove such a principle’s practical utility. In this paper, we have demonstrated that our distributed locomotion controller is a viable approach to hexapod robot wallung.

Acknowledgments This work was supported by Grant N00014-90-J-1545 to R. D. B. from the Office of Naval Research and Grant NGT-50588 from NASA Goddard. Additional support was provided by the Cleveland Advanced Manufacturing Program through the Center for Automation and Intelligent Systems Research and the Howard Hughes Medical Institute. H. J. C. gratefully acknowledges the support 07 the NSF through Grant BNS8810757.

References Beer, R. D. 1990. Intelligence as Adaptive Behavior: An Experiment in Computational Neuroethology. Academic Press, San Diego. Beer, R. D., Chiel, H. J., and Sterling, L. S. 1989. Heterogeneousneural networks for adaptive behavior in dynamic environments. In Advances in Neural lnformation Processing Systems 1 , D. S . Touretzky, ed., pp. 577-585. Morgan Kaufmann, San Mateo, CA. Bizzi, E., and Mussa-Ivaldi, F. A. 1990. Muscle properties and the control of arm movement. In An lnvitation to Cognitive Science, Volume 2: Visual Cognition and Action, D. N. Osherson, S. M. Kosslyn, and J. M. Hollerbach, eds., pp. 213242. MIT Press, Cambridge, MA. Brooks, R. A. 1989. A robot that walks; emergent behaviors from a carefully evolved network. Neural Comp. 1(2), 253-262. Burrows, M. 1989. Processing of mechanosensory signals in local reflex pathways of the locust. ]. Exp. Biol. 146, 209-227.


365

Chiel, H. J., and Beer, R. D. 1989. A lesion study of a heterogeneous neural network for hexapod locomotion. Proc. Int. J. Conf. Neural Networks [IJCNN 891, I, 407-414. Chiel, H. J., Beer, R. D., Quinn, R. D., and Espenschied, K. S. In press. Robustness of a distributed neural network controller for locomotion in a hexapod robot. To appear in IEEE Transactions on Robofics and Automation. Cruse, H. 1990. What mechanisms coordinate leg movement in walking arthropods? Trends Neurosci. 13(1), 15-21. Graham, D. 1985. Pattern and control of walking in insects. Adu. Insect Physiol. 18, 31-140. Graham, D. 1977. Simulation of a model for the coordination of leg movements in free walking insects. Biol. Cybernet. 26, 187-198. Maes, P., and Brooks, R. A. 1990. Learning to coordinate behaviors. Proc. Eighth Natl. Conf. AI [AAAI-90], 796-802. Pearson, K. G. 1976. The control of walking. Sci. Am. 235, 72-86. Pearson, K. G., Fourtner, C. R., and Wong, R. K. 1973. Nervous control of walking in the cockroach. In Control of Posture and Locomotion, R. 8. Stein, K. G. Pearson, R. S. Smith, and J. B. Bedford, eds., pp. 495-514. Plenum Press, New York. Pearson, K. G., and Franklin, R. 1984. Characteristics of leg movements and patterns of coordination in locusts walking on rough terrain. Int. J. Robotics Res. 3(2), 101-112 Wilson, D. M. 1966. Insect walking. Annu. Rev. Entomol. 11, 103-122.

Received 9 July 1991; accepted 9 October 1991.

This article has been cited by: 1. QiDi Wu, ChengJu Liu, JiaQi Zhang, QiJun Chen. 2009. Survey of locomotion control of legged robots inspired by biological concept. Science in China Series F: Information Sciences 52:10, 1715-1729. [CrossRef] 2. Tao Geng , Bernd Porr , Florentin Wörgötter . 2006. A Reflexive Neural Network for Dynamic Biped Walking ControlA Reflexive Neural Network for Dynamic Biped Walking Control. Neural Computation 18:5, 1156-1196. [Abstract] [PDF] [PDF Plus] 3. Philip Holmes, Robert J. Full, Dan Koditschek, John Guckenheimer. 2006. The Dynamics of Legged Locomotion: Models, Analyses, and Challenges. SIAM Review 48:2, 207. [CrossRef] 4. P. Arena, L. Fortuna, M. Frasca, G. Sicurella. 2004. An Adaptive, Self-Organizing Dynamical System for Hierarchical Control of Bio-Inspired Locomotion. IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) 34:4, 1823-1837. [CrossRef] 5. D.W. Marhefka, D.E. Orin, J.P. Schmiedeler, K.J. Waldron. 2003. Intelligent control of quadruped gallops. IEEE/ASME Transactions on Mechatronics 8:4, 446-456. [CrossRef] 6. K.S. Narendra. 1996. Neural networks for control theory and practice. Proceedings of the IEEE 84:10, 1385-1406. [CrossRef] 7. H. Cruse, C. Bartling, G. Cymbalyuk, J. Dean, M. Dreifert. 1995. A modular artificial neural net for controlling a six-legged walking system. Biological Cybernetics 72:5, 421-430. [CrossRef] 8. J. J. Collins, S. A. Richmond. 1994. Hard-wired central pattern generators for quadrupedal locomotion. Biological Cybernetics 71:5, 375-385. [CrossRef] 9. Shinichi Kimura, Masafuni Yano, Hiroshi Shimizu. 1994. A self-organizing model of walking patterns of insects. Biological Cybernetics 70:6, 505-512. [CrossRef]

Communicated by John Platt and David Willshaw

Multilayer Perceptron Learning Optimized for On-Chip Implementation: A Noise-Robust System Alan F. Murray Department of Electrical Engineering, University of Edinburgh, Edinburgh EH9 3JL, Scotland

1 Requirement

This paper describes an approach to multilayer perceptron (MLP) learning that is optimized for hardware implementation. Experimental results to date are promising, and it is the express aim of this paper to present these results concisely - a detailed mathematical analysis will follow in a subsequent, longer publication. Error backpropagation (Rumelhart et al. 1986) has achieved remarkable success as an algorithm for solving hard classification problems with MLP networks. It is, however, not readily amenable to VLSI integration, and the distinction it draws between hidden and output nodes renders it hostile to analog circuit forms. Use of the mathematical chain rule, to calculate the effect of a weight connecting to a hidden unit on the errors {Q} in the output layer, renders the error calculation scheme for hidden units different from, and more complicated than that for output units. The Virtual Targets learning scheme circumvents this problem by introducing an explicit "desired state," or target for each of the hidden units, which is updated continuously, and stored along with the synapse weights. While this means that a target state must be stored for each input pattern and hidden node, it simplifies and renders homogeneous the process of weight evolution for all neurons. Furthermore, since a target state is already stored for each output neuron, the scheme essentially removes the distinction during learning between hidden and output nodes. Analog integrated circuits based on the virtual targets strategy will therefore be flexible in architectural terms, as all units will be configurable as either output or hidden layer neurons. The fundamental idea of adapting the internal representation, either as well as, or instead of the weights, is not itself new (Rohwer 1990; Grossman et al. 1990; Krogh et al. 1990). However, these pieces of work were not optimized for hardware implementation. The fundamental difference is that simplicity of implementation has been made the primary goal in the work described in this paper, to produce a system optimized for analog VLSI. There are also several important differences in detail between the work described in this paper and these earlier, similar Neural Computation

4, 366-381 (1992)


MLP Learning for On-Chip Implementation

367

approaches. These will be indicated in the next section. Within the boundaries of the tasks it has been applied to, the new scheme does not suffer from such problems, and is furthermore apparently viable as an analog VLSI learning process. Far from being impeded in its efficiency by synaptic inaccuracy in the form of electrical noise, performance is actually enhanced b y the presence of this unavoidable uncertainty. This contradicts the commonly held view that, since analog systems are inherently "inaccurate," because of noise, the requirement for high weight precision during MLP (and other) learning procedures cannot be satisfied with such systems. Equating noise-limited accuracy with digital wordlength-limited accuracy is therefore misleading, and the existing dogma regarding the need for very high "precision" during MLP learning should be reevaluated. In Section 2 of this paper, the Virtual Targets learning approach is detailed, and some experimental results shown in Section 3. Finally, in Sections 4 and 5, a preliminary analog learning chip architecture is proposed, and some conclusions are drawn. 2 "Virtual Targets" Method In an kJ:K MLP Network

The J hidden- and K output-layer neurons obey the usual equations [e.g., ok = f(Xk), where xk = C wkjoj]. Weights evolve in response to the presentation of a pattern p via perceptron-like equations similar to those used in backpropagation. (2.1) (2.2)

where, for instance, output layer errors are ekp = 8kp - okp, where { Z k p } are the target states. The terms ob etc. represent the derivatives of the activation function, 60kp/6Xkpr which effectively discourage learning on weights that connect to neurons that are firmly OFF or ON. The terms ojp and oIp discourage learning on weights that connect from neurons that are firmly OFF. vweights represents learning speed. Note in passing that equations 2.1 and 2.2 involve information local to the layers that are being connected. The departure from backpropagation is the presence of an error signal for the hidden nodes - this is the crux of the virtual targets scheme. Weights are initialized to random values. The learning scheme then operates as follows: 1. Apply input pattern {oip}, and read out the states {ojp}, hidden and output nodes.

2. Assign targets for the hidden nodes,

Ojp

= ojP.

{okp}

of the

Alan F. Murray

368

3. Repeat (1) and (2) for all input patterns. 4. Present patterns in random order, allowing weights to evolve according to equations 2.1 and 2.2, and targets { O j p } according to k=K

"jP = ??targets 6t

EkpWkj

(2.3)

k=O

where qtargets is the target learning speed. In simulation, equation 2.3 must be multiplied by an additional term of the form Ojp(l - Ojp), to restrain hidden node targets to the range 0 5 Olp 5 1. When the target values are stored on-chip as charge on capacitors, this "saturation" will occur naturally. Operations (1) -+ (3) are "initialization" steps, and only (4), with equations 2.1-2.3, describes learning. During learning via equations 2.12.3, whenever the output targets { O k p } are achieved for a particular input pattern p (i.e., the net has learned that pattern successfully) the hidden node targets {Ojp} for that pattern are reinitialized as in (2) above. When a pattern p has been learned successfully, the errors { ~ k p }are, by definition, small. Equations 2.1 and 2.2 will no longer cause weight and target modifications, respectively. The target values { O j p } may, however, not be equal to the states {ojp}, and may merely be exerting a force on { Wji} via equation 2.2 in the appropriate direction. It is therefore necessary to introduce this "reset" mechanism, to cause the learning process to cease to react to pattern p - at least until the weights change to corrupt the successful coding of pattern p . There are several differences in detail between this and the earlier, similar algorithms (Krogh et al. 1990). In Rolwer's work (Rohwer 1990) the hidden node activations are made explicitly the primary independent variables, with the weights related to them by a set of linear equations. In the closest ancestor of the virtual targets method, the CHIR (CHoice of Internal Representation) algorithm of Grossman et al. (1990), the phases of learning that affect weights and target states are distinct, in contrast to the simpler virtual targets algorithm, which adapts weights and targets simultaneously, at different, constant speeds. It: is interesting to note that Grossman's group has evolved a derivative of CHIR that avoids the need to store target states, at the expense of a more complex weight update scheme (Nabutovsky et al. 1990). In the most general exposition of the target concept, (referred to here as the KTH algorithm) Krogh et al. (1990) have encapsulated the above two examples, and the virtual targets scheme, in a single formalism. In the notation of this paper, they introduce values for the hidden node "net-computed" states {ojp} of the form

Here, the parameter T controls the extent to which the hidden node states, or "internal representations" feel the effect of the normal forward flow


369

of data (high T ) and the backward flow of error information (low T ) . As Krogh et al. (1990) point out, the CHIR algorithm alternates between high and low T, depending on which set of weights is being adjusted. In the virtual targets method, the two phases of learning are coincident. The net-computed values of the internal representations {ojp} represent the high-T limit, while the target states {Ojp} are retrieved in the low-T limit. During learning these two are reconciled, and the virtual targets algorithm, with the added target reset feature, may be viewed as simply one manifestation of the KTH algorithm. The KTH algorithm can thus be configured to mimic the virtual targets scheme, in which limit it will perform similarly. In all of these target based schemes, the target states {5jP} are taking the pIace of the chain rule in the full backpropagation algorithm, to act as intermediaries in transmitting error ‘informationthrough the intermediate layers. Equations 2.1-2.3 describe the learning scheme’s response to a single pattern {oip} applied to the inputs, with output states compared to their target values (6kp). This is not how MLP training normally proceeds. To mimic conventional MLP learning, each of equations 2.1-2.3 should be allowed to evolve for a short time with each of the input patterns applied, in random order, and repeatedly. The simulation experiments in the next section were performed in exactly this way. It is then possible to derive equations that describe the time evolution of the error signals {CM} and {qP}over one learning epoch (a complete set of input presentations). These are

and (2.6)

where &bc = o i b o i c C d O d b o d c 2 0. Clearly, the individual terms in equations 2.5 and 2.6 can be variously positive or negative, according to the values of the weights and errors. However, the presence of two competing terms in equation 2.6, one modulated by the Vweights and the other by qtargetsis illuminating. This represents the competing forces of perceptron learning on the weights { Wq}, which aims to reduce {qP},and movement of the targets {6jp} to reduce { ~ k p } , which may increase {qP}.We might expect, therefore, that the will not decrease monotonically during learning, particularly during the early stages of learning, when the { E M } are substantial. In addition, equation 2.5 does not guarantee gradient descent in the { Ekp}, since equations 2.1-2.3 were constructed pragmatically, rather than with the express aim of producing gradient descent. We should not therefore be surprised to see occasional “hill climbing” in the output error.

370

Alan F. Murray

3 Experimental Results

The method was applied initially to the standard MLP test problems of Parity and Encoder-Decoder tasks, to determine its functional viability. No fundamental problems were encountered, apart from a tendency to get stuck in local minima, in exactly the same way as in backpropagation learning. To attempt to avoid these minima, noise was injected at the synapse and activity levels, with no serious aspiration that learning would survive such rigors. In other words, each synaptic weight { W,b} in forward pass mode was augmented by a noise source of variable strength, and each value of x, = C W,bob was similarly corrupted. Noise sources of up to 20% on either and both of these quantities were introduced. Somewhat surprisingly, learning on these conceptually simple ”hard” problems was actually improved by the presence of high levels of noise, and the network became stuck in local minima less frequently. Including noise in the backward calculations 2.1-2.3 neither improve nor degrade this result. Figure 1 shows an example of a 4-bit parity learning cycle, with a particularly “bad” set of (randomly chosen) initial weights. The noisefree network immediately settles into a local minimum, where it stays indefinitely. With noise, however, excursions from the local minimum are made around 2000 and 5300 learning epochs, and a solution is finally found at around 6600 epochs. The temporary minima in the error signal are not associated with one pathologically noisy learning epoch, and the hill-climbing seen in Figure 1 takes place over several noisy learning epochs. Learning is clearly enhanced by the presence of noise at a high level (around 20%) on both synaptic weights and activities. This result is surprising, in light of the normal assertion that backpropagation requires at least 16-bit precision during learning, and that analog VLSI is unsuitable for backpropagation. The distinction is that digital inaccuracy, determined by the significance of the least significant bit (LSB), implies that the smallest possible weight change during learning is 1 LSB. Analog inaccuracy is, however, fundamentally different, in being noise limited. In principle, infinitesimally small weight changes can be made, and the inaccuracy takes the form of a spread of ”actual” values of that weight as noise enters the forward pass. The underlying “accurate” weight does, however, maintain its accuracy as a time average, and the learning process is sufficiently slow to effectively “see through the relatively low levels of noise in an analog system. The implication is that while analog noise may introduce inappropriate changes in the {Wab} and {Olp}, the underlying trend reflects the accurate synaptic weight values, and makes the appropriate averaged adjustments. The incidental finding - that higher levels of noise actually assist learning - is not so easily explained, although injection of noise into adaptive filter training algorithms is not unusual. These two findings concerning noise would seem to be perfectly general, and have ramifica-


371

0.6

WITHOUT NOISE 0.5

d

fi

5 0.4

E 22

0.3

0.2

WITH NOISE 0.1

OO

5000

Figure 1: Learning cycle for 4-bit parity, with the same set of "bad" initial conditions, with and without noise. tions for all learning processes where weights evolve incrementally, and slowly. In Figure 2 the network settles into a poor local minimum shortly after learning commences, with a mean output error of around 0.45. At around 1300 learning epochs, a better, but still local minimum is found (output error 2 0.27). As the inset shows, the network climbs smoothly out of this local minimum. This effect can be explained as follows. As the network enters the local minimum, a subset of the output patterns is becoming coded correctly (definition of a local minimum). The patterns in this local subset are driving weight changes via equations 2.1-2.3. During this process, the output errors { E @ } are reducing, as are the hidden node errors { q p } . Once all of the patterns in the subset are "learned," the { F @ } are all zero, and the target reset mechanism sets the {qP}to zero, abruptly. In effect, the hidden node targets, and their associated errors, have been acting as elastic forces on learning, which are suddenly

Alan F. Murray

372

ERROR 1.2 1

0.8 0.6

MAXIMUMSINGLEERROR HIDDENLAYERERROR MEANOUTPUTERROR

_____ __________ -

!

0.4 0.2

0

So00

LEARNING EPOCHS

Figure 2: Learning cycle for parity operation showing hill-climbing in output error (inset). removed. If the local minimum is poor, the input patterns not in the local subset assert themselves via equations 2.1-2.3, and the system climbs out of the poor minimum. Only when the minimum is "good enough" as defined by the error criterion does it persist - as it does at N 7000 epochs in Figure 2. This unusual and surprising feature allows the system to respond appropriately to both "poor" and "good" minima, as defined by the output error criterion. It is, I believe, a consequence of the target method, coupled with the reset mechanism outlined in Section 2. Parity and Encoder-Decoder tasks are not representative of the real classification/generalizationproblems that an MLP might aspire to solve. As an example of a "real" classification task, with both training and test data sets, the Oxford/Alvey vowel database formed the vehicle for a 542711 MLP to learn to classlfy vowel sounds from 18 female and 15 male speakers, using the virtual targets strategy. The data appear as the analog outputs of 54 band-pass filters, for 11different vowel sounds, and 33 speakers. Figure 3 shows an example of a learning cycle involving the first 5 female speakers. The figure charts the evolution of the modulus of the average output error, the hidden node error, and the maximum single-bit output error. This latter error is included to avoid the situation where the average output error is extremely low in the presence of a number of large bit-errors in the output layer, simply because the number


373

ERROR

I

0

MAXIMUM SINGLE-BIT ERROR - OUTPUT LAYER

100 200 300 400 LEARNING TIME (SIMULATION EPOCHS)

500

Figure 3: Mean output error, hidden node error, and maximum single-bit error for learning 11 vowel sounds from 5 female speakers, with noise. of outputs is large. Such errors are not acceptable in a classification problem such as this. The maximum single-bit output error is the largest of all the output bit errors, for all nodes and all patterns. Not until it falls can the network be deemed to have learned the training set completely. Over some 500 simulation (learning) epochs, the errors can be seen to be reduced - not all monotonically. The presence of the noise alluded to above is fairly obvious - and is more so in Figure 4, which shows the results of two learning experiments, with and without noise. The experiments were in every other respect identical, using the same set of randomized initial weights. The noise-free traces are smoother, but learning is protracted over the noise-injected example. A solution is found in the absence of noise, and indeed local minima were found to be rare in this set of experiments, with or without noise. The results in Figure 4 are, however, dramatic, and characteristic of other similar tests. In each case, learning was ended when the maximum single bit error dropped below 0.1, and the noise signal was reduced in magnitude when this error dropped below 0.4. Interestingly, the generalization ability is also improved by the presence of noise on the synapses and activities -

Alan F. Murray

374

I

MEAN ERROR. HIDDEN LAYER O.WI

L Y

~

LMRNING WITH NOISE LEARNING WITHOUT NOISE

-’*

MEAN ERROR OUTPUTLAYER 0 IW 200 300 4al sw SIMULATION TIME (LEARNING EPOCHS)

Figure 4 Mean output error, hidden node error, and maximum single-bit error for learning 11 vowel sounds from 5 female speakers.

by up to 5%,and the results given above are for a ”noisy” network. The hidden layer errors peak before falling, while the output errors fall more or less monotonically. This is entirely consistent with the observations regarding equation 2.5, and the competing pressures on the hidden node errors. In all, the following experiments were conducted, with the results shown in Table 1. The results presented are averaged values over several learning experiments, with standard deviations in parentheses. These generalization results are broadly similar to those obtained using standard backpropagation, with a momentum term of 0.1, on a 54:2711 network (Watts 1990), although the backpropagation learning times were considerably longer (Table 2). The backpropagation results are taken from single ”best-case” experiments, without the averaging process in Table 1,


375

Table 1: Simulation Results - Virtual Targets Strategy Training set Test set

Learning time Generalization epochs % of test set correct mean (SD) Mean (SD) Best result

(%I ~~

5 female 10 female 5 male 10 male

~

13 female 8 female 10 male 5 male

196 (40) 280 (125) 308 (160) 186 (60)

(%I

~

~~

64 (4.3) 69 (2.4) 65 (3.0) 75 (3.5)

75 72 69 80

Table 2 Simulation Results - Backpropagation Algorithm Training set Test set

Learning time Generalization epochs % of test set correct

(96) 5 females 10 females 5 males 10 males

13 females 8 females 10 males 5 males

1452 1375 1826 1386

66 70 76 78

and as such are prone to the deleterious effects of a small database. Particularly when the training set is small (5 individuals), wide variations in learning "quality" between different experiments are to be expected. This accounts for the difference in generalization results between Table 1 and Table 2. The Virtual Targets strategy has, therefore, similar learning and generalization properties to backpropagation. This is hardly surprising, as it has the same conceptual roots. It is, however, optimized for implemen-tation in analog VLSI, as the following section indicates. In an attempt to clarify the role of noise, Figure 5 shows the effect of different levels of noise on a learning cycle with the same initial conditions. Initially, learning time is reduced by noise injection, as Figure 4 suggests. Increasing the noise level must, however, eventually swamp the data totally, and prevent the classification from being captured at all. This effect is seen in the upper trace in Figure 5, where learning times increase exponentially, at noise level of around 40%. However, the generalization ability (the measure of the quality of learning, as evidenced by the MLP's ability to classify unseen data correctly) rises essentially monotonically. Figure 5 suggests that a level of around 10-20% noise offers an optimal compromise between extended learning time for high levels of noise, and lower generalization ability for lower levels. The

5

-

500

10%

20%

30%

40%

PERCENTAGE OF SYNAITIC/ACTIVITY NOISE

Figure 5: Learning time and generalization ability as a function of injected noise level. most useful observation to be made at this stage is that corrupting the training data with noise is held to have the same effect as penalizing high curvature in decision boundaries - in other words it causes the network to draw sweeping curves through the decision space, rather than fitting convoluted curves to the individual training data points (Bishop 1990). In this way, underlying trends are modeled, while fine "irrelevant" detail is ignored. These two findings concerning noise would seem to be perfectly general in the neural context, and have ramifications for all learning processes where weights evolve incrementally, and slowly. Noise sources were inserted initially to model the noise known to be present in analog systems. As the method now stands, the noise sources are being used to improve both learning speed and quality. Analog noise takes the form of both DC and AC inaccuracies in voltages, currents, and device characteristics. DC offsets are essentially canceled out by


377

(a) FORWARD INFORMATION R O W Synapse

Neuron

qe_l%? 141 ,.I

-

..................

w aJ' 0J'

I

Wajoj

,"

0-

aj O j

0a

(b) BACKWARD INFORMATION FLOW Upd.1~ Wab via(l)or(2)

+jij

Slme

1 ap I and

updslc YIP (3)

o a ( 1 - Oa)

=::::::::::::::::::::. ,*.I

,*,

ib iP

,=,

Wib ip

Error from layer above

Figure 6: Proposed VLSI implementation scheme for information and error flow in the forward (a) and backward (b) directions, respectively. including the chip in the learning process [i.e., "chip-in-the-loop," as described by INTEL in the context of their E T A " chip (Holler et al. 198911. Natural AC noise is in general too weak to provide the high levels of noise that give optimal results here. However, Alspector has reported an elegant solution to this problem in the context of stochastic learning systems (Alspector et al. 1991) involving a small overhead in digital feedback register circuitry. Preliminary experiments suggest that in the work reported in this paper, the exact form of the noise is not critical, and that a simplified version of Alspector's circuitry will suffice. 4 Implementation

Figure 6 shows the flow of information in a virtual targets network. The forward flow of information is that of a standard MLP. In parallel with this, however, error signals are calculated and passed backward via equations 2.1-2.3. This implies that each synapse circuit must perform backward multiplication as implied by equation 2.3, at the same time as the multiplication for C W&,. We have already demonstrated (Murray et al. 1990, 1991) that multiplication can be performed using as few as 3 MOSFETs, with the bulk of the synapse circuit devoted to weight storage. The area and complexity overhead of a "two-way," single-multiplicand multiplier is therefore slight. Also indicated in Figure 6 is the storage requirement for the (adaptive) target states on the hidden units. This could be achieved via a set of on-chip memory elements for each of the {8jp}. It is more likely to be achieved optimally via a single set of on-chip

378

Alan F. Murray

memories for {ijj}, loading a new set for each pattern p along with the inputs, and reading the adapted targets {ijjp} along with the outputs. The synapse must be capable of incrementing and decrementing its own weight, while passing state signals forward, and error signals backward. Incrementing and decrementing capacitively stored weights is not difficult (Murray 1989) and will involve only two or three MOSFETs. The update equation 2.3 for target states is extremely simple, and thus easily implemented local to the neuron circuit. Looked at critically, this is actually no less complex than backpropagation. In fact, the requirement that the current version of the target state be retrieved after each presentation and stored along with the rest of the exemplar from the training set requires an input/output pad for each of the hidden neurons. However, the distinction between hidden and output neurons has been removed to the extent that the update scheme for weights to hidden and output neurons is identical, and furthermore both sets of neurons now include a local temporary target storage memory. Chips based on the target method will therefore be architecturally flexible, as the exact role of the neurons in the network‘s function (input, output, or hidden) need not be completely determined at the chip design stage. It is this feature, rather than a significant difference in raw complexity, that renders this scheme more amenable to VLSI. We have not yet designed the chip associated with this architecture. At this early stage, however, it is useful to propose an architecture that demonstrates the advantages of target-based methods. Figure 7 shows how a 5-neuron network would be arranged on silicon. It is intended that both neural activity values { x i } and post-activatiofinction outputs (0,) be accessible as multiplexed network “inputs,” to allow maximum flexibility in how input data are presented to the device. If the network were a backpropagation MLP, the input:hidden:output node assignment would have to be defined at the design stage in order to equip the hidden node elements with the circuitry essential to the chain-rule portion of backpropagation. In a target-based chip, however, every neuron has a locally stored target value, and associated target update circuitry. It is the use of these targets that defines which nodes are input, output, or hidden. For instance, to configure this chip as a 2:2:1 network, the input target circuits’ function would be ignored as irrelevant. The output target is loaded with each training input as normal, and updated target values ignored. The hidden node targets are loaded from off-chip memory along with the input and output vectors, and updated targets are loaded back into off-chip memory at the end of each training epoch. It would be a simple matter to reconfigure the chip as a 2:1:2 network, by a redefinition of the target usage, without any change in the chip’s architecture. Clearly this advantage is much greater in a realistically larger network. I believe that target-based algorithms, of which this is only the most recent variant, have much to offer here. They are not, to be sure, truly adaptive systems, requiring as they do extra storage and control, and


379

ALL NEURONS IDENTICAL

OFF-CHIP MEMORY FOR INPUTS, OUTPUT AND HIDDEN NODE TARGETS

Figure 7 Proposed targets chip architecture. being unable to function without supervision. However, the conceptual simplicity of the target idea lends an architectural simplicity to chips designed to support it. Furthermore, target algorithms allow uncomplicated update equations, and may even have some direct advantages as described above with respect to hill-climbing in the output error. The reset and learning speed controls on the targets themselves afford an additional means of influencing learning electronically and dynamically. We are now developing 1.5-pm CMOS devices to support target-based learning, and will report the results from these as soon as possible. 5 Conclusions

A scheme has been described that offers learning speeds and generalization ability slightly better than backpropagation, but is both conceptually simpler, and designed for flexible implementation as analog integrated circuitry. It differs from other target-based algorithms in being more pragmatic in its methods. Immunity to analog inaccuracy is high - in fact high levels of artificially introduced noise assist learning. The noise inherent in analog

380

Alan F. Murray

circuitry is therefore not a problem, and circuitry will be included to inject controlled noise into the learning arithmetic. We are currently testing the scheme's capabilities more exhaustively, in preparation for full implementation using analog pulse-firing VLSI, in conjunction with dynamic weight and target storage, and also using nonvolatile amorphous silicon memory devices (Reeder 1991).

Acknowledgments The author is grateful to Lionel Tarassenko (University of Oxford) for his encouragement (tempered with healthy scepticism), and constantly useful advice. Financial support from the Science and Engineering Research Council, and the CEC (ESPRIT BRA NERVES project) made this work possible.

References Alspector, J., Gannett, J. W., Haber, S., Parker, M. B., and Chu, R. 1991. VLSIefficient technique for generating multiple uncorrelated noise sources and its application to stochastic neural networks. I E E E Trans. Circuits Syst. 38(1), 109-123. Bishop, C. 1990. Curvature-driven smoothing in feedforward networks. PYOC. Int. Neural Networks Conf. 749-752. Grossman, T., Meir, R., and Domany, E. 1990. Learning by choice of internal representations. In Neural Information Processing Systems (NIPS) Conference 1989, pp. 73-80. Morgan Kaufmann, San Mateo, CA. Holler, M., Tam, S., Castro, H., and Benson, R. 1989. An electrically trainable artificial neural network (ETANN) with 10240 "floating gate" synapses. Int. joint Conf. Neural Networks -lJCNN89 191-196. Krogh, A., Thorbergsson, G. I., and Hertz, J. A. 1990. A cost function for internal representations. In Neural Information Processing Systems, pp. 733740. Morgan Kaufmann, San Mateo, CA. Murray, A. F. 1989. Pulse arithmetic in VLSI neural networks. I E E E MICRO 9(6), 64-74. Murray, A. F., Brownlow, M., Hamilton, A., I1 Song Han, Reekie, H. M., and Tarassenko, L. 1990. Pulse-firing neural chips for hundreds of neurons. In Neural Information Processing Systems (NIPS)Conference, pp. 785-792, Morgan Kaufmann, San Mateo, CA. Murray, A. F., Del Corso, D., and Tarassenko, L. 1991. Pulse-stream VLSI neural networks - mixing analog and digital techniques. I E E E Trans. Neural Networks 193-204. Nabutovsky, D., Grossman, T., and Domany, E. 1990. Learning by CHIR without storing internal representations. Complex Syst. 4, 519-541. Reeder, A. A., Thomas, I. P.,Smith, C., Wittgreffe, J., Godfrey, D., Hajto, J., Owen, A., Snell, A. J., Murray, A. F., Rose, M., and LeComber, P G. 1991. Appli-


381

cation of analogue amorphous silicon memory devices to resistive synapses for neural networks. Int. Conf. Neural Networks (Munich), pp. 253-259. Rohwer, R. 1990. The ”moving targets” training algorithm. In Neural Information Processing Systems, pp. 558-565, Morgan Kaufmann, San Mateo, CA. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1, James L. McClelland and David E. Rumelhart, eds., pp. 318-362. The MIT Press, Cambridge, MA. Watts, S. 1990. Computation in neural networks: A comparison of the multilayer perceptron and the hierarchical pattern recognition networks for classification problems. M. Sc. Thesis, University of Oxford.

Received 5 June 1991; accepted 15 October 1991.

This article has been cited by: 1. D.J. Mayes, A.F. Murray, H.M. Reekie. 1999. Non-Gaussian kernel circuits in analogue VLSI: implications for RBF network performance. IEE Proceedings - Circuits, Devices and Systems 146:4, 169. [CrossRef] 2. Takio Kurita, Hideki Asoh, Shinji Umeyama, Shotaro Akaho, Akitaka Hosomi. 1996. Influence of noises added to hidden units on learning of multilayer perceptrons and structurization of networks. Systems and Computers in Japan 27:11, 64-73. [CrossRef] 3. Leonard Neiberg, David Casasent. 1994. High-capacity neural networks on nonideal hardware. Applied Optics 33:32, 7665. [CrossRef] 4. Thorsteinn Rögnvaldsson . 1994. On Langevin Updating in Multilayer PerceptronsOn Langevin Updating in Multilayer Perceptrons. Neural Computation 6:5, 916-926. [Abstract] [PDF] [PDF Plus]

Communicated by Todd K. Leen

Computing the Karhunen-Loeve Expansion with a Parallel, Unsupervised Filter System Reiner Lenz Mats bsterberg Department EE, Linkoping University, S-58183 Linkoping, Sweden

We use the invariance principle and the principles of maximum information extraction and maximum signal concentration to design a parallel, linear filter system that learns the Karhunen-Loeve expansion of a process from examples. In this paper we prove that the learning rule based on these principles forces the system into stable states that are pure eigenfunctions of the input process.

1 Introduction Unsupervised learning neural networks try to describe the structure of the space of input signals. One type of unsupervised learning neural networks is closely related to the traditional statistical method of principal component analysis. One of the earliest attempts to compute one eigenfunction of a class of input signals is the principal component analyzer developed by Oja (see Oja 1989 for a recent overview). Oja’s analyzer is based on the idea that the system is essentially a similarity detector or a correlation unit. This leads naturally to a Hebbian learning rule that updates the correlation coefficients so that the output of the analyzer is maximized. Oja showed that this analyzer could learn the first eigenvector of the input process. Recently, Sanger (1989) generalized Oja’s result by showing how different one-dimensional analyzers could be used to compute a number of different eigenfunctions. The system proposed by Sanger consists of a number of analyzers of the type introduced by Oja that are connected in a serial fashion: The first analyzer works on the original input signal, then the contribution from the first analyzer is subtracted from the original signal and the modified signal is fed into the next analyzer. Analyzer number i is thus trained with that part of the signal which is unexplained by the previous analyzers 1,.. . ,i - 1 of the system. Neural Computation 4, 382-392 (1992)


Computing the Karhunen-Loeve Expansion

383

In this paper we propose a system that produces output signals that preserve the structure of the input data space, that extract a maximum amount of information and that are maximally concentrated. These properties will be discussed in detail later on. The main aim of the paper is to show that the system that is constructed this way learns the principal components of the input process in parallel. The basic parts of the system are essentially the same correlation units as those used in Oja’s and Sanger’s systems. In the following we will call one such correlation unit (together with its update unit) a basic unit. The fundamental difference between the system proposed in this paper and the earlier systems lies in the learning rule that is used to update the basic units. In this paper we will show that our new learning rule makes it possible to train the basic units in parallel. We will also show that the communication between the basic units is minimal. By this we mean that the only information that is sent from unit i to unit j is the response of unit i to the actual input pattern. A given unit has therefore no information about the internal coefficients of the other units in the system. This leads to a clear modular structure of the overall system. The learning rule proposed in this paper is based on a combination of three principles: the structure preserving principle, the maximum information extraction principle, and the maximum concentration principle. The structure preserving or invariance principle is one of the oldest design principles in pattern recognition (see, for example, Lenz 1990). It is based on the idea that the measurement or feature extraction process should preserve the structures in pattern space. The standard example is rotation invariant edge detection in image processin:. Here pattern space is structured in the sense that patterns differing only by their orientations are considered to be equal. An edge detector should thus produce features that are independent of the orientation of the edge in the image. The maximum information principle is by now well known in neural network research. This principle states that the space of computed output signals should contain as much information about the space of input signals as possible. This maximum information principle was recognized as a basic principle for neural network design by Linsker (see Linsker 1988) and it was shown to be of fundamental importance in the grouptheoretical filter design strategy (see Lenz 1990). In Lenz (1990) it was also shown that the invariance principle and the maximum information principle are closely related and that they lead to identical solutions. The amount of extracted information is, however, much easier to handle algorithmically and we will therefore base our design on the requirement of maximum information extraction. In our analytical studies we showed that the solutions obtained with this approach can be described in terms of eigenfunctions of the input covariance function. These two principles are, however, not strong enough to force the system into a unique stable state: using only the maximum information

384

Reiner Lenz and Mats Osterberg

principle usually leads to basic units that correspond to arbitrary (orthogonal) mixtures of pure eigenfunctions. Therefore we propose to combine the maximum information principle with the maximum concentration principle. The idea behind this principle is the following: Assume we have two systems C and S that extract an equal amount of information from the input process. Assume further that system C tries to concentrate this information into as few output channels as possible, whereas system S tries to spread this information equally over all channels. A single output channel of system C will (in the mean) have either very strong or very weak output signals whereas a typical output channel in system S will produce medium strength signals. It should be clear that the signals produced by system C are much easier to evaluate: In the simplest case one only has to observe which units produced high outputs and which units were silent. The output signals of such a system are thus well suited for further processing by threshold units. This may be advantageous if the different units in the system communicate via noisy communication channels because in the simplest cases it may be sufficient to transmit on/off signals. We will therefore require that our system should extract a maximum amount of information from the input signals and that it should concentrate this information as much as possible. The dynamics of a similar system that also generalizes Oja’s approach was investigated by Leen. This system tries to maximize the sum of the variances of the output signals. It also penalizes the interaction between different units. The behavior of the system depends critically on the value of a free coupling constant that describes the relation between the variance part and the interaction part of the update rule. The dynamics of this system (and a related system) is investigated in detail in Leen (1990). The outline of this paper is as follows: first we will introduce the quality function that we use to measure the quality of the current set of filter coefficients. Then we describe the update rule that optimizes these coefficents by performing a gradient search. We show that this update rule leads to a parallel filter system with a minimal amount of internal communications. The main purpose of this paper is to show that the stable states of the system correspond to the pure eigenfunctions of the input process. In previous papers (see Lenz 1991; Lenz and Osterberg 1991a,b,c) we investigated this, and several similar systems based on the same ideas. We applied them to various pattern recognition problems like edge and line detection, texture analysis, and OCR reading. All these experiments showed that the system is indeed able to recognize the structures in pattern space. In the edge- and line-detection application we demonstrated, for example, that the learned filter functions are indeed the eigenfunctions of the input covariance functions. These functions recognized as optimal edge and line filters constructed in our work on group theoretical filter design.


385

2 The Structure of the System

Based on the heuristical considerations described in the introduction we use a quality function of the form:

where W is the matrix of the weight coefficients of the system. The heuristical considerations are formalized in the following construction: We assume that the pattern vectors have K components and that the system consists of N basic units. The coefficients of the system are collected in the K x N matrix W. We require that all columns of this matrix have unit length, i.e., the units are represented by unit vectors. If p denotes the input vector and o the output vector then we have o = p x W. The components of u will be denoted by o k : ok is thus the output of the unit number k. The output covariance matrix S is defined as the matrix of all the expected values of the products o i . oj. This value is denoted by (oi . oj) where a pair of brackets denotes the expectation operation. Using these notations we introduce the quality function Q defined as

where Qv(W) = det S = det ((0, . u,)) measures the amount of extracted information by the variation of the output signals. The function Qc(W) measures the concentration within the output vectors; it is defined as Qc(w) = CE,(oT)(1- ( o f ) ) . If Wk(t) is the coefficient vector of unit k at time t and if w,k(t) is the Zth entry in this vector then at each iteration the following operations are performed by the basic units of the system: 0

The correlation units compute the output signals:

0

The update units compute the new state vectors of the units:

0

Later on we will see that it is convenient to have normed coefficient vectors. The update units will therefore also normalize the state vectors after each iteration.


386

In our implementation we use a gradient based learning rule, i.e., we select A WJk(t)proportional to the partial derivative 8 Wfk:

Q/a

Qc( W) (a/awIk)Qv (w) - Qv ( w) (a/aw~k) Qc( w) Q: ( W) We will first show that this derivative is a function of (up,) and (OkpJ). -

Computing (a/aw,)Qc(W) gives

-

a

-(o:)(l

- (0;))

h l k

= 2(okpl) - 4(o:)(okpI))

(2.3)

Since the derivative of (0:) is 2(0kpf). To compute the derivative (a/awfk)Qv(W) we note first that only the kth column and the kth line of the matrix S = ((ulol))depends on the weight Wlk. We have Qv(

w) =

(0:)

det S k

+

(OfO~)",~k)(-l)fr+'fA(''''k)

det (sk l ) f k

l#k,i#k

=

(0;)

detskk +

( % ~ k ) ( ~ , ~ k ) p ~

(2.4)

I#k,iZk

where A(i,j , k) is defined as ( 1

ifi>kandi>k (2.5)

[

0

ifik

s k k denotes the (N- 1) x (N- 1)submatrix of ((uioj)) obtained by deleting the kth row and the kth column. is the (N - 2 ) x (N - 2 ) submatrix of ((oioj)) obtained by deleting row k, row i, column j , and column k. All these submatrices (and thus pi) are independent of Wlk. The derivative is then given by

Since the derivative of (oiok)(ojok) is (ojok)(oipl) + (oiok)(ojpr). From these expressions we see that a unit can compute the increments without the knowledge of the internal weights of the other units, i.e., the increments for the unit k are functions of the input patterns, the output values, and the previous values of the weight coefficients within this unit.


387

Output Odt)

Output ON(t)

Figure 1: The learning filter system

We see therefore that the learning rule can indeed be implemented in a parallel system of the form shown in Figure 1.

3 Computation of the Optimal Weight Matrix

We will now investigate the quality function Q( W) and compute its maximum points. In the following we assume that the system has already stabilized so that the weight coefficients do not change anymore. This constant weight matrix will be denoted by W. The output vector at time t is then computed as o ( t ) = p(t) x W and the covariance matrix of the output values is then given by

s = ((0iOj)) = (0’0)= (W’p’pw)= W’(p’p)w = W’TW

(3.1)

where T is the covariance matrix of the input patterns. We now assume that we know the first- and second-order statistical properties of the pattern process: especially that the mean vector is zero and the covariance matrix T is given. Our goal is to find the matrices W that have centered and normed columns and that are maximum points of the quality function: Q( W) max. We assume also that all eigenvalues of the covariance matrix T are different.


388

In our analysis we will use the singular value decomposition of the weight matrix W. This decomposition (see Golub and Van Loan 1986) can be described as follows: Theorem 1. Assume W is a K x N matrix with K > N . Then we can find orthogonal matrices U and V of size K and N , respectively,and a matrix D of size K x N such that W = UDV

(3.2)

For the diagonal elements d k of D we have dll 2 d z 2 2 “d 2 0 and the other elements in D are all zero. The decomposition W = UDV is called the singular value decomposition or the SVD of W. In the following we will use the term diagonal matrix for all matrices that have zero entries outside the diagonal. The matrix D in the previous theorem is therefore a diagonal matrix although it is rectangular. The N x N unit matrix will be denoted by EN. But we will also use EN for rectangular matrices if they are essentially equal to the unit matrix, that is if dii = 1 for all i = 1. . .N and dij = 0 for all i # j . This should not lead to any confusion since the size of the matrices are always clear from the context. In the next series of theorems we investigate the variation part of the quality function as a function of U, D, and V. 1. If W = UDV then the variation is independent of the V: Qv(w)= Qv(UDV) = Qv(U0)

Theorem 2.

Qv(W) = det(D’U‘TUD)= Qv(UD)

(3.3)

2. If we select W = UD and if the columns of Ware unit vectors then

w = U D = UEN

(3.4)

From equation 2.3 and Theorem 1 we get Qv( W) = det S

= det(V’D’U’TUDV) = det(D’U’TUD)

In the following we investigate thus Qv as a function of U and D alone. We select V = EN or W = UD. Since U was orthogonal we get W’W = D’U’UD = D’D. If the columns of W are unit vectors then we get W’W = D’D = EN and since the elements of D are nonnegative we see that D = E N . For our further investigations we need some facts about Gram determinants: Definition 1. Assume that al, . . . ,aL is a set of vectors of equal length and that y,! = aiaj are the scalar products between pairs of two such vectors. Then we define


389

(see Grobner 1966, p . 108) the Gram determinant G(al,...,aL)of these vectors as the determinant of the matrix Y = ( yij ): G(al,... , a L ) = det (aia,) = det (yij)

= detY

(3.5)

We can now show that the variation function is a Gram determinant:

Theorem 3. There is a matrix X such that Qv(W) = G(xl,.. . , X N )

(3.6)

= det (x:xj)

X has the form X = TUENwhere ? is a diagonal matrix with nonnegative entries.

From Qv( W) = det(EhU’TUEN) we see that we may assume that T is a diagonal matrix with nonnegative diagonal elements (if it is not replace U by U = U1U2 where both Ui are orthogonal and where U1 diagonalizes T; this is possible since T is a covariance matrix and therefore positive semidefinite). We can thus find a matrix ? such that T = .f.2. Qv( W)

= det(EkU’TUEN)= det(EkU’T2UEN)= det(X’X)

Theorem 4. Assume that r1 2 . . . 2 matrix T. Then we have

Qv(W I ~ I

TN

(3.7)

are the N largest eigenvalues of the (3.8)

. - J N

Equality holds if U = EK. In the theory of Gram determinants (see Grobner 1966, p; 108) it is shown that 2

G(x1,.. .,X N ) L G(xi)G(x2,.. . ,X N ) = 11x1 11 G(x2,...,X N - I )

(3.9)

where equality holds if the vector x1 is orthogonal to all the vectors x2 . . .X N . From the previous theorem we know that Qv( W) = G(x1, .. . , X N ) where the xi are the columns of the matrix TU. Now U is orthogonal and therefore llxi112 5 71. Repeated application gives the theorem. The results derived so far are collected in the following theorem:

Theorem 5. The maximum value of the variationfunction is equal to the product of the N largest eigenvalues of the input process. This value is obtained by the filter system consisting of the eigenvectors belonging to these eigenvalues (and by any orthogonal transformation of them). In thefollowing we denote the matrix consisting of the sorted eigenvectorsof T by UO.With this notation we have: Qv(W)

I Qv(UOENV)= T I . . . ~ N

where W runs over all matrices with unit column vectors.

(3.10)


390

We now investigate filter systems with a SVD of the form UOENV and we will see how Qc can be used to select an optimal matrix V. We note also that the matrix UbTUo = T is the diagonal matrix with the eigenvalues since Uo consists of the eigenvectors of T.

Theorem 6. IfQ4(W) = Cr=l(oE)2 and W = UOENVthen Qc(W)

=

trace(ELU6TUOEN) - Q4( W)

=

trace(EL?EN) - Q4(W) =

N T,

-

Q4(W)

(3.11)

n=l

Proof. (3.12) n=l

n=l

and since the trace is invariant under orthogonal transformations: N

C(oi) =

trace ( ( O k o ) ) )

= traces = trace(W‘TW)

n=l

=

trace(V’E&U;TUoENV) = traCe(EhTEN)

(3.13)

Theorem 7. If W = UOENVand V = ( vij ) then (3.14)

If v, is the: mth column of V then we find that (0;) is the mth diagonal element of V’ELU~TUOENV. This is equal to VLTV, and we get (0;) =

Ck Tkvzm* Theorem 8. If TI 2 r2 2 . . . 2

T N:

(3.15) Since v is orthogonal we have: (0:)’ = Ck,/TkT/z&vf1 5 7: Ck,l v:lv~l = T:. Selecting ull = 1 and vil = 0 for i > 1 shows that this maximum value can be obtained if we select the first column of V as the first unit vector. V is orthogonal and we find therefore also that ‘u11 = 1 implies D l i = 0 for i > 1. From this we conclude that V has the form:

with an (n

c:=,(o;)~

- 1) x (n - 1) matrix V,. By induction we find that the sum = Ck,l,mT k q v z m v f m is maximal if we select = EN.

v

Finally some comments on the case where several eigenvalues are equal. For simplicity we assume that all eigenvalues are equal: T~ =


. . . = TN of

v

391

= r. In this case we can see that also Q ~ ( U V O ), is independent N

Q~(UV O ), =

C (Ok)2 = C T k ~ $ m v ~ m= m=l

k , b

N

T2

C

Z ' ~ m V ~= m Nr2

(3.16)

k,l,m=l

The quality function is in this case independent of V and all systems of the form W = UoENV with arbitrary, orthogonal matrices V are maximum points.

4 Summary and Conclusions

In the last section we proved the following two properties of our filter system: 3 . The variation value is maximal for all filter systems of the form W =

UOENV, where UOis the eigenvector matrix of the input covariance matrix and V is an arbitrary orthogonal matrix. 2. For all filter systems with maximum variation value the concentration value is minimal for the filter system consisting of the pure eigenvectors . The derivation shows also that the key component in our new quality function is the fourth-order term; this term forces the system into stable states consisting of pure eigenvectors. This was shown under the assumption that the system has already stabilized but our simulation experiments show that this is also the case even if the covariance matrix of the input process has to be estimated during the learning process.

References Golub, G. H., and Van Loan, C. F. 1986. Matrix Computations. North-Oxford Academic. Grobner, W. 1966. Matrizenrechnung. B-I Hochschultaschenbiicher. Bibliographisches Institut, Mannheim. Leen, T. K. 1990. Dynamics of learning in linear feature-discovery networks. Network 2(1), 85-105. Lenz, R. 1990. Group Theoretical Methods in Image Processing. Lecture Notes in Computer Science (Vol. 413). Springer-Verlag, Berlin. Lenz, R. 1991. On probabilistic invariance. NeuTal Networks 4(5), 627-641. Lenz, R., and Osterberg, M. 1990. Learning filter systems. Proc. Int. Neural Networks Conf. Paris. Lenz, R., and Osterberg, M. 1991a. Filtering: Invariance, information extraction and learning. In Progress in Neural Networks (in press). Lenz, R., and Osterberg, M. 1991b. Learning filter systems with maximum correlation and maximum separation properties. In SPlE Proc. Applications of Artificial Neural Networks II, Orlando.

392


Lenz, R., and Osterberg, M. 1991~.A parallel learning filter system that learns the kl-expansion from examples. In Proc. First IEEE-SP Workshop on Neural Networks for Signal Processing, Princeton, pp. 121-130. Linsker, R. 1988. Self-organizationin a perceptual network. IEEE Cornput. 21(3),

105-117. Oja, E. 1989. Neural networks, principal components, and subspaces. Int. 1. Neural Syst. 1,61-68. Sanger, T. D. 1989. Optimal unsupervised learning in a single-layer linear feedforward neural network. Neural Networks 2(6), 459-474.

Received 6 May 1991; accepted 7 October 1991

This article has been cited by: 1. Reiner Lenz, Mats Österberg, Jouni Hiltunen, Timo Jaaskelainen, Jussi Parkkinen. 1996. Unsupervised filtering of color spectra. Journal of the Optical Society of America A 13:7, 1315. [CrossRef]

Communicated by James McClelland

Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks C. L. Giles C. B. Miller NEC Research Institute, 4 Independence Way, Princeton, NJ 08540 USA

D. Chen H. H. Chen G. Z. Sun Y. C. Lee University of Ma yland, Institute for Advanced Computer Studies, Department of Physics and Astronomy, College Park, M D 20742 USA

We show that a recurrent, second-order neural network using a realtime, forward training algorithm readily learns to infer small regular grammars from positive and negative string training samples. We present simulations that show the effect of initial conditions, training set size and order, and neural network architecture. All simulations were performed with random initial weight strengths and usually converge after approximately a hundred epochs of training. We discuss a quantization algorithm for dynamically extracting finite state automata during and after training. For a well-trained neural net, the extracted automata constitute an equivalence class of state machines that are reducible to the minimal machine of the inferred grammar. We then show through simulations that many of the neural net state machines are dynamically stable, that is, they correctly classify many long unseen strings. In addition, some of these extracted automata actually outperform the trained neural network for classification of unseen strings. 1 Introduction

Grammatical inference, the problem of inferring grammar(s) from sample strings of a language, is a hard problem, even for regular grammars; for a discussion of the levels of difficulty see Gold (1978) and Angluin and Smith (1983). Consequently, there have been many heuristic algorithms developed for grammatical inference, which either scale poorly with the number of states of the inferred automata or require additional information such as restrictions on the type of grammar or the use of queries (Angluin and Smith 1983). For a summary of inference methods, see Neural Computation 4, 393-405 (1992)


394

C. L. Giles et al.

Fu (1982) and Angluin and Smith (1983) and the recent, comprehensive summary by Miclet (1990). The history of finite state automata and neural networks is a long one. For example, Minsky (1967) proved that ”Every finite-state machine is equivalent to, and can be simulated by some neural network.” More recently the training of first-order recurrent neural networks that recognize finite state languages was discussed by Williams and Zipser (1989), Cleeremans et al. (1989), and Elman (1990). The recurrent networks were trained by predicting the next symbol and using a truncation of the backward recurrence. Cleeremans et al. (1989) concluded that the hidden unit activations represented past histories and that clusters of these activations can represent the states of the generating automaton. Mozer and Bachrach (1990) apply a neural network approach with a second-order gating term to a query learning method (Rivest and Schapire 1987). These methods (Rivest and Shapire 1987; Mozer and Bachrach 1990) require active exploration of the unknown environments, and produce very good finite state automata (FSA) models of those environments. We discuss a recurrent neural network solution to grammatical inference and show that second-order recurrent neural networks learn fairly well small regular grammars with an infinite number of strings. This greatly expands on our previous work (Giles et al. 1990; Liu et al. 1990), which considered only regular grammars of unusual state symmetries. Our approach is similar to that of Pollack (1990)and differs in the learning algorithm (the gradient computation is not truncated) and the emphasis on what is to be learned. In contrast to Pollack (1990), we emphasize that a recurrent network can be trained to exhibit fixed-point behavior and correctly classify long, previously unseen, strings. Watrous and Kuhn (1992) illustrate similar results using another complete gradient calculation method. We also show that from different trained neural networks, a large equivalence class of FSA can be extracted. This is an important extension of the work of Cleeremans et al. (1989) where only the states of the FSA were extracted. This work illustrates a method that permits not only the extraction of the states of the FSA, but the full FSA itself. 2 Grammars

2.1 Formal Grammars and Grammatical Inference. We give a brief introduction to formal grammars and grammatical inference; for a thorough introduction, we recommend, respectively, Harrison (1978) and Fu (1982). Briefly, a grammar G is a four tuple {N,T,P,S}, where N and T are sets of nonterminals and terminals (alphabet of the grammar), P a set of production rules, and S the start symbol. For every grammar, there exists a language L, a set of strings of the terminal symbols, that the grammar generates or recognizes. There also exist automata that recognize and generate that grammar. In the Chomsky hierarchy of phrase structured

Second-Order Recurrent Neural Networks

395

grammars, the simplest grammar and its associated automata are regular grammars and FSA. This is the class of grammars we will discuss here. It is important to realize that all grammars whose string length and alphabet size are bounded are regular grammars and can be recognized and generated, maybe inefficiently, by finite state automata. Grammatical inference is concerned mainly with the procedures that can be used to infer the syntactic rules (or production rules) of an unknown grammar G based on a finite set of strings Z from L(G), the language generated by G, and possibly also on a finite set of strings from the complement of L(G) (Fu 1982). Positive examples of the input strings are denoted as 2, and negative examples as Z-. We replace the inference algorithm with a recurrent second-order neural network, and the training set consists of both positive and negative strings. 2.2 Grammars of Interest. To explore the inference capabilities of the recurrent neural net, we have chosen to study a set of seven relatively simple grammars originally created and studied by Tomita (1982) and recently by Pollack (19901, Giles etal. (19911, and Watrous and Kuhn (1991). We hypothesize that formal grammars are excellent learning benchmarks; that no feature extraction is required since the grammar itself constitutes the most primitive representation. For very complex grammars, such as the regular grammar that represents Rubik‘s cube, the feature extraction hypothesis might break down and some feature extraction method, such as diversity (Rivest and Shapire 1983, would be necessary. The grammars shown here are simple regular grammars and should be learnable. They all generate infinite languages over {0,1}* and are represented by finite state automata of between three and six states. Briefly, the languages these grammars generate can be described as follows:

#1 - l’,

#2 - ( 1 0 )*, #3 - an odd number of consecutive 1s is always followed by an even number of consecutive Os, #4 - any string not containing ”000” as a substring,

#5 - even number of 0s and even number of Is, (see Giles et LIZ. 1990, p. 383), (our interpretation of Tomita #5), #6 - number of Is - number of 0s is a multiple of 3, #7-o*

1*o* 1’.

The FSA for Tomita grammar #4 is given in Figure lc. Note that this FSA contains a so-called “garbage state,” that is, a nonfinal state in which all transition paths lead back to the same state. This means that the recurrent

396

C. L. Giles et al.

Figure 1: Finite state automata (=A) for Tomita’s 4th grammar. Initial state nodes are cross-hatch. Final state nodes are drawn with an extra surrounding circle. Transitions induced by a “0” input are shown with solid lines, and transitions induced by a “1” with dashed lines. (a,b) FSA derived from the statespace partitioning of two neural networks that learned the grammar starting from different initial weight conditions. ( c ) The ideal, minimal FSA for Tomita’s 4th grammar. Machines (a) and (b) both reduce via a minimization algorithm to machine (c). neural net must not only learn the grammar but also its complement and thus correctly classify negative examples. Not all FSA will have garbage states. Such an FSA recognizes a language only when the entire string is seen. In this case there are no situations where ”illegal characters” occur - there are no identifiable substrings that could independently cause a string to be rejected.

3 Recurrent Neural Network 3.1 Architecture. Recurrent neural networks have been shown to have powerful capabilities for modeling many computational structures;


397

an excellent discussion of recurrent neural network models and references can be found in Hertz et al. (1991). To learn grammars, we use a secondorder recurrent neural network (Lee et al. 1986; Giles et al. 1990; Sun et al. 1990; Pollack 1990). This net has N recurrent hidden neurons labeled Sj; L special, nonrecurrent input neurons labeled Ik; and N2 x L real-valued weights labeled Wqk. As long as the number of input neurons is small compared to hidden neurons, the complexity of the network only grows as O ( p ) ,the same as a linear network. We refer to the values of the hidden neurons collectively as a state vector S in the finite N-dimensional space [0,1IN.Note that the weights Wijk modify a product of the hidden Sj and input & neurons. This quadratic form directly represents the state transition diagrams of a state process - {input,state} + {nextstate}. This recurrent network accepts a time-ordered sequence of inputs and evolves with dynamics defined by the following equations:

where g is a sigmoid discriminant function. Each input string is encoded into the input neurons one character per discrete time step t. The above equation is then evaluated for each hidden neuron Si to compute the next state vector S of the hidden neurons at the next time step t + 1. With unary encoding the neural network is constructed with one input neuron for each character in the alphabet of the relevant language. This condition might be restrictive for grammars with large alphabets. 3.2 Training Procedure. For any training procedure, one must consider the error criteria, the method by which errors change the learning process, and the presentation of the training samples. The error function €0 is defined by selecting a special “response” neuron SO, which is either on (SO > 1 - E ) if an input string is accepted, or off (SO < c) if rejected, where E is the response tolerance of the response neuron. We define two error cases: (1)the network fails to reject a negative string 2- (i.e., So > c); (2) the network fails to accept a positive string 2, (i.e., SO< 1 - c). For these studies, the acceptance or rejection of an input string is determined only at the end of the presentation of each string. The error function is defined as

where TO is the desired or target response value for the response neuron SO. The target response is defined as TO = 0.8 for positive examples and TO = 0.2 for negative. The notation Sf) indicates the final value of So, that is, after the final input symbol.

398

C. L. Giles et al.

The training is an on-line (real-time) algorithm that updates the weights at the end of each sample string presentation (assuming there is an error Eo > .5& with a gradient-descent weight update rule: (3.3)

where cr is the learning rate. We also add a momentum term, an additive update to A WI,,, which is q, the momentum, times the previous A Wlmn.To determine A Wlmn,the aS“/ldWlmn must be evaluated. From the recursive network state equation, we see that (3.4)

where 9”is the derivative of the discriminant function. In general, f and ” f - 1 can be replaced by any t and t - 1, respectively. These partial derivative terms are calculated iteratively as the equation suggests, with one iteration per input symbol. This on-line learning rule is a secondorder form of the recurrent net of Williams and Zipser (1989). The initial terms aSY’/dWl,, are set to zero. After the choice of the initial weight values, the dS~”/dW,,, can be evaluated in red time as each input 1;) enters the network. In this way, the error term is forward-propagated and accumulated at each time step t. However, each update of dS~’)/aWl,,, requires OW4 x L2) terms. For N >> L, this update is 0(N4),which is the same as a linear network. This could seriously inhibit the size of the recurrent net if it remains fully interconnected. 3.3 Presentation of Training Samples. The training data consist of a series of stimulus-response pairs, where the stimulus is a string over {O,l}*, and the response is either ”1” for positive examples or ”0” for negative examples. The positive and negative strings, Z+and I-,are generated by a source grammar prior to training. Recall that at each discrete time step, one symbol from the string is presented to the neural network. There was no total error accumulation as occurs in batch learning; training occurred after each string presentation. The sequence of strings during training may be very important. To avoid too much bias (such as short versus long, positive versus negative), we randomly chose the initial training set of 1024 strings, including Tomita’s original set, from the set of all strings of length less than 16 (65,535 strings). As the network starts training, the network gets to see only some small randomly-selected fraction of the training data, about 30 strings. The remaining portion of the data is called “pretest” training data, which the network gets to see only after it either classifies all 30 examples correctly (i.e., for all strings ]El < 4,or reaches a maximum number of epochs (one epoch = the period during which the network


399

processes each string once). This total maximum number of epochs is 5000 and is set before training. When either of these conditions is met, the network checks the pretest data. The network may add up to 10 misclassified strings in the pretest data. This prevents the training procedure from driving the network too far toward any local minima that the misclassified strings may represent. Another cycle of epoch training begins with the augmented training set. If the net correctly classifies all the training data, the net is said to converge. This is a rather strict sense of convergence. The total number of cycles that the network is permitted to run is also limited, usually to about 20. An extra end symbol is added to the string alphabet to give the network more power in deciding the best final state S configuration. For encoding purposes this symbol is simply considered as another character and requires another input neuron. Not that this does not increase the complexity of the FSA! In the training data, the end symbol appears only at the end of each string.

3.4 Extracting State Machines. As the network is training (or after training), we apply a procedure for extracting what the network has learned - that is, the network's current conception of the FSA it is learning (or has learned). The FSA extraction process includes the following steps: (1) clustering of FSA states, (2) constructing a transition diagram by connecting these states together with the alphabet labeled arcs, (3) putting these transitions together to make the full digraph-forming loops, and (4) reducing the digraph to a minimal representation. The hypothesis is that during training, the network begins to partition (or quantize) its state space into fairly well-separated, distinct regions or clusters, which represent corresponding states in some finite state automaton (see Fig. 1). See Cleeremans et al. (1989) for another clustering method. One simple way of finding these clusters is to divide each neuron's range [0,1] into q partitions of equal width. Thus for N hidden neurons, there exist 9N possible partition states. The FSA is constructed by generating a state transition diagram, that is, associating an input symbol with the partition state it just left and the partition state it activates. The initial partition state, or start state of the FSA, is determined from the initial value of S('=O). If the next input symbol maps to the same partition state value, we assume that a loop is formed. Otherwise, a new state in the FSA is formed. The FSA thus constructed may contain a maximum of 9N states; in practice it is usually much less, since not all partition states are reached by S(f). Eventually this process must terminate since there are only a finite number of partitions available; and, in practice, many of the partitions are never reached. The derived FSA can then be reduced to its minimal FSA using standard minimization algorithms (Hopfcroft and Ullman 1979). (This minimization process does not change the performance of the FSA; the unminimized FSA has the same time complexity as the minimized FSA. The process just rids the FSA of redundant, unnecessary states and

400

C. L. Giles et al.

reduces the space complexity.) The initial value of the partition parameter is 9 = 2 and is increased only if the extracted FSA fails to correctly classify the 1024 training set. It should be noted that this FSA extraction method may be applied to any discrete-time recurrent net, regardless of order or hidden layers. Of course this simple partitioning or clustering method could prove difficult for large numbers of neurons. 4 Results - Simulations

At the beginning of each run, the network is initialized with a set of random weights, each weight chosen between [-l.O,l.O]. Unless otherwise noted, each training session has its own unique initial weight conditions. The initial value of the neurons Sir=') = 6io; though simulations with these values chosen randomly on the interval [0.0,1.Ol showed little significant difference in convergence times. For on-line training the initial hidden neuron values are never reset. The hidden neuron values update as new inputs are seen and when weights change due to string misclassification. The simulations shown in Table 1 focus on Tomita’s grammar #4. However, our studies of Tomita’s other grammars suggest that the results presented are quite general and apply to any grammar of comparable complexity. Column 1 is a run identification number. The only variables were the number of hidden neurons (3,4,5), in column 2, and the unique random initial weights. In column 3 the initial training set size is 32. Column 4 contains the initial training set size plus all later errors made on the training set. The number of epochs for training is shown in Column 5. If the network does not converge in 5000 epochs, we say the network has failed to converge on a grammar. The number of errors in the test set (both positive and negative) consisting of the original universe of all strings up to length 15 (65,535-1024 strings) is shown in columns 6, 6a, and 6b, and is a measure of the generalization capacity of the trained net. For columns 6 and 6a the error tolerance t is respectively < 0.2 and < 0.5. As expected if the error tolerance is relaxed, the trained network correctly classifies significantly more of the 65K test set. In column 6b are the number of errors for a randomly chosen set of 850,000 strings of length 16 to 99 with an error tolerance < 0.5. The information related to the extraction of FSA from the trained neural network is in columns 7-9. The number of neuron partitions (or quantizations) necessary to obtain the FSA that correctly recognizes the original training set is shown in column 7. The partition parameter q is not unique and all or many values of 9 will actually produce the same minimal FSA if the grammar is well learned. The number of states of the unminimized extracted FSA is shown in column 8. The extracted FSA is minimized and in column 9 the number of states of the minimal extracted FSA is shown. The minimal FSA for the grammar Tomita #4 is 4 states (see Fig. lc) if the empty string is accepted and 5 states if the empty string is rejected.

-

S P P P

P P P P P -

-

01 9 9 S 01 P 9 9

P

E E E E Z Z

E

S Z -

-

Z Z Z

P 8 L ZI 9

E

El

P P P

8

S

El

s

01

S

-

-

P P P

E E Z

E Z

0 0

-

0 98

09E 0 0 0 0

ZLo09 0 0 EZ1

-

-

PE 61I S8 Z 0

0 0 0 0 0

0 0

0 99

0 OE

I

PZ

0 0 0

0

z

I 996 Z 0 0

01

0 0 0

006 0 0 19 8 28 0 082 0

A

1z $.

88PI P LLlZ 0 1SE LZPZ

- L S LI S IZ P 8 P Zl P P 9 - -

F

2P 'm Lit

20

2

::5.

E

e)cI

*3 *3

Y

9.

ZE

zs

ZE

ZP

ZE ZE

ES

SP

ZE

s9

ZE

-

-

PW3 PIP3

ZE

ZP ZS 9s ZP

PW3 PSO I 19 L6 P6 PZ I z1I 98 sz1 96

ZOI 1 OL €01 EL

PL

zs

ZE ZE ZE

ZE ZE ZE ZE ZE ZE ZE

99

ES

ZS

ZP PS

zz I

zs

-

E E E E E E E E E E

-

P P P P P P P P P P P

__ -

s

LE ZE ZE ZE

PS LP PS ZP

S

ZE

zs

P

Q J!

a

D

ZE ZE

ZE ZE

ES

PZI PZ 1 LO1 LL SZ I EE I 0s 6P 88 Z8

S S S

S LE ZP --

! z S

8 G

I

kt

a q

8

m

I

!.mi

Y

I+'

*

P

L

3. x L 6

6S 8P €01 zo1 09 8s __.

r

E F ii

3

f

$ 2.

b!

D

3

g

1

a

iB

t

rI v

402

C. L. Giles et al.

The empty string was not used in the training set; consequently, the neural net did not always learn to accept the empty string. However, it is straightforward to include the empty string in the training set. In Figure l a and b are the extracted FSA for two different successful training trial runs (#lo4 and #104b in Table 1) for a four-neuron neural network. The only difference between the two trials is the initial weight values. The minimized FSA (Hopfcroft and Ullman 1979) for Figure l a and b is shown in Figure lc. All states in the minimized FSA are final states with the exception of state 0, which is a garbage state. For both cases and in all trials in Table 1 that converged, the minimized extracted FSA is the same as the minimal FSA of Tomita #4. What is interesting is that some extracted FSA, for example, trial Runs #60 and #104e, will correctly classify all unseen strings whereas the trained neural networks, from which the FSA were extracted, will not.

5 Conclusions

Second-order recurrent neural networks are capable of learning small regular grammars rather easily and generalizing very well on unseen grammatical strings. The training results of these neural networks for small simple grammars are fairly independent of the initial values of the weight space and usually converge using an incremental on-line, forward-propagation, training algorithm. For a well-trained neural net, the generalization performance on long (string lengths < 100) unseen strings can be perfect. A heuristic method was used to extract FSA from the neural network, both during and after training. (It would be interesting if a neural network could also learn to extract the proper FSA.) Using a standard FSA minimization algorithm, the extracted FSA can be reduced to an equivalent minimal-state FSA. Note that the minimization procedure reduces only the space complexity of the FSA; the time complexity of the minimized and unminimized FSA remains the same. From the extracted FSA, minimal or not, the production rules of the learned grammar are evident. There are some interesting aspects to the extracted FSA. Surprisingly, each of the unminimized FSA shown in the table is unique, even those with the same number of states (i.e., see Runs #105b,d,i,j). For the simple grammar Tomita #4, nearly all networks converged during training (learned the complete training set). For all cases that converged, it is possible to extract state machines that are perfect, i.e., the FSA of the unknown source grammar. For these cases the minimized, extracted FSA with the same number of states constitute a large equivaZence cZass of neuralnet-generated FSA, that is, all unminimized FSA are equivalent and have the same performance on string classification. This equivalence class extends across neural networks that vary both in size (number of neurons) and


403

initial conditions. Thus, the extracted FSA give some indication of how well the neural network learns the grammar. In fact, for some of the well-trained neural nets, for example, run #104, all extracted, minimized FSA for a large range of partition parameters (250) are the same as the ideal FSA of the source grammar. We speculate that for these well-trained neural nets, the extracted, minimal FSA will be independent of the choice of the partition parameter. These perfect FSA outperform some of the trained neural networks in correct classification of unseen strings. (By definition, a perfect FSA will correctly classify all unseen strings.) This is not surprising due to the possibility of error accumulation as the neural network classifies long unseen strings (Pollack 1990). However, when the neural network has learned the grammar well, its generalization performance is also perfect (for all strings tested). Thus, the neural network can be considered as a tool for extracting an FSA that is representative of the unknown grammar. Once the FSA is extracted, it can be used independently of the trained neural network. Can we make any arguments regarding neural net capacity and scalability? In our simulations the number of states of the minimal FSA that was extracted was comparable to the number of neurons in the network; but the actual extracted, unminimized FSA had many more states than neurons. However, for Runs #105e and #104h the neural network actually learned an elegant solution, the perfect FSA of the grammar (no minimization was necessary). The question of FSA state capacity and scalability is unresolved. Further work must show how well these approaches can model grammars with large numbers of states and what FSA state capacity of the neural net is theoretically and experimentally reasonable. How a complete-gradient calculation approach using second-order recurrent networks compares to other gradient-truncation, first-order methods (Cleeremans et al. 1989; Elman 1990) is another open question. Surprisingly, a simple clustering approach derives useful and representative FSA from a trained or training neural network. Acknowledgments We would like to acknowledge useful discussions with M. W. Goudreau, S. J. Hanson, G. M. Kuhn, J. McClelland, J. B. Pollack, E. Sontag, D. S. Touretzky, and R. L. Watrous. The University of Maryland authors gratefully acknowledge partial support through giants from AFOSR and DARPA. References Angluin, D., Smith, C . H. 1983. Inductive inference: Theory and methods. ACM Comput. Sum. 15(3), 237.

404

C. L. Giles et al.

Cleeremans, A., Servan-Schreiber, D., and McClelland, J. 1989. Finite state automata and simple recurrent networks. Neural Comp. 1(3), 372. Elman, J. L. 1990. Finding structure in time. Cog. Sci. 14, 179. Fu, K. S. 1982. Syntactic Pattern Recognition and Applications. Prentice-Hall, Englewood Cliffs, NJ. Giles, C. L., Sun, G. Z., Chen, H. H., Lee, Y. C., and Chen, D. 1990. Higher order recurrent networks and grammatical inference. In Advances in Neural Information Systems 2, D. S. Touretzky, ed., p. 380. Morgan Kaufmann, San Mateo, CA. Giles, C. L., Chen, D., Miller, C. B., Chen, H. H., Sun, G. Z., and Lee, Y. C. 1991. Grammatical inference using second-order recurrent neural networks. In Proceedings of the International Joint Conference on Neural Networks, IEEE 91CH3049-4, Vol. 2, p. 357. IEEE. Gold, E. M. 1978. Complexity of automaton identification from given data. Inform. Control 37, 302. Harrison, M. H. 1978. Introduction to Formal Language Theory. Addison-Wesley, Reading, MA. Hertz, J., Krogh, A., and Palmer, R. G. 1991. Introduction to the Theory of Neural Computation, p. 163. Addison-Wesley, Redwood City, CA. Hopfcroft, J. E., and Ullman, J. D. 1979. Introduction to Automata Theory,Languages and Computation, p. 68. Addison-Wesley, Reading, MA. Lee, Y. C., Doolen, G., Chen, H. H., Sun, G. Z., Maxwell, T., Lee, H. Y., and Giles, C. L. 1986. Machine learning using a higher order correlational network. Physica D 22-DU-31, 276. Lu, Y. D., Sun, G. Z., Chen, H. H., Lee, Y. C., and Giles, C. L. 1990. Grammatical inference and neural network state machines. In Proceedings of the International Joint Conference on Neural Networks, IJCNN-90-WASH-DC, Vol. I, p. 285. Lawrence Erlbaum, Hillsdale, NJ. Miclet, L. 1990. Grammatical inference. In Syntactic and Structural Pattern Recognition Theoryand Applications, H. Bunke and A. Sanfeliu, eds., Chap. 9. World Scientific, Singapore. Minsky, M. L. 1967. Computation: Finiteand InfiniteMachines, Chap. 3.5. PrenticeHall, Englewood Cliffs, NJ. Mozer, M. C., and Bachrach, J. 1990. Discovering the structure of a reactive environment by exploration. Neural Comp. 2(4), 447. Pollack, J. B. 1990. The Induction of Dynamical Recognizers. Tech. Rep. 90-JPAutomata, Dept. of Computer and Information Science, Ohio State University. Rivest, R. L., and Schapire, R. E. 1987. Diversity-based inference of finite automata. Proc. Twenty-Eight Annu. Symp. Found. Comput. Sci., p. 78. Sun, G. Z., Chen, H. H., Giles, C. L., Lee, Y. C., and Chen, D. 1990. Connectionist pushdown automata that learn context-free grammars. In Proceedings of the International Joint Conference on Neural Networks, IJCNN-90-WASH-DC, Vol. I, p. 577. Lawrence Erlbaum, Hillsdale, NJ. Tomita, M. 1982. Dynamic construction of finite-state automata from examples using hill-climbing. Proc. Fourth Annu. Cogn. Sci. Conf., p. 105.


405

Watrous, R. L., and Kuhn, G. M. 1992. Induction of finite-state languages using second-order recurrent networks. Neural Comp. 4(3), 406-414. Williams, R. J., and Zipser, D. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural Comp. 1(2), 270. Received 6 June 1991; accepted 29 October 1991.

This article has been cited by: 1. Jinmiao Chen, N.S. Chaudhari. 2009. Segmented-Memory Recurrent Neural Networks. IEEE Transactions on Neural Networks 20:8, 1267-1280. [CrossRef] 2. Ueli Rutishauser, Rodney J. Douglas. 2009. State-Dependent Computation Using Coupled Recurrent NetworksState-Dependent Computation Using Coupled Recurrent Networks. Neural Computation 21:2, 478-509. [Abstract] [Full Text] [PDF] [PDF Plus] 3. E. Kolman, M. Margaliot. 2008. A New Approach to Knowledge-Based Design of Recurrent Neural Networks. IEEE Transactions on Neural Networks 19:8, 1389-1401. [CrossRef] 4. Miguel Delgado, Manuel P. Cuellar, Maria Carmen Pegalajar. 2008. Multiobjective Hybrid Optimization and Training of Recurrent Neural Networks. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 38:2, 381-403. [CrossRef] 5. Dezhe Z Jin. 2008. Decoding spatiotemporal spike sequences via the finite state automata dynamics of spiking neural networks. New Journal of Physics 10:1, 015010. [CrossRef] 6. Henrik Jacobsson. 2006. The Crystallizing Substochastic Sequential Machine Extractor: CrySSMExThe Crystallizing Substochastic Sequential Machine Extractor: CrySSMEx. Neural Computation 18:9, 2211-2255. [Abstract] [PDF] [PDF Plus] 7. Peter Tiňo , Ashely J. S. Mills . 2006. Learning Beyond Finite Memory in Recurrent Networks of Spiking NeuronsLearning Beyond Finite Memory in Recurrent Networks of Spiking Neurons. Neural Computation 18:3, 591-613. [Abstract] [PDF] [PDF Plus] 8. André Grüning. 2006. Stack-like and queue-like dynamics in recurrent neural networks. Connection Science 18:1, 23-42. [CrossRef] 9. Henrik Jacobsson . 2005. Rule Extraction from Recurrent Neural Networks: ATaxonomy and ReviewRule Extraction from Recurrent Neural Networks: ATaxonomy and Review. Neural Computation 17:6, 1223-1263. [Abstract] [PDF] [PDF Plus] 10. Zhi-Hua Zhou. 2004. Rule extraction: Using neural networks or for neural networks?. Journal of Computer Science and Technology 19:2, 249-253. [CrossRef] 11. A. Vahed, C. W. Omlin. 2004. A Machine Learning Method for Extracting Symbolic Knowledge from Recurrent Neural NetworksA Machine Learning Method for Extracting Symbolic Knowledge from Recurrent Neural Networks. Neural Computation 16:1, 59-71. [Abstract] [PDF] [PDF Plus] 12. P. Tino, M. Cernansky, L. Benuskova. 2004. Markovian Architectural Bias of Recurrent Neural Networks. IEEE Transactions on Neural Networks 15:1, 6-15. [CrossRef]

13. Peter Tiňo , Barbara Hammer . 2003. Architectural Bias in Recurrent Neural Networks: Fractal AnalysisArchitectural Bias in Recurrent Neural Networks: Fractal Analysis. Neural Computation 15:8, 1931-1957. [Abstract] [PDF] [PDF Plus] 14. Wolfgang Maass , Thomas Natschläger , Henry Markram . 2002. Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on PerturbationsReal-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations. Neural Computation 14:11, 2531-2560. [Abstract] [PDF] [PDF Plus] 15. Stephen José Hanson , Michiro Negishi . 2002. On the Emergence of Rules in Neural NetworksOn the Emergence of Rules in Neural Networks. Neural Computation 14:9, 2245-2268. [Abstract] [PDF] [PDF Plus] 16. Chia-Feng Juang. 2002. A TSK-type recurrent fuzzy network for dynamic systems processing by neural network and genetic algorithms. IEEE Transactions on Fuzzy Systems 10:2, 155-170. [CrossRef] 17. Michael Schmitt . 2002. On the Complexity of Computing and Learning with Multiplicative Neural NetworksOn the Complexity of Computing and Learning with Multiplicative Neural Networks. Neural Computation 14:2, 241-301. [Abstract] [PDF] [PDF Plus] 18. J.C. Principe, V.G. Tavares, J.G. Harris, W.J. Freeman. 2001. Design and implementation of a biologically realistic olfactory cortex in analog VLSI. Proceedings of the IEEE 89:7, 1030-1051. [CrossRef] 19. Peter Tiňo , Bill G. Horne , C. Lee Giles . 2001. Attractive Periodic Sets in Discrete-Time Recurrent Networks (with Emphasis on Fixed-Point Stability and Bifurcations in Two-Neuron Networks)Attractive Periodic Sets in Discrete-Time Recurrent Networks (with Emphasis on Fixed-Point Stability and Bifurcations in Two-Neuron Networks). Neural Computation 13:6, 1379-1414. [Abstract] [PDF] [PDF Plus] 20. Edward Kei Shiu Ho , Lai Wan Chan . 2001. Analyzing Holistic Parsers: Implications for Robust Parsing and SystematicityAnalyzing Holistic Parsers: Implications for Robust Parsing and Systematicity. Neural Computation 13:5, 1137-1170. [Abstract] [PDF] [PDF Plus] 21. Stefan C. Kremer . 2001. Spatiotemporal Connectionist Networks: A Taxonomy and ReviewSpatiotemporal Connectionist Networks: A Taxonomy and Review. Neural Computation 13:2, 249-306. [Abstract] [PDF] [PDF Plus] 22. A. Paccanaro, G.E. Hinton. 2001. Learning distributed representations of concepts using linear relational embedding. IEEE Transactions on Knowledge and Data Engineering 13:2, 232-244. [CrossRef] 23. R.C. Carrasco, M.L. Forcada. 2001. Simple strategies to encode tree automata in sigmoid recursive neural networks. IEEE Transactions on Knowledge and Data Engineering 13:2, 148-156. [CrossRef]

24. Rafael C. Carrasco , Mikel L. Forcada , M. Ángeles Valdés-Muñoz , Ramón P. Ñeco . 2000. Stable Encoding of Finite-State Machines in Discrete-Time Recurrent Neural Nets with Sigmoid UnitsStable Encoding of Finite-State Machines in Discrete-Time Recurrent Neural Nets with Sigmoid Units. Neural Computation 12:9, 2129-2174. [Abstract] [PDF] [PDF Plus] 25. A. Blanco, M. Delgado, M. C. Pegalajar. 2000. Extracting rules from a (fuzzy/crisp) recurrent neural network using a self-organizing map. International Journal of Intelligent Systems 15:7, 595-621. [CrossRef] 26. S. Lawrence, C.L. Giles, S. Fong. 2000. Natural language grammatical inference with recurrent neural networks. IEEE Transactions on Knowledge and Data Engineering 12:1, 126-140. [CrossRef] 27. Morten H Christiansen, Nick Chater. 1999. Toward a Connectionist Model of Recursion in Human Linguistic Performance. Cognitive Science 23:2, 157-205. [CrossRef] 28. N. Srinivasa, N. Ahuja. 1999. A topological and temporal correlator network for spatiotemporal pattern learning, recognition, and recall. IEEE Transactions on Neural Networks 10:2, 356-371. [CrossRef] 29. P. Tino, M. Koteles. 1999. Extracting finite-state representations from recurrent neural networks trained on chaotic symbolic sequences. IEEE Transactions on Neural Networks 10:2, 284-302. [CrossRef] 30. C.L. Giles, C.W. Omlin, K.K. Thornber. 1999. Equivalence in knowledge representation: automata, recurrent neural networks, and dynamical fuzzy systems. Proceedings of the IEEE 87:9, 1623-1640. [CrossRef] 31. Chun-Hsien Chen, V. Honavar. 1999. A neural-network architecture for syntax analysis. IEEE Transactions on Neural Networks 10:1, 94-114. [CrossRef] 32. A.B. Tickle, R. Andrews, M. Golea, J. Diederich. 1998. The truth will come to light: directions and challenges in extracting the knowledge embedded within trained artificial neural networks. IEEE Transactions on Neural Networks 9:6, 1057-1068. [CrossRef] 33. S. Das, O. Olurotimi. 1998. Noisy recurrent neural networks: the continuous-time case. IEEE Transactions on Neural Networks 9:5, 913-936. [CrossRef] 34. P. Frasconi, M. Gori, A. Sperduti. 1998. A general framework for adaptive processing of data structures. IEEE Transactions on Neural Networks 9:5, 768-786. [CrossRef] 35. C.W. Omlin, K.K. Thornber, C.L. Giles. 1998. Fuzzy finite-state automata can be deterministically encoded into recurrent neural networks. IEEE Transactions on Fuzzy Systems 6:1, 76-89. [CrossRef] 36. Yoram Singer . 1997. Adaptive Mixtures of Probabilistic TransducersAdaptive Mixtures of Probabilistic Transducers. Neural Computation 9:8, 1711-1733. [Abstract] [PDF] [PDF Plus]

37. Alan D. Blair, Jordan B. Pollack. 1997. Analysis of Dynamical RecognizersAnalysis of Dynamical Recognizers. Neural Computation 9:5, 1127-1142. [Abstract] [PDF] [PDF Plus] 38. Ephraim Nissan, Hava Siegelmann, Alex Galperin, Shuky Kimhi. 1997. Upgrading automation for nuclear fuel in-core management: From the symbolic generation of configurations, to the neural adaptation of heuristics. Engineering with Computers 13:1, 1-19. [CrossRef] 39. Hava T. Siegelmann. 1996. RECURRENT NEURAL NETWORKS AND FINITE AUTOMATA. Computational Intelligence 12:4, 567-574. [CrossRef] 40. S.C. Kremer. 1996. Comments on "Constructive learning of recurrent neural networks: limitations of recurrent cascade correlation and a simple solution". IEEE Transactions on Neural Networks 7:4, 1047-1051. [CrossRef] 41. Christian W. Omlin, C. Lee Giles. 1996. Stable Encoding of Large Finite-State Automata in Recurrent Neural Networks with Sigmoid DiscriminantsStable Encoding of Large Finite-State Automata in Recurrent Neural Networks with Sigmoid Discriminants. Neural Computation 8:4, 675-696. [Abstract] [PDF] [PDF Plus] 42. Paolo Frasconi, Marco Gori, Marco Maggini, Giovanni Soda. 1996. Representation of finite state automata in Recurrent Radial Basis Function networks. Machine Learning 23:1, 5-32. [CrossRef] 43. Y. Bengio, P. Frasconi. 1996. Input-output HMMs for sequence processing. IEEE Transactions on Neural Networks 7:5, 1231-1249. [CrossRef] 44. J.-P.S. Draye, D.A. Pavisic, G.A. Cheron, G.A. Libert. 1996. Dynamic recurrent neural networks: a dynamical analysis. IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) 26:5, 692-706. [CrossRef] 45. C.W. Omlin, C.L. Giles. 1996. Rule revision with recurrent neural networks. IEEE Transactions on Knowledge and Data Engineering 8:1, 183-188. [CrossRef] 46. Kam-Chuen Jim, C.L. Giles, B.G. Horne. 1996. An analysis of noise in recurrent neural networks: convergence and generalization. IEEE Transactions on Neural Networks 7:6, 1424-1438. [CrossRef] 47. Giovanna Castellano, Anna Maria Fanelli, Marcello Pelillo. 1995. Iterative pruning in second-order recurrent neural networks. Neural Processing Letters 2:6, 5-8. [CrossRef] 48. R. Alquézar , A. Sanfeliu . 1995. An Algebraic Framework to Represent Finite State Machines in Single-Layer Recurrent Neural NetworksAn Algebraic Framework to Represent Finite State Machines in Single-Layer Recurrent Neural Networks. Neural Computation 7:5, 931-949. [Abstract] [PDF] [PDF Plus] 49. Peter Tiňo , Jozef Šajda . 1995. Learning and Extracting Initial Mealy Automata with a Modular Neural Network ModelLearning and Extracting Initial Mealy Automata with a Modular Neural Network Model. Neural Computation 7:4, 822-844. [Abstract] [PDF] [PDF Plus]

50. Jun Tani, Naohiro Fukumura. 1995. Embedding a grammatical description in deterministic chaos: an experiment in recurrent neural learning. Biological Cybernetics 72:4, 365-370. [CrossRef] 51. Peter Manolios , Robert Fanelli . 1994. First-Order Recurrent Neural Networks and Deterministic Finite State AutomataFirst-Order Recurrent Neural Networks and Deterministic Finite State Automata. Neural Computation 6:6, 1155-1173. [Abstract] [PDF] [PDF Plus] 52. John L. Johnson. 1994. Pulse-coupled neural nets: translation, rotation, scale, distortion, and intensity signal invariance for images. Applied Optics 33:26, 6239. [CrossRef] 53. Zheng Zeng , Rodney M. Goodman , Padhraic Smyth . 1993. Learning Finite State Machines With Self-Clustering Recurrent NetworksLearning Finite State Machines With Self-Clustering Recurrent Networks. Neural Computation 5:6, 976-990. [Abstract] [PDF] [PDF Plus] 54. Mikel L. Forcada, Marco GoriNeural Nets, Recurrent . [CrossRef]

Communicated by James McClelland

Induction of Finite-State Languages Using Second-Order Recurrent Networks Raymond L. Watrous Siemens Corporate Research, 755 College Road East, Princeton, NJ 08540 USA Gary M.Kuhn Center for C o ~ m ~ n j c a f i oResearch, ns IDA, Thanet Road, Princefon, N]08540 USA

Second-order recurrent networks that recognize simple finite state languages over {0,1}* are induced from positive and negative examples. Using the complete gradient of the recurrent network and sufficient training examples to constrain the definition of the language to be induced, solutions are obtained that correctly recognize strings of arbitrary length. 1 Introduction We address the problem of inducing languages from examples by considering a set of finite state languages over (0,l)" that were selected for study by Tomita (1982):

L1. 1* L2. (lo)*

L3. no odd-length 0-string anywhere after an odd-length 1-string L4. not more than 2 0s in a row L5. bit pairs, # O h

+ #1Os = 0 mod 2

L6. abs(#ls - #Os) = 0 mod 3

L7. 0*1*0*1* Tomita also selected for each language a set of positive and negative examples (summarized in Table 1) to be used as a training set. By a method of heuristic search over the space of finite state automata with up to eight states, he was able to induce a recognizer for each of these languages (Tomita 1982). Recognizers of finite-state languages have also been induced using first-order recurrent connectionist networks (Elman 1990; Williams and Zipser 1988; Cleeremans et al. 1989). Generally speaking, these results Neural Computation 4, 406-414 (1992) @ 1992 Massachusetts Institute of Technology

Induction of Finite-State Languages

407

were obtained by training the network to predict the next symbol (Cleeremans et al. 1989; Williams and Zipser 1988), rather than by training the network to accept or reject strings of different lengths. Several training algorithms used an approximation to the gradient (Elman 1990; Cleeremans et aZ. 1989) by truncating the computation of the backward recurrence. The problem of inducing languages from examples has also been approached using second-order recurrent networks (Pollack 1990; Giles e f al. 1990). Using a truncated approximation to the gradient, and Tomita's training sets, Pollack reported that "none of the ideal languages were induced (Pollack 1990). On the other hand, a Tomita language has been induced using the complete gradient (Giles etal. 1991). The present paper also reports the induction of several Tomita languages using the complete gradient with certain differences in method from Giles et al. (1991). 2 Method

2.1 Architecture. The network model consists of one input unit, one threshold unit, N state units, and one output unit. The output unit and each state unit receive a first-order connection from the input unit and the threshold unit. In addition, each of the output and state units receives a second-order connection for each pairing of the input and threshold unit with each of the state units. For N = 3, the model is mathematically identical to that used by Pollack (1990); it has 32 free parameters. 2.2 Data Representation. The symbols of the language are represented by byte values, that are mapped into real values between 0 and 1 by dividing by 255. Thus, the ZERO symbol is represented by octal 040 (0.1255). This value was chosen to be different from 0.0, which is used as the initial condition for all units except the threshold unit, which is set to 1.0. The ONE symbol was chosen as octal 370 (0.97255). All strings are terminated by two occurrences of a termination symbol that has the value 0.0.

2.3 Training. The Tomita languages are characterized in Table 1 by the number of grammatical strings of length 10 or less (out of a total of 2047 strings). The Tomita training sets are also characterized by the number of grammatical strings of length 10 or less included in the training data. For completeness, Table 1 also shows the number of grammatical strings in the training set of length greater than 10. A comparison of the number of grammatical strings with the number included in the training set shows that while Languages 1 and 2 are very sparse, they are almost completely covered by the training data, whereas Languages 3-7 are more dense, and are sparsely covered by the training sets. Possible consequences of these differences are considered in discussing the experimental results.

Raymond L. Watrous and Gary M. Kuhn

408

Table 1: Number of Grammatical and Ungrammatical Strings of Length 10 or Less for Tomita Languages and Number of Those Included in the Tomita Training Sets. Grammatical strings

Ungrammaticalstrings

Length 5 10 Longer strings Length 5 10 Longer strings Language Total Training in training set Total Training in training set 1 2 3 4 5 6 7

11 6 652 1103 683 683 561

9 5 11 10 9 10 11

1 2 1

2

2036 2041 1395 944 1364 1364 1486

8 10 11 7 11 11 6

1 2 1 1 2

A mean-squared error measure was defined with target values of 0.9 and 0.1 for accept and reject, respectively. The target function was weighted so that error was injected only at the end of the string, The complete gradient of this error measure for the recurrent network was computed by a method of accumulating the weight dependencies backward in time (Watrous ef al. 1990). This is in contrast to the truncated gradient used by Pollack (1990) and to the forward-propagation algorithm used by Giles et al. (1991). The networks were optimized by gradient descent using the BFGS algorithm (Luenberger 1984). A termination criterion of was set; it was believed that such a strict tolerance might lead to smaller loss of accuracy on very long strings. No constraints were set on the number of iterations. Five networks with different sets of random initial weights were trained separately on each of the seven languages described by Tomita using exactly his training sets (Tomita 19821, including the null string. The training set used by Pollack (1990) differs only in not including the null string (Pollack 1991). 2.4 Testing. The networks were tested on the complete set of strings up to length 10. Acceptance of a string was defined as the network having a final output value of greater than 0.9 - T and rejection as a final value of less than 0.1 T, where 0 5 T 0.95) and the differences between the 8 component model and the other models are much larger. Weigend et a!. point out that for time series prediction tasks such as the sunspot task a much more interesting measure of performance is the ability of the model to predict more than one time step into the future. 15This was the model favored by Priestley (1991a) in a recent evaluation of classical statistical approaches to this task.

Simplifying Neural Networks

485

Table 2: Summary of average relative variance of five different models on the one-step sunspot prediction problem. Method TAR RBF WRH Soft-share - 3 Comp. Soft-share - 8 Comp.

Test arv 0.097 0.092 0.086 0.077 f0.0029 0.072 f 0.0022

One way to approach the multistep prediction problem is to use iterated single-step prediction. In this method, the predicted output is fed back as input for the next prediction and all other input units have their values shifted back one unit. Thus the input typically consists of a combination of actual and predicted values. We define the predicted value for time t, obtained after 1 iterations to be it,1. The prediction error will depend not only on I but also on the time ( t - I ) when the iteration was started. In order to account for both effects, Weigend e f al. suggested the average relative 1-times iterated prediction variance as a performance measure for iterated prediction:

where M is the number of different start times for iterated prediction and 6 is the estimated standard deviation of the set of target values. In Figure 4 we plot this measure (computed over the test set from 1921 to 1955) as a function of the number of prediction iterations for the simulations using the 3 and 8 component complexity measures, the Tong and Lim model (TAR), and the model from Weigend et al., which produced the lowest single step am (WRH). The plots for the 3 and 8 component complexity models are the averages over 10 simulations with the error bars indicating the plus or minus one standard deviation intervals. Once again, the differences between the 3 and 8 component models are significant for all numbers of iterations. The differences between the adaptive gaussian complexity measure and the fixed complexity measure used by Weigend et al. are not as dramatic on the sunspot task as they were in the shift detection task. The explanation for this may be seen in Figures 5 and 6, which show a typical set of weights learned by the soft-sharing model with 8 mixture components and the corresponding final mixture probability density. The distinct weight groups seen clearly in the shift detection task (Fig. 2) are not as apparent in the weights for the sunspot task and the final weight distribution for the sunspot task is very smeared out except for one very strong sharp component near 0. It is clear that the fixed model assumed

Steven J. Nowlan and Geoffrey E. Hinton

486

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3 0-2 0.1

(

Figure 4: Average relative I-times iterated prediction variance versus number of prediction iterations for the sunspot time series from 1921 to 1955. Closed circles represent the TAR model, open circles the WRH model, closed squares the 3 component complexity model, and open squares the 8 component complexity model. One deviation error bars are shown for the 3 and 8 component complexity models.

by Weigend et al. is much more appropriate for the sunspot prediction task than it was for the shift detection task.

7 A Minimum Description Length Perspective

A number of authors have suggested that when attempting to approximate a n unknown function with some parametric approximation scheme (such as a network), the proper measure to optimize combines an estimate of the cost of the misfit with a n estimate of the cost of describing the parametric approximation (Akaike 1973; Rissanen 1978; Barron and Barron 1988). Such a measure is often referred to as a minimum description


487

Figure 5 A diagram of weights discovered for the sunspot prediction problem by a model which employed an 8 component mixture for the complexity cost. Weights are shown for all 8 hidden units. For each unit, the weights coming from the 12 inputs are shown in a row with the single weight to the output immediately above the row. The biases of the hidden units, which are not shown, were, with one exception, small negative numbers very close in value to most of the other weights in the network. The first three units in the left column all represent the simple rule that the number of sunspots depends on the number in the previous year. The last two units in this column compute a simple moving average. The three units on the right represent more interesting rules. The first captures the 11 year cycle, the second recognizes when a peak has just passed, and third appears to prevent the prediction from rising too soon if a peak happened 9 years ago and the recent activity is low.

length criterion (MDL), and typically has the general form

MDL

=

c

messages

- log p(message)

+

c

- logp(parameter).

parameters

For a supervised network, the parameters are the weights and the messages are the desired outputs. If we assume that the output errors are gaussian and that the weights are encoded using a mixture of gaussians probability model the description length is approximated by equation 4.1. The expression in equation 4.1 does not include the cost of encoding the means and variances of the mixture components or the mixing proportions of the mixture density. Since the mixture usually contains a small number of components (fewer than 10 usually) and there are only three parameters associated with each component, the cost of encoding these parameters is negligible compared to the cost of encoding the

488


Figure 6: Final mixture probability density for the set of weights shown in Figure 5. The density is dominated by a narrow component centered very near zero, with the remaining components blending into a skewed distribution with a peak around 0.5. weights in most networks of interest.16 In addition, since the number of components in the distribution does not change during the optimization, if the component parameters are all encoded with the same fixed precision, the cost of the mixture parameters is simply a constant offset, which is ignored in the optimi~ation.'~ There is one important aspect of estimating the cost of describing a weight that we have ignored. We have assumed that the cost of a weight is the negative logarithm of a probability density function evaluated at the weight value, but this ignores the accuracy with which the weight must be described. We are really interested in the probability mass of a particular small interval of values for the parameter, and this means that we should integrate our density function over this interval to estimate the cost of each weight. We have implicitly assumed that this integration 161norder to provide enough data to fit the mixture density, one should have an order of magnitude more weights than components in the mixture. "This ignores the possibility of not encoding the parameters of components whose mixing proportions approach 0.


489

region has the same (infinitesimal) width for every weight, and so the probability of a weight is simply proportional to the density function. This ignores the fact that most networks are generally much more sensitive to small changes in some weight values than others, so some weights need to be encoded more accurately than others.I8 The sensitivity of a network to a small change in a weight is determined by the curvature of the error surface. One could evaluate the curvature by computing the Hessian and make the width of the integration region for each weight inversely proportional to the curvature along each weight dimension. To be perfectly accurate, one would need to integrate the joint probability density function for all of the weights over a region determined by the complete Hessian (since the directions of maximum curvature are often not perfectly aligned with the weight axes). This process would be computationally very costly, and an adequate approximation might be obtainable by using a diagonal approximation to the Hessian and treating each weight independently (as advocated by le Cun et al. 1990). We see no reason why our method of estimating the probability density should not be combined with a method for estimating the integration interval. For small intervals, this is particularly easy since the probability mass is approximately the width of the interval times the height of the density function so these two terms are additive in the log probability domain. 8 A Bayesian Perspective

As a number of authors have recently pointed out (Weigend et al. 1991; Nowlan 1991; MacKay 1991; Buntine and Weigend 1991), equation 1.1 can be derived from the principles of Bayesian inference. If we have a set of models MI,M 2 )... , which are competing to account for the data, Bayesian inference is concerned with how we should update our belief in the relative plausibility of each of these models in light of the data D. If P(M; I D ) is the plausibility of model M; given we have observed D, Bayes rule states

where P ( D I Mi) is a measure of how well model i predicts the data and P ( M , ) is our belief in the plausibility of model i before we have seen any data. Here P ( D ) is simply a normalizing factor to ensure our beliefs add up to one. If we are only interested in comparing alternate models, P ( D ) can be ignored and in the log domain equation 8.1 becomes equation 1.1 with the data-misfit cost equal to logP(D I Mi) and the complexity ''Other things being equal, we should prefer networks in which the outputs are less sensitive to the precise weight values, since then the weight values can be encoded imprecisely without causing large output errors.

490


cost equal to logP(M,). If we are only considering a single network architecture, P(Mj) becomes a prior distribution over the set of possible weights.’’ What equation 8.1 highlights is that in the Bayesian framework our complexity cost should be independent of our data. This is certainly true when the complexity is the sum of the squares of the weights, and also holds for models such as the one used by Weigend et al. However, the mixture densities discussed in this paper are clearly not independent of the data and cannot be regarded as classical Bayesian priors. The complexity cost we are using corresponds more closely to a Bayesian hyperprior (Jaynes 1986; Gull 1988). We have specified a particular family of distributions from which the prior will be drawn but have left the parameters of the prior (T,, p,, a,) undetermined. Members of this family of distributions have the common feature of favoring sets of weights in which the weights in a set are clustered about a small number of values.20When using a hyperprior, we can deal with the hyperparameters either by marginalizing over them (in effect, integrating them out) (Buntine and Weigend 19911, or by allowing the data (i.e., the weights) to determine their values a posteriori.” We have used this second approach, which is advocated by Gull (1988), who has shown that the use of such flexible hyperpriors can lead to considerable improvement in the quality of image reconstructions (Gull 1989; Skilling 1989) compared to the use of more classical priors. The trick of optimizing 7, rather than a, (discussed at the end of Section 4) may also be justified within the Bayesian framework. To estimate our hyperparameters, we should properly specify prior distributions for each. If these priors are uninformative:’ then the estimated values of the hyperparameters are determined entirely by the data. A parameter like a, is known as a scale parameter (it affects the width of the distribution) while parameters like pj are known as location parameters (they affect the position of the distribution). (See Jeffreys 1939 for further discussion.) An uninformative prior for a location parameter is uniform in the parameter, but an uninformative prior for a scale parameter is uniform in the log of the parameter ke., uniform in 7, rather than a,,Gull 1988). It is more consistent from this perspective to treat 7 and pj similarly, rather than u,and pj.

”Much more interesting results are obtained when we apply this framework to making choices among many architectures, see MacKay for some elegant examples (MacKay 1991). 20Thelocations of these clusters will generally be different for different sets of weights. 211nprinciple, both approaches will lead to the same posterior distribution over the weights and the same ultimate choice of weights for the network. The difference lies in whether we are searching over a joint space of weights and hyperparameters or using prior analytic ,simplifications to reduce the search to some manifold in weight space alone. =A prior that contains no initial bias except for a possible range constraint.


491

9 Summary

The simulations we have described provide evidence that the use of a more sophisticated model for the distribution of weights in a network can lead to better generalization performance than a simpler form of weight decay, or techniques that control the learning time. The better generalization performance comes at the cost of greater complexity in the optimization of the weights. The effectiveness of the technique is likely to be somewhat problem dependent, but one advantage offered by the more sophisticated model is its ability to automatically adapt the model of the weight distribution to individual problems.

Acknowledgments This research was funded by grants from the Ontario Information Technology Research Center and the Canadian Natural Science and Engineering Research Council and the Howard Hughes Medical Institute. Hinton is the Noranda fellow of the Canadian Institute for Advanced Research.

References Akaike, H. 1973. Information theory and an extension of the maximum likelihood principle. In Proceedings 2nd International Symposium on Information Theory, B. N. Petrov and E Csaki, eds., pp. 267-281. Akademia Kiado, Budapest, Hungary. Barron, A. R., and Barron, R. L. 1988. Statistical learning networks: A unifying view. In 1988 Symposium on the Interface: Statistics and Computing Science, Reston, Virginia, April 21-23. Baum, E. B., and Haussler, D. 1989. What size net gives valid generalization? Neural Comp. 1, 151-160. Buntine, W. L., and Weigend, A. S. 1991. Bayesian back-propagation. Complex Systems 5(6), 603-643. Dempster, A. P., Laird, N. M., and Rubin, D. 8. 1977. Maximum likelihood from incomplete data via the EM algorithm. Proc. R. Stat. SOC.Ser. B 39, 1-38. Gull, S. E 1988. Bayesian inductive inference and maximum entropy. In Maximum Entropy and Bayesian Methods in Science and Engineering, G. J. Erickson and C. R. Smith, eds. Kluwer Academic, Dordrecht, The Netherlands. Gull, S. F. 1989. Developments in maximum entropy data analysis. In Maximum Entropy and Bayesian Methods (8th Workshop), J. Skilling, ed., pp. 53-71. Kluwer Academic, Dordrecht, The Netherlands. He, X., and Lapedes, A. 1991. Nonlinear Modelling and Prediction by Successive Approximation Using Radial Basis Functions. Tech. Rep. LA-UR-91-1375, Los Alamos National Laboratory.

492


Hinton, G. E. 1986. Learning distributed representations of concepts. In Proceedings of the Eighth Annual Conference of the Cognitive Science Society, pp. 1-12. Erlbaum, Hillsdale, NJ. Hinton, G. E. 1987. Learning translation invariant recognition in a massively parallel network. In Proc. Conf. Parallel Architectures and Languages Europe, pp. 1-13. Eindhoven, The Netherlands. Jaynes, E. T. 1986. Bayesian methods: General background. In Maximum Entropy and Bayesian Methods in Applied Statistics, J. H. Justice, ed., pp. 1-25. Cambridge University Press, Cambridge. Jeffreys, H. 1939. Theory of Probability. Oxford University Press, Oxford. Later editions 1948, 1961, 1983. Kohonen, T. 1977. Associative Memoy: A System-Theoretical Approach. Springer, Berlin. Lang, K. J., Waibel, A. H., and Hinton, G. E. 1990. A time-delay neural network architecture for isolated word recognition. Neural Networks 3, 2343. LeCun, Y. 1987. Modeles connexionnistes de l’apprentissage. Ph.D. thesis, Universite Pierre et Marie Curie, Paris, France. LeCun, Y. 1989. Generalization and Network Design Strategies. Tech. Rep. CRGTR-89-4, University of Toronto. LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. 1990. Handwritten digit recognition with a backpropagation network. In Advances in Neural Information Processing Systems 2, D. S. Touretzky, ed., pp. 39-04. Morgan Kaufmann, San Mateo, CA. LeCun, Y., Denker, J., Solla, S., Howard, R. E., and Jackel, L. D. 1990. Optimal brain damage. In Advances in Neural Information Processing Systems 2,598-605, D. S. Touretzky, ed. Morgan Kaufmann, San Mateo, CA. MacKay, D. J. C. 1991. Bayesian modeling and neural networks. Ph.D. thesis, Computation and Neural Systems, California Institute of Technology, Pasadena, CA. McLachlan, G. J., and Basford, K. S. 1988. Chapters 1 and 2. In Mixture Models: Inference and Applications to Clustering, G. J. McLachlan and K. E. Basford, eds., pp. 1-69. Marcel Dekker, New York. Morgan, N., and Bourlard, H. 1989. Generalization and Parameter Estimation in FeedforwardNets: Some Experiments. Tech. Rep. TR-89-017, International Computer Science Institute, Berkeley, CA. Mozer, M. C., and Smolensky, P. 1989. Using relevance to reduce network size automatically. Connection Sci. 1(1),3-16. Nowlan, S. J. 1991. Soft competitive adaptation: Neural network learning algorithms based on fitting statistical mixtures. Ph.D. Thesis, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA. Plaut, D. C., Nowlan, S. J., and Hinton, G. E. 1986. Experiments on Learning by Backpropagation. Tech. Rep. CMU-CS-86-126, Carnegie Mellon University, Pittsburgh, PA. Priestley, M. B. 1991a. Non-linear and Non-stationary Time Series Analysis. Academic Press, San Diego. Priestley, M. 8. 1991b. Spectral Analysis and Time Series. Academic Press, San Diego.

Simplifying NeuraI Networks

493

Rissanen, J. 1978. Modeling by shortest data description. Automatica 14, 465471. Rumelhart, D. E., McClelland, J. L., and the PDP research group. 1986. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vols. I and 11. MIT Press, Cambridge, MA. Skilling, J. 1989. Classic maximum entropy. In Maximum Entropy and Bayesian Methods (8th Workshop), J. Skilling, ed. Kluwer Academic, Dordrecht, The Netherlands. Tong, H., and Lim, K. S. 1980. Threshold autoregression, limit cycles, and cyclical data. I. R. Stat. SOC.Ser. B 42, 245-253. Weigend, A. S., Rumelhart, D. E., and Huberman, B. A. 1990. Predicting the future: A connectionist approach. In Proceedings of the 1990 Connectionist Models Summer School, T. J. Sejnowski, G. E. Hinton, and D. S. Touretzky, eds., pp. 105-116. Morgan Kaufmann, San Mateo, CA. Weigend, A. S., Rumelhart, D. E., and Huberman, B. A. 1991. Generalization by weight-elimination with application to forecasting. In Advances in Neural Information Processing Systems 3, R. P. Lippmann, J. E. Moody, and D. S. Touretzky, eds., pp. 875-882. Morgan Kaufmann, San Mateo, CA. ~~

Received 15 October 1991; accepted 2 December 1991.

This article has been cited by: 1. Huisheng Zhang, Wei Wu, Fei Liu, Mingchen Yao. 2009. Boundedness and Convergence of Online Gradient Method With Penalty for Feedforward Neural Networks. IEEE Transactions on Neural Networks 20:6, 1050-1054. [CrossRef] 2. Gursel Serpen. 2008. Hopfield Network as Static Optimizer: Learning the Weights and Eliminating the Guesswork. Neural Processing Letters 27:1, 1-15. [CrossRef] 3. Kazumi Saito, Ryohei Nakano. 2007. Bidirectional clustering of weights for neural networks with common weights. Systems and Computers in Japan 38:10, 46-57. [CrossRef] 4. Jung Hoon Lee, Konstantin K. Likharev. 2007. Defect-tolerant nanoelectronic pattern classifiers. International Journal of Circuit Theory and Applications 35:3, 239-264. [CrossRef] 5. N. Garcia-Pedrajas, C. Hervas-Martinez, D. Ortiz-Boyer. 2005. Cooperative Coevolution of Artificial Neural Network Ensembles for Pattern Classification. IEEE Transactions on Evolutionary Computation 9:3, 271-302. [CrossRef] 6. Kenji Suzuki. 2004. Determining the receptive field of a neural filter. Journal of Neural Engineering 1:4, 228-237. [CrossRef] 7. A. Sfetsos, C. Siriopoulos. 2004. Time Series Forecasting with a Hybrid Clustering Scheme and Pattern Recognition. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans 34:3, 399-405. [CrossRef] 8. Anil Kumar Ghosh, Smarajit Bose. 2004. Backfitting neural networks. Computational Statistics 19:2, 193-210. [CrossRef] 9. E. Trentin, M. Gori. 2003. Robust combination of neural networks and hidden markov models for speech recognition. IEEE Transactions on Neural Networks 14:6, 1519-1531. [CrossRef] 10. J. Ghosn, Yoshua Bengio. 2003. Bias learning, knowledge sharing. IEEE Transactions on Neural Networks 14:4, 748-765. [CrossRef] 11. C. Alippi, C. de Russis, V. Piuri. 2003. A neural-network based control solution to air-fuel ratio control for automotive fuel-injection systems. IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews) 33:2, 259-268. [CrossRef] 12. Liu Yong, Zou Xiu-fen. 2003. From designing a single neural network to designing neural network ensembles. Wuhan University Journal of Natural Sciences 8:1, 155-164. [CrossRef] 13. Zhe Chen , Simon Haykin . 2002. On Different Facets of Regularization TheoryOn Different Facets of Regularization Theory. Neural Computation 14:12, 2791-2846. [Abstract] [PDF] [PDF Plus] 14. Shun-Feng Su, Chan-Ben Lin, Yen-Tseng Hsu. 2002. A high precision global prediction approach based on local prediction approaches. IEEE Transactions on

Systems, Man and Cybernetics, Part C (Applications and Reviews) 32:4, 416-425. [CrossRef] 15. Fi-John Chang, Hsiang-Fan Hu, Yen-Chang Chen. 2001. Counterpropagation fuzzy-neural network for streamflow reconstruction. Hydrological Processes 15:2, 219-232. [CrossRef] 16. Mirko van der Baan, Christian Jutten. 2000. Neural networks in geophysical applications. Geophysics 65:4, 1032. [CrossRef] 17. Zhimin Huo, Maryellen L Giger, Charles E Metz. 1999. Physics in Medicine and Biology 44:10, 2579-2595. [CrossRef] 18. B. Igelnik, Yoh-Han Pao, S.R. LeClair, Chang Yun Shen. 1999. The ensemble approach to neural-network learning and generalization. IEEE Transactions on Neural Networks 10:1, 19-30. [CrossRef] 19. Sheng Ma, Chuanyi Ji. 1999. Performance and efficiency: recent advances in supervised learning. Proceedings of the IEEE 87:9, 1519-1535. [CrossRef] 20. Ansgar H L West, David Saad. 1998. Journal of Physics A: Mathematical and General 31:45, 8977-9021. [CrossRef] 21. Michael D. Lee . 1998. Neural Feature Abstraction from Judgments of SimilarityNeural Feature Abstraction from Judgments of Similarity. Neural Computation 10:7, 1815-1830. [Abstract] [PDF] [PDF Plus] 22. Kwabena Agyepong, Ravi Kothari. 1997. Controlling Hidden Layer Capacity Through Lateral ConnectionsControlling Hidden Layer Capacity Through Lateral Connections. Neural Computation 9:6, 1381-1402. [Abstract] [PDF] [PDF Plus] 23. Nathan Intrator§, Shimon Edelman. 1997. Learning low-dimensional representations via the usage of multiple-class labels. Network: Computation in Neural Systems 8:3, 259-281. [CrossRef] 24. Rajesh P. N. Rao, Dana H. Ballard. 1997. Dynamic Model of Visual Recognition Predicts Neural Response Properties in the Visual CortexDynamic Model of Visual Recognition Predicts Neural Response Properties in the Visual Cortex. Neural Computation 9:4, 721-763. [Abstract] [PDF] [PDF Plus] 25. Sepp Hochreiter, Jürgen Schmidhuber. 1997. Flat MinimaFlat Minima. Neural Computation 9:1, 1-42. [Abstract] [PDF] [PDF Plus] 26. Tsung-Nan Lin, C.L. Giles, B.G. Horne, Sun-Yuan Kung. 1997. A delay damage model selection algorithm for NARX neural networks. IEEE Transactions on Signal Processing 45:11, 2719-2730. [CrossRef] 27. Chuanyi Ji, Sheng Ma. 1997. Combinations of weak classifiers. IEEE Transactions on Neural Networks 8:1, 32-42. [CrossRef] 28. X.M. Gao, X.Z. Gao, J.M.A. Tanskanen, S.J. Ovasaka. 1997. Power prediction in mobile communication systems using an optimal neural-network structure. IEEE Transactions on Neural Networks 8:6, 1446-1455. [CrossRef]

29. Klaus Prank, Mirko Kloppstech, Steven Nowlan, Terrence Sejnowski, Georg Brabant. 1996. Random Secretion of Growth Hormone in Humans. Physical Review Letters 77:9, 1909-1911. [CrossRef] 30. Peter M. Williams. 1996. Using Neural Networks to Model Conditional Multivariate DensitiesUsing Neural Networks to Model Conditional Multivariate Densities. Neural Computation 8:4, 843-854. [Abstract] [PDF] [PDF Plus] 31. Lizhong Wu , John Moody . 1996. A Smoothing Regularizer for Feedforward and Recurrent Neural NetworksA Smoothing Regularizer for Feedforward and Recurrent Neural Networks. Neural Computation 8:3, 461-489. [Abstract] [PDF] [PDF Plus] 32. Guozhong An . 1996. The Effects of Adding Noise During Backpropagation Training on a Generalization PerformanceThe Effects of Adding Noise During Backpropagation Training on a Generalization Performance. Neural Computation 8:3, 643-674. [Abstract] [PDF] [PDF Plus] 33. Colin Campbell , C. Perez Vicente . 1995. The Target Switch Algorithm: A Constructive Learning Procedure for Feed-Forward Neural NetworksThe Target Switch Algorithm: A Constructive Learning Procedure for Feed-Forward Neural Networks. Neural Computation 7:6, 1245-1264. [Abstract] [PDF] [PDF Plus] 34. R Naimimohasses, D M Barnett, D A Green, P R Smith. 1995. Measurement Science and Technology 6:9, 1291-1300. [CrossRef] 35. Gerson Lachtermacher, J. David Fuller. 1995. Back propagation in time-series forecasting. Journal of Forecasting 14:4, 381-393. [CrossRef] 36. Mohammad Bahrami. 1995. Issues on representational capabilities of artificial neural networks and their implementation. International Journal of Intelligent Systems 10:6, 571-579. [CrossRef] 37. Hong Pi , Carsten Peterson . 1994. Finding the Embedding Dimension and Variable Dependencies in Time SeriesFinding the Embedding Dimension and Variable Dependencies in Time Series. Neural Computation 6:3, 509-520. [Abstract] [PDF] [PDF Plus] 38. B. Apolloni, G. Ronchini. 1994. Dynamic sizing of multilayer perceptrons. Biological Cybernetics 71:1, 49-63. [CrossRef] 39. Nathan Intrator . 1993. Combining Exploratory Projection Pursuit and Projection Pursuit Regression with Application to Neural NetworksCombining Exploratory Projection Pursuit and Projection Pursuit Regression with Application to Neural Networks. Neural Computation 5:3, 443-455. [Abstract] [PDF] [PDF Plus] 40. Marcus FreanConnectionist Architectures: Optimization . [CrossRef]

Communicated by David MacKay

NOTE

Exact Calculation of the Hessian Matrix for the Multilayer Percep tron Chris Bishop Neural Networks Group, AEA Technology, Harwell Laboratory, Oxfordshire, OX11 ORA, United Kingdom

The elements of the Hessian matrix consist of the second derivatives of the error measure with respect to the weights and thresholds in the network. They are needed in Bayesian estimation of network regularization parameters, for estimation of error bars on the network outputs, for network pruning algorithms, and for fast retraining of the network following a small change in the training data. In this paper we present an extended backpropagation algorithm that allows all elements of the Hessian matrix to be evaluated exactly for a feedforward network of arbitrary topology. Software implementation of the algorithm is straightforward. 1 Introduction

Standard training algorithms for the multilayer perceptron use backpropagation to evaluate the first derivatives of the error function with respect to the weights and thresholds in the network. There are, however, several situations in which it is also of interest to evaluate the second derivatives of the error measure. These derivatives form the elements of the Hessian matrix. Second derivative information has been used to provide a fast procedure for retraining a network following a small change in the training data (Bishop 1991). In this application it is important that all elements of the Hessian matrix be evaluated accurately. Approximations to the Hessian have been used to identify the least significant weights as a basis for network pruning techniques (Le Cun et d.1990), as well as for improving the speed of training algorithms (Becker and Le Cun 1988; Ricotta et al. 1988). The Hessian has also been used by MacKay (1991) for Bayesian estimation of regularization parameters, as well as for calculation of error bars on the network outputs and for assigning probabilities to different network solutions. MacKay found that the approximation scheme of Le Cun et al. (1990) was not sufficiently accurate and therefore included off-diagonal terms in the approximation scheme. Neural Computation

4, 494-501 (1992)


Exact Calculation of the Hessian Matrix

495

In this paper we show that the elements of the Hessian matrix can be evaluated exactly using multiple forward propagation through the network, followed by multiple backward propagation. The resulting algorithm is closely related to a technique for training networks whose error functions contain derivative terms (Bishop 1990). In Section 2 we derive the algorithm for a network of arbitrary feedforward topology, in a form that can readily be implemented in software.. The algorithm simplifies somewhat for a network having a single hidden layer, and this case is described in Section 3. Finally a brief summary is given in Section 4. 2 Evaluation of the Hessian Matrix

Consider a feedforward network in which the activation zi of the ith unit is a nonlinear function of the input to the unit: zi

= f (ai)

in which the input of other units

(2.1) ai

is given by a weighted linear sum of the outputs

where wi, is the synaptic weight from unit j to unit i, and Bi is a bias associated with unit i. Since the bias terms can be considered as weights from an extra unit whose activation is fixed at zk = fl, we can simplify the notation by absorbing the bias terms into the weight matrix, without loss of generality. We wish to find the first and second derivatives of an error function E, which we take to consist of a sum of terms, one for each pattern in the training set, P

where p labels the pattern. The derivatives of E are obtained by summing the derivatives obtained for each pattern separately. To evaluate the elements of the Hessian matrix, we note that the units in a feedforward network can always be arranged in "layers," or levels, for which there are no intralayer connections and no feedback connections. Consider the case in which unit i is in the same layer as unit n, or in a lower layer (i.e., one nearer the input). The remaining terms, in which unit i is above unit n, can be obtained from the symmetry of the Hessian matrix without further calculation. We first write

Chris Bishop

496

where we have made use of equation 2.2. The first equality in equation 2.4 follows from the fact that, as we shall see later, the first derivative only through ai. We now introduce a set of quantities g n depends on w,, defined by (2.5) Note that these are the quantities that are used in standard backpropagation. The appropriate expressions for evaluating them will be obtained shortly. Equation 2.4 then becomes

where again we have used equation 2.2. We next define the quantities

(2.8)

The second derivatives can now be written in the form

where f’(a) denotes d f l d a . The {gri} can be evaluated from a forward propagation equation obtained as follows. Using the chain rule for partial derivatives we have 8ar dat (2.10) g/, = --

c dai

da,

where the sum runs over all units r that send connections to unit 1. (In fact, contributions only arise from units which lie on paths connecting unit i to unit 1.) Using equations 2.1 and 2.2 we then obtain the forward propagation equation (2.11)

The initial conditions for evaluating the {ga} follow from the definition of equation 2.7, and can be stated as follows. For each unit i in the network, (except for input units, for which the corresponding {gli} are not required), set gii = 1 and set gli = 0 for all units I # i, which are in the same layer as unit i or which are in a layer below the layer containing unit i. The remaining elements of gli can then be found by forward propagation using equation 2.11. The number of forward passes needed to evaluate all elements of {gli} will depend on the network topology, but will typically scale like the number of (hidden plus output) units in the network.


497

The quantities {a,} can be obtained from the following backpropagation procedure. Using the definition in equation 2.5, together with the chain rule, we can write (2.12)

where the sum runs over all units Y to which unit n sends connections. Using equations 2.1 and 2.2 then gives (2.13)

This is just the familiar backpropagation equation. Note that the first derivatives of the error function are given by the standard expression

dE,- ajzj -

(2.14)

dWij

which follows from equations 2.2 and 2.5. The initial conditions for evaluation of the {a,} are given, from equations 2.2 and 2.5, by (2.15)

where rn labels an output unit. Similarly, we can derive a generalized backpropagation equation that allows the {bni} to be evaluated. Substituting the backpropagation formula 2.13 for the { a n }into the definition of bni, equation 2.8, we obtain (2.16)

which, using equations 2.7 and 2.8, gives bni = f ” ( a n ) g n i

Cr wrnar + f’(0,) Cr wrvbri

(2.17)

where again the sum runs over all units r to which unit n sends connections. Note that, in a software implementation, the first summation in equation 2.17 will already have been computed in evaluating the {a,} in equation 2.13. The derivative dldui that appears in equation 2.16 arose from the derivative d/aw, in equation 2.4. This transformation, from w, to ui, is valid provided w, does not appear explicitly within the brackets on the right-hand side of equation 2.16. This is always the case, because we considered only units i in the same layer as unit n, or in a lower layer. Thus the weights wrn are always above the weight wij and so the term dw,/awii is always zero.

Chris Bishop

498

The initial conditions for the backpropagation in equation 2.17 follow from equations 2.7, 2.8, and 2.15,

where we have defined (2.19)

Thus, for each unit i (except for the input units), the b,i corresponding to each output unit m are evaluated using equations 2.18 and 2.19, and then the b,i for each remaining unit n (except for the input units, and units n that are in a lower layer than unit i) are found by backpropagation using equation 2.17. Before using the above equations in a software implementation, the appropriate expressions for the derivatives of the activation function should be substituted. For instance, if the activation function is given by the sigmoid: 1

f ( a ) = 1 + exp(-a)

(2.20)

then the first and second derivatives are given by (2.21) For the case of linear output units, we have f ( u ) = u, f‘(u) = 1, and f”(a) = 0, with corresponding simplification of the relevant equations. Similarly, appropriate expressions for the derivatives of the error function with respect to the output unit activations should be substituted into equations 2.15 and 2.19. Thus, for the sum of squares error defined by 1

(2.22)

where t, is the target value for output unit m, the required derivatives of the error become (2.23) Another commonly used error measure is the relative entropy (Solla et ul. 1988) defined by

Ep=C{tm1nzm+(~-tm)In(~-zm)} m

(2.24)


499

The derivatives of Ep take a particularly elegant form when the activation function of the output units is given by the sigmoid of equation 2.20. In this case, we have, from equations 2.15, 2.19, and 2.21,

To summarize, the evaluation of the terms in the Hessian matrix can be broken down into three stages. For each pattern p , the {zn} are calculated by forward propagation using equations 2.1 and 2.2, and the {gli} are obtained by forward propagation using equation 2.11. Next, the {on} are found by backpropagation using equations 2.13 and 2.15, and the {b,i} are found by backpropagation using equations 2.17, 2.18, and 2.19. Finally, the second derivative terms are evaluated using equation 2.9. (If one or both of the weights is a bias, then the correct expression is obtained simply by setting the corresponding activationts) to +1.) These steps are repeated for each pattern in the training set, and the results summed to give the elements of the Hessian matrix. The total number of distinct forward and backward propagations required (per training pattern) scales like the number of (hidden plus output) units in the network, with the number of operations for each propagation scaling like N , where N is the total number of weights in the network. Evaluation of the elements of the Hessian using equation 2.9 requires of order fl operations. Since the number of weights is typically much larger than the number of units, the overall computation will be dominated by the evaluations in equation 2.9. 3 Single Hidden Layer Many applications of feedforward networks make use of an architecture having a single layer of hidden units, with full interconnections between adjacent layers, and no direct connections from input units to output units. Since there is some simplification to the algorithm for such a network, we present here the explicit expressions for the second derivatives. These follow directly from the equations given in Section 2. We shall use indices k and k‘ for units in the input layer, indices I and I’ for units in the hidden layer, and indices m and rn’ for units in the output layer. The Hessian matrix for this network can be considered in three separate blocks as follows. (A) Both weights in the second layer:

(B) Both weights in the first layer:

Chris Bishop

500

(C) One weight in each layer:

(3.3) where H, is defined by equation 2.19. If one or both of the weights is a bias term, then the corresponding expressions are obtained simply by setting the appropriate unit activation(s) to fl.

4 Summary In this paper, we have derived a general algorithm for the exact evaluation of the second derivatives of the error function, for a network having arbitrary feedforward topology. The algorithm involves successive forward and backward propagations through the network, and is expressed in a form that allows for straightforward implementation in software. The number of forward and backward propagations, per training pattern, is at most equal to twice the number of (hidden plus output) units in the network, while the total number of multiplications and additions scales like the square of the number of weights in the network. For networks having a single hidden layer, the algorithm can be expressed in a particularly simple form. Results from a software simulation of this algorithm, applied to the problem of fast network retraining, are described in Bishop (1991).

References Becker, S., and LeCun, Y. 1988. Improving the convergence of back-propagation learning with second order methods. In Proceedings of the Connectionist Models Summer School, D. S. Touretzky, G. E. Hinton, and T. J. Sejnowski, eds., p. 29. Morgan Kaufmann, San Mateo, CA. Bishop, C. M. 1990. Curvature-driven smoothing in feedforward networks. In Proceedings of the lnternational Neural Network Conference, Paris, Vol. 2, p. 749. To be published in I E E E Transactions on Neural Networks. Bishop, C. M. 1991. A fast procedure for re-training the multilayer perceptron. Int. I. Neurat Syst. 2(3), 229-236. LeCun, Y., Denker, J. S., and Solla, S. A. 1990. Optimal brain damage. In Advances in Neural lnformation Processing Systems, Vol. 2, D. S. Touretzky, ed., p. 598. Morgan Kaufmann, San Mateo, CA. MacKay, D. J. C. 1991. A practical Bayesian framework for backprop networks. Neural Comp. 4(3), 448472.


501

Ricotta, L. I?, Ragazzini, S., and Martinelli, G. 1988. Learning of word stress in a suboptimal second order backpropagation neural network. In Proceedings IEEE International Conference on Neural Networks, San Diego, Vol. 1, p. 355. Solla, S. A., Levin, E., and Fleisher, M. 1988. Accelerated learning in layered neural networks. Complex Syst. 2, 625-640.

Received 5 August 1991; accepted 2 October 1991.

This article has been cited by: 1. Krzysztof M. Graczyk, Piotr Płonski, Robert Sulej. 2010. Neural network parameterizations of electromagnetic nucleon form-factors. Journal of High Energy Physics 2010:9. . [CrossRef] 2. Maria P. Cadeddu, David D. Turner, James C. Liljegren. 2009. A Neural Network for Real-Time Retrievals of PWV and LWP From Arctic Millimeter-Wave Ground-Based Observations. IEEE Transactions on Geoscience and Remote Sensing 47:7, 1887-1900. [CrossRef] 3. Shangling Song, Kazuhiko Ohnuma, Zhi Liu, Liangmo Mei, Akira Kawada, Tomoyuki Monma. 2009. Novel biometrics based on nose pore recognition. Optical Engineering 48:5, 057204. [CrossRef] 4. Kar-Ann Toh. 2008. Deterministic Neural ClassificationDeterministic Neural Classification. Neural Computation 20:6, 1565-1595. [Abstract] [PDF] [PDF Plus] 5. D. Erdogmus, O. Fontenla-Romero, J.C. Principe, A. Alonso-Betanzos, E. Castillo. 2005. Linear-Least-Squares Initialization of Multilayer Perceptrons Through Backpropagation of the Desired Response. IEEE Transactions on Neural Networks 16:2, 325-337. [CrossRef] 6. L. M. Raff, M. Malshe, M. Hagan, D. I. Doughan, M. G. Rockley, R. Komanduri. 2005. Ab initio potential-energy surfaces for complex, multichannel systems using modified novelty sampling and feedforward neural networks. The Journal of Chemical Physics 122:8, 084104. [CrossRef] 7. Yoshua Bengio . 2000. Gradient-Based Optimization of HyperparametersGradient-Based Optimization of Hyperparameters. Neural Computation 12:8, 1889-1900. [Abstract] [PDF] [PDF Plus] 8. F. Vivarelli, P. Fariselli, R. Casadio. 1997. The prediction of protein secondary structure with a Cascade Correlation Learning Architecture of neural networks. Neural Computing & Applications 6:1, 57-62. [CrossRef] 9. R. Parisi, E.D. Di Claudio, G. Orlandi, B.D. Rao. 1996. A generalized learning paradigm exploiting the structure of feedforward neural networks. IEEE Transactions on Neural Networks 7:6, 1450-1460. [CrossRef] 10. H.H. Thodberg. 1996. A review of Bayesian neural networks with an application to near infrared spectroscopy. IEEE Transactions on Neural Networks 7:1, 56-72. [CrossRef] 11. Chris M. Bishop . 1995. Training with Noise is Equivalent to Tikhonov RegularizationTraining with Noise is Equivalent to Tikhonov Regularization. Neural Computation 7:1, 108-116. [Abstract] [PDF] [PDF Plus] 12. Barak A. Pearlmutter . 1994. Fast Exact Multiplication by the HessianFast Exact Multiplication by the Hessian. Neural Computation 6:1, 147-160. [Abstract] [PDF] [PDF Plus]

13. Chris M. Bishop. 1994. Neural networks and their applications. Review of Scientific Instruments 65:6, 1803. [CrossRef] 14. David J. C. MacKay . 1992. The Evidence Framework Applied to Classification NetworksThe Evidence Framework Applied to Classification Networks. Neural Computation 4:5, 720-736. [Abstract] [PDF] [PDF Plus]

Communicated by Thomas Brown

NMDA-Based Pattern Discrimination in a Modeled Cortical Neuron Bartlett W. Me1 Computation and Neural Systems Program, Division of Biology, 216-76, California Institute of Technology, Pasadena, C A 91125 U S A

Compartmental simulations of an anatomically characterized cortical pyramidal cell were carried out to study the integrative behavior of a complex dendritic tree. Previous theoretical (Feldman and Ballard 1982; Durbin and Rumelhart 1989; Me1 1990; Me1 and Koch 1990; Poggio and Girosi 1990) and compartmental modeling (Koch et al. 1983; Shepherd et al. 1985; Koch and Poggio 1987; Rall and Segev 1987; Shepherd and Brayton 1987; Shepherd ef al. 1989; Brown e t al. 1991) work had suggested that multiplicative interactions among groups of neighboring synapses could greatly enhance the processing power of a neuron relative to a unit with only a single global firing threshold. This issue was investigated here, with a particular focus on the role of voltagedependent N-methyl-D-asparate (NMDA) channels in the generation of cell responses. First, it was found that when a large proportion of the excitatory synaptic input to dendritic spines is carried by NMDA channels, the pyramidal cell responds preferentially to spatially clustered, rather than random, distributions of activated synapses. Second, based on this mechanism, the NMDA-rich neuron is shown to be capable of solving a nonlinear pattern discrimination task. We propose that manipulation of the spatial ordering of afferent synaptic connections onto the dendritic arbor is a possible biological strategy for pattern information storage during learning. 1 Introduction

The cerebral neocortex and its connections account for almost 90% of the human brain by weight (Hofman 1989). Our understanding of its functions can be significantly advanced by an accurate input-output model for the individual pyramidal cell, the principal and most numerous neocortical cell type. How does an individual cortical pyramidal cell integrate its synaptic inputs over space and time? Which spatiotemporal patterns of synaptic activation in the dendritic tree lead to the firing of action potentials, and which do not? In short, what kind of "device" is the cortical pyramidal cell? Neural Computation

4, 502-517 (1992)

NMDA-Based Discrimination in a Cortical Neuron

503

The response of a pyramidal neuron depends on a large number of variables, such as the detailed anatomy of the dendritic tree, numerous biophysical cell parameters, the number and distribution of synaptic inputs to the dendritic tree and their firing frequencies, and the strengths and time courses of the several excitatory and inhibitory synaptic conductances that are known to exist in these cells. The direct study of the input-output behavior of a dendritic tree under conditions of varying synaptic stimulation is not currently possible in the laboratory for technical reasons. The primary tool currently available for the study of complex dendritic trees is compartmental modeling, used to compute the time course of currents, voltages, and conductances in an electrical circuit model of an arbitrary neuron (Rall 1964; Perkel and Mulloney 1978; Traub 1982; Koch et al. 1983; Shepherd et al. 1985; Koch and Poggio 1987; Rall and Segev 1987; Shepherd and Brayton 1987; Shepherd et al. 1989; Brown et al. 1991). Our initial hypothesis as to the integrative behavior of the neocortical pyramidal cell was derived from a simple abstract neuron type, called a sigma-pi unit (Rumelhart et al. 1986). The sigma-pi unit computes its response as a sum of contributions from a set of multiplicative clusters of inputs, and is in this sense a neural instantiation of a polynomial function. Three factors have recommended this single-unit abstraction as the seed for a biological model. First, direct monosynaptic connections between units with multiplicative input stages can encode a very general class of nonlinear associations (Feldman and Ballard 1982; Giles and Maxwell 1987; Durbin and Rumelhart 1989; Me1 1990; Me1 and Koch 1990; Poggio and Girosi 1990). Second, the learning of such associations can be achieved with a simple Hebbian rule of a general type known to exist in the central nervous system (Bliss and L0mo 1973; Brown et al. 1988; Sejnowski and Tesauro 1989). Third, numerous suggestions have been made over the past decade that voltage-dependent membrane mechanisms in dendritic trees could underlie the multiplicative interactions among neighboring synapses needed for this type of model neuron (Feldman and Ballard 1982; Koch et a2. 1983; Shepherd et al. 1985; Koch and Poggio 1987; Rall and Segev 1987; Shepherd and Brayton 1987; Brown et al. 1988; Shepherd et al. 1989; Durbin and Rumelhart 1989; Me1 1990; Me1 and Koch 1990; Poggio and Girosi 1990; Brown et al. 1991). 2

T h e Biophysical Model

We sought evidence for sigma-pi-like behavior in modeling studies of a serially reconstructed layer 5 pyramidal cell from cat visual cortex (Douglas et al. 1991). The compartmental modeling program, called NEURON (Hines 19891, mapped lengths, diameters, and interconnectivity pattern of the dendritic branches into an equivalent electrical circuit that could be stimulated and probed at will. The dendritic tree shown in Figure 1 was

504

Bartlett W. Me1

Figure 1: Testing for cluster sensitivity. Default membrane parameters were R, = 21,000 Ct-cm2, C, = 1.0 pF/cm2, Ri = 306 0-cm taken from a study of a layer 5 pyramidal cell in rat visual cortex (Stratford et al. 1990). (A-F) ZeroNMDA condition. In A, 80 spines were selected at random, designated by black circles. Each spine contained a synapse with a 0.8 nS AMPA conductance. In this and all other simulation runs, each synapse was stimulated asynchronously at 100 Hz for 100 msec, with a randomized interspike interval. Synaptic voltage traces are shown in B, and voltage at the cell body is shown in C where four spikes were counted. Experiment was repeated in (D-F)with 80 synapses in clusters of size 6 (plus one partial cluster of size 2). All synapses within a cluster normally fell within one 20 pm dendritic compartment. Subsynaptic voltage traces in E are larger on average than those in 8, reflecting the fact that the postsynaptic depolarization i s indeed boosted where synapses are closely grouped. However, each AMPA synapse injects less current postsynaptically in its strongly depolarized state. The response at the soma (F), a measure of total synaptic current, is thus attenuated. Continued facing page.


505

K

H

I

Figure 1: Continued. (G-L) High-NMDA condition. Experimentswere repeated with NMDA peak conductance gN = 0.9 nS, and gA = 0.1 nS. A preference is now seen for clusters of size 6 (J-L) vs. clusters of size 1 (G-I) since the NMDA channels become more effective as they become more strongly depolarized.

thus converted into 903 coupled electrical compartments, one for each 20 pm length of dendrite. Membrane for unmodeled dendritic spines was incorporated into dendritic branches as in Stratford et al. (1990). All default simulation parameters are collected in Table 1. In each simulation run, a set of locations (usually 80, but up to 200) was selected from across the dendritic tree, marking sites to be synaptically stimulated. At each location, an electrical compartment representing a cylindrical spine (1.0 x 0.15 pm) was attached to the associated dendritic compartment. The distal end of each spine contained an

Bartlett W. Me1

506

excitatory synapse with both a fast non-NMDA, or AMPA, conductance, and a slow, voltage-dependent NMDA conductance (Mayer and Westbrook 1987). The voltage-independent AMPA conductance was modeled as an alpha-function (see Stratford et al. 1990): G A = g,Kte-'/' K = e / r , where peak conductance gAvaried between 0.1 to 1 nS, and time to peak was T = 1 msec. The NMDA conductance depended exponentially on membrane voItage V , and time, modeled as follows (Jahr and Stevens 1990a,b; Zador et al. 1990):

with peak conductance gNalso in the range of 0.1 to 1.0 nS, 71 = 40 msec, 7 2 = 0.33 msec, 17 = 0.33/mM, [Mg2+]= 1 mM, and y = 0.06/mV. The long NMDA time constant leads to the possibility of conductance summation across spikes in a high-frequency train; in most simulations, the NMDA Figure 2: Facing page. (A) Ten randomly generated synaptic layouts were run for each cluster size between 1 and 9 with only AMPA channels; summary plot shows predicted monotonic decline in cell response as a function of increasing cluster size. (B) In case of high dendritic NMDA, response of cell shows initially increasing, nonmonotonic dependence on cluster size. Extensive exploration of the biophysical parameter space indicated that the observed NMDA-based cluster sensitivity phenomenon is extremely robust. Parameters varied in plots C-F above are indicated in the upper left corner of each plot. Complete list of parameter variations (usualIy taken one or a few at a time) included membrane resistivity from 10,000 to 100,000 R-cm2 (C), membrane capacitance from 0.9 to 2.0 pF/cm2, cytoplasmic resistivity from 100 to 362 R cm (several configurations were taken from Stratford et al. 1990); synaptic conductance waveforms were changed from saturating to non-saturating curves (D), spine membrane correction was used or not, count of activated synapses was varied from 80 to 200 (E). Synaptic strengths were normally uniform across the entire tree; in one run (F), peak synaptic conductances were set proportional to local dendritic input resistance, leading to a factor of 30 variation in conductance from the finest to the thickest dendritic branches. Spine neck resistance was also increased in one run from 173 MR 6-fold to 1038 MR, which led to a significant reduction, but did not abolish, the cluster sensitivity (i.e. there persisted a 50% increase in average cell response as clusters grew to from size 1 to size 3). NMDA-based cluster sensitivity was also seen for both apical and basal dendrites individually, though absolute difference between clustered and unclustered activation patterns as measured at a passive soma was significantly larger for basal (40%) than apical (18%)dendrites. Finally, in a pilot study on a smaller layer 3 pyramidal cell (Douglas et al. 1991), cluster sensitivity was seen at spike train frequencies of 20 and 100 Hz, or when only a single spike was delivered to each synapse.


507

conductance was assumed to saturate to gN,corresponding to the case of many presynaptic transmitter molecules and few receptor sites. In one case (Fig. 2D), NMDA conductance was allowed to summate across presynaptic spikes, and could thus achieve values much larger than gN. In most experiments, all of the AMPA conductances were assigned the same peak value, as were all of the NMDA conductances, independent of location on the dendritic tree. The NMDA channel is normally discussed for its role in synaptic plasticity (see Collingridge and Bliss 1987; Brown ef al. 1988). For present 3

A

ID mns

default parameters

1.5

M r

m D. 0.5

0.0

0

2

4

6

8

1

0

0

22

4 4

6

6 8

81

0

Cluster Size

Cluster Size C

non-saturating synapses

M

75

m R

m a 05

,a111

0.0 0

2

4

6

8

1

0

0

2

Cluster Size

4

6

8

Cluster Size

E 2.5-

conductances scaled to input resistance (factor of= variation)

2.0

m 1.0

, 0

2

4

6

0 8

Cluster Size

1

. 0

0 0

2

4

6

Cluster Size

8

I

Bartlett W. Me1

508

Table 1: Default Simulation Parameters.

Rm

Cm

Ri Input resistance (soma) Time constant (soma)

21, OOO R-cm2 1.0 pF/cm2 306 (2-cm 51 MR 19 msec

Resting potential -70 mV Dendritic spines I 1.0 pm, D = 0.15 pm, neck resistance = 173 MR Synaptic stimulation 100 Hz with randomized interspike interval 80, stimulated asynchronously Active synapses gA= 0.1-1.0 nS AMPA conductance AMPA time constant T = 1 msec NMDA conductance gN = 0.1 to 1.0 nS NMDA time constants TI = 40 msec, ~2 = 0.33 msec NMDA Parameters 77 = 0.33/mM, [Mg2+1 = 1 mM, and y = 0.06/mV Synaptic reversal potential 0 mV (both AMPA and NMDA) Compartments 903 plus 1 per activated dendritic spine Compartment length 20 pm Integration time step 0.1 msec

purposes, however, the crucial difference between AMPA and NMDA channels relates to their dependence on voltage. Briefly, the synaptic current injected through an AMPA channel declines steadily as the membrane in which it sits is depolarized from rest. The depolarizing effect of one AMPA synapse thus reduces the effectivenessof neighboring AMPA synapses, and vice versa. In contrast, NMDA current increases steadily as the membrane is depolarized from -70 to -30 mV (Mayer and Westbrook 1987), such that NMDA synapses can be more effective when activated in groups. Except for NMDA channels, the dendritic membrane was electrically passive. The soma contained Hodgkin-Huxley-style sodium and potassium channels in addition to a passive leak conductance. At the beginning of each run, the entire cell was set to a resting potential of -70 mV. A 100-Hz spike train was then delivered asynchronously to each of the preselected synapses, and the response at the soma was "recorded over a 100-msec period. The primary measure of cell response was the number of spikes generated at the soma during the first 100 msec of synaptic stimulation, but passive-soma response measures were also used with similar results, including peak somatic potential, and time integral of somatic potential. Our main hypothesis held that if sufficiently endowed with NMDA channels, the cell would respond selectively to patterns of stimulation in which the activated synapses were clustered., rather than uniformly distributed, across the dendritic tree. We first examined sensitivity to


509

cluster size in a dendritic tree with only AMPA channels (Fig. 1A-F). Sample runs are shown for clusters of size 1 (A,B,C) and 6 (D,E,F). As expected, clusters of 6 led to higher average subsynaptic depolarization (E vs. B), and hence less current injection per synapse. In consequence, the cell response was significantly reduced (F vs. C). This experiment was repeated with 90% of the total peak conductance assigned to NMDA channels.' In contrast to the pure AMPA case, a larger response was seen at the cell body when synapses were distributed in clusters of 6 0, K, L), as compared to clusters of 1 (G, H, I). Note that "clustering" here refers to the spatial configuration of activated synapses, and not an underlying inhomogeneity in the anatomical distribution of dendritic spines. To confirm these effects statistically, 10 random layouts were generated for each cluster size from 1 to 9, for both zero-NMDA and highNMDA conditions. Figure 2A shows the predicted monotonic decline in cell response as a function of increasing cluster size when only AMPA channels were present. In contrast, Figure 2B shows for the high-NMDA case an initially increasing, nonmonotonic dependence on cluster size. The initial increase in cell response as clusters grow larger is explained by the average increase in membrane potential within each cluster, which leads to increased average effectiveness of the NMDA synapses. The subsequent fall-off of the response at larger cluster sizes is due to the diminishing driving potential as the membrane under each cluster saturates to the synaptic reversal potential. The specific cluster size at which the somatic response peaked, in this case 5, was not fixed, but typically varied over a range of 5 to 9 depending on the value of peak conductance assigned to the activated synapses. This basic effect of nonmonotonic cluster sensitivity was observed to hold under a very wide range of biophysical parameter manipulations (Fig. 2C-F). We conclude that a cortical pyramidal cell with a major proportion of its excitatory synaptic current carried by NMDA channels could respond preferentially to clustered distributions of activated synapses. In a pilot study on a smaller layer 2-3 pyramidal cell, a continuous increase in cluster sensitivity was seen as the proportion of peak NMDA conductance [i.e., g,/(g, +g,)] was increased from 0 to 100%. In the current study, it was observed that a small proportion of AMPA conductance was useful in rapidly boosting the local membrane potential, and hence neighboring NMDA channels, into the voltage range where the latter begin to open significantIy. The actual proportion of NMDA conductance in cortical pyramidal cells has not yet been determined, though its contribution to 'Note that due to its dependence on voltage, the actual NMDA conductance achieved during synaptic stimulation will not normally approach the nominal maximum value EN.For example, with a 90% proportion of NMDA peak conductance, that is, gN/(gN+ gA)= 0.9, the peak EPSC in response to a synchronous single-shock activation of 100 randomly placed excitatory synapses for the cell and parameters of Fig.1 is only 3 W % due to current through NMDA channels. However, at least 85% of the total synaptic charge in this case is camed by the NMDA conductance, since its time constant is relatively much longer.

510

Bartlett W. Me1

normal synaptic activation in these cells is known to be large (Miller et al. 1989; Fox et al. 1989; see also Keller et al. 1991 with respect to hippocampal granule cells). 3 A Basis for Nonlinear Pattern Discrimination

Regarding the biological utility of dendritic cluster sensitivity, we observe that any mechanism that allows a cell to distinguish certain patterns of stimulation from others can act as a basis for memory. For example, if the subset of afferents activated in stimulus condition A terminate in a clustered configuration on the dendrite of our pyramidal cell, then pattern A will fire the cell with high probability, and will thus be “preferred.” Another pattern, B, for which no such clustering condition holds, will with high probability fail to fire the cell. To establish a first empirical lower bound o n the nonlinear pattern discrimination capacity of the single pyramidal cell, we sought to load a set of pattern “preferences” onto the dendritic tree in just this way. One hundred gray-level images of natural scenes were used as a convenient set of complex high-dimensional patterns. A simple visual preprocessing Figure 3 Facing page. Gray-level images of natural scenes were used as a convenient set of complex high-dimensional training and test patterns. Each image was mapped onto a sparse binary pattern of activity over a set of orientationselective visual filters. Four filters were centered at each of the 64 by 64 image pixel locations, sensitive to horizontal, vertical, and right- and left-oblique intensity edges, giving a total population of 16,384 orientation-selective visual ”units.” A global thresholding operation guaranteed that exactly 80 visual units were active in response to any presented image. Each visual unit gave rise to an afferent “axon” that could provide input to the dendritic tree. A training image was “loaded by arranging that the 80 afferents activated by the image terminated in a clustered configuration on the dendritic tree that had been previously observed to fire the cell.- After 50 images were loaded in this way, at most 4000 afferents made a synapse onto the dendrite-fewer in actuality due to feature overlap among training images. All remaining uncommitted visual afferents (> 12,384) were randomly mapped onto the dendrite as well, in order that any presented image should drive exactly 80 synapses in the dendritic tree. This resulted in a total of 16,384 synaptic ”contacts” onto the dendritic surface, at a maximum density of one spine per micrometer of dendritic length. While a biologically unrealistic number of synapses for this single cell (5000 more probable), this control measure guaranteed that any observed preference for trained vs. control images in the experiments of Figure 4 was the result of the pattern of synaptic clustering, and not a result of total synaptic activation. Note that the response properties of the orientation-selective visual units and the pattern of afferentation described here are not intended as models of cortical visual processing; any other type of (nonvisual) patterns and preprocessing stage might have been used.


511

step was used to map each image onto a sparse binary pattern of “neural” activity, consisting of 80 active lines within a bundle of 16,384 orientationselective visual “axons” (Fig. 3). Another source of patterns or another type of preprocessing could have been used for these experiments; the visual modality was chosen only for convenience, and was not intended as a model of any aspect of cortical visual processing. Fifty images were chosen to act as a training set, 50 as a control set. For the first training image, a layout of 80 synapses in clusters of 8 was generated that produced at least one spike at the soma during 100 msec of 100 Hz stimulation (about 50% of clustered layouts of this kind generated at least one spike). The 80 orientation-selectivevisual afferents driven by the training image were then randomly mapped onto the 80 dendritic locations specified by the precomputed clustered layout (Fig. 3). The process was repeated for each training image in turn,except that a visual afferent, once mapped onto the dendritic tree, was never remapped in the

80 features activated by each image

16,384 oriented feature detectors

- \

t / image \

512

Bartlett W. Me1

processing of a subsequent training image. When h i s loading procedure was complete, all remaining uncommitted visual afferents were randomly mapped to the dendrite as well, so that any image, trained or untrained, activated exactly 80 synapses of the same strength and with the same intensity. Five different image types were then presented to the cell, including intact training images, three types of corrupted training images, and control images. Average cell responses to each during 100 msec are shown in Fig. 4. The most prominent feature of these results is the cell's overwhelming selectivity for the 50 trained vs. 50 control images. Only 1 in 100 control images presented to the cell in two complete runs generated a single spike. In contrast, 87%of the trained images generated at least one spike, with an average spike count of 1.25. Intermediate responses were seen for the three categories of corrupted training images. It is important to note that the procedure used for "storing" training patterns in the dendritic tree was designed to make the task of discriminating training vs. control patterns as difficult as possible for a linear neuron with only a global thresholding nonlinearity (e.g., a perceptron), since every presented pattern activated the same number of synaptic inputs, with the same synaptic strength. The much greater probability of firing an action potential in response to a trained (clustered) vs. control (diffuse) pattern is thus evidence that the dendritic tree is a more powerful pattern discriminator than a simple perceptron. Significantly reduced cell responses to linear superpositions of 2 training patterns is further evidence for this conclusion (Fig. 4). We may relate the standard measure of pattern discrimination capacity used for the simple perceptron to the present situation. Cover (1965) proved that an N-input perceptron has a capacity of 2N patterns, counting both training and control patterns (see Hertz et al. 1991). This result means that for 2N randomly chosen patterns, some randomly designated as training patterns, others as controls, the probability that a perceptron can perfectly discriminate all training from control patterns is 1/2. This simple result cannot be applied in the present situation, however, as we consider here instead the ability of the neuron to discriminate k training patterns (in this case 50) from all other possible patterns acting as controls. In this more demanding case, we do not require classification performance to be perfect, but rather that it fall within an acceptable error tolerance. For example, in the present nonoptimized experiment with 50 stored training images, the probability of misclassification was 14%(13% false negatives, 1 % false positives). 4 Conclusions

The single statistical parameter, cluster size, has proved to be a remarkably good predictor of responses to complex patterns of synaptic in-

513


Decline in Cell Response as Training Patterns are Degraded

1.o

0.5

0.0

Training Pamerns

50% 20% Conuption Corruption

Test Patterns

Figure 4: Average cell responses to five types of presented images, combined over two runs with 50 training and control images interchanged. Largest responses were to whole training images and smallest to control images. In three intermediate cases, progressively attenuated responses were seen to (1) random 50/50 composites of features taken from two training images, (2) training images with 20%, and (3) 50% of their visual features randomly replaced. Pictorial representation of each category is only suggestive of these manipulations; corruptions to training images were actually carried out within the internal orientation-selective visual representation. The 50/50 composite case provides direct evidence that the dendritic tree is capable of nonlinear discrimination, since a thresholded linear neuron would respond equivalently to any such superposition of training patterns.

514

Bartlett W. Me1

put in an NMDA-rich neuron. Further, a simple experiment has shown how the ability to respond selectively to statistically inhomogeneous distributions of activated synapses could act as a basis for nonlinear pattern discrimination within a dendritic tree. It must be noted, however, that other membrane mechanisms may exist in pyramidal dendrites in addition to NMDA channels that could mimic, enhance, alter, or abolish the cluster-based integrative behavior observed here. Nonetheless, the fact that NMDA channels can by themselves underlie a capacity for nonlinear pattern discrimination in dendritic trees is, in and of itself, useful knowledge. Beyond the widely accepted proposition that modifiable synaptic weights are the likely sites for neural information storage, these results highlight the probable importance of synaptic ordering in the generation of cortical cell responses as well. Given the rich combinatorics that are possible in choosing spatial arrangements of lo4 synapses in a dendritic tree, the true discrimination capacity of the single pyramidal cell is likely to be much greater than the level that has been empirically demonstrated here. In this vein, preliminary experiments have been conducted on an abstract cluster-sensitiveneuron with 10,000synapses in which the directloading procedure described above was replaced by a Hebb-type learning rule. Much larger discrimination capacities were easily achieved in this way for the abstract neuron (unpublished observations); application of a similar learning scheme within the present biophysical modeling context awaits further work (but see Brown et al. 1991). Elsewhere (Me1 1990; Me1 and Koch 19901, it has been proposed that the three-dimensional geometry of intersecting axonal and dendritic arbors in the neocortical column are ideally suited to allow a given axon to establish synaptic contacts at many different dendritic loci as dictated by the constraints of a learning task. It must also be noted that the pyramidal cell is not alone in the cortex, but is a member of a group of cells within the cortical microcolumnar structure (Jones 1981; Peters and Kara 1987) within which a high degree of interaction is likely to occur. Within such a network of neurons, however, the issue of storage capacity is profound, and will need to be addressed in the future both analytically and through more sophisticated simulations. Acknowledgments We are greatly indebted to Rodney Doulas and Kevan Martin for providing us with anatomical data from two cortical pyramidal cells, to John Moore and Michael Hines for their compartmental modeling program, NEURON, and to Tom Tromey for creating a 3-D graphical interface to NEURON. Thanks to qvind Bernander, Christof Koch, Ken Miller, Ernst Niebur, and Stephanie Me1 for useful discussions and many helpful comments on the manuscript. This work was supported by grants from the


515

the National Institutes of Mental Health, the Office of Naval Research, and the James S. McDonnell Foundation.

References Bliss, T. V. P., and L0m0, T. 1973. Long-lasting potentiation of synaptic transmission in the dentate area of the anaesthetized rabbit following stimulation of the perforant path. J. Physiol., 232, 331-356. Brown, T. H., Chang, V. C., Ganong, A. H., Keenan, C. L., and Kelso, S. R. 1988. Biophysical properties of dendrites and spines that may control the induction and expression of long-term synaptic potentiation. In Long-Term Potentiation: From Biophysics to Behavior, 201-264. Alan R. Liss, New York. Brown, T. H., Mainen, Z. F., Zador, A. M., and Claiborne, B. J. 1991. Selforganization of hebbian synapses in hippocampal neurons. In Advances in Neural lnformation Processing Systems, Vol. 3, R. Lippmann, J. Moody, and D. Touretzky, eds., pp. 3945. Morgan Kaufmann, Palo Alto. Collingridge, G. L., and Bliss, T. V. P. 1987. NMDA receptors: Their role in long-term potentiation. Trends Neurocsi. 10, 288-293. Cover, T. M. 1965. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE Trans. Elect. Comput. 14,326-334. Douglas, R. J., Martin, K. A. C., and Whitteridge, D. 1991. An intracellular analysis of the visual responses of neurons in cat visual cort. J. Physiol. 440, 659-696. Durbin, R., and Rumelhart, D. E. 1989. Product units: A computationally powerful and biologically plausible extension to backpropagation networks. Neural Cornp. 1, 133. Feldman, J. A., and Ballard, D. H. 1982. Connectionist models and their properties. Cog. Sci. 6, 205-254. Fox, K., Sato, H., and Daw, N. W. 1989. The location and function of NMDA receptors in cat and kitten visual cortex. 1.Neurosci. 9, 2443-2454. Giles, C. L., and Maxwell, T. 1987. Learning, invariance, and generalization in high-order neural networks. Appl. Opt. 26(23), 4972-4978. Hertz, J.,Krogh, A., and Palmer, R. G. 1991. Introduction to the Theory of Neural Computation. Addison-Wesley, Redwood City, CA. Hines, M. 1989. A program for simulation of nerve equations with branching geometries. lnt. J. Biomed. Comput. 24, 55-68. Hofman, M. A. 1989. On the evolution and geometry of the brain in mammals. Prog. Neurobiol. 32, 137-158. Jahr, C. E., and Stevens, C. F. 1990a. A quantitative description of NMDA receptor-channel kinetic behavior. J. Neuroscience 10, 1830-1837. Jahr, C. E., and Stevens, C. F. 1990b. Voltage dependence of NMDA-activated macroscopic conductances predicted by single-channel kinetics. J. Neurosci. 10, 3176-3182. Jones, E. G. 1981. Anatomy of cerebral cortex: Columnar input-output relations.

516

Bartlett W. Me1

In The Organization of CerebralCortex, F. 0.Schmitt, F. G. Worden, G. Adelman, and S. G. Dennis, eds. MIT Press, Cambridge, MA. Keller, B. U., Konnerth, A., and Yaari, Y.1991. Patch clamp analysis of excitatory synaptic currents in granule cells of rat hippocampus. J. Physiol. 435, 275293. Koch, C., and Poggio, T. 1987. Biophysics of computation: Neurons, synapses, and membranes. In Synaptic Function, G. E. Edelman, W. F. Gall, and W. M. Cowan, eds., pp. 637-698. Wiley, New York. Koch, C., Poggio, T., and Torre, V. 1983. Nonlinear interactions in a dendritic tree: Localization, timing, and role in information processing. Proc. Natl. Acad. Sci. U.S.A. 80, 2799-2802. Mayer, M. L., and Westbrook, G. L. 1987. The physiology of excitatory amino acids in the vertebrate central nervous system. Prog. Neurobiol. 28, 197-276. Mel, B. W. 1990. The sigma-pi column: A model for associative learning in cerebral neocortex. CNS Memo #6, Computation and Neural Systems Program, California Institute of Technology. Mel, B. W., and Koch, C. 1990. Sigma-pi learning: On radial basis functions and cortical associative learning. In Advances in AJeuralInformation Processing Systems, Vol. 2, D. S. Touretzsky, ed. Morgan Kaufmann, San Mateo, CA. Miller, K. D., Chapman, B., and Stryker, M. P. 1989a. Visual responses in adult cat visual cortex depend on N-methyl-D-aspartate receptors. Proc. Natl. Acad. Sci. U.S.A. 86, 5183-5187. Perkel, D.H., and Mulloney, 8.1978. Electrotonic properties of neurons: steadystate compartmental model. J. Neurophysiol. 41, 621-639. Peters, A., and Kara, D. A. 1987. The neuronal composition of area 17 of rat visual cortex. IV. The organization of pyramidal cells. J. Comp. Neurol. 260, 573-590. Poggio, T., and Girosi, F. 1990. Regularization algorithms for learning that are equivalent to multilayer networks. Science 247, 978-982. RalI, W. 1964. Theoretical significance of dendritic trees for neuronal inputouput relations. In Neural Theoy and Modeling, R. F. Reiss, ed., pp. 73-97. Stanford University Press, Stanford, CA. Rall, W., and Segev, I. 1987. Functional possibilities for synapses on dendrites and on dendritic spines. In SynapticFunction, G. E. Edelman, W. F. Gall, and W. M. Cowan, eds., pp. 605-636. Wiley, New York. Rumelhart, D. E., Hinton, G. E., and McClelland, J. L. 1986. A general framework for parallel distributed processing. In Parallel Distributed Processing: Explorations in theMicrostructureof Cognition, Vol. 1, D. E. Rumelhart, J. L. McClelland, eds., pp. 45-76. Bradford, Cambridge, MA. Sejnowski, T. J., and Tesauro, G. 1989. The Hebb rule for synaptic plasticity: Algorithms and implementations. In Neural Models of Plasticity, J. H. Byrne and W. Berry, eds., pp. 94-103. Academic Press, Cambridge, MA. Shepherd, G. M., and Brayton, R. K. 1987. Logic operations are properties of computer-simulated interactions between excitable dendritic spines. Neurosci. 21, 151-166. Shepherd, G. M., Brayton, R. K., Miller, J. P., Segev, I., Rinzel, J., and Rall, W. 1985. Signal enhancement in distal cortical dendrites by means of in-


517

teractions between active dendritic spines. Proc. Natl. Acad. Sci. U.S.A. 82, 2192-2195. Shepherd, G. M., Woolf, T. B., and Carnevale, N. T. 1989. Comparisons between active properties of distal dendritic branches and spines: Implications for neuronal computations. Cog. Neurosci. 1, 273-286. Stratford, K., Mason, A., Larkman, A., Major, G., and Jack, J. 1990. The modeling of pyramidal neurons in visual cortex. In The Computing Neuron, R. Durbin, C. Miall, and G. Mitchison, eds., pp. 296-321. Addison-Wesley, Wokingham, England. Traub, R. D. 1982. Simulation of intrinsic bursting in CA3 hippocampal neurons. Neurosci. 7, 1233-1242. Zador, A., Koch, C., and Brown, T. H. 1990. Biophysical model of a hebbian synapse. Proc. Natl. Acad. Sci. U.S.A. 87, 671845721.

Received 25 July 1991; accepted 26 November 1991.

This article has been cited by: 2. Eduardo Mizraji, Juan Lin. 2010. Logic in a Dynamic Brain. Bulletin of Mathematical Biology . [CrossRef] 3. Yingxue Wang, Shih-Chii Liu. 2010. Multilayer Processing of Spatiotemporal Spike Patterns in a Neuron with Active DendritesMultilayer Processing of Spatiotemporal Spike Patterns in a Neuron with Active Dendrites. Neural Computation 22:8, 2086-2112. [Abstract] [Full Text] [PDF] [PDF Plus] 4. Paul A. Rhodes. 2008. Recoding Patterns of Sensory Input: Higher-Order Features and the Function of Nonlinear Dendritic TreesRecoding Patterns of Sensory Input: Higher-Order Features and the Function of Nonlinear Dendritic Trees. Neural Computation 20:8, 2000-2036. [Abstract] [PDF] [PDF Plus] 5. Michael Schmitt . 2002. On the Complexity of Computing and Learning with Multiplicative Neural NetworksOn the Complexity of Computing and Learning with Multiplicative Neural Networks. Neural Computation 14:2, 241-301. [Abstract] [PDF] [PDF Plus] 6. C. Rasche, R.J. Douglas. 2001. Forward- and backpropagation in a silicon dendrite. IEEE Transactions on Neural Networks 12:2, 386-393. [CrossRef] 7. Panayiota Poirazi , Bartlett W. Mel . 2000. Choice and Value Flexibility Jointly Contribute to the Capacity of a Subsampled Quadratic ClassifierChoice and Value Flexibility Jointly Contribute to the Capacity of a Subsampled Quadratic Classifier. Neural Computation 12:5, 1189-1205. [Abstract] [PDF] [PDF Plus] 8. P. C. Bressloff, S. Coombes. 2000. Solitary Waves in a Model of Dendritic Cable with Active Spines. SIAM Journal on Applied Mathematics 61:2, 432. [CrossRef] 9. J. Köhn , F. Wörgötter . 1998. Employing the Z-Transform to Optimize the Calculation of the Synaptic Conductance of NMDA and Other Synaptic Channels in Network SimulationsEmploying the Z-Transform to Optimize the Calculation of the Synaptic Conductance of NMDA and Other Synaptic Channels in Network Simulations. Neural Computation 10:7, 1639-1651. [Abstract] [PDF] [PDF Plus] 10. Barak A. Pearlmutter . 1995. Time-Skew Hebb Rule in a Nonisopotential NeuronTime-Skew Hebb Rule in a Nonisopotential Neuron. Neural Computation 7:4, 706-712. [Abstract] [PDF] [PDF Plus] 11. Federico Girosi , Michael Jones , Tomaso Poggio . 1995. Regularization Theory and Neural Networks ArchitecturesRegularization Theory and Neural Networks Architectures. Neural Computation 7:2, 219-269. [Abstract] [PDF] [PDF Plus] 12. Christopher J. Coomber. 1995. Compartmental modelling with artificial neural networks. Neural Processing Letters 2:1, 13-18. [CrossRef] 13. Bartlett W. Mel . 1994. Information Processing in Dendritic TreesInformation Processing in Dendritic Trees. Neural Computation 6:6, 1031-1085. [Abstract] [PDF] [PDF Plus]

14. Erik De Schutter , James M. Bower . 1993. Sensitivity of Synaptic Plasticity to the Ca2+ Permeability of NMDA Channels: A Model of Long-Term Potentiation in Hippocampal NeuronsSensitivity of Synaptic Plasticity to the Ca2+ Permeability of NMDA Channels: A Model of Long-Term Potentiation in Hippocampal Neurons. Neural Computation 5:5, 681-694. [Abstract] [PDF] [PDF Plus] 15. Barry Horwitz, Olaf Sporns. 1993. Neural modeling and functional neuroimaging. Human Brain Mapping 1:4, 269-283. [CrossRef]

Communicated by Wilfrid Rall

The Impact of Parallel Fiber Background Activity on the Cable Properties of Cerebellar Purkinje Cells Moshe Rapp Yosef Yarom Idan Segev Department of Neurobiology, Institute of Life Sciences, The Hebrew University, Jerusalem, 92904, Israel

Neurons in the mammalian CNS receive 104-105 synaptic inputs onto their dendritic tree. Each of these inputs may fire spontaneously at a rate of a few spikes per second. Consequently, the cell is bombarded by several hundred synapses in each and every millisecond. An extreme example is the cerebellar Purkinje cell (PC) receiving approximately 100,000 excitatory synapses from the parallel fibers (p.f.s) onto dendritic spines covering the thin dendritic branchlets. What is the effect of the p.f.s activity on the integrative capabilities of the PC? This question is explored theoretically using analytical cable models as well as compartmental models of a morphologically and physiologically characterized PC from the guinea pig cerebellum. The input of individual p.f.s was moaeled as a transient conductance change, peaking at 0.4 nS with a rise time of 0.3 msec and a reversal potential of f60 mV relative to rest. We found that already at a firing frequency of a few spikes per second the membrane conductance is several times larger than the membrane conductance in the absence of synaptic activity. As a result, the cable properties of the PC significantly change; the most sensitive parameters are the system time constant (70)and the steady-state attenuation factor from dendritic terminal to soma. The implication is that the cable properties of central neurons in freely behaving animals are different from those measured in slice preparation or in anesthetized animals, where most of the synaptic inputs are inactive. We conclude that, because of the large conductance increase produced by the background activity of the p.f.s, the activity of the PC will be altered from this background level either when the p.f.s change their firing frequency for a period of several tens of milliseconds or when a large population of the p.f.s fires during a narrow time window. Neural Computation 4, 518-533 (1992) @ 1992 Massachusetts Institute of Technology

Modulation of Cable Properties

519

1 Introduction The processing of synaptic information by nerve cells is a function of the morphology and the electrical properties of the membrane and cytoplasm, as well as the site and properties of the synaptic inputs that impinge onto the dendritic tree. The theoretical studies of W. Rall (1959, 1964, 1967, 1977) have shown that in a passive tree the relevant parameters that determine this processing are the input resistance of the cell (RN), the average electrotonic length of the dendrites (LA"), the system time constant ( T ~ ) ,the rise time (tpeak) and magnitude of the synaptic conductance change, as well as the reversal potential (Esyn) of the synaptic current. Rall's studies, followed by the study of Jack and Redman (1971a), suggested experimental methods for estimating these parameters. Consequently, several neuron types were characterized based on these parameters (e.g., Jack and Redman 1971b; Barrett and Crill 1974; Brown et al. 1981; Stratford et al. 1989; Nitzan et al. 1990). These studies were performed either on anesthetized animals or on slice preparations where most of the background synaptic activity is absent. However, neurons in the CNS are part of a large network and, therefore, each neuron receives thousands of synaptic inputs which may have ongoing activity of a few spikes per second. Such a massive input will generate a sustained conductance change which will modify the integrative capabilities of the neuron (Holmes and Woody 1989). The present study explores the effect of this conductance change on the integrative properties of the cell, using detailed cable and compartmental models of a morphologically and physiologically characterized Purkinje cell (PC) from the guinea pig cerebellum (Rapp 1990; Segev et al. 1991). This cell has a large and complex dendritic tree (Fig. 1) with several hundred spiny branchlets (Fig. 2) that are densely studded with spines. Each of these spines receives an excitatory (asymmetrical) synaptic input from the parallel fiber (p.f.) system, summing to a total of 100,000 p.f.s impinging on a single PC. In such a large system, even a low spontaneous activity rate of each of the p.f.s will generate a high frequency of synaptic input to the PC. The consequences of such an input for the input/output properties of the PC are discussed. N

2 Model

Simulations were performed using cable and compartmental models of the cell shown in Figure 1. The cable properties of the same cell were estimated from intra-somatic recordings performed in the presence of 2.5 mM Csf ions that abolish the marked subthreshold membrane rectification of these cells (Crape1 and Penit-Soria 1986; Rapp 1990; Segev et al. 1991). The process of utilizing the model to estimate the specific membrane resistivity (R,) and capacitance (CJ, and the specific

520

M. Rapp, Y. Yarom, and I. Segev

Figure 1: The modeled Purkinje cell. This cell was reconstructed following an intracellular injection of horseradish peroxidase. The cable parameters of the cell were also characterized in the presence of 2.5 mM Cs+ ions to abolish the marked rectification that exists near the resting potential. The input resistance ( R N )was 12.9 MR and the system time constant (TOO) was 46 msec. Assuming a total spine number of 100,000, each with an area of 1 pm2, the soma-dendritic surface area sums to 149,500 pm2. The optimal matching between the morphology and the cable properties of the cell implies that Rm is 440 R.cm2 at the soma and 110,000 R . cm2 at the dendrites, Cm is 1.64 pF/cm2 and Xi is 250 R . cm. Figure 2: Facing page. Sholl diagram, in morphological units, of the cell shown in Figure 1. This cell was represented by 1500 cylindrical segments and a spherical soma. Red lines denote the spiny branchlets that are studded with approximately 10 spines per 1 pm dendritic length. Each of these spines receives an excitatory input from a parallel fiber. These branchlets consist of 85% of the dendritic area (without the spines). The cell consists of 473 dendritic terminals, some of which terminate less than 100 pm away from the soma whereas others extend to a distance close to 400 pm from the soma. Because the parallel fibers impinge only on the spiny branchlets, these branchlets are electrically "stretched" when the parallel fibers are active.


521

resistivity of the cytoplasm (RJ is elaborated in Segev et al. (1989) and Nitzan et al. (1990). For the PC under study the optimal match between the cell morphology and electrical measurements was obtained with R, = 440 R . cm2 at the soma and 110,000 R . cmz at the dendrites, a uniform

I

0

100

200

300 (Pm>


522

C, of 1.64 pF/cm2 and R; of 250 R . cm. These values yielded an input resistance, RN, of 12.9 MR and a system time constant, TO, of 46 msec, both matching the corresponding experimental values. Morphological measurements of the cell in Figure 1 showed that the total length of dendritic branchlets bearing spines is 10,230 pm. These branchlets are marked by the red lines of Figure 2, which demonstrates Sholl diagram of the cell in Figure 1. The density of spines on the spiny branchlets has been reported to range between 10 and 14 per 1 pm dendritic length (Harris and Stevens 1988a,b); we estimated the total number of spines in the modeled tree to be 100,000, each with an area of 1 pm2 (Harris and Stevens 1988a,b). The total soma-dendritic surface area (including spines) of the modeled neuron was 149,500 pm2. In most of the simulations spines were incorporated into the model globally, rather than by representing each spine as a separate segment, since the latter would be computationally very slow. It can be shown that, for plausible values of R,, Ri, and C m , when current flows from the dendrite into the spine, the membrane area of the spine can be incorporated into the membrane of the parent dendrite (Fig. 3A). The reason for this is that for this direction of current flow, the spine base and the spine head membranes are essentially isopotential (Jack et al. 1975; Rall and Rinzel 1973; Segev and Rall 1988). This approximation is valid when , the average cable length of the cable parameters such as RN, T ~ and dendrites, La,, are estimated from the model. It does not hold when the voltage response of the p.f. input impinging onto the spine head is of interest. In this case spines receiving synaptic input were represented in full (Fig. 38). To incorporate the spines globally into the model we modified the method suggested by Holmes (19891. In this method the original dimensions of the parent dendrite are preserved and the membrane area of the spines is effectively incorporated into the dendritic membrane by changing the values of R, and C,. When both spines and parent dendrite have the same specific membrane properties (R, and C,), the transformed values are:

Rh

= R,

fF

and

Ch

= C,F

and (2.1) where areadendis the area of the parent dendrite without the spines and areaspinesis the membrane area of the spines at that dendritic segment. Note that this transformation preserves the time constant of the original spiny dendrite. The present study focuses on the effect of the conductance change induced by the p.f. input on the cable properties of the PC. In this case the effective membrane resistivity at the spine heads receiving the input (RmqadSpinH) is reduced as compared to the membrane resistivity of


523

A When measuringcable propenies ('To, R,.

R AF), the membrane area of both passive and synaptically activated spines IS Incorporatedinto the parent dendrite membrane

... ,o.*,

B

.1

I I 1,

............................ ......................................... I I

1 1 , , , , , , 1 1 1 1 1 1 , 1 , , , , , , 1 1

m e n measuring synaptic potential. spines receiving synaptic input remain "unincorporated

Figure 3: Schematic representation of the method used to globally incorporate dendritic spines into the membrane of the parent dendrite. (A) The case where the current flows from the dendrite into the spines (arrows in left schematic). (B) The case where current flows from the spines into the dendrite. In (A), equation 2.2 is first used to calculate an effective R, value for the whole segment (middle schematic); then equation 2.1 is utilized for calculating new specific values for the membrane of the parent branch to effectively incorporate the spine area into the parent dendrite area. In (B),spines receiving synaptic input remain unincorporated, whereas passive spines are incorporated into the parent dendrite membrane as in (A).

the nonactivated spines and of the parent segment Assuming that the input can be approximated by a steady-state conductance change (see below), the effective time constant of the spiny segment (dendrite plus spines) is between the time constant of the dendritic membrane . C), and that of the activated spines membrane (Rm,actspinesC,). Now the first step in utilizing equation 2.1 is to find a single R; value for the whole dendritic segment (parent dendrite plus all spines) such that (R; . C), approximates the effective time constant of the original (nonhomogeneous) segment. Our calculations show that for an electrically short spiny segment, R; is the reciprocal of the sum of the specific +


524

conductances of the two membrane types, weighted by their relative area: 1

.:=I/[(&)(-)+(

areatotal

Rrn,actspines

) (areaactspines)] areatotal

(2.2)

where arearestis the membrane area of the parent dendrite and of the nonactivated spines, areaactspines is the area of the activated spines, and areatotal= (area,,, areaactspines) (see also Fleshman et al. 1988). Having a single Rh value as calculated in equation 2.2 for the whole segment (Fig. 3A, middle panel), equation 2.1 can then be used (with R; replacing R,) for incorporating the spines area into the parent dendrite area (Fig. 3A, right panel). Transient (compartmental) models were studied using SPICE (Segev et al. 1985, 1989). These models were utilized to simulate the voltage response of the parallel fiber input. To facilitate the computations, only a representative group of 200 randomly selected spines were modeled individually. In each run a different group of 200 such spines was selected. The rest of the spines were globally incorporated into the parent branchlets membrane as discussed above. Each of the representative spines was simulated by an R-C circuit, modeling the passive membrane properties of the spine, and an additional synaptic path consisting of a transient conductance change, gsyn(t),in series with a synaptic battery, Esyn. This compartment was connected to the parent dendrite by an axial resistance, representing the longitudinal resistivity of the spine neck (Segev and Rall 1988). The representative spines were activated synchronously at a high frequency, w, such that w = N .0/200 where N is the number of p.f.s and 6 is the original (low) firing frequency of the p.f.s. For example, if each of the N = 100,000 p.f.s is activated at 0 = 2 Hz (200,000 inputs/sec), then each of the 200 representative spines is activated once every msec (w = 1000 Hz), thus preserving the total number of inputs per unit time. The validity of this approximation was examined by increasing the number of representative spines to 400 and decreasing w by a factor of 2. These changes resulted in only minor differences in both the cable properties and the membrane voltage produced at the modeled cell. Also, the input resistance, RN, as calculated using SPICE matched the analytical results of the cable models (see below). These control tests have led us to conclude that for the range of input frequencies tested, because of the large number of inputs impinging on the tree, the results of the present study are essentially independent of the exact timing of the input and/or the location of the activated spines, provided that the total conductance change per unit time is preserved. Intuitively, a dendritic tree which is bombarded by a massive transient synaptic input must experience approximately a steady conductance change (Rall 1962). This effective conductance change, gsteady, can then be utilized in steady-state (segmental) cable models to compute analytically the impact of the synaptic activity on the cable properties of the cell.

+


525

For each transient synaptic input, gsyn(f),activated at a frequency 8, the effective steady conductance change is

1

W

gsteady

6

(2.3)

gsyn(f)dt

When 200 representative spines were used, w replaced 8 in equation 2.3. In that case, each of these spines has an effective R, value, (2.4) where areaspineis the membrane area of the spine and grest= (areaspine/ R,,rest) is the resting conductance of the spine membrane (without the synaptic input). This Rm,actspinesvalue, over an area, areaactspines = 200 pm2 (200 spines), was utilized in equation 2.2 to calculate the effective R; value of the spiny branchlets, with areatotalbeing the total membrane area of ull spiny brunchlets plus spines. In this way the total synaptic conductance was equally distributed over the whole membrane surface of the spiny branchlets. In our computations, gsyn(f)was modeled with an ”alpha function,” g s y n ( t ) = gmax(f/tpeak)exp{l

-

(t/tpeak))

(2.5)

where gmaX is the peak synaptic conductance change and tpeak is the rise time. For the alpha function the integral in equation 2.3 is

1

00

gsyn(f)d t = grnaxtpeak exp(1)

(2.6)

With the values used in the present study @,ax = 0.4 nS; fpe& = 0.3 msec), this integral is 0.33 nS.msec. For 6 = 2 Hz (w = 1000 Hz) we get from equation 2.3 that gsteady = 0.33 nS. Thus, with a spine area of 1 pm2 and R,,,,,, of 110,000 R . cm2, equation 2.4 implies that the effective membrane resistivity of each of the activated spines is as small as 30 R . cm2. Utilizing equation 2.2 with this value (with arearest= 144,538 pm2 and areaactspines = 200 lm2), the effective R; at the spiny brunchlets was 18,140 R . cm2 (rather than 110,000 R . cm2),suggesting that the activation of the p.f.s have a marked effect on the cable properties of the PC. Having R, and Ri values for the tree, the algorithm developed by Rall (1959) was implemented to analytically calculate the effect of p.f. activation on the soma input resistance (RN),the average input resistance at the terminals (RT), the average cable length of the dendrites (Lon),and the average voltage attenuation factor from the dendritic tips into the soma AFT+^); see also Nitzan et al. (1990). The modifications needed to include nonuniform R, distribution is given in Fleshman et ul. (1988). The system time constant ( T ~was ) computed by “peeling” the voltage response to a brief current pulse produced by the corresponding compartmental model (Rall 1969).

526


The values for the unitary synaptic conductance change are based on the patch-clamp studies by Hirano and Hagiwara (1988) and Llano et al. (1991). The value for g,, was estimated from the minimal peak of the current produced by the p.f. input whereas the value for tpeak was estimated from the experimental rise time (measured at 22°C) corrected to 37"C, assuming a Qlo of 3. 3 Results

The voltage responses of the modeled PC to the p.f. input are shown in Figure 4 for an input frequency (0) of 0.5 Hz (Fig. 4A) and 5 Hz (Fig. 4B). In each of these panels the continuous lines depict the voltage response at an arbitrarily chosen spine head (top trace), at the base of this spine (middle trace), and at the soma (lower trace). The bottom panels (C and D) show the corresponding input into each of the 200 representative spines (w = 250 Hz in Fig. 4C and 2500 Hz in Fig. 4D). Dashed lines in A and B are the voltage responses at the corresponding sites when a steady state conductance change (gsteady), as defined in equation 2.3, is imposed on each of the 200 representative spines. This steady input shown by the dashed line in the corresponding lower panels is 0.08 nS in Figure 4C and 0.8 nS in Figure 4D (in the latter case the dashed line is masked by the heavy line). Unlike the case of relatively low input frequency (Fig. 4C) where each transient input is seen individually, at high frequency (Fig. 4D) the temporal summation of individual inputs resulted in the heavy line. The figure demonstrates that already at a low firing rate of 0.5 Hz (Fig. 4A) the p.f. input produces a peak depolarization of approximately 15 mV at the spine head, 12 mV at the spine base, and 8 mV at the soma. At 5 Hz (Fig. 481, the maximal depolarization at the spine head is 45 mV, 42 mV at the spine base, and 30 mV at the soma. An important difference in the voltage produced by the two input frequencies is in the rate of the voltage buildup. At a low frequency (Fig. 4A), the depolarization reaches a steady-state value after more than 100 msec whereas at high frequency (Fig. 4B) the steady-state value is reached after approximately 50 msec (see Discussion). Figure 4 also shows that the steady-state approximation (dashed lines) faithfully reproduces the voltage response of the cell already at the low firing rate of 0.5 Hz (Fig. 4A). The agreement between the results of the transient input and the steady-state input implies that, indeed, the depolarization along the tree is essentially a function of w . g,, . fpeak (equations 2.3 and 2.5). Thus, w,g,,,, and t p a k are interchangeable. Another point to note is that, unlike the case of a localized synaptic input where a significant voltage attenuation is expected, when a distributed input is activated the depolarization along the soma-dendritic surface is rather uniform (compare the voltage at the spine head and at the soma).


527

B

A

hI time (msec)

time (msec)

D

C

0.2

1

O.O

A

0.4

1

10.8 - j

0.3

h

h

rn

v) 0.8

5 0 . 2

C

3M 0.4

3

0.1

0.0

time (msec)

I

50

75

id0

time (msec)

Figure 4: Depolarization produced by background activity of the parallel fibers. (A,C), 8 = 0.5 Hz; (B,C), 8 = 5 Hz. Bottom frames show the conductance input into each of the 200 representative spines modeled in full. Each of these spines receives transient conductance input at a frequency w = 8 (100,000/200) = 5008; individual inputs are clearly seen at the low firing rate (C).At the higher frequency (D) the transient synaptic conductance changes are summed-up to produce the heavy line. Dashed line in (C) and (D) [masked by the heavy line in (D)] shows the corresponding steady-state conductance change as calculated from equation 2.3. Upper panels (A and B) show the resultant depolarization at the head of one of the activated spines (upper curve), at its base (middle curve) and at the soma (lower curve). Dashed lines show the results when a steady-state conductance change is used. Note the excellent agreement between the results using transient inputs and the results obtained with a steady-state conductance change.

How d o the different cable properties of the cell and the soma depolarization depend on the frequency of the p.f. input? Figure 5A shows


528

B

A 0.4

1

*

3 4 frequency (Hz)

, . o h

I

5 1

D

C 151

o

?

h r ! 9 4 5 frequency (Hz)

ohfrequency (Hz)

Figure 5: The cable properties of the PC critically depend on the input frequency (8) of the parallel fibers. The graphs show this dependence on 8 for (A) the depolarization produced at the PC soma, (B)the average electrotonic length of PC dendrites, (C) the soma input resistance, and (D), the system line constant. The points in (A) were calculated using SPICE; the points in ( E D ) were calculated analytically. These changes will influence the efficacy of the p.f. input. that the depolarization at the soma changes steeply at low frequencies

and tends to saturate at higher frequencies. The saturated value of somatic depolarization induced by the p.f. input was found to be $Esyn, namely 40 mV for the parameters chosen in the present study. The effect of the p.f. input frequency on the average cable length (LJ of the dendritic tree is shown in Figure 5B. Due to the high R, value at rest (at 0 = 0 Hz) L,, is 0.13. The tree is electrically "stretched" into L,, = 0.31 (a factor of 2.4) at 5 Hz. At this range of frequencies the input resistance at the soma (Fig. 5C) is decreased from 12.9 MS1 at


529

0 Hz to 6.5 MR at 5 Hz (50%),whereas the average input resistance at the terminal tips (RT)is reduced by only 20% (from 104 to 83 MR, not shown). The system time constant (Fig. 5D) is the most sensitive parameter; it is reduced from 46 to 12.1 msec (a factor of 3.8, see also Rall 1962). This implies that, at 5 Hz, the effective membrane conductance is four times larger than the membrane conductance at 0 Hz. Another parameter that was calculated (not shown) is the average steady state attenuation factor (AFT-5) from the terminal tips to the soma. This parameter is increased from 8.8 (at 0 Hz) to 28 (at 5 Hz), a factor of 3.2. For the explicit relation between RT and AFT,s (see Rall and Rinzel 1973). We conclude that the background activity of the p.f.s significantly changes the cable properties of the PC dendrites with the following relative sensitivity: ro > AFT,^ > Lo, > RN > RT. The somatic depolarization resulting from p.f.s activation rises steeply as a function of the input frequency. Already at relatively low firing frequency of 5 Hz it reaches 3/4 of the maximal depolarization that the p.f.s can produce at the Pc's soma (Fig. 5A). This depolarization develops rather smoothly with a rate that increases as the input frequency increases.

4 Discussion

The present study demonstrates that the background activity of the parallel fibers has a dramatic effect on the functional properties of cerebellar Purkinje cells. Already at a low firing rate of a few Hz, the membrane conductance of the PC significantly increases. As a result, both the system time constant, 70, and the input resistance, RN, decrease by several fold, whereas the electronic length, L,,, and the voltage attenuation factor, AFT,^ (not shown) increase (Fig. 5E!-D). This background activity is expected to significantly depolarize the PC (Fig. 4A & B and Fig. 5A). The effect of the background activity on the cable properties of the cell strongly depends on the time-integral of the synaptic conductance change and on the frequency of the background activity (equation 2.3). The same general results hold also for other neurons from the mammalian CNS receiving a large number of synaptic contacts, each of which may be activated spontaneously at a frequency of a few spikes/sec. Indeed, similar conclusions have been recently reached by Bernander et al. (1991) who modeled the effect of background activity on the cable properties of a reconstructed layer V pyramidal cell in the visual cortex. We therefore suggest that the effective cable properties and the "resting potential" of these neurons in the behaving animal are different from those measured in the slice preparation or in anesthetized animals (Holmes and Woody 1989; Abbott 1991; Amit and Tsodyks 1992). The results of the present study have several interesting implications for the integrative capabilities of central neurons. The massive background activity (and the corresponding increase in the membrane con-

530


ductance) implies that single p.f. inputs essentially loose their functional meaning and only the activation of a large number of p.f.s will significantly displace the membrane potential. It should be noted, however, that other more efficient individual inputs (which may also contact a different dendritic region and do not participate in the background activity) can have a marked effect on the input/output characteristics of the PC. An example is the powerful climbing fiber input which forms as many as 200 synaptic contacts on the soma and main dendrite of the Purkinje cell. When activated, the whole dendritic tree and soma are strongly depolarized; this produces a short burst of high-frequency firing at the PC axon (Llinas and Walton 1990). As demonstrated in Figure 5A, the soma depolarization (the excitatory synaptic current reaching the soma) is a nonlinear function of the input firing rate. The higher the frequency of the background activity, the larger the number of excitatory synapses that need to be recruited to depolarize the cell by a given amount. This is in contrast to the presently used neuron network models, where the current input into a modeled "cell" is assumed to be linearly related to the input firing rate. The saturation of the soma depolarization at relatively low firing rates (Fig. 5A) implies a narrow dynamical range for the detection of changes in p.f. input frequency. Figure 4A and B show that for any given input frequency, several tens of milliseconds are required for the voltage to reach a steady-state value. Therefore, changes in the frequency of the p.f. input will be "detected only if the change lasts for a relatively long period. Furthermore, because of the change in 70, the time course of the voltage change is a function of the input frequency. The higher the input frequency, the faster the build-up of the potential toward its steady value. Hence, at higher background frequencies, more synapses are required to shift the membrane potential by a given amount, but the time course of this shift is faster. For example, suppose that the frequency of the background activity of the p.f.s is 1 Hz (100 synapses/msuc); the resulting depolarization (relative to 0 Hz) is 13 mV (Fig. 4A). Increasing the frequency to 2 Hz (an additional 100 synapses/msec) will further depolarize the soma by 8 mV, provided that the frequency change lasts at least 50 msec (about 2.5 times T~ corresponding to 2 Hz; Fig. 4D). These dynamic alternations in the depolarization of the PC soma will modulate the cell firing rate. Inhibitory inputs onto the PC originate primarily from the stellate cells (dendritic inhibition) and from the basket cells (mainly somatic inhibition). Since our model has a very leaky soma (low somatic Rm), the basket cell input is essentially built-in to the model. The number of stellate cells that contact a single PC is much smaller than the number of p.f.s. It is expected, therefore, that the background activity onto the PC will be dominated by the activity of the p.f.s. Our preliminary simulations suggest that the inhibition induced by the stellate cells can act only locally,


531

at a given dendritic region, and have only a minor effect on the somatic membrane potential produced by the p.f. activity. Finally, it has been clearly shown that the PC's dendrites are endowed with a variety of voltage-sensitive channels (LlinAs and Sugimori 1980a,b). In response to synaptic inputs, these channels produce both subthreshold nonlinearities as well as full blown dendritic spikes. The effect of these nonlinearities on the results of the present report will be explored in a future study.

Acknowledgments This work was supported by a grant from the Office of Naval Research and a grant from the Israeli Academy of Sciences. We thank our colleague Shaul Hochstein for critical comments on this work.

References Abbott, L. F. 1991. Realistic synaptic input for model neural networks. Network (in press). Amit, D. J., and Tsodyks, M. V. (1992). Effective neurons and attractor neural networks in cortical environment. Network (in press). Barrett, J. N., and Crill, W. E. 1974. Specific membrane properties of cat motoneurones. J. Physiol. (London) 293,301-324. Bernander, O., Douglas, R. J., Martin, K. A. C., and Koch, C. 1992. Synaptic background activity determines spatio-temporal integration in single pyramidal cells. Proc. Natl. Acad. Sci. U S A . 88, 11569-11573. Brown, T. H., Fricke, R. A., and Perkel, D. H. 1981. Passive electrical constants in three classes of hippocampal neurons. J. Neurophysiol. 46, 812-827. Crapel, F., and Penit-Soria, J. 1986. Inward rectification and low threshold calcium conductance in rat cerebellar Purkinje cells. J. Physiol. (London) 372, 1-23. Fleshman, J. W., Segev, I., and Burke, R. E. 1988. Electrotonic architecture of type-identified Q motoneurons in the cat spinal cord. J. Neurophysiol. 60, 60-85. Harris, K. M., and Stevens, J. K. 1988a. Dendritic spines of rat cerebellar Purkinje cells: Serial electron microscopy with reference to their biophysical characteristics. J. Neurosci. 12, 455469. Harris, K. M., and Stevens, J. K. 198813. Study of dendritic spines by serial electron microscopy and three-dimensional reconstruction. Neurol. Neurobioi. 37, 179-199. Hirano, T., and Hagiwara, S. 1988. Synaptic transmission between rat cerebellar granule and Purkinje cells in dissociated cell culture: Effects of excitatoryamino acid transmitter antagonists. Proc. Natl. Acad. Sci. U.S.A. 85, 934-938. Holmes, W. R. 1989. The role of dendritic diameter in maximizing the effectiveness of synaptic inputs. Brain Res. 478, 127-137.

532


Holmes, W. R., and Woody, C. D. 1989. Effect of uniform and non-uniform synaptic "activation-distribution" on the cable properties of modeled cortical pyramidal cells. Brain Res. 505, 12-22. Jack, J. J. B., Noble, D., and Tsien, R. W. 1975. Electrical Curretit Flow in Excitable Cells. Oxford University Press, UK. Jack, J. J. B., and Redman, S. J. 1971a. The propagation of transient potentials in some linear cable structures. J. Physiol. (London) 215, 283-320. Jack, J. J. B., and Redman, S. J. 1971b. An electrical description of the motoneuron, and its application to the analysis of synaptic potentials. J. Physiol. (Lottdon) 215, 321-352. Llano, I., Marty, A., Armstrong, C. M., and Konnerth, A. 1991. Synaptic and agonist-induced excitatory current of Purkinje cells in rat cerebellar slices. J. Physiol. (London) 431, 183-213. Llinb, R. R., and Sugimori, M. 1980a. Electrophysiological properties of in vitro Purkinje cell somata in mammalian cerebellar slices. J. Physiol. (London) 305, 171-195. Llinbs, R. R., and Sugimori, M. 1980b. Electrophysiological properties of in vitro Purkinje cell dendrites in mammalian cerebellar slices. J. Physiol. (London) 305, 197-213. LlinAs, R. R., and Walton, K. D. 1990. Cerebellum. In The Synaptic Organization of the Brain, G. M. Shepherd, ed., pp. 214-245. Oxford University Press, Oxford. Nitzan, R., Segev, I., and Yarom, Y. 1990. Voltage behavior along irregular dendritic structure of morphologically and physiologically characterized vagal motoneurones in the guinea pig. J. Neurophysiol. 63, 333-346. Rall, W. 1959. Branching dendritic trees and motoneuron membrane resistivity. Exp. Neurol. 1, 491-527. Rall, W. 1962. Theory of physiological properties of dendrites. Ann. NY Acad. Sci. 96, 1071-1092. Rall, W. 1964. Theoretical significance of dendritic trees for neuronal inputoutput relations. In Neural Theory and Modeling, R. Reiss, ed., pp. 73-97. Stanford University Press, Stanford. Rall, W. 1967. Distinguishing theoretical synaptic potentials computed for different soma-dendritic distributions of synaptic input. J. Neurophysiol. 30, 1138-11 68. Rall, W. 1969. Time constants and electronic length of membrane cylinders in neurons. Biophys. J . 9, 1483-1168. Rall, W. 1977. Core conductor theory and cable properties of neurons. In Handbook of Physiology, Vol. 1, Pt. 1. The Nervous System, E. R. Kandel, ed., pp. 39-97. America1 Physiology Society, Bethesda, MD. Rall, W., and Rinzel, J. 1973. Branch input resistance and steady attenuation for input to one branch of a dendritic neuron model. Biophys. J. 13, 648-688. Rapp, M. 1990. The passive cable properties and the effect of dendritic spines on the integrative properties of Purkinje cells. M.Sc. Thesis, submitted to the Hebrew University, Jerusalem. Segev, I., and Rall, W. 1988. Computational study of an excitable dendritic spine. J. Neurophysiol. 60, 499-523.


533

Segev, I., Fleshman, J. W., Miller, J. I?, and Bunow, B. 1985. Modeling the electrical properties of anatomically complex neurons using a network analysis program: Passive membrane. Biol. Cyber. 53, 27-40. Segev, I., Fleshman, J. W., and Burke, R. E. 1989. Compartmental models of complex neurons. In Methods in Neuronal Modeling: From Synapses to Networks, C. Koch and I. Segev, eds., pp. 171-194. Bradford Books, Cambridge, MA. Segev, I., Rapp, M., Manor, Y., and Yarom, Y. 1992. Analog and digital processing in single nerve cells: Dendritic integration and axonal propagation. In Single Neuron Computation, T. McKenna, J. Davis, and S. F. Zornetzer, eds. Academic Press, Orlando, FL (in press). Stratford, K., Mason, A., Larkman, A., Major, G., and Jack, J. 1989. The modelling of pyramidal neurons in the visual cortex. In The Computing Neuron, R. Durbin, C. Miall, and G. Mitchison, eds., pp. 296-321. Addison-Wesley, Wokingham, England.

Received 23 July 1991; accepted 12 December 1991.

This article has been cited by: 2. Mark E. Nelson. 2010. Electrophysiological models of neural processing. Wiley Interdisciplinary Reviews: Systems Biology and Medicine n/a-n/a. [CrossRef] 3. Zhijun Yang, Matthias H. Hennig, Michael Postlethwaite, Ian D. Forsythe, Bruce P. Graham. 2009. Wide-Band Information Transmission at the Calyx of HeldWide-Band Information Transmission at the Calyx of Held. Neural Computation 21:4, 991-1017. [Abstract] [Full Text] [PDF] [PDF Plus] 4. I. B. Kulagina. 2009. Impact of Structural Characteristics of Reconstructed Motoneurons on Their Excitability (a Simulation Study). Neurophysiology 41:2, 116-121. [CrossRef] 5. M. Ozer, L. J. Graham. 2008. Impact of network activity on noise delayed spiking for a Hodgkin-Huxley model. The European Physical Journal B 61:4, 499-503. [CrossRef] 6. Arvind Kumar, Sven Schrader, Ad Aertsen, Stefan Rotter. 2008. The High-Conductance State of Cortical NetworksThe High-Conductance State of Cortical Networks. Neural Computation 20:1, 1-43. [Abstract] [PDF] [PDF Plus] 7. M. Rudolph , A. Destexhe . 2003. Characterization of Subthreshold Voltage Fluctuations in Neuronal MembranesCharacterization of Subthreshold Voltage Fluctuations in Neuronal Membranes. Neural Computation 15:11, 2577-2618. [Abstract] [PDF] [PDF Plus] 8. David M. Halliday . 2000. Weak, Stochastic Temporal Correlation of Large-Scale Synaptic Input Is a Major Determinant of Neuronal BandwidthWeak, Stochastic Temporal Correlation of Large-Scale Synaptic Input Is a Major Determinant of Neuronal Bandwidth. Neural Computation 12:3, 693-707. [Abstract] [PDF] [PDF Plus] 9. Amit Manwani , Christof Koch . 1999. Detecting and Estimating Signals in Noisy Cable Structures, I: Neuronal Noise SourcesDetecting and Estimating Signals in Noisy Cable Structures, I: Neuronal Noise Sources. Neural Computation 11:8, 1797-1829. [Abstract] [PDF] [PDF Plus] 10. Richard Kempter, Wulfram Gerstner, J. van Hemmen. 1999. Hebbian learning and spiking neurons. Physical Review E 59:4, 4498-4514. [CrossRef] 11. Richard Kempter , Wulfram Gerstner , J. Leo van Hemmen , Hermann Wagner . 1998. Extracting Oscillations: Neuronal Coincidence Detection with Noisy Periodic Spike InputExtracting Oscillations: Neuronal Coincidence Detection with Noisy Periodic Spike Input. Neural Computation 10:8, 1987-2017. [Abstract] [PDF] [PDF Plus] 12. Gary R. Holt, Christof Koch. 1997. Shunting Inhibition Does Not Have a Divisive Effect on Firing Rates*Shunting Inhibition Does Not Have a Divisive

Effect on Firing Rates*. Neural Computation 9:5, 1001-1013. [Abstract] [PDF] [PDF Plus] 13. G. L. Yuen, P. E. Hockberger, J. C. Houk. 1995. Bistability in cerebellar Purkinje cell dendrites modelled with high-threshold calcium and delayed-rectifier potassium channels. Biological Cybernetics 73:4, 375-388. [CrossRef]

14. P C Bressloff. 1995. Journal of Physics A: Mathematical and General 28:9, 2457-2469. [CrossRef] 15. Paul Bressloff. 1995. Spatiotemporal processing in neural networks with random synaptic background activity. Physical Review E 51:5, 5064-5073. [CrossRef] 16. Bartlett W. Mel . 1994. Information Processing in Dendritic TreesInformation Processing in Dendritic Trees. Neural Computation 6:6, 1031-1085. [Abstract] [PDF] [PDF Plus] 17. J. D. Evans, G. C. Kember. 1994. Analytical solutions to the multicylinder somatic shunt cable model for passive neurones with differing dendritic electrical parameters. Biological Cybernetics 71:6, 547-557. [CrossRef] 18. Andrew A. V. Hill, Donald H. Edwards, Rodney K. Murphey. 1994. The effect of neuronal growth on synaptic integration. Journal of Computational Neuroscience 1:3, 239-254. [CrossRef] 19. L. F. Abbott. 1994. Decoding neuronal firing and modelling neural networks. Quarterly Reviews of Biophysics 27:03, 291. [CrossRef] 20. P C Bressloff. 1994. Journal of Physics A: Mathematical and General 27:12, 4097-4113. [CrossRef] 21. Mark E. Nelson . 1994. A Mechanism for Neuronal Gain Control by Descending PathwaysA Mechanism for Neuronal Gain Control by Descending Pathways. Neural Computation 6:2, 242-254. [Abstract] [PDF] [PDF Plus] 22. Paul Bressloff, John Taylor. 1993. Spatiotemporal pattern processing in a compartmental-model neuron. Physical Review E 47:4, 2899-2912. [CrossRef]

Communicated by Lany Abbott

Activity Patterns of a Slow Synapse Network Predicted by Explicitly Averaging Spike Dynamics John Rinzel Mathematical Research Branch, National Institute of Diabetes and Digestive and Kidney Diseases, National Institutes of Health, Befhesda,M D 20892 U S A

Paul Frankel Division of Applied Mathematics, Brown University, Providence, RI 02912 USA and Mathematical Research Branch, National Institute of Diabetes and Digestive and Kidney Diseases, National Institutes ofHealth, Bethesda, M D 20892 USA

When postsynaptic conductance varies slowly compared to the spike generation process, a straightforward averaging scheme can be used to reduce the system’s complexity. Our model consists of a HodgkinHuxley-like membrane description for each cell; synaptic activation is described by first order kinetics, with slow rates, in which the equilibrium activation is a sigmoidal function of the presynaptic voltage. Our work concentrates on a two-cell network and it applies qualitatively to the activity patterns, including bistable behavior, recently observed in simple in vitro circuits with slow synapses (Kleinfeld e t al. 1990). The fact that our averaged system is derived from a realistic biophysical model has important consequences. In particular, it can preserve certain hysteresis behavior near threshold that is not represented in a simple ad hoc sigmoidal input-output network. This behavior enables a coupled pair of cells, one excitatory and one inhibitory, to generate an alternating burst rhythm even though neither cell has fatiguing properties. 1 Introduction

When modeling the dynamic activity of cell ensembles descriptions are often used that do not account for action potential generation (e.g., Hopfield and Tank 1986).Cell activity might be represented by instantaneous spike frequency, or as mean (time-averaged) membrane potential. Although one assumes that a justifiable procedure for averaging over the spikes can be carried out explicitly, activity equations, based on sigmoidal Neural Computation

4, 534-545 (1992)

An Averaged Model for a Slow Synapse Network

535

input-output relations, are usually presented ad hoc without direct connection to the biophysical properties of neurons. Here we consider a situation in which averaging can be performed systematically starting from the bottom up. In various neurons postsynaptic responses are much slower than spikes (Adams ef al. 1986; Kleinfeld ef al. 1990; Syed et al. 1990). We exploit these differing time scales in order to average explicitly the action potential dynamics and thereby derive a simplified model that involves only the slowly varying synaptic activation variables. We apply our model and use phase plane methods to describe nonlinear behavior seen in recent experiments with two-cell in vitro invertebrate networks (Kleinfeld et al. 1990). For example, neurons that are mutually inhibitory can exhibit bistable behavior in which one cell is firing and the other not, or vice versa (Section 3). Switching between these two modes can be induced with current stimuli that are not too brief. Our model cells, like the real neurons (Kleinfeld et al. 1990), exhibit sustained repetitive firing but not endogenous bursting. Yet we find, surprisingly, that an inhibitory cell and an excitatory cell may interact via slow synapses in order to generate an oscillatory burst pattern (Section 4). In our case, this rhythm occurs because an isolated cell’s input-output (I/O) relation can exhibit hysteresis: for a certain range of input intensity, the cell membrane can either fire repetitively or be stationary, as can the standard Hodgkin-Huxley (HH) model (Rinzel 1978). The alternating burst pattern does not depend on a fatiguing or postinhibitory rebound mechanism (Perkel and Mulloney 1974), nor on synaptic self-feedback (Wilson and Cowan 1972). It would not be predicted with an ad hoc model employing sigmoidal 1/0 relations. 2 Averaging an HH Model with Slow Synapses

We consider a two-cell system (each cell may synapse on the other but not on itself) whose model equations are:

it,

= - ~ , o n ( ~ 1H ,I ) - 811 S]l

m(vd &I

=

4,

=

G

@I)

-

(vl- V q m ) + 1,

n,

A, km(v,)- sl11,

i, j = 1 , 2 (i # j )

(2.1) (2.2) (2.3)

The first two equations (2.1-2.2) for each cell constitute an HH-like model for spike behavior; I , o n ( n~ ), = g N a m3, (v) (v - vNa) + g K n4 (v - vK) + gL (v - DL) contains the voltage-activated inward (say, sodium) and outward (potassium) currents and the leakage current. Sodium activation, rn, is assumed so fast that it achieves steady state instantaneously: m = rn,(v) = l/{l+exp[(Om-v)/km]}.For potassium activation, n, the voltagedependent ”steady-state” and “time constant” functions are given, respectively, by n,(v) = 1/{1+exp[(O, -v)/k,]} and 7, (v) = 7,/cosh[(0.2v)/0.3]. The postsynaptic activation, sIl, in cell j specifies the fraction

536

John Rinzel and Paul Frankel

of maximal conductance, gi,; it obeys first order kinetics in which the steady-state level, s,(D) = 1/{1+exp[2(B,, -v)/ksm]}, depends on presynaptic voltage of cell i. The equations and variables are dimensionless. Voltages have been divided by the sodium reversal potential, V N ~ .An effective membrane time constant, T,, is defined by using a typical conductance, C,. Time is scaled relative to r,,,, and all conductances are relative to G,. The applied current Ii is dimensionless after dividing by C,VN,. Parameter values are listed in the legend to Figure 1. Our model of synaptic activation dynamics is highly idealized, and it does not distinguish whether the rate-limiting step is in the postsynaptic channel kinetics, in transmitter binding or in handling processes in the cleft, or in the release mechanism. Nevertheless, results obtained with the simplified description agree qualitatively with exploratory simulations using more detailed models. Here, by "slow" synapses we mean that rs is large relative to 1 and 7., Thus, since s,, changes slowly compared to D and n we proceed as follows. First, we determine the I/O properties of each spike-generating subsystem with s,, treated as a parameter. Then, in equation 2.3, we replace sm(ui) by its average, S,, over the spike activity of cell i (which depends on sji). This average is defined by (2.4)

where v,(t;sj,)is the oscillating time course for a periodic solution (period T ) to equations 2.1-2.2 with slI held constant. For a time-independent, , ) , value of steady-state solution of equations 2.1-2.2, D,,,, = D ~ , ~ ~ ( s , the Srn(sll)equals s ~ ( D ~ , which ~ ~ ) , satisfies equation 2.4 trivially. With the above replacement (equation 2.4) we obtain the reduced twovariable system for the averaged synaptic activations, s,, which approximate the exact quantities s,,:

Only recently have averaging procedures been applied to biomathematical models in which the slow variables feedback to influence the fast dynamics; a rigorous basis is just beginning to emerge (Dvofik and SiSka 1989; Pernarowski 1990; Rinzel and Lee 1986). To illustrate the effect of slow synaptic dynamics, we consider a case of unidirectional inhibitory coupling (Fig. 1): gz1 =: 0, uSyn= DK. When the silent cell #1 is stimulated s12 increases slowly and eventually inhibits the follower cell #2. After the stimulus to cell #1is turned off, cell #2 resumes firing, with some delay, when 512 decreases sufficiently. Interpretation of the on-off transitions is based on the 1 / 0 relation for cell # 2 (see legend). In this relation, S,(s12) measures the steady response of # 2 for the input s12 treated as a parameter (lower panel of Fig. 1). Numerical integration of equations 2.1-2.3 was carried out using the software package PHASEPLANE (Ermentrout 1990); the Adams-Bashforth method was used with At = 0.1. To evaluate the functions S,(sij)

537


Vl

312

061 0.0

O.*

0.0

1, 0

,

100

fi

200

30U

400

I

500

Time

O0 00 00

0 0I 7

0 ~ 0,8 0 14 U

s12 (znhzbztory)

Figure 1: Suppression of autorhythmicity by "slow" synaptic inhibition. (Three upper panels) Cell #2 fires repetitively when uncoupled due to steady bias current, 12 = 0.667. When cell #1 is stimulated (11 = 0.66667, 180 5 t 5 270) inhibition, 512, to # 2 slowly increases; # 2 turns off when s12 exceeds 8,(= 0.123). After stimulus 11 is removed, inhibition slowly wears off and # 2 resumes firing when s12 decreases below eHB(" 0.106). Time courses obtained by numerical integration of equations 2.1-2.3. (Lowest panel) Input/output relation of # 2 with 512 treated as parameter. Ordinate is integrated output (equation 2.4), computed with AUTO (Doedel 1981). osc denotes steady repetitive firing state; ss denotes time-independent stationary state. Dotted lines indicate unstable state. Coexistence of osc and ss for OHB < 512 < 8, leads to hysteresis for on-off transitions of cell # 2 in upper panels. All variables and parameters are dimensionless (see text). Values given here are used throughout except as noted: gNa = 4.0, V N ~= 1.0, 6, = 0.3, k, = 0.12, g K = 4.0, V K = -0.1, 8, = 0.1, kn = 0.1, Tn = 4.5, 81 = 0.0333, 01 == 0.1, g12 = 5.55556, ~ S y n= -0.1, 8syn == 0.43, ksyn = 0.12, T~ = 150, A12 = 1.0.

we used AUTO (Doedel1981), which computes periodic and steady-state solutions of nonlinear differential equations as a function of a parameter.

538


For all dynamic simulations, each Ii includes a small additive stochastic component to diminish delay effects that can occur when a slowly modulated stationary state loses stability (Rinzel and Baer 1988). This noise is uniformly distributed in the interval (-0.005,0.005), and is sampled at each function call. Although this best mimics machine noise or roundoff error, a similar decrease in the ramp effect was obtained by modeling "white" noise with a standard deviation of 0.001. Without noise the system behaves similarly, but our phase plane analysis becomes harder to explain. 3 Mutual Inhibition, Bistability, and the "Synaptic" Phase Plane -

Two mutually inhibitory neurons (vSy,= ZIK for each cell) may exhibit a unique stationary activity pattern if the steady bias currents applied to the cells are in an appropriate range. If only one cell is intensely stimulated, it will fire at a high rate and suppress the other cell. If both receive strong stimulation they will fire steadily but at a rate reduced from their uncoupled rates by inhibition. Less intuitive is an intermediate case of nonuniqueness when the system is bistable: either cell is firing and the other silent. We apply phase plane concepts (Edelstein-Keshet 1988) to interpret this latter case for our averaged model (equation 2.5). The nullcline for S,, is the curve along which St, = 0, and from equation 2.5 this curve is just the graph of the averaged 1 / 0 relation. Thus, in Figure 2, the 321 nullcline is obtained by replotting the lower panel of Figure 1; for 512 we interchange roles becomes the independent variable along the ordinate) and again replot. These nullclines in the "synaptic" phase plane have a restricted interpretation. In particular, the portions corresponding to unstable oscillations of equations 2.1 and 2.2 (open circles and dotted lines on osc branches) are not directly meaningful, since averaging is not justified in such cases. The sign of ill is not determined by the values of S,, and S,, alone when they are in regions where the nullcline is multibranched; one needs to know whether cell i is either oscillating or nearly stationary. Similarly, to identify positions along a trajectory where a cell starts or stops firing one must apply the correct state-dependent threshold values (0, and &B). If approximate "synaptic" trajectories are sought by integrating equation 2.5 (without equations 2.1 and 2.2) it is necessary to keep track of which branch is currently applicable so that branch switching can be carried out properly. An intersection of these nullclines does represent a steady state of equation 2.5, which can be stable only if the associated states (stationary or periodic) of equations 2.1 and 2.2 are stable. In Figure 2 there are two such stable steady states (open squares). These patterns correspond, in the full model, to cell #1 firing and #2 silent, or vice versa. Similar bistability was found experimentally (compare Kleinfeld et al. 1990,


0.5

539

-

0.3 S21

0.1

-0.1

-

I ,

-0.1

0.3

0.1

0.5

312

v1

-

- 4 . 00.0

A 311

9 321

0.0

0.4

0.0 0

500

1000

1500

Time

Figure 2: Bistability for two cells coupled by slow inhibition. Phase plane for synaptic activation variables (upper panel). Nullclines for the approximating "averaged equation 2.5 are graphs of the cells' 1 / 0 relations (from equation 2.4, computed numerically with AUTO); curve for Sij corresponds to activity of cell j driven by slow synapse from cell i. Each nullcline has two branches, for either stationary or periodic behavior (labeled ss or osc, respectively) of the spikegenerating subsystem equations 2.1-2.2. Stability of these behaviors indicated as follows: cell 1 stable states -;cell 1unstable states . . . . . .; cell 2 stable states 0; cell 2 unstable states 0. Of the several nullcline intersections, two represent stable states (denoted by 0): one cell on (sq x 0.4) and the other off (s,i = 01, or vice versa. Heavy curve corresponds to "synaptic" trajectory for simulation of switching experiment; time courses shown in lower panels. Initially, #2 is on and then, following square current step to #1 (duration, 180; intensity, 1.5), #2 is off and #1 is on. Return to initial state occurs after identical current step to # 2. Currents 1; are so strong that during stimulation, the cell is depolarized beyond firing level. Phase plane trajectory is shown dashed during stimulus application. Parameters particular to this case: 11 = 12 = 0.66667, g21 = 5.55556.

540


Fig. 8-9 to our Fig. 2) and demonstrated by using square current pulses to switch dominance in the pair from one cell to the other. In our analogous simulation which starts with cell #2 active, we see that current to #1 must be delivered long enough not only for #2 to stop firing (when sI2rises above 6), but for the ”synaptic” trajectory to cross the separatrix (the 45” diagonal, in this case of identical cells) between the two attracting states. Observe that there is a delay between the termination of the current and when #1 starts firing. It is clear from this graphic representation that firing commences only after inhibition, ~ 2 1 , from #2 drops below OHB. This delay phenomenon illustrates the controlling role played by the slow synaptic variables, and the importance of considering their transient behavior for proper interpretation of firing patterns. We offer a few additional observations regarding the “synaptic” phase plane. As one expects, the trajectory crosses the 512 nullcline vertically (lower right), and similarly, during the reswitching phase, the S21 nullcline is crossed horizontally. Also, the dependence of activity patterns on the steady bias currents can be related to changes in the nullclines. For example, a strong depolarizing bias to cell #2 would shift the 321 nullcline rightward (with some change in shape) and thereby preclude bistability. The only stable state would be with #2 firing and #1 off. A classical treatment of mutual inhibition might assume a cell’s I/O relation (equation 2.4) is monotonic. In that case one could also predict bistability for the pair over a range of bias currents (Kleinfeld et ul. 1990). While our model cells exhibit hysteresis in their 1 / 0 relations, this does not significantly influence the behavior of our mutually inhibitory pair over parameter ranges we have explored. However, if one of the cells is excitatory, hysteresis can lead to a qualitatively different pattern, in which the cells can establish a slow alternating oscillatory rhythm. We explore this next.

4 A Two-Cell Burster The averaged 1 / 0 relation for a cell which receives steady or slow excitatory synaptic input is illustrated in Figure 3 (upper). The hysteretic effect near threshold is quite substantial for these simplified HH spikegenerating dynamics. If the target cell (#2), when brought to firing, could slowly inhibit the sending cell then 512 might decrease enough (below 0,) so that #2 would turn off. With #2 silent, the output, ~ 1 2 ,of #1 could recover to its prior level and thereby reinitiate firing in #2. As this cycle repeats we would observe rhythmic bursting. Figure 4 illustrates such an oscillation for two model cells: one excitatory and the other inhibitory. The bias currents and synaptic weights are set so that the activity threshold for the excitatory cell lies between the on and of activity levels for the inhibitory cell, that is, the B12 nullcline crosses midway through the overlapping osc and ss branches of the 321

541


00

ff, 8 " ~ 0.5

1.o

sIz (exatatory)

1.0

0.9 0.8

0.7 n

0.6 0.5 0.4

0.3 0.0

0.5

1.0

512

Figure 3: Input/output relation for cell that receives steady or slow synaptic excitation. (Upper panel) Integrated activity (equation 2.4) of cell #2 as function of excitation from cell #1; 512 treated as parameter in equations 2.1-2.2, which are solved using AUTO. Labels as in Figure 1 (lower); dotted lines denote unstable state. Repetitive firing and stationary behavior coexist over range 8, < s12 < 8 ~ (8, = 0.132,8HB FZ 0.317). For large 512, repetitive firing is precluded; response is stable steady depolarization. (Lower panel) Potassium activation n versus 512 for output states shown in upper panel. For osc branch, maximum and minimum values of n during a cycle of repetitive firing are shown. Note, in coexistence range, nmin for stable and unstable osc branches are nearly equal and close to ss, indicating that the v - n trajectories for these periodic states are extremely close during slow recovery phase of action potential near the rest state. Parameters particular to this case: g12 = 2.0,12 = 0.

8

542


Figure 4: Bursting oscillation of an excitatory-inhibitory cell pair with slow synapses. Phase plane for synaptic activation variables (upper panel). Nullcline for Sl2 is defined and obtained as in Figure 2; similarly for 521, with 1 / 0 relation for excitation from Figure 3. There is no intersection of nullclines here that corresponds to a stable steady state of averaged equation 2.5. Rather, these equations are assumed to have a stable periodic solution (not shown) that approximates the bursting oscillation (closed curve with arrowheads) of the full model, equations 2.1-2.3. Since s12 is slower than s21 (A12 = 0.333, A21 = 1.01, the synaptic trajectory basically traverses the hysteresis loop of the 321 nullcline. Time courses (lower panel) show alternating activity pattern of the burst rhythm. On-off transitions of spiking are labeled on time courses and in phase plane. Neither cell is firing during phase a-b; both are firing during c-d. Ragged appearance of v during spiking is due to plotting artifact. Parameters particular to this case: 11 = 0.66667,12 = 0.0.

nullcline (upper panel). Thus, when #2 is active and s21 is high 512 decreases, but when #2 is inactive and szl is low then 512 increases from low levels. The activity of #1 lags somewhat behind that of #2 because we have set A12 to be 1/3 of AZ1. Thus, in the time courses, w e see that 512 rises


543

and falls more slowly than sZ1. If s12were even slower, the closed orbit in the phase plane would have nearly vertical jumps between phases during which it would ride the upper or lower branch of the hysteresis loop. Notice that cell #2 stops firing early, that is, the trajectory drops vertically before it reaches the left knee. This feature can be explained by considering the V - n phase plane of the action potential dynamics. For s12 in the overlap regime (Fig. 3 upper), equations 2.1 and 2.2 have a stable steady state surrounded by the stable limit cycle of repetitive firing. These two attractors are separated by an unstable periodic orbit (dotted osc). Between the depolarizing phases of successive spikes, the variable n falls to a minimum as the V - n trajectory passes near the steady state (Fig. 3 lower). Moreover, the stable cycle is extremely close to the unstable cycle during this phase. Hence, small fluctuations from noise (perhaps analogous to channel noise or action currents attenuated through gap junctions) and/or the dynamic variation of 512 can perturb the trajectory across the unstable cycle into the domain of attraction of the steady state and thereby can prematurely terminate the repetitive firing mode. The bursting behavior of Figure 4 depends on hysteresis in the 1 / 0 relation for at least one, but not necessarily both, of the cells. Nonmonotonicity for the cell receiving inhibition is not required. The reverse situation (with sI2 faster and with hysteresis in its nullcline) could also lead to bursting. We note that if both I/O relations were monotonic, as in the classical treatment, bursting would not be possible. Of course, bursting in our example would be precluded if a strong depolarizing bias were applied to #l. In this case, the 512 nullcline would approximately translate upward and both cells would fire continuously. The robustness of bursting is determined by how each parameter shifts the nullclines. As long as the nullclines intersect in the same generic manner, bursting is obtained. For example, the bias current 11 given to cell #1 permits a large range of possible values for bursting (0.6,l.l). On the other hand, the bias current I2 has a much smaller permissible range (-0.03,0.02). 5 Discussion

For two model neurons with slow synaptic coupling we have obtained a reduced model by averaging over the HH-like spike-generating dynamics. Phase plane methods were applied to the reduced synaptic activation equations to predict and interpret activity patterns for the full model. We illustrated our approach by considering examples of nonlinear behavior: bistability for mutual inhibitory cells and bursting for an excitatory-inhibitory pair. A model of mutually excitatory cells can also exhibit multistable behavior, for example, with the two states of on-on and of-of. It yields to essentially the same analysis as the inhibitory pair model and therefore is not discussed in this work.

544


The illustrated mechanism for rhythmic bursting does not involve slowly varying intrinsic membrane currents, such as a slow voltagedependent conductance or a calcium-modulated current. Here, an isolated cell does not burst. Nor does the mechanism reflect nonlinearities associated with the cells’ connectivity, for example, autocatalytic effects of synaptic self-excitation (Wilson and Cowan 1972). The primary factor is hysteresis in the near-threshold behavior for repetitive firing of the spike-generating dynamics. This feature should not be viewed as extraordinary. There are a few generic ways that oscillations can arise as parameters are varied (Rinzel and Ermentrout 1989). The Hopf bifurcation to periodic behavior is associated with a nonzero minimum frequency and the onset may involve bistability with a coexistent stable oscillation and a stable steady state over an interval of parameter values adjacent to the threshold value. This is the case for our V - n system, as well as for the standard HH model (Rinzel 1978). For a different generic onset behavior (referred to as Type I in Rinzel and Ermentrout 1989) the spike frequency rises smoothly from zero. Such a spike-generator in our setup, considered by us and by G. B. Ermentrout (private communication), leads to monotonic averaged 1 / 0 relations and hence cannot cause bursting. Repetitive firing in the cultured Aplysia neurons (Kleinfeld et al. 1990) exhibits a sudden onset with nonzero frequency, as in the Hopf bifurcation mechanism for our model cells. However, hysteresis as in Figures 1 and 3 was not reported for these experiments. Thus it is uncertain whether these neurons would support bursting oscillations of the sort we have found. Nevertheless, it is intriguing that hysteresis in a simple HH-like membrane with simple model synapses that have no fatiguing properties can lead to network-mediated oscillations.

References Adams, P. R., Jones, S. W., Pennefather, P., Brown D. A., Koch, C., Lancaster, C. 1986. Slow synaptic transmission in frog sympathetic ganglia. J. E x p . Biol. 124, 259-285. Doedel, E. J. 1981. AUTO: A program for the automatic bifurcation analysis of autonomous systems. Congr. Numer. 30, 265-284. Edelstein-Keshet, L. 1988. Mathematical Models in Biology. Random House, New York. Ermentrout, G. B. 1990. PHASEPLANE: The dynamical systems tool, Version 3.0. Brooks/Cole Publishing Co., Pacific Grove, CIA. DvoEiik, I., and SiSka, J. 1989. Analysis of metabolic systems with complex slow and fast dynamics. Bull. Math. Biol. 51(2),255-274. Hopfield, J. J., and Tank, D. W. 1986. Computing with neural circuits: A model. Science 233, 625-633.


545

Kleinfeld, D., Raccuia-Behling, F., and Chiel, H. J. 1990. Circuits constructed from identified Aplysia neurons exhibit multiple patterns of persistent activity. Biophys. J. 57, 697-715. Perkel, D. H., and Mulloney, 8. 1974. Motor pattern production in reciprocally inhibitory neurons exhibiting postinhibitory rebound. Science 185, 181-183. Pernarowski, M. 1990. The mathematical analysis of bursting electrical activity in pancreatic beta cells. Ph.D. Thesis, University of Washington. Rinzel, J. 1978. On repetitive activitiy in nerve. Fed. Proc. 37(14), 2793-2802. Rinzel, J., and Lee, Y. S. 1986. On different mechanisms for membrane potential bursting. In Nonlinear Oscillations in Biology and Chemisty, H. G. Othmer, ed. Lecture Notes in Biomathematics, Vol. 66, pp. 19-33. Springer, New York. Rinzel, J., and Baer, S. M. 1988. Threshold for repetitive activity for a slow stimulus ramp: A memory effect and its dependence on fluctuations. Biophys. J. 54,551-555. Rinzel, J., and Ermentrout, G. B. 1989. Analysis of neural excitability and oscillations. In Methods in Neuronal Modeling: From Synapses to Networks, C. Koch and I. Segev, eds., pp. 135-171. MIT Press, Cambridge, MA. Syed, N. I., Bulloch, G. M., and Lukowiak, K. 1990. In vitro reconstruction of the respiratory central pattern generator of the mollusk Lymnaeu. Science 250,282-285. Wilson, R. H., and Cowan, J. D. 1972. Excitatory and inhibitory interactions in localized populations of model neurons. Biophys. J. 12, 1-24.

Received 8 July 1991; accepted 16 September 1991

This article has been cited by: 2. Oren Shriki , David Hansel , Haim Sompolinsky . 2003. Rate Models for Conductance-Based Cortical Neuronal NetworksRate Models for Conductance-Based Cortical Neuronal Networks. Neural Computation 15:8, 1809-1841. [Abstract] [PDF] [PDF Plus] 3. Eugene M. Izhikevich, Frank C. Hoppensteadt. 2003. Slowly Coupled Oscillators: Phase Dynamics and Synchronization. SIAM Journal on Applied Mathematics 63:6, 1935. [CrossRef] 4. Bo Cartling. 1996. Response Characteristics of a Low-Dimensional Model NeuronResponse Characteristics of a Low-Dimensional Model Neuron. Neural Computation 8:8, 1643-1652. [Abstract] [PDF] [PDF Plus] 5. François Chapeau-Blondeau , Nicolas Chambet . 1995. Synapse Models for Neural Networks: From Ion Channel Kinetics to Multiplicative Coefficient wijSynapse Models for Neural Networks: From Ion Channel Kinetics to Multiplicative Coefficient wij. Neural Computation 7:4, 713-734. [Abstract] [PDF] [PDF Plus] 6. Bard Ermentrout . 1994. Reduction of Conductance-Based Models with Slow Synapses to Neural NetsReduction of Conductance-Based Models with Slow Synapses to Neural Nets. Neural Computation 6:4, 679-695. [Abstract] [PDF] [PDF Plus] 7. Paul Frankel, Tim Kiemel. 1993. Relative Phase Behavior of Two Slowly Coupled Oscillators. SIAM Journal on Applied Mathematics 53:5, 1436. [CrossRef]

Communicated by Nancy Kopell

Phase Coupling in Simulated Chains of Coupled Oscillators Representing the Lamprey Spinal Cord Thelma L. Williams Department of Physiology, St.George's Hospital Medical School, Tooting, London S W17 ORE, United Kingdom

Previous application of a mathematical theory of chains of coupled oscillators to the results of experiments on the lamprey spinal cord led to conclusions about the mechanisms of intersegmental coordination in the lamprey. The theory provides no direct link, however, to electrophysiological data obtained at the cellular level, nor are the details of the neuronal circuitry in the lamprey known. In this paper, a variant of the theory is developed for which the relevant variables can potentially be measured. This theory will be applied to measurements on simulated oscillators, based on a network that has been postulated to constitute the basic circuitry of the segmental oscillator in the lamprey. A linear approximation to the equations is derived, and it will be shown that the behavior of simulated chains of these oscillators obeys the predictions of this approximation. 1 Introduction

Many neural systems produce periodic activity, and may thus be regarded as oscillators or as systems of oscillators. Kopell and Ermentrout have developed a mathematical theory of chains of coupled oscillators, in which the effect of the coupling between two oscillators is dependent on the phase difference between them (Ermentrout and Kopell 1984; Kopell and Ermentrout 1986,1988,1990; Kopell etal. 1991). The results of experiments on the lamprey spinal cord have been interpreted in the context of this mathematical theory, to reveal properties of the intersegmental coordinating system. In particular, it was concluded that ascending coupling dominates descending coupling in determining the value of the intersegmental phase lag (Williams et al. 1990; Sigvardt et al. 1991). A disadvantage of the theory is that in the absence of a direct interpretation in terms of membrane potentials and synaptic transmission it is difficult to judge how well the underlying assumptions of the theory are met by the biological system, or to get a biologically intuitive grasp of what the analysis is about. To bridge some of the conceptual gap between the mathematical analysis and the behavior of real neurons, I have developed an operational variant of the theory, which can be directly applied Neural Computation 4, 546-558 (1992) @ 1992 Massachusetts Institute of Technology

Chains of Coupled Oscillators Applied to Lamprey Spinal Cord

547

to an assemblage of simulated neurons. I have also derived a linear approximation to the equations governing the behavior of a chain of such oscillators, and I will show, for specific examples of coupling between neurons in neighboring segments, that the behavior of a coupled chain of such oscillators is in accordance with the predictions of the theory. 2 The Central Pattern Generator for Locomotion in the Lamprey ___

The lamprey spinal cord can produce rhythmic motor patterns of ventral root activity (fictive locomotion) in the absence of any patterned input (Cohen and Wallen 1980). The cyclic pattern consists of alternating bursts of activity in the left and right ventral roots of each spinal cord segment, with a rostral-caudal delay of activation along the length of the cord. Faster swimming is accomplished by a decrease in the cycle duration at all segments. The intersegmental time delay scales with the cycle duration, so that the delay between segments remains a constant fraction of the cycle, at about 0.01 (Wall& and Williams 1984). It is this intersegmental phase lag of about 1%of a cycle per segment that is the focus of the mathematical analysis. 3 The Mathematical Theory

The Kopell-Ermentrout formulation considers a chain of unit oscillators, each of which has intrinsic frequency w. It is assumed that the oscillators are coupled to their nearest neighbors (Fig. 1A). Ascending coupling may differ from descending coupling, but both are assumed to be uniform along the cord. The effect of the coupling between two neighboring oscillators is mediated by ”H-functions,” which give the change in frequency caused by the coupling from one oscillator to another, as a function of the phase difference between them. It is further assumed that this effect is additive; thus the frequency Ok of the kth oscillator in the interior of a chain of length n is given by (Kopell and Ermentrout 1988) flk = wk

+HA(&)

+ HD(-($k-I)

1 [CD(-#A)/(dotted lines). (B) Simulated membrane potential of ipsilateral E cells in segments 1, 3, 5, 7, and 9 of a simulated chain of 10 segments with ascending and descending coupling as in (A). (C)Intersegmental phase lag as a function of segment number (solid circles), average values (with standard deviations) from 4 cycles. The variability is due primarily to the discrete nature of the simulation (one cycle of activity occupies approximately 30 simulated time steps). The solid squares give 4~ and #A as determined in (A). The solid line is drawn from equation 6.4 using the ratio of slopes measured from (A).


555

where both a and /3 > 0, so that C A ( ~has ) a positive slope, C&$) a negative one. With this linear approximation equation 3.1 for a chain of entrained oscillators becomes =W

+a(d'k - 4

~-)P(dk-1

-d

~ ) , 1 1. The curve drawn in Figure 3C is calculated from equation 6.4 with n = 10 and f = 0.6. This curve gives the predicted values of the intersegmental phase lags for a chain of 10 oscillators with coupling as in Figure 3A. The filled circles in Figure 3B are the results of a simulation of such a chain, and it can be seen that the intersegmental delay was approximately equal to 4~over most of the chain, with a boundary layer at the rostral end. The simulation agrees well with that predicted from the data of Figure 3A, obtained from a single pair of oscillators, and shown by the solid line. For simulations using longer chains, the width of the boundary layer at the rostral end remained essentially unchanged, and the intersegmental phase lag for the additional segments was approximately equal to 4 ~ This . is also predicted by equation 6.4, even for f as great as 0.9 (for n 2 10). The direction of travel of the wave of activation depends on the sign of the zero-crossing phase lag of the dominant coupling. Thus in the coupling shown in Figure 3A, a rostral-to-caudal wave would result with either ascending or descending coupling dominant, since both have negative zero-crossings. For the coupling regime of Figure 4, on the other hand, the direction of wave travel depends on which coupling is dominant. For ascending coupling dominant, the wave travels rostral-tocaudal; for descending dominant, it travels caudal-to-rostral. These examples illustrate the fallacy in the intuitive argument that the direction of wave travel should be in the direction of the dominant coupling. For symmetric coupling, f = 1, and the solution to equation 6.3 is given by (6.5)

Thelma L. Williams

556

3 I

C I I

-1.5%

7

-0.05

0.00

0.05

Intersegmental phase lag ( $ 1

B

-0.02

! 0

A

I

2

-

I

4

6

8

1

Segment number ( k

0

I

Figure 4: The effect of dominance on intersegmental phase lag. (A) Ascending coupling (C,) as in Figure 2; descending coupling (CD) the same but with synaptic strengths reduced from 0.05 to 0.02 times the intrasegmental strengths, making ascending coupling dominant. (B)Intersegmental phase lags from simulated chains with different coupling strengths. Relative ascending/descending synaptic strengths in upper to lower curve, in order, were 0.01/0.05,0.05/0.05, 0.05/0.02, 0.05/0.01. Note that for symmetric coupling (0.05/0.05), the phase lag varies along the entire length of the chain, as predicted by equation 6.5, and changes sign in the center. This represents a traveling wave beginning in the center and traveling in opposite directions toward the head and tail ends. For asymmetric coupling the traveling waves were of approximately uniform speed except in the boundary layer, the width of which decreased with increasing dominance. For a chain of greater length, the phase lags for the asymmetric regimes remained virtually unchanged over the first 10 segments (lower 2 curves) or last 10 segments (upper curve), and remained approximately equal to the zero-crossing of the dominant coupling for the remaining segments.


557

which represents a linear change in phase lag along the chain. This prediction was also confirmed, as seen in the triangles of Figure 4B. One of the assumptions implicit in the application of the mathematical analysis is that the coupling signals received by an oscillator d o not alter those transmitted by that oscillator. This assumption is likely to be met if the coupling does not significantly distort the relative timing and magnitude of the activity of the cells within a unit oscillator. In this study the strength of the coupling synapses was low compared to intrasegmental synaptic strengths (0.01 to 0.05), and it was found that the intersegmental phase lag was close but not precisely equal to the zero-crossing of the dominant coupling (as in Figure 4B). Thus this study has demonstrated that intrasegmental coupling between cells comprising the unit oscillator can give rise to well-behaved C-functions and behavior that obeys the predictions of the mathematical analysis. In particular, with asymmetric coupling of equally activated oscillators, the intersegmental phase lag is uniform over the length of the chain, except for a small boundary layer at one end, and is approximately equal to the zero-crossing phase lag of the dominant coupling. Acknowledgments

I am grateful to Jim Buchanan for introducing me to the McClellandRumelhart software and showing me his unpublished results for the lamprey CPG, and to Graham Bowtell for helpful suggestions on equations 6.4 and 6.5. This work was supported by the SERC. References Buchanan, J. T. 1990. Simulations of lamprey locomotion: Emergent network properties and phase coupling. Eur. ]. Neurosci., Suppl. 3, 184 (abstract). Buchanan, J. T. 1992. Neural network simulations of coupled locomotor oscillators in the lamprey spinal cord. Bid. Cybernet. 66, 367-374. Buchanan, J. T., and Grillner, S. 1987. Newly identified 'glutamate interneurons' and their role in locomotionin the lamprey spinal cord. Science 236,312-314. Buchanan, J. T., and Grillner, S. 1988. A new class of small inhibitory interneurones in the lamprey spinal cord. Brain Res. 438, 404407. Cohen, A. H. 1987. Effects of oscillator frequency on phase-locking in the lamprey central pattern generator. ]. Neurosci. Methods 21, 113-125. Cohen, A. H., and Wall&, P. 1980. The neuronal correlate of Iocomotion in fish. 'Fictive swimming' induced in an in vitro preparation of the lamprey. Exp. Brain Res. 41, 11-18. Ermentrout, G. B., and Kopell, N. 1984. Frequency plateaus in a chain of weakly coupled oscillators. SlAh4 1.Math. Anal. 15, 215-237. Ermentrout, G. B., and Kopell, N. 1991. Multiple pulse interactions and averaging in systems of coupled neural oscillators. ]. Math. B i d . 29, 195, 217.

558

Thelma L. Williams

Grillner, S., Buchanan, J. T., and Lansner, A. 1988. Simulation of the segmental burst generating network for locomotion in lamprey. Neurosci. Lett. 89, 31-35. Grossberg, S. 1978. A theory of visual coding, memory, and development. In: E. L. J. Leeuwenberg, and H. F. J. M. Buffart (eds.) Formal Theories of Visual Perception. New York Wiley. Kopell, N., and Ermentrout, G. 8. 1986. Symmetry and phase-locking in chains of weakly coupled oscillators. Comm. Pure Appl. Math. 39, 623-660. Kopell, N., and Ermentrout, G. 8. 1988. Coupled oscillators and the design of central pattern generators. Math. Biosci. 90, 87-109. Kopell, N., and Ermentrout, G. B. 1990. Phase transitions and other phenomena in chains of coupled oscillators. SIAM J. Appl. Math. 50, 1014-1052. Kopell, N.,Ermentrout, G. B., and Williams, T. L. 1991. On chains of oscillators forced at one end. S l A M J. Appl. Math. 51, 10-31. McClelland, J. L., and Rumelhart, D. E. 1988. Explorations in Parallet Distributed Processing: A Handbook of Models, Programs, and Exercises. MIT Press, Cambridge. Rovainen, C. M. 1974. Synaptic interactions of identified nerve cells in the spinal cord of the sea lamprey. J. Comp. Neurol. 154, 189-206. Sigvardt, K. A., Kopell, N., Ermentrout, G. B., and Remler, M. P. 1991. Effects of local oscillator frequency on intersegmental coordination in the lamprey locomotor CPG: theory and experiment. SOC.Neurosci. Abst. 17, 122. Sigvardt, K. A., and Williams, T. L. 1991. Models of central pattern generators as oscillators: The lamprey locomotor CPG. Semin. Neurosci. 4, 37-46. Wallen, P., and Williams, T.L. 1984. Fictive locomotion in the lamprey spinal cord in nitro compared with swimming in the intact and spinal animal. J. Physiol. 347, 22.5239. Williams, T. L., Sigvardt, K. A., Kopell, N., Ermentrout, G. B., and Remler, M. P. 1990. Forcing of coupled nonlinear oscillators: Studies of intersegmental coordination in the lamprey locomotor central pattern generator. J. Neurophysiol. 64,862-871.

Received 13 August 1991; accepted 11 November 1991.

This article has been cited by: 2. A. Rabinovitch, M. Gutman, I. Aviram. 2003. Reaction-diffusion dynamics in an oscillatory medium of finite size: Pseudoreflection of waves. Physical Review E 67:3. . [CrossRef] 3. F. K. Skinner, C. Wu, L. Zhang. 2001. Phase-coupled oscillator models can predict hippocampal inhibitory synaptic connections. European Journal of Neuroscience 13:12, 2183-2194. [CrossRef] 4. KAREN A. SIGVARDT, WILLIAM L. MILLER. 1998. Analysis and Modeling of the Locomotor Central Pattern Generator as a Network of Coupled Oscillators. Annals of the New York Academy of Sciences 860:1 NEURONAL MECH, 250-265. [CrossRef]

Communicated by Kenneth Miller

Understanding Retinal Color Coding from First Principles Joseph J. Atick Zhaoping Li A. Norman Redlich School of Natural Sciences, Institute for Advanced Study, Princeton, NJ 08540 USA A previously proposed theory of visual processing, based on redundancy reduction, is used to derive the retinal transfer function including color. The predicted kernels show the nontrivial mixing of spacetime with color coding observed in experiments. The differences in color coding between species are found to be due to differences among the chromatic autocorrelators for natural scenes in different environments. 1 Introduction

The retinas of many species code color signals in a nontrivial way that is strongly coupled to their coding of spatial and temporal information. For example, in the primate retina many color coding ganglion cells are excited by "green" light' falling on the centers of their receptivefields on the retina, but their response is inhibited by " r e d light falling in a concentric annulus about the green center - called the surround region of their receptive field. There are also red center, green surround cells (Derrington e t d . 1984)as well as rarer types involving blue cones. Such arrangements, which can be termed "single-opponency," are not the only types found in nature. For example, freshwater fish such as goldfish and carp have a different type of coding called "double-opponency" (Daw 1968). Their ganglion cells are color opponent - they calculate the difference between the outputs of different cone types at each spatial location - and they are spatially opponent (with a center surround receptive field) but their color and spatial encoding are mostly decoupled. One of the challenges for a theory of retinal processing is to account for the difference between this double-opponent goldfish code and the single-opponent primate code, as well as the range of intermediate response types observed in other species. 'In this paper, we use "green" and "red to denote light with spectral frequencies exciting primarily the cones with medium and long spectral sensitivities, respectively.

Neural Computation 4, 559-572 (1992)


560

J. J. Atick, Z. Li, and A. N. Redlich

In this paper, we demonstrate that the computable theory of early visual processing reported by Atick and Redlich (1990, 1992 henceforth references I and II) can explain this variety of retinal processing types. As explained at length there, the theory hypothesizes that the purpose of retinal coding is to reduce both redundancy and noise in the visual image. The idea of redundancy reduction as an efficiency goal in the sensory system was first proposed by Attneave (1954) and Barlow (1961). In the retina, redundancy in space, time, and color comes from the fact that the pixel by pixel representation of natural scenes, which is the representation formed by the photoreceptors, contains a high degree of correlations among pixels. Therefore, many pixels redundantly represent the same information. Actually with color, there is an additional source of correlation between the photoreceptor outputs coming from the overlapping spectral domains of the three cone types. To improve efficiency, the retina can recode the photoreceptor signal to eliminate correlations in space, time, and color. In refs. I and 11, it was assumed that the retina, being only the first stage in the visual pathway, can eliminate only the simplest form of redundancy, which comes from pixel-pixel correlations: second-order correlation. It makes sense for the retina to eliminate second-order correlation since it is the source of the largest fraction of redundancy in the image, and it can be eliminated relatively easily through a linear transformation that decorrelates the input (photoreceptor) signal. As shown in I and 11, decorrelation together with noise reduction does give a retinal transfer function that agrees with available data from contrast sensitivity experiments. Here we take that analysis one step further and solve for the system that decorrelates at the two point level both in space and color. What we find is that the differences seen in the color coding of primates and fish can be attributed to plausible differences in the color correlation matrix for the two species. More precisely, we note that the degree of overlap between the R and G cones in primates is greater than the corresponding overlap in fish (the R and G spectral sensitivity peaks are separated by only 32 nm for the primates but by 90 nm for the fish). This difference in photoreceptor sampling is well known to be attributed to differences between the primate visual environment and the environment under water (Lythgoe 1979). What we show in this paper is that this sampling difference has very pronounced effects on the subsequent neural processing needed to achieve decorrelation. In fact it will enable us to account for single vs. double opponency coding. In passing, we should mention that we limit our analysis to the two cone ( R and G ) system, since in primate retina these photoreceptors occur with equal density and are more abundant than the blue cones. In fact the blue cones constitute only 15%of the total cone population in the entire retina while in the fovea they are virtually nonexistent. We also confine ourselves to color coding by linear cells, which implies cells in the primate parvocellular pathway.

Understanding Retinal Color Coding

561

It is important to point out, however, that the mixing between space, time, and color that we derive here does not come only from decorrelation. In fact, we use here a correlation matrix which itself does not mix space-time with color, though such mixing in the correlation matrix can easily be included in our analysis and it only accentuates the effect found here - for the more general analysis (see Atick et al. 1990). It is actually noise filtering, together with redundancy reduction, which produces the nontrivial interactions. Noise filtering is a prerequisite for achieving single opponency, and it also explains the observed differences between psychophysical contrast sensitivity measurements in the luminance and chromatic channels. We should point out that the earliest attempt to explain color opponency using decorrelation was made by Buchsbaum and Gottschalk (1983), also inspired by Barlow (1961). However, their work did not include the spatiotemporal dimensions, nor did it include noise, so it does not account for the observed nontrivial coupling of space-time and color.

2 Decorrelation and Color Coding

As in refs. I and II, we make the hypothesis that the purpose of retinal processing is to produce a more efficient representation of the incoming information by reducing redundancy. With the assumption of linear processing, the retina can eliminate only the simplest form of redundancy, namely second-order correlations. However, second-order decorrelation cannot be the only goal of retinal processing, since in the presence of noise as was argued in I1 decorrelation alone would be a dangerous computational strategy. This is due to the fact that after decorrelation both useful signal and noise are coded in a way that makes their distinction no longer possible (they both have the properties of random noise). Thus for decorrelation, or more generally redundancy reduction, to be a viable computational strategy, there must be a guarantee that no significant input noise be allowed to pass. The way we handle this noise here is similar to the approach in II for the purely spatial domain: we first lowpass filter to diminish noise and then decorrelate as if no noise existed. Figure 1 is a schematic of the processing stages we assume take place in the retina. We should emphasize that this is meant to be an effective computational model and is not necessarily a model of anatomically distinct stages in the retina. As shown in the figure, the intensity signal L ( x , t , A), depending on space, time, and spectral wavelength A, is first transduced by the photoreceptors to produce a discrete set of photoreceptor outputs, P y x , t ) = J dACa(A)L(x, t , A)

(2.1)

562


-

Sampling

Low-pass

P’

MU‘

Whitening

Ka’

Figure 1: Schematic of the signal processing stages for the model of the retina used here. At the first stage, images are sampled by the photoreceptors to produce the chromatic signals, Pa. These are subsequently lowpass filtered by Mnb to eliminate noise, and then decorrelated to reduce redundancy by Kab. The functions Cn(A) are the spectral sensitivities of the two (more generally three) photoreceptor types, a = 1 , 2 for R, G, respectively. Following transduction, the photoreceptor signals are lowpass filtered by a function Mab(x,t; x’: t’) to reduce noise. Having filtered out the noise, the final stage in Figure 1 is to reduce the redundancy using the transfer function Kab(x.t; x’, t’) that produces decorrelated retinal outputs. Thus the output 0 is related to the input P through

0 = K .(M (P + n) + no)

(2.2)

where na(x. t) is input noise including transduction and quantum noise, while no(x, t) is noise (e.g., synaptic), which is added following the filter M. Such post-filter noise, though it may be small, must be included because it is very significant from an information theoretic point of view: it sets the scale (accuracy) for measuring the signal at the output of the filter M. We have introduced boldface to denote matrices in the 2 x 2 color space; also in equation 2.2 each . denotes a space-time convolution. To derive both filters M and K, we require some knowledge of the statistical properties of the luminance signals L ( x , t , A): the statistical properties of natural scenes. For our linear analysis here, we only need the chromatic-spatiotemporal autocorrelator, which is a matrix of the form P y x , t;x’, t’) = ( y x l t)Pb(X1t ) ) =

/dXdA’Ca(X)Cb(A’) (L(x, t , X)L(x’, t’, A’))

where ( ) denotes ensemble average. Unfortunately, not much is known experimentally about the entries of the matrix Rab(x,t ; x ’ , t’). Thus, to gain insight into the color coding problem we are forced to make some assumptions. First, we assume translation invariance in space and time: R is then only a function of x - x’, and t - t’, so it can be Fourier transformed to R”(f, w),where f and w are spatial and temporal frequency,


563

respectively. Second, we assume Rab(f,w)can be factorized into a pure spatiotemporal correlator times a 2 x 2 matrix describing the degree of overlap between the R and G systems. This second assumption is not absolutely necessary, since it is possible to perform the analysis entirely for the most general form of Rob(f,w)(see Atick et al. 1990). However, this assumption does make it much simpler to analyze and explain our theoretical results. We also examine color coding only under conditions of slow temporal stimulation or near zero temporal frequency. In that case, we do have available Field's (1987) experimental measurement of the spatiotemporal correlator: R(f, 0) = G/lflz with 10the mean background luminance of the input signal. Using this R(f), and making the further simplification that the mean squared R and G photoreceptor outputs are roughly equal, we arrive at (2.3)

where Y < 1 is a parameter describing the amount of overlap of R and G. We should emphasize, that we do not advocate this simple form of Rab as necessarily the one found in nature. More complex Rab can similarly be dealt with but they produce qualitatively similar solutions. The next step is to use this autocorrelator to derive a noise filter Mab(f) (from now on we drop explicit w dependence). In ref. ll, the principle used to derive M(f), without color, was to maximize the mutual information between the output of the filter and the ideal input signal [the signal L(f, w)without noise], while constraining the total entropy of the output signal. The resulting lowpass filter cannot be complete, however, since it does not include the effects of the optics, but these can be incorporated by multiplying by the optical modulation transfer function (MTF). As discussed in detail in ref. 11, in the absence of color (one channel problem), this gives

with N2 the input noise power. Here, the exponential term approximates the optical MTF, which has been independently measured (Campbell and Gubisch 1966); we use typical values for the parameters cy and f c . Although, as shown in ref. 11, this filter matches the spatial experimental data well, other filters can also give good results. For example, one may use a maximum log likelihood principle, equivalent in our case to using mean squared estimation. The really important property all such filters must have, however, is that their shape must depend on the signal to noise ( S I N ) at their input. To see how color correlations (two channel problem) affect the spatiotemporal lowpass filtering, it is helpful to rotate in color space to the


564

basis where the color matrix is diagonal. For the simple color matrix in equation 2.3, this is a 45" rotation by

to the luminance, G + R, and chromatic, G -- R, channels [in vector notation, the red and green channels are denoted by R = (1,O)and G = (0,l)]. In this G & R basis, the total correlation matrix, equation 2.3, plus the contribution due to noise is

where the noise, (nunb)= S a b p , is assumed equal in both the R and G channels, for simplicity. In the G & R basis, the two color channels are decoupled. Thus, the corresponding spatiotemporal filters M* (f) are found by applying our single-channel results, equation 2.4, independently to each channel. The R(f) appropriate to each G f R channel is from equation 2.5,

(2.6) Notice that the two channels differ only in their effective S/N ratios: (S/N)* = J r n ( h / N )

which depend multiplicatively on the color eigenvalues 1 & r. In the luminance channel, G + R, the signal to noise is increased above that in either the R or G channel alone, due to the summation over the R and G signals. The filter M + (f), therefore, passes relatively higher spatial and temporal frequencies, increasing spatiotemporal resolution, than without the R plus G summation. On the other hand, the chromatic channel, G - R, has lower S I N , proportional to 1 - r, so its spatiotemporal filter M - (f) cuts out higher spatial and temporal frequencies, thus sacrificing spatiotemporal resolution in favor of color discriminability. The complete filter in the original basis is finally obtained by rotating from the diagonal basis back by 45" =

1 1 -1 1

2 (1

)(

M+(f) 0

M-(f)

[Again M+(f) is given by equation 2.4 with R(f) --t R+(f) in equation 2.6.1 After filtering noise, the next step is to reduce redundancy using the transfer function K u b ( f , ~ ) . 2At the photoreceptor level, most of the *By redundancy here, as in ref. 11, we mean the difference between the average information H calculated using the joint probabilities for the neural outputs, and the sum of the "bit" entropies Hi, calculated treating the ith neuron completely inde-

xi


565

redundancy is due to second-order statistics: autocorrelation. If we ignore noise for the moment, then this redundancy can be eliminated, as shown in ref. 11, by a linear transformation Kab(x- x') that diagonalizes the correlation matrix RQb(x- x') so that at second-order the signals are statistically independent: K . R . KT = D with D a diagonal matrix both in color and space-time. This, does not, however, uniquely specify K since the matrix D is still an arbitrary diagonal matrix. In the spatiotemporal case, we found a unique solution by requiring a translationally invariant, local set of retinal filters: the approximation where all retinal ganglion cells (in some local neighborhood, at least) have the same receptive fields, except translated on the retina, and these fields sum from only a nearby set of photoreceptor inputs. These assumptions force D to be proportional to the unit matrix: D = p l , with proportionality constant p. This gives in frequency space, the whitening filter

In generalizing this to include color, we note that when D is proportional to the unit matrix, the mean squared outputs [ ( K I N T ) % for output o",] of all ganglion cells are equal. This equalization provides efficient use of optic nerve cables (ganglion cell axons) if the set of cables for the cells in a local neighborhood has similar information-carrying capacity. We therefore continue to take D proportional to the identity matrix in the combined space-time-color system. Taking D proportional to the identity, however, leaves a symmetry, since one can still rotate by a 2 x 2 orthogonal matrix Uib, that is, K(f) -+ UoK(f), which leaves D proportional to the identity (Ugh is a constant matrix depending only on one number, the rotation angle; it satisfies UoUi = 1). This freedom to rotate by Uo will be eliminated later by looking at how much information (basically S I N ) is carried by each ganglion cell output. We shall insist that no optic nerves are wasted carrying signals with very low SIN. Returning to the situation with noise, the correlation matrix to be is the one for the signal after filtering by M diagonalized here by Kab(f) (see Fig. 1). To derive Kab(f),we go back to the G R basis where M"(f) is diagonal in color space. Then we can again apply the single-channel analysis from Atick and Redlich (1992) to each channel separately. This gives two functions K+(f) that are chosen to separately whiten the G rt R channels, respectively. Since the complete frequency space correlators in

*

pendently. More precisely, H = - &ljk,,,} P[i,k...} log(P{i,k...)), using the complete joint probabilities P{i,k...} = P(O,, 0,,. . .) for the neural (e.g. photoreceptor) outputs Oi with space-time-color index i, while Hi = - C P(0i) log[P(Oi)]. The difference between H and C,H, measures the amount of statistical dependence of the neural signals on each other: the more dependent, the greater the redundancy, since then more bits effectively carry the same information. Reducing redundancy amounts to finding a transformation on the signals Oi so that after the transformation the ratio H / Hi is lowered.

xi

566

J,

J. Atick, Z . Li, and A. N. Redlich

the two channels after filtering by M*(f) are M:(f)(R*(f) K+(f) are therefore

+ N2)+I$,

the

12

Kdf) =

(2.7)

[M’.

(1)(R* if) +

+ %]

where is the power of the noise that is added following the filter Mub(f) (see e uation 2.2). Equation 2.7 generalizes the whitening filter K(f) = p/R(f) to the case with noise. Now putting equation 2.7 together with equations 2.4 and 2.6, we obtain the complete retinal transfer function the one measured experimentally -

J-4

The right-most matrix transforms the G , R inputs into the G f R basis. These two channels are then separately filtered by K*M*. Finally, the rotation UQ,to be specified shortly, determines the mix of these two channels carried by individual retinal ganglion cells. 3 Properties of Solutions

We now use our theoretical solution (equation 2.8) to explain the observed color processing. Specifically, we now show how such diverse processing types as those found in goldfish and primates are both given by equation 2.8 but for different values of the parameter r in the color correlation matrix. For the case of goldfish, where, as argued in the introduction, one expects only small overlap between R and G (r is small), the two channels in the diagonal basis have eigenvalues 1f r, which are comparable: (I - r ) / ( l + r) 1. This means that both channels will on average be carrying roughly the same amount of information and will handle signals of comparable S I N . Thus the filters K+(f)M+(f)and K-(f)M-(f) are very similar. In fact they are both bandpass filters as shown in Figure 2A for some typical set of parameters. Since these channels are already nearly equalized in S I N , there is no need to rotate them using Ue,so that matrix can be set to unity. Therefore, the complete solution (equation 2.8) when acting on the input vectors R,G , gives two output channels corresponding to two ganglion cell types: N

Z1 = ( G + R ) K+M+ 2 2 = (G -R) K-M-

(3.1)

If we Fourier transform these solutions to get their profiles in space, we arrive at the kernels KRb( x- x’) shown in Figure 3 for some typical set of


300

567

F

A

.1

1

100

10

300 7 100

t

B

30 10

3 1 .1

1

10

100

Spatial frequency (c/deg)

Figure 2: (A,B) The luminance and chromatic channels for the goldfish, A, and for primates, B. In both figures the curve that is more bandpass-like is the luminance G + R channel, while the other is the G - R channel. Parameters used are Io/N = 5.0, CY = 1.4, fc = 22.0 c/deg, NO = 1.0 for both A and B. The only difference between the A and B is T : for A, Y = 0.2 while for B, r = 0.85. parameters. The top row is one cell type acting on the R and G signals, and the bottom row is another cell type. These have the properties of double opponency cells. Moving to primates, there is one crucial difference which is the expectation that Y is closer to 1 since the overlap of the spectral sensitivity curves of the red and green is much greater: the ratio of eigenvalues ( l - y ) / ( l + r ) AN

where the first state represents the orthant of th.e initial point P and the second state represents the orthant of the focal point toward which the flow is directed. The table that gives the symbolic location of the focal point for each orthant is defined here as the truth table. Now consider the connection between the flows in the piecewise linear equations, and the truth table. Call the current Boolean state S1 and

Nonlinear and Symbolic Dynamics of Neural Networks

627

the Boolean state toward which the flow is directed, given by the truth table, S2. If the distance between S1 and S2 is 0, then all initial conditions in orthant S1 are directed towards the focal point in S1 leading to a stable steady state in the differential equation. If the distance between S1 and S2 is 1 then trajectories from all initial conditions in S1 are directed across the common boundary between S1 and S2. Now suppose the distance between S1 and S2 is greater than 1; for example, let the two states differ in n loci. Then the flow from S1 can be directed to any of the n different orthants that lie a distance of 1 from S1 and n - 1 from S2. The boundary that is crossed depends on the initial condition in S1. As a consequence of the above properties the allowed transitions can be represented as a directed graph on an N-cube. This directed graph is called the state transition diagram. As the dynamics of equation 2.7 evolve, the trajectories may pass into different orthants in phase space. Thus a symbolic sequence is generated corresponding to the sequence of orthants visited along the trajectory. These symbolic sequences are consistent with the allowed transitions from the state transition diagram on the N-cube. The state transition diagram for equation 2.7 has the following property. Each edge is oriented in oneand only one direction. This can be established using simple arguments (Glass 1975, 1977a,b). Since we assume that for - each i the value of Ai(y1,. . . ,y,-1, y,+l,. . . ,y ~ does ) not depend on yi ke., wii = O), an edge cannot be directed in two directions. From the construction of the state transition diagram, the number of directed edges in the state transition diagram is equal to the distance between each state on the left-hand side of the truth table, and the subsequent state on the right-hand side. Each column on the right-hand side of the truth table contributes ZN-' to the total distance, and there are N columns so that the total distance is N x 2N-1. This is equal to the total number of edges of the N-cube. Since no edge can be oriented in 2 directions, it follows that every edge has one unique orientation. 3.4 Steady States and Limit Cycles. A problem of general interest is to make assertions concerning the qualitative dynamics of equation 2.7 based solely on the state transition diagram. Previous work established rules to find stable steady states and limit cycles (Glass and Pasternack 1978). Very briefly, if the N edges at any given vertex of the N-cube are all directed toward it, then in the corresponding orthant of phase space there will be a stable steady state. These steady states, which are called extremal steady states, have been the main focus in the study of neural networks (Cowan and Sharp 1988). For an oscillation to result, a necessary condition is that there be a cyclic path in the state transition diagram. This is not, however, a sufficient condition to guarantee stability or uniqueness of the oscillation. In some circumstances, a much more powerful result can be found. A cyclic attractor is defined as a configuration on the N-cube that is analogous to a stable limit cycle in a differential equation. A cyclic attractor of length n is a cyclic path through n vertices

628

John E. Lewis and Leon Glass

of the N-cube such that (1) the edge between successive vertices on the cycle is directed from one to the next in sequence; (2) for any vertex on the cycle, there are N - 2 adjacent vertices that are not on the cycle, and the edge(s) from each of these adjacent vertices idare) directed toward the cycle. If there is a cyclic attractor in the state transition diagram then in the associated piecewise linear differential equations there is either a stable unique limit cycle in phase space such that all points in all orthants associated with the cyclic attractor approach the limit cycle in the limit t 4 00, or there is an asymptotic oscillatory approach to a point Pf.The point Pf is analogous to a stable focus with each of the n coordinates involved in the cyclic attractor approaching zero. The proof of this result relies on the explicit algebraic computation of the limiting properties of the Poincare map, giving the return to a threshold hyperplane. The Poincare map is (3.4)

where z is an (N - 1) vector on a threshold hyperplane, A is an (N - 1) x (N - 1) positive matrix, 4 is a nonnegative (N- 1)vector, and the brackets represent the inner product. For this system, the limiting properties of equation 3.4 on iteration follow using the Perron theorem (Glass and Pasternack 1978). 3.5 Chaotic Dynamics. Chaotic dynamics are aperiodic dynamics in a deterministic system in which there is a sensitivity to the initial state of the system so that two initial conditions, arbitrarily close to one another diverge exponentially over time (Ruelle 1989). Since the flow in any given orthant is always focused toward a single point, it is not obvious that equation 2.7 can display chaotic dynamics. However, as we will show in Section 4 [see also Lewis and Glass (1991)1, numerical integration shows chaotic dynamics in some systems. We have not yet found criteria for chaotic dynamics based on the state transition diagram on the N-cube.

4 Dynamics in Model Networks

In this section we illustrate the dynamics that we have observed so far in equation 2.7. Since we are interested in neural networks, we assume the special case given by equations 2.8 and 2.9, and we assume unless otherwise stated that for all j , the functions Gj(yj) are the same with uj = 1 and b, = 0, and 7;= 7 for all i. Likewise all terms of the connection matrix, wll, are either 1 or 0. Each of the N elements in the network has the same number of inputs, np.

Example 1: Steady States. Consider the network in Figure ZA, where the symbol y2 -I y1 implies y2 inhibits yl (w12:= 1) and T = 0.5. The integration of the dynamics starting from several initial conditions is


629

A

B

yz ( -0.5,0.f

I

'

'

I

(0.5, -0.5)

C 0 l r [ ;

1 0 00

1 1

1 0 0 0

Figure 2: (A) Schematic diagram of a neural network in which there is mutual inhibition. (B) Integration of the PL equations in the phase plane, r = 0.5. The heavy dots indicate the focal points. (C) State transition diagram on the 2-cube (yly2) and the associated truth table. shown in Figure 2B, and the N-cube state transition diagram and truth table are shown in Figure 2C. There are two stable steady states.

Example 2: Stable Limit Cycle. A second example is the cyclic inhibitory loop shown in Figure 3A with N = 3. For T = 0.5, this system gives a unique stable limit cycle oscillation, associated with the cyclic attractor in the state transition diagram (Fig. 3B) (Glass 1975, 1977a,b; Glass and Pasternack 1978). Classification of stable limit cycles using the result in Section 3.4 has been considered previously. The number of distinct cyclic attractors under the symmetry of the N-cube is 1, 1, 3, 18 in dimensions 2, 3, 4, 5, respectively (Glass 1977a). Example 3: Multiple Limit Cycles in a 5-DNetwork. Now consider the dynamics of the 5-element network shown in Figure 4A (n, = 2) with r E (1,2). The state transition diagram for this network is shown in


630

A

B

0 1 0 0 1 1

0 0 1

1 0 1

1 0 0

Figure 3: (A) Schematic diagram of a neural network composed of 3 elements. (B) State transition diagram on the 3-cube ( Y i i j 2 i j 3 ) and the associated truth table. There is a cyclic attractor passing through the states 001, 101, 100, 110, 010, 011.

Figure 4B. Let each vertex on one 4-cube represent all the vertices of the 5-cube in which the first digit of the 5-tuple is 0 and each vertex on the other 4-cube represent all the vertices of the 5-cube in which the first digit is 1. Each vertex on one 4-cube is connected to the equivalent vertex on the other. From numerical integration, there are 8 stable cycles that have different symbolic sequences for the range of T considered. The sequences of states for each of these cycles are shown in Table 1, and can also be followed on the state transition diagram. Each state is represented by the 5-tuple y l y 2 y 3 y 4 y 5 .

Figure 4: Facing page. (A) The 5-element network described in Example 3. All connections are inhibitory and of uniform magnitude (i.e., wij = 1). (B) The state transition diagram for the network in (A). The upper 4-cube represents all states in which the first locus is 1; the lower 4-cube represents all states in which the first locus is 0. See text for a more detailed description.

631


A

10110

11110

I

1xxxx

11110

oxxxx


632

Table 1: Limit Cycles in Example 3. Cycle 1 Cycle 2 Cycle 3 Cycle 4 Cycle 5 Cycle 6 Cycle 7

Cycle 8

10010 00010 00011 00001 00101 00100 01100 01000 01001 00001 10001 10000

11010 01010 01011 00011 00111 00101 00100 01100 01000 01001 11001 11000

10010 00010 00011 00111 00101 00100 01100 01000

01001 11001 10001 10000

10010 00010 01010 01011 00011 00111 00101 00100 01100 01000 01001 11001 11000 11010

10010 00010

00011 00111 00110 01110 01100 01000 01010 01011 01001 11001

10001 10000

10010 00010 01010 OlOll

00011 00001 00101 00100 01100 01000 01001 11001 11000 10000

10010 00010 00011 00111 00110 00100 01100 01000 01010 01011 01001 00001 10001 10000

10010 00010 00011 00111 00110 01110 01010 01011 01001 11001 10001 10000

The stability of each of these cycles depends on the value of 7 . For example, Figure 5 shows the three different stable cycles for T = 1.9. From left to right the panels show the time series corresponding to cycle 4, 5, and 6 from Table 1. To illustrate the bifurcations, we consider the returns to a 4-dimensional face 3 3 separating two neighboring orthants in phase space. The state transition diagram can be used to choose &. In this example, there is not one state transition that is common to all 8 cycles. However, the transition 01100 + 01000 is common to all cycles except cycle 7. By plotting the point of intersection of the trajectory with this hyperplane as the value of T is varied for different initial conditions, the regions of parameter space for which each of the 8 cycles are stable can be observed. Projections of the bifurcation diagram constructed in this way onto the yi-axes are shown in Figure 6. In such diagrams, more than one branch for a given value of 7 indicates that either there are multiple cycles, or that one or more cycles have multiple crossings of F3. In Figure 6, each different branch represents a unique cycle. We have numerically analyzed the bifurcations shown here. Briefly, the bifurcation occurring near T = 1.29 appears to be a subcritical Hopf bifurcation. Increasing T above this value causes cycles 2 and 3 to lose stability (upper and lower branches). Cycle 1 maintains its stability through this point (middle branch). Near T = 1.66, an exchange of stability from cycle 1 to cycle 4 occurs. Cycles 5 and 6 gain stability near T = 1.79 in a bifurcation that is similar to that occurring with cycles 2 and 3 for 7 = 1.29. Cycles 7 and 8 are stable for values of T E (1,1.25). Cycles 5 and 6 are identical under a relabeling transformation. To make this more clear, consider the sequences of the state transitions in


cycle 5

cycle 4

0.0 -

yz -0.4 -0.8-

633

cycle 6

mmm

0.0-

y4 -0.4-0.8 -

0.0

y5 -0.4 -0.8

1 0

10

20

time

Figure 5: Multistability of cycles for the network described in Example 3 (Fig. 4A). Three different cycles are stable for T = 1.9 and are shown here by choosing three different initial conditions. The time axis is arbitrary.

Table 1 corresponding to the two cycles. As mentioned earlier, each state is represented by the 5-tuple y l y 2 y 3 y 4 y 5 . The relabeling transformation is the following: switch locus 1 with 3 and locus 2 with 4. In other words, the 5-tuple y & y 3 y 4 y 5 becomes y3y4y&Y5. Performing this transformation on one of the cycles shows that the sequences of state transitions are the same, and thus the cycles are the same. This symmetry is also evident in the connectivity of the network (Fig. 4A). A similar relationship exists between cycles 2 and 3 and cycles 7 and 8.

Example 4: Chaotic Dynamics in a 6-D Network. The 6-element network (n, = 3) in Figure 7 exhibits chaotic dynamics for some parameters. A projection of the attractor onto the y2-1~4 plane is shown in Figure 8A. We consider a face, F.4 separating the orthants defined by 011011 and 010011. Figure 8B shows the density histogram for the times between


634

A

B Yl

C Y4

D Y5

1.o

1.s

2.0

z

Figure 6: Bifurcation diagram for returns to the face F3 and values of T from 1.001 to 1.999 in steps of 0.001. Each panel (A-D) shows the projections onto the different axes. 2000 successive returns to F4 and Figure 8C shows the density for a single variable y4 on each return to F4.We also consider the evolution of the density histograms for successive returns to F4 for a set of 2000 initial conditions in which y4 was varied, and the other variables were held constant. Figure 8D-F shows that by the 20th return, the histograms have reached a density that is the same as that of a single trajectory (Fig. SC). The approach to an invariant density and the observation of the same invariant density along a single trajectory constitute numerical evidence that this system is ergodic and has a unique invariant density, two features common to many chaotic systems (Lasota and Mackey 1985). Now we consider the effects of varying 7 on the dynamics of this network. The dynamics are tracked by plotting the values of y4 on 30 successive crossings of F4as T is varied. Figure 9A shows the resulting bifurcation diagram. As T is increased from 7 = 1.2, the dynamics change


635

Figure 7 The 6-element network discussed in Example 4. from a simple limit cycle to aperiodic behavior. For larger values of 7, a limit cycle is evident again. In the aperiodic region, there are at least 4 periodic windows, spaced nearly symmetrically about T = 1.5. This simple example shows how r can influence the network dynamics. Since the step function nonlinearity in equation 2.9 is not realistic as a model for most biological processes, it is important to clarify the dynamics when continuous nonlinear functions are used in equation 2.8. We consider the continuous gain function, (4.1) where p is a positive constant, and equation 4.1 approaches a step function in the limit p -+ 00. A 4th order Rung-Kutta integration scheme (At = 0.01) was used to solve the equations. As the value of B increases, the continuous system exhibits a complex sequence of bifurcations. By using a method similar to that described for Example 3, a bifurcation diagram was constructed for values of between 7.0 and 12.0 (Fig. 9B). The value of y4 is plotted as the solution trajectory crosses the y3 = 0 hyperplane in a negative sense. For each value of p, a transient of 300 crossings was allowed before the next 30 points were plotted. A different example of a chaotic 6-dimensional network also shows a complex sequence of bifurcations as a continuous sigmoidal function is steepened (Lewis 1991; Lewis and Glass 1991). Further study of the bifurcations in these systems is needed.


636

SINGLE TRAJECTORY

A

-

(1 11

Y2

-I

-1

RETURN MAPS

D 1st

I

0

Y4

B

E

1

,hr3

#

~,

I_j

0 0

5

15

10

IWO

,

I

I

3rd

,

20

Crossing times

C

1 #Irn]

F

,_A,

loo0

1

20th

0

-0.2

-0.1 Y4

0.0

y, return

Figure 8: (A) Projection of the dynamics onto the y2-~4 plane for r = 1.5. (B) shows the density histogram for the times between successive crossings of F4. (C) The density histogram of y4 for 2000 successive crossings of F 4 on a single trajectory. (D-F) The density histograms of y4 for the lst, 3rd, and 20th returns to F 4 using 2000 different initial conditions in which equally spaced values of y4 were chosen between -0.2 and 0, with yl = -0.293862, y2 = 0.478693, lj3 = 0.0, y5 = 0.028766, and y6 = 0.270764. Example 5: Chaotic Dynamics in a Network of 50 Elements. We now consider the dynamics of a larger network consisting of 50 elements with np = 5 and T = 2.5. Details concerning the network are in Lewis (1991) and will be provided on request. In this network, a search of 100 randomly chosen initial conditions revealed no steady states or limit cycles. As in previous examples, the value of a single variable on the return of the trajectory to an ( N - 1)-dimensional face, 5 , is considered. Figure 10A shows the density histograms of yl on & (left panel) and the times between returns (right panel) for 500 successive returns of a single trajectory. Figure 1OB shows the density histograms for y~ and the return times for a first return map constructed by taking initial condi-

637


A

-0.1

Y4

I

1.2

1.4

I

i

1.6

1.8

z

6

-0.4 7

'.

..

I

1

I

I

1

8

9

10

11

12

P Figure 9: (A) Bifurcation diagram showing the value of y4 on 30 successive crossings of F4 after a sufficient transient, for different values of T . (B) Bifurcation diagram as a function of P for the continuous network described in Example 4. After a transient, the values of y4 are plotted when the trajectory crosses the y3 = 0 hyperplane in a negative sense, 30 consecutive times. tions on F5where all initial values were constant except y1 which was varied from -3.0 to -1.0 (as in Example 4). These density histograms are similar to those of a single trajectory (Fig. 10A) after only one return to F5. Calculating a first return map for a smaller interval of yl, between -2.1 and -1.9, again reveals similar density histograms (Fig. 1OC). This


638

SINGLE TRAJECTORY

A

RETURN MAPS

B

n

100

80

1

C

-3.0

-2.5

-2.0

Yl

-1.5

I

-

-1.0

0

1

.

,

I

!

*

I

I

I

2000 4000 6000 8000 lo000

crossing time

Figure 10: (A) Left panel: The density histogram of yl on Fs for 500 successive crossings of a single trajectory. Right panel: The density histogram for the corresponding times between successive crossings of 35. (B) Left panel: The density histogram of yl on the first return map constructed for 500 different initial conditions on Fs in which the value of yl was varied between -3.0 and -1.0. Right panel: The density histogram of the corresponding crossing times for the data in the left panel. (C) Same as (B) but using initial values of yl between -2.1 and -1.9. system is chaotic and only a small number of passes through phase space is required for nearby trajectories to diverge. 5 Discussion Neural networks in nature display a wide range of complex dynamic behavior, ranging from more or less regular periodic behavior, to complex fluctuation that is difficult to characterize. The current paper shows that


639

complex dynamics can also be found in commonly used mathematical models for neural networks. The dynamics can be classified by using the state transition diagram, which links the wiring diagram of the neural network to the coarse-grained activity patterns in the network. The simple structure of the mathematical equations enables us to demonstrate uniqueness and stability of limit cycle oscillations in some special circumstances. We comment briefly on the various dynamics found in these networks. We then discuss some open theoretical questions. The extremal steady states in these networks are easily identified using the state transition diagram. Recent theoretical studies (Amit 1989) have linked such steady states with memories in neural networks, but we are not aware of physiological studies supporting such an identification. Neural network limit cycle oscillations have been proposed as models for rhythmogenesis in a large variety of invertebrate and vertebrate systems (Friesen and Stent 1978; Matsuoka 1985). These studies considered networks of a specific connectivity and some analytical results have been obtained for the oscillatory properties of these systems (Matsuoka 1985; Cohen 1988). The current approach provides techniques for associating patterns of oscillation with the underlying connectivity of the network (Glass and Young 1979). A novel behavior demonstrated here is multistability of limit cycle oscillations, where parameter changes of the network can lead to changes in the stability of the various behaviors (Figs. 5 and 6). This behavior is interesting in light of recent experimental studies on multifunctional invertebrate neural networks (Harris-Warrickand Marder 1991; Meyrand et al. 19911, where different types of oscillatory behaviors can be exhibited by a single network. The simple networks here also support chaotic dynamics. Although the possible role of chaotic dynamics in normal and pathological functioning in neurobiology was raised several years ago (Guevara et al. 1983; Harth 1983) clear identification of chaos in neural systems has been possible only in rather simple systems in which there is a periodic forcing of neural tissue (Matsumoto et al. 1987; Takahashi et al. 1990). There have also been claims that neural activity in more complex situations is chaotic (Rapp et al. 1985; Skarda and Freeman 1987; Babloyantz and Destexhe 1987). The existence of chaotic dynamics in models of abstract neural networks has also been investigated. Kurten and Clark (1986) used spectral and dimensional analysis to identify chaos in a neural network model of 26 elements, each described by 2 ordinary differential equations and interconnected in a pseudorandom manner with each element receiving 7 inputs (both excitatory and inhibitory). Sompolinsky et al. (1988) have shown that some continuous models of neural networks will show a transition to chaotic dynamics as a gain parameter is varied. They proved this result in the thermodynamic limit (i.e., in an infinitely large network). Finally, Kepler et al. (1990) showed that for a specific formulation

640


of a neural network implemented as an electronic circuit, chaotic dynamics could be observed in three dimensions. Their investigation focused, however, on the dynamics of four-dimensional networks. A compelling question is to identify and classify network connectivities that are capable of generating chaotic dynamics. Several mathematical questions are raised by this work. Previously we reported that assuming the same connection parameters for each element (i.e., w,j = l and np inputs to each element), the lowest dimension in which chaotic dynamics was observed is 6 (Lewis and Glass 1991). However, when the w,, are randomly chosen real numbers (with wii = 01, some networks of 5 elements have shown such behavior (less than 0.05% of networks tested). The general system, equation 2.7 has shown chaos in dimensions 4 and higher; in these cases the truth tables consisted of functions that do not correspond to those possible in neural network models. Preliminary studies of the prevalence of the various sorts of dynamic behavior have been carried out. For 2 and 3 input systems in dimension up to 20, chaotic dynamics appear to be a relatively rare phenomenon found in less than 1%of trials in which there were 20 initial conditions for each of 1000 different networks. The number of different attractor basins in these networks is also very small (usually less than 10 attractors, even in dimension 20). However, systematic numerical studies require searching in huge parameter spaces, since one is interested in studying the effects of the numbers of inputs, the thresholds, and the connectivity. The simplicity of numerically integrating the piecewise linear equations facilitate such studies. A difficult mathematical question is to analyze the bifurcations as the piecewise linear functions are replaced by continuous functions. Numerical results indicate that in systems with cyclic attractors, the limit cycles maintain stability over a large range of steepness of the sigmoidal function, but there is no proof of this (Glass 197%). The bifurcations in more complex networks that display chaos require further analysis. An especially interesting question is how chaos arises in these systems whose dynamics are dissipative within every coarse-grained orthant of phase space. This work provides a conceptually simple way to correlate the connectivity and dynamics of simple models of neural networks. This provides a foundation for the investigation of more realistic models of neural networks and complex rhythms observed in the laboratory.

Acknowledgments This research has been supported by funds from the Natural Sciences and Engineering Research Council of Canada and the Fonds F. C. A. R. du Quebec.


641

References Amit, D. J. 1989. Modeling Brain Function: The World of Attractor Neural Networks. Cambridge University Press, Cambridge. Babloyantz, A., and Destexhe, A. 1987. Chaos in neural networks. Proc. Int. Conf. Neural Networks, San Diego, CA, pp, 1-9. Cohen, M. A. 1988. Sustained oscillations in a symmetric cooperative-competitive neural network Disproof of a conjecture about content adressable memory. Neural Networks 1, 217-221. Cowan, J. D., and Sharp, D. H. 1988. Neural nets. Q. Rev. Biophys. 21,305427. Friesen, W. O., and Stent, G. S. 1978. Neural circuits for generating rhythmic movements. Annu. Rev. Biophys. Bioeng. 7, 37-61. Glass, L. 1975. Combinatorial and topological methods in nonlinear chemical kinetics. J. Chem. Phys. 63, 1325-1335. Glass, L. 1977a. Combinatorial aspects of dynamics in biological systems. In Statistical Mechanics and Statistical Methods in Theory and Application, U. Landman, ed., pp. 585-611. Plenum, New York. Glass, L. 197%. Global analysis of nonlinear chemical kinetics. In Statistical Mechanics, Pt. B, B. J. Berne, ed., pp. 311-349. Plenum, New York. Glass, L., and Pasternack, J. S. 1978. Stable oscillations in mathematical models of biological control systems. J. Math. Biology 6, 207-223. Glass, L., and Young, R. 1979. Structure and dynamics of neural network oscillators. Brain Res. 179, 207-218. Guevera, M. R., Glass, L., Mackey, M. C., and Shrier, A. 1983. Chaos in neurobiology. I E E E Trans. Syst. Man Cybern. SMC-13, 790-798. Harris-Warrick, R. M., and Marder, E. 1991. Modulation of neural networks for behavior. Annu. Rev. Neurosci. 14, 39-57. Harth, E. 1983. Order and chaos in neural systems: Approach to the dynamics of higher brain functions. IEEE Trans. Syst. Man Cybern. SMC-13, 782-789. Hopfield, J. J, 1984. Neurons with graded response have collective computational properties like those of two-state neurons. Proc. Nutl. Acad. Sci. U.S.A. 81, 3088-3092. Keister, W., Ritchie, A. E., and Washburn, S. H. 1951. The Design of Switching Circuits. D. Van Nostrand, Toronto. Kepler, T. B., Datt, S., Meyer, R. B., and Abbott, L. F. 1990. Chaos in a neural network circuit. Physica D 46, 449-457. Kiirten, K. E., and Clark, J. W. 1986. Chaos in neural systems. Phys. Lett. A 114, 413-418. Lasota, A,, and Mackey, M. C. 1985. Probabilistic Properties of Deterministic Systems. Cambridge University Press, Cambridge. Lewis, J. E. 1991. Dynamics of neural networks and respiratory rhythm generation. M.Sc. Thesis, McGill University. Lewis, J. E., and Glass, L. 1991. Steady states, limit cycles, and chaos in models of complex biological networks. Int. I. Bijkrc. Chaos. 1, 477-483. Matsumoto, G., Aihara, K., Hanyu, Y., Takahashi, N., Yoshizawa, S., and Nagumo, J, 1987. Chaos and phase locking in normal squid axons. Phys. Lett. A 123, 162-166.

642


Matsuoka, K. 1985. Sustained oscillations generated by mutually inhibiting neurons with adaptation. Biol. Cybern. 52, 367-376. Meyrand, P., Simmers, J., and Moulins, M. 1991. Construction of a pattern generating circuit with neurons of different networks. Nature (London) 351, 60-63. Rapp, P., Zimmerman, I. D., Albano, A. M., deGuzman, G. C., Greenbaum, N. N., and Bashore, T. R. 1985. Experimental studies of chaotic neural behavior: Cellular activity and electroencephalographicsignals. In Nonlinear Oscillations in BiologyandChernistry, H. G. Othmer, ed., pp. 175-205. SpringerVerlag, Berlin. Ruelle, D. 1989. Clzaotic Evolution and Strange At tractors. Cambridge University Press, Cambridge. Skarda, C. A., and Freeman, W. J. 1987. How brains make chaos in order to make sense of the world. Behav. Brain Sci. 10, 161-195. Sompolinsky, H., Crisanti, A., and Sommers, H. J. 1988. Chaos in random neural networks. Phys. Rev. Lett. 61,259-262. Takahashi, N., Hanyu, Y., Musha, T., Kubo, R., and Matsumoto, G. 1990. Global bifurcation structure in periodically stimulated giant axons of squid. Physica D 43,318-334.

Received 19 August 1991; accepted 3 January 1992.

This article has been cited by: 1. R. Edwards, P. Driessche, Lin Wang. 2007. Periodicity in piecewise-linear switching networks with delay. Journal of Mathematical Biology 55:2, 271-298. [CrossRef] 2. Randall D. Beer. 2006. Parameter Space Structure of Continuous-Time Recurrent Neural NetworksParameter Space Structure of Continuous-Time Recurrent Neural Networks. Neural Computation 18:12, 3009-3051. [Abstract] [PDF] [PDF Plus] 3. Xiao-Song Yang, Yan Huang. 2006. Complex dynamics in simple Hopfield neural networks. Chaos: An Interdisciplinary Journal of Nonlinear Science 16:3, 033114. [CrossRef] 4. Qingdu Li, Xiao-Song Yang. 2006. Chaotic dynamics in a class of three dimensional Glass networks. Chaos: An Interdisciplinary Journal of Nonlinear Science 16:3, 033101. [CrossRef] 5. Jaeseung Jeong, Yongho Kwak, Yang In Kim, Kyoung J. Lee. 2005. Dynamical Heterogeneity of Suprachiasmatic Nucleus Neurons Based on Regularity and Determinism. Journal of Computational Neuroscience 19:1, 87-98. [CrossRef] 6. Asa Ben-Hur, Hava T. Siegelmann. 2004. Computation in gene networks. Chaos: An Interdisciplinary Journal of Nonlinear Science 14:1, 145. [CrossRef] 7. K. Aihara. 2002. Chaos engineering and its application to parallel distributed processing with chaotic neural networks. Proceedings of the IEEE 90:5, 919-930. [CrossRef] 8. Hidde de Jong. 2002. Modeling and Simulation of Genetic Regulatory Systems: A Literature Review. Journal of Computational Biology 9:1, 67-103. [CrossRef] 9. R. Edwards, L. Glass. 2000. Combinatorial explosion in model gene networks. Chaos: An Interdisciplinary Journal of Nonlinear Science 10:3, 691. [CrossRef] 10. K. Pakdaman, C. Grotta-Ragazzo, C. Malta. 1998. Transient regime duration in continuous-time neural networks with delay. Physical Review E 58:3, 3623-3627. [CrossRef] 11. Thomas Mestl, R. Bagley, Leon Glass. 1997. Common Chaos in Arbitrarily Complex Feedback Networks. Physical Review Letters 79:4, 653-656. [CrossRef] 12. R. D. Hangartner, P. Cull. 1995. A ternary logic model for recurrent neuromime networks with delay. Biological Cybernetics 73:2, 177-188. [CrossRef]

NOTE

Communicated by Charles Stevens

Cortical Cells Should Fire Regularly, But Do Not William R. Softky Christof Koch Computation and Neural Systems Program, California Institute of Technology,Pasadena, CA 92225 USA

When a typical nerve cell is injected with enough current, it fires a regular stream of action potentials. But cortical cells in v i m usually fire irregularly, reflecting synaptic input from presynaptic cells as well as intrinsic biophysical properties. We have applied the theory of stochastic processes to spike trains recorded from cortical neurons (Tuckwell 1989) and find a fundamental contradiction between the large interspike variability observed and the much lower values predicted by well-accepted biophysical models of single cells. Over 10,000 extracellular spike trains were recorded from cells in cortex of the awake macaque monkey responding to various visual stimuli. These trains were recorded from V1 (Knierim and Van Essen 1992) and MT (Newsome et al. 1989). Traces were chosen from well-isolated, fastfiring, nonbursting neurons. Because the firing frequency varied over the course of the stimulus presentation, each interspike interval At (i.s.i.) was assigned to 1 of 10 histograms for that cell, with each histogram representing a narrow range of instantaneous firing rates, for example, 50-100 Hz, 250-300 Hz. From each histogram we computed a measure of the variability of the spike train, the dimensionless coefficient of variation (CV), which is the ratio of the standard deviation to the mean of the i.s.i. histogram:

The approximate CV values measured here are in good agreement with other reports of CV (Douglas and Martin 1991; Burns and Webb 1976): interspike intervals are near-random, and close to that expected for the i.s.i. histogram of a pure Poisson process (i.e., CV M 0.5-1; see Fig. 1). We attempted to account for this observed variability using a simple integrate-and-fire model requiring N random (Poisson) impulse inputs to reach the threshold (Tuckwell 1989). For such a neuron, CV = 1/a. An absolute refractory period t o reduces this value when is near to Neural Computation 4, 643-646 (1992)


William R. Softky and Christof Koch

644

Observed and Predicted Neuronal Variability

0.5’’

0.4.-

0.3~-

0.2.-

0.1 ~-

Figure 1: Comparison of the randomness measure CV as a function of interspike interval for three different data sets: (1) experimentally recorded, nonbursting, macaque cortical neurons (MT and V1; empty squares; we observed no systematic difference between the two data sets); (2) detailed compartmental simulation of a reconstructed layer V pyramidal cell (filled, connected squares); (3) different integrate-and-fire models with refractory period of 1.0 msec and N EPSPs required to fire (crosses and jagged lines). Crosses are predictions by integrate-and-fire models with N = 1 (top), N = 4 (middle), and N = 51 (bottom). Jagged lines show simulated leaky integrators with N = 51: T, = 0.2 msec (top) or T,,, = 13 msec (bottom). Conventional parameters (i.e., T, > 10 msec and N > 50) fail to account for the high variability observed.

(Tuckwell 1989). Numerical simulations with a leak term 7 , = RC show >> 7,. CV can also increase that CV increases significantly only for during periods of very strong inhibition, but such inhibition was not found in a recent electrophysiological search (Berman et al. 1991). Because most researchers estimate that 100 or more inputs are required to trigger a cell (Douglas and Martin 1991; Abeles 1991), as well as 7 , 2 10 msec and

Firing of Cortical Cells

645

to 2 1.O msec, the above models predict that CV should be far lower than is seen in the monkey data for the high firing rates observed (see Fig. 1). There remains the possibility that more realistic Hodgkin-Huxley neurons (whose firing currents are continuous functions of voltage) might be able to amplify input irregularities more effectively than the highly simplified integrate-and-fire neuron above, which has a discontinuous firing threshold and no such sensitive voltage regime. We expect that this difference would be significant only in a neuron whose soma spends most of its “integration time” resting just below threshold (unlike the cortical cells in question, which have high firing rates and hence no stationary resting potential during periods of peak activation). But the only persuasive test would be the simulation of a Hodgkin-Huxley-like neuron in the presence of random synaptic input. We therefore simulated a biophysically very detailed compartmental model of an anatomically reconstructed and physiologically characterized layer V pyramidal cell (Bernander et al. 1991). The model included not only the traditional Hodgkin-Huxley currents, but five additional 820 active currents at the cell body (INa,INa-p, Ic,, IDR, IA, IM,IK(c~)), compartments, and a passive decay time of T~ = 13 msec. Spatially distributed random (Poisson) excitatory synaptic conductance inputs gave rise to strong somatic EPSPs with mean amplitudes around 1.6 mV. We provided enough synaptic input to this cell to generate 200 spike trains (with mean frequencies comparable to the spike trains recorded from monkey) and subjected them to the same analysis. The resulting CV values agree with the simple integrator models, and disagree strongly with the monkey data (see Fig. 1). In addition, the number of spikes n in each simulated train varied by no more than a few percent, a much smaller amount than the fi variation observed for real cells. Therefore, we conclude that the present knowledge of pyramidal cell biophysics and dynamics is unable to account for the high CV seen in fast-firing monkey visual cortex neurons: these cells should fire regularly, but do not. Neither the data nor the model used here are controversial. But they are not consistent with each other. Only a few situations could cause near-random, fast firing in these monkey cells: for example, strong synaptic conductance changes that create a very fast effective time constant (7, < 0.2 msec; see Fig. 11, or nonrandom synaptic input, which is highly synchronized on a millisecond scale (Abeles 1991; Koch and Schuster 1992). In the absence of such phenomena, the Central Limit Theorem makes these cells observed near-random spiking inconsistent with their assumed role as devices that temporally integrate over many inputs. Thus, it may well be that the time scale of cortical computation is much faster than previously realized.

William R. Softky and Christof Koch

646

Acknowledgments This research was funded by a NSF Presidential Young Investigator Award, by the Office of Naval Research, a n d by the James S. McDonnell Foundation.

References Abeles, M. 1991. Corticonics. Cambridge University Press, New York. Berman, N., Douglas, R., Martin, K., and Whitteridge, D. 1991. Mechanisms of inhibition in cat visual cortex. J. Physiol. 440,697-722. Bernander, O., Douglas, R., Martin, K., and Koch, C. 1991. Synaptic background activity determines spatio-temporal integration in single pyramidal cells. Proc. Natl. Acad. Sci. U.S.A. 88, 11569-11573. Burns, B., and Webb, A. C. 1976. The spontaneous activity of neurones in the cat's cerebral cortex. Proc. R. SOC.London B 194,211-223. Douglas, R., and Martin, K. 1991. Opening the grey box. Trends Neurosci. 14, 286-293. Knierim, J., and Van Essen, D. 1992. Neuronal responses to static textural patterns in area V1 of the alert macaque monkey. J. Neurophysiol. 67, 961-980. Koch, C., and Schuster, H. 1992. A simple network showing burst synchronization without frequency locking. Neural Comp. 4,211-223. Newsome, W., Britten, K., Movshon, J. A., and Shadlen, M. 1989. Single neurons and the perception of motion. In Neural Mechanisms of Visual Perception, D. Man-Kit Lam and C. Gilbert, eds., pp, 171-198. Portfolio Publishing Co., The Woodlands, TX. Tuckwell, H. C. 1989. Stochastic processes in theneurosciences. Society for Industrial and Applied Mathematics, Philadelphia. ~-

____

Received 29 October 1991; accepted 4 February 1992.

This article has been cited by: 1. Kanaka Rajan, L. Abbott, Haim Sompolinsky. 2010. Stimulus-dependent suppression of chaos in recurrent neural networks. Physical Review E 82:1. . [CrossRef] 2. Jörn Davidsen, Peter Grassberger, Maya Paczuski. 2008. Networks of recurrent events, a theory of records, and an application to finding causal signatures in seismicity. Physical Review E 77:6. . [CrossRef] 3. Gyuchang Lim, Soo Yong Kim, Kyungsik Kim, Dong-In Lee, Myung-Kul Yum. 2008. Regularity Analysis of Inter-Out-of-Equilibrium State Intervals in Financial Markets. Journal of the Physical Society of Japan 77:3, 033801. [CrossRef] 4. Hideo Hasegawa. 2007. Generalized rate-code model for neuron ensembles with finite populations. Physical Review E 75:5. . [CrossRef] 5. A. N. Burkitt. 2006. A Review of the Integrate-and-fire Neuron Model: I. Homogeneous Synaptic Input. Biological Cybernetics 95:1, 1-19. [CrossRef] 6. Gleb Basalyga , Emilio Salinas . 2006. When Response Variability Increases Neural Network Robustness to Synaptic NoiseWhen Response Variability Increases Neural Network Robustness to Synaptic Noise. Neural Computation 18:6, 1349-1379. [Abstract] [PDF] [PDF Plus] 7. Hiroya Nakao. 2006. Population Coding by Globally Coupled Phase Oscillators. Journal of the Physics Society Japan 75:3, 034001. [CrossRef] 8. Nicolas P Cottaris, Sylvia D Elfar. 2005. How the retinal network reacts to epiretinal stimulation to form the prosthetic visual input to the cortex. Journal of Neural Engineering 2:1, S74-S90. [CrossRef] 9. B. Scott Jackson . 2004. Including Long-Range Dependence in Integrate-and-Fire Models of the High Interspike-Interval Variability of Cortical NeuronsIncluding Long-Range Dependence in Integrate-and-Fire Models of the High Interspike-Interval Variability of Cortical Neurons. Neural Computation 16:10, 2125-2195. [Abstract] [PDF] [PDF Plus] 10. S. Mikula, E. Niebur. 2004. Correlated Inhibitory and Excitatory Inputs to the Coincidence Detector: Analytical Solution. IEEE Transactions on Neural Networks 15:5, 957-962. [CrossRef] 11. Shawn Mikula , Ernst Niebur . 2003. The Effects of Input Rate and Synchrony on a Coincidence Detector: Analytical SolutionThe Effects of Input Rate and Synchrony on a Coincidence Detector: Analytical Solution. Neural Computation 15:3, 539-547. [Abstract] [PDF] [PDF Plus] 12. Jörn Davidsen, Heinz Schuster. 2002. Simple model for 1/f^{α} noise. Physical Review E 65:2. . [CrossRef]

13. Hideo Hasegawa. 2001. An Associative Memory of Hodgkin-Huxley Neuron Networks with Willshaw-Type Synaptic Couplings. Journal of the Physics Society Japan 70:7, 2210-2219. [CrossRef] 14. Stefano Fusi , Mario Annunziato , Davide Badoni , Andrea Salamon , Daniel J. Amit . 2000. Spike-Driven Synaptic Plasticity: Theory, Simulation, VLSI ImplementationSpike-Driven Synaptic Plasticity: Theory, Simulation, VLSI Implementation. Neural Computation 12:10, 2227-2258. [Abstract] [PDF] [PDF Plus] 15. A. N. Burkitt , G. M. Clark . 2000. Calculation of Interspike Intervals for Integrate-and-Fire Neurons with Poisson Distribution of Synaptic InputsCalculation of Interspike Intervals for Integrate-and-Fire Neurons with Poisson Distribution of Synaptic Inputs. Neural Computation 12:8, 1789-1820. [Abstract] [PDF] [PDF Plus] 16. Quentin Pauluis , Stuart N. Baker . 2000. An Accurate Measure of the Instantaneous Discharge Probability, with Application to Unitary Joint-Event AnalysisAn Accurate Measure of the Instantaneous Discharge Probability, with Application to Unitary Joint-Event Analysis. Neural Computation 12:3, 647-669. [Abstract] [PDF] [PDF Plus] 17. Paul C. Bressloff , S. Coombes . 2000. Dynamics of Strongly Coupled Spiking NeuronsDynamics of Strongly Coupled Spiking Neurons. Neural Computation 12:1, 91-129. [Abstract] [PDF] [PDF Plus] 18. Hideo Hasegawa. 2000. Responses of a Hodgkin-Huxley neuron to various types of spike-train inputs. Physical Review E 61:1, 718-726. [CrossRef] 19. G. Mato. 1999. Stochastic resonance using noise generated by a neural network. Physical Review E 59:3, 3339-3343. [CrossRef] 20. Douglas A. Miller , Steven W. Zucker . 1999. Computing with Self-Excitatory Cliques: A Model and an Application to Hyperacuity-Scale Computation in Visual CortexComputing with Self-Excitatory Cliques: A Model and an Application to Hyperacuity-Scale Computation in Visual Cortex. Neural Computation 11:1, 21-66. [Abstract] [PDF] [PDF Plus] 21. Guido Bugmann, Chris Christodoulou, John G. Taylor. 1997. Role of Temporal Integration and Fluctuation Detection in the Highly Irregular Firing of a Leaky Integrator Neuron Model with Partial ResetRole of Temporal Integration and Fluctuation Detection in the Highly Irregular Firing of a Leaky Integrator Neuron Model with Partial Reset. Neural Computation 9:5, 985-1000. [Abstract] [PDF] [PDF Plus] 22. Herman P. Snippe. 1996. Parameter Extraction from Population Codes: A Critical AssessmentParameter Extraction from Population Codes: A Critical Assessment. Neural Computation 8:3, 511-529. [Abstract] [PDF] [PDF Plus] 23. K. Schmoltzi, H. Schuster. 1995. Introducing a real time scale into the Bak-Sneppen model. Physical Review E 52:5, 5273-5280. [CrossRef]

24. Marius Usher , Martin Stemmler , Christof Koch , Zeev Olami . 1994. Network Amplification of Local Fluctuations Causes High Spike Rate Variability, Fractal Firing Patterns and Oscillatory Local Field PotentialsNetwork Amplification of Local Fluctuations Causes High Spike Rate Variability, Fractal Firing Patterns and Oscillatory Local Field Potentials. Neural Computation 6:5, 795-836. [Abstract] [PDF] [PDF Plus] 25. Ying -Cheng Lai, Raimond L. Winslow, Murray B. Sachs. 1994. A model of selective processing of auditory-nerve inputs by stellate cells of the antero-ventral cochlear nucleus. Journal of Computational Neuroscience 1:3, 167-194. [CrossRef]

NOTE

Communicated by Andrew Barto

A Simplified Neural-Network Solution through Problem Decomposition: The Case of the Truck Backer-Upper Robert E. Jenkins Ben P. Yuhas* The Applied Physics Laboratory, The Johns Hopkins University, Baltimore, M D 21218 U S A Nguyen and Widrow (1990) demonstrated that a feedforward neural network could be trained to steer a tractor-trailer truck to a dock while backing up. The feedforward network they used to control the truck contained 25 hidden units and required tens of thousands of training examples. The training strategy was to slowly expand the region in which the controller could operate, by starting with positions close to the dock and after a few thousand iterations moving the truck a little farther away. We found that a very simple solution exists requiring only two hidden units in the controller. The solution was found by decomposing the problem into subtasks. The original goal was to use the solution to these subtasks to reduce training time. What we found was a complete solution. Nevertheless, this example demonstrates how building prior knowledge into the network can dramatically simplify the problem. The problem is composed of three subtasks. First, the truck must be oriented so that the trailer is nearly normal to the dock. This is accomplished by continuously driving Ltrailer to zero by tilting the cab in the proper direction. Then, having gotten itrailer to zero or near zero, the cab must be straightened out to keep it there. Thus a restoring spring constant on Ltrailer is needed to drive Ltrailer to 0, and a restoring spring constant on i c a b is needed to straighten out the cab as Ltrailer approaches 0. This subnetwork depends upon the values of itrailer and Lcab and is independent of position. Once the truck is correctly oriented, the remaining objective is to dock at Y = 0. An acceptable solution is found to be independent of X, as long as the truck is not started too close to the left edge. An X dependence could be introduced to amplify the movement to Y = 0 when the truck is closer to the dock. This X dependence is equivalent to turning up the gain on the transfer function, and would best be captured by a multiplicative control term (Xtimes Y) using 0-T units. The truck and the controller are shown in Figure 1. The specific weights used were adjusted based on observed performance, balancing between sensitivity and damping. This controller was able to successfully *Current address: Bellcore MRE 2E-330, Morristown, NJ 07962-1910 USA.

Neural Computation 4,647-649 (1992) @ 1992 Massachusetts Institute of Technology

Robert E. Jenkins and Ben P. Yuhas

648

cx,,,:

)

cab

cab angle

traller

angle

Y

Figure 1: The truck and the network used to control it. The state of the truck is described by the X, Y coordinates of the back of the trailer along with three angles: the trailer relative to the dock, Ltrailer, the cab relative to the trailer, i c a b , and the angle of the wheel relative to the cab iwheel. The weights used do not constitute a unique solution. Increasing the input to hidden weights while maintaining their ratio (for correct stability) can be approximately compensated for by reducing the hidden to output weights and vice versa.

back the truck up to the dock from all random locations we observed, as long as the back of the trailer started at least 0.7 times the trailer length away from the left wall. This example demonstrates how intuitively decomposing the problem can be used to initialize the neural network's weights. In this specific example, by identifying the components of the problem and embedding their solutions in the network, a solution to the larger problem was obtained.

Neural-Network Solution through Problem Decomposition

649

References Nguyen, D. H., and Widrow, B. 1990. Neural networks for self-learning control systems. I E E E Control Syst. Mag. 18-23.

Received 17 December 1991; accepted 4 March 1992.

Communicated by Christoph von der Malsburg

~

Learning to Segment Images Using Dynamic Feature Binding Michael C. Mozer Department of Computer Science and Institute of Cognitive Science, University of Colorado, Boulder, CO 803094430 U S A

Richard S. Zemel Department of Computer Science, University of Toronto, Toronto, Ontario M5S 1A4

Marlene Behrmann Department of Psychology and Faculty of Medicine and Rotman Research lnstitute of Baycrest Centre, University of Toronto, Toronto, Ontario M5S ZAl

Christopher K. 1. Williams Department of Computer Science, University of Toronto, Toronfo, Ontario M5S ZA4

Despite the fact that complex visual scenes contain multiple, overlapping objects, people perform object recognition with ease and accuracy. One operation that facilitates recognition is an early segmentation process in which features of objects are grouped and labeled according to which object they belong. Current computational systems that perform this operation are based on predefined grouping heuristics. We describe a system called MAGIC that learns how to group features based on a set of presegmented examples. In many cases, MAGIC discovers grouping heuristics similar to those previously proposed, but it also has the capability of finding nonintuitive structural regularities in images. Grouping is performed by a relaxation network that attempts to dynamically bind related features. Features transmit a complexvalued signal (amplitude and phase) to one another; binding can thus be represented by phase locking related features. MAGIC’S training procedure is a generalization of recurrent backpropagation to complexvalued units. 1 Introduction

Recognizing an isolated object in an image is a demanding Computational task. The difficulty is greatly compounded when the image contains Neural Computation 4, 650-665 (1992)


Learning to Segment Images

651

multiple objects because image features are not grouped according to which object they belong. Without the capability to form such groupings, it would be necessary to undergo a massive search through all subsets of image features. For this reason, most machine vision recognition systems include a component that performs feature grouping or image segmentation (e.g., Guzman 1968; Lowe 1985; Marr 1982). Psychophysical and neuropsychological evidence suggests that the human visual system performs a similar operation (Duncan 1984; Farah 1990; Kahneman and Henik 1981; Treisman 1982). Image segmentation presents a circular problem: Objects cannot be identified until the image has been segmented, but unambiguous segmentation of the image requires knowledge of what objects are present. Fortunately, object recognition systems do not require precise segmentation: Simple heuristics can be used to group features, and although these heuristics are not infallible, they suffice for most recognition tasks. Further, the segmentation-recognition cycle can iterate, allowing the recognition system to propose refinements of the initial segmentation, which in turn refines the output of the recognition system (Hinton 1981; Hanson and Riseman 1978; Waltz 1975). A multitude of heuristics have been proposed for segmenting images. Gestalt psychologists have explored how people group elements of a display and have suggested a range of grouping principles that govern human perception. For example, there is evidence for the grouping of elements that are close together in space or time, that appear similar, that move together, or that form a closed figure (Rock and Palmer, 1990). Computer vision researchers have studied the problem from a more computational perspective. They have investigated methods of grouping elements of an image based on nonaccidental regdarities-feature combinations that are unlikely to occur by chance when several objects are juxtaposed, and are thus indicative of a single object. Kanade (1981) describes two such regularities, parallelism and skewed symmetry, and shows how finding instances of these regularities can constrain the possible interpretations of line drawings. Lowe and Binford (1982) find nonaccidental, significant groupings through a statistical analysis of images. They evaluate potential feature groupings with respect to a set of heuristics such as collinearity, proximity, and parallelism. The evaluation is based on a statistical measure of the likelihood that the grouping might have resulted from the random alignment of image features. Boldt et al. (1989) describe an algorithm for constructing lines from short line segments. The algorithm evaluates the goodness of fit of pairs of line segments in a small neighborhood based on relational measures (collinearity, proximity, and contrast similarity). Well matched pairs are replaced by longer segments, and the procedure is repeated. In these earlier approaches, the researchers have hypothesized a set of grouping heuristics and then tested their psychological validity or computational utility. In our work, we have taken an adaptive approach to the

652

M. C. Mozer et al.

Figure 1: Examples of randomly generated two-dimensional geometric contours. problem of image segmentation in which a system learns how to group features based on a set of examples. We call the system MAGIC, an acronym for multiple-object adaptive grouping of image components. In many cases MAGIC discovers grouphg heuristics similar to those proposed in earlier work, but it also has the capability of finding nonintuitive structural regularities in images. MAGIC is trained on a set of presegmented images containing multiple objects. By "presegmented" we mean that each image feature is labeled as to which object it belongs. MAGIC learns to detect configurations of the image features that have a consistent labeling in relation to one another across the training examples. Identifying these configurations then allows MAGIC to label features in novel, unsegmented images in a manner consistent with the training examples. 2 The Domain

Our initial work has been conducted in the domain of two-dimensional geometric contours, including rectangles, diamonds, crosses, triangles, hexagons, and octagons. The contours are constructed from four primitive feature types-oriented line segments at 0", 45", 90", and 135"-and are laid out on a 25 x 25 grid. At each location on the grid are units, called feature units, that represent each of the four primitive feature types. In our present experiments, images contain two contours. We exclude images in which the two contours share a comrnon edge. This permits a unique labeling of each feature. Examples of several randomly generated images containing rectangles and diamonds are shown in Figure 1. 3 Representing Feature Labelings

Before describing MAGIC, we must first discuss a representation that allows for the labeling of features. von der Malsburg (1981; von der


653

Malsburg and Schneider 1986), Gray et al. (1989), Eckhorn et al. (1988), and Strong and Whitehead (1989), among others, have suggested a biologically plausible mechanism of labeling through temporal correlations among neural signals, either the relative timing of neuronal spikes or the synchronization of oscillatory activities in the nervous system. The key idea here is that each processing unit conveys not just an activation value-average firing frequency in neural terms-but also a second, independent value that represents the relative phase of firing. The dynamic grouping or binding of a set of features is accomplished by aligning the phases of the features. A flurry of recent work on populations of coupled oscillators (e.g., Baldi and Meir 1990; Grossberg and Somers 1991; Eckhorn et al. 1990; Kammen et al. 1990) has shown that this type of binding can be achieved using simple dynamic rules. However, most of this work assumes a relatively homogeneous pattern of connectivity among the oscillators and has not attempted to tackle problems in computer vision such as image segmentation, where each oscillator represents an image feature, and more selective connections between the oscillators are needed to simulate the selective binding of appropriate subsets of image features. A few exceptions exist (Goebel 1991a,b; Hummel and Biederman 1992; Lumer and Huberman 1991; Sporns et al. 1991); in these systems, the pattern of connectivity among oscillators is specified by simple predetermined grouping heuristics.' In MAGIC, the activity of a feature unit is a complex value with amplitude and phase components. The phase represents a labeling of the feature, and the amplitude represents the confidence in that labeling. The amplitude ranges from 0 to 1, with 0 indicating a complete lack of confidence and 1 indicating absolute certainty. There is no explicit representation of whether a feature is present or absent in an image. Rather, absent features are clamped off-their amplitudes are forced to remain at 0-which eliminates their ability to influence other units, as will become clear when the activation dynamics are presented later.

4 The Architecture When an image is presented to MAGIC, units representing features absent in the image are clamped off and units representing present features are assigned random initial phases and small amplitudes. MAGIC'S task is to assign appropriate phase values to the units. Thus, the network performs a type of pattern completion. 'In the Sporns et a / . model, the coupling strength between two connected units changes dynamically on a fast time scale, but this adaptation is related to achieving temporal correlations, not learning grouping principles.

654

M. C. Mozer et al.

Figure 2: The architecture of MAGIC. The lower (input) layer contains the feature units; the upper layer contains the hidden units. Each layer is arranged in a spatiotopic array with a number of different feature types at each position in the array. Each plane in the feature layer corresponds to a different feature type. The grayed hidden units are reciprocally connected to all features in the corresponding grayed region of the feature layer. The lines between layers represent projections in both directions.

The network architecture consists of two layers of units, as shown in Figure 2. The lower (input) layer contains the feature units, arranged in spatiotopic arrays with one array per feature type. The upper layer contains hidden units that help to align the phases of the feature units; their response properties are determined by training. There are interlayer connections, but no intralayer connections. Each hidden unit is reciprocally connected to the units in a local spatial region of all feature arrays. We refer to this region as a patch; in our current simulations, the patch has dimensions 4 x 4. For each patch there is a corresponding fixed-size pool of hidden units. To achieve uniformity of response across the image, the pools are arranged in a spatiotopic array in which neighboring pools respond to neighboring patches and the patch-to-pool weights are constrained to be the same at all locations in the array. The feature units activate the hidden units, which in turn feed back to the feature units. Through a relaxation process, the system settles on an assignment of phases to the features. One might consider an alternative architecture in which feature units were directly connected to one another (Hummel and Biederman 1992). However, this architecture is in principle not as powerful as the one we propose because it does not allow for higher order contingencies among features.


655

5 Network Dynamics

The dynamics of MAGIC are based on a mean-field approximation to a stochastic network of directional units, described in Zemel et al. (1992). A variant of this model was independently developed by Gislh et al. (1991). These papers provide a justification of the activation rule and error function in terms of an energy minimization formalism. The response of each feature unit i, x,,is a complex value in polar form, ( a J p, , ) , where a, is the amplitude and p J is the phase. Similarly, the response of each hidden unit j , y, has components (b,,4,). The weight connecting unit i to unit j , wIJ,is also complex valued, having components (p,,,O,,). The activation rule we propose is a generalization of the dot product to the complex domain. The net input to hidden unit j at time step t 1 is

+

net,(t

+ 1) = x ( t ) . w, =

EM)q

=

({(CJal(t)p]l

cos[pl(t) -

+

'12

sin[pl(t) -

(CJal(t)p,l

3

where the asterisk denotes the complex conjugate. The net input is passed through a squashing nonlinearity that maps the amplitude of the response from the range 0 + co to 0 - 1 but leaves the phase unaffected: net,(t) 11 [ ~ ( t ) ]

y,(t)

=

X T W

where m,(t) is the magnitude of the net input, Inet,(t)l, and Ik is the modified Bessel function of the first kind and order k. The squashing function Il(rn)/Io(m) is shown in Figure 3. The intuition underlying the activation rule is as follows. The amplitude (confidence) of a hidden unit, b,, should be monotonically related to how well the feature response pattern matches the hidden unit weight vector, just as in the standard real-valued activation rule. Indeed, one can readily see that if the feature and weight phases are equal ( p , = O,,), the rule for bl reduces to the real-valued case. Even if the feature and weight phases differ by a constant ( p , = O,, c), bJ is unaffected. This is a critical property of the activation rule: Because absolute phase values have no intrinsic meaning, the response of a unit should depend only on the relative phases. That is, its response should be rotation invariant. The activation rule achieves this by essentially ignoring the average difference

+

M. C. Mozer et al.

656

a l r

Figure 3: The squashing function G = I ~ ( m ) / I ~ ( m )The . amplitude of the net input to a unit is passed through this function to obtain the output amplitude.

in phase between the feature units and the weights. The hidden phase,

91, reflects this average difference? The flow of activation from the hidden layer to the feature layer follows the same dynamics as the flow from the feature layer to the hidden layer: net,(t

+ 1)= y(t + 1). w,

and net,(t)11 [ml(t)l m d t ) I0 b I ( f ) J if feature i is present in the image, or xl(t) = 0 otherwise. Note that update is sequential by layer: the feature units activate the hidden units, which then activate the feature units. In MAGIC, the weight matrix is constrained to be Hermitian, i.e., w,I = "5. This is a generalization of weight symmetry to the complex domain. Weight symmetry ensures that MAGIC will converge to a fixed point. The proof of this is a generalization of Hopfield's (1984) result to complex units, discrete-time update, and a two-layer architecture with sequential layer updates and no intralayer connections. x z ( t )=

~~

2To elaborate, the activation rule produces a 9, that yields the minimum of the following expression: [a1cosp,

d, =

~

P,~cos(Q,,

+ q,)]* + [aI sinp, - P,, sin(@,,+ 9,)12

I

This is a measure of the distance between the feature and weight vectors given a free parameter 9, that specifies a global phase shift of the weight vector.


657

6 Learning Algorithm During training, we would like the hidden units to learn to detect configurations of features that reliably indicate phase relationships among the features. For instance, if the contours in the image contain extended horizontal lines, one hidden unit might learn to respond to a collinear arrangement of horizontal segments. Because the unit's response depends on the phase pattern as well as the activity pattern, it will be strongest if the segments all have the same phase value. We have experimented with a variety of algorithms for training MAGIC, including an extension of soft competitive learning (Nowlan 1990)to complex-valued units, recurrent backpropagation (Almeida 1987; Pineda 19871, backpropagation through time (Rumelhart et al. 19861, a backpropagation autoencoder paradigm in which patches of the image are processed independently, and an autoencoder in which the patches are processed simultaneously and their results are combined. The algorithm with which we have had greatest success, however, is a relatively simple single-step error propagation algorithm. It involves running the network for a fixed number of iterations and, for each iteration, using backpropagation to adjust the weights so that the feature phase pattern better matches a target phase pattern. Each training trial proceeds as follows: 1. A training example is generated at random. This involves selecting two contours and instantiating them in an image. The features of one contour have target phase 0" and the features of the other contour have target phase 180".

2. The training example is presented to MAGIC by setting the initial amplitude of a feature unit to 0.1 if its corresponding image feature is present, or clamping it at 0.0 otherwise. The phases of the feature units are set to random values in the range 0" to 360". 3. Activity is allowed to flow from the feature units to the hidden units and back to the feature units.

4. The new phase pattern over the feature units is compared to the target phase pattern (see step l),and an error measure is computed:

where in, is the magnitude of the net input to feature unit i, pi is the actual phase of unit i, and p i is the target phase. This is a log likelihood error function derived from the formalism described in Zemel et al. (1992). In this formalism, the activities of units represent a probability distribution over phase values. The error function is

M. C. Mozer et al.

658

the asymmetric divergence between the actual and target phase distributions. The aim is to minimize the difference between the target and actual phases and to maximize the amplitude, or confidence, of the response. The error measure factors out the absolute difference between the target and actual phases. That is, E is minimized when j j , - p l is equal for all i, regardless of the value of fj, - p l . 5. Using a generalization of backpropagation to complex valued units, error gradients are computed for the feature-to-hidden and hiddento-feature weights.

6. Steps 3-5 are repeated for a maximum of 30 iterations. The trial is terminated if the error increases on five consecutive iterations. 7. Weights are updated by an amount proportional to the average error gradient over iterations. The constraint that wil = zuI; is enforced in proby modifying w , ~in proportion to V,, Vf and modifying w,, portion to V; + V,, where V, denotes the gradient with respect to the weight to i from j. To achieve a translation-invariant response of the hidden units, hidden units of the same "type" responding to different regions of the image are constrained to have the same weights. This is achieved by having single set of underlying weight parameters that is replicated across the hidden layer. The appropriate gradient descent algorithm for these parameters is to adjust them in proportion to the sum of the gradients with respect to each of their instantiations.

+

The algorithm is far less successful when a target phase pattern is given just on the final iteration or final k iterations, rather than on each iteration. Surprisingly, the algorithm operates little better when error signals are propagated back through time. The simulations reported below use a learning rate parameter of 0.005 for the amplitudes and 0.02 for the phases. On the order of 10,000 learning trials are required for stable performance, although MAGIC rapidly picks up on the most salient aspects of the domain. 7 Simulation Results

We trained a network with 20 hidden units per pool on examples like those shown in Figure 1. The resulting weights are shown in Figure 4. Each hidden unit attempts to detect and reinstantiate activity patterns that match its weights. One clear and prevalent pattern in the weights is the collinear arrangement of segments of a given orientation, all having the same phase value. When a hidden unit having weights of this form responds to a patch of the feature array, it tries to align the phases of the patch with the phases of its weight vector. By synchronizing the phases


659

-

1

Figure 4: Complex feature-to-hidden connection weights learned by MAGIC. In this simulation, there are connections from a 4 x 4 patch of the image to a pool of 20 hidden units. (Theseconnections are replicated for each patch in the image to achieve a uniformity of hidden unit response.) The connections feeding into each hidden unit are presented on a light gray background. Each hidden unit has a total of 64 incoming w e i g h t s 4 x 4 locations in its receptive field and four feature types at each location. The weights are further grouped by feature type (dark gray background), and for each feature type they are arranged in a 4 x 4 pattern homologous to the image patch itself. The area of a circle is proportional to the amplitude of the corresponding weight, the orientation of the internal tick mark represents the phase angle. Due to the symmetry constraint, hidden-to-feature weights (not shown) mirror the feature-to-hidden weights.

M. C. Mozer et al.

660

1

1; I

Iterabon 6

Iterabon 10

..-; 1

h

-

Iterauon 25

Figure 5: An example of MAGIC segmenting an image. The "iteration" refers to the number of times activity has flowed from the feature units to the hidden units and back. The phase value of a feature is represented by a gray level. The cyclic phase continuum can be approximated only by a linear gray level continuum, but the basic information is conveyed nonetheless. of features, it acts to group the features. Thus, one can interpret the weight vectors as the rules by which features are grouped. Whereas traditional grouping principles indicate the conditions under which features should be bound together as part of the same object, the grouping principles learned by MAGIC also indicate when features should be segregated into different objects. For example, the weights of the vertical and horizontal segments are generally 180" out of phase with the diagonal segments. This allows MAGIC to segregate the vertical and horizontal features of a rectangle from the diagonal features of a diamond (see Fig. 1, left panel). We had anticipated that the weights to each hidden unit would contain two phase values at most because each image patch contains at most two objects. However, some units make use of three or more phases, suggesting that the hidden unit is performing several distinct functions. As is the usual case with hidden unit weights, these patterns are difficult to interpret. Figure 5 presents an example of the network segmenting an image. The image contains two rectangles. The top left panel shows the features


661

of the rectangles and their initial random phases. The succeeding panels show the network's response during the relaxation process. The lower right panel shows the network response at equilibrium. Features of each object have been assigned a uniform phase, and the two objects are 180" out of phase. The task here may appear simple, but it is quite challenging due to the illusory rectangle generated by the overlapping rectangles. 8 Alternative Representation of Feature Labeling

To perform the image segmentation task, each feature unit needs to maintain two independent pieces of information: a label assigned to the feature and a measure of confidence associated with the label. In MAGIC, these two quantities are encoded by the phase and amplitude of a unit, respectively. This polar representation is just one of many possible encodings, and requires some justification due to the complexity of the resulting network dynamics. An alternative we have considered-which seems promising at first glance but has serious drawbacks-is the rectangular coordinate analog of the polar representation. In this scheme, a feature unit conveys values indicating belief in the hypotheses that the feature is part of object A or object B, where A and B are arbitrary names. For example, the activities ( 1 , O ) and ( 0 , l ) indicate complete confidence that the feature belongs to object A or B,respectively, (0,O) indicates that nothing is known about which object the feature belongs to, and intermediate values indicate intermediate degrees of confidence in the two hypotheses. The rectangular and polar representations are equivalent in the sense that one can be transformed into the other.3 The rectangular scheme has two primary benefits. First, the activation dynamics are simpler. Second, it allows for the simultaneous and explicit consideration of multiple labeling hypotheses, whereas the polar scheme allows for the consideration of only one label at a time. However, these benefits are obtained at the expense of presuming a correspondence between absolute phase values and objects. (In the rectangular scheme we described, A and B always have phases 0" and 90°, respectively, obtained by transforming the rectangular coordinates to polar coordinates.) The key drawback of absolute phase values is that a local patch of the image cannot possibly determine which label is correct. A patch containing, say, several collinear horizontal segments can determine only that the segments should be assigned the same label. Preliminary simulations indicate that the resulting ambiguity causes severe difficulties in processing. In contrast, the polar scheme allows the network to express the relative labelings of two segments-e.g., that they should be assigned the same label-without needing to specify the particular label. 'Yann Le Cun (personal communication, 1991) has independently developed the notion of using the rectangular encoding scheme in the domain of adaptive image segmentation.

M. C. Mozer et al.

662

9 Current Directions

We are currently extending MAGIC in several directions, which we outline here. 0

0

0

We have not addressed the question of how the continuous phase representation is transformed into a discrete object label. One may simply quantize the phase values such that all phases in a given range are assigned the same label. This quantization step has the extremely interesting property that it allows for a hierarchical decomposition of objects. If the quantization is coarse, only gross phase differences matter, allowing one object to be distinguished from another. As the quantization becomes finer, an object is divided into its components. Thus, the quantization level in effect specifies whether the image is parsed into objects, parts of objects, parts of parts of objects, etc. This hierarchical decomposition of objects can be achieved only if the phase values reflect the internal structure of an object. For example, in the domain of geometric contours, MAGIC would not only have to assign one contour a different phase value than another, but it would also have to assign each edge composing a contour a slightly different phase than each other edge (assuming that one considers the edges to be the "parts" of the contour). Somewhat surprisingly, MAGIC does exactly this because the linkage between segments of an edge is stronger than the linkage between two edges. This is due to the fact that collinear features occur in images with much higher frequency than do corners. Thus, the relative frequency of feature configurations leads to a natural principle for the hierarchical decomposition of objects. Although MAGIC is trained on pairs of objects, it has the potential of processing more than two objects at a time. For example, with three overlapping objects, MAGIC attempts to push each pair 180" out of phase but ends up with a best constraint satisfaction solution in which each object is 120" out of phase with each other. We are exploring the limits of how many objects MAGIC can process at a time. Spatially local grouping principles are unlikely to be sufficient for the image segmentation task. Indeed, we have encountered incorrect solutions produced by MAGIC that are locally consistent but globally inconsistent. To solve this problem, we are investigating an architecture in which the image is processed at several spatial scales simultaneously. Fine-scale detectors respond to the sort of detail shown in Figure 4, while coarser-scale detectors respond to more global structure but with less spatial resolution.


663

Simulations are under way to examine MAGIC’S performance on real-world images-overlapping handwritten letters and digitswhere it is somewhat less clear to which types of patterns the hidden units should respond. 0

Behrmann et al. (1992) are conducting psychological experiments to examine whether limitations of the model match human limitations.

Acknowledgments This research was supported by NSF Presidential Young Investigator award IRI-9058450, Grant 90-21 from the James S. McDonnell Foundation, and DEC external research Grant 1250 to MM; by a National Sciences and Engineering Research Council Postgraduate Scholarship to Rz; and by an NSERC operating grant to MB. Our thanks to Paul Smolensky, Radford Neal, Geoffrey Hinton, and Jurgen Schmidhuber for helpful comments regarding this work.

References Almeida, L. 1987. A learning rule for asynchronous perceptrons with feedback in a combinatorial environment. Proceedings of the I E E E First Annual lnternational Conference on Neural Networks, Vol. 2, M. Caudill and C. Butler, eds., pp. 609-618. EEE Publishing Services, San Diego, CA. Baldi, P., and Meir, R. 1990. Computing with arrays of coupled oscillators: An application to preattentive texture discrimination. Neural Cornp. 2, 45&471. Behrmann, M., Zemel, R. S., and Mozer, M. C. 1992. Perceptual organization and object-based attention. Manuscript in preparation. Boldt, M., Weiss, R., and Riseman, E. 1989. Token-based extraction of straight lines. IEEE Transact. Syst. Man Cybern. 19, 1581-1594. Duncan, J. 1984. Selective attention and the organization of visual information. j . Exp. Psychol. General 113, 501-517. Eckhorn, R., Bauer, R., Jordan, W., Brosch, M., Kruse, W., Munk, M., and Reitboek, H. J. 1988. Coherent oscillations: A mechanism of feature linking in the visual cortex? Biol. Cybern. 60, 121-130. Eckhorn, R., Reitboeck, H. J., Arndt, M., and Dicke, P. 1990. Feature linking via synchronization among distributed assemblies: Simulations of results from cat visual cortex. Neural Comp. 2, 293-307. Farah, M. J. (1990). Visual Agnosia. The MIT Press/Bradford Books, Cambridge, MA. Gislen, L., Peterson, C., and Soderberg, 8. 1991. Rotor Neurons-Basic Formalism and Dynamics (LU TP 91-21). University of Lund, Department of Theoretical Physics, Lund, Sweden. Goebel, R. 1991a. An oscillatory neural network model of visual attention, pattern recognition, and response generation. Manuscript in preparation.

664

M. C. Mozer et al.

Goebel, R. 1991b. The role of attention and short-term memory for symbol manipulation: A neural network model that learns to evaluate simple LISP expressions. In Cognition and Computer Programming, K. F. Wender, F. Schmalhofer, and H. D. Boecker, eds. Ablex Publishing Corporation, Norwood, NJ. Gray, C. M., Koenig, P., Engel, A. K., and Singer, W. 1989. Oscillatory responses in cat visual cortex exhibit intercolumnar synchronization which reflects global stimulus properties. Nature (London) 338, 334-337. Grossberg, S., and Somers, D. 1991. Synchronized oscillations during cooperative feature linking in a cortical model of visual perception. Neural Networks 4,453466. Guzman, A. 1968. Decomposition of a visual scene into three-dimensional bodies. AFIPS Fall Joint Comput. Conf. 33, 291-304. Hanson, A. R., and Riseman, E. M. 1978. Computer Vision Systems. Academic Press, New York. Hinton, G. E. 1981. A parallel computation that assigns canonical object-based frames of reference. In Proceedings of the Seventh lnternational Joint Conference on Artificial Intelligence, pp. 683-685. Morgan Kaufmann, Los Altos, CA. Hopfield, J. J. 1984. Neurons with graded response have collective computational properties like those of two-state neurons. Proc. Natl. Acad. Sci. U.S.A. 81, 3088-3092. Hummel, J. E., and Biederman, I. 1992. Dynamic binding in a neural network for shape recognition. Psychol. Rev., in press. Kahneman, D., and Henik, A. 1981. Perceptual organization and attention. In Perceptual Organization, M. Kubovy and J. R. Pomerantz, eds., pp. 181-211. Erlbaum, Hillsdale, NJ. Kammen, D., Koch, C., and Holmes, P. J. 1990. Collective oscillations in the visual cortex. In Advances in Neural lnformation Processing Systems2, D. S. Touretzky, ed., pp. 76-83. Morgan Kaufmann, San Mateo, CA. Kanade, T. 1981. Recovery of the three-dimensional shape of an object from a single view. Artificial Intell. 17,409-460. Lowe, D. G. 1985. Perceptual Organization and Visual Recognition. Kluwer Academic Publishers, Boston. Lowe, D. G., and Binford, T. 0. 1982. Segmentation and aggregation: An approach to figure-ground phenomena. Proceedings of the DARPA IUS Workshop, pp. 168-178. Palo Alto, CA. Lumer, E., and Huberman, 8. A. 1992. Binding hierarchies: A basis for dynamic perceptual grouping. Neural Comp. 4, 341-355. Marr, D. 1982. Vision. Freeman, San Francisco. Nowlan, S. J. 1990. M a x likelihood competition in RBF networks. Tech. Rep. CRGTR-90-2. Toronto, Canada: University of Toronto, Department of Computer Science, Connectionist Research Group. Pineda, F. 1987. Generalization of back propagation to recurrent neural networks. Phys. Rev. Lett. 19, 2229-2232. Rock, I., and Palmer, S. E. 1990. The legacy of Gestalt psychology. Sci. Amer. 263, 84-90. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing: Ex-


665

plorations in theMicvostructure of Cognition. VolumeI: Foundations, D. E. Rumelhart and J. L. McClelland, eds., pp. 318-362. The MIT Press/Bradford Books, Cambridge, MA. Sporns, O., Tononi, G., and Edelman, G. M. 1991. Modeling perceptual grouping and figure-ground segregation by means of active reentrant connections. Proc. Natl. Acad. Sci. 88, 129-133. Strong, G. W., and Whitehead, B. A. 1989. A solution to the tag-assignment problem for neural networks. Behav. Brain Sci. 12, 381433. Treisman, A. 1982. Perceptual grouping and attention in visual search for features and objects. J. Exp. Psychol. Human Percept. Perform. 8, 194-214. von der Malsburg, C. 1981. The correlation theory of brain function (Internal Report 81-2). Goettingen: Department of Neurobiology, Max Planck Institute for Biophysical Chemistry. von der Malsburg, C., and Schneider, W. 1986. A neural cocktail-party processor. Bid. Cybern. 54, 29-40. Waltz, D. A. 1975. Generating semantic descriptions from drawings of scenes with shadows. In The Psychology of Computer Vision, P. H. Winston, ed., pp. 19-92. McGraw-Hill, New York. Zemel, R. S., Williams, C. K. I., and Mozer, M. C. 1992. Adaptive networks of directional units. Submitted for publication.

Received 29 August 1991; accepted 25 February 1992.

This article has been cited by: 2. S. Weng, H. Wersing, J.J. Steil, H. Ritter. 2006. Learning Lateral Interactions for Feature Binding and Sensory Segmentation From Prototypic Basis Interactions. IEEE Transactions on Neural Networks 17:4, 843-862. [CrossRef] 3. D. Wang. 2005. The Time Dimension for Scene Analysis. IEEE Transactions on Neural Networks 16:6, 1401-1426. [CrossRef] 4. Mary A. Peterson, Daniel W. Lampignano. 2003. Implicit Memory for Novel Figure-Ground Displays Includes a History of Cross-Border Competition. Journal of Experimental Psychology: Human Perception and Performance 29:4, 808-822. [CrossRef] 5. Michael C. Mozer. 2002. Frames of reference in unilateral neglect and visual perception: A computational perspective. Psychological Review 109:1, 156-185. [CrossRef] 6. Shaun P. Vecera, Edward K. Vogel, Geoffrey F. Woodman. 2002. Lower region: A new cue for figure-ground assignment. Journal of Experimental Psychology: General 131:2, 194-205. [CrossRef] 7. Richard S. Zemel, Marlene Behrmann, Michael C. Mozer, Daphne Bavelier. 2002. Experience-dependent perceptual grouping and object-based attention. Journal of Experimental Psychology: Human Perception and Performance 28:1, 202-217. [CrossRef] 8. Edward K. Vogel, Geoffrey F. Woodman, Steven J. Luck. 2001. Storage of features, conjunctions, and objects in visual working memory. Journal of Experimental Psychology: Human Perception and Performance 27:1, 92-114. [CrossRef] 9. Shaun P. Vecera , Kendra S. Gilds . 1998. What Processing Is Impaired in Apperceptive Agnosia? Evidence from Normal SubjectsWhat Processing Is Impaired in Apperceptive Agnosia? Evidence from Normal Subjects. Journal of Cognitive Neuroscience 10:5, 568-580. [Abstract] [PDF] [PDF Plus] 10. Robert L. Goldstone. 1998. PERCEPTUAL LEARNING. Annual Review of Psychology 49:1, 585-612. [CrossRef] 11. Hermina J.M. Tabachneck-Schijf, Anthony M. Leonardo, Herbert A. Simon. 1997. CaMeRa: A Computational Model of Multiple Representations. Cognitive Science 21:3, 305-350. [CrossRef] 12. DeLiang Wang, David Terman. 1997. Image Segmentation Based on Oscillatory CorrelationImage Segmentation Based on Oscillatory Correlation. Neural Computation 9:4, 805-836. [Abstract] [PDF] [PDF Plus]

Communicated by Haim Sompolinsky

Stimulus-Dependent Assembly Formation of Oscillatory Responses: 111. Learning Peter Konig Bernd Janosch Thomas B. Schillen Max-Planck-lnstitut filr Hirnforschung, Deutschordenstrasse 46,6000 Frankfurt 71, Germany

A temporal structure of neuronal activity has been suggested as a potential mechanism for defining cell assemblies in the brain. This concept has recently gained support by the observation of stimulusdependent oscillatory activity in the visual cortex of the cat. Furthermore, experimental evidence has been found showing the formation and segregation of synchronously oscillating cell assemblies in response to various stimulus conditions. In previous work, we have demonstrated that a network of neuronal oscillators coupled by synchronizing and desynchronizing delay connections can exhibit a temporal structure of responses, which closely resembles experimental observations. In this paper, we investigate the self-organization of synchronizing and desynchronizing coupling connections by local learning rules. Based on recent experimental observations, we modify synchronizing connections according to a two-threshold learning rule, involving synaptic potentiation and depression. This rule is generalized to its functional inverse for weight changes of desynchronizing connections. We show that after training, the resulting network exhibits stimulus-dependent formation and segregation of oscillatory assemblies in agreement with the experimental data. These results indicate that local learning rules during ontogenesis can suffice to develop a connectivity pattern in support of the observed temporal structure of stimulus responses in cat visual cortex.

1 Introduction During recent years, the temporal structure of neuronal activity has attracted much interest due to its potential role in visual processing (von der Malsburg 1981; Singer 1990). Based on theoretical considerations, it has been proposed that the temporal correlation of neuronal responses could be used by the mammalian brain to solve the binding problem (von der Malsburg 1981). In particular, the synchronization of oscillatory Neural Computation 4,666-681 (1992) @ 1992 Massachusetts Institute of Technology

Stimulus-DependentOscillatory Responses

667

responses would allow the unique definition of neuronal assemblies representing sensory stimuli (von der Malsburg and Schneider 1986). The presence of stimulus-driven oscillations of neuronal activity has been found in the olfactory bulb of the rabbit and in the cat visual cortex (Freeman 1975; Gray and Singer 1987,1989; Eckhorn et al. 1988). Furthermore, both the stimulus-dependent synchronization and desynchronization of oscillatory neuronal responses have recently been demonstrated: light bars moving collinearly were shown to induce synchronous activity of neuronal responses (Gray et al. 1989). In contrast, two superimposed light bars moving in different directions activated two synchronously oscillating cell assemblies that exhibited no constant phase relationship among one another (Engel et al. 1991). Previously, we presented a network model of coupled delayed nonlinear oscillators that exhibits stimulus-dependent assembly formation of oscillatory responses in close analogy to experimental observations (Konig and Schillen 1990,1991; Schillen and Konig 1990, 1991). The network incorporates two types of delay connections that actively synchronize and desynchronize oscillatory assemblies. As discussed in Schillen and Konig (1991), we consider the stimulus-dependent active desynchronization to be an important complement to synchronizing mechanisms if the temporal structure of neuronal activity is to be employed in sensory processing. In this paper, we investigate the self-organization of synchronizing and desynchronizing delay connections by local learning rules. Motivated by recent experimental observations in rat visual cortex (Artola et al. 1990) we modify synchronizing connections according to a twothreshold learning rule, involving synaptic potentiation and depression. We generalize this modification rule to its functional inverse for weight changes of the desynchronizing connections. We show that the resulting network exhibits stimulus-dependent assembly formation of oscillatory responses which are similar to those found in physiological experiments (Engel et al. 1991). 2 Oscillatory Network

Model

We investigate the temporal structure of neuronal activity in a network of coupled delayed nonlinear oscillators (Konig and Schillen 19911. Elementary oscillators consist of an excitatory unit u,, coupled with delay ~~i to an inhibitory unit ui, which projects back to unit u, with delay ri, (Fig. 1A). When externaI stimulus input is applied to the excitatory unit ue, this system exhibits a stimulus-dependent transfer from a stable fixed point to a limit cycle oscillation. For details of the dynamics of the system refer to Konig and Schillen (1991). Oscillatory elements are coupled by synchronizing and desynchronizing delay connections as described before: Synchronizing connections

668

P. Konig, B. Janosch, and T. B. Schillen

B

Figure 1: (A) Basic oscillatory element implemented by coupling an excitatory unit (0) with an inhibitory unit (0) using delay connections. An additional unit (0) allows for external input of a stimulus. t, time; x ( t ) , unit activity; F ( x ) , sigmoidal output function with threshold 8; w,coupling weight; T , delay time; i e ( t ) , external input. Subscripts: e, excitatory unit; i, inhibitory unit. For further details see Konig and Schillen (1991). (B)Oscillatory elements coupled by two types of delay connections. Connections between the excitatory unit of one oscillator and the inhibitory unit of another (w:),dashed) synchronize the activity of the respective oscillators (Konig and Schillen 1991). Connections between excitatory units (wg), dotted) are used to desynchronize oscillatory activity (Schillen and Konig 1991). Note that the panel is meant to demonstrate only the employed types of coupling connections and not the model's connectivity pattern. Notation: Throughout this paper, w(')denotes the coupling weight between oscillators which are r oscillator positions apart (r-nearest neighbor coupling).

(~2')

originate from the excitatory unit of one oscillator and terminate at the inhibitory unit of another (Fig. lB, dashed) (Konig and Schillen 1991). Couplings between the excitatory unit of one oscillator and the excitatory unit of an other (w:)) are used to achieve desynchronization of oscillatory activity (Fig. lB, dotted) (Schillen and Konig 1991). We choose the configuration of the experiment by Engel et al. (1991) for our investigation of self-organization in an oscillatory network. In this experiment, several cells with receptive fields of different preferred orientations at closely overlapping retinal locations were stimulated with either a single or two overlapping moving light bars (Fig. 48). Using these two stimulus conditions, both synchronization and desynchronization of oscillatory neuronal responses could be observed.

Stimulus-Dependent Oscillatory Responses

669

In our model, we represent this experimental situation (Engel et al. 1991) by a one-dimensional chain of 16 oscillators (Fig. 4A). Each oscillator represents a population of cells with one particular orientation preference. For all the oscillators, we assume a continuous sequence of preferred orientations in steps of 11.25'. The receptive fields of all oscillators are located at the same retinal position. We simulate stimulus light bars by distributions of input activity corresponding to a gaussian tuning of preferred orientations with 30" tuning width (Fig. 4C, left). The orientation of a stimulus bar is represented by centering the input distribution appropriately. In the network's initial configuration, each oscillator is coupled by synchronizing and desynchronizing delay connections with all other oscillators in the chain. Initially, coupling weights wz)and wg) are homogeneously distributed. Note that, thus, no topology is imposed a priori on the coupling connections. Simulations were performed with the MENS modeling environment (Schillen 1991) on a VAXstation 3100. 3 Learning Rules

We train the network with a pseudorandom sequence of input stimuli corresponding to the model's 16 different preferred orientations. Each stimulus is presented for a period of At - 200msec. Synaptic weight changes occur at the end of each stimulus presentation. Based on recent observations by Artola et al. (1990) in slices from rat visual cortex, we use the following two-threshold learning rule (ABS rule) for weight changes of the synchronizing connections in our network:

(~2))

Au(t) = €iiPre(t)f~~s(iipost(t)) where t is time, Aw is the synaptic weight change, E is a rate constant, ,i and iipost are respective mean activities of the pre- and postsynaptic unit, and

{:::

ii 5 0,

fABs(il) =

C1 : 01 < 6 5 02 02< 3

with C1 < Co < 0 < C2

is a two-threshold function of the postsynaptic activity with thresholds O1 and 0, (Fig. 2, solid; Artola etal. 1990). We calculate a unit's mean activity u during a period of stimulus presentation according to 1 t+At ii(t) = a(t')dt'+ao At t where the offset a0 ensures the positivity of the integral. In physiological terms, lz could correspond to an indicator of integrated neuronal activity (At - 200 msec) such as intracellular concentrations of second messengers

/


670

r------I

I I

I I

-------

1

I

t

I

-1

I

L

-

a

I I

-------

Figure 2: Functions fAsS of postsynaptic activity employed with the twothreshold ABS (solid) and anti-ABS (dashed) learning rules. ABS rule: Mean postsynaptic activity a exceeding threshold 02 leads to synaptic weight potentiation (LTP in Artola et al. 1990). Threshold 62 is above the ”spiking” threshold 0 of the output function F ( x ) of units in our model. Activity u intermediate between thresholds 81 and & leads to a depression of synaptic weights (LTD in Artola et al. 1990). Threshold O1 is below ”spiking” threshold 0. For little postsynaptic activity, ii 5 61, we assume a small negative value forfABS. This provides a slow degeneration of synaptic connections that consistently fail to excite their postsynaptic target cells. ABS rule learning is employed for modifications of synchronizing delay connections in our model. Anti-ABS rule: For anti-ABS weight changes we exchange the regions of potentiation and depression of the ABS learning rule: 82 < a, depression; 81 < ii 5 02, potentiation. Anti-ABS learning applies to modifications of desynchronizing delay connections in our network.

like Ca2+ or IP3 (Berridge and Irvine 1989). Sign changes of synaptic weights are excluded by applying a lower boundary to the modified weights: w(t At) = max[O, w(t) A w ( t ) ] . This ensures that a synapse maintains its excitatory specificity during training. Even though synaptic modifications are fairly balanced for each unit, total synaptic weights can slowly diverge. Therefore, we keep a unit’s total synaptic weight constant by normalization (von der Malsburg 1973). For the network‘s desynchronizing connections (wg)), we generalize the above modification scheme to a corresponding two-threshold antiABS learning rule. This is achieved by modifying function f A ~ sas depicted in Figure 2 (dashed, C2 < Co < 0 < C,). Note that we allow the ABS learning rule of weights wz)to also affect each oscillator’s intrinsic excitatory connection w2’.However, we do not

+

+


671

modify inhibitory connections w!:)since there is currently no physiological evidence for plasticity of inhibitory synapses. 4 Learning Synchronizing and Desynchronizing Delay Connections

When the network is presented with training stimuli, the described learning functions lead to rapid changes of synaptic weights. After several hundred stimulus presentations only those synchronizing connections that couple oscillators differing by not more than 22.5" in their preferred orientations maintain nonzero synaptic weights. For all other synchronizing connections synaptic weights degenerate to near zero (Fig. 3A). The desynchronizing connections develop a bipartite weight distribution that provides active desynchronization between oscillators that differ by 22.5", . . . ,56.25" in their orientation preferences (Fig. 3B). Since the dynamics of desynchronizing weight changes is more susceptible to imbalances in the sequence of training stimuli, we use a reduced learning rate tee= 0.1 eel. Figure 3C shows the asymptotic weight distributions after 3200 stimulus presentations at the end of the training phase. These distributions closely resemble those implemented in a previous network (Schillen and Konig 1991). Note that for the synchronizing connections the tuning width of synaptic weights (orientation difference between maximum and half maximum weight; l5? corresponds to only about half the width of the orientation tuning assumed for each neuronal population (30"). After training is completed we disable further weight changes. Then we test the network with (1) a single stimulus (Fig. 4, left column), and (2) two superimposed stimuli of orthogonal orientation (Fig. 4, right column) as in the experimental situation (Engel et a2. 1991; Schillen and Konig 1991). With a single stimulus, the activity of all responding cells is well synchronized. This identifies the neuronal populations that belong to the single oscillatory assembly representing the stimulus (Fig. 4E, left). In the case of the two superimposed stimuli, each stimulus is represented by its respective assembly of synchronously oscillating cells. However, because of the desynchronizing connections a rapidly varying phase relation is found between the two assemblies (Fig. 4D, right). Averaging over 20 epochs results in a reduced cross-correlationfunction with a random phase relation (Fig. 4E, right). Without this active desynchronization the oscillatory responses to the superimposed stimuli become completely synchronized (data not shown). [See Schillen and Konig (1991) for a detailed discussion of the desynchronization of assemblies defined by temporal structure of neuronal activity.] The observed synchronization and desynchronization of oscillatory assemblies in this model agrees well with the experimental evidence (Engel et a2. 1991). In a second investigation, we train a network with pairs of input stimuli corresponding to random combinations of two different stimulus orientations. In order to provide a better resolution of orientation differ-

672

A

P. Konig, 8. Janosch, and T. B. Schillen

0.8I

1

source unit

16 source unit

32

Figure 3: Development of synaptic weights during training. (A) Development of weights wi)of synchronizing connections. Depicted are the coupling weights from all 16 excitatory units in the network (Fig. 4A) to inhibitory unit 8. Each presentation of a training stimulus lasts 200msec. (B) Development of weights &) of desynchronizing connections. The panel shows the weights from the network‘s other excitatory units to excitatory unit 8. (C) Weight distributions for the coupling connections shown in (A) and (B) after 3200 stimulus presentations. wi),solid; wL),dashed. Parameters: for modifying synchronizing connections: 01 = 1.0, 02 = 2.4, CO = -3.0, C1 = -10.0, C2 = 8.0; for modifying desynchronizing connections: 01 = 1.2, 0, == 2.6, CO= -3.0, C1 = 8.0, C2 = -10.0; C,w$)= 3.35; C,w$:’= 3.35; wj:’ = -1.0; input strength 0.55; other parameters as in Schillen and Konig (1991). (D) Weight distributions analogous to (C) after training a network of 32 oscillatory elements with superimposed input stimuli of randomly paired orientations. Solid, dashed: ABS learning rule; doffed,dot dashed: one-threshold Hebb-like learning rule (Ca = C1). ences, we use a network comprising 32 oscillatory elements. We normalize the superimposed distributions of input activity of the stimulus pairs to the same maximum activity as that of a single stimulus. Training with these randomly paired stimuli leads to an equivalent weight distribution of synchronizing and desynchronizing connections as the training with

673


A

B

90'

135'

0'

45'

@

time

time

Figure 4: Stimulus-dependent assembly formation. (A) 16 oscillators representing orientation-selective neuronal populations with closely overlapping receptive field locations. (B) Stimulus conditions of one single (left) and two superimposed (right) light bars. (C) Distributions of input activity representing the two stimulus conditions in (B). (D) Activity traces from the two units hatched in (A). (E) Mean normalized auto- (dashed) and cross- (solid) correlations of activity from the units shown in (D). Mean of 20 epochs of 20 T; T , period length of isolated oscillator. Normalization by geometric mean of the two auto-correlations. Parameters: input activity as specified in (C), weight distributions zu$) and zug) as shown in Figure 3C, other parameters as in Schillen and Konig (1991).

674


single stimuli (Fig. 3D, thick lines). As expected, testing the network after training with (1) the single stimulus and (2) the two superimposed orthogonal stimuli shows again the synchronization and desynchronization of oscillatory assemblies in agreement with the experimental observations (Engel et al. 1991). Next, we use the previous simulation to assess the dependence of the obtained connectivity pattern on the presence of the two thresholds of the ABS learning rule. For this purpose, we establish a one-threshold learning rule by choosing Co = CI,which renders threshold O1 ineffective. The resulting modification scheme then corresponds to a one-threshold Hebb-like learning rule operating on mean activities a in an oscillatory network. Training the network again with randomly paired stimuli leads to a broader tuning of the resulting weight distributions as compared to training with the two-threshold rule (Fig. 3D, thin lines): Synchronizing connections are maintained to all units in the network and corresponding coupling weights to neighboring units are reduced. Similarly, the distribution of desynchronizing connections is spread over most units, leading to strongly reduced weight contributions for each individual unit. Testing the trained network with the two superimposed orthogonal stimuli shows that with this connectivity pattern the network is no longer able to exhibit the stimulus-dependent segregation of oscillatory assemblies. With the one-threshold learning rule, the increased coupling length of synchronizing connections and the flat weight distribution of desynchronizing connections result in bulk synchronization of all activated oscillators. 5 Conclusions

In this paper, we have described the self-organization of synchronizing and desynchronizing coupling connections in a network of neuronal oscillators. The results presented demonstrate that local learning rules can establish a network that exhibits stimulus-dependent formation and segregation of oscillatory assemblies in agreement with experimental evidence (Engel et al. 1991). With respect to known physiological data it is particularly interesting that these learning rules do not need to incorporate any assumptions as to the oscillatory character of the networks stimulus responses. During training the employed learning rules make use of a measure of mean neuronal activity, integrated on a long time scale, to develop a network that is able to operate in its functional response on rather short time scales. This indicates that during ontogenesis physiologically related local modification schemes can suffice to develop a connectivity pattern in support of the observed temporal structure of stimulus responses in cat visual cortex. The simulations presented above show that the two-threshold synaptic modification rule found in in vitro preparations of rat visual cortex


675

(Artola et al. 1990; Fig. 2, solid) is well suited to account for the development of synchronizing connections in our network. Synaptic weights are increased for those connections for which the activity u of the postsynaptic unit attains the learning rule's potentiation domain by exceeding upper threshold 02 (0, < ii). If the postsynaptic activity falls into the depression domain (0, < ii 5 the weights of the pertaining connections are decreased. With only little postsynaptic activity (ii < &), synaptic weights are subject to a slow decay, leading to an elimination of those connections that consistently fail to activate their postsynaptic target cells. This modification scheme stabilizes synchronizing connections between units, the orientation preferences of which differ by not more than 22.5" (Fig. 3A). The resulting distribution of synchronizing connections is the basis for the representation of a stimulus by an assembly of synchronously oscillating cells. Since the tuning width of the weight distribution w:) corresponds to only half the orientation tuning assumed for each neuronal population, the obtained connectivity pattern avoids to completely synchronize the entire network without compromising coarse coding of stimulus orientation. Generalizing the described ABS learning rule to anti-ABS weight changes for desynchronizing connections (Fig. 2, dashed) leads to a bipartite weight distribution wg) (Fig. 3B). In this case, desynchronizing connections between units with similar orientation preferences are depressed, while those between units differing 22.5', . . .,56.25' in their preferred orientations are potentiated. The resulting distribution of desynchronizing connections is the basis for the uncorrelated activity between two assemblies representing two superimposed stimulus bars of orthogonal orientation (Schillen and Konig 1991). Training our network with a one-threshold Hebb-like learning rule results in an increased coupling length of synchronizing connections and reduced weight contributions of desynchronizing connections (Fig. 3D). With such a connectivity pattern the network exhibits bulk synchronization of all activated oscillators. Thus, the self-organization with the one-threshold Hebb-like learning rule cannot establish the stimulusdependent segregation of oscillatory assemblies in our network. The physiological evidence for the ABS learning rule has been established for excitatory synapses on excitatory cells (Artola et al. 1990). At present, the evidence for plasticity of excitatory synapses on inhibitory cells is only indirect (BuzsAki and Eidelberg 1982; Kairiss etal. 1987; Taube and Schwartzkroin 1987). Nevertheless, we consider our application of the ABS modification rule to the network's synchronizing connections to be justified since excitatory synapses on inhibitory cells have many features in common with those on excitatory cells (Sah et al. 1990). Currently, there is no physiological evidence for the anti-ABS learning rule which we apply to the desynchronizing connections in our network. However we postulate anti-Hebbian mechanisms for several reasons. Reductions of synaptic weight in response to associated pre- and postsynaptic neuronal

676

I? Konig, B. Janosch,and T. B. Schillen

activity are present within the depression domain of the ABS modification scheme (Artola et al. 1990). Similarly, a decrease of synaptic weights associated with postsynaptic activity has been found in several paradigms of long-term depression (It0 1989; Stanton and Sejnowski 1990). Modulatory systems could conceivably gate anti-ABS synaptic modifications by adjusting membrane potentials or thresholds determining synaptic depression and potentiation. In our simulations, we use anti-ABS weight changes as a canonical extension of the ABS learning rule, appropriate to establish connections between units with uncorrelated activity. Similarly, other authors have introduced anti-Hebbian learning for the purpose of decorrelating activity in their networks (Barlow 1989; Rubner and Schulten 1990; Foldihk 1990; Hopfield et aZ. 1983). We did not include synaptic modifications of the network's inhibitory connections in our investigation since no plasticity of inhibitory synapses has yet been found. A learning rule with a depression and a potentiation domain of synaptic weights has been introduced by Bienenstock et al. (1982) for the modeling of deprivation experiments. The single threshold that separates depression and potentiation domain is adjusted as a nonlinear function of mean postsynaptic activity. This dynamic threshold adjustment is used to maintain, on average, constant total synaptic: weights of units in the network. In an oscillatory network, a corresponding dynamic threshold adaptation would have to generalize the algorithm of Bienenstock et al. (1982) to a functional dependence on global correlations in the network. However, such an approach would give up the principle of locality proposed for the learning rules in the current investigation. Therefore we ensure the convergence of total synaptic weights by normalization (von der Malsburg 1973). The applied learning rule involves the time averaging of pre- and postsynaptic activity. This is done as neuronal plasticity involves mediators with long time constants as, for example, Gaff and IP3 (Berrigde and Irvine, 1989). However, the exact sequence of synchrony detection, corresponding to the multiplication of pre- and postsynaptic activity in our learning rule, and time averaging, corresponding to the integral over a period of 200 msec, needs further experimental investigation: Stanton and Sejnowski (1990) provide evidence for a fast detection of synchrony in in vitro recordings of hippocampal slices. On the other hand a temporal contiguity of visual stimuli presented to both eyes of 200 msec has been found to be sufficient for the maintenance of binocularity in cat visual cortex (Altmann et al. 19871, a process subject to plastic changes in paradigms as alternating monocular deprivation. These two results demonstrate that the exact sequence of synchrony detection and temporal averaging for the paradigm under consideration has yet to be established. Furthermore, in the developing cortex incomplete myelinization leads to long conduction delays. Thus it may not be possible to synchronize oscillatory signals on a fine temporal scale, although connections appropriate for this task have to be developed. That the self-organization of such


677

a network, suited for stimulus-dependent assembly formation on a fine temporal scale, is possible by using only response properties on a temporal scale of a few hundreds of millisecond is demonstrated by our results. Hartmann and Driie (1990) presented a network model that represents continuous line segments by neuronal assemblies defined by correlated activity. They also report on an approach for the self-organization of synchronizing connections by a Hebb-like learning rule. Since Hartmann and Driie do not aim at simulating the experimentally observed segregation of superimposed stimuli (Engel et ~2.1991;Schillen and Konig 1991) their model is not designed to cope with this problem. As a consequence, Hartmann and Driie do not address the self-organization of a desynchronizing mechanism appropriate for this task. The self-organization of neural networks with oscillatory activity has also been investigated in models of the olfactory bulb (Freeman et al. 1988; Taylor and Keverne 1991). In a conditioning paradigm of odor discrimination, Taylor and Keverne (1991) use a Hebb-like learning rule to associate a frequency shift of oscillatory responses with a conditioned pattern. This frequency shift contrasts with the behavior of the model presented in this paper. Our system of delay-coupled units with nonlinear output functions exhibits only a weak dependence of the frequency of oscillatory responses on the coupling weights (Konig and Schillen 1991). Thus in our network, synaptic weight changes induced by learning do not interfere with the correlation of activity between units. An investigation of the ABS learning rule (Artola et al. 1990) with respect to associative memory has been presented by Hancock et al. (1991). Hancock et al. discuss the error correcting properties of the two-threshold learning rule in comparison to error backpropagation and conventional Hebbian learning algorithms. Wang et al. (1990) addressed the discrimination of superimposed patterns by temporal structure of activity in an associative memory of neuronal oscillators. In their network, synchronizing and desynchronizing coupling connections are implemented by excitatory and inhibitory connections, originating concurrently from the same unit. A Hebb-like learning rule is applied to both types of connections, which are allowed in addition to change their excitatory and inhibitory specificities. To circumvent these physiologically implausible assumptions requires the introduction of additional interneurons, which would, however, affect the temporal characteristics of the network. These and other considerations (Konig and Schillen 1991; Schillen and Konig 1991) have determined our choice of synchronizing and desynchronizing coupling connections, and the introduction of anti-ABS weight changes for the desynchronizing connections. Kleinfeld and Sompolinsky (1989) have investigated the temporal structure of neuronal activity in a modified Hopfield network. In their application two types of coupling connections with different time scales are used to learn stereotyped sequences of activity patterns, as observed

678


in biological central pattern generators. Connections operating on a short time scale serve to store the activity patterns of the memory. Connections with a long time scale represent the transitions between the consecutive patterns of a sequence. In this way, the network can learn temporal sequences by a local Hebb-like learning rule. Another application of the temporal structure of neuronal activity to learning and recall in an associative memory has been described by Abbott (1990). In this application, switching the network between stationary and oscillatory modes of activity, on different time scales, is used for initiating and terminating learning in the memory. Similarly, the effect of different time scales on the learning in neural networks has been studied by Baldi and Pineda (1991). These authors discuss switching between supervised and unsupervised learning in oscillatory and nonoscillatory networks. In our current investigation we used simulated light bars as stimuli for the self-organization of synchronizing and desynchronizing coupling connections between neuronal oscillators. Integrating stimulus responses on a long time scale during learning, the network developed the appropriate functional behavior for responses on short time scales. If trained with single stimulus bars (Fig. 3C), ABS rule learning established synchronized activity within the oscillatory assembly that represented the stimulus. At the same time, anti-ABS weight changes developed desynchronizing connections that allowed the segregation of superimposed but distinct additional stimuli. Furthermore, if trained with two superimposed stimulus bars of randomly paired orientations (Fig. 3D), the network still detected the single bar as the underlying coherent stimulus, generalizing over the random combinations of the two presented stimulus bars. Contrastingly, training the network with a one-threshold Hebb-like learning rule could not establish the stimulus-dependent segregation of oscillatory assemblies. In general, in a natural environment, the temporal structure of which neuronal responses would have to be synchronized and desynchronized in order to constitute neuronal assemblies appropriate for the representation of the visual field is not clear, a priori. Here, the self-organization of the neuronal network during ontogenesis provides a means to develop adequate coupling connections in response to the interaction with the environment. The coherent presentation of stimulus features, integrated on a long time scale during learning, can then lead to the development of synchronizing and desynchronizing connections, which are appropriate for the formation and segregation of neuronal assemblies on a short time scale in the mature organism. The work presented in this paper indicates that physiologically related local learning rules suffice to establish neuronal networks that, after training, exhibit a stimulus-dependent assembly formation of oscillatory responses in agreement with the observations in cat visual cortex.


679

Acknowledgments It is our pleasure to thank Wolf Singer and Alain Artola for valuable discussions and comments on the first draft of this paper. Renate Ruhl provided excellent graphic assistance. Many thanks also to Neil Steinberg for improving the English. This work has been supported in part by the Deutsche Forschungsgemeinschaft (SFB 185).

References Abbott, L. F. 1990. Modulation of function and gated learning in a network memory. Proc. Natl. Acad. Sci. U.S.A. 87, 9241-9245. Altmann, L., Luhmann, H. J., Greul, J. M., and Singer, W. 1987. Functional and neuronal binocularity in kittens raised with rapidly alternating monocular occlusion. J. Neurophysiol. 58, 965-980. Artola, A., Brocher, S., and Singer, W. 1990. Different voltage-dependent thresholds for inducing long-term depression and long-term potentiation in slices of rat visual cortex. Nature (London) 347, 69-72. Baldi, P., and Pineda, F. 1991. Constrastive learning and neural oscillations. Neural Comp. 3, 526-545. Barlow, H. 1989. Unsupervised learning. Neural Comp. 1, 295-311. Berridge, N. J. and Irvine, R. F. 1989. Inositol phosphates and cell signaling. Nature (London) 341, 197-205. Bienenstock, E. L., Cooper, L. N., and Munro, I? W. 1982. Theory for the development of neuron selectivity: Orientation specificity and binocular interaction in visual cortex. J. Neurosci. 2, 3248. Buzsdki, G., and Eidelberg, E. 1982. Direct afferent excitation and long-term potentiation of hippocampal interneurons. J. Neurophysiol. 48, 597-607. Eckhorn, R., Bauer, R., Jordan, W., Brosch, M., Kruse, W., Munk, M., and Reitboeck, H. J. 1988. Coherent oscillations: A mechanism of feature linking in the visual cortex? Biol. Cybern. 60, 121-130. Engel, A. K., Konig, P., and Singer, W. 1991. Direct physiological evidence for scene segmentation by temporal coding. Proc. Natl. Acad. Sci. U.S.A. 88, 91369140. Foldidk, P. 1990. Forming sparse representations by local anti-Hebbian learning Biol. Cybern. 64, 165-170. Gray, C. M., and Singer, W. 1987. Stimulus-specificneuronal oscillations in the cat visual cortex: A cortical functional unit. SOC.Neurosci. Abstr. 13(404.3). Gray, C. M., and Singer, W. 1989. Stimulus-specific neuronal oscillations in orientation columns of cat visual cortex. Proc. Natl. Acad. Sci. U.S.A. 86, 1698-1702. Gray, C. M., Konig, P., Engel, A. K., and Singer, W. 1989. Oscillatory responses in cat visual cortex exhibit inter-columnar synchronization which reflects global stimulus properties. Nature (London) 338, 334-337. Freeman, W. J. 1975. Mass Action in the Nervous System. Academic Press, New York.

680

I? Konig, B. Janosch, and T. B. Schillen

Freeman, W. J., Yao, Y., and Burke, B. 1988. Central pattern generating and recognizing in olfactory bulb: A correlation learning rule. Neural Networks 1, 277-288. Hancock, P. J. B., Smith, L. S., and Phillips, W. A. 199'1. A biologically supported error-correcting learning rule. Neural Comp. 3, 201-212. Hartmann, G., and Driie, S. 1990. Self organization of a network linking features by synchronization. In Parallel Processing in Neural Systems and Computers, R. Eckmiller, G. Hartmann, and G. Hauske, eds., pp. 361-364. Elsevier, Amsterdam. Hopfield, J. J., Feinstein, D. I., and Palmer, R. G. 1983. "Unlearning" has a stabilizing effect in collective memories. Nature (London) 304, 158-159. Ito, M. 1989. Long-term depression. Annu. Rev. Neurosci. 12, 85-102. Kairiss, E. W., Abraham, W. C., Bilkey, D. K., and Goddard, G . V. 1987. Field potential evidence for long-term potentiation of feed-forward inhibition in the rat dentate gyms. Brain Res. 401, 87-94. Kleinfeld, D., and Sompolinsky, H. 1989. An associative network model for central pattern generators. In Methods in Neuronal Modeling: From Synapses to Networks, C. Koch and I. Segev, eds. MIT Press, Cambridge, MA. Konig, P., and Schillen, T. B. 1990. Segregation of oscillatory responses by conflicting stimuli - Desynchronizing connections in neural oscillator layers. In Parallel Processing in Neural Systems and Computers, R. Eckmiller, G. Hartmann, and G. Hauske, eds., pp. 117-120. Elsevier, Amsterdam. Konig, P., and Schillen, T. B. 1991. Stimulus-dependent assembly formation of oscillatory responses: I. Synchronization. Neural Comp. 3, 155166. Rubner, J., and Schulten, K. 1990. Development of feature detectors by selforganization. Biol. Cybern. 62, 193-199. Sah, l?, Hestrin, S., and Nicoll, R. A. 1990. Properties of excitatory postsynaptic currents recorded in vitro from rat hippocampal interneurons. I. Physiol. 430,605-616. Schillen, T. B. 1991. Designing a neural network simulator - The M E N 5 modelling environment for network systems 11. Comp. Appl. Biosci. 7, 431446. Schillen, T. B., and Konig, P. 1990. Coherency detection by coupled oscillatory responses - Synchronizing connections in neural oscillator layers. In Parallel Processing in Neural Systems and Computers, R. Eckmiller, G . Hartmann, and G. Hauske, eds., pp. 139-142. Elsevier, Amsterdam. Schillen, T. B., and Konig, P. 1991. Stimulus-dependent assembly formation of oscillatory responses: 11. Desynchronization. Neural Comp. 3, 167-177. Singer, W. 1990. Search for coherence: A basic principle of cortical self-organization. Concepts Neurosci. 1, 1-26. Stanton, I? K., and Sejnowski, T. J. 1990. Associative long-term depression in the hippocampus induced by Hebbian covariance. Nature (London) 339,215-218. Taube, J. S., and Schwartzkroin, P. A. 1987. Intracellular recording from hippocampal CAI interneurons before and after development of long-term potentiation. Brain Res. 419,32-38. Taylor, J. G., and Keverne, E. B. 1991. Accessory olfactory learning. Biol. Cybern. 64, 301-305.


681

von der Malsburg, C. 1973. Self-organization of orientation sensitive cells in the striate cortex. Kybernetik 14, 85-100. von der Malsburg, C. 1981. The correlation theory of brain function. Internal Report 81-2, Max-Planck-Institute for Biophysical Chemistry, Gottingen, Germany. von der Malsburg, C., and Schneider, W. 1986. A neural cocktail-party processor. Biol. Cybern. 54, 2940. Wang, D., Buhmann, J., and von der Malsburg, C. 1990. Pattern segmentation in associative memory. Neural Comp. 2, 94-106. ~~

~

Received 5 August 1991; accepted 16 March 1992.

This article has been cited by: 2. Walter G. Sannita. 2009. Neuronal functional diversity and collective behaviors: a scientific case. Cognitive Processing 10:S1, 17-22. [CrossRef] 3. Walter G. Sannita. 2008. Neuronal Functional Diversity and Collective Behaviors. Journal of Biological Physics 34:3-4, 267-278. [CrossRef] 4. Antonino Raffone , Gezinus Wolters . 2001. A Cortical Mechanism for Binding in Visual Working MemoryA Cortical Mechanism for Binding in Visual Working Memory. Journal of Cognitive Neuroscience 13:6, 766-785. [Abstract] [PDF] [PDF Plus] 5. H. J. Kappen. 1997. Stimulus-dependent correlations in stochastic networks. Physical Review E 55:5, 5849-5858. [CrossRef] 6. Ruby Klink, Angel Alonso. 1997. Morphological characteristics of layer II projection neurons in the rat medial entorhinal cortex. Hippocampus 7:5, 571-583. [CrossRef] 7. Pieter R. Roelfsema, Andreas K. Engel, Peter König, Wolf Singer. 1996. The Role of Neuronal Synchronization in Response Selection: A Biologically Plausible Theory of Structured Representations in the Visual CortexThe Role of Neuronal Synchronization in Response Selection: A Biologically Plausible Theory of Structured Representations in the Visual Cortex. Journal of Cognitive Neuroscience 8:6, 603-625. [Abstract] [PDF] [PDF Plus] 8. H. Sompolinsky , M. Tsodyks . 1994. Segmentation by a Network of Oscillators with Stored MemoriesSegmentation by a Network of Oscillators with Stored Memories. Neural Computation 6:4, 642-657. [Abstract] [PDF] [PDF Plus]

Communicated by Vincent Torre

Seeing Beyond the Nyquist Limit Daniel L. Rudemian* William Bialek* Department of Physics and Department of Molecular and Cell Biology, University of California at Berkeley, Berkeley, C A 94720 U S A

In many biological systems the primary transduction of sensory stimuli occurs in a regular array of receptors. Because of this discrete sampling it is usually assumed that the organism has no knowledge of signals beyond the Nyquist frequency. In fact, higher frequency signals are expected to mask the available lower frequency information as a result of aliasing. It has been suggested that these considerations are important in understanding, for example, the design of the receptor lattice in the mammalian fovea. We show that if the organism has knowledge of the probability distribution from which the signals are drawn, outputs from a discrete receptor array can be used to estimate signals beyond the Nyquist limit. In effect, a priori knowledge can be used to de-alias the image, and the estimated signal above the Nyquist cutoff is in fact coherent with the real signal at these high frequencies. We address initially the problem of stimulus reconstruction from a noisy receptor array responding to a Gaussian stimulus ensemble. In this case, the best reconstruction strategy is a simple linear transformation. In the more interesting (and natural) case of nongaussian stimuli, optimal reconstruction requires nonlinear operations, but the higher order correlations in the stimulus ensemble can be used to improve the estimate of super-Nyquist signals.

1 Introduction

Sensory stimuli reaching an organism are initially encoded in the activity of an array of sensory neurons. From this discrete pattern of activity and a priori information about the stimulus ensemble, the organism must be able to reconstruct important aspects of the stimulus. Since the animal's performance will be limited by the properties of the receptor array, it is important that its design take full advantage of any knowledge of the stimulus ensemble. 'Present address: NEC Research Institute, 4 Independence Way, Princeton, NJ 08540. Neural Computafion 4, 682-690 (1992)


Seeing Beyond the Nyquist Limit

683

Each element in the receptor array encodes information about a particular subset of the stimulus. In the cochlea, for example, each hair cell responds to a limited frequency range. In the retina each photoreceptor has a limited aperture. Presumably different arrangements of these apertures result in receptor signals that convey more or less information about the outside world. In particular, the combination of receptor sampling, receptor noise, and the statistics of natural signals can lead to nontrivial optimization problems for the design of the receptor array. One example is the compound eye, where some insects sacrifice angular resolution in favor of collecting more photons per receptor and hence providing better intensity resolution; these tradeoffs can be understood semiquantitatively in terms of information theoretic optimization principles (Snyder et al. 1977). In the compound eye one can easily demonstrate that the discreteness of the photoreceptor array leads to aliasing: Behavioral and neural reactions to moving gratings are reversed if the spatial frequency of the grating exceeds the Nyquist limit. But gratings are highly unnatural stimuli. In this work we ask how the constraints of discrete sampling limit performance under conditions where the stimuli are chosen at random from some “natural” distribution that is presumed known to the organism. We find that it is in fact possible to reconstruct meaningful information beyond the Nyquist limit, and comment on the implications of this result for retinal design. 2 The Receptor Array

We will consider an infinite array of receptors lying on a one-dimensional lattice at the points x, = no, where a is the receptor spacing and n is any integer. Each element has the same receptive field profile or aperture function f ( x ) centered at the location of that element. The output of the nth element is

Jw

(2.1) + vn where $ ( x ) is the stimulus and v,,is the receptor noise (see Fig. 1). We yn =

-w

d x f ( x - x n ) 4(x)

shall assume the noise is additive, gaussian, and independent in each channel, (qmvn)= cr2Smn. Such an array will act as a simple approximation to a system of sensory cells, which in general have nonlinearities as well as nontrivial temporal responses. 3 Stimulus Reconstruction

The organism is faced with the problem of estimating various features of the stimulus $(x) using the receptor cell outputs {y,}. Everything that we know about $ ( x ) by virtue of observing the {y,} is summarized by

Daniel L. Ruderman and William Bialek

684

Neb.

Nois.

Activity

Activity

$-

Nola0

Activity

Figure 1: A stimulus periodically sampled with noisy filters. the posterior probability distribution P[$ I {yll}]. Using Bayes' theorem we have

Given that we have seen the activities {y,} it is this distribution that tells us the best 4 to choose as the original stimulus. Note that P[{y,,}]acts only as a normalization factor since it is assumed that the activities {y,} are known. The important aspects of this distribution are characterized by W Y n } I 41 and PMI. Since the noise is gaussian and independent in each channel we have

where

F,,[$] = /dxf(x

-

x,) $(x) =

1;

e?k""f(k) $ * ( k )

(3.3)

is the filter activity uncorrupted by noise. We define $ ( k ) , the Fourier transform of $(x), a s

4 ( k ) = /dxe-'kXd(x)

(3.4)

The question of choosing the distribution of the stimuli, P[$], is more complicated. As a simple first step we shall assume the stimuli are drawn


685

from a gaussian ensemble characterized by a (two-sided) power spectrum S(k), so (3.5) The treatment of nongaussian stimuli will be addressed below. Gaussian stimuli and noise lead to a gaussian distribution for P[4 I {yn)l:

The mean of the distribution is &(k) and its width is determined by C(k, k‘). From the distribution Pj4 1 {yn}] we must choose some particular 4Jk) that is the ”best” estimate of the actual signal. The most useful estimator depends on the costs for making different kinds of errors, but there are two natural choices: maximum likelihood, in which we find the 4e(k) that satisfies

(3.7) and the conditional average,

$t(k)

=

10 4 ~ 1I 4

{~n)14(k)

(3.8)

Here J D4 means a functional integral over all signals 4(x). In the simple gaussian case considered here both approaches lead to the same result, (3.9) which is just &(k), the mean of the distribution P [ 4 I {yn}].Here ko = 2r/a and y(k) is the Fourier transform of the receptor outputs; since the yn lie on a lattice we have y(k) = y(k + Mko) for all integer M. Thus the most likely stimulus can be derived from the receptor activities through a linear transformation. In principle there is no limit to the spatial frequency of the reconstruction. The accuracy is limited by noise, filter strength, and aliasing among frequencies separated by integer multiples of the lattice wavevector. A formula analogous to equation 3.9, derived as the linear filter that minimizes reconstruction error, has been known since the 1950s (Stewart 1956). With any signal estimate it is important to know its degree of confidence. In the current example this is given by the width of the distri-

686


bution of & ( k ) , which is characterized by the correlation function of the distribution: kof(k)S (k)f * (k’)S(k:’)C,,6(k -k‘ - t~ko) C(k.k’) = 2 ~ S ( k ) O (k k’)(3.10) 02+( k o / 2 ~C ) , lf(k+nko)12S(k+nko) This correlation function is not diagonal when there is aliasing; the Fourier components become coupled and thus covary. When the noise variance is small and aliasing is absent the formula reduces to (3.11) The variance goes to zero as the noise diminishes, thus increasing the reliability of the estimate. This correlation function plays an important role in the estimation of nongaussian stimuli. 4 Examples

Consider the simplest case of sampling. The signal is band-limited to below the Nyquist sampling frequency ko/2, and the sampling is done by noiseless delta-function filters [o= 0, f ( k ) = 11. Then the reconstruction is exact and takes the familiar Shannon-Whittaker form (Shannon 1948) (4.1) A de-aliasing reconstruction example is shown in Figure 2. The signal (solid line) is drawn from a white noise distribution with a maximum frequency of 6 cycles. Such a signal requires at least 13 (= 2 x 6 1) sampling points to reconstruct exactly. In this case it is sampled at 11 points (diamonds) by noiseless delta-function filters and reconstructed based on these samples. The first reconstruction (dotted line) is done using the Shannon-Nyquist formula and displays a classic case of aliasing. The second reconstruction (dashed line) uses the optimal filter derived above, which takes into account the actual power spectrum. This a priori knowledge allows the filter to de-alias the signal, producing a more accurate reconstruction.

+

5 What‘s going on?

To understand why we can estimate signals above the Nyquist frequency it is helpful to look at a simple example. Consider a signal whose power spectrum includes the frequencies k and k + k,, but none higher. The reconstruction gives, in the limit of large signal-to-noise ratio, (5.1)


687

Figure 2: Stimulus reconstruction showing aliasing and de-aliasing. Solid line, signal; diamonds, sample points; dotted line, reconstruction with aliasing; dashed line, reconstruction using a priori statistics. See text. Thus the ratio of the reconstructed stimuli at the two frequencies is

Mk) + kn)

&(k

-

S ( k ) f(k)

S(k + kn)f(k + kn)

(5.2)

The signal estimate comes from partitioning the aliased signals according to their relative power spectra and filter strengths. In this way, by knowing the power spectrum, the signal can be estimated at frequencies beyond the Nyquist limit. 6 Is Phase Preserved in Super-Nyquist Frequencies?

In order for a reconstruction to be useful there must be phase coherence between the signal and the estimate. The mean squared error in estimating the signal amplitude at a frequency k is (I$e(k) - d(k)12) =

(I$t-(k)12)+ (t$(k)12) - 2 ~ e ( $ ~ , * k M ( k ) )

(6.1)

688


The last term involves the cosine of the phase difference between the signal and its estimate. Clearly the larger the phase coherence, the better the estimate will be on average since the error will be minimized. To quantify the preservation of phase we examine the coherence between the estimate and the signal: (6.2) Here ( $ * ( k ) & ( k ) ) is an average over all signals and noise. The dependence of &(k) on the signal is through y(k). We find that (6.3)

Thus there is positive covariance at a frequency as long as there is signal power and the filter will pass the frequency. The covariance is reduced by aliasing and by the finite level of receptor cell noise 0,but both of these are continuous, gradual effects. Just as the reconstruction of the stimulus does not fail abruptly when the signal-to-noise ratio falls below unity, similarly the reconstruction does not fail abruptly as soon as aliasing is possible. It is interesting to consider what happens in the case of nongaussian stimuli. Here standard methods of statistical physics can be used to calculate conditional averages as a perturbation series in the higher order correlations, which will involve the correlation function of the posterior distribution. The result is that our best estimate of a nongaussian stimulus involves nonlinear processing of the receptor outputs even if the input/output relations of the receptors themselves are linear, as assumed here. Nongaussian behavior, however, means that different Fourier components of the stimulus are correlated with one another, unlike the case of the (stationary) gaussian distribution. In natural images, whose statistics are scale-invariant (Field 1987) this means that high- and low-frequency signals are correlated, and of course this helps in the reconstruction of super-Nyquist data.

7 Conclusions An organism is faced with the problem of estimating stimuli from noisy neural responses. We have shown that a priori knowledge of the stimulus ensemble leads to meaningful stimulus estimation beyond the naive Nyquist limit. Such applications of prior statistical knowledge are widely used in regularizing ill-posed inverse problems (Turchin et al. 1971). Although we have worked a one-dimensional example, the theory is readily extended to higher dimensions. An interesting follow-up to this problem is to consider the choice of optimal sampling filter or ”receptive field,” f ( x ) . A given power spectrum


689

and system design constraints lead to a variational problem that can be solved for the optimal filter. Here "optimal" may be defined in terms of minimizing the mean-squared error in reconstruction, for example, or accurately detecting a particular stimulus feature. Is any of this relevant to biology? Several years ago a number of authors were concerned with the consequences of aliasing for mammalian vision (Yellot 1982, 1983, 1984; Miller and Bernard 1983; Hirsch and Hylton 1984; Bossomaier et ai. 1985). Questions were raised about the role of disorder in the receptor lattice, which might serve to reduce aliasing, as opposed to the more conventional view that our remarkable acuity [and hyperacuity (Westheimer 198l)l is based on a highly ordered receptor array. From equation 6.3 we suggest that this debate missed a crucial point. Under natural conditions the effect of aliasing is essentially the same as that of noise-it causes a gradual reduction in the quality of our estimates of the true image at high spatial frequencies. If the noise is very large, in fact, aliasing is really not a problem at all: From equation 6.3 we see that at high noise levels the gradual decline in correlations at higher frequencies is governed solely by the signal-to-noise ratio, and the confusion triggered by aliasing has no effect. Recent experiments on the photocurrents and noise produced by cones from the primate fovea (Schnapf etal. 1990) can be analyzed to show that the noise level is in fact quite high: The equivalent contrast noise per pixel is of order 30% if one assumes an integration time appropriate to preattentive vision (W. Bialek, unpublished). We suggest that, as a result of the high noise level, antialiasing is not a major design constraint in the mammalian retina. It would be interesting to design experiments that would test the ability of human or animal observers to "de-alias" natural signals in the sense described here.

Acknowledgments We thank J. S. Joseph, J. I? Miller, and F. M. Rieke for helpful discussions. Work in Berkeley was supported in part by a Presidential Young Investigator award from the National Science Foundation (to W. B.), supplemented by funds from Sun Microsystems, Cray Research, and the NEC Research Institute, and by a graduate fellowship from the Fannie and John Hertz Foundation (to D. L. R.). References Bossomaier, T. R. J., Snyder, A. W., and Hughes, A. 1985. Irregularity and aliasing: solution? Vision Res. 25, 145-147. Field, D. 1987. Relations between the statistics of natural images and the response properties of cortical cells. J. Opt. SOC.Am. 4, 2379.

690


Hirsch, J., and Hylton, R. 1984. Quality of the primate photoreceptor lattice and limits of spatial vision. Vision Res. 24, 347-355. Miller, W.H., and Bernard, G. D. 1983. Averaging over the foveal receptor aperture curtails aliasing. Vision Res. 23, 1365-1369. Schnapf, J. L., Nunn, B. J., Meister, M., and Baylor, D. A. 1990. Visual transduction in cones of the monkey Macaca fascicularis. 1, Physiol. 427, 681-713. Shannon, C.E. 1948. A mathematical theory of communication. Befl Sys. Tech. J. 27, 379. Snyder, A. W., Stavenga, D. G., and Laughlin, S. B. 1977. Spatial information capacity of compound eyes. J. Comp. Physiol. 116,183-207. Stewart, R. M. 1956. Statistical design and evaluation of filters for the restoration of sampled data. Proc. IRE 44, 253-257. Turchin, V. F., Kozlov, V. P., and Malkevich, M. S. 1971. The use of mathematicalstatistics methods in the solution of incorrectly posed problems. Soviet Phys. Uspekhi. 13, 681-703. Westheimer, G. 1981. Visual hyperacuity. Prog. Sens. Physiol. 1, 1-30. Yellot, J. I., Jr. 1982. Spectral analysis of spatial sampling by photoreceptors: Topological disorder prevents aliasing. Vision Res. 22, 1205-1210. Yellot, J. I., Jr. 1983. Spectral consequences of photoreceptor sampling in the rhesus retina. Science 221, 382-385. Yellot, J. I., Jr. 1984. Image sampling properties of photoreceptors: A reply to Miller and Bernard. Vision Res. 24, 281-282.

Received 26 November 1991; accepted 31 January 1992.

This article has been cited by: 2. Daniel Ruderman. 1994. The statistics of natural images. Network: Computation in Neural Systems 5:4, 517-548. [CrossRef]

Communicated by Joseph Atick

Local Synaptic Learning Rules Suffice to Maximize Mutual Information in a Linear Network Ralph Linsker IBM Research Division, T. I. Watson Research Center, P. 0.Box 218, Yorktown Heights, NY 10598 USA

A network that develops to maximize the mutual information between its output and the signal portion of its input (which is admixed with noise) is useful for extracting salient input features, and may provide a model for aspects of biological neural network function. I describe a local synaptic Iearning rule that performs stochastic gradient ascent in this information-theoretic quantity, for the case in which the inputoutput mapping is linear and the input signal and noise are multivariate gaussian. Feedforward connection strengths are modified by a Hebbian rule during a "learning" phase in which examples of input signal plus noise are presented to the network, and by an anti-Hebbian rule during an "unlearning" phase in which examples of noise alone are presented. Each recurrent lateral connection has two values of connection strength, one for each phase; these values are updated by an anti-Hebbian rule. 1 Introduction

The idea of designing a processing stage so as to maximize the mutual information (MI) between its output and the signal portion of its input (which is admixed with noise) is attractive as a way to use sensory input optimally, and to extract statistically salient input features (Linsker 1988; Atick and Redlich 1990). For the idea to be practical for use by biological systems or in large synthetic networks, it is important that the required optimization be implemented by a local algorithm-one that uses only information currently available at the node or connection that is to be modified. This paper presents such an algorithm for the case in which the input-output transformation is linear and the signal and noise distributions are multivariate gaussian. The algorithm performs stochastic gradient ascent in the MI. Local network algorithms have been described for several tasks that differ from the present one but are related to it: (1)principal component analysis (PCA) (Foldiik 1989; Leen 1991; Sanger 1989), which identifies high-variance linear combinations of inputs, but does not take account of noise; (2) smoothing and predictive filtering (Atick and Redlich 1991), Neural Computation

4, 691-702 (1992)


Ralph Linsker

692

\

Figure 1: Linear network, showing feedforward paths (solid lines) and lateral recurrent connections (dashed lines). which approximate MI maximization in certain limiting cases; and (3) MI maximization in a probabilistic winner-take-all network (Linsker 1989b), a nonlinear case in which only one output node "fires" at any given time, simplifying the computation of the MI. The paper is organized as follows: Section 2 states the optimization problem. The algorithm is presented in Section 3, illustrated by a numerical example in Section 4, and discussed in a broader context in Section 5. Mathematical details are given in the Appendix. 2

The Optimization Problem

A linear feedforward network (indicated by the solid-line paths in Fig. 1) is presented with a sequence of input vectors X. Each vector is the sum of an input signal S and input noise N,where S and N are independently drawn from multivariate gaussian distributions whose means are zero. The network's output is Z f CX + u, where the noise u, added at each output is an independent gaussian random variable of zero mean and nonzero variance. The ensemble-averaged mutual information R ( Z ,S) between Z and S, also called the Shannon information rate for the S Z mapping, is the information that the output Z "conveys about" the input signal S. It equals H ( Z )-H(Z I S), where H ( Z ) denotes the entropy of Z , and H ( Z I S) --f

Local Synaptic Learning Rules

693

denotes the average over S of the conditional entropy of Z given S. [Note that H ( Z I S) equals the entropy of the noise contribution to Z, which is CN v.1 The entropy of a gaussian distribution having covariance Q is (Shannon and Weaver 1949), apart from an irrelevant constant term,

+

H = (1/2)IndetQ

(2.1)

Therefore

R ( Z , S )= H ( Z ) - H ( Z

I S) = (1/2)[IndetQL-lndetQU]

(2.2)

where

+ + + + (2.3) QU + + (2.4) qL = ((S + N ) ( S + N)') = (SS') + (NN'),9' = ( N N T ) ,and Y = (4). QL

= =

([C(S N) v][C(S N) 4') = CqLCT+ r ((CN v)(CN v)') = CqUCT+ r

Both S and N may have statistical correlations. Correlations in the input noise N may arise because of sensor design, or because correlations were induced at earlier stages of a multistage processing network, or because N represents environmental features that we do not want the network to respond to (e.g., "semantic noise"). In any case, if the network is to learn to maximize R ( Z ,S), it must learn to distinguish input signal from noise. We will do this using a training process that consists of two phases. In "learning" or L phase, the network is shown examples of X = S S N . In "unlearning" or U phase, the network is shown examples X = N of input noise alone. Performing gradient ascent on R ( Z ,S) then consists of alternately performing gradient ascent on H ( Z ) during L phase, and gradient descent on H ( Z I S) during U phase. Each of these tasks is performed, during its respective phase, by the local algorithm described in the next section.' 3 Local Synaptic Modification Rule

3.1 Performing Gradient Ascent in the Entropy of the Output Activity. We show first how to perform gradient ascent in the entropy H of the output of a linear mapping. (Derivations are relegated to the Appendix.) Let X be an input vector, Y 5 CX, and Z = Y v be the output. X has covariance matrix 4 = (XX'), and the output Z has covariance

+

'Throughout the paper, the covariance matrix of the input X is denoted by q = (XX'), and that of the output Z = CX + v by Q = (ZZ'). When used, the superscript L or U specifies whether the covariance matrix refers to the "learning" or "unlearning" phase. Since X = S+N during L phase and X = N during U phase, the expressions for qLsU and QLsU used in equations 2.3 and 2.4 are obtained. Angle brackets denote an ensemble average, and superscript T denotes the transpose.

Ralph Linsker

694

Q = ( Z Z T )= CqCT + r. We assume for now that r is independent of C. Using equation 2.1 we obtain aH/aCni

=

(Q-'Cq)ni

(3.1)

If the Q-' factor were absent, we would obtain the gradient ascent "batch update" learning rule: AC,, = raH/aC,, = r(Cq),, = y(Y,X,). The local Hebbian rule AC,, = yY,X, would perform stochastic gradient ascent in H . The Q-' factor, however, introduces a complicated dependence on the activities at all other nodes. To compute equation 3.1 using a local learning rule, we augment the Z (solid lines of Fig. 1) by feedforward network that maps X + Y adding lateral connections of strength F,, (dashed lines) from each node m to each node n (including m = n). The lateral connections do not directly affect the network output Z; they are used only to compute the weight changes AC. We choose the strength matrix to be --f

F=i-aQ

(3.2)

( a > 0, and i is the identity matrix) and recursively define a sequence of activity vectors y(t) by

y(0)

= Y;

Y(t

+ 1) = Y + Fy(t)

(3.3)

If cy is chosen so that y(t) converges (see Appendix), then ay(m) = Q-'Y and we obtain the batch update learning rule Acm = raH/acni

(3.4)

= ra(yn(m)Xt)

The Hebbian rule AC,, = yayn(m)XI, using the iterated activity y(w) rather than Y, performs stochastic gradient ascent in H. The lateral connection strengths F depend on C through Q. An estimate Q of Q is computed as a running average, or trace, over recent input presentations. The initial Q is chosen arbitrarily, and Qnmis updated at each presentation by AQnm =

(l/M)(YnYm + rnm

-

(3.5)

Qnm)

+

(If r is not explicitly known, Z,Z, may be used in place of Y,Ym rnm.1 We define the strength F,, = 6,, - CYQ,,. Thus AF,, contains the antiHebbian term [-(a/M)Y,Y,]. An empirically useful, although not theoretically guaranteed, way to keep a at a proper value for convergence of y(t) is to monitor whether y(t) has "settled down" by a specified time T. For example, define p as the sum over all nodes n (optionally averaged over a running window of recent presentations) of b n ( T + l ) -y,(T)] x [y,(T) -y,(T- l)]. If y(T) has nearly converged to y(w), IpI will be smaller than a specified tolerance 6 . (The converse is not guaranteed.) If p > 6 , y(t) is converging too slowly and N should be increased. If p < - 6 , y(t) is oscillating and (Y should


695

be decreased. (An analytic condition for convergence is discussed in the Appendix. That condition cannot directly be used to choose a, since it makes use of the eigenvalues of Q, which are not available to the network.) To summarize, the local algorithm for stochastic gradient ascent in H is: Choose initial C (e.g., randomly) and Q (e.g., = 0). Then do repeatedly the following steps, which we will refer to as ”Algorithm A“:

+ v. 2. Update lateral connections: Change 0 using equation 3.5. 1. Select X; compute Y = CX, Z

=Y

3. Recursive activation: Compute {y(t), t I T

+ 1) using equation 3.3

with F = I - NQ. 4. Convergence check (e.g., using p as above). If fied, go back to step 3.

Q

needs to be modi-

5. Update feedforward connections: Change Cni by ACni

= rayn(T)Xi.

During a start-up period, train Q until it converges, but leave C unchanged, by skipping steps 3-5. 3.2 Gradient Ascent in Mutual Information; ”Learning” and ”Unlearning” Phases. Combining the two training phases (see equations 2.2 and 3.41, we obtain for the batch update rule

ACni = yaR(Z, S)/dCwi = r[aL(yn(m;L)Xi)L- ~ u ( Y ~u )( X~i ); ~(3.6) ] Here (. . . ) d denotes an average over a set of presentations appropriate to phase +that is, X = S + N for L phase, and X = N for U phase. Each iterated activity y,(m; 4)is computed using the lateral connection matrix F d = I - a+Qb appropriate to that phase. Stochastic gradient ascent in R(Z,S) is obtained by starting with arbitrary C, QL,and QU,and repeatedly performing Algorithm A alternately (1) for L phase, updating QL,aL (if necessary), and C; and (2) for U phase, updating QU,QU (if necessary), and C. 3.3 Constraints and ”Resource Costs”. In general, maximizing the above R ( Z ,S ) will cause elements to increase without limit. Some “resource cost” penalty function P, or some explicit constraint on C, is therefore typically included as part of the optimization problem. Maximizing [ R ( Z , S )- P] instead of R ( Z , S ) poses no additional difficulty, provided dP/dC,i can be computed locally (at each node or connection) by the network. TWOuseful cases are P 0: C,(1 -gn)’, where (1)g n = CiC’,j or (2) gn = V,,, and where

v, ZE (Y’,)L = (CqLCT),,

(3.7) is the output variance at node n during L phase (before adding output noise v,).

Ralph Linsker

696

3.4 A Constraint on the Number of Discriminable Output Values at Each Node. Realistic nonlinear processing elements typically have

limited dynamic range. This limits the number of output values that can be discriminated in the presence of output noise. One way to introduce a similar limitation in our linear network is to penalize deviations from V, = 1 as above. In this subsection we explore a different way to limit the effective dynamic range of each node. We multiply the output noise at each node n by the factor V;”, so that increasing V, does not change the number of discriminable output values. That is, Z, = Y, $- v, with v, = VA”v; and (v;,) = P (= constantL2 When this is done, the variance of each v, depends on C (through V,) and contributes additional terms to the gradient of R ( Z ,S). We obtain (see Appendix for derivation of equations 3.8 and 3.10)

(note the term in ,b’ is new). To derive a learning rule for batch update, we must express [(Q4)-lInn in terms of local quantities. To do this, we recursively compute (during each phase 4) ~ ’ ( 0 4) ; = v’;

+

+

~ ’ ( t 1; 4)= V’ F’yy’(t; 9)

(3.9)

The prime on y’ indicates that the input to the recursion is now v’, rather than CX as before. We find

where

To derive a learning rule for update after each presentation, we associate with each node n an estimate or trace W,?of W$ obtained by choosby AW,? = (l/M)[a$y’,(co;4)~;-Wt]. ing an initial W 4 , then updating W,? The learning rule is then as stated following equation 3.6, with the addition that y’(t; 4)is recursively computed and equations 3.10 and 3.11 are used. *We could instead have rescaled the output Y, to unit variance before adding output noise of constant variance @, yielding Z, = V,”2Y, + I(,. The resulting R ( Z ,S) is the same in both cases.


697

4 Illustrative Example

The operation of the local learning rule for update after each presentation in the case of constrained dynamic range (previous subsection) is illustrated by a numerical example. Each input signal vector S is a set of values at D uniformly spaced sites on a one-dimensional ”retina” having periodic boundary conditions. S is obtained by convolving white noise with a gaussian filter, so that (SiSj) = exp[-(sij/~a)~] (apart from a negligible deviation due to the boundary condition), where sij = min(/i- j l , D - [i - j l ) . The input noise is white; each component Ni of noise vector N is an independent gaussian random variable of mean zero and variance 7. Thus (NiN,) = 76i,. There are D’ output nodes n. The initial C,i values are drawn independently from a uniform distribution with mean zero. Initial Q(4) and I%,($) are set to zero. Parameter values used are D = 16; D’= 8; so = 3; input noise variance 7 = 0.25; output noise variance 0 = 0.5; y =5x aL = 0.445 to 0.475 (automatically adjusted to keep the for T = 16); (YU = 1; M = 400. C is convergence measure IpJ < t = held fixed for the first 800 input presentations to allow Q4 to converge. During the development of C, a running estimate (trace) of each output variance V, is used to rescale C,i (multiplying it by Vi”2) so that each V,,remains close to unity. [This rescaling of C for each node n is done for convenience, and has no effect on the value of R ( Z ,S), as discussed in the previous subsection.] Note that y was conservatively chosen for slow but accurate convergence of C; no attempt was made to optimize the rate of C development. The resulting weight vector (C,, , . . . , C,D) for each output node n in general spans the entire “retina.” [No penalty for long connections has been included; see Linsker (1989a) for discussion of a similar case.] Since the input covariance matrix is Toeplitz ( ( S S T ) q is a function of i - j), the eigenvectors of ( S S T ) are states (in weight space) having definite spatial frequency k. It is therefore more revealing to show the development of the of c for various n, rather than Fourier components (&, . . . , &, . . . , exhibiting the C,, themselves. Figure 2 shows the squared magnitude lC,k1* for two of the eight nodes n, and the sum of this quantity over all eight nodes, at several stages during training. The summed squared magnitude develops a ”bandpass filter” appearance for the following reasons. For each n, C,i starts as white noise and loses its high spatial frequency components during development, since the input signal is spatially correlated over short distances. The Fourier components of the connection strength also tend to decrease at low spatial frequencies k, where the input signal-to-noise ratio is largest, since devoting a large connection strength to transmitting a high-SNR component would be wasteful of the limited available output variance. Bandpass filter solutions have also been found for similar MI maximiza-

en,)

698

Ralph Linsker

Figure 2: Example of development of Fourier transform C n k of C,; for output nodes n, input sites i, and spatial frequencies k. Plotted are lC,z# (dotted and dashed curves) for two of the eight nodes n, and C,,le,,k(* (solid curve), vs. \k\, at (a) start of development (random C); (b,c) two intermediate times; (d) final state to which C has converged. Corresponding values of R(Z, S) are (a) 2.00, (b) 2.33, (c) 2.87, (d)3.04. See text for parameter values used. tion problems, in special cases where the form of C has been restricted to a translationally invariant Ansatz (e.g., C,, =_ C[(n/D’)- i/D]) [cf. Linsker (1989a) and Atick and Redlich (1990)l. 5 Discussion

It is useful and perhaps striking that learning rules constructed from simple Hebbian and anti-Hebbian modification of feedforward and lateral connections can perform gradient ascent in the mutual information to any desired accuracy, with no need for additional network complexity or nonlocal rules. The maximization of mutual information appears to have value for extracting statistical regularities, building ”feature” analyzers, and generating fruitful comparisons with biological data. For synthetic networks, the existence of a local algorithm increases the likelihood of feasible and efficient hardware implementations. For biological networks, such an algorithm is crucial if the proposed optimality principle or some variant of it is to be seriously considered as a candidate for a general task that a neural processing stage may learn to perform (Linsker 1988).


699

5.1 Relation to PCA. The optimization principle is related to PCA in the special case that the input noise covariance (A")= 71 and output noise variance ,4 + 0. Then the D' output nodes develop to span a D'-dimensional leading PCA subspace of the input space. Although the present MI-maximization algorithm is necessarily more complex, it resembles in some respects the PCA algorithm proposed by Foldihk (1989). An extension of PCA that is of practical interest for improving image SNR (Green et al. 1988) corresponds to the case in which the input noise N may have arbitrary correlations and one wants to select those components of the input that have maximum SNR in order to reconstruct a lessnoisy version of the input. The present work provides a local network algorithm suitable for selecting the appropriate components. 5.2 More General Distributions and Mappings. In the case of gaussian-distributed inputs to a linear processing stage, treated here, MI maximization strikes a balance between (1) maximizing the fraction of each node's output variance that reflects variance in the input signal as opposed to the input noise; (2) removing correlations among different nodes' output signal values, thereby reducing redundancy (Barlow 1989), when the output noise variance P is small; and (3) introducing redundancy to mitigate the information-destroying effects of output noise when P is large. The present algorithm can be applied to signal and noise distributions that are not multivariate gaussian, as well as to multistage processing systems in which a nonlinear transformation may be applied between stages. In these more general cases, the algorithm can be expected to strike a balance among the same three goals, although in general the MI will not be thereby maximized (since the MI reflects higher order correlations as well). The present learning algorithm, and extensions thereof, are likely to be well suited to nonlinear multistage networks in which one wishes to analyze low-order statistical correlations at each stage, with the goal of extracting higher-order featural information after a sequence of stages. 5.3 Lateral Connectivity and the Role of Selective "Unlearning". Two features of the lateral connectivity in the present work are of interest. First, the lateral connection strengths F,, depend on the correlation between output activities. (They may in general also depend on the distance between output nodes; we have omitted this for simplicity.) Second, two phases-"learning" and ''unlearning''-are used for distinguishing input signal from input noise, both of which may have correlational structure. The lateral connection strength depends on which phase the network is in at a given time. It would be of great interest to know to what extent biological learning rules and network architectures may exhibit qualitatively similar features.

700

Ralph Linsker

Two-phase learning, with input signal present in one phase and absent in the other, has played a role in some earlier work directed toward other goals.

1. A Boltzmann machine learns to generate activation patterns over a subset of nodes whose probability distributions are as nearly as possible the same during two phases: (i) when the nodes' activities are clamped by the inputs, and (ii) when they are free-running (Hinton and Sejnowski 1983). 2. "G-maximization" produces a single node whose output is maximally different, in the presence of input correlations, from what it would be if its inputs were uncorrelated (Pearlmutter and Hinton 1986). 3. The AR-Prule for reinforcement learning (Barto and Anandan 1985) uses a rule similar to Hebbian update when the output is "successful," and anti-Hebbian update otherwise. We may draw an analogy between reinforcement learning and the present work on unsupervised learning: "Success" may indicate that the network is responding to combinations of input that are relevant to the intended goal (hence to combinations that constitute input "signal"), while "failure" may indicate that the input combinations that evoked the response constitute "semantic noise." Extensions of the present algorithm along these lines may be applicable to reinforcement learning. 4. Finally, Crick and Mitchison (1983) have suggested that dream sleep may serve a selective "unlearning" role, by suppressing parasitic or otherwise undesired network interactions. The present algorithm offers a concrete example of the utility of such an "unlearning" process in a simple network. 6 Appendix

Derivation of equation 3.1: Since Q is a positive definite symmetric matrix, it can be shown that 1ndetQ = TrlnQ, and that the differential quantity dH = (l/Z)d(TrlnQ) = (1/2)Tr(Q-'dQ). Also, dQmp/dCni = (C9)mi6p,+ (C9),iSm,, where S denotes the Dirac delta function. Therefore, dH/aC,i = (Q-'Cq),j. Derivation of equation 3.4: The recursive method is due to Jacobi. Equations 3.2 and 3.3 yield ay(ca) = aCp",,F'Y = a(I - F)-'Y = Q-'Y. Combining this result with equation 3.1 yields the desired result. Convergence condition for a (see text following equation 3.5): Let f denote the eigenvalues of F, and let A+ denote the maximum and minimum eigenvalues of Q (0 < A- < A+). Iff a < 2/A+, then max I f ] < 1 and the series (I - F)-' = CZJ' converges.


701

+

For faster convergence, max If1 should be small. When a = 2/(A+ and equals (A+ - XL)/(A+ L). The convergence rate is slow when max If1 is close to unity, i.e., when the condition number of Q, A+/A-, is large. Note that the variance of the output noise term v, can be used to control the condition number: increasing all output noise variances by a constant increases A+ and A- by the same amount, hence decreases the condition number and improves convergence. Derivation of equations 3.8 and 3.10: V , = (CqLCT),, yields W,/X,, = 2(C9L)n,6,n.We also have

+

L), max If\ is minimized

Q& = (CqLCT)rnp+ P V m 6 m p

(A.1)

hence

+

dQL,,/dCm = (CqL)mi6pn+ (CqL)pJmn 2P(CqL)ni6mn6pn

(A.2)

and d[(1/2) lndetQLI/dCni = [(QL)-'C9']ni

+ P[(QL)-']nn(C9L)ni

(A.3)

Subtracting the corresponding expression involving QU yields equation 3.8. Next, using equation 3.9 and (vkv;) = PS,, we obtain

a+(yk(CO; 4 )v;)

a+( [CZ, (F4)tv'1n 4 ) = ( [(Q'I-'

v'] n

)

= &n[(Q4)-1]nm(vkv;)= P [ ( Q ' ) - l ] n n

(A.4)

Combining this result with (C9L),1= (YnXI)Lyields equation 3.10.

References Atick, J. J., and Redlich, A. N. 1990. Towards a theory of early visual processing. Neural Comp. 2, 308-320. Atick, J. J., and Redlich, A. N. 1991. Predicting ganglion and simple cell receptive field organizations. Int. J. Neural Syst. 1, 305-315. Barlow, H. B. 1989. Unsupervised learning. Neural Camp. 1,295-311. Barto, A. G., and Anandan, P. 1985. Pattern-recognizing stochastic learning automata. IEEE Trans. Sys. Man Cybern. 15, 360-375. Crick, F. H. C., and Mitchison, G. 1983. The function of dream sleep. Nature (London) 304, 111-114. FoldiBk, I? 1989. Adaptive network for optimal linear feature extraction. In Proc. IEEEIINNS Intern. Joint Conf. Neural Networks, Washington, DC, Vol. 1, pp. 401-405. IEEE Press, New York. Green, A. A., Berman, M., Switzer, P., and Craig, M. D. 1988. A transformation for ordering multispectral data in terms of image quality with implications for noise removal. I E E E Trans. Geosci. Remote Sensing 26, 65-74. Hinton, G. E., and Sejnowski, T. J. 1983. Optimal perceptual inference. Proc. I E E E Conf. Computer Vision, 448453. Leen, T. K. 1991. Dynamics of learning in linear feature-discovery networks. Network 2, 85-105.

Ralph Linsker

702

Linsker, R. 1988. Self-organization in a perceptual network. Computer 21(March), 105-11 7. Linsker, R. 1989a. An application of the principle of maximum information preservation to linear systems. In Advances in Nmral Information Processing Systems 1, D. S. Touretzky, ed., pp. 186-194. Morgan Kaufmann, San Mateo, CA. Linsker, R. 1989b. How to generate ordered maps by maximizing the mutual information between input and output signals. Neural Comp. 1,402-411. Pearlmutter, B. A., and Hinton, G. E. 1986. G-maximization: An unsupervised learning procedure for discovering regularities. In Neural Networks for Computing, J. S. Denker, ed., pp. 333-338. American Institute Physics, New York. Sanger, T. 1989. An optimality principle for unsupervised learning. In Advances in Neural Information Processing Systems 1 , D. S. Touretzky, ed., pp. 11-19. Morgan Kaufmann, San Mateo, CA. Shannon, C. E., and Weaver, W. 1949. The Mathematical Theory of Communication. Univ. of Illinois Press, Urbana. .-~

Received 27 December 1991; accepted 28 February 1992.

This article has been cited by: 2. Jim W. Kay, W. A. Phillips. 2010. Coherent Infomax as a Computational Goal for Neural Systems. Bulletin of Mathematical Biology . [CrossRef] 3. Ryotaro Kamimura. 2009. Structural enhanced information and its application to improved visualization of self-organizing maps. Applied Intelligence . [CrossRef] 4. Hao Miao, Xiaodong Li, Jing Tian. 2008. An EME blind source separation algorithm based on generalized exponential function. Journal of Electronics (China) 25:2, 262-267. [CrossRef] 5. Thomas D. Coates, Jr.. 2008. Neural Interfacing: Forging the Human-Machine Connection. Synthesis Lectures on Biomedical Engineering 3:1, 1-112. [CrossRef] 6. Hongtao Du, Hairong Qi, Xiaoling Wang. 2007. Comparative Study of VLSI Solutions to Independent Component Analysis. IEEE Transactions on Industrial Electronics 54:1, 548-558. [CrossRef] 7. R. Kamimura. 2006. Cooperative Information Maximization With Gaussian Activation Functions for Self-Organizing Maps. IEEE Transactions on Neural Networks 17:4, 909-918. [CrossRef] 8. Simone Fiori . 2005. Nonlinear Complex-Valued Extensions of Hebbian Learning: An EssayNonlinear Complex-Valued Extensions of Hebbian Learning: An Essay. Neural Computation 17:4, 779-838. [Abstract] [PDF] [PDF Plus] 9. X. Li, R. Du, X.P. Guan. 2004. Utilization of Information Maximum for Condition Monitoring With Applications in a Machining Process and a Water Pump. IEEE/ASME Transactions on Mechatronics 9:4, 711-714. [CrossRef] 10. Gurinder Atwal. 2004. Dynamic plasticity in coupled avian midbrain maps. Physical Review E 70:6. . [CrossRef] 11. Simone Fiori . 2003. Closed-Form Expressions of Some Stochastic Adapting Equations for Nonlinear Adaptive Activation Function NeuronsClosed-Form Expressions of Some Stochastic Adapting Equations for Nonlinear Adaptive Activation Function Neurons. Neural Computation 15:12, 2909-2929. [Abstract] [PDF] [PDF Plus] 12. Gal Chechik . 2003. Spike-Timing-Dependent Plasticity and Relevant Mutual Information MaximizationSpike-Timing-Dependent Plasticity and Relevant Mutual Information Maximization. Neural Computation 15:7, 1481-1510. [Abstract] [PDF] [PDF Plus] 13. Fabian J. Theis , Andreas Jung , Carlos G. Puntonet , Elmar W. Lang . 2003. Linear Geometric ICA: Fundamentals and AlgorithmsLinear Geometric ICA: Fundamentals and Algorithms. Neural Computation 15:2, 419-439. [Abstract] [PDF] [PDF Plus]

14. S. Fiori. 2002. Information-theoretic learning for FAN network applied to eterokurtic component analysis. IEE Proceedings - Vision, Image, and Signal Processing 149:6, 347. [CrossRef] 15. Ryotaro Kamimura, Taeko Kamimura, Thomas R. Shultz. 2001. Information Theoretic Competitive Learning and Linguistic Rule Acquisition. Transactions of the Japanese Society for Artificial Intelligence 16, 287-298. [CrossRef] 16. D. R. C Dominguez, M Maravall, A Turiel, J. C Ciria, N Parga. 1999. Numerical simulation of a binary communication channel: Comparison between a replica calculation and an exact solution. Europhysics Letters (EPL) 45:6, 739-744. [CrossRef] 17. Ralph Linsker . 1997. A Local Learning Rule That Enables Information Maximization for Arbitrary Input DistributionsA Local Learning Rule That Enables Information Maximization for Arbitrary Input Distributions. Neural Computation 9:8, 1661-1665. [Abstract] [PDF] [PDF Plus] 18. Marc M. Van Hulle. 1997. The Formation of Topographic Maps That Maximize the Average Mutual Information of the Output Responses to Noiseless Input SignalsThe Formation of Topographic Maps That Maximize the Average Mutual Information of the Output Responses to Noiseless Input Signals. Neural Computation 9:3, 595-606. [Abstract] [PDF] [PDF Plus] 19. Anthony J. Bell , Terrence J. Sejnowski . 1995. An Information-Maximization Approach to Blind Separation and Blind DeconvolutionAn Information-Maximization Approach to Blind Separation and Blind Deconvolution. Neural Computation 7:6, 1129-1159. [Abstract] [PDF] [PDF Plus] 20. Michael Haft, Martin Schlang, Gustavo Deco. 1995. Information theory and local learning rules in a self-organizing network of Ising spins. Physical Review E 52:3, 2860-2871. [CrossRef] 21. Marco Idiart, Barry Berk, L. F. Abbott. 1995. Reduced Representation by Neural Networks with Restricted Receptive FieldsReduced Representation by Neural Networks with Restricted Receptive Fields. Neural Computation 7:3, 507-517. [Abstract] [PDF] [PDF Plus] 22. G. Deco , W. Finnoff , H. G. Zimmermann . 1995. Unsupervised Mutual Information Criterion for Elimination of Overtraining in Supervised Multilayer NetworksUnsupervised Mutual Information Criterion for Elimination of Overtraining in Supervised Multilayer Networks. Neural Computation 7:1, 86-107. [Abstract] [PDF] [PDF Plus] 23. Dawn M. Adelsberger-Mangan, William B Levy. 1994. The influence of limited presynaptic growth and synapse removal on adaptive synaptogenesis. Biological Cybernetics 71:5, 461-468. [CrossRef] 24. L. F. Abbott. 1994. Decoding neuronal firing and modelling neural networks. Quarterly Reviews of Biophysics 27:03, 291. [CrossRef]

25. Dawn M. Adelsberger-Mangan, William B. Levy. 1993. Adaptive synaptogenesis constructs networks that maintain information and reduce statistical dependence. Biological Cybernetics 70:1, 81-87. [CrossRef] 26. Riccardo BoscoloIndependent Component Analysis . [CrossRef]

Communicated by Terrence J. Sejnowski

On the Information Storage Capacity of Local Learning Rules Giinther Palm Vogt-Institute for Brain Research, University of Diisseldorf D-4000 Diisseldorf, Germany

A simple relation between the storage capacity A for autoassociation and H for heteroassociation with a local learning rule is demonstrated: H = 2A. Both values are bounded by local learning bounds: A 5 LA and H 5 LH. L H = LA is evaluated numerically.

1 Introduction Neural networks with modifiable synaptic connections are now the standard modeling paradigm for learning and associative memory. The recent scientific literature on this subject contains an enormous number of such models, all of them very similar in their basic structure, their rules for synaptic modification, and their qualitative behavior, but different in many details. This paper is concerned with local two-term rules for synaptic modification used in large networks of nonlinear model neurons. We do not compare the different retrieval procedures in detail. For this reason we shall mostly concentrate on the storage procedures, i.e., on the first box in Figure 1. The criterion for a comparison and evaluation of different local synaptic storage procedures or learning rules will be the information storage capacity. This is essentially the channel capacity of the channel depicted in Figure 1, or-as in the next section-of the first box alone. More explicitly, we shall consider the amount of information about the input set S that can be obtained from the storage matrix M . One could agree on calling the capacity of the first box in Figure 1 the ”storage capacity,” the capacity of the last box, the ”retrieval capacity” and the capacity of the whole channel, the “memory capacity.” There are two essentially different cases, namely autoassociative and heteroassociative storage. In the next section we will demonstrate a relationship between the storage capacities for these two cases. Finally, in the last section we shall determine these capacities numerically. Neural Computation 4, 703-711 (1992) @ 1992 Massachusetts Institute of Technology

Giinther Palm

704

4

storage

Information to be stored

,passage

/ -

of time

state of the storage medium

S

> retrieval

I+

retrieved information

M

Figure 1: The memory channel.

2 Capacity of Local Storage Procedures, Heteroassociation versus

Autoassociation The most common synaptic arrangement in the cerebral cortex (and the hippocampus) is the simple dyadic synapse. It connects just between two neurons, the presynaptic and the postsynaptic one. Therefore there are just two natural, locally available activity signals: the presynaptic and the postsynaptic activity. Consequently we concentrate on two-term local synaptic rules, i.e., learning rules where the change of synaptic connectivity depends only on two variables x and y, which represent the pre- and postsynaptic activities. We consider two essentially different memory tasks, namely heteroassociation and autoassociation. In autoassociation a set S ( n ) = {x', . . . ,x"} of "patterns" (n-vectors) is stored by forming the connectivity matrix M = CEl R(xT,xF), where the function or rule Rjx,y) determines explicitly the amount of synaptic connectivity change for a pair (x,y) of preand postsynaptic activity values. In heteroassociation a set S(n,m ) = {(x',yl), . . . , (x",y")} of pairs of patterns is stored by forming the matrix M = C,&(xf,y;). The amount of information about S that can be retrieved from M is the storage capacity H ( n ,m ) for a heteroassociative n x m matrix, and A ( n )

Local Learning Rules

705

for an autoassociative n x n matrix (see also Palm 1980, 1982). Section 2 illustrates how this information can actually be obtained. Now the asymptotic storage capacity (per synapse) is defined as the limit A = limA(n)/n2 and H = limH(n,m)/(m.n). Clearly these two limits depend crucially on the properties of the sets S ( n ) or S(n,m) that are to be stored. In the framework considered here the sets S are assumed to be randomly generated: each component xi of each vector x is an independent binary random variable with the same probability q of becoming 1. Similarly the output patterns for heteroassociation are independently generated and p = prob[yr = 11. In the case of heteroassociation there is only one reasonable retrieval strategy. Given the storage matrix M and one input pattern x p one can try to obtain an estimate for the corresponding pattern yp by means of threshold detection in each component. Thus the problem of retrieving the outputs yp to their corresponding inputs x p from the matrix can be reduced to the problem of retrieving one component yj of one output pattern yf’ from the jth column of the matrix M and the input vector x b . The simplest and in fact most efficient estimate for yj can be obtained from the inner product of x p and this column (compare equation 3.1 below). This observation shows that the capacity H ( n ,m ) does not really depend on the second parameter m. We therefore also choose the parameters p ( n ) , q(n), and M ( n ) defining the sets S(n,m) as depending only on n, not on m. When we want to compare heteroassociation to autoassociation we have in addition to choose p ( n ) = q(n). One of the essential features of associative memory models is the distributed nature of the memory: different memories are allowed to overlap and thus to affect the same synapses, so that each entry in the synaptic connectivity matrix M may contain the superposition of several memory traces, i.e., for most index pairs i, j the sum CkR(xf,y); should have more than one nonzero contribution. This also implies a distributed representation of information in the patterns to be stored, i.e., in a usual pattern x there should be more than one nonzero component xi. This has the important consequence that a nontrivial “distance relation” between different patterns can be defined by the amount of overlap between them, whereas patterns without overlap all have the same maximal distance from each other (see also Palm 1988,1990). To represent this quest for a distributed representation in an associative memory, we explicitly introduce the following restriction on S: Distributedness Most of the patterns xk and yk occurring in the set S must have more than one nonzero component. For stochastic pattern generation distributedness simply means that p ( n ) > l / n and q(n) > l / n . Without this restriction it would be possible to store isolated activity patterns, each in one row (or column) of the memory matrix M, and address them by activity patterns with only one nonzero component in the place of the corresponding row (or column).

Giinther Palm

706

postsynaptic activity I

0

Figure 2: The four parameters rl ,r2, r3, r4 describing a local learning rule.

This would trivially lead to a storage capacity of H = 1, at least in the setup of binary activity vectors, or two-state neurons.

Proposition 0. Without requiring distributedness a heteroassociative storage capacity of H = 1 can be achieved, even with binary synapses. We now want to study the dependence of A = A(R) or H = H ( R ) on the choice of the rule R. To simplify the discussion, we restrict ourselves to network models with fwo-state neurons (as in the spin-glass Iiterature), although many of the subsequent arguments could be extended to more general neuron models. This restriction means that the variables x and y determining the local synaptic change, can have only two values, here a and 1, where a is usually either 0 or -1, and thus a rule R ( x , y) is defined by 4 numbers (rl ,r2, r3, r4) as in Figure 2. The following two propositions contain some simple observations based on linear algebra and elementary information theory. They are proved in Palm (1988).

Proposition 1. Let N be the set of all local rules R for which A(R) = 0. (1) N is a linear subspace of the space R of all local rules. (2) If Ro E N, then A(R) = A(R Ro) and H ( R ) = H ( R Ro) for any R E R.

+

+

Proposition 2. The subspace N of R is three-dimensional. It is spanned by ( l , l , l , l )(-1,1, , -l,l), and (-l,-l,l,l). The (up to a constant factor unique) rule C which is orthogonal to N is C = (1,-1, -1,l). There are two constants A # 0 and H # 0 such that A ( R )= A and H ( R ) = H for every R $ N.

+

Definition. A local rule R is called Hebb-like, if ( R I C) = rl - r2 - r3 r4 > 0, anti-Hebb, if ( R 1 C) < 0, and noninteractive if ( R I C) = 0. In the remainder of this section we demonstrate a simple relation between A and H , namely 2A = H .


707

Before we proceed let us recall the assumptions needed for the next three propositions (3-5). We assume local two-term rules, two-state neurons, stochastic pattern generation, and an arbitrary but fixed choice of the parameters M ( n ) and p ( n ) = q(n) as n goes to infinity. Proposition 3. H(n, i + j )

= H(n, i)

+ H(n,j) and H(i +j, n) 2H(i,n ) +

HO', n). Proof. Obvious.

Thus supH(n,m)/(rnn ) = liiH(n, rn)/(rn n ) = lipH(n, m)/(rnn ) n.m

+

Proposition 4. limn[A(n 1) - A(n)]/n = H for p ( n ) = 9(n). Proof. The matrix M realizing A(n + 1) can be decomposed into an n x n autoassociative matrix M', which approximately realizes the storage capacity A(n), two times the same vector x (appearing as an n x 1 and a 1 x n matrix) and one number. This decomposition shows that IA(n + 1)- A(n) - H ( n , l)l/n -+ 0. 0 Proposition 5. H = 2A for p ( n ) = q(n).

Proof. By Proposition 4 we can write [A(i)- A(i - l)]/i = H where e(i) goes to zero and thus e(i) < e for i 2 L. So, n

n--tm lim A(n)/n2

+ e(i),

n

=

limx[A(i)- A(i - l)]/n2 = l i m x [ H + e(i)]i/n2

=

lim H[n(n I ) - L ( L + 1)]/2n2

i=L

+

i=L

n

+ lim C e(i)i/n2= ~

/ 2

i=L

The result of Proposition 5 is intuitively plausible since autoassociation always leads to a symmetric storage matrix, and a symmetric matrix contains roughly half the information of an arbitrary matrix of the same size. 3 Capacity of Heteroassociative Storage and Retrieval, Evaluation of H and A

In this section we discuss certain particular choices of the parameters M ( n ) ,p ( n ) , and 9(n) for the case of heteroassociation, leading in particular to the optimal choice for fixed p ( n ) = p . For heteroassociative storage there is only one natural retrieval procedure: Given S = { ( x k , $ ) : k = 1,. . . , M } and the matrix M, we take each xk (k = 1,. . . ,M) and form xkM. Then we choose a vector 0 of detection thresholds and form yk = h(xkM - 0), i.e., yf = h[(xkM- O)i], where h ( x ) = 1 if x > 0 and h ( x ) = 0 otherwise. This vector y" is taken

Gunther Palm

708

as an estimate for the desired output vector yk and we have to estimate the amount of information that is needed to correct the errors, i.e., the deviation between @ and y" for an optimally chosen threshold 0. The estimate yf for the jth component of the kth output pattern yk is obtained as

In this expression, N contains no information about yf and can be regarded as noise, S can be regarded as the signal containing the information on $, and 0 = 0j can be regarded as a detection threshold. Obviously the rule R should be chosen in such a way that S is large for yf = 1 and small for yf = 0. We define the signal-to-noise ratio r for the detection problem as Y := E[C,$R($,

1) - Cj$R(xf, O ) ] / a ( N )

where a(N)denotes the standard deviation of N,given the input xk. This signal detection problem has been analyzed by several authors (Palm 1988, 1990; Willshaw and Dayan 1990; Nadal and Toulouse 1990) and there is one result that is of particular interest here. It concerns the so called sparse kigk-fidelity limit. In this limit the parameters p(n), 9(n), and M(n) are chosen in such a way that p(n)/ 9(n) as well as the error probabilities converge to zero as n goes to infinity. In this limit a capacity of 1/(2 In 2) can be achieved with optimal threshold detection. Thus H = 1/(2 In 2) for this choice of parameters p(n), 9(n),and M ( n ) . Furthermore the local rule achieving maximal signal-to-noise ratio Y cart be identified (see Palm 1990; Willshaw and Dayan 1990)and its value for Y is (3.2)

12 = I/bP(1 - PI1

From this relation one can immediately see that high fidelity, i.e., Y co and nonzero capacity ( a 0), can be achieved only if p(1 - p ) goes to zero. Here it is reasonable to break the inherent symmetry and require that p ---t 0. With this knowledge we can also determine H for another extreme choice of parameters: the no-fidelity or error-full case. In this case we take p(n) = p and 9(n) = 9, both constant, and let M ( n ) increase to extremely large values so that r 0 and both error probabilities converge to 112 as TI 4 00. Still one can retrieve (at least in principle) some information from the memory matrix. Given p19 and M and therefore r, we can estimate the error probabilities by means of the cumulative gaussian distribution G as e = G(-r/2). -+

--f

-+


709

Since e + 1/2 and thus r + 0, we can approximate G(-r/2) as e = G(-r/2)

=

1/2 - r/(2v%)

(3.3)

l W

(3.4)

Thus

H

=

N P ) -P

-

e)/Pll

-

PowPo)}

Here I ( p ) = - p log p - (1 - p ) log,(l - p ) and p l = p(l - e) = pe (1 - p)(l - e). The second order approximation to I@’) around p is

+

po

(In 2 ) W ) = (In 2MP) + (P’-P)ln[(l -P)/Pl-

+ (1 - p ) e and

(P’-P)2/[2P(l -PI1 (3.5)

If we insert equations 3.2, 3.3, and 3.5 into equation 3.4 we obtain H

= l / ( In ~ 2)

(3.6)

By Proposition 5 we obtain A = 1/(27r In 2). Interestingly these values turn out to be independent of p and 9. We summarize these observations in the following proposition. Proposition 6. In the sparse limit p + 0 a capacity of H = 1/(2 In 2) can be achieved with high fidelity retrieval. In the nonsparse case for arbitrary fixed p , a capacity of H = l / ( In~ 2) can always be achieved in the no-fidelity limit. Of course, it does not make too much sense to let M ( n ) increase so quickly that both error probabilities go to 1/2. We have analyzed more judicious choices of M ( n ) numerically, and it turns out that one can always reach values above the no-fidelity limit, although we have never been able to reach the sparse high-fidelity limit. For p = 9 = 1/2 the no-fidelity limit actually seems to be the optimum. We finally consider the general case where p ( n ) = 9(n) = p . In the case of heteroassociation it turns out that H does not depend on the choice of 9(n), so we can choose 9(n) = p ( n ) for easier comparison to autoassociation. We may define the local learning bound Lh(p) or L,(p) as the optimal value for H , or A, respectively, that can be achieved for p(n)[=9(n)] = p and the best choice of M ( n ) . Our numerical investigations of this bound suggest the following proposition. Proposition 7. Lh(p) decreases monotonically from L h ( 0 ) := lim L h ( p ) = 1/ (2 In 2) to L h (1/2) = 1/ ( T In 2). By Proposition 5 again L h = 2L,. Our final result as stated in Propositions 6 and 7 is certainly important for a large class of memory models based on local learning rules in neural networks. In particular, the result on autoassociation can be used as an upper bound for the memory capacities that can be achieved with concrete local learning rules and concrete retrieval procedures, like the fixed-point information retrieval capacities for a number of Hopfield-like

710

Giinther Palm

Figure 3: The local learning bound LH as a function of the density p of ones. spin-glass models (e.g., Hopfield 1982; Amit ef al. 1987; Horner 1989; Tsodyks and Feigelman 1988). We have recently been able to determine the function Lh(p) numerically; a plot is provided in Figure 3.

References Amit, D. J., Gutfreund, H., and Sompolinsky, H. 1987. Information storage in neural networks with low levels of activity. Phys. Rev. A 35, 2293-2303. Hopfield, J. J. 1982. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. U.S.A. 79, 2554-2558. Horner, H. 1989. Neural networks with low levels of activity: king vs. McCulloch-Pitts neurons. Z. Phys. B 75, 133-136. Nadal, J. P., and Toulouse, G. 1990. Information storage in sparsely coded memory nets. Network I, 61-74. Palm, G . 1980. On associative memory. Biol. Cybern. 36, 19-31. Palm, G. 1982. Rules for synaptic changes and their relevance for the storage of information in the brain. In Cybernetics and Systems Research, R. Trappl (Ed.), pp. 277-280. North-Holland Publishing Company, Amsterdam. Palm, G. 1988. On the asymptotic information storage capacity of neural networks. In Neural Computers, C. Von der Malsburg and R. Eckmiller (Eds.), pp. 271-280. Springer-Verlag, Berlin.


711

Palm, G. 1990. Local learning rules and sparse coding in neural networks. In Advanced Neural Computers, R. Eckmiller (Ed.), pp. 145-150. Elsevier-Science Publishers B.V., North-Holland. Tsodyks, M. V., and Feigelman, M. V. 1988. The enhanced storage capacity in neural networks with low activity level. Europhys. Left. 6, 101-105. Willshaw, D., and Dayan, P. 1990. Optimal plasticity from matrix memories: What goes up must come down. Neural Comp. 2,85-93.

Received 11 December 1990; accepted 18 March 1992.

This article has been cited by: 2. Gal Chechik , Isaac Meilijson , Eytan Ruppin . 2001. Effective Neuronal Learning with Ineffective Hebbian Learning RulesEffective Neuronal Learning with Ineffective Hebbian Learning Rules. Neural Computation 13:4, 817-840. [Abstract] [PDF] [PDF Plus] 3. Sepp Hochreiter , Jürgen Schmidhuber . 1999. Feature Extraction Through LOCOCODEFeature Extraction Through LOCOCODE. Neural Computation 11:3, 679-714. [Abstract] [PDF] [PDF Plus]

Communicated by Haim Sompolinsky

Learning Curves for Error Minimum and Maximum Likelihood Algorithms Y. Kabashima S. Shinomoto Department of Physics, Kyoto University, Kyoto 606, Japan

For the problem of dividing the space originally pad tionec ~y a blurred boundary, every learning algorithm can make the probability of incorrect prediction of an individual example E decrease with the number of training examples t. We address here the question of how the asymptotic form of ~ ( tas ) well as its limit of convergence reflect the choice of learning algorithms. The error minimum algorithm is found to exhibit rather slow convergence of E ( t ) to its lower bound E ~ &(f) , - EO O(t-2/3).Even for the purpose of minimizing prediction error, the maximum likelihood algorithm can be utilized as an alternative. If the true probability distribution happens to be contained in the family of hypothetical functions, then the boundary estimated from the hypothetical distribution function eventually converges to the best choice. Convergence of the prediction error is then E ( t ) - E O O(t-'). If the true distribution is not available from the algorithm, however, the boundary generally does not converge to the best choice, but instead E ( t ) - EI fO(t-'/2), where > EO > 0.

-

-

-

1 Introduction

The original purpose of machine learning is to adjust the machine parameters so as to reproduce the input-output relationship implied by the examples. Learning situations can be classified into two cases depending upon whether or not the machine is in principle able to completely reproduce the individual examples. In the case that the machine is able to reproduce examples, the remaining interest is the estimate of generalization error: the probability of the incorrect prediction E of a novel example provided that the machine has succeeded to reproduce t examples. The problem has currently been resolved by two means: computational theoretical, and statistical mechanical. First, the idea of PAC learning by Valiant (1984) was applied by Baum and Haussler (1989) to the worst case estimate of the generalization error of the neural network models. Second, a statistical mechanical theory for typical case estimate of the generalization error is formulated under the Bayes formula by Levin et Neural Computation 4, 712-719 (1992) @ 1992 Massachusetts Institute of Technology

Error Minimum and Maximum Likelihood

713

al. (1990). Amari et al. (1992) classified the asymptotic scaling forms of the ) four types. The statistical theory is not restricted learning curves ~ ( tinto to the case that a machine can reproduce the raw examples. Actually, two among the four types of the scaling forms are concerned with the case that the examples shown by the supervisor are more or less noisy. We take up here the convergence of prediction error for dividing the space originally partitioned by a blurred boundary. The purpose of the learning would not be unique in this case; one may seek the probability distribution of the classification, or one may seek the best boundary so as to minimize the prediction error for individual examples. The maximum likelihood algorithm and the error minimum algorithm are the corresponding standard strategies for these motivations. The two strategies are identical if the family of hypothetical distribution functions for the maximum likelihood algorithm are stepwise and symmetrical [see Rissanen (1989); Levin et al. (1990)l. In the case that the hypothetical distribution functions are smooth, however, the two strategies are generally different from each other. We found that the convergence of the error minimum algorithm is rather slow. In this algorithm, c ( t ) converges to the lower bound EO with O(t-2/3).We will explain the source of the the asymptotic form E ( t ) - E O fractional exponent 2/3 theoretically. Even for the purpose of minimizing prediction error, we can use the maximum likelihood algorithm as an alternative. In this case, the boundary can be defined as a hypersurface on which the hypothetical probabilities for alternative classes balance with each other. If the true probability distribution is available from the algorithm, the prediction error converges rapidly as E ( t ) - EO O(t-'). In the case that the true distribution is not available from the algorithm, the boundary generally does not converge to the best choice, but ~ ( t-)~1 +O(t-'/2),where EI > EO > 0.

-

-

-

2 Numerical Simulation

We first show the result of numerical simulation of the following simple partition problem. Every example consists of the real input x E [0,1]and the binary output s = f l . Real number x is drawn independently from the uniform distribution over the interval, p ( x ) = 1. The probability of ~ p ( s = -1 I getting s = f l depends on x as p ( s = fl I x) = 0.1 0 . 7 and X ) = l-p(s = +1 1 x ) (Fig. la). We examined the following three strategies for the partition of the interval: (1) the error minimum algorithm, ( 2 ) the maximum likelihood algorithm with the family of probability functions qw(s = +1 I x ) = w 0.7x, and (3) the maximum likelihood with q,(s = +1 I X ) = w 0 . 4 ~ . The error minimum algorithm seeks the partition that minimizes the total number of the left points with s = +1 and the right points with s = -1. As the number of examples increases the partition point x, is

+

+

+

714

Y. Kabashima and S. Shinomoto

t

Figure 1: (a) The original probability distribution p ( s = +1 I x ) = 0.1 + 0 . 7 ~ . (b) The best partition (0 = 4/7 for the error minimum. (c-e) The best hypothetical distribution functions for the maximum likelihood with pw(s = +1 I x) = w + 0.7x,w + 0.4x, and 6 + (1 - 26)8(x - w).

expected to approach the point t o = 417 at which the probabilities for the alternative classes balance: p ( s = $1 I to)= p(s = -1 I to)(Fig. lb). For the given partition at xo, the probability of incorrect prediction is E = EO a(x, - to)’, where EO = 9/28 and a = 0.7’. In this algorithm, the possible position of the optimal partition x, is given by the interval of adjacent points (xi, x,) and the error measure has to be averaged over the interval.

+


715

The maximum likelihood algorithm seeks the optimal parameter value

w,, which maximizes the likelihood function, (2.1) S ,X

The original probability distribution is available from the algorithm (2), and the optimal parameter w,is expected to approach wo = 0.1, which minimizes the Kullback divergence. As a result, the optimal partition x, estimated by sW,,(s= +l I x,) = sw(,(s= -1 I x,) eventually approaches to the best choice, t o = 4/7 (Fig. lc). On the other hand, algorithm ( 3 ) does not contain the true distribution function, and the optimal parameter is expected to approach to a value w1, which minimizes the Kullback divergence. In this case, the optimal partition x, approaches to a point (7 remote from to= 4/7 (Fig. Id). Note again that the maximum likelihood is identical to the error minimum if the family of hypothetical functions is stepwise and symmetrical (Fig. le), although this is rather exceptional as a maximum likelihood algorithm. In the numerical simulation, the three algorithms are carried out to obtain the optimal partition x, and the prediction error E for the set of t examples drawn from the distribution, p(s I x)p(x). The average of the prediction error E taken over 1000 sets of examples is plotted in Figure 2. The plots of E ( t ) - EO for (1) and (2) exhibit the scaling E ( t ) - EO O(t-"), with the exponents IY = 0.670 f0.004 and (Y = 1.012+L 0.008, respectively. The prediction error according to the algorithm (3)does not converge to the lower bound EO but to € I ( > E O ) . The mean square deviation of E ( t ) from ~1 in case ( 3 ) is depicted in Figure 3, where we can see & ( t )- €1 f O (t-") with the exponent N = 0.520f0.004. These results are examined in the next section. N

-

3 Theoretical Interpretation

In order to elucidate the nontrivial exponent obtained from the error minimum (l), we wish to consider here the simpler situation that the examples are arranged at regular intervals of l / t , assuming the same form for p(s I x). Let each example be denoted by the sequence j from 1 to t. The probability of sj taking the value s = fl is given by p(s I x = j / t ) . The total number of errors for the partition between i and i + 1 is

E,

=

k(l+ s,)/2 + f:(1

~

]=1

s,)/2

(3.1)

j=i+l

The expectation value of the number of errors is estimated as

(6) (Ern)+ ( a / t ) ( i - m)' N

(3.2)

where m is the best partition which minimizes the difference of the alternative probabilities, Ip(s = +1 I x = m / t ) - p ( s = -1 I x = m / t ) ] . O n

Y.Kahashima and S. Shinomoto

716

G-E >

average prediction error

0

lo2 k

k

I O3 1O4

1O5

103

104

105

Figure 2: Average of E ( t ) - E O . (b), (c), and (d) correspond to cases (l),(2), and (31, respectively. The lines for (1) and (2) were drawn from the least square fit: E ( t ) - EO IX t P with the exponents N = 0.670 f 0.004 and 1.012 f 0.008, respectively. the other hand, the mean square deviation of the difference E; - E,, is approximated as

AE2 = ( ( E ; - Em)') - ( ( E i - Em))*

-

(i - rnl

(3.3)

This is the result of a "random walk" of E; (see Fig. 4). Thus the optimal partition i that minimizes the number of errors E; can fluctuate around rn. The order of the deviation is estimated by the balance (AE( (Ei - Em), which implies Ii - m( O(t2/3),or fx, - (01

-

-

0(t-'/3)

-

(3.4)


\

Figure 4: Schematic representation of the fluctuation of E; around ( E i ) . where w = limf-m(wo(f)). Thus the deviation of w, from w is of the order of t-'/2. Deviation of the position of the boundary xu from = limf+m(x,(t))is proportional to the one of w o from w. In the case that the limit [ is identical to the best choice, = (0, the prediction error is estimated as

> s >> 1 (see Fig. lc). Because of this it might be wise to use numerical integration then implement AS:) in look-up tables.

David J. C. MacKay

734

-20

-10

0

10

20

30

Figure 7 Demonstration of expected mean marginal information gain. The mean marginal information gain was computed for the network demonstrated in Figure 2b,c. The region of interest was defined by 100 data points from the test set. The gray level represents the utility of a single observation as a function of where it is made. The darkest regions are expected to yield little information, and white corresponds to large expected information gain. The contours that are superposed represent the moderated output of the network, as shown in Figure 2c. The mean marginal information gain is quantified: the gray scale is linear from 0 to 0.0025 nats.

5 Discussion 5.1 Moderated Outputs. The idea of moderating the outputs of a classifier in accordance with the uncertainty of its parameters should have wide applicability, for example, to hidden Markov models for speech recognition. Moderation should be especially important where a classifier is expected to extrapolate to points outside the training region. There is presumably a relationship of this concept to the work of Seung et al. (1991) on generalization ”at nonzero temperature.”

Evidence Framework Applied to Classification Networks

735

If the suggested approximation to the moderated output and its derivative is found unsatisfactory, a simple brute force solution would be to set up a look-up table of values of +(a,s2) and +'(u,s2). It is likely that an implementation of marginalization that will scale up well to large problems will involve Monte Carlo methods (Neal 1992). 5.2 Evidence. The evidence has been found to be well correlated with generalization ability. This depends on having a sufficiently large amount of data. There remain open questions, including what the theoretical relationship between the evidence and generalization ability is, and how large the data set must be for the two to be well correlated, how well these calculations will scale up to larger problems, and when the quadratic approximation for the evidence breaks down. 5.3 Mean Marginal Information Gain. This objective function was derived with active learning in mind. It could also be used for selection of a subset of a large quantity of data, as a filter to weed out fractions of the data that are unlikely to be informative. Unlike Plutowski and White's (1991) approach this filter depends only on the input variables in the candidate data. A strategy that selectively omits data on the basis of their output values would violate the likelihood principle and risk leading to inconsistent inferences. A comparison of the mean marginal information gain in Figure 7 with the contours of the most probable networks output in Figure 2b indicates that this proposed data selection criterion offers some improvements over the simple strategy of just sampling on and near decision boundaries: the mean marginal information gain shows a plausible preference for samples in regions where the decision boundary is uncertain. On the other hand, this criterion may give artifacts when applied to models that are poorly matched to the real world. How useful the mean marginal information gain will be for real applications remains an open question.

Acknowledgments This work was supported by a Caltech Fellowship and a Studentship from SERC, UK. References Baum, E. 8. 1991. Neural net algorithms that learn in polynomial time from examples and queries. I E E E Truns. Neurul Networks 2(1), 5-19. Bishop, C. M. 1992. Exact calculation of the Hessian matrix for the multilayer perceptron. Neurul Comp. 4(4), 494-501.

David J. C. MacKay

736

Bridle, J. S. 1989. Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In Neurocomputing: Algorithms, Architectures and Applications, F. Fougelman-Soulie and J. Hkrault, eds., pp. 227-236. Springer-Verlag, Berlin. Hinton, G. E., and Sejnowski, T. J. 1986. Learning and relearning in Boltzmann machines. In Parallel Distributed Processing, Rumelhart et al., eds., pp. 282317. The MIT Press, Cambridge. Hopfield, J. J. 1987. Learning algorithms and probability distributions in feedforward and feed-back networks. Proc. Natl. Acad. Sci. U.S.A. 84,8429-8433. Hwang, J-N., Choi, J. J., Oh, S., and Marks, R. J., I1 1991. Query-based learning applied to partially trained multilayer perceptrons. IEEE Trans. Neural Networks 2(1), 131-136. MacKay, D. J. C. 1992a. Bayesian interpolation. Neural Comp. 4(3), 415-447. MacKay, D. J. C. 1992b. A practical Bayesian framework for backprop networks. Neural Comp. 4(3), 448-472. MacKay, D. J. C. 1992~.Information-based objective functions for active data selection. Neural Comp. 4(4), 589-603. Neal, R. M. 1992. Bayesian training of backpropagation networks by the Hybrid Monte Carlo Method. University of Toronto CRG-TR-92-1. Plutowski, M., and White, H. 1991. Active selection of training examples for network learning in noiseless environments. Dept. Computer Science, UCSD, TR 90-011. Seung, H. S., Sompolinsky, H., and Tishby, N. 1991. Statistical mechanics of learning from examples. Preprint, Racah Institute of Physics, Israel. Solla, S. A., Levin, E., and Fleisher, M. 1988. Accelerated learning in layered neural networks. Complex Syst. 2, 625-640. Spiegelhalter, D. J., and Lauritzen, S. L. 1990. Sequential updating of conditional probabilities on directed graphical structures. Networks 20, 579-605. ~~

Received 20 November 1991; accepted 18 February 1992.

This article has been cited by: 2. Ashok K. Mishra, Mehmet Özger, Vijay P. Singh. 2010. Wet and dry spell analysis of Global Climate Model-generated precipitation using power laws and wavelet transforms. Stochastic Environmental Research and Risk Assessment . [CrossRef] 3. J V Marcos, R Hornero, D Álvarez, I T Nabney, F del Campo, C Zamarrón. 2010. The classification of oximetry signals using Bayesian neural networks to assist in the detection of obstructive sleep apnoea syndrome. Physiological Measurement 31:3, 375-394. [CrossRef] 4. Zahra Moravej, Mohammad Pazoki, Ali Akbar Abdoos. 2010. Wavelet transform and multi-class relevance vector machines based recognition and classification of power quality disturbances. European Transactions on Electrical Power n/a-n/a. [CrossRef] 5. Arvind Tolambiya, S. Venkatraman, Prem K. Kalra. 2010. Content-based image classification with wavelet relevance vector machines. Soft Computing 14:2, 129-136. [CrossRef] 6. D. P. Vetrov, D. A. Kropotov, N. O. Ptashko. 2009. An efficient method for feature selection in linear regression based on an extended Akaike’s information criterion. Computational Mathematics and Mathematical Physics 49:11, 1972-1985. [CrossRef] 7. Paulo J. G. Lisboa, Terence A. Etchells, Ian H. Jarman, Corneliu T. C. Arsene, M. S. Hane Aung, Antonio Eleuteri, Azzam F. G. Taktak, Federico Ambrogi, Patrizia Boracchi, Elia Biganzoli. 2009. Partial Logistic Artificial Neural Network for Competing Risks Regularized With Automatic Relevance Determination. IEEE Transactions on Neural Networks 20:9, 1403-1416. [CrossRef] 8. D. Kropotov, N. Ptashko, D. Vetrov. 2009. Relevant regressors selection by continuous AIC. Pattern Recognition and Image Analysis 19:3, 456-464. [CrossRef] 9. D. Kropotov, D. Vetrov. 2009. General solutions for information-based and Bayesian approaches to model selection in linear regression and their equivalence. Pattern Recognition and Image Analysis 19:3, 447-455. [CrossRef] 10. Ashok K. Mishra, Mehmet Özger, Vijay P. Singh. 2009. Trend and persistence of precipitation under climate change scenarios for Kansabati basin, India. Hydrological Processes 23:16, 2345-2357. [CrossRef] 11. Shirish Shevade, S. Sundararajan. 2009. Validation-Based Sparse Gaussian Process Classifier DesignValidation-Based Sparse Gaussian Process Classifier Design. Neural Computation 21:7, 2082-2103. [Abstract] [Full Text] [PDF] [PDF Plus] 12. Huanhuan Chen, P. Tino, Xin Yao. 2009. Probabilistic Classification Vector Machines. IEEE Transactions on Neural Networks 20:6, 901-914. [CrossRef]

13. J.G. Hincapie, R.F. Kirsch. 2009. Feasibility of EMG-Based Neural Network Controller for an Upper Extremity Neuroprosthesis. IEEE Transactions on Neural Systems and Rehabilitation Engineering 17:1, 80-90. [CrossRef] 14. Andrew E. Mercer, Chad M. Shafer, Charles A. Doswell, Lance M. Leslie, Michael B. Richman. 2009. Objective Classification of Tornadic and Nontornadic Severe Weather Outbreaks. Monthly Weather Review 137:12, 4355. [CrossRef] 15. A. K. Mishra, Vijay P. Singh. 2009. Analysis of drought severity-area-frequency curves using a general circulation model and scenario uncertainty. Journal of Geophysical Research 114:D6. . [CrossRef] 16. Tetsuya Shimokawa, Tadanobu Misawa, Kyoko Suzuki. 2008. Neural representation of preference relationships. NeuroReport 19:16, 1557-1561. [CrossRef] 17. Stefan Lessmann, Bart Baesens, Christophe Mues, Swantje Pietsch. 2008. Benchmarking Classification Models for Software Defect Prediction: A Proposed Framework and Novel Findings. IEEE Transactions on Software Engineering 34:4, 485-496. [CrossRef] 18. Shian-Chang Huang, Tung-Kuang Wu. 2008. Combining wavelet-based feature extractions with relevance vector machines for stock index forecasting. Expert Systems 25:2, 133-149. [CrossRef] 19. Satohiro Tajima, Masato Inoue, Masato Okada. 2008. Bayesian-Optimal Image Reconstruction for Translational-Symmetric Filters. Journal of the Physical Society of Japan 77:5, 054803. [CrossRef] 20. E. Ronchi, S. Conroy, E. A. Sundén, G. Ericsson, A. Hjalmarsson, C. Hellesen, M. G. Johnson, M. Weiszflog, JET-EFDA Contributors. 2008. A neural networks framework for real-time unfolding of neutron spectroscopic data at JET. Review of Scientific Instruments 79:10, 10E513. [CrossRef] 21. Chang Kook Oh, James L. Beck, Masumi Yamada. 2008. Bayesian Learning Using Automatic Relevance Determination Prior with an Application to Earthquake Early Warning. Journal of Engineering Mechanics 134:12, 1013. [CrossRef] 22. Cristian Sminchisescu, Atul Kanaujia, Dimitris N. Metaxas. 2007. BM³E : Discriminative Density Propagation for Visual Tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence 29:11, 2030-2044. [CrossRef] 23. BegÜm Demir, Sarp Erturk. 2007. Hyperspectral Image Classification Using Relevance Vector Machines. IEEE Geoscience and Remote Sensing Letters 4:4, 586-590. [CrossRef] 24. Muhammad Aqil, Ichiro Kita, Akira Yano, Soichi Nishiyama. 2007. Neural Networks for Real Time Catchment Flow Modeling and Prediction. Water Resources Management 21:10, 1781-1796. [CrossRef] 25. Andrew I. Schein, Lyle H. Ungar. 2007. Active learning for logistic regression: an evaluation. Machine Learning 68:3, 235-265. [CrossRef]

26. Faming Liang. 2007. Annealing stochastic approximation Monte Carlo algorithm for neural network training. Machine Learning 68:3, 201-233. [CrossRef] 27. Alexander Schmolck, Richard Everson. 2007. Smooth relevance vector machine: a smoothness prior extension of the RVM. Machine Learning 68:2, 107-135. [CrossRef] 28. Feng Zheng, Yejun Qin, Kun Chen. 2007. Sensitivity map of laser tweezers Raman spectroscopy for single-cell analysis of colorectal cancer. Journal of Biomedical Optics 12:3, 034002. [CrossRef] 29. K. Kobayashi, F. Komaki. 2006. Information Criteria for Support Vector Machines. IEEE Transactions on Neural Networks 17:3, 571-577. [CrossRef] 30. G.C. Cawley, N.L.C. Talbot, G.J. Janacek, M.W. Peck. 2006. Sparse Bayesian Kernel Survival Analysis for Modeling the Growth Domain of Microbial Pathogens. IEEE Transactions on Neural Networks 17:2, 471-481. [CrossRef] 31. S. Gareth Pierce, Yakov Ben-Haim, Keith Worden, Graeme Manson. 2006. Evaluation of Neural Network Robust Reliability Using Information-Gap Theory. IEEE Transactions on Neural Networks 17:6, 1349-1361. [CrossRef] 32. Faming Liang . 2005. Evidence Evaluation for Bayesian Neural Networks Using Contour Monte CarloEvidence Evaluation for Bayesian Neural Networks Using Contour Monte Carlo. Neural Computation 17:6, 1385-1410. [Abstract] [PDF] [PDF Plus] 33. Shovan K. Majumder, Nirmalya Ghosh, Pradeep K Gupta. 2005. Relevance vector machine for optical diagnosis of cancer. Lasers in Surgery and Medicine 36:4, 323-333. [CrossRef] 34. Shuang Yang, Antony Browne. 2004. Neural network ensembles: combining multiple models for enhanced performance using a multistage approach. Expert Systems 21:5, 279-288. [CrossRef] 35. S. Sigurdsson, P.A. Philipsen, L.K. Hansen, J. Larsen, M. Gniadecka, H.C. Wulf. 2004. Detection of Skin Cancer by Classification of Raman Spectra. IEEE Transactions on Biomedical Engineering 51:10, 1784-1793. [CrossRef] 36. Vasily Belokurov, N. Wyn Evans, Yann Le Du. 2004. Light-curve classification in massive variability surveys - II. Transients towards the Large Magellanic Cloud. Monthly Notices of the Royal Astronomical Society 352:1, 233-242. [CrossRef] 37. D. Chakraborty, N.R. Pal. 2003. A novel training scheme for multilayered perceptrons to realize proper generalization and incremental learning. IEEE Transactions on Neural Networks 14:1, 1-14. [CrossRef] 38. T. Van Gestel , J. A. K. Suykens , G. Lanckriet , A. Lambrechts , B. De Moor , J. Vandewalle . 2002. Bayesian Framework for Least-Squares Support Vector Machine Classifiers, Gaussian Processes, and Kernel Fisher Discriminant AnalysisBayesian Framework for Least-Squares Support Vector

Machine Classifiers, Gaussian Processes, and Kernel Fisher Discriminant Analysis. Neural Computation 14:5, 1115-1147. [Abstract] [PDF] [PDF Plus] 39. A.P. Engelbrecht. 2001. A new pruning heuristic based on variance analysis of sensitivity information. IEEE Transactions on Neural Networks 12:6, 1386-1399. [CrossRef] 40. S. Chen, S.R. Gunn, C.J. Harris. 2001. The relevance vector machine technique for channel equalization application. IEEE Transactions on Neural Networks 12:6, 1529-1532. [CrossRef] 41. Tony A. Plate , Joel Bert , John Grace , Pierre Band . 2000. Visualizing the Function Computed by a Feedforward Neural NetworkVisualizing the Function Computed by a Feedforward Neural Network. Neural Computation 12:6, 1337-1353. [Abstract] [PDF] [PDF Plus] 42. S. Roberts, I. Rezek, R. Everson, H. Stone, S. Wilson, C. Alford. 2000. Automated assessment of vigilance using committees of radial basis function analysers. IEE Proceedings - Science, Measurement and Technology 147:6, 333. [CrossRef] 43. James Tin-Yau Kwok. 2000. The evidence framework applied to support vector machines. IEEE Transactions on Neural Networks 11:5, 1162-1173. [CrossRef] 44. A.K. Jain, P.W. Duin, Jianchang Mao. 2000. Statistical pattern recognition: a review. IEEE Transactions on Pattern Analysis and Machine Intelligence 22:1, 4-37. [CrossRef] 45. S. J. Roberts, W. D. Penny. 2000. Real-time brain-computer interfacing: A preliminary study using Bayesian learning. Medical & Biological Engineering & Computing 38:1, 56-61. [CrossRef] 46. D.J.C. Mackay, M.N. Gibbs. 2000. Variational Gaussian process classifiers. IEEE Transactions on Neural Networks 11:6, 1458-1464. [CrossRef] 47. David J. C. MacKay . 1999. Comparison of Approximate Methods for Handling HyperparametersComparison of Approximate Methods for Handling Hyperparameters. Neural Computation 11:5, 1035-1068. [Abstract] [PDF] [PDF Plus] 48. Bernhard Schottky, David Saad. 1999. Journal of Physics A: Mathematical and General 32:9, 1605-1621. [CrossRef] 49. P.J. Edwards, A.F. Murray, G. Papadopoulos, A.R. Wallace, J. Barnard, G. Smith. 1999. The application of neural networks to the papermaking industry. IEEE Transactions on Neural Networks 10:6, 1456-1464. [CrossRef] 50. J.T.-Y. Kwok. 1999. Moderating the outputs of support vector machine classifiers. IEEE Transactions on Neural Networks 10:5, 1018-1031. [CrossRef] 51. S.J. Roberts, D. Husmeier, I. Rezek, W. Penny. 1998. Bayesian approaches to Gaussian mixture modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence 20:11, 1133-1142. [CrossRef]

52. H.H. Thodberg. 1996. A review of Bayesian neural networks with an application to near infrared spectroscopy. IEEE Transactions on Neural Networks 7:1, 56-72. [CrossRef] 53. R. Rovatti, R. Ragazzoni, Zs. M. Kovàcs, R. Guerrieri. 1995. Adaptive Voting Rules for k-Nearest Neighbors ClassifiersAdaptive Voting Rules for k-Nearest Neighbors Classifiers. Neural Computation 7:3, 594-605. [Abstract] [PDF] [PDF Plus] 54. Chris M. Bishop. 1994. Neural networks and their applications. Review of Scientific Instruments 65:6, 1803. [CrossRef] 55. Graeme Manson, Keith Worden, S. Gareth Pierce, Thierry DenoeuxUncertainty Analysis . [CrossRef]

Communicated by Fernando Pineda

Rotor Neurons: Basic Formalism and Dynamics Lars G i s l h Carsten Peterson Bo Soderberg Department of Theoretical Physics, University of Lund, Solvegatan 14A, 5-22362 Lund, Sweden

Rotor neurons are introduced to encode states living on the surface of a sphere in D dimensions. Such rotors can be regarded as continuous generalizations of binary (Ising) neurons. The corresponding mean field equations are derived, and phase transition properties based on linearized dynamics are given. The power of this approach is illustrated with an optimization problem-placing N identical charges on a sphere such that the overall repulsive energy is minimized. The rotor approach appears superior to other methods for this problem both with respect to solution quality and computational effort needed. 1 Background

Standard McCulloch-Pitts neurons are characterized by sigmoidal updating equations

vi = g(ui) = tanhui

(1.1)

where the local field u; is given by

and the inverse temperature 1/T sets the gain. The neurons are binary in the high gain ( T + 0) limit. In feed-back networks (Hopfield and Tank 1985) with a quadratic energy function in terms of binary neurons sir (1.3)

the iterative solutions of the mean field equations (equations 1.1 and 1.2) represent approximate minima to E for appropriate values of T, where v; =< si >T. In the more general case one has (1.4) Neural Computation 4,

737-745 (1992)


L. Gislen, C. Peterson, and B. Soderberg

738

In a series of papers, we have investigated the generalization of this approach to multistate (Potts) neurons, which are superior in situations where one wants only one of the si (q)to be "on" and the others "off." In effect, equation 1.1 is replaced by'

Such a constrained encoding turns out to be crucial for many optimization applications (Peterson and Soderberg 1989; Peterson 1990a). Potts neurons also play a crucial role in deformable templates or so-called elastic net methods (Durbin and Willshaw 1987). In feedforward networks with exclusive classifications Potts neuron encoding of the output layer could be profitable (Lonnblad et al. 1991). In the present paper we investigate the generalization from binary neurons to the case of a continuum of states on a D-dimensional sphere, and apply the method to the problem of optimal configuration of charges on a sphere. 2 Rotor Neurons

Consider the general problem of minimizing an energy function E ( s l , . . . , SN) with respect to a set of N D-dimensional unit vectors si (hereafter denoted rotors) Is11 = 1

(2.1)

A locally minimal configuration must satisfy S;

=

-ViE/IViEI

(2.2)

Local optimization consists of iterating these equations until convergence. This is in general not a very good method for finding the global minimum: the configurations easily get stuck in local minima. A more careful method is simulated annealing (Kirkpatrick et al. 1983), where a thermal distribution 0: exp(-E/T) is simulated. The temperature T is very slowly lowered, until a stable state results. This method is very time-consuming, if a good result is desired. For this kind of problem, we suggest a mean field theory (MET) rotor approach analogous to what is used for combinatorial optimization problems (Hopfield and Tank 1985; Peterson and Soderberg 1989). 2.1 Mean Field Equations. Consider a thermal distribution of configurations, characterized by the partition functioiz Z,

z=

J

e-E[sl/rdsl . . . dsN

'Using [0,1] representation rather than the [-1,1] of equation 1.1.

(2.3)

Rotor Neurons

739

where the simplified notation dsi means that the integration is to be performed only over the direction of sir and normalized such that Jdsi = 1. For simplicity, consider first a single integral I = J H ( s )ds, over the directions of a D-dimensional unit vector s, with H an arbitrary function. It can be rewritten as /H(s)ds

= /H(v)b(s - v)dsdv K

I

H(v)e".(*-")dsdvdu

(2.4)

Performing the s integral, one is left with dvdu .IH(v)e-V'U+F(U)

IK where u

=

(2.5)

lul and F(u) is defined by

1

F ( u ) = log eU" ds

(2.6)

For a D-dimensional integral, this evaluates to (2.7) where 1, are modified Bessel functions. Repeating this trick N times with the multiple integral Z , we obtain

Z K 1exp [-E[v]/T-xv;.u,+xF(ui) 1

i

1

dvldul . . . dvNduN (2.8)

Next, we seek a saddlepoint of the effective potential, appearing in the argument of the exponent in the integrand in equation 2.8, by demanding its derivatives with respect to u; and v, to vanish. This results in the mean field equations (2.9) (2.10) where u, = ui/u,. They give v, as the average of s, in the local field V,E(v). In Table 1, g ( u ) is listed for different values of D. When D = 1, equation 1.1 is recovered. The corresponding curves are shown in Figure 1. In the large D limit, the shape of g is given by lim g(Du) =

dG73-1

(2.11) 2u We regard this system as a neural network, with vi as a generalized neuron, ui as its input (local field), and g as a generalized sigmoid function. The obvious dynamics consists in iterating equations 2.9 and 2.10. The performance of the neural network thus defined will depend on T; this is to be discussed next. D-03

L. Gislbn, C . Peterson, and 8.Soderberg

740

Table 1: Proper g(u) for Different Dimensions D.

1.0

,

I

08

0.6

04

0.2

4.

2.

0.

6.

a.

LO.

Figure 1: Graphs of g ( u ) for different dimensions D. 2.2 Critical Temperature Estimation. From equation 2.10 we infer that for small ui, vi NN u;/D. Making the simplifying assumption that E is rotationally invariant, we can approximate E for small v with a quadratic form 1 (2.12) E = Eo - - C w j j V i 'v, O(V')

+

I,]

But then v1 = ... = VN = 0 is a fixpoint of the updating equations 2.9 and 2.10. Linearizing these for small 21, we obtain (2.13)

Rotor Neurons

741

Figure 2: Schematic evolution of a D = 2 rotor initialized close to the center, for T < T,. If the temperature is higher than the critical temperature

where Xminjmax are the extreme eigenvalues of w,this trivial fixpoint is stable under synchronous updating, and the system is in a symmetric phase. For a lower T , it becomes unstable, and the mean fields vi will be repelled by the origin. For low enough temperature they will stabilize close to the sphere v: = 1 (cf. Fig. 2). The dynamics is thus very different from that of local optimization and simulated annealing, where moves take place on the surface. For serial updating, things are similar, although T, is different. In the special case of a constant self-coupling wii = p, we have instead 1

- - max(X,,,, ,D

T

-/A)

(2.15)

Thus, for a large set of energy functions, we can estimate T, in advance. A good strategy is then to initialize the system close to the symmetry point, with T close to T,, and slowly anneal while the system settles. When finally a stable state is reached, a possible solution to the minimization problem is extracted.

L. Gislen, C. Peterson, and 8.Soderberg

742

For both types of updating, it turns out to be advantageous to be able to adjust the self-coupling to achieve maximum stability. This is done by adding a term -([j/2) Civj!to the energy. For a more detailed discussion of [j and T,, and of serial versus synchronous updating, the reader is referred to Peterson and Soderberg (1989). 3 Placing Charges on a Sphere

We now turn to the specific problem of the equilibrium configuration of N equal charges on a sphere ( D = 3). With Coulomb interaction between the charges the energy function is given by

where we for dynamic reasons have added a [ j term as discussed above. With this energy function the local field ui (cf. equation 2.9) is given by

with the corresponding updating equation (see Table 1) V, = ii,(cothu; - l / u l )

(3.3)

The critical temperature T, for this system in serial updating mode is, for reasonable values (cf. equation 2.14) 7-c =

(P + 3 / 3

(3.4)

The :i-term also affects the low temperature behavior, controlling the tendency for the system to remain in a local minimum. A good final configuration, with the charges uniformly distributed over the sphere, should be stable. This means that the updated uis should satisfy ul(t)' v,(t - 1) > 0

(3.5)

A necessary condition for accomplishing this can be derived for large N. The result is

The role of is thus twofold: it controls the phase transition and the dynamic behavior at low temperatures. Equipped with prior estimates of T, and Po, the algorithm for a given problem size can take the following "black box" form: 1. Compute T, and

Po according to equations 3.4 and 3.6.

Rotor Neurons

743

MFT SA 0

1

GD

100

10

Figure 3: Time consumption as a function of problem size (N)for the MFT rotor (MFT), gradient descent (GD), and simulated annealing (SA) algorithms. Both axes are logarithmic. The three lines correspond to N,N2, and N3, respectively.

2. lnitialize with v,"= 0.01 ' rand[-l,l). 3. Update all v,s in turn according to equations 3.2 and 3.3. 4. Decrease the temperature, T

-+ 0.95

. T.

5. Repeat from step 3, until the saturation (Peterson and Soderberg 1989) C,q / N > 0.99.

6. Extract configuration by setting s, = i!j1, Using this prescription we have computed configurations of 3, 5, 10,20, 30, and 100 charges, respectively. In Figure 3 the time consumption as a function of problem size is shown. As in the case of other MFT applications (Peterson and Soderberg 1989; Gislen et al. 1989, 19921, the number of iterations needed for convergence empirically shows no dependence on problem size. Hence, the time consumption scales as J@ for the MET rotor algorithm. As for the quality of the solution, the MET rotor model gives the correct solutions where these are explicitly known (N= 2, 3, 4, and 6). For larger problems we have compared the solutions with those from a gradient descent (GD) and a simulated annealing (SA) algorithm. In GD one charge at a time is moved on the sphere a step 0: -gradient/N. In SA, a maximum step size 0; 1/N was used, with the annealing schedule T 4 0.95. T . These algorithms were clocked when the energy was within 1%of the MFT rotor result. The time consumption to achieve this is

L. Gislbn, C. Peterson, and B. Soderberg

744

Figure 4: Evolutions of rotors for an N = 32 problem as T decreases. Open and filled dots represent charges placed in front and the back of the sphere, respectively. The two graphs are generated to provide a stereo effect. shown in Figure 3. In Figure 4 the evolution of the rotors for an N = 32 problem is shown. Comparing the MET rotor approach with the conventional ones we find that for the MET rotor algorithm the number of sweeps needed to reach a satisfactory solution is practically independent of problem size, while for the other methods it is (with optional step size) roughly proportional to the problem size. As for quality, the final energies obtained by the MET rotor approach were always equal to or lower than the ones obtained with the other approaches. We have also run the MET rotor algorithm for a D = 3 system, where we substituted the appropriate sigmoid in equation 3.3 with a properly scaled tanh function

ui = i,tanh(ui/3)

(3.7)

We find that the algorithm performs as well (if not better) with respect to the number of sweeps needed to obtain a good solution. The reason for this investigation is that this sigmoid is more natural in a VLSI implementation.

4 Summary The formalism and dynamics for D-dimensional feedback rotor neurons have been developed. For D = 1 one recovers the conventional sigmoidal updating equations. As a first test bed for this approach in higher dimensions we applied the method to the problem of finding optimal charge configuration on a D = 3 sphere. The performance of the rotor method appears to be superior to that of gradient descent and simulated annealing for this problem.

Rotor Neurons

745

Other potential problem areas of more practical use are, e.g., curve detection in the early vision system (Zucker et al. 1990), or the reconstruction of tracks from signals (Peterson 1990b). The D > 1 updating equations can of course also be used in feedforward multilayered networks.

References Durbin, R., and Willshaw, G. 1987. An analogue approach to the travelling salesman problem using an elastic net method. Nature (London) 326, 689. Gislen, L., Peterson, C., and Soderberg, B. 1989. Teachers and classes with neural networks. lnt. J. Neural Syst. 1, 3. Gislen, L., Peterson, C., and Soderberg, B. 1991. Scheduling high schools with neural networks. Lund university preprint LU TP 91-9 (to appear in Neural Cornp.). Hopfield, J. J., and Tank, D. W. 1985. Neural computation of decisions in optimization problems. Bid. Cybern. 52, 141. Kirkpatrick, S., Gelatt, C. D., and Vecchi, M. P 1983. Optimization by simulated annealing. Science 220, 671. Lonnblad, L., Peterson, C., and Rognvaldsson, T. 1991. Using neural networks to identify jets. Nuclear Phys. B 349, 675. Peterson, C. 1990a. Parallel distributed approaches to combinatorial optimization. Neural Comp. 2, 261. Peterson, C. 1990b. Neural networks and high energy physics. In Proceedings of International Workshop on Software Engineering, Artificial Intelligence and Expert Systems for High Energy and Nuclear Physics, Lyon Villeurbanne, France, March , 2990, D. Perret-Gallix and W. Woijcik, eds. Editions du CRNS, Paris. Peterson, C., and Soderberg, 8. 1989. A new method for mapping optimization problems onto neural networks. lnt. J. Neural Syst. 1, 3. Zucker, S. W., Dobbins, A., and Iverson, L. 1990. Two stages of curve detection suggest two styles of visual computation. Neural Comp. 1, 68.

Received 17 September 1991; accepted 27 January 1992.

This article has been cited by: 2. Carsten Peterson, Ola Sommelius, Bo Söderberg. 1996. Variational approach for minimizing Lennard-Jones energies. Physical Review E 53:2, 1725-1731. [CrossRef] 3. I. M. Elfadel . 1995. Convex Potentials and their Conjugates in Analog Mean-Field OptimizationConvex Potentials and their Conjugates in Analog Mean-Field Optimization. Neural Computation 7:5, 1079-1104. [Abstract] [PDF] [PDF Plus]

Communicated by Michael Jordan

Refining PID Controllers Using Neural Networks Gary M. Scott Department of Chemical Engineering, University of Wisconsin, Madison, WI 53706 U S A

Jude W. Shavlik Department of Computer Sciences, University of Wisconsin, Madison, WI 53706 U S A

W. Harmon Ray Department of Chemical Engineering, University of Wisconsin, Madison, W f 53706 U S A

The K B A N N(Knowledge-Based Artificial Neural Networks) approach uses neural networks to refine knowledge that can be written in the form of simple propositional rules. We extend this idea further by presenting the MANNCON (Multivariable Artificial Neural Network Control) algorithm by which the mathematical equations governing a PID (Proportional-Integral-Derivative) controller determine the topology and initial weights of a network, which is further trained using backpropagation. We apply this method to the task of controlling the outflow and temperature of a water tank, producing statistically significant gains in accuracy over both a standard neural network approach and a nonlearning PID controller. Furthermore, using the PID knowledge to initialize the weights of the network produces statistically less variation in testset accuracy when compared to networks initialized with small random numbers. 1 Introduction

Research into the design of neural networks for process control has largely ignored existing knowledge about the task at hand. One form this knowledge (often called the “domain theory”) can take is embodied in traditional controller paradigms. The recently developed KBANN approach (Towel1 et al. 1990) addresses this issue for tasks for which a domain theory (written using simple, nonrecursive propositional rules) is available. The basis of this approach is to use the existing knowledge to determine an appropriate network topology and initial weights, such that the network begins its learning process at a “good” starting point. Neural Computation 4, 746-757 (1992)


Refining PID Controllers

747

This paper describes the MANNCONalgorithm, a method of using a traditional controller paradigm to determine the topology and initial weights of a network. The use of a PID controller in this way eliminates network-design problems such as the choice of network topology (i.e., the number of hidden units) and reduces the sensitivity of the network to the initial values of the weights. Furthermore, the initial configuration of the network is closer to its final state than it would normally be in a randomly configured network. Thus, the MANNCONnetworks perform better and more consistently than the standard, randomly initialized three-layer approach. The task we examine here is learning to control a nonlinear MultipleInput, Multiple-Output (MIMO) system. There are a number of reasons to investigate this task using neural networks. First, many processes involve nonlinear input-output relationships, which matches the nonlinear nature of neural networks. Second, there have been a number of successful applications of neural networks to this task (Bhat and McAvoy 1990; Jordan and Jacobs 1990; Miller et al. 1990). Finally, there are a number of existing controller paradigms that can be used to determine the topology and the initial weights of the network. The next sections introduce the MANNCONalgorithm and describe an experiment that involves controlling the temperature and outflow of a water tank. The results of our experiment show that our network, designed using an existing controller paradigm, performs significantly better (and with significantly less variation) than a standard, three-layer network on the same task. The concluding sections describe some related work in the area of neural networks in control and some directions for future work on this algorithm. In the course of this article, we use many symbols not only in defining the topology of the network, but also in describing the physical system and the PID controller. Table 1 defines these symbols and indicates the section of the paper in which each is defined. The table also describes the various subscripts to these symbols. 2 Controller Networks

The MANNCONalgorithm uses a Proportional-Zntegrul-Derivative (PID) controller (Stephanopoulos 19841, one of the simplest of the traditional feedback controller schemes, as the basis for the construction and initialization of a neural network controller. The basic idea of PID control is that the control action u (a vector) should be proportional to the error, the integral of the error over time, and the temporal derivative of the error. Several tuning parameters determine the contribution of these various components. Figure 1 depicts the resulting network topology based on the PID controller paradigm. The first layer of the network, that from ysp (desired process output or setpoint) and ~ ( ~ - 1(actual ) process output

G. M. Scott, J. W. Shavlik, and W. Harmon Ray

748

Table 1: Definitions of Symbols. Symbol

d = [ F d , Td] u = [Fc,FH] y = [F(h),7'1 e E

GI F T h

K, 71, TD AT

w 4yl nu,

(n) (n - 1) SP

d

C H

Definition Process disturbances Process inputs Process outputs Simple error Precompensated error Precompensator matrix Flow rate Temperature Height PID tuning parameters Time between control actions Network weights based on PID controller Error signal at plant output Error signal at plant input

Section Introduced Section 2 Section 2 Section 2 Section 2 Section 2 Section 2 Figure 1 Figure 1 Figure 2 Section 2 Section 2 Section 2 Section 3 Section 3

Subscripts Value at current step Value at previous step Setpoint Disturbance Cold water stream Hot water stream

of the past time step), calculates the simple error (e). A simple vector difference, e = ysp- y

accomplishes this. The second layer, that between e, ~ ( ~ and ~ ~e,1calcu, lates the actual error to be passed to the PID mechanism. In effect, this layer acts as a steady-state precompensator (Ray 19811, where E = Gle

and produces the current error and the error signals at the past two time steps. This compensator is a constant matrix, GI, with values such that interactions at steady state between the various control loops are eliminated. The final layer, that between E and u ( ~(controller ) output/plant input), calculates the controller action based on the velocity form of the discrete PID controller:


749

Figure 1: MANNCON network showing weights that are initializedusing ZieglerNichols tuning parameters. See Table 1 for definitions of symbols. where KC, TIC, and r x are the tuning parameters mentioned above, and AT is the discrete time interval between controller actions. This can be rewritten as Uc(n)= W ( n - 1 )

+ WCOEl(n) + W C 1 E l ( n - l ) + W C ~ E I ( ~ - ~ )

where WCO, w C 1 , and w c 2 are constants determined by the tuning parameters of the controller for that loop. A similar set of equations and constants ( W H O w, H 1 , w ~ 2 exist ) for the other controller loop. Figure 2 shows a schematic of the water tank (Ray 1981) that the network controls. This figure also shows the variables that are the controller variables (Fc and FH),the tank output variables [F(h)and TI, and the disturbance variables (Fd and Td). The controller cannot measure the disturbances, which represent noise in the system. MANNCONinitializes the weights of the network in Figure 1 with values that mimic the behavior of a PID controller tuned with ZieglerNichols (Z-N) parameters (Stephanopoulos 1984) at a particular operating condition (the midpoint of the ranges of the operating conditions). Using the KBANNapproach (see Appendix), it adds weights to the network such that all units in a layer are connected to all units in all subsequent layers, and initializes these weights to small random numbers several orders of magnitude smaller than the weights determined by the PID parameters. We scaled the inputs and the outputs of the network to be in the range [0,1].


750

Cold Stream ( a t Tc-)

Fc

Figure 2: Stirred mixing tank requiring outflow and temperature control. See Table 1 for definitions of symbols. Initializing the weights of the network in the manner given above assumes that the activation functions of the units in the network are linear, that is, o],llWar

=z

w)IoJ

but the strength of neural networks lie in their having nonlinear (typically sigmoidal) activation functions. For this reason, the MANNCONsystem initially sets the weights (and the biases of the units) so that the linear response dictated by the PID initialization is approximated by a sigmoid over the output range of the unit. For units that have outputs in the range [-1,1], the activation function becomes o1,sigmoid =

2 1 exp(-2.31

+

-1

Cw,~o~)

which approximates the linear response quite well in the range [-0.6,0.6]. Once MANNCON configures and initializes the weights of the network, it uses a set of training examples and backpropagation to improve the accuracy of the network. The weights initialized with PID information, as well as those initialized with small random numbers, change during backpropagation training. 3 Experimental Details

We compared the performance of three networks that differed in their topology and/or their method of initialization. Table 2 summarizes the

75 1

Refining PID Controllers Table 2: Topology and Initialization of Networks. ~

Network

Topology

Weight Initialization

Three-layer (14hidden units) PID topology PID topology

Random

~

1. Standard neural network 2. MANNCONnetwork I

3. MANNCON network I1

Random Z-N tuning

network topology and weight initialization method for each network. In this table, “PID topology” is the network structure shown in Figure 1. “Random” weight initialization sets all weights to small random numbers centered around zero. We trained the networks using backpropagation over a randomly determined schedule of setpoint y,, and disturbance d changes that did not repeat. The setpoints, which represent the desired output values that the controller is to maintain, are the temperature and outflow of the tank. The disturbances, which represent noise, are the inflow rate and temperature of a disturbance stream. The magnitudes of the setpoints and the disturbances each formed gaussian distributions centered at 0.5. The number of training examples between changes in the setpoints and disturbances were exponentially distributed. For example, the original setpoints could be an output flow of 0.7 liters/sec at a temperature of 40°C. After 15 sec (which represents 15 training examples since time is discretized into one-second slices), the setpoints could change to new values, such as a flow of 0.3 liters/sec at 35°C. The flow rate and the temperature of the disturbance stream also varied in this manner. We used the error at the output of the plant (y in Fig. 1) to determine the network error (at u) by propagating the error backward through the plant (Jordan and Rumelhart 1990). In this method, the error signal at the input to the process is given by

where 0;Yi represents the simple error at the output of the water tank and SUiis the error signal at the input of the tank. Since we used a model of the process and not a real tank, we can calculate the partial derivatives from the process model equations. We periodically interrupted training and tested the network over a different (but similarly determined) schedule. Results are averaged over 10 runs for each of the networks. We also compare these networks to a (nonlearning) PID controller, that had its tuning parameters determined using a standard design methodology (Stephanopoulos 1984). Using the MIDENTprogram of the CONSYD package (W. Harmon Ray Research Group 1989), we fit a linear, first-order

752


model to the outputs of the system when stimulated by random inputs. We then determined an appropriate steady-state precompensator (Ray 1981) and Z-N tuning parameters for a PID controller (Stephanopoulos 1984) using this model. Further details and additional experimentation are reported in Scott (1991).

4 Results

Figure 3 compares the performance of the three networks. As can be seen, the MANNCON networks show an increase in correctness over the standard neural network approach. Statistical analysis of the errors using a t test show that they differ significantly ( p = 0.005). Furthermore, while the difference in performance between MANNCON network I and MANNCON network I1 is not significant, the difference in the variance of the testing error over different runs is significant (p = 0.005). Finally, the MANNCON networks perform significantly better ( p = 0.0005) than the nonlearning PID controller tuned at the operating midpoint. The performance of the standard neural network represents the best of several trials with a varying number of hidden units ranging from 2 to 20. A second observation from Figure 3 is that the MANNCONnetworks learned much more quickly than the standard neural-network approach. The MANNCON networks required significantly fewer training instances

DO Training Instances

Figure 3: Mean square error of networks on the testset as a function of the number of training instances presented.


753

Table 3: Comparison of Network Performance. Method 1. Standard neural network 2. MANNCONnetwork I 3. MANNCONnetwork I1 4. PID control (Z-N tuning)

Mean square error

Training instances

0.0103 f 0.0004 0.0090 f 0.0006 0.0086 f 0.0001

25,200 f2,260 5,000 f 3,340 640 f 200

0.0131

to reach a performance level within 5% of its final error rate. Table 3 summarizes the final mean error for each of these three network paradigms, as well as the number of training instances required to achieve a performance within 5% of this value.

5 Related Research

A great deal of research in both recurrent networks and in using neural networks for control share similarities with the approach presented here. The idea of returning the output of the network (and of the system) from the previous training instance to become part of the input for the current training instance is quite common in works pertaining to control (Jordan and Jacobs 1990; Miller ef al. 1990). However, this idea also appears in problems pertaining to natural language (Gori ef al. 1989) and protein folding (Maclin and Shavlik 1991). In the area of process control, there have been many approaches that use neural networks. Introductions to neural network control are given by Bavarian (1988), Franklin (19901, and Werbos (1990). Feedforward controllers, in which the neural network learns to mimic the inverse of the plant are discussed by Psaltis et al. (1988), Hosogi (1990))Li and Slotine (19891, and Guez and Bar-Kana (1990). Chen (1990) proposes an adaptive neural network controller where the controller uses a system of two neural networks to model the plant. Herndndez and Arkun (1990) propose a Dynamic Matrix Control (DMC) scheme in which the linear model is replaced by a neural network model of the process. Narendra and Parthasarathy (1990) propose a method of indirect adaptive control using neural networks in which one network is used as a controller while the second is the identification model for the process. Bhat and McAvoy (1990) propose an Internal Model Controller (IMC) that utilizes a neural network model of the process and its inverse. Systems that use neural networks in a supervisory position to other controllers have been developed by Kumar and Guez (1990) and Swiniarski (1990). The book by Miller et al. (1990) gives an overview of many of the techniques mentioned here.


754

6 Future Work

In training the MANNCON initialized networks, we found the backpropagation algorithm to be sensitive to the value of the learning rate and momentum value. There was much less sensitivity in the case of the randomly initialized networks. Small changes in either of these values could cause the network to fail to learn completely. The use of a method involving adaptive training parameters (Battiti 19901, and especially methods in which each weight has its 0 ~ 7 nadaptive learning rate (Jacobs 1988; Minai and Williams 1990) should prove useful. Since not all weights in the network are equal (that is, some are initialized with information while some are not), the latter methods would seem to be particularly applicable. Another question is whether the introduction of extra hidden units into the network would improve the performance by giving the network ”room” to learn concepts that are completely outside of the given domain theory. The addition of extra hidden units as well as the removal of unused or unneeded units is still an area with much ongoing research. Some “ringing” occurred in some of the trained networks. A future enhancement of this approach would be to create a network architecture that prevented this ringing from occurring, perhaps by limiting the changes in the controller actions to some relatively small values. Another important goal of this approach is the application of it to other real-world processes. The water tank in this project, while illustrative of the approach, was quite simple. Much more difficult problems (such as those containing significant time delays) exist and should be explored. There are several other controller paradigms that could be used as a basis for network construction and initialization. There are several different digital controllers, such as Deadbeat or Dahlin’s, that could be used in place of the digital PID controller used in this project. DMC and IMC are also candidates for consideration for this approach. Finally, neural networks are generally considered to be “black boxes,” in that their inner workings are completely uninterpretable. Since the neural networks in this approach are initialized with information, it may be possible to in some way interpret the weights of the network and extract useful information from the trained network.

7 Conclusions We have shown that using the MANNCON algorithm to structure and initialize a neural-network controller significantly improves the performance of the trained network in the following ways: 0

Improved mean testset accuracy

Refining PID Controllers 0

Less variability between runs

0

Faster rate of learning

755

The MANNCON algorithm also determines a relevant network topology without resorting to trial-and-error methods. In addition, the algorithm, through initialization of the weights with prior knowledge, gives the backpropagation algorithm an appropriate direction in which to continue learning. Finally, since the units and some of the weights initially have physical interpretations, it seems that the MANNCONnetworks would be easier to interpret after training than standard, three-layer networks applied to the same task. Appendix: Overview of the KBANNAlgorithm The KBANNalgorithm translates symbolic knowledge into neural networks by defining the topology and connection weights of the network (Towell ef al. 1990). It uses knowledge in the form of PROLOG-like clauses, to define what is known about a topic. As an example of the KBANN method, consider the simple knowledge base in Figure 4a, which defines the membership in category A. Figure 4b represents the hierarchical structure of these rules where solid lines and dashed lines represent necessary and prohibitory dependencies, respectively. Figure 4c represents the neural network that results from a translation of this knowledge base. Each unit in the neural network corresponds to a consequent or an antecedent in the knowledge base. The solid and dashed lines represent heavily weighted links in the neural network. The dotted lines represent the lines added to the network to allow refinement of the knowledge base.

/Ft\ \ if B, C then A if G. not(F) then B if 1, J then C

Figure 4: Translation of a knowledge base into a neural network using the

KBANNalgorithm.

756


Acknowledgments G. M. S. was supported under a National Science Foundation Graduate Fellowship. J. W. S. was partially supported by Office of Naval Research Grant N00014-90-J-1941 and National Science Foundation Grant IRI-9002413. W. H. R. was partially supported by National Science Foundation Grant CPT-8715051.

References Battiti, R. 1990. Optimization methods for back-propagation: Automatic parameter tuning and faster convergence. In International joint Conference on Neural Networks, Vol. I, pp. 593-596, Washington, DC. Lawrence Erlbaum. Bavarian, B. 1988. Introduction to neural networks for intelligent control. IEEE Control Syst. Mag. 8, 3-7. Bhat, N., and McAvoy, T. J. 1990. Use of neural nets for dynamic modeling and control of chemical process systems. Comput. Chem. Eng. 14,573-583. Chen, E-C. 1990. Back-propagation neural networks for nonlinear self-tuning adaptive control. IEEE Control Syst. Mag. 10. Franklin, J. A. 1990. Historical perspective and state of the art in connectionist learning control. In 28th Conference on Decision and Control, Vol. 2, pp. 17301735, Tampa, FL. IEEE Control Systems Society. Gori, M., Bengio, Y., and DeMori, R. 1989. Bps: A learning algorithm for capturing the dynamic nature of speech. In International Joint Conference on Neural Networks, Vol. 11, pp. 417-423. San Diego, CA. IEEE. Guez, A., and Bar-Kana, I. 1990. Two degree of freedom robot adaptive controller. In American Control Conference, Vol. 3, pp. 3001-3006. San Diego, CA. IEEE. Hernandez, E., and Arkun, Y. 1990. Neural network modeling and an extended DMC algorithm to control nonlinear systems. In American Control Conference, Vol. 3, pp. 2454-2459. San Diego, CA. IEEE. Hosogi, S. 1990. Manipulator control using layered neural network model with self-organizing mechanism. In International joint Conference on Neural Networks, Vol. 2, pp. 217-220. Washington, DC. Lawrence Erlbaum. Jacobs, R. A. 1988. Increased rates of convergence through learning rate adaptation. Neural Networks 1, 295-307. Jordan, M. I., and Jacobs, R. A. 1990. Learning to control an unstable system with forward modeling. In Advances in Neural Information Processing Systems, Vol. 2, pp. 325-331. San Mateo, CA. Morgan Kaufmann. Jordan, M. I., and Rumelhart, D. E. 1990. Forward models: Supervised learning with a distal teacher. Occasional Paper #40, Massachusetts Institute of Technology (to appear in Cog. Sci.). Kumar, S. S., and Guez, A. 1990. Adaptive pole placement for neurocontrol. In International Ioint Conference on Neural Networks, Vol. 2, pp. 397400. Washington, DC. Lawrence Erlbaum.


757

Li, W., and Slotine, J.-J. E. 1989. Neural network control of unknown nonlinear systems. In American Control Conference, Vol. 2, pp. 1136-1141. San Diego, CA. IEEE. Maclin, R., and Shavlik, J. W. 1991. Refining domain theories expressed as finite-state automata. In Eighth International Workshop on Machine Learning. Morgen Kaufmann, San Mateo, CA. Miller, W. T., Sutton, R. S., and Werbos, P. J., eds. Neural Networks for Control. The MIT Press, Cambridge, MA. Minai, A. A., and Williams, R. D. 1990. Acceleration of back-propagation through learning rate and momentum adaptation. In International Joint Conference on Neural Networks, Vol. I, pp. 676-679. Washington, DC. Lawrence Erlbaum. Narendra, K. S., and Parthasarathy, D. 1990. Identification and control of dynamical systems using neural networks. IEEE Transact. Neural Networks 1(1), 4-27. Psaltis, D., Sideris, A,, and Yamamura, A. A. 1988. A multilayered neural network controller. I E E E Control Syst. Mag. 8, 17-21. Ray, W. H. 1981. Advanced Process Control. McGraw-Hill, New York. Scott, G. M. 1991. Refining PID controllers using neural networks. Master’s project, University of Wisconsin, Department of Computer Sciences, May. Stephanopoulos, G. 1984. Chemical Process Control: A n Introduction to Theory and Practice. Prentice-Hall, Englewood Cliffs, NJ. Swiniarski, R. W. 1990. Novel neural network based self-tuning PID controller which uses pattern recognition technique. In American Control Conference, Vol. 3, pp. 3023-3024. San Diego, CA. IEEE. Towell, G. G., Shavlik, J. W., and Noordewier, M. 0. 1990. Refinement of approximate domain theories by knowledge-base neural networks. In Eighth National Conference on Artificial Intelligence, pp. 861-866. AAAI Press, Menlo Park, CA. W. Harmon Ray Research Group. 1989. Department of Chemical Engineering, University of Wisconsin, Madison. CONSYD: Computer-Aided Control System Design. Werbos, P. J. 1990. Neural networks for control and system identification. In 28th Conference on Decision and Control, Vol. 1, pp. 260-265. Tampa, FL. IEEE Control Systems Society. ~

Received 6 September 1991; accepted 28 February 1992.

This article has been cited by: 2. Gary M. Scott , W. Harmon Ray . 1994. Neural Network Process Models Based on Linear Model StructuresNeural Network Process Models Based on Linear Model Structures. Neural Computation 6:4, 718-738. [Abstract] [PDF] [PDF Plus] 3. Jude W. Shavlik. 1994. Combining symbolic and neural learning. Machine Learning 14:3, 321-331. [CrossRef]

Communicated by Francoise Fogelman-Soulie

Ultrasound Tomography Imaging of Defects Using Neural Networks Denis M. Anthony Evor L. Hines David A. Hutchins J. T. Mottram Department of Engineering, Warwick University, Coventry, England

Simulations of ultrasound tomography demonstrated that artificial neural networks can solve the inverse problem in ultrasound tomography. A highly simplified model of ultrasound propagation was constructed, taking no account of refraction or diffraction, and using only longitudinal wave time of flight (TOF). TOF data were used as the network inputs, and the target outputs were the expected pixel maps, showing defects (gray scale coded) according to the velocity of the wave in the defect. The effects of varying resolution and defect velocity were explored. It was found that defects could be imaged using time of flight of ultrasonic rays. 1 Introduction

One of the main techniques for use in the nondestructive testing (NDT) of materials is ultrasound. Defects in a material may be detected due to a change in the time of flight (TOF) through the material between two fixed points, amplitude changes in the ultrasonic waves, or by examining the transmitted waveforms. Many measurements need to be taken to scan a material, and this can be time-consuming and costly. Ultrasound tomography, where several measurements are taken simultaneously, may be used to reconstruct an image of the test material. Algorithms exist to reconstruct an image, but there are difficulties to overcome, in that the ultrasonic waves, unlike X-rays, may not be assumed to travel in straight lines, and are affected by diffraction and refraction. Artificial neural networks (ANNs) may be used to solve problems without specifying an algorithm. In one common technique, backpropagation, inputs and target outputs are presented to the network, and the network forms connection strengths between inputs, through "hidden units" through to outputs, such that the outputs attempt to match the targets, in an iterative period of "training." The inethod is essentially an error gradient descent technique. Neural Computation 4, 758-771 (1992)


Ultrasound Tomography Imaging of Defects

759

ANNs have been used (Watanabe and Yoneyama 1990), to recognize digits scanned by ultrasonic "eyes" and digits the network was not trained on were reconstructed with some success. ANNs have determined the radius of a cylindrical object, given the real and imaginary pressures from an array of 16 transducers surrounding the object (Conrath et al. 1989). ANNs may be used to process data, for example, in medical ultrasound, delay noise has been reduced in simulations where an ANN preprocessed echo delays prior to beamforming (Nikoonahad and Liu 1990). Other NDT applications using ANNs include the inversion of eddy currents to give flaw size (Mann et al. 1989). ANNs have also been used in other tomographic modalities, e.g., laser scattering tomography (Gonda et al. 1989). The aim of this paper is to report work that has been performed using simulation experiments to solve the inverse problem in ultrasound imaging. The final target is to produce tomographic maps of anisotropic composite materials containing defects that may be the result of service load conditions (e.g., with carbon fiber reinforced plastics). These are particularly difficult to image using ultrasound as the orientation of the fibers in the resin matrix affects the wave propagation, and the fibers act as channels for the waves, thus adding to the above mentioned diffraction and refraction effects. ANNs might be successful in solving this mapping probIem. Prior to attempting the more difficult anisotropic case, it was decided to initially determine whether isotropic media could be imaged. Problems in this type of material would be expected to be less severe than the anisotropic case, and thus the more difficult problem would be unlikely to be solved unless the simpler case was shown to be soluble. Simulations were undertaken as a prelude to experimental data acquisition, to allow tighter control and for speed of development. 2 Tomography

Computer aided tomography (CAT) is a method by which a series of views of an object may be used to form an image. A typical CAT application is X-ray imaging. In conventional X-ray, the rays cover a plane, while in CAT a series of scans are produced each with a thickness or height of 1 cm or less. The series of views are repeated at different angles. While X-ray is the best known CAT technique, other modalities have been imaged in this way including nuclear scintigram images, microwaves, electron microscopy, and ultrasound. Reviews of tomography may be found in, for example, Scudder (1978) and Mueller et al. (1979). In principle if the projection data are represented by a matrix X, and the absorption (assuming X-ray tomography) by Y, then there is a matrix A that gives the relation between the projection data and absorption: X=AY

(2.1)

760

D. M. Anthony et al.

and to reconstruct an image one needs to find the inverse A-’ or, if it does not exist, a pseudoinverse A*. However, the sizes of the matrices are typically far too large to allow matrix inversion to be used as a reconstruction technique, and some other method needs to be found. Common methods used include iterative algebraic reconstruction technique (ART) and Fourier transform techniques; ART is discussed further in Section 5. Certain methods specific to applications also exist, ( e g , diffraction methods in ultrasound). ANNs have been used (Obellianne et al. 1985) in simulated tomography reconstruction, where simulated projection data were processed in a conventional reconstruction algorithm. An ANN employing backpropagation was then used to improve the image by reducing the effects of noise. 3 The Simulation Model

A thin sheet specimen of square cross section was created, which was assumed to have constant in-plane wave velocity in all directions ke., isotropic). A defect lying in the plane may in principle be of any size, however, a finite imaging resolution will limit the representation or display of the defect. The resolution will be (say) N by N. It was assumed that if the defect lies in any of the N 2 cells, the cell will be designated as having the defect velocity, and if a cell contains no part of a defect it will be designated as background material wave velocity. Thus where a defect lies across the boundary of two cells, both cells will be assumed to have the defect wave velocity. Figure 1 illustrates the scanning method that was adopted for the simulations. An ultrasonic transmitter was assumed to move around the outside of the pixel area, and the ultrasonic signal to be received at all other receiver locations not on the same edge as the transmitter. (The transducers around the specimen may act as transmitters or receivers.) In subsequent discussions the actual defect dimensions will be referred to as the ”ideal” image. The image that results from a given output image resolution will be called the ”target” image. The ANN will be given inputs that will be the TOF of ultrasound rays, and will be trained to give the associated target pixel values. 4 Statistics on Ray Paths

-

If the ANN is to be able to determine the location and size of a defect, there must be a difference between the inputs (TOF data) with a defect and those with no defect. If no rays pass through the defect, the TOF is not affected in the above model, and the ANN will not detect a defect, let alone be able to locate it. For each defect in a randomly selected sample of 500 single defects, the paths were examined to determine whether any


761

Figure 1: Transducers setup for tomography. Conventional setup using parallel beams at angle Oi (left). Fan ray tomogram as used for ANN simulation (right). The material is assumed to be split into N x N cells, within each cell wave velocity is assumed constant; here 100 cells are shown. The ultrasonic beam is sent from transmitter (marked as closed circles around the perimeter of the square material), and are received by transducers (open circles) along the ray path indicated by arrows. paths went through the defect, and how many defects had not a single ray path going through them. There is a relationship between the number of the transducers, i.e., their number, and the size of objects that may be detected (see Fig. 1). If transducers are arranged 4 x 4 along the edges, and each is connected to all 4 on each of the other 3 edges, then 192 paths are created (16 x 3 x 4, i.e., 16 transducer positions, each with 12 active receivers). In 500 defects with 192 potential paths for each, 26.8% of paths went through a defect. Of the 500 input patterns, only 22 had no path at all. Of the defects with no path, 12 had zero dimensions (the height or width randomly allocated could be zero), and of the 10 defects remaining, the largest had an area of 0.32 of a pixel. Thus only very small defects are likely to be completely missed by all ray paths, and as Figure 2 shows, most defects are crossed by many paths.

5 Time of Flight Calculations Initially a very simple model is proposed. The ultrasound wave propagation is assumed to travel in straight lines, with constant velocity in the medium, and with a lower constant velocity in the defects. No account is taken of diffraction, refraction, reflection, or the effects of anisotropy, only longitudinal waves are assumed, and no account is taken of shear waves, etc. Consider a specimen surrounded by a medium of constant

762


Figure 2: (a) Histogram of defect path:total path for defects with a ray path through them. This figure shows that most defect ray paths were small in comparison to the total path. (b) Number of paths passing through defects, showing the majority of defects have many ray paths passing through them. propagation. The specimen can be split into cells within which a constant velocity may be assumed. The simplistic assumption is that the TOF may be computed by taking a straight line between the transmitter and any of the 12 receivers, and by working out the distance traveled in the cells (Mueller et al. 1979). For a given angle of incidence Oi. the TOF for ray j at angle i, which is the measured value, is given by

(5.1) where D is the total distance between transmitter and receiver, I$) is the length of the kth cell traversed by the jth ray at orientation i, C, is the


763

velocity of wave propagation in the medium surrounding the specimen, Ck the (constant) wave velocity in the kth cell, and N is the number of cells. If one denotes nk = (1/Ck) - (l/C,,,), one has a parameter to be reconstructed ( n k ) subject to

where ~ i = j Tij - (D/C,). This forms a set of M equations, if M rays are considered, in N unknowns, which may be solved using algebraic reconstruction techniques (ART), convolution methods, Fourier analysis, or diffraction methods (Mueller et al. 1979). In this study a solution to the equations will be attempted using ANNs. The Tij will be given as inputs and the required image as targets to the ANN. In principle, a linear network can solve this equation, as the set of equations is linear. Nonlinear nets will be employed since in practice nonlinearities may be encountered, for the TOF calculations have built in assumptions that make the linear equations an approximation to reality only. Thus the nonlinear net needs to be tested in the linear simulation because it will be used in studies based on experimental data. The distances are calculated using simple geometry, where the transducer locations and the rectangular defect coordinates are known. In a reconstruction each transducer would in turn be the transmitter, and the remaining transducers on the other three sides would be receivers. The net will be required to solve the inverse problem of equation 5.2, i.e.,

6 Experimental Procedures Given an array of transducers, the effects of varying the resolution of the final image and the ratio of defect wave velocity to background material wave velocity were explored. The imaging of rectangular single defects of arbitrary dimensions was attempted. A set of 16 transducers were assumed to be arranged spaced equally around a material of square cross section. TOF paths were computed from all transmitters to every other receiver not on the same edge. This gave 192 values. There was redundancy in the data as each path was given twice, due to the symmetry of the problem, but as in real simulations, especially in anisotropic media, these TOF would not be identical due to noise and other effects; it was therefore decided to include all values. The net with all inputs used is potentially more liable to fall into uninteresting one to one mappings (i.e., act as a look-up table) than one using half the inputs, and thus one needs to test the net for this potential problem, if one is later to use all inputs in an experimental study. These


764

were fed into an input layer of 192 units, which were fed into a hidden layer of 48 units, and the hidden layer was connected to an output layer that gave pixel values for a reconstructed image. The figure of 48 hidden units is somewhat arbitrary; unfortunately there is little theoretical justification for using any particular architecture. Too large a number of units in the network will give slow convergence, and may not generalize well; too few units will not allow a solution to the problem. The architecture used in this study gave convergence, and the nets showed similar performance on test data as training data, thus the nets did generalize. The nets also converged in a reasonable time. In these simulations a fan-beam geometry is assumed (see Fig. 1). The equivalent setup in parallel beam tomography would be to have K , the number of projections, as given by

where Nd is the total number of receivers, as each transmitter sends rays to the receivers on all other three sides. It is known in such a system that to avoid aliasing (Platzer 1981), the number of projections must be given by

Thus the minimum number of receivers that will avoid aliasing is given by

N >8T

"-3

(6.4)

In Obellianne et al. (1989) it was found that recognition of simulated "lungs" dropped to very low levels when the ratio of receivers to projections fell below the minimum specified by equation 6.2, though the error level of the output image was able to be reduced substantially by the network. The figure of 12 receivers used in this study thus should not suffer from the problem of aliasing. In all simulations error backpropagation was employed (Rumelhart et al. 1986). The network parameters of learning rate (TI) and momentum (0)need to be given values. The lower the value of 71 the more likely convergence is achieved, but very low values may slow convergence unnecessarily. An (Y close to unity speeds convergence, and Rumelhart ef a/. (1986) suggest a high (1 of about 0.9 with a low 7 is superior to a higher 7 with no momentum term. An 77 of 0.001 and (Y of 0.9 were used, and these values were found to allow convergence in all cases.


765

6.1 Defect Types. Arbitrary dimensioned rectangular defects were randomly created for the training set. The resolution of the target image was set, and if a pixel contained any of a defect, that pixel was allocated to the defect, otherwise it was allocated to the background material. Pixels were given values according to the wave velocity of defect or background. Thus images were overrepresented in size as defects were rounded up in size for the target pixel map, and a gray scale was produced whereby white indicates the highest velocity and black zero velocity. Initially 4 by 4 resolution images were created using a defect velocity 10% that of the background velocity. This large reduction in velocity value due to the presence of a defect may not be realistic, but a large difference in velocities was found to allow easier network learning. As Figure 3 shows, the ANN restored image locates the defect with some success. 6.2 Increasing Resolution. Increasing the resolution of the output pixel map to 10 by 10 gives better restored images than the 4 by 4 case (see Fig. 3). (Note: The number of the receivers remains at 4 by 4.) Defects whose length-to-width ratio is high are not detected in either resolution. This is likely to be due to the fact that the defect affects the TOF between transmitter and receiver very little, and the ANN has insufficient variability of its inputs to reconstruct the defect. The ability of a network to converge is often shown by plotting total sum square error (tsse) against number of epochs of training, which is defined as tsse =

C C(tP- o')* p=l NP

"i=l

I

(6.5)

where Np is the number of data patterns and N" the number of outputs, 't and up are the target and output values, respectively, for the pth pattern and ith target or output value. Figure 4 shows tsse per pixel of networks with various resolutions, which shows the trend toward lower error as the resolution (number of pixels in the reconstructed image) increases. 6.3 Repeatability. To test whether the ANNs were giving consistent and repeatable results, several runs were made using different initial random weights. As the ANNs were found to converge within two to five epochs (see Fig. 4) the nets were trained for 10 epochs each. The nets showed very similar behavior on repetition. The tsse in every case converged within 2-3 epochs, and was similar for both training and test (i.e., using data that the net had not been trained on) data. The absolute value of weights from input to hidden and hidden to output nodes slowly increased with time.


766

1

I

Figure 3: Target (left), restored (center), and ideal (right) images of randomly created (single) rectangles. The ideal images are the actual dimensions of the simulated defect. The target image is created by using a given pixel resolution (top 2 images 4 x 4, bottom 3 images 10 x 10) so the defect is rounded up in size; this is the image the ANN is given during training as a target. The ANN after training gives the center image when presented with the TOF data.

6.4 Amendments to Backpropagation Algorithm. To improve images two changes were made to the training algorithm. The majority bias problem occurs when most of the outputs of the net are set to the same value most of the time, and the net can reduce the tsse quickly by fixing the outputs at this value (i.e., the network gets “ s t u c k at these


Key:

767

-2by 2,-.-.3 by 3, - 4 by 4, ...10 by lOpuds ~

Use per Pixel against epoch for 4 by 4 Rcaolution, Single Dcfed

Key: - 10% D c f s l Velocity. -.50%Dcfm Velocity

Figure 4: tsse per pixel for various resolutions against number of epochs of training-single defect (top graph). As the resolution increases the tsse drops. Lower graph-tsse per pixel for 4 by 4 resolution against epochs of training. Single defect for 2 different defect velocities. The tsse for the defect velocity 50% that of background material is lower than the tsse of a defect with wave velocity 10% that of background material (lower graph). However, subjectively the image with the lower tsse was not improved. Bottom figures show, in order, target and output from defect velocity 10% that of background material, target and output for 50% defect velocity.

768


outputs). As Chiu et al. (1990) have shown, the use of thresholding may improve images where the majority bias problem is apparent. In the case of small defects, most of the outputs are set to the background material, and thus this may partly explain the poor performance of the network to detect small defects. Thresholding was applied to the network, whereby any output that was within 10% of its target contributed no error to the backpropagation algorithm. This would be expected to allow the network to concentrate on the smaller number of defect pixel outputs. However, the performance was not improved in 10 repeated runs using this technique where the criterion of good performance was receiver operating characteristic curves (ROC, see Section 6.5). To speed convergence of nets, dynamic parameter tuning may be employed, as in Vogl et al. (1988). In this technique the learning rate and momentum are altered according to whether the tsse is decreasing or increasing. Where the tsse decreases from one epoch to the next, rl is increased by a scalar (we used 1.1>,and where the tsse increases o is set to zero, and 71 reduced by a scalar value (we used 0.7). In 10 repeated runs little advantage was found using this meth-od in terms of speed of convergence or the performance of the net. In every case 71 increased for the first few epochs (2-5 typically) and then reduced. 6.5 Ratio of Defect to Background Wave Velocity. Reducing the difference between background and defect velocities decreases the tsse for single defects (Fig. 4). However, looking at the reconstructed images, the ANN does not perform any better, and subjectively appears to be worse (see Fig. 4). It might be the case that the ANN reduces the tsse by setting the output pixels to a value midway between defect and background. The different slopes of the two cases indicate that they may be learning different mappings. To test the relative performance of the nets receiver operating characteristic (ROC) curves were constructed. This technique is well known in medical informatics, and for a description see ToddPokropek (1983) and Swets (1979). Essentially a graph is plotted of the true positive rate against false positive rate for a series of confidence levels. This technique has been applied to ANN analysis (Meistrell 1990; Anthony 1991). An area significantly above 0.5 indicates that the network is not giving random classifications. The area under the ROC curves for 10 repeated runs was calculated for the defect velocity of 50 and of 10% that of the background velocity. Using the t test of means, there was no significant difference between the two sets ( p > 0.1). To test further the effect of defect velocity, a training set was constructed which had defect velocity 90% that of background. There was a significant difference between this set and, separately, both the other two sets (10 and 50%, p < 0.0001) (see Table 1). For 10 versus 50% the variances were not significantly different. For both 10 and 50% versus 90% defect velocity, there was a difference in the variances, and this was taken account of in the t test using Satterthwaite’s approximation


769

Table 1: Area under ROC curve for various defect velocities Defect/background velocity Mean area Standard deviation 0.1 0.5 0.9

0.639 0.634 0.562

0.009 0.009 0.018

for degrees of freedom (SAS 1988) (The SAS package was used to perform the tests.). 7 Conclusion

A simplistic ultrasound simulation has been constructed. An ANN solved the inverse problem, i.e., given the TOF data it was able to construct a pixel image. The conclusions that may be drawn from this study are as follows: 0

0

0

0

0

The network behavior was repeatable. As the resolution of single arbitrary dimensioned rectangular defects increases, the tsse per pixel decreases. This may be due to a reduction in rounding u p of defects for the targets. The defect velocity is critical. If the defect velocity is close to the background velocity, the defect recognition rate reduces. This may have importance when analyzing experimental data. If the data show small differences for TOF comparing defect and nondefect paths, one may need to preprocess the data to increase the dynamic range. The use of thresholding errors from the output layer did not solve the majority bias problem. Dynamic learning did not improve the convergence in these simulations.

Using ANN techniques may provide alternative methods of reconstructing tomographic images. It may be the case that conventional tomographic reconstruction will continue to be used in isotropic materials, and that ANNs may be used in anisotropic materials. Another possibility would be the marriage of the two techniques, such that the conventional method produces an output which is improved by a neural net. Future work planned by the authors will explore aIternative network designs on the simulation data [e.g., neural trees (Sankar and Mammone 1991)l. The ability of ANNs to detect small defects using simulation data has been tested, and will shortly be published (Anthony et al. 1991). An

770


experimental tomographic setup has been constructed, which will allow data to be collected on specimens with known defects. The data will be analyzed to determine the most appropriate transformation of the data for input to the neural net. Initial study indicates the ratio of Fourier transform amplitudes, or the area under the Fourier transform, may allow a single number from each ultrasonic wave to be used. These parameters seem to be affected when a defect lies in the path of the ultrasonic wave. If experimental data are reconstructed with some success, the trained neural network could be implemented in hardware to form a n imaging system. Acknowledgments This project is funded by the Science and Engineering Research Council

(S.E.R.C.), U.K. References Anthony, D. M. 1991. The use of artificial neural networks in classifying lung scintigrams. Ph.D. thesis, University of Warwick. Anthony, D. M., Hines, E. L., Hutchins, D., and Mottram, J. T. 1992. Simulated tomography ultrasound imaging of defects. In Series in Neural Networks, Springer, Berlin. Chiu, W. C., Anthony, D. M., Hines, E. L., Forno, C., Hunt, R., and Oldfield, S. 1990. Selection of the optimal Mlp net for photogrammetric target processing. Zasted Conf. Artificial Intelligence App. Neural Networks, pp. 180-183. Conrath, B. C., Daft, C. M. W., and OBrien, W. D. 1989. Applications of neural networks to ultrasound tomography. Ultrasonics Symposium, 1007-1010. Gonda, T., Kakiuchi, H., and Moriya, K. 1989. In situ observation of internal structures in growing ice crystals by laser scattering tomography. J. Crystal Growth 102,179-184. Mann, J. M., Schmerr, L. W., and Moulder, J. C. 1989. Neural network inversion of uniform-field eddy current data. Materials Evaluation, Jan. 34-39. Meistrell, M. L. 1990. Evaluation of neural network performance by receiver operating characteristic (ROC)analysis: examples from the biotechnology domain. In Computer Methods and Programs in Biomedicine, pp. 73-80. Elsevier, Amsterdam. Mueller, R. K., Kaveh, M., and Wade, G. 1979. Reconstructive tomography and applications to ultrasonics. IEEE Proc. 67(4), 567-587. Nikoonahad, M., and Liu, D. C. 1990. Medical ultrasound imaging using neural networks. Electronic Lett. 26, 545-546. Obellianne, C., Fogelman-Soulie, F., and Galibourg, G. 1989. Connectionist models for image processing. In From Pixels to Features, COST23 Workshop, J. C. Simon, ed., pp. 185-196. Elsevier, North Holland. Platzer, H. 1981. Optical image processing. Proc. 2nd Scandinavian Conf. Zmage Analysis, pp. 128-139.


771

Rumelhart, D. E., Hinton, G. E., and Williams,N. 1986. Learning representations by back-propagating errors. Nature (London) 323(9), 533-536. Sankar, A., and Mammone, R. J. 1991. Optimal pruning of neural tree networks for improved generalization. Proc. 4th International Joint Conference on Neural Networks, Seattle. IEEE. SAS Users' Guide Statistics. Volume 5, 798. Scudder, H. J. 1978. Introduction to computer aided tomography. IEEE Proc. 66(6),62-37. Swets, J. A. 1979. ROC analysis applied to the evaluation of medical imaging techniques. Invest. Rudiol. 14(2), 109-121. Todd-Pokropek, A. E. 1983. The comparison of a black and white and a color display: An example of the use of receiver operating characteristic curves. IEEE Transact. 19-23, MI-2. Vogl, T. P., Mangis, J. K., Rigler, A. K., Zink, W. T., and Alkon, D. L. 1988. Accelerating the convergence of the back-propagation method. Biol. Cybern. 59,257-263. Watanabe, S., and Yoneyama, M. 1990. Ultrasonic robot eyes using neural networks. IEEE Transact. Ultrason.Ferroelect. Freq. Control 37(3), 141-147. Received 18 June 1991; accepted 1 December 1991.

This article has been cited by:

Communicated by Richard Lippmann

Improving the Accuracy of an Artificial Neural Network Using Multiple Differently Trained Networks William G. Baxt Department of Emergency Medicine and Medicine, University of California, San Diego Medical Center, Sari Diego, C A 92103-8676 U S A When either detection rate (sensitivity) or false alarm rate (specificity) is optimized in an artificial neural network trained to identify myocardial infarction, the increase in the accuracy of one is always done at the expense of the accuracy of the other. To overcome this loss, two networks that were separately trained on populations of patients with different likelihoods of myocardial infarction were used in concert. One network was trained on clinical pattern sets derived from patients who had a low likelihood of myocardial infarction, while the other was trained on pattern sets derived from patients with a high likelihood of myocardial infarction. Unknown patterns were analyzed by both networks. If the output generated by the network trained on the low risk patients was below an empirically set threshold, this output was chosen as the diagnostic output. If the output was above that threshold, the output of the network trained on the high risk patients was used as the diagnostic output. The dual network correctly identified 39 of the 40 patients who had sustained a myocardial infarction and 301 of 306 patients who did not have a myocardial infarction for a detection rate (sensitivity) and false alarm rate (1-specificity) of 97.50 and 1.63%, respectively. A parallel control experiment using a single network but identical training information correctly identified 39 of 40 patients who had sustained a myocardial infarction and 287 of 306 patients who had not sustained a myocardial infarction ( p = 0.003). 1 Introduction

Artificial neural networks have been shown to be a powerful pattern recognition paradigm (Widrow and Hoff 1960; Rumelhart et al. 1986; McClelland and Rumelhart 1988; Weigend et al. 1990). It has been recently demonstrated that artificial neural networks can be applied to the analysis of clinical data (Hudson etal. 1988; Smith et al. 1988; Saito and Nakano 1988; Kaufman et al. 1990; Hiraiwa et al. 1990; Cios et al. 1990; Marconi et al. 1989; Eberhart et al. 1991; Mulsant and Servan-Schreiber 1988; Bounds et al. 1990; Yoon et al. 1989). Both retrospective and prospective studies Neural Computation

4, 772-780 (1992)


Improving the Accuracy of a Neural Network

773

of the application of this technology to the diagnosis of acute myocardial infarction have revealed that the network can perform substantially more accurately than physicians (Baxt 1991a,b) or other electronic data processing paradigms (Goldman et LIZ. 1988; Pozen et al. 1984). The performance of an artificial neural network is highly dependent on the composition of the data on which it is trained. The ability of a network to identify a pattern is directly related to the representation of that pattern in the data used to train the network. In settings in which the networks are trained to make categorical decisions, this appears to translate into the known reciprocal relationship between detection and false alarm rates. It was observed that if a network trained to recognize the presence of myocardial infarction is trained on a pattern set derived from a patient population in which the likelihood of myocardial infarction was high and in which most patients appeared to have sustained a myocardial infarction, the network performed with a high detection rate but less than optimized false alarm rate. Similarly, when the network was trained on a population in which the likelihood of myocardial infarction was low and in which most patients did not appear to have sustained a myocardial infarction, the network performed with a low false alarm rate but less than optimized detection rate. A number of strategies were developed to try to simultaneously improve both detection rate and false alarm rate by using a single predesigned pattern set. Training sets with varied numbers of patients who had and had not sustained a myocardial infarction as well as varied numbers of patients who on presentation appeared to have sustained and not sustained a myocardial infarction were utilized to train the network. Training parameters were also relaxed in order to aIlow the network greater generalization. These strategies all failed to improve accuracy. Because it had been observed that networks could be trained to optimize one end of the categorical spectrum, it was reasoned that it might be possible to design an algorithm that could utilize two networks in concert, each trained to optimize either detection rate or false alarm rate. To this end, two networks were trained separately, one on a low-risk population of patients and the other on a high-risk population of patients, and then used simultaneously to test unknown patterns. The following reports on the results accrued from this strategy. 2 Methods

The architecture of the artificial neural network used in this study is illustrated in Figure l. It consisted of 20 input units, two layers of 10 internal or hidden units, and 1 output unit (Baxt 1991a,b). The artificial neural network simulator for this project was written specifically for this study in C and run on a UNIX work station at 1.5 mflops. The training algorithm used was standard backpropagation (Widrow and Hoff 1960;

William G. Baxt

774

2 O X 1 O X 1 O X 1 BACK PROPAGATION NETWORK

INPUT U N I T S

HIDDEN U N I T S

HIDDEN UNITS

ObTPUT UN I T

Figure 1: Neural network architecture. Rumelhart et a!. 1986; McClelland and Rumelhart 1988) with the Stornetta and Huberman (1987) modification. The network utilized in this study used inputs representing the encoded presenting complaints, past history, physical and electrocardiographic findings of adult patients presenting to the emergency department with anterior chest pain. These clinical variables are listed in Table 1. The network output was trained to represent the presence or absence of acute myocardial infarction (0 = absence; 1 = presence of myocardial infarction). The network training process consisted of the retrospective selection of a large number of patients who had presented with anterior chest pain in whom the presence or absence of acute myocardial infarction was known. Training consisted of the repeated sequential presentation of the training set to the network until the error in the output stopped decreasing. Overtraining was prevented by the testing of the network on a subgroup of patterns to which it had never been exposed. When the error made on this set stopped decreasing, training was considered optimized. Two groups of patients were utilized in the study. The first group, or low-risk (LR) group, consisted of 346 patients who presented to the emergency department and were felt to have a low risk for the presence of myocardial infarction. This group was utilized to both train and test. The second group, or high-risk (HR) group, consisted of 350 patients who had been admitted to the coronary intensive care unit to rule out the presence of myocardial infarction. All these patients were felt to have a high likelihood for myocardial infarction when initially evaluated. This group was used only to train. The breakdown of these groups is summarized in Table 2.

775

Improving the Accuracy of a Neural Network Table 1: Input Variables. History

Past history

Examination

Agea

Past AM1

Sex

Angina

Left anterior location of pain

Diabetes

Jugular venous 2-mm ST elevation distension 1-mm ST elevation Rales ST depression

Hypertension

Electrocardiogram

T wave inversion

Nausea and vomiting Significant ischemic change

Diaphoresis Syncope Shortness of breath Palpitations Response to nitroglycerin “Analog coded.

Table 2: Patient Groups. LRA group

Low Likelihood Group Nonmyocardial infarction 153 Myocardial infarction 20 Weight file derived from training LRA

LRB group 153 20 LRB

High Likelihood Group Nonmyocardial infarction 230 Myocardial infarction 120 Weight file derived from training HR The training parameters of learning rate and momentum were set a t 0.05 and 0.9, respectively. Five pattern sets were constructed and utilized to train the networks. Sets #1-2: the 346 low-risk patients were divided into two groups (LRA and LRB), each with half of the patients w h o did not sustain a myocardial infarction and half of the patients who did

776

William G. Baxt

sustain a myocardial infarction. Set #3: the 350 high-risk patients were used to construct one high-risk pattern set (HR). In order to demonstrate that the two-network strategy was responsible for any improvement in accuracy and not the information on which the networks were trained, two additional pattern sets were constructed that contained all the information present in both the low- and high-risk patient groups. Sets #4-5: each of the low-risk pattern sets was combined with the high-risk training set (LRA + HR and LRB + HR) and used to train one network each. Twenty percent of the patients who did and did not sustain a myocardial infarction were removed from each of the five sets and used as a training validation set. Training was considered optimized when the total number of errors made on the validation set stopped decreasing using an output of 0.5 as the discriminator between the presence and absence of myocardial infarction ( 2 0.5 = myocardial infarction; < 0.5 = no myocardial infarction). The dual-trained networks were tested on the two low-risk pattern sets (LRA and LRB). This was accomplished programmatically by using both networks simultaneously; one network trained on one of the lowrisk pattern sets (LRA or LRB patterns) and the other network trained on the high-risk pattern sets (HR patterns). The dual network was always tested on the pattern set to which it had not been exposed (ie., LRA patterns tested on the dual network trained on the LRB pattern set). Two diagnostic outputs were generated. The output from the network trained on the low-risk pattern sets was chosen as the diagnostic output unless it was greater than 0.05. In this case, the output from the second network trained on the high-risk pattern set was used as the diagnostic output. This number was chosen by empirical trials using the training set with cut-off values between 1 and 0.001. An output of 0.5 was again utilized to distinguish between the presence and absence of a myocardial infarction. The entire process is outlined in Figure 2. Each of the low risk pattern sets was also tested on a single network trained on the two sets of the combined low- and high-risk patterns (LRA + HR, LRB + HR). Each low-risk pattern set was tested on the network trained on the combined set that had been formed by the union of the high-risk pattern set with the low-risk pattern set to which it had not been exposed. 3 Results

The low-risk patients were tested because this group represented the typical patient group presenting to the emergency department in whom the diagnosis of acute myocardial infarction would be entertained. The two low-risk pattern sets (LRA and LRB patterns) were tested on the dual network trained on the low-risk pattern set to which it had not been exposed. The dual network correctly identified 39 of 40 patients who


777

TWIN

TEST

Figure 2: Separated training and testing algorithm used to develop the dual network system. Table 3: Network Testing.

Myocardial infarction Correct Incorrect Detection rate Nonrnyocardial infarction Correct Incorrect False alarm rate

Dual network

Single network

39

39

1

1

97.50%

97.50%

301

287

5 1.66%

19 6.21%

p

0.003

had sustained a myocardial infarction [detection rate 97.5% (95% C.I.= 95-loo%)] and 301 of 306 patients who had not sustained a myocardial infarction [false alarm rate 1.63% (95% C.I. = 0.2-3.0%)]. The full process of designation and elimination is illustrated in Table 3. The single networks trained on the control combined patterns sets were tested on the low-risk pattern sets to which they had not been exposed. The two single networks identified 39 of 40 patients who had sustained a myocardial infarction [detection rate 97.5% (95% C.I. = 95loo%)] and 287 of 306 patients who had not sustained a myocardial infarction [false alarm rate 6.2% (95% C.I.= 3.5-7.6%)]. The chi-square analysis of the difference in false alarm rates of the single and dual network was carried out by constructing 2 x 2 contingency tables from the summarized results of the experimental and control testing illustrated in

778

William G . Baxt

Table 3. The McNemar symmetry chi-square analysis of this table had a p value of 0.001 and Yates corrected chi-square analysis had a p value of 0.003. 4 Discussion

The purpose of this study was to develop a strategy that could take advantage of the fact that artificial neural networks trained to make categorical decisions can individually be trained to optimize either detection rate or false alarm rate. By utilizing dual networks, one network was used to define those patients who had not sustained a myocardial infarction while the other network was used to define those patients who had sustained a myocardial infarction. The approach used signal strength analysis to determine which network to use to make the final diagnostic decision. The signal or output of the network trained on the low-risk patterns was measured for each pattern the network processed. If the signal level rose above a set threshold, this was taken as an indication that that patient may have sustained a myocardial infarction. At this point, the analysis was shifted to the network trained on the high-risk patterns and, thus, trained to identify the presence of myocardial infarction. Although the empirically chosen signal cut-off of 5.0 x lo-’ appears to be low, it should be pointed out that the mean output generated by the network trained on the low-risk patterns set from patients who had not sustained a myocardial infarction was 1.0 x lop6. Variation of this threshold has a significant impact on the accuracy of the network and care had to be taken to empirically choose the most optimized separation point. By separating the analysis in this manner, the negative reciprocal effects on detection rate and false alarm rate accrued by improving one or the other seems to have been obviated. Although single network processing is inherently highly accurate (Baxt 1991a,b), simultaneous optimization of both detection rate and false alarm rate could be achieved only by use of the two separate weight sets in concert. This strategy yielded a higher detection rate and lower false alarm rate than the use of weight sets derived from any other single optimized patient pattern set. The fact that the separated training and dual analysis imparted a higher accuracy to network function is supported by the result obtained from the control pattern sets. The same training information was available to the single network as to the dual network, yet the single network approach was not able to perform as accurately as the dual network. Separated training appears to impart a higher degree of accuracy to the performance of the artificial neural network in this setting; however, it should be pointed out that the improvement in accuracy is not profound. In other settings this small improvement in accuracy may not be significant. In the setting of the diagnosis of a disease such as myocardial infarction, which is a disease of both low incidence and a disease with


779

a major penalty for misdiagnosis, the optimization gained by the use of separated training is highly desirable. Misdiagnosis resultant from a less than optimized detection rate can lead to significant morbidity or even mortality. Because myocardial infarction is a disease of low incidence, a less than optimized false alarm rate will lead to the unwarranted admission of large numbers of patients to the coronary care unit and hospital (Goldman et al. 1988). The major drawback of this study is that this strategy has been tested only retrospectively. This approach must be tested prospectively on a large number of patients before it can be statistically fully validated. It may also be possible to duplicate these results using a single network and single training pattern, since this approach was not fuily exhausted. Furthermore, this methodology was tested only in this specific setting and may not be transportable to other diagnostic settings or other applications. However, if the methodology is validated, it may also be applicable to other applications of artificial neural network technologies. Acknowledgments

I thank Doctors Hal White and David Zipser for their help with the technical aspects of this study and Kathleen James for her help in the preparation of this manuscript. References Baxt, W. G. 1991a. Use of an artificial neural network for data analysis in clinical decision-making: The diagnosis of acute coronary occlusion. Neural Comp. 2,480-489. Baxt, W. G. 1991b. Use of an artificial neural network for the diagnosis of myocardial infarction. Ann. Intern. Med. 115, 843-848. Bounds, D. G., Lloyd, I? J., and Mathew, B. G. 1990. A comparison of neural network and other pattern recognition approaches to the diagnosis of low back disorders. Neural Networks 3, 583-591. Cios, K. J., Chen, K., and Langenderfer, R. A. 1990. Use of neural networks in detecting cardiac diseases from echocardiographic images. I E E E Eng. Med. Biol. Mag. 9, 58-60. Eberhart, R. C., Dobbins, R. W., and Hutton, L. V. 1991. Neural network paradigm comparisons for appendicitis diagnosis. Proceedings of the Fourth Annual I E E E Symposium on Computer-Based Medical Systems, 298-304. Goldman, L., Cook, E. F., Brand, D. A., Lee, T. H., Rouan, G . W., Weisberg, M. C., Acampora, D., Stasiulewicz, C., Walshon, J., Terranova, G., Gottlieb, L., Kobernick, M., Goldstein-Wayne,B., Copen, D., Daley, K., Brandt, A. A., Jones, D., Mellors, J., and Jakubowski, R. 1988. A computer protocol to predict myocardial infarction in emergency department patients with chest pain. N . Engl. 1.Med. 318, 797-803.

780

William G. Baxt

Hiraiwa, A., Shimohara, K., and Tokunaga, Y. 1990. EEG topography recognition by neural networks. IEEE Eng. Med. Biol. Mag. 9, 39-42. Hudson, D. L., Cohen, M. E., and Anderson, M. F. 1988. Determination of testing efficacy in carcinoma of the lung using a neural network model. Symposium on Computer Applications in Medical Care 1988 Proceedings: 12th Annual Symposium, Washington, DC 12, 251-255. Kaufman, J. J., Chiabera, A., Hatem, M., et al. 1990. A neural network approach for bone fracture healing assessment. IEEE Eng. Med. Biol. Mag. 9, 23-30. Marconi, L., Scalia, F., Ridella, S., Arrigo, P., Mansi, C., and Mela, G. S. 1989. An application of back propagation to medical diagnosis. Proceedings of the International Joint Conference on Neural Networks, Washington, DC 2, 577. McClelland, J. L., and Rumelhart, D. E. 1988. Training hidden units. In Explorations in Parallel Distributed Processing, J. L. McClelland and D. E. Rumelhart, eds., pp. 121-160. The MIT Press, Cambridge, MA. Mulsant, G. H., and Servan-Schreiber, E. 1988. A connectionist approach to the diagnosis of dementia. Symposium on Computer Applications in Medical Care 1988 Proceedings: 12th Annual Symposium, Washington, DC 12, 245-250. Pozen, M. W., DAgostino, R. B., Selker, H. P., Sytkowski, P. A., and Hood, W. B., Jr. 1984. A predictive instrument to improve coronary-care-unit admission practice in acute ischemic heart disease: A prospective multicenter clinical trial. N. Engl. J. Med. 310, 1273-1278. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing: Explorations in theMicrostructure ofCognition, D. E. Rumelhart and J. L. McClelland, eds., pp. 318-364. The MIT Press, Cambridge, MA. Saito, K., and Nakano, R. 1988. Medical diagnostic expert system based on PDP model. Proceedings of the International Joint Conference on Neural Networks, San Diego 2, 255-262. Smith, J. W., Everhart, J. E., Dickson, W. C., Knowler, W. C., and Johannes, R. S. 1988. Using the ADAP learning algorithm to forecast the onset of diabetes mellitus. Symposium on Computer Applications in Medical Care 1988 Proceedings: 22th Annual Symposium, Washington, DC 12, 261-265. Stornetta, W. S., and Huberman, B. A. 1987. An improved three-layer, backpropagation algorithm. In Proceedings of the I E E E First International Conference on Neural Networks, M. Caudill and C. Butler, eds. SOS Printing, San Diego, CA. Weigend, A. S., Huberman, B. A., and Rumelhart, D. E. 1990. Predicting the future: A connectionist approach. Stanford PDP Research Group, April. Widrow, G., and Hoff, M. E. 1960. Adaptive Switching Circuits. Institute of Radio Engineering Western Electronic Show and Convention. Convention Record, Part 4,96104. Yoon, Y. O., Brobst, R. W., Bergstresser, P. R., and Peterson, L. L. 1989. A desktop neural network for dermatology diagnosis. J. Neural Network Comp. 43-52.

Received 22 October 1991; accepted 6 February 1992.

This article has been cited by: 2. M.A. Moussa. 2004. Combining Expert Neural Networks Using Reinforcement Feedback for Learning Primitive Grasping Behavior. IEEE Transactions on Neural Networks 15:3, 629-638. [CrossRef] 3. G.P. Zhang. 2000. Neural networks for classification: a survey. IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews) 30:4, 451-462. [CrossRef] 4. Klaus Prank, Clemens Jürgens, Alexander von zur Mühlen, Georg Brabant. 1998. Predictive Neural Networks for Learning the Time Course of Blood Glucose Levels from the Complex Interaction of Counterregulatory HormonesPredictive Neural Networks for Learning the Time Course of Blood Glucose Levels from the Complex Interaction of Counterregulatory Hormones. Neural Computation 10:4, 941-953. [Abstract] [PDF] [PDF Plus] 5. Kamal M. Ali, Michael J. Pazzani. 1996. Error reduction through learning multiple descriptions. Machine Learning 24:3, 173-202. [CrossRef] 6. Vassilios Petridis , Athanasios Kehagias . 1996. A Recurrent Network Implementation of Time Series ClassificationA Recurrent Network Implementation of Time Series Classification. Neural Computation 8:2, 357-372. [Abstract] [PDF] [PDF Plus] 7. William G. Baxt, Halbert White. 1995. Bootstrapping Confidence Intervals for Clinical Input Variable Effects in a Network Trained to Identify the Presence of Acute Myocardial InfarctionBootstrapping Confidence Intervals for Clinical Input Variable Effects in a Network Trained to Identify the Presence of Acute Myocardial Infarction. Neural Computation 7:3, 624-638. [Citation] [PDF] [PDF Plus] 8. David Hamilton, Peter J. Riley, Ueber J. Miola, Ahmed A. Amro. 1995. A feed forward neural network for classification of bull's-eye myocardial perfusion images. European Journal of Nuclear Medicine 22:2, 108-115. [CrossRef] 9. Michael L. Astion, Mark H. Wener, Ronald G. Thomas, Gene G. Hunder, Daniel A. Bloch. 1994. Application of neural networks to the classification of giant cell arteritis. Arthritis & Rheumatism 37:5, 760-770. [CrossRef] 10. William G. Baxt. 1993. A neural network trained to identify the presence of myocardial infarction bases diagnostic decision on nonlinear relationships between input variables. Neural Computing & Applications 1:3, 176-182. [CrossRef]

ARTICLE

___ Communicated by Richard Lippmann

Rule-Based Neural Networks for Classification and Probability Estimation Rodney M. Goodman Charles M. Higgins John W. Miller Depv tli??f I t of E/ectvicn/ E q i i i t w l I is, Cnlifomin Irisfihfe of E c l i i ~ o / o ~PIo~s,n d ~ w iCA , 92125 USA

In this paper we propose a network architecture that combines a rulebased approach with that of the neural network paradigm. Our primary motivation for this is to ensure that‘the knowledge embodied in the network is explicitly e‘ncoded in-the form of understandable rules. This enables the network’s decision to be understood, and provides an audit trail of how that decision was arrived at. We utilize an information theoretic approach to leafning a model of the domain knowledge from examples. This model takes ;he form of a set of probabilistic conjunctive rules between discrete input evidence variables and output class variables. These rules are then mapped onto the weights and nodes of a feedforward neural network resulting in a directly specified architecture. The network acts as parallel Bayesian classifier, but more importantly, can also output posterior probability estimates of the class variables. Empirical tests on a number of data sets show that the rulebased classifier performs comparably with standard neural network classifiers, while possessing unique advantages in terms of knowledge representation and probability estimation. 1 Introduction

The rule-based knowledge representation paradigm is well established as a powerful model for higher level cognitive processes (Newell and Simon 1972; Chomsky 1957), whereas the connectionist paradigm seems very well suited to modeling lower level perceptual processes. In particular, rule-based expert systems have proven to be a successful software methodology for automating complex decision-making tasks. Primary advantages of this approach include the Facility for explicit kiiozuledge re;’reseiztatiori in the form of rules and objects, and the ability of a rule-based Nrirval Coiiipirtatiorr 4, 781-804 (1992)

@ 1992 Massachusetts liistitute of Technology

782

Rodney M. Goodman et al.

system’s reasoning to be understood by humans. However, current rulebased systems are firndmentally restricted in speed of execution, and hence in their applicability to real-time systems, because of the serial computations performed in present inference processing schemes. In addition, current rule-based systems are brittle in their ability to deal with the uncertainties inherent in real-world information and lack any ability to generalize to novel problems. Neural network paradigms, on the other hand, are typically quite adept at modeling problems that occur in pattern recognition, visual perception, and control applications. This ability is due (at least in part) to their inherent robustness in the presence of noise, the lack of which plagues the implementation of rule-based systems in practice. In addition, neural networks are inherently parallel, and special-purpose parallel neural network hardware implementations promise quantum leaps in processing speeds, suitable for real-time systems. However, neural networks, as presently implemented, are poor at explaining their reasoning in human understandable terms because they embed domain knowledge in the implicit form of weights and hidden nodes. The network is thus very much of a “black-box” solution, whose structure and reasoning are relatively inaccessible to higher level reasoning or control processes, such as the human user. In many areas of expertise such as medical, legal, or life-critical domains, it is an absolute requirement that an autonomous reasoning agent be able to explain its decisions to a higher level authority such as a judge. We are therefore led to ask whether it is possible to amalgamate the rule-based and connectionist approaches into a hybrid scheme, combining the better aspects of both, while eliminating the drawbacks peculiar to each in the process. A natural common ground on which to combine these approaches is that of probability. We show that by referencing both rule-based systems and neural networks to the common normative frame of probability, a novel and practical architecture emerges. In this paper we propose a hybrid rule-based connectionist approach that overcomes many of the problems outlined above. Our ultimate goal is the automatic learning of rule-based expert systems that can perform inference in parallel when implemented on neural network architectures. For the purposes of this paper, however, we concentrate on the problem of classification and posterior probability estimation, implemented on rule-based feedforward neural nets. We show how probabilistic rules can be used as a natural method for describing the high-order correlation information in discrete (or categorical) data, and how the hidden units of a feedforward net can easily implement such rules. Furthermore, we show how information theory and minimum description length theory can be used to learn only the most important of these rules, thus directly specifying the network architecture in terms of hidden units and connectivity. Finally, we show that output probabilities may be estimated using a parallel Bayesian approach, which is a natural extension of a first-order Bayes classifier. The architecture proposed in this paper is

Rule-Bascd Neural Networks

783

therefore novel for a number of reasons. First, it avoids iterative network training processes (such as backpropagation) by dimctl!y specifying network weights in terms of probability estimates derived from the example data. Second, the hidden nodes of the network are automatically learned from the data without having to specify the number of such nodes. This approach leads to the advantage that network parameters are directly interpretable in terms of rules with associated weights of evidence between the nodes. Third, given that it is usually necessary to assume some form of conditional independence among the input variables in order to render the probability estimation problem tractable, the proposed classification ~~t independence scheme is novel in that it uses d f ? f a - d e p " ~ [ ~ econditional assumptions only to the extent jiisfifieil by the data. Networks that learn from example form the basis of many current connectionist paradigms. The success of the backpropagation (Rumelhart ct a / . 1986) and related algorithms is that, given a specific architecture in terms of input, hidden, and output nodes, the connection weights between these nodes needed to model the high-order correlations in the example data can be easily learned. Learning the network nrclrifccfure itself, and generating true output probability estimates is a considerably more difficult task for current neural network paradigms. It is interesting to note that Uttley (1959), conceived of a network in which all higher order input-output correlations were stored. This network stored a number of probabilities exponential in the number of input variables, but contained the information necessary for calculating the conditional probability of any set of output variables, given any other set of input variables. In principle, this provided a method of calculating output probabilities at the expense of exponentially many of what we would now call hidden units, many of which were redundant in the sense of not contributing to the output information. Networks whose architectures include high-order connections chosen rnridoiiily were of course among the very early neural network models (Rosenblatt 1962; Aleksander 1971). At the other extreme, in a previous paper we showed how simple first-order correlations could be used to successfully predict output probabilities (Goodman r t a / . 19891, provided the data were well specified by such low-order information. Between these extremes lie approaches that make subjective prior judgments about conditional independence to decide 7diicI1 higher order conjunctive probabilities to store, such as the Bayesian networks described by Pearl (1988), Lansner and Ekeberg (19891, and Kononenko (1989). This paper develops in the following way. First, we outline our rulebased network architecture. Second, we describe our methodology for learning a set of probabilistic production rules from example data, using an information theoretic appruach. Third, we show how these rules are then mapped onto the nodes and links of a feedforward neural network in such a manner that the network computes posterior class probabilities using a Bayesian formalism. We conclude with a comparative evaluation


784

of the approach using five data sets, including medical diagnosis and protein structure prediction. 2 A Rule-Based Classifier Architecture

We consider the problem of building a classifier that relates a set of K discrete feature variables (or attributes) comprising the set Y = { YI. . . . . YK} to a discrete class variable X. Each attribute variable takes values in the alphabet {y!. . . . .y;”’}. 1 5 1 5 K, where ml is the cardinality of the lth attribute alphabet. The class variable X takes discrete values from the set {x,. . . . . x,,,},where rn is the cardinality of the class. We also assume that we are given an initial labeled training set of N examples where each example is of the form {Y, = G ,. . . . YK = yi, X = x,}. The supervised learning problem we set ourselves is to learn a classifier that when presented with future unseen attribute vectors (which may be either partial or complete) will estimate the posterior probability of each class. We may then wish to output either these probabilities, or the class variable with the highest probability as the decision made by the classifier. Note that we are particularly interested in real data sets in which the classification is often nondeterministic or noisy, that is, there exists class overlap and hence a fundamental ambiguity in the mapping from Y to X. In this case there is no perfect classifier for the problem and the performance of the classifier as measured by its error rate will be nonzero, and bounded below by the optimal Bayes error rate p:. The rule-based architecture we propose takes the form of a three-layer feedforward network as shown in Figure 1.

Input Atbiibutes

Conjmaive Rulcs

Figure 1: Architecture of the rule-based classifier.

outplc class

Rule-Based Neural Networks

785

The input nodes correspond to each possible attribute-value pair in the input attribute space. The hidden layer consists of a set of lRJ conjunction detector nodes (or AND gates), one for each rule in the set of rules R. These hidden nodes detect the conjunction of one or more input . a conjuncattribute-value pairs of the form { Y1 = Y;. . . . . YI = ~ } When tion is detected the rule fires and the node outputs a 1. When the node is not firing it outputs a 0. The output layer consists of one node for each output class. The action of a rule firing contributes an activation from the hidden rule node into one or more output class nodes. The contribution into the ith output node from rule r, is given by the link weight zu,,,and represents the weight of evidence for the conclusion, given the occurrence of the left-hand side conjunction of attribute values. Each rule node together with its output link can therefore be considered to be implementing an lth-order conjunctive rule of the form

IF { Y1 = Y;. . . . . YI= yr} THEN X

= x,

with STRENGTH zu,,

The rule has a conjunction of input attribute-value pairs on its left-hand side (LHS), and a particular class attribute-value pair on its right-hand side (RHS). The weights w,, can be positive or negative depending on whether the rule supports the truth or falsity of the RHS conclusion. Each output node accumulates the inputs feeding into it from the rules that have fired and outputs a quantity that is a function of the particular activation function and threshold used in the node. Our design problem is then to implement a set of rules and associated weights, together with a suitable set of output activation functions and thresholds, such that the output of each class node is an estimate of the corresponding class probability. 3 Learning Rules Using Information Theory We now consider how to learn the set of rules R from the given training data such that the classifier will operate in the desired manner. Clearly we do not want to implement nll possible conjunctive rules, as the size of the hidden layer will be exponential in the size of the input attributes. Rather we require a sirfflcrcntly good set of rules that allows the network to both load the training data and to generalize to new data while having a performance that approaches the optimum Bayes risk. Alternatively, given a fixed resource constraint in terms of l R I allowed hidden units, we should implement the bcsf (72 rules, according to some "goodness" criterion. Let us rephrase the previously defined rule in terms of a prohnhilistic production rule of the form:

If si then x, with probability p

786


where p is the conditional probability p ( x , 1 s,), and sf represents the particular conjunction of attribute-value pairs found in the LHS of the rule. We wish to have a measure of the utility or goodness of such a rule. In a Hebbian sense such a rule might be considered good if the occurrence of the LHS conjunction of variables is strongly correlated with the RHS. Alternatively, such a rule might be considered good if the transition probability p is near unity. For example, a rule with p = 1 indicates a deterministic rule in which the occurrence of s1implies X = x, with certainty. However, we will take an information theoretic approach to this problem, and consider that the goodness of such a rule can be measured by the average bits of information that the occurrence of the LHS sf gives about the RHS X = x,. We have introduced such a measure, called the ]-measure (Goodman and Smyth 1989), which can be defined as

This measure possesses a variety of desirable properties as a rule information measure, not the least of which is the fact that it is unique as a nonnegative measure of the information that si gives about X (Blachman 1968). As can be seen the ]-measure is the product of two terms. The first is p(s,), the probability that the LHS will occur. This term can be viewed as a preference for generality or simplicity in our rules; that is, the left-hand side must occur relatively often for a rule to be deemed useful. The other term is the cross-entropy of X and X given s,, and as such is well-founded measure of the goodness of fit between our a posteriori belief about X and our a priori belief (Shore and Johnson 1980). Hence, maximizing the product of the two terms, ](X; s,), is equivalent to simultaneously maximizing both the simplicity of the hypothesis si and the goodness of fit between sf and a perfect predictor of X. The simplicity of s, directly corresponds to the number of attribute-value conjunctions in the rule LHS, that is, the rule order. Low-order rules have less LHS conditions, and thus a higher p(s,). There is a natural trade-off involved here because, typically, one can easily find high-order rules (less probable s,s) that are accurate predictors, but one has a preference for more general low-order rules (more probable s,s). The ]-measure thus provides a unique method of not only ranking the goodness of a set of rules, but also being able to tell if a more specialized rule (one with more LHS conditions) is better or worse than a more general rule. This basic trade-off between accuracy and generality (or goodness of fit and simplicity) is a fundamental principle underlying various general theories of inductive inference (Angluin and Smith 1984; Rissanen 3989). We use the ]-measure in a search algorithm (Smyth and Goodman 1992) to search the space of all possible rules relating the LHS attributes Y to the RHS class X and produce a ranked candidate set S of the IS1 most informative rules that classify X. The search proceeds in a depth


787

first manner starting with a particular LHS conjunction and progressively specializes the rule until bounds indicate that a higher measure cannot be achieved by specializing turther. The search is potentially exponential but in practice is highly constrained by small sample estimators and information theoretic bounds that heavily penalize searching higher order rules (empirical results demonstrating this effect are given in Smyth and Goodman 1992). In addition, higher order rules that have lower information content than a corrcywnciing lower order (more general) rule can be omitted from the final rule list. From this large candidate set of rules S we next produce the final set of rules R, which will be implemented in the classifier. 4 Rule Pruning Using a Minimum Description Length Model

~

We have already described how we find an initially large candidate set of rules S that models tlie data. I t is well known, both empirically and from theory, that there is a trade-off between the complexity of the model and the quality of generalization performance. A model that is too simple will not have the representational power to capture the regularities of the environment, whereas a model that is too complicated may well overfit the training data and generalize poorly. When we speak here of generalization we are referring to the system’s mean performance in terms of classification accuracy (or a similar function) evaluated over some infinitely large independent test data set. The notion of Occam’s razor has been used to promote model parsimony: choose the simplest model that perfectly explains the data. Unfortunately this presupposes that there exists a model under consideration that can explain the data perfectly in this manner. In practical problems this is unlikely to be the case, siiice there is often a n ambiguity in tlie mapping from attribute space to the class labels. In this stochastic setting a more general version of Occam’s razor has been introduced (Rissanen 1984, 1987) under the title of minimum description length (MDL). The MDL principle is simple to state: choose the model that results in the least description length, where the description length is calculated by first sending a message describing the model [the complexity term L ( M ) ] ,followed by a message encoding the data given the model [the goodness-of-fit term L ( D I M ) ] . Thus we minimize: UMI

+ L ( D 1 Mi

MDL can be viewed as entirely equivalent to a Bayes maximum a posteriori (MAP) principle (where one chooses the model that maximizes the joint probability of the data and the model) by virtue of the fact that the description lengths are directly related to prior probabilities. We will refer primarily to tlie description length framework as it is somewhat more intuitive.


788

In the context of applying MDL to the problem at hand we seek a pruned rule set R 2 S, which possesses near-minimal description length among all possible subsets-finding the optimal solution is clearly intractable in the general case. For a more general discussion of search in MDL contexts see Smyth (1991). The algorithm we propose is a simple greedy procedure that, starting from an initially empty set of rules, continues to add the next-best rule to the current set, and terminates at a local minimum of the description length function. In more detail the algorithm is described as follows: 4.1 MDL Rule Pruning Algorithm. 1. Let R = { }.

2. Find the rule r E S such that when R U r is evaluated on the training data as a classifier the sum of the goodness of fit and the complexity of r is minimized. 3. Remove rule r from S .

4. If the description length of length of R then stop.

5. Else let R

= R UY

R U r is greater than the description

and return to step 2 .

At this point in the discussion we can treat the classifier itself as a ”black box” that simply takes a rule set R, a set of unlabeled test data, and produces probability estimates of the class labels. We will describe this ”black box” in detail in the next section. Let us first look at the other part of the algorithm, which we have not defined in detail, namely the calculation of description length. Suppose we have N training samples. For the ith sample, 1 5 i _< N, let xt,,(i) be an index to the true class, i.e., the training label. Let p[xtme(i)]be the classifier’s estimate of this class given the ith attribute vector. Hence, the length in bits to describe the data given the model (the classifier) is

The complexity term, the length in bits to describe the classifier itself, may be arrived at by a number of arguments. In principle we need to send a message describing for each rule its left-hand side component, its right-hand side component, and an estimate of the transition probability of the rule. One of the key factors in proper application of MDL is the precision with which these probabilities are stated. It is a well known general result that very often the optimal precision for model parameters is proportional to fl,or about (1/2) log N bits per parameter. In practice this term dominates as N becomes large over the specification of the rule components. Since these specification terms also depend on the


789

particular coding scheme (or the prior bias in Bayesian terminology), we choose to ignore these terms in the optimization or search and propose that the complexity be simply proportional to the (3/2) log N precision terms. This penalty scheme has been widely used in practice by other authors (Rissanen and Wax 1988; Tenorio and Lee 1990, et al.). Hence, for rule set R the complexity is assessed as

As we shall discuss later in the section on empirical results, this simple pruning algorithm is very effective at discovering parsimonious rule sets that account for the data. Heuristically, for multivalued class problems, we can understand the behavior of the algorithm as initially trying to account for each class by a single accurate rule for each, and then integrating rules that cover regions of the attribute space with high accuracy. In particular, as we shall discuss in the next section, by evaluating the performance of the classifier on each candidate rule, we can match the rule set to the nature of the classifier (conditional independence in this case). If I R I is the number of rules in the final rule set then it is straightforward to show that the complexity of the pruning algorithm approximately scales as NISIIRI’. Typically IR(> r

-

(3.11)

which climbs to an asymptotic value of ,/ti/. for m m. To compare this with intuition, consider that the sample mean of {XI.x2.. . . . x,,,} approaches to within a variance of 1/2mrr, so that (3.12) which makes equation 2.22 agree with equation 3.11 for large enough 11. In this sense, the statistical mechanical theory of learning differs from conventional Bayesian estimation only in its choice of an unconventional performance criterion APP. 3.2 Small Training Set Size. We can demonstrate overtraining even for this elementary example when the size of the training set m is small and the the error on the training set f T is reduced too much. It will be

Griff L. Bilbro and David E. Van den Bout

846

sufficient to consider equation 2.12 for r n ~ ~0 ~occurs only when the error of the prior estimate for the unknown constant is larger than the width of the prior distribution. Furthermore it can occur only when m is not too large, since we have assumed mtc 20 the slznpes of the curves are almost unchanged, even though the vertical scale is different: saturation occurs at about the same value of in. Even when the prior greatly overrepresents poor nets, their effect on the prediction rapidly diminishes with training set size. This is important because in actual training, the effect of the initial configuration is also quickly lost. For ~n < 20 the predictions are not valid in any case, since

852

Griff L. Bilbro and David E. Van den Bout

our simple error function does not reflect the actual probability even approximately for in < k in these nets. It is for m < 20 where the only significant differences between the two families of curves occur. We have also been able to draw the same conclusions from less structured prior densities generated by assigning positive normalized random numbers to intervals of the domain. In fact, we were not able to produce curves worse than those of Figure 3. Moreover, we generally find that TLS predicts that about twice as many samples as neurons are needed to train competitive nets of other sizes. The previous curves were produced with large total sample sets N = 1000. We subsequently reran the experiments with N = 100 with essentially identical results (we d o not plot them because the two are indistinguishable). It is reassuring that the predictions of the theory are reliable even for relatively small sample sets.

6 Conclusion

We have derived the TLS theory of learning using the principle of maximum entropy instead of a combination of statistical mechanics and Bayes' rule. We show that TLS can be applied generally to learning and modeling. We apply it to learning a constant and to learning a one-dimensional density in the annealed approximation of TLS. We considered the effects of varying the number of examples m, the target training error CT (or equivalently P), and the choice of prior density o(')(w). These experiments on learning a density are consistent with learning a binary output (Bilbro and Snyder 19901, a ternary output (Chow et al. 19911, and a continuous output (Bilbro and Klenin 1990). We find if saturation occurs for m substantially less than the total number of available samples, say m < T / 2 , that in is a good predictor of sufficient training set size. Moreover there is evidence from a reformulation of the learning theory based on the grand canonical ensemble that also supports this statistical approach (Klenin 1990). For small problems the theory is very easy to use. We have no experience yet in applying the theory to large problems with more than 50 unknown weights or parameters. The TLS theory appears promising as a predictor of sufficient training set size; however, another question remains. It is difficult to obtain large overtraining effects from this theory even though some traces of overtraining remain: The curves of Figure 3 can be made to cross for small m and poor prior. In equation 3.16 of the elementary example, it is possible to obtain a positive F T , ~ However, ~ ~ . it appears that some aspects of overtraining do not survive the annealed approximation. This is consistent with the experience of other workers (Solla 1990).

Maximum Entropy and Learning Theory

853

References Bilbro, G. L., and Klenin, M. 1990. Thermodynamic models of learning: Applications. Unpublished. Bilbro, G. L., and Snyder, W. E. 1990. Learning theory, linear separability, and noisy data. CCSP-TR-90/7, Center for Communications and Signal I'rocessing, Box 7914, Raleigh, N C 27695-7914. Chow, M. Y., Bilbro, G. L., and Yee, S. 0. 1991. Application of learning theory to single-phase induction motor incipient fault detection artificial neural networks. Irit. J. N m r d S y s t . 2(1,2), 91-100. DeSieno, D. 1988. Adding a conscience to competitive learning. I n I E E E / r f k r riatiorin1 Corifcr~'~~cc O I I Nc~rrrnlNc?i[lorks, pp. 1:117-1:124. Jaynes, E. T. 1979. Where Do We Stand on Maximum Entropy? I n M n s i r r i ~ m Eritropy Forrnalisrrr, R. D. Leven and M. Tribus, eds., pp. 17-118. The MIT Press, Cambridge, MA. Klenin, M. 1990. Learning models and thermostatistics: A description of overtraining and generalization capacities. NETR-90/3, Center for Communications and Signal Processing, Neural Engineering Group, Box 7914, Raleigh, NC 27695-7914. Levin, E., Tishby, N., and Soll'i, S. A. 1990. A statistical approach to learning and generalization in layered neural networks. Proc. l E E E 78(10), pp. 1568-1 574. Melton, M., Plian, T., Reeves, U., and Van den Bout, D. E. 1992. The TInMANN VLSI Chip. I E E E Trms. N w r n l Nrtzvorks 3(3). Ritter, H. 1991. Asymptotic lcvel density for a class of vector quantization processes. I E E E Trmis. N w r d Nrtworks 2(1), pp. 173-175. Rumelhart, D., and McClelland, J. 1986. Pnrnllrl Distributed Proccssirig: Ex$ratioris ir7 the Micr~stritrtrrrc~ of Copition, Chapter 5. The MIT Press, Cnmbridge, MA. Schwartz, D. B., Solla, S. A., Samalan, V. K., and Denker J. S. 1990. Exhaustive learning. Ntwrd Cornp. 2(3), pp. 374-385. Solla, S. A. 1990. Private communication. Tishby, N., Levin, E., and Solla, S. A. 1989. Consistent inference of probabilities in layered networks: Predictions and generalization. IJCNN, pp. 11:403410. IEEE, New York. ~

Received 1 November 1991; accepted 25 March 1992

This article has been cited by: 2. David J. Miller , Lian Yan . 2000. Approximate Maximum Entropy Joint Feature Inference Consistent with Arbitrary Lower-Order Probability Constraints: Application to Statistical ClassificationApproximate Maximum Entropy Joint Feature Inference Consistent with Arbitrary Lower-Order Probability Constraints: Application to Statistical Classification. Neural Computation 12:9, 2175-2207. [Abstract] [PDF] [PDF Plus] 3. Lian Yan, D.J. Miller. 2000. General statistical inference for discrete and mixed spaces by an approximate application of the maximum entropy principle. IEEE Transactions on Neural Networks 11:3, 558-573. [CrossRef] 4. Shie-Jue Lee, Mu-Tune Jone. 1996. An extended procedure of constructing neural networks for supervised dichotomy. IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) 26:4, 660-665. [CrossRef]

Communicated by Les Valiant

Tight Bounds on Transition to Perfect Generalization in Perceptrons Yuh-Dauh Lyuu Igor Rivin NEC Research Institute, Princeton, N ] 08540 U S A "Sudden" transition to perfect generalization in binary perceptrons is investigated. Building on recent theoretical works of Gardner and Derrida (1989) and Baum and Lyuu (1991), we show the following: for a > a , = 1.44797. if (Y n examples are drawn from the uniform distribution on { + l , -1)" and classified according to a target perceptron w tE { + l , -l}"as positive if w t x 2 0 and negative otherwise, then the expected number of nontarget perceptrons consistent with the examples is 2-@(fi); the same number, however, grows exponentially 2@(") if a < a,. Numerical calculations for n up to 1,000,002 are reported.

. .,

-

1 Introduction

We are interested in the following problem of sample complexity for valid generalization. Let rn examples be drawn from a probability distribution D and classified as positive or negative according to some threshold net f. We would like to pinpoint the value m where the expected number of non-target nets in a class of threshold nets, R,that consistently classify those examples shifts from exponential to negligible. Recently, Sompolinsky et al. (1990) suggest a phenomenon when the target function and the hypothesis function are both simple perceptrons (Minsky and Papert 1969) with fl weights and no hidden units, and the distribution D is uniform over the vectors {il.-1}". Using less rigorous methods of statistical physics, they argue that for m > 1.24n, perfect generalization is achieved as n + m; that is, with high probability, the target function is the only one consistent with the data set. Such conclusion finds agreement in simulation. (A conclusion similar to theirs is reached by Gyorgyi (1990) using the annealed approximation method.) Baum and Lyuu (1991)prove their conclusion rigorously for tn > 2.0821 n by showin that, then, the expected number of consistent hypotheses is only 2-@(&, which is also an upper bound on the probability for perfect generalization not to hold. Related theoretical work has been done by Baum and Haussler (1989). It is shown that if one can find a choice of weights such that at least a fraction 1 - ~ / of 2 a set of rn > 32zo/c ln(32 n ) / f random examples are Neural Computation 4, 854-862 (1992)


Transition to Perfect Generalization in Perceptrons

855

correctly classified, then, independent of the distribution D, one has high confidence of having error probability less than f on future examples. Here, ZLJ is the number of weights in the threshold net (i.e., the nets in IH) and I I is the number of nodes. Experiments by Bauni (1990) indicate >> io >> 1, that if D is uniform over the interior of the unit cube, and then the error rate is in fact F = zo/m for multilayer perceptrons. The following is an outline of our result. Let zu, E { +l. -l}” be the target perceptron. We are given a set of o. I I examples, where each example is a vector x drawn uniformly from {+l.-1}“ and classified as positive or negative according to whether XJ, . x 0 or not. For ( r > ( i t 1.44797. . ., the expected number of lion-ill, perceptrons consistent with the examples is shown to be 2-“)‘\’‘I, which is also an upper bound on the probability that there exists one such perceptron; the same number, however, grows exponentially 2“’”) if ( I i( I ( . . This is done in Section 4. Such an ( i t has been reached by Gardner and Derrida (1989). But their proof is not entirely correct; worse, it leads to false conclusions (Remark 4.3). We note that our result improves the one given by Baum and Lyuu (1991). Numerical calculations for I i up to 1.000.002 are reported in Section 5. 2 Definitions and Terminology

We follow the definitions and terminology used by Baum and Lyuu (1991). The number of examples will be denoted by 171 = ( I ti. Without loss of generality, assume I i > 0 is even and the target perceptron is ZLJ, =

( + I .+ I . . . . .

1

Let P(i) denote the probability that a perceptron y E {+l.-l}” at Ham~ a random x E { + l . -1 }”. Define ming distance i from 7 1 ~misclassifies

and

1-1

As there are exactly

perceptrons at Hamming distance i from zi+, the target perceptron, z , denotes the expected number of consistent perceptrons that are at Hamming perdistance i from zi+. Clearly, A(n.( I ) is the expected number of non-ZL’, ceptrons y E {+l.-l}” that consistently classify 171 random examples.

Yuh-Dauh Lyuu and Igor Rivin

856

Let b(k;n. p ) be the probability that n Bernoulli trials with probabilities p for success and q = 1 -p for failure result in k successes and n-k failures. Denote the number of successes in n trials by S,,,. Hence b(k;n.p) = Pr(S,,,p= k ) . Define &,k = b ( n / 2 + k;n. 112) = Pr(S,,,1/2= n / 2 + k ) , which measures the probability that the result of n Bernoulli trials, S,,,,, deviates from the mean, n/2, by k. We let all.k = 0 for Ikl > n/2. Note that aii.-k

= an.k

and

Throughout this paper, p will be 1/2. Finally, h ( n ) N g ( n ) means h ( n ) / g ( n )+ 1 as n

-

m.

3 Preliminary and Some Known Results

We list known results used by our proof in the next section. The first result can be easily proved from Stirling’s formula (Bollobas 1985, p. 4). Fact 3.1 (Baum and Lyuu 1991, Lemma 3.5) For an even integer a,

The next result is an exact recurrence relation for the P(i)s. Since its proof is very involved, we refer the interested reader to Baum and Lyuu (1991). Fact 3.2 (Baum and Lyuu 1991, Theorem 3.8) For i even, we have

and

P(i) - P(i - 1) = 0 We will need good bounds on the tails of normal distribution and sum of binomial distributions. The following two facts will be used later. Fact 3.3 (Bollobas 1985, Chernoff bound)

Fact 3.4 (Bollobiis 1985, p. 9) As x

--t

03,

857


We also need the DeMoivre-Laplace Limit Theorem to approximate the probability that S,,1 l 2 lies within a certain range near its mean, 11/2, by normal distributions. Note its range of applicability as expressed in how 111 and 122 may grow as I I and that their difference must go to infinity as n -, x.

Fact 3.5 (Bollobas 1985, p. 13) Suppose h, < I I ~ , I I ? - hl l11lI + lhzl = o [ ( r ~ / 4 ) ~ / 'Let ] . / I , = x,(i1/4)"~.Tlzeri Pr(n/2

-

x as 11

- x ,nrid

- a. 1''

+ hl 5 S,, 1/2 5 12/2 + 1 1 ~ )

1

-

e-l2I2dt

11

The following simple lemma will also be used.

Lemma 3.6 For a i l / i 5 1112(i may depend O H i f ) , we Iznve I

all-,,^!, = 0 (1Vfi) /=(I

as n

--t

m.

Proof. For simplicity, let both

IZ

and i be even. Now,

which is the last equation above from Fact 3.1. Symmetrically, one can prove that

A

= 0(

1 / 6 )

Hence, A = O ( m i n ( l / f i . 1/4)], which clearly proves the lemma no matter how i may depend on 11. Q.E.D. 4 The Main Result

The main result of this paper, to be proved in this section, is the following.

Theorem 4.1

By a proof that is as involved as that of Fact 3.2, Baum and Lyuu (1991, pp. 394-395) show that 2 , = 2-R(r')for any ( I > 1 when i 2 ,212. Hence, we only have to consider those 2 , terms with 1 5 i 5 ~ / 2 .We shall assume n/2 is odd and i even without loss of generality. The probability that a perceptron y at Hamming distance z from ?1+ classifies m random examples is calculated as follows. Assume, without loss of generality, that y's first i components are -1 and the rest are +l.

858


Let Z denote the sum of the first i components of the example x and Y that of the other iz - i components. Clearly, Y + Z and Y Z denote ZUr. x and y . x, respectively. For y to classify x as does, we must have: Y + Z 2 0 iff Y - Z 2 0 and Y + Z < 0 iff Y - Z < 0. Hence, the probability that y classifies rn random examples consistently is ~

~ ( i dlf ) 1 - P(i) = [Pr(O 5 5 Y) + Pr(O < -Z 5 Y) + Pr(O 5 Z < -Y)+ Pr(O < -Z < -Y)]"'

z

where Y is the sum of 11 - i random numbers fl,whereas Z is the sum of i random numbers +l. So, R(i) is equal to

J which, after some manipulations, lead to

[

((lJ+f

y=o

5+

?=-I/

q

-1

all-,

-

2

C

I/= ( l l - J ) / Z

all-, ,al

l,=-(l,-l)/2z=l/

1

From Lemma 3.6 we know that the second term within the brackets in the above formula is of order o(1). Hence we end up with

(4.1) Remark 4.2 We remark that Y and Z have limiting distributiom that are normal with expectation 0 and standard deviations 6 1 2 and 4 1 2 , respectively. First, for i = o ( n ) ,we have

(n - i ) / 2

2 N

-

T

fi

2" - 1

'J

1

Jic.-i,

Hence, P ( i ) < c for some constant c as n be o(n/ log3n ) , we have

+ m.

Further limiting i to


859

So we have shown that, for i = o(n/ln3 IZ), each z, is of size 2-"('") 2- 0. We proceed to bound each z, where i = 12( t z / In3 n ) . Observe that

Since the second term in 4.2 is o(1) as n substitute it with

as 2

& s:,fi~-'/' ~

(&

+

e-"i2ilz) dy

=

cx; by using Fact 3.3, we can

=

o ( 1 ) by Fact 3.4. The first

term of 4.2 is, by Fact 3.5,

The above two approximations combined with formula 4.1, we have

which is [Gardner and Derrida 1989, Eq. (2311 (4.4) where x = 1 - ( i / ~ ) . As in Gardner and Derrida (1989), we have

rv

e ~ ~ { -1 i ni-il-i)

lii(l-i)+rb

In[

$

tan-I(fi)+o(lj]}

(4.5)

One can easily show that, for ( t > 1.44797.. ., the exponent is less than zero for all x, whereas, for < 1.44797.. ., the exponent can be positive for some constant x [i.e., some i = @(ti)]. Thus we conclude our proof.


860

Remark 4.3 Here we comment on the reasons why Gardner and Derrida's argument is not correct. Using the Central Limit Theorem, they jump directly from Remark 4.2 to the claim that 4.1 = 4.3 = 4.4 without any qualification on the size of i, i.e., without going through the steps that we took for the case i = o(n/ log3n ) . Another way of saying the same thing is they take the DeMoivre-Laplace Limit Theorem, Fact 3.5, and immediately apply it to 4.1 to obtain 4.3, without taking into account that theorem's range of applicability. Hence, their result does not apply when i is, for example, "small." In fact, their formula 4.5, used without qualifications, would imply that A(n,rr)= 2-'(") for any o > ac, which is false by Theorem 4.1. 5 Numerical Results and Conclusion

We use the Mathernatica 2.0 program in Figure 1 to calculate A(n,tr). Figure 2 contains the log base e plot of A(n.1.448) against n. Although Theorem 4.1 predicts that it should eventually approach -@(fi), this graph fits perfectly the linear function 0.800727 - 9.8459 x 1OP6n. The explanation is that, at N = 1.448, which is very close to rxc = 1.44797. . ., terms of size 2-'(") in A(n.1.448) may dominate those of size 2-'(fi) for relatively big n before they start to be dominated. In Figure 3, we pick N = 2, which is much farther from the critical value ac, plot

Nbinom[ n-, i-,

Nbinom2[ i-,

prec-:

prec-:

LogGamma[ N[ n + 1, prec ] ] . LogGamma[ N[ i + 1, prec ] ] LogGamma[ N[ n - i + 1, prec ] I ]

40 ] := Exp[

40 ] := Exp[ LogGamma[ N[ 2 i + 1, prec ] ] ( 2 LogGamma[ N[ i + 1, prec ] ] + ( 2 i ) Log[ " 2, prec 1 1 ) I

P[ n-, 0 1 = 0 P[ n-, i-] := P[ n, i ] = If[ EvenQ[ i ] , P[ n, i - 1 1, P[ n, i - 1 ] + Nbinom2[ ( i - 1 ) / 2 ] Nbinom2[ ( n - i ) / 2 ] ] A[ n-,

alpha-] := Block[ ( sum = 0,pi = 0 ), For [ i = 1, I C=

n,

i++, If[ OddQ[ i 1, pi += Nbinom2[ ( i - 1 ) / 2 ] Nbinom2[ ( n - i + 1) / 2 ] 1; term = Nbinom[ n, i ] ( i - pi )"( n alpha ); sum += term 1; sum ]

Figure 1: Mathematica program used to calculate A(n.a ) .

861


0 0 0

0

0

Figure 2: A plot of log,[A(rz.1.448)].

Figure 3: A plot of [log,A(rz.2.0)]*.

862


[log, d ( n , 2.0)]*against n, and find it to be perfectly fit by the linear function -7241.51 2.46168n (drawn as a line), showing A ( n .2.0) grows at roughly ecm as the theorem says. In the other direction, we quickly get exponential blow-up when ( v is even just a little bit below oc. For example, d(ll000,002.1.447)E= 3.44762391 x We have not been able to prove rigorously the stronger claim (Gyorgyi 1990; Sompolinsky et al. 1990) that, for an (1' around 1.24, the probability that there exist nontarget perceptrons consistent with ( I n random examples, where N > o', is negligible. Our result puts n' < 1.44797. . . .

+

Acknowledgments The authors thank Eric B. Baum and David Jagerman for discussions. The authors also thank two anonymous referees whose suggestions substantially improved the presentation and accessibility of this paper.

References Baum, E. B. 1990. What can back propagation and k-nearest neighbor learn with feasible sized sets of examples? In Neurd Networks, E"AASIP Workshop 1990 Proceedings, L. B. Almeida and C. J. Wellekens, eds., pp. 2-25. Lecture Notes in Computer Science Series, Springer-Verlag, Berlin. Baum, E. B., and Haussler, D. 1989. What size net gives valid generalization? Neurul Comp. 1, 151-160. Baum, E. B., and Lyuu, Y.-D. 1991. The transition to perfect generalization in perceptrons. Neirral Comp. 3, 386-401. Bollobas, B. 1985. Random Graphs. Academic Press, New York. Gardner, E., and Derrida, B. 1989. Three unfinished works on the optimal storage capacity of networks. I. Phys. A: Math. Gen. 22, 1983-1994. Gyorgyi, G. 1990. First-order transition to perfect generalization in a neural network with binary synapses. Plzys. Rev. A 41, 7097-7100. Minsky, M., and Papert, S. 1969. Perceptrons: An lntrodirctioii to Cornputatioizal Geometry. The MIT Press, Cambridge, MA. Sompolinsky, H., Tishby, N., and Seung, H. 1990. Learning from examples in large neural networks. Phys. Rev. Lett. 65(13), 1683-1686. _

~

_

_

_

_

~

~

Received 14 November 1991; accepted 26 March 1992.

This article has been cited by: 2. Shao C. Fang, Santosh S. Venkatesh. 1999. Learning finite binary sequences from half-space data. Random Structures and Algorithms 14:4, 345-381. [CrossRef] 3. David Haussler, Michael Kearns, H. Sebastian Seung, Naftali Tishby. 1997. Rigorous learning curve bounds from statistical mechanics. Machine Learning 25:2-3, 195-236. [CrossRef]

Communicated by Harry Barrow

Learning Factorial Codes by Predictability Minimization Jurgen Schmidhuber Departmerit of Coriipirter Scicriic, Utiizlcrsity of Colorndo, Boirlder, CO 80309 U S A

I propose a novel general principle for unsupervised learning of distributed nonredundant internal representations of input patterns. The principle is based on two opposing forces. For each representational unit there is an adaptive predictor which tries to predict the unit from the remaining units. In turn, each unit tries to react to the environment such that it minimizes its predictability. This encourages each unit to filter “abstract concepts” out of the environmental input such that these concepts are statistically independent of those on which the other units focus. I discuss various simple yet potentially powerful implementations of the principle that aim at finding binary factorial codes (Barlow e t al. 1989), i.e., codes where the probability of the occurrence of a particular input is simply the product of the probabilities of the corresponding code symbols. Such codes are potentially relevant for (1) segmentation tasks, (2) speeding u p supervised learning, and (3) novelty detection. Methods for finding factorial codes automatically implement Occam’s razor for finding codes using a minimal number of units. Unlike previous methods the novel principle has a potential for removing not only linear but also nonlinear output redundancy. Illustrative experiments show that algorithms based on the principle of predictability minimization are practically feasible. The final part of this paper describes an entirely local algorithm that has a potential for learning unique representations of extended input sequences. 1 Introduction

Consider a perceptual system being exposed to an unknown environment. The system has some kind of internal ”state” to represent external events. We consider the general case where the state is an n-dimensional distributed representation yl’ (a vector of real-valued or binary code symbols) created in response to the pth input vector XI’. An ambitious and potentially powerful objective of unsupervised learning is to represent the environment such that the various parts of the representation are statistically imleyeiideiit of each other. In other words, we would like to have methods for decomposing the environment into entities that belong together and do not have much to do with other Nritrnl Comprtntioii 4, 863-879 (1992)


864

Jurgen Schmidhuber

entities' ("learning to divide and conquer"). This notion is captured by the concept of "discovering factorial codes" (Barlow et a / . 1989). The aim of "factorial coding" is the following: Given the statistical properties of the inputs from the environment, find iizverfible internal representations such that the occurrence of the ith code symbol y; is independent of any of the others. Such representations are called factorial because they have a remarkable and unique property: The probability of the occurrence of a particular input is simply the product of the probabilities of the corresponding code symbols. Among the advantages of factorial codes are as follows: 1. "Optimal" ii7piit se,gi~eiztation. An efficient method for discovering mutually independent features would have consequences for many segmentation tasks. For instance, consider the case where the inputs are given by retinal images of various objects with mutually independent positions. At any given time, the activations of nearby "pixels" caused by the same object are statistically correlated. Therefore a factorial code would not represent them by different parts of the internal storage. Instead, a factorial code could be created by finding input transformations corresponding to the abstract concept of "object position": Positions of different objects should be represented by different code symbols.

2. Speeding zip siipervised Ieamiizg. As Becker (1991) observes, if a representation with uncorrelated components is used as the input to a higher level linear supervised learning network, then the Hessian of the supervised network's error function is diagonal, thus allowing efficient methods for speeding up learning (note that statistical independence is a stronger criterion than the mere absence of statistical correlation). Nonlinear networks ought to profit as well.

3. Occam's razor. Any method for finding factorial codes automatically implements Occam's razor which prefers simpler models over more complex ones, where simplicity is defined as the number of storage cells necessary to represent the environment in a factorial fashion. If there are more storage cells than necessary to implement a factorial code, then the independence criterion is met by letting all superfluous units emit constant values in response to all inputs. This implies storage efficiency, as well as a potential for better generalization capabilities. 4. Novelty detection. As Barlow et al. (1989) point out, with factorial codes the detection of dependence between two symbols indicates hitherto undiscovered associations. 'The G-Max algorithm (Pearlmutter and Hinton 1986) aims at a related objective: It tries to discover features that account for input redundancy. G-Max, however, is designed for single output units only.

Learning Factorial Codes

865

Barlow et al. do not present an efficient general method for finding factorial codes (but they propose a few sequential ”nonneural” heuristic methods). Existing “neural” methods for decreasing output redundancy (e.g., Linsker 1988; Zemel and Hinton 1991; Oja 1989; Sanger 1989; Foldikk 1990; Rubner and Schulten 1990; Silva and Almeida 1991) are mostly restricted to the linear case and do not aim at the ambitious general goal of statistical independence. In addition, some of these methods require gaussian assumptions about the input and output signals, as well as the explicit calculation of the derivative of the determinant of the output covariance matrix (Shannon 1948). The main contribution of this paper is a simple but general “neural” architecture (plus the appropriate objective functions) for finding factorial codes. I would not be surprised, however, if the general problem of finding factorial codes turned out to be NP-hard. In that case, gradient-based procedures as described herein could not be expected to always find factorial codes. The paper at hand focuses on the novel basic principle without trying to provide solutions for the old problem of local maxima. Also, the purpose of this article is not to compare the performance of algorithms based on the novel principle to the performance of existing sequential ”nonneural” heuristic methods (Barlow et a!. 1989). The toyexperiments described below are merely for illustrative purposes. 2 Formulating the Problem

Let us assume iz different adaptive input processing represeiztatioiinl modides that see the same single input at a time. The output of each module can be implemented as a set of neuron-like units. Throughout this paper I focus on the simplest case: One output unit (also called a representational unit) per module. The ith module (or unit) produces an output value yr E [0.1] in response to the current external input vector XI’. In what follows, P ( A ) denotes the probability of event A , P ( A I €3) denotes the conditional probability of event A given B, y, denotes the mean of the activations of unit i, and E denotes the expectation operator. The methods described in this paper are primarily devoted to finding binary or at least quasibinary codes. Each code symbol participating in a quasibinary code is either 0 or 1 in response to a given input pattern or emits a constant value in response to every input pattern. Therefore, binary codes are a special case of quasibinary codes. Most of our quasibinary codes will be created by starting out from real-valued codes. Recall that there are three criteria that a binary factorial code must fulfill: 1 . The binary criterion: Each code-symbol should be either 1 or 0 in response to a given input pattern.

Jiirgen Schmidhuber

866

2. The invertibility criterion: It must be possible to reconstruct the input from the code. In cases where the environment is too complex (or too noisy) to be fully coded into limited internal representations (i.e., in the case of binary codes where there are more than 2dim(y) input patterns), we want to relax the invertibility criterion. In that case, we still want the internal representations to convey maximal information about the inputs. The focus of this paper, however, is on situations like the ones studied in Barlow et al. (1989): Noisefree environments and sufficient representational capacity in the representational units. In the latter case, reversibility is equivalent to Infomax as per Linsker (1988). 3. The independence criterion: The occurrence of each code symbol ought to be independent of all other code symbols. If the binary criterion is fulfilled, then we may rewrite the independence criterion by requiring that

The latter condition implies that y, does not depend on {yk. k # i}. In other words, E(y, I {yk. k # i}) is computable from a constant. Note that with real-valued codes the criterion Vi : E(y, I {yk. k # i}) = E(y,) does not necessarily imply that the yk are independent. 3 The Basic Principle and Architecture

For each representational unit i there corresponds an adaptive predictor P;,which, in general, is nonlinear. With the pth input pattern xp, Pi’s input is the concatenation of the outputs of all units k # i. Pi’s onedimensional output is trained to equal the expectation E(y, I ( 6 , k # i}). It is well known that this can be achieved by letting P; minimize2

6

(3.1)

With the help of the n predictors one can define various objective functions for the representational modules to enforce the three criteria listed above (see Sections 4 and 5). Common to these methods is that all units are trained to take on values that minimize mutual predictability via the predictors: Each unit tries to extract features from the environment such that no combination of n - 1 units conveys information about the remaining unit. In other words, no combination of n - 1 units should allow better predictions of the remaining unit than a prediction based on 2Cross-entropyis another objective function for achieving the same goal. In the experiments, however, the conventional mean squared error based function 3.1 led to satisfactory results.


867

a constant. I call this the principle of intrarepresetitational predictability miniinizatioiz or, somewhat shorter, the principle of predictability minimization. A major novel aspect of this principle that makes it different from previous work is that it uses adaptive submodules (the predictors) to define the objective functions for the subjects of interest, namely, the representational units themselves. Following the principle of predictability minimization, each representational module tries to use the statistical properties of the environment to protect itself from being predictable. This forces each representational module to focus on aspects of the environment that are independent of environmental properties upon which the other modules focus. 4 Objective Functions for the Three Criteria

Sections 4.1, 4.2, and 4.3 provide objective functions for the three criteria. Sections 4.4, 4.5, and 4.6 describe various combinations of these objective functions. Section 4.7 hints at a parameter tuning problem. A way to overcome it (my preferred method for implementing predictability minimization) is presented in Section 5. 4.1 An Error Function for the Independence Criterion. For the sake of argument, let us assume that at all times each PI is as good as it can be, meaning that PI always predicts the expectation of yl conditioned on the outputs of the other modules, E(y, I {y!. k # i } ) . (In practice, the predictors will have to be retrained continually.) In the case of quasibinary codes the following objective function H is zero if the independence criterion is met:

H

=

1 -Ex [Pr 2 1

-

yl]’

(4.1)

P

This term for mutual predictability minimization aims at making the outputs independent-similar to the goal of a term for maximizing the determinant of the covariance matrix under gaussian assumptions (Linsker 1988). The latter method, however, tends to remove only linear predictability, while the former can remove nonlinear predictability as well (even without gaussian assumptions), due to possible nonlinearities learnable by nonlinear predictors. 4.2 An Objective Function for the Binary Criterion. A well-known objective function V for enforcing binary codes is given by

Maximizing this term encourages each unit to take on binary values. The contribution of each unit i is maximized if E(yi) is as close to 0.5

868

Jiirgen Schmidhuber

as possible. This implies maximal entropy for unit i under the binary constraint, i.e., i wants to become a binary unit that conveys maximal information about its input. 4.3 An Error Function for the Invertibility Criterion. The following is a simple, well-known method for enforcing invertibility: With pattern p , a reconstructor module receives the concatenation of all as an input and is trained to emit as an output the reconstruction zp of the external input xp. The basic structure is an auto encoder. The auto encoder’s objective function, to be minimized, is defined as

4.4 Combining Error Terms. A straightforward way of combining V, I, and H is to maximize the total objective function

where a, j3, and y are positive constants determining the relative weighting of the opposing error terms. Maximization of 4.3 tends to force the representational units to take on binary values that maximize independence in addition to minimizing the reconstruction error.3 4.5 Removing the Variance Term: Real-Valued Codes. If with a specific application we want to make use of the representational capacity of real-valued codes and if we are satisfied with decorrelated (instead of independent) representational units, then we might remove the V-term from 4.3 by setting a = 0. In this case, we want to minimize

Note that with real-valued units the invertibility criterion theoretically can be achieved with a single unit. In that case, the independence criterion would force all other units to take on constant values in response to all input patterns. In noisy environments, however, it may turn out to be advantageous to code the input into more than one representational unit. This has already been noted by Linsker (1988) in the context of his Infomax principle. 30ne might think of using Lagrangian multipliers (instead of arbitrary a,0, y) to rigidly enforce constraintssuch as independence. However, to use them the constraints would have to be simultaneously satisfiable. Except for special input distributions this seems to be unlikely (see also Section 4.7).


869

4.6 Removing the Global Invertibility Term. Theoretically it is sufficient to do without the auto encoder and set d = 0 in 4.3. In this case, we simply want to maximize

T=flV-yH The H-term counteracts the possibility that different (near-)binary units convey the same information about the input. Setting , j = 0 means to maximize information locally for each unit while at the same time trying to force each unit to focus on different pieces of information from the environment. Unlike with autoassociators, there is I Z O global invertibility term. Note that this method seemingly works diametrically opposite to the sequential, heuristic, non-neural methods described by Barlow et nl. (1989), where the sum of bit entropies is minimized instead of being maximized. How can both methods pursue the same goal? One may put it this way: Among all invertible codes, Barlow et a / . try to find those closest to something similar to the independence criterion. In contrast, among all codes fulfilling the independence criterion (ensured by sufficiently strong ?), the above methods try to find the invertible ones. 4.7 A Disadvantage of the Above Methods. Note that a factorial code causes nonmaximal V and therefore nonmaximal T for all methods with (1 > 0 except for rare cases (such as if there are 2” equally probable different input patterns). This means that with a given problem there is some need for parameter tuning of the relative weighting factors, due to the possibility that the various constraints may not be satisfiable simultaneously (see footnote 3). The method in the next section avoids this necessity for parameter tuning by replacing the term for variance maximization by a predictor-based term for conditioned variance maximization.

5 Local Conditioned Variance Maximization This is the author’s preferred method for implementing the principle of predictability minimization. It does not suffer from the parameter tuning problems involved with the V-term above. It is extremely straightforward and reveals a striking symmetry between opposing forces. Let us define

Recall that Py is supposed to be equal to E(y, I {yi.k # i } ) ,and note that 5.1 is formally equivalent to the sum of the objective functions E , of the predictors (equation 3.1).

Jurgen Schmidhuber

870

As in Section 4.6 we drop the global invertibility term and redefine the total objective function T to be maximized by the representational modules as

T

=

v c - yH

(5.2)

Conjecture. I conjecture that if there exists a quasibinary factorial code for a given pattern ensemble, then among all possible (real-valued or binary) codes T is maximized with a quasi-binary factorial code, even if 7 =o. If this conjecture is true, then we may forget about the H-term in 5.2 and simply write T = VC. In this case, all representational units simply try to maximize the same function that the predictors try to minimize, namely, Vc. In other words, this generates a symmetry between two forces that fight each other-one trying to predict and the other one trying to escape the predictions. The conjecture remains unproven for the general case. The long version of this paper, however, mathematically justifies the conjecture for certain special cases and provides some intuitive justification for the general case (Schmidhuber 1991). In addition, algorithms based solely on Vc-maximization performed well in the experiments to be described below. 6 "Neural" Implementation

In a realistic application, of course, it is implausible to assume that the errors of all P; are minimal at all times. After having modified the functions computing the internal representations, the Pi must be trained for some time to assure that they can adapt to the new situation. Each of the n predictors, the n representational modules, and the potentially available autoassociator can be implemented as a feedforward backpropagation network (e.g., Werbos 1974). There are two alternating passes-one for minimizing prediction errors and the other one for maximizing T . Here is an off-line version based on successive "epochs" (presentations of the whole ensemble of training patterns):

PASS 1 (minimizing prediction errors): Repeat for a "sufficient" number of training epochs: 1. For all p: 1.1. For all i: Compute fl. 1.2. For all i: Compute Pr. 2. Change each weight w of each P, according to

where rp is a positive constant learning rate.


871

PASS 2 (minimizing predictability): 2. For all p: 2.1. For all i: Compute yr. 2.2. For all i: Compute Py. 2.3. If an autoassociator is involved, compute zp.

2 . Change each weight u of each representational moduleaccording to

where 7/R is a positive constant learning rate. The weights of the P, do not change during this pass, but all other weights do change. Note that PASS 2 requires backpropagation of error signals through the predictors (uiithout changing their zoeights) and then through their n 1 input units (which are the output units of the representatiorial modules) down to the weights of the representational modules. ~

The off-line version above is perhaps not as appealing as a more local procedure where computing time is distributed evenly between PASS 2 and PASS 1: An on-line version. An extreme on-line version does not sweep through the whole training ensemble before changing weights. Instead it processes the same single input pattern xp (randomly chosen according to the input distribution) in both PASS 1 and PASS 2 and immediately changes the weights of all involved networks simultaneously, according to the contribution of xp to the respective objective functions. Simultaneous updating of the representations and the predictors, however, introduces a potential for instabilities. Both the predictors and the representational modules perform gradient descent (or gradient ascent) in changing functions. Given a particular implementation of the basic principle, experiments are needed to find out how much on-line interaction is permittable. With the toy-experiments reported below, on-line learning did not cause major problems. It should be noted that if T = Vc = C,Ep, (Section 5 ) , then with a given input pattern we may compute the gradient of Vc with respect to both the predictor weights and the weights of the representation modules in a single pass. After this we may simply perform gradient descent in the predictor weights and gradient ascent in the remaining weights (it is just a matter of flipping signs). This was actually done in the experiments. Local maxima. Like all gradient ascent procedures, the method is subject to the problem of local maxima. A standard method for dealing with local maxima is to repeat the above algorithm with different weight initializations (using a fixed number nF of training epochs for each repetition) until a (near-)factorial code is found. Each repetition corresponds

872

Jiirgen Schmidhuber

to a local search around the point in weight space defined by the current weight initialization. Shared hidden unifs. It should be mentioned that some or all of the representational modules may share hidden units. The same holds for the predictors. Predictors sharing hidden units, however, will have to be updated sequentially: No representational unit may be used to predict its own activity.

7 Experiments

All the illustrative experiments described below are based on T as defined in Section 5, with y = 0. In other words, the representational units try to maximize the same objective function Vc that the predictors try to minimize. All representational modules and predictors were implemented as 3-layer backpropagation networks. All hidden and output units used logistic activation functions and were connected to a bias-unit with constant activation. Parameters such as learning rates and number of hidden units were not chosen to optimize performance-there was no systematic attempt to improve learning speed. Daniel Prelinger and Jeff Rink implemented on-line and off-line systems based on Section 6 [see details in Schmidhuber (1991) and Prelinger (1992)l. The purpose of this section, however, is not to compare on-line and off-line versions but to demonstrate that both can lead to satisfactory results. With the off-line version, the sufficient number of consecutive epochs in PASS 1 was taken to be 5. With the on-line system, at any given time, the same single input pattern was used in both PASS 1 and PASS 2. The learning rates of all predictors were 10 times higher than the learning rates of the representational modules. An additional modification for escaping certain cases of local minima was introduced (see Schmidhuber 1991; Prelinger 1992). The significance of nonlinearities. With many experiments it turned out that the inclusion of hidden units led to better performance. Assume that dim(y) = 3 and that there is an XOR-like relationship between the activations of the first two representational units and the third one. A linear predictor could not possibly detect this relationship. Therefore the representational modules could not be encouraged to remove the redundancy. The next subsections list some selected experiments with both the online and the off-line method. In what follows, the term "local input representation" means that there are dim(x) different binary inputs, each with only one nonzero bit. The term "distributed input representation" means that there are 2d'm(x) different binary inputs. With all experiments, a representational unit was considered to be binary if the absolute difference


873

between its possible activations and either the maximal or the minimal activation permitted by its activation function never exceeded 0.05. Local maxima. With some of the experiments, multiples of 10,000 training epochs were employed. In many cases, however, the representational units settled into a stable code long before the training phase was over (even if the code corresponded to a suboptimal solution). The repetitive method based on varying weight initializations (Section 6) sometimes allowed shorter overall learning times (using values T I E of the order of a few 1000). A high number of repetitions increases the probability that a factorial code is found. Again it should be emphasized, however, that learning speed and methods for dealing with local maxima are not the main objective of this paper.

7.1 Uniformly Distributed Inputs. With the experiments described in this subsection there are 2di"'(!/)different uniformly distributed input patterns. This means that the desired factorial codes are the full binary codes. In the case of a factorial code all predictors emit 0.5 in response to every input pattern (this makes all conditioned expectations equal to the unconditioned expectations). Experimenf 1. Off-line, dim(y) = 2, dim(x) = 4, local input representation, 3 hidden units per predictor, 4 hidden units shared among the representational modules. Ten test runs with 20,000 epochs for the representational modules were conducted. In 8 cases this was sufficient to find a binary (factorial) code. Experiment 2. On-line, dim(y) = 2, dim(x) = 2, distributed input representation, 2 hidden units per predictor, 4 hidden units shared among the representational modules. Ten test runs were conducted. Less than 3000 pattern presentations (equivalent to ca. 700 epochs) were always sufficient to find a binary factorial code. Experiment 3 . Off-line, dim(y) = 4,dim(x) = 16, local input representation (16 patterns), 3 hidden units per predictor, 16 hidden units shared among the representational modules. Ten test runs with 20,000 epochs for the representational modules were conducted. In 1 case the system found a n invertible factorial code. In 4 cases it created a near-factorial code with only 15 different output patterns in response to the 16 input patterns. In 3 cases it created only 13 different output patterns. In 2 cases it created only 12 different output patterns. Experiment 4. On-line, dim(y) = 4, dim(x) = 4, distributed input representation (16 patterns), 6 hidden units per predictor, 8 hidden units shared among the representational modules. Ten test runs were conducted. In all cases but one the system found a factorial code within less than 4000 pattern presentations (corresponding to less than 300 epochs).

Jiirgen Schmidhuber

874

7.2 Occam’s Razor at Work. The experiments in this section are meant to verify the effectiveness of Occam’s razor, mentioned in the introduction. It is interesting to note that with nonfactorial codes predictability minimization prefers to reduce the number of used units instead of minimizing the sum of bit-entropies as per Barlow et d. (1989). This can be seen by looking at an example described by Mitchison in the appendix of the paper of Barlow et al. This example shows a case where the minimal sum of bit-entropies can be achieved with an expansive local coding of the input. Local representations, however, maximize mutual predictability: With local representations, each unit can always be predicted from all the others. Predictability minimization tries to avoid this by creating nonlocal, nonexpansive codings.

Experiment 1 . Off-line, dim(y) = 3, dim(x) = 4, local input representation, 3 hidden units per predictor, 4 hidden units shared among the representational modules. Ten test runs with 10,000 epochs for the representational modules were conducted. In 7 cases the system found a binary factorial code: In the end, one of the output units always emitted a constant value. In the remaining 3 cases, the code was at least binary and invertible. Experimenf 2. Off-line, dim(y) = 4, dim(x) = 4, local input representation, 3 hidden units per predictor, 4 hidden units shared among the representational modules. Ten test runs with 10,000 epochs for the representational modules were conducted. In 5 cases the system found a binary factorial code: In the end, two of the output units always emitted a constant value. In the remaining cases, the code did not use the minimal number of output units but was at least binary and invertible. Experiment 3. On-line, dim(y) = 4, dim(x) = 2, distributed input representation, 2 hidden units per predictor, 4 hidden units shared among the representational modules. Ten test runs with 250,000 pattern presentations were conducted. This was sufficient to always find a quasibinary factorial code: In the end, two of the output units always emitted a constant value. In 7 out of 10 cases, less than 100,000 pattern presentations (corresponding to 25,000 epochs) were necessary. 7.3 Nonuniformly Distributed Inputs. The input ensemble considered in this subsection consists of four different patterns denoted by x“, xb, xc, and 9, respectively. The probabilities of the patterns were

P(f)=

1 9

-;

P(Xb) =

2 -, 9

P(x‘)

2 9

= -,

4 9

P ( 2 )= -

This ensemble allows for binary factorial codes, one of which is denoted by the following code F : y”

= (1, l)‘,

yb = (0,

yc = ( l , O ) T ,

y” = (0.0)’


875

With code F , the total objective function VCbecomes V [ = 2. A nonfactorial but invertible (information-preserving) code is given by

With code B, V C = 19/10, which is only 1/10 below V:. This already indicates that certain local maxima of the internal state's objective function may be very close to the global maxima. Experiment 1 . Off-line, dim(y) = 2, dim(x) = 2, distributed input representation with x" = ( O . O ) r , xb = ( O . l ) T , x' = (l.O)', 1"' = ( l . l ) r ,1 hidden unit per predictor, 2 hidden units shared among the representational modules. Ten test runs with 2000 epochs for the representational modules were conducted. Here one epoch consisted of the presentation of 9 patterns--x" was presented once, X' was presented twice, xL was presented twice, x" was presented four times. In 7 cases, the system found a global maximum corresponding to a factorial code. In the remaining cases the code was not invertible. Experiment 2 (Occam's Razor). As experiment 1, but with dim(y) = 3. In all but one of the 10 test runs the system developed a factorial code (including one unused unit). In the remaining test run the code was at least invertible. With local input representation and dim(x) = 4, dim(y) = 2, the success rate dropped below 50%. With dim(y) = 3, the system usually found invertible but rarely factorial codes. This reflects the fact that with certain input ensembles there is a trade-off between redundancy and invertibility: Superfluous degrees of freedom among the representational units may increase the probability that an information-preserving code is found, while at the same time decreasing the probability of finding an optimal factorial code.

8 Predictability Minimization and Time Let us now consider the case of input sequences. This section describes an entirely local method designed to find unambiguous, nonredundant, reduced sequence descriptions. The initial state vector yI'(0) is the same for all sequences p. The input at time t > 0 of sequence p is the concatenation xP(t) o yP(t - I) of the input x'(t) and the last internal state yP(t - 1). The output is yJ'(t) itself. We minimize and maximize essentially the same objective functions as described above. That is, for the ith module, which now needs recurrent connections to itself and the other modules, there is again an adaptive predictor P,, which need not be recurrent. P,'s input at time t is the concatenation of

Jiirgen Schmidhuber

876

the outputs $ ( t ) of all units k # i. Pi’s one-dimensional output Py(t) is trained to equal the expectation of the output yi, given the outputs of the other units, E(y, 1 {yk(t).k # i } ) ,by defining Pi’s error function as

In addition, all units are trained to take on values that maximize

E=

C T(t) t

where T (t) is defined analogously to the respective stationary cases. The only way a unit can protect itself from being predictable from the other units is to store properties of the input sequences that are independent of aspects stored by the other units. In other words, this method will tend to throw away redundant temporal information much as the systems in Schmidhuber (1992a,b). For computing weight changes, each module looks back only to the last time step. In the on-line case, this implies an entirely local learning algorithm. Still, even when there are long time lags, the algorithm theoretically may learn unique representations of extended sequences-as can be seen by induction over the length of the longest training sequence: 1. y can learn unique representations of the beginnings ofall sequences. 2. Suppose all sequences and subsequences with length < k are uniquely rep-

resented in y. Then, by looking back only one time step at a time, y can learn unique representations of all subsequences with length k. The argument neglects all on-line effects and possible cross-talk. On-line variants of the system described above were implemented by Daniel Prelinger. Preliminary experiments were conducted with the resulting recurrent systems. These experiments demonstrated that there are entirely local sequence learning methods that allow for learning unique representations of all subsequences of nontrivial sequences (like a sequence consisting of 8 consecutive presentations of the same input pattern represented by the activation of a single input unit). Best results were obtained by introducing additional modifications (like other error functions than mean squared error for the representational modules). A future paper will elaborate on sequence learning by predictability minimization. 9 Concluding Remarks, Outlook

Although gradient methods based on predictability minimization cannot always be expected to find factorial codes-due to local minima and the


877

possibility that the problem of finding factorial codes may be NP-hardthey have a potential for rcmoving kinds of redundancy that previous linear methods were not able to remove. This holds even if the conjecture in Section 5 ultimately proves to be false. In many realistic cases, however, approximations of nonredundant codes should be satisfactory. It remains to be seen whether predictability minimization can be useful to find nearly nonredundant representations of real-world inputs. In ongoing research it is intended to apply the methods described herein to problems of unsupervised image segmentation (in the case of multiple objects), as well as to unsupervised sequence segmentation. There is a relationship of predictability minimization to more conventional “competitive” learning schemes: In a certain sense, units compete for representing certain ”abstract” transformations of the environmental input. The competition is not based on a physical ”neighborhood” criterion but on mutual predictability. Unlike most previous schemes based on ”winner-take-all” networks, output representations formed by predictability minimization may have multiple “winners,” as long as they stand for independent features extracted from the environment. One might speculate about whether the brain uses a similar principle based on ”representational neurons” trying to escape the predictions of ”predictor neurons.” Sincc the principle allows for entirely local sequence learning algorithms (in space and time), it might be biologically more plausible than methods such as ”backpropagation through time” etc. Predictability minimimtion also might be useful in cases where different representational modules see different inputs. For instance, if a binary feature of one input ”patch” is predictable from features extracted from neighboring ”patches,” then representations formed by predictability minimization would tend to not use additional storage cells for representing the feature. The paper at hand adopts a general viewpoint on predictability minimization by focusing on the general case of nonlinear nets. In some cases, however, it might be desirable to restrict the computational power of the representational modules and/or the predictors by making them linear or semilinear. For instance, a hierarchical system with successive stages of computationally limited modules may be useful for reflecting the hierarchical structure of certain environments. Among the additional topics covered by the longer version of this report (Schmidhuber 1991) are general remarks on unsupervised learning and information-theoretic aspects, a “neural” approach to finding binary factorial codes 7oithoict using predictors, implementations of predictability minimization using binary stochastic units, the relationship of predictability minimization to recent sequence chunking methods, and combinations of goal-directed learning and unsupervised predictability minimization.

878

Jurgen Schmidhuber

Acknowledgments Thanks to Daniel Prelinger a n d Jeff Rink for implementing and testing the algorithms. Thanks to Mike Mozer, Daniel Prelinger, Radford Neal, Luis Almeida, Peter Dayan, Sue Becker, Rich Zemel, and Clayton McMillan for valuable comments and suggestions that helped to improve the paper. This research was supported by NSF PYI award IRI-9058450, Grant 9021 from the James S. McDonnell Foundation, and DEC external research Grant 1250 to Michael C. Mozer.

References Barlow, H. B., Kaushal, T. P., and Mitchison, G. J. 1989. Finding minimum entropy codes. Neural Comp. 1, 412423. Becker, S. 1991. Unsupervised learning procedures for neural networks. Int. J. Neural Syst. 2(1 & 2), 17-33. Foldiiik, P. 1990. Forming sparse representations by local anti-hebbian learning. Biol. Cybernet. 64,165-170. Linsker, R. 1988. Self-organization in a perceptual network. ZEEE Comput. 21, 105-117. Oja, E. 1989. Neural networks, principal components, and subspaces. Znt. J. Neural Syst. 1(1), 61-68. Pearlmutter, B. A., and Hinton, G. E. 1986. G-maximization: An unsupervised learning procedure for discovering regularities. In Neural Networks for Computing: American lnstitute of Physics Conference Proceedings 252, J. S. Denker, ed., Volume 2, pp. 333-338. Morgan Kaufmann. Prelinger, D. Diploma thesis. 1992. Institut fur lnformatik, Technische Universitat Miinchen, in preparation. Rubner, J., and Schulten, K. 1990. Development of feature detectors by selforganization: A network model. Biol. Cybernet. 62, 193-199. Sanger, T. D. 1989. An optimality principle for unsupervised learning. In Advances in Neural Information Processing Systems I, D. s. Touretzky, ed., pp. 1119. Morgan Kaufmann, San Mateo, CA. Schmidhuber, J. H. 1991. Learning factorial codes by predictability minimization. Tech. Rep. CU-CS-565-91, Dept. of Comp. Sci., University of Colorado at Boulder, December. Schmidhuber, J. H. 1992a. Learning complex, extended sequences using the principle of history compression. Neural Comp. 4(2), 234-242. Schmidhuber, J. H. 1992b. Learning unambiguous reduced sequence descriptions. In Advances in Neural Information Processing Systems 4, J. E. Moody, S. J. Hanson, and R. P. Lippman, eds., pp. 291-298. Morgan Kaufmann, San Mateo, CA. Shannon, C. E. 1948. A mathematical theory of communication (part 111). Bell System Tech. J. XXVII, 623-656.


879

Silva, F. M., and Almeida, L. B. 1991. A distributed decorrelation algorithm. In Neural Netzuorks, Aduniices nrid Applications, Erol Gelenbe, ed. North-Holland, Amsterdam. Werbos, P. J. 1974. Beyond regression: New tools for prediction and analysis in the behavioral sciences. Ph.D. thesis, Harvard University. Zemel, R. S., and Hinton, G. E. 1991. Discovering viewpoint-invariant relationships that characterize objects. In Aduarices in Naira/ Iiiforinatioii Proccssirig Systeins3, D. S. Lippman, J. E. Moody, and D. S. Touretzky, eds., pp. 299-305. Morgan Kaufmann, San Mateo, CA.

Received 2 January 1992; accepted 14 April 1992.

This article has been cited by: 2. Ella Bingham, Ata Kabán, Mikael Fortelius. 2009. The aspect Bernoulli model: multiple causes of presences and absences. Pattern Analysis and Applications 12:1, 55-78. [CrossRef] 3. Alexander S. Klyubin, Daniel Polani, Chrystopher L. Nehaniv. 2007. Representations of Space and Time in the Maximization of Information Flow in the Perception-Action LoopRepresentations of Space and Time in the Maximization of Information Flow in the Perception-Action Loop. Neural Computation 19:9, 2387-2432. [Abstract] [PDF] [PDF Plus] 4. Neil Burgess, Francesca Cacucci, Colin Lever, John O'Keefe. 2005. Characterizing multiple independent behavioral correlates of cell firing in freely moving animals. Hippocampus 15:2, 149-153. [CrossRef] 5. N.N. Schraudolph. 2004. Gradient-Based Manipulation of Nonparametric Entropy Estimates. IEEE Transactions on Neural Networks 15:4, 828-837. [CrossRef] 6. M.A. Sanchez-Montanes, F.J. Corbacho. 2004. A New Information Processing Measure for Adaptive Complex Systems. IEEE Transactions on Neural Networks 15:4, 917-927. [CrossRef] 7. Sepp Hochreiter , Jürgen Schmidhuber . 1999. Feature Extraction Through LOCOCODEFeature Extraction Through LOCOCODE. Neural Computation 11:3, 679-714. [Abstract] [PDF] [PDF Plus] 8. Suzanna Becker, Mark Plumbley. 1996. Unsupervised neural network learning procedures for feature extraction and classification. Applied Intelligence 6:3, 185-203. [CrossRef] 9. Jürgen Schmidhuber, Martin Eldracher, Bernhard Foltin. 1996. Semilinear Predictability Minimization Produces Well-Known Feature DetectorsSemilinear Predictability Minimization Produces Well-Known Feature Detectors. Neural Computation 8:4, 773-786. [Abstract] [PDF] [PDF Plus] 10. Lucas Parra , Gustavo Deco , Stefan Miesbach . 1996. Statistical Independence and Novelty Detection with Information Preserving Nonlinear MapsStatistical Independence and Novelty Detection with Information Preserving Nonlinear Maps. Neural Computation 8:2, 260-269. [Abstract] [PDF] [PDF Plus] 11. J. Schmidhuber, S. Heil. 1996. Sequential neural text compression. IEEE Transactions on Neural Networks 7:1, 142-146. [CrossRef] 12. Anthony J. Bell , Terrence J. Sejnowski . 1995. An Information-Maximization Approach to Blind Separation and Blind DeconvolutionAn Information-Maximization Approach to Blind Separation and Blind Deconvolution. Neural Computation 7:6, 1129-1159. [Abstract] [PDF] [PDF Plus]

13. Peter Dayan, Richard S. Zemel. 1995. Competition and Multiple Cause ModelsCompetition and Multiple Cause Models. Neural Computation 7:3, 565-579. [Abstract] [PDF] [PDF Plus] 14. David J. Field . 1994. What Is the Goal of Sensory Coding?What Is the Goal of Sensory Coding?. Neural Computation 6:4, 559-601. [Abstract] [PDF] [PDF Plus] 15. Jürgen Schmidhuber , Daniel Prelinger . 1993. Discovering Predictable ClassificationsDiscovering Predictable Classifications. Neural Computation 5:4, 625-635. [Abstract] [PDF] [PDF Plus]

Communicated by Halbert White

How to Incorporate New Pattern Pairs without Having to Teach the Previously Acquired Pattern Pairs H. M. Wabgaonkar A. R. Stubberud Electrical and Computer Engineering Department, University of California, lrvine, California 92717 U S A

In this paper, we deal with the problem of associative memory synthesis. The particular issue that we wish to explore is the ability to store new input-output pattern pairs without having to modify the path weights corresponding to the already taught pattern pairs. The approach to the solution of this problem is via interpolation carried out in a Reproducing Kernel Hilbert Space setting. An orthogonalization procedure carried out on a properly chosen set of functions leads to the solution of the problem. 1 Introduction

Associative memories can be implemented conveniently by employing neural networks as function approximators (Cybenko 1989; Stinchcombe and White 1989) or function interpolators (Wabgaonkar and Stubberud 1990). The problem of discrete associative memory synthesis is as follows. Given a finite discrete set of input-output pairs {(xi.yi)}, where the input patterns xi are in E" (the n-dimensional Euclidean space), and the corresponding outputs yi are in R'; the problem is to find an interpolating function f such the / ( x i ) = y;, for all i. It can be shown that a three-layer neural network with an input layer, a single hidden layer, and an output layer of processing units with linear characteristics can be employed to solve the above problem (Cybenko 1989; David 1975). The required function as realized by the network can be written as

f(x)=Ca,g;(x) x

E

E"

(1.1)

I

where g; : €"- > R' represents the characteristic or the activation function of the ith processing unit in the hidden layer. If the number of pattern pairs is N, then N functions gi must be chosen such that the matrix G = k,(xi)] is of maximal rank. If this condition is satisfied, the vector a of the coefficients { a j } can be obtained uniquely. In this paper, we are concerned with the problem of storing successive data pairs without having to alter the weights corresponding to the Neural Computation 4,880-887 (1992) @ 1992 Massachusetts Institute of Technology

Incorporation of New Pattern Pairs

881

previously stored data pairs for the net modeled by 1. Specifically, we require that the previously acquired coefficients a,, i = 1, . . . . K should not have to be modified for storing ”future pairs” ( x ~ ” . Y K + ~(xK+’. ) yK+2) etc. Imposing the interpolation conditions f ( x ’ ) = yI, i = 1 . . . . . K , from 1.1 we see that our objective will be achieved if the matrix G is lower-triangular, i.e.,

g , ( x ’ ) = 0.

for j > i

(1.2)

A convenient framework to solve the discrete associative memory synthesis problem by interpolation is the Reproducing Kernel Hilbert Space (RKHS) framework. Intuitively, our approach to the solution of the particular data storage problem is as follows. We define a kernel K ( x .y), x . y in E”, (called the reproducing kernel for reasons to be mentioned later), , (.. .) such that g,(x) = K ( x . k ) and K(x1, X I ) = ( K ( x .X I ) . K ( x . x ’ ) ) ~where indicates an appropriate inner product and the subscript x indicates the inner product to be taken with respect to the variable x . This allows us to convert the ”spatial” orthogonality condition of 1.2 into an orthogonality condition on the functions K ( . . x j ) , j = 1,.. . . N. Any standard orthogonalization technique such as the Gram-Schmidt procedure can then be applied to achieve the desired orthogonality of the successive neuronal activation functions 8,. In Section 2, we begin by briefly reviewing some of the relevant properties of RKHS. The approaches to the general interpolation problem and to the particular pattern storage problem are presented. A specific example of an RKHS presented in Section 3 is the Generalized Fock Space (GFS) framework originally proposed by de Figuereido and co-workers (1980, 1990). Finally, we present an example illustrating the proposed technique. 2 RKHS: A Brief Overview

In this section, we present a very brief overview of RKHS theory relevant to our work. The proofs of the general results related to RKHS can be found in Aronszajn (1950), Davis (1975), and de Figueiredo and Dwyer (1980); they will not be presented here. Next, we indicated how the required associative memory synthesis can be carried out via interpolation within an RKHS framework. This sets the stage for the particular orthogonality problem, which is also solved in this section. 2.1 Preliminaries. Let D be a compact subset (containing the points x i ) of E”; let H be a Hilbert space of continuous real-valued functions defined on D. Let (.. .) indicate a real-valued, symmetric inner product with respect to which H is complete. Since D is compact and the functions g of H are continuous, the functions g are also bounded. Therefore, each

H. M. Wabgaonkar and A. R. Stubberud

882

of the linear point evaluation functions L given by

L ( g ) = g(x). x E D.

vg E H

(2.1)

is also bounded. Hence, by the Frechet-Riesz Theorem, for each bounded linear functional L on H , there exists an element (a function) 1 of H , called the representer of the function L, given by

L ( g ) = (g. 1).

vg E H

(2.2)

Since each of the point evaluation functions on H is bounded, there exists a function of two variables K(x.y), with x . y in D, such that (Aronszajn 1950)

1. for each fixed y in D, K(x.y) considered as a function of x is in H; 2. for every function g in H, and for every y in D, the following reproducing property holds:

Due to property (2) above, K(...) is called (the) reproducing kernel (r.k.1 of the RKHS H. If a space has an r.k., it is unique. Also, in our case, K(...) happens to be a real-valued, symmetric function, i.e.,

K(x.y)

= K(y.x)

(2.4)

We note in passing that, although not usually recognized in this way, the familiar sinc function given by

is an r.k. for the space S of finite-energy, Fourier-transformable, band-limited signals (Yao 1967). This kernel is used for reconstructing a signal element of S from its samples. 2.2 Interpolation. The significant advantage of using an RKHS framework is the ability to extract the representers 1 of the functional L- the representer 1 of the functional L is given by

UY)

= L*[K(X.Y)l

(2.6)

where the subscript x indicates that L operates on K as if K were a function of the variable x alone. In particular, for the point evaluation functionals Li given by

L;V)=f(x');

xi in D

(2.7)


883

the corresponding representers I, are given by

l , ( x ) = K(x. x')

(2.8)

We intend to use these representer functions I, for interpolation. Specifiy,) such cally, we assume that we are given N input-output data pairs (1'. that x' are in D; x' are distinct. The RKHS H with the r.k. K is formed out of real-valued continuous functions on D. We are now in a position to make the following claim:

Claim: Under tkeassuinptions 011 D and H , tke matrix G = [l,(x')]is ofrna.uirna/ rank; where ll are tke representers gizien by 2.8. For the proof of the above claim refer to Wabgaonkar and Stubberud (1990) and de Figueiredo (1990). At this stage, it follows from 2.8 that the required function can be written as (see 1.1) N

f(x)=

ELI, K ( x . x');

xED

(2.9)

I=1

In view of the above claim, the coefficient vector a can be determined uniquely so as to satisfy the interpolation constraints stated earlier: f(x')

= y,.

i

=

1.. . . . N

(2.10)

For a properly chosen RKHS H , it is possible to choose (as we indicate in the next section),

K ( x .x') = h ( X T X 1 )

(2.11)

for a suitably defined function k : R' - > R'. With this choice, equation 2.9 is in complete conformity with equation 1.1 and with the three-layer neural net structure described in the previous section. 2.3 Orthogonality. Using equation 2.9, the orthogonality condition of 1.2 becomes

K(x1.x') = 0 for j > i

(2.12)

But, due to the property of the r.k., we have

K(x1. x ' )

=

( K ( x .x'). K ( x . XI)),

(2.13)

Therefore, we must have K ( . . x l ) orthogonal to K(..x') for j > i. This immediately suggests the use of the Gram-Schmidt procedure for orthogonalization. Recall that I, = K(..x'). Then, the following equations (Kreyszig 1978) describe the required procedure:

(2.14) (2.15) (2.16)

H. M. Wabgaonkar and A. R. Stubberud

884

The functions {g,}satisfy the required orthogonality property. Of course, the additional price to be paid for this property is the extra computation needed for the orthogonalization. From equations 2.12 and 2.15, it follows that llhI1l2= W')

(2.17)

This considerably simplifies the norm computations which are perhaps the most demanding of the computations in the algorithm. Finally, we note that the above procedure is data dependent. The possibility of employing symbolic computations for implementing the above procedure is currently being explored. 3 An RKHS Example

In this section, we consider a specific example of an RKHS called the Generalized Fock Space (GFS) due to de Figueiredo and co-workers (1980). Its application to the synthesis of the Hamming Net was proposed recently (de Figueiredo 1990). Here, we present it in a somewhat different context of the discrete associative memory synthesis problem. Finally, we give a numerical example. 3.1 The GFS. Suppose, the domain D (to which the inputs xi belong) is a compact subset of E", such that

D = { X E E";

1 1 ~ 1 2. 4 Concluding Remarks

In this paper, we have presented a constructive approach to the solution of the discrete associative memory synthesis problem. The synthesis is carried out in an RKHS framework. A very desirable feature of this type of the setting is the ability to store the given successive pattern pairs


887

without having to retrain the previously acquired path weights. This attribute of the proposed approach is also demonstrated. Given a multivariate function defined on a nondiscrete domain, the choice of the projection directions a n d their number are not obvious. O u r current work is directed toward the extension of this approach to the storage of such a function defined on a nondiscrete domain.

References Aroszajn, N. 1950. Theory of reproducing kernels. Trmis. A m Math. SOC. 68, 337-404. Cybenko, G. 1989. Approximation by superposition of a sigmoidal function. CSRD Report no. 856, University of Illinois, Urbana, IL. Also published in Math. Control, Signals and Systems, 1989. Davis, I? J. 1975. Ziiterpolaticiri mid Approxiination, Dover, New York. de Figueiredo, R. J. I? 1990. An optimal matching score-net for pattern classification. Proc. IJCNN, San Diego, CA, pp. 909-916. de Figueiredo, R. J. I?, and Dwyer, T. A. W. 1980. A best approximation framework and implementation for simulation of large-scale non-linear systems. ZEEE Trails. Circuits S y s t . 27, 1005-1014. Kreyszig, E. 1978. Iiitroductory Firrictiorznl Arzahysis zclith Ayplicntioris, pp. 157-1 59. Wiley, New York. Stinchcombe, M., and White, H. 1989. Universal approximation using feedforward networks with non-sigmoid hidden layer activation functions. Proc. International Joint Conference on Neural Networks (IJCNN), Washington, D.C., pp. 613-617. Stubberud, A. R., and Wabgaonkar, H. 1990. Synthesis of discrete associative memories by multivariate interpolation. Tech. Rep. ECE Dept., Univ. of California-Irvine, Irvine, CA. Wabgaonkar, H., and Stubherud, A. 1990. Approximation and estimation techniques for neural networks. IEEE 1990 Conference on Decision and Control, Honolulu, December. Yao, K. 1967. Applications of reproducing kernal Hilbert spaces bandlimited signal models. Iiifo. Coil. 11, 429-444. ~

~

~~

Received 20 December 1990, ‘iccepted 19 March 1992.


Communicated by Geoffrey Hinton

Local Learning Algorithms Leon Bottou Vladimir Vapnik AT&T Bell Laboratories, Holmdel, N J 07733 U S A

Very rarely are training data evenly distributed in the input space. Local learning algorithms attempt to locally adjust the capacity of the training system to the properties of the training set in each area of the input space. The family of local learning algorithms contains known methods, like the k-nearest neighbors method (kNN) or the radial basis function networks (RBF), as well as new algorithms. A single analysis models some aspects of these algorithms. In particular, it suggests that neither kNN or RBF, nor nonlocal classifiers, achieve the best compromise between locality and capacity. A careful control of these parameters in a simple local learning algorithm has provided a performance breakthrough for an optical character recognition problem. Both the error rate and the rejection performance have been significantly improved. 1 Introduction

Here is a simple local algorithm: For each testing pattern, (1) select the few training examples located in the vicinity of the testing pattern, ( 2 ) train a neural network with only these few examples, and (3) apply the resulting network to the

testing pattern. Such an algorithm looks both slow and stupid. Indeed, only a small part of the available training examples is used to train our network. Empirical evidence however defeats this analysis. With proper settings, this simple algorithm improves significantly the performance of our best optical character recognition networks. A few years ago, V. Vapnik devised a theoretical analysis for such local algorithms, briefly discussed in Vapnik (1992). This analysis introduces a new component, named locality, in the well-known trade-off between the capacity of the learning system and the number of available examples. This paper attempts to explain, and demonstrates, that such an algorithm might be very efficient for certain tasks, and that its underlying ideas might be used with profit. The voluminous equations of the theoretical analysis, however, will not be discussed in this paper. Their complexity, mostly related to the imperfection of the current generalization theories, would introduce unnecessary noise in this discussion. Neural Computation 4, 888-900 (1992)


Local Learning Algorithms

889

In Section 2, we show that handling rejections in some pattern recognition problems requires different properties for a learning device in different areas of the pattern space. In Section 3, we present the idea of a local learning algorithm, discuss related approaches, and discuss the impact of the locality parameter on the generalization performance. In Section 4, we demonstrate the effectiveness of a local learning algorithm on a real size optical character recognition task. 2 Rejection

An ideal isolated character recognition system always assigns the right symbolic class to a character image. But a real recognizer might commit an error or perform a rejection. Errors are very expensive to correct. A zipcode recognition system, for instance, might erroneously send a parcel to the other end of the world. Therefore, the system. should reject a pattern whenever the classification cannot be achieved with enough confidence. Having this pattern processed by a human being is usually less expensive than fixing an error. Selecting a proper confidence threshold reduces both the cost of handling the rejected patterns and the cost of correcting the remaining errors. The quality of such a pattern recognition system is measured by its rejection curve (cf. Fig. 4). This curve displays the possible compromises between the number of rejected patterns and the numbers of remaining errors. Two very different situations reduce the classification confidence and might cause a rejection (cf. Fig. 1). 0

0

Patterns might well be ambiguous. For instance, certain people write their "1" like other people write their "7." This cause of rejection is inherent to the problem. Ambiguities arise because important information, like the contextual knowledge of the writing style, is not provided as input to the system. Knowing exactly the probability distribution of the patterns and classes would not eliminate such rejections. A pattern would still be rejected if its most probable class does not win by a sufficient margin. Patterns might be unrelated to the training data used for defining the classifier. For instance, many atypical writing styles are not represented in the training database. Low probability areas of the pattern space are poorly represented in the training set. The decision boundary of our classifier in such areas is a mere side effect of the training algorithm. These boundaries are just irrelevant. This second cause of rejection is a direct consequence of the finite nature of the training set. Knowing exactly the probability distri-

890

Leon Bottou and Vladimir Vapnik

Rejection by ambiguity. The class of this pattern cannot be determined with enough confidence

Rejection by lack of data. We have not enough examples to find a decision boundary in that area with enough confidence.

Figure 1: This is a piece of an imaginary pattern space. Gray and black circles are examples of two classes. Thin lines are the actual Bayesian decision boundary between these two classes. Both crosses represent rejected patterns. bution of the pattern and classes would reveal the exact Bayesian decision boundaries everywhere. This latter cause of rejection has rarely been studied in the literature.' Its mere definition involves nonasymptotical statistics closely related to the generalization phenomenon. A high capacity learning system is able to model accurately and with high confidence the parts of the decision boundary that are well described by the training examples. In these areas, both rejections and misclassifications are rare. The same system however produces unreliable high confidence decision boundaries in the poorly sampled areas of the pattern space: Rejections are rare, but misclassifications are frequent. Alternatively, a low capacity learning system builds low confidence boundaries in the poorly sampled areas of the pattern space. This system rejects more atypical patterns, but reduces the number of misclassifications. Unfortunately, such a device performs poorly in the well sampled areas. Unable to take profit of the abundant data, it builds poor decision boundaries, and rejects almost everything, because everything looks too ambiguous. 'A Bayesian approach has been suggested in Denker and Le Cun (1991)for estimating error-bars on the outputs of a learning system. This useful information affects the interpretation of the outputs, and might improve the rejection performance, as suggested by our reviewer. This method could as well improve the rejection of local algorithms.


891

In fact, different properties of the learning algorithm are required in different areas of the input space. In other words, the ”local capacity” of a learning device should match the density of training examples. 3 Local Learning Algorithms

It is now generally admitted that the generalization performance is affected by a global trade-off between the number of training examples and the capacity of the learning system. Various parameters monotonically control the capacity of a learning system (Guyon et al. 19921, including architectural parameters (e.g., the number of hidden units), preprocessing parameters (e.g., the amount of smoothing), or regularization parameters (e.g., the weight decay). The best generalization is achieved for some optimal values of these capacity controI parameters, which depend on the size of the training set. This fact holds for rejection or for raw performance, in the case of pattern recognition, regression, or density estimation tasks. Whenever the distribution of patterns in the input space is uneven, a proper local adjustment of the capacity can significantly improve the overall performance. Such a local adjustment requires the introduction of capacity control parameters whose impact is limited to individual regions of the pattern space. Here are two ways to introduce such parameters: 0

0

Our experiment (cf. Section 4) illustrates the first solution. For each testing pattern, we train a learning system with the training examples located in a small neighborhood around the testing pattern. Then we apply this trained system to the testing pattern itself. The parameters of the locally trained system d e facto affect only the capacity of the global system in the small neighborhood defined around each testing pattern. We shall show below that the k-nearest-neighbor (kNN) algorithm is just a particular case of this approach. Like kNN, such systems are very slow. The recognition speed is penalized by the selection of the closest training patterns and by the execution of the training algorithm of the local learning system. In the second solution, the structure of the learning device ensures that each parameter affects the capacity of the system in a small neighborhood only. For example, we might use a separate weight decay per radial basis function (RBF) unit in an RBF network. Each weight decay parameter affects the capacity of the network only locally. Similarly, architectural choices in a modular network (Jacobs ct nl. 1991) have a local impact on the capacity of the global system. Since such a system is trained at once, the recognition time is not affected by the local nature of the learning procedure.

892


In the next subsections, we shall describe a general statement for local learning algorithms, discuss related approaches, and explain how the locality parameter affects the generalization performance. 3.1 Weighted Cost Function. We present here a general statement for local learning algorithms. Let us define J[y.fil,(x)] as the loss incurred when the network gives an answer fw(x) for the input vector x, when the actual answer is y. The capacity control parameters y directly or indirectly define a subset W, of the weight space (Guyon et al. 1992). A nonlocal algorithm searches this subset for the weight vector w*(y), which minimizes the empirical average of the loss over a training set (XI.yl ). . . . . (XI.yl). (3.1)

For each neighborhood defined around a point xo, a local algorithm searches for a weight vector w*(xo. b. y), which minimizes a weighted empirical average of the loss over the training set.

The weighting coefficients are defined by a kernel K ( x - x o , b ) of width b centered on point xo. Various kernels might be used, including square kernels and smooth kernels (cf. Fig. 2).

Figure 2 A square kernel selects only those examples located in a specific neighborhood. A smooth kernel gives a different weight to all examples, according to their position with respect to the kernel. The locality parameter, b, measures the "size" of the neighborhood.

-


893

Since a separate minimization is performed in each neighborhood, the capacity control parameters -, and the kernel width b can be adjusted separately for each neighborhood. 3.2 Related Approaches. This formulation allows for many variations concerning the class of function f n , ( x ) ,the number of neighborhoods, the shape of the kernels K(x Xg.b ) , the scaling laws for the kernel width b, or the parameters 7. Selecting the class of constant functions with respect to the input vectors x , and using a quadratic loss J(y. y) = (y-5)' leads to several popular algorithms, like the kNN method or the RBF networks. In each specific neighborhood, such algorithms try to find a constant approximation 5' of the desired output: ~

y' = Arg Min ii

1 ' K ( x - xo. b ) (yl - 5)' 1 I= I

-

1

(3.3)

For instance, consider a pattern recognition problem. If the pattern XI belongs to the tzth class, the nth coefficient of the corresponding desired output GI is equal to 1; the others coefficients are equal to 0. 0

For each testing pattern xo, we consider a square kernel whose width is adjusted to contain exactly k examples. The optimum y of (3.3) is the mean of the desired outputs of the k closest patterns. Its highest coefficient, then, corresponds to the most represented class among the k closest patterns to xo. This is the k-izenrest neighbors (kNN) algorithm.

If we use a smooth kernel instead of a square kernel, minimizing (3.3) for each testing pattern xo computes estimates of the posterior probability of the classes. This is the Pnrzen windows algorithm. 0

We consider now R fixed neighborhoods, defined by the centers x: and the standard deviation b,+ of their gaussian kernels. Minimizing (3.3) in each neighborhood computes the weighted average y: of the desired values of the training examples. To evaluate the output yGl&,al(x)of the complete system for an input pattern x , we merge these weighted averages according to the values of the X kernels on x. (3.4) This is a radial basis functions (RBF) network (Broomhead and Lowe 1988; Moody and Darken 1989).


894

3.3 Locality and Capacity. Theoretical methods developed for nonlocal algorithms apply to each local optimization. In particular, the best value for the capacity control parameters (Guyon et al. 1992) depend on the number of training examples. In the context of a local algorithm, however, the effective number of training examples is modulated by the width b of the kernels. For instance, a square kernel selects from the training set a subset, whose cardinality depends on the local density of the training set and on the kernel width b. The classical trade-off between capacity and number of examples must be reinterpreted as a trade-off between capacity and locality. If we increase the locality by reducing b, we implicitly reduce the effective number of training examples available for training the local system. kNN and the RBF networks use small kernels and a class of constant functions. There is no reason, however, to believe that the best results are obtained with such a low capacity device. Conversely, big multilayer networks are nonlocal (b = m), but have a high capacity. Modular networks (Jacobs et al. 1991) sit somewhat between these two extremes. The kernel functions are embodied by a "gating network," which selects or combines the outputs of the modules, according to the input data and sometimes the outputs of the modules themselves. Another phenomenon makes the situation slightly more complex: The capacity control parameters can be adjusted separately in each neighborhood. This local adjustment is more accurate when the kernel width is small. On the other hand, there is little to adjust in a very low capacity device.

4 Experiments

This section discusses experiments of a simple local learning algorithm on a real size pattern recognition problem. Comparisons have been carried out (1) with a backpropagation network, and (2) with the kNN and Parzen windows algorithms. 0

0

A backpropagation network is a nonlocal algorithm, with a comparatively high capacity. Comparison (1) shows that introducing locality and reducing the capacity improves the resulting performance of such a network. A kNN classifier is an extremely local algorithm, with a very low capacity. Comparison (2) shows that reducing the locality and increasing the capacity again improves the resulting performance.

4.1 A Simple Local Learning Algorithm. We have implemented a simple local algorithm:


895

For each testing pattern so,a linear classifier is trained on the k closest , the Euclidian distance. This training examples, ( x I . y l1 . . . . . ( x ~ . y k ) for trained network then is applied to the testing pattern xg. The effective number of examples, k, is much smaller than the number of weights in the linear classifier. Therefore, a strong weight decay *, is required to reduce the capacity of the linear classifier. A weight decay, however, pulls the weights toward some arbitrary origin. For isotropy reasons, the origin of the input space is translated on the testing pattern xo,by subtracting xo from all the selected training patterns. This also has the advantage of reducing the eigenvalue spread of the hessian matrix. The training procedure computes the explicit minimum of a quadratic cost incorporating a weight decay term. The positive weight decay , ; ensures that the required matrix inversion is possible.

Since the new coordinate system is centered on the testing pattern, the output of the network on this testing pattern is equal to the bias vector. The highest output determines which class is selected for pattern xo. If, however, the difference between the highest output and the second highest output is less than a certain threshold, the pattern is rejected. A simple heuristic rule controls the capacity versus locality trade-off within each neighborhood. It adjusts the locality and leaves the capacity constant. In other words, the same value of the weight decay parameter 7 is used in all neighborhoods. The locality is usually controlled by a kernel size b, which should increase when the density of training example decreases. In fact, selecting the k closest training examples is equivalent to having a square kernel whose size is somewhat adjusted according to the local density of training examples. We just use the same value for k in all neighborhoods. Although this system is extremely inefficient, it implements a wide range of compromises between locality and capacity, according to the values of only two parameters, a locality parameter k and a regularization parameter 7 . We have found that this quality was attractive for a first experiment. 4.2 Results and Comparisons. We trained several systems on the same training set composed of normalized 16 x 16 images of 7291 handwritten digits and 2549 printed digits. Performance has been measured on a test set of 2007 handwritten digits. The same database was used in Le Cun et a/. (1990). Table 1 gives the raw error and the rejection at 1% error for various systems. The ”raw error” is the percentage of misclassifications when no rejection is performed. The “rejection for 1% error” is the percentage

896

LPon Bottou and Vladimir Vapnik

0I 2 3 4 5 6 7 89 n....

.

rpys v:

10 clauiflcm'oa lvliu

Figure 3: "LeNet": Layers i, ii, iii, and iv compute 192 features. Layer v performs the classification. In this paper, we replace layer v by a local algorithm. Table 1: Results on an Optical Character Recognition Task.

Raw error Human LeNet kNN Parzen Local

(on segmented digits) zz 2.5% (on segmented digits) 5.1% (on LeNet features) 5.1% (on LeNet features) 4.7% (on LeNet features) 3.3%

Rejection for 1%error n.a. 9.6% n.a. 10.8% 6.2%

of rejected pattern when the rejection threshold is adjusted to allow less than 1% misclassification on the remaining patterns. The 2.5% human performance on the segmented and preprocessed digits provides a reference point (Sackinger and Bromley, personal communication). The nickname "LeNet" designates the network described in Le Cun et al. (1990). This five-layer network performs both the feature extraction and the classification of 16 x 16 images of single handwritten digits. Four successive convolutional layers extract 192 translation invariant features; a last layer of 10 fully connected units performs the classification (cf. Fig. 3 ) . This network has achieved 5.1% error and 9.6% rejection for 1% error.* 'This is slightly worse than the 4.6%raw error and 9% rejection for 1%error reported in Le Cun et nl. (1990). This is due (1) to a slightly different definition of the rejection performance (1%error on the remaining patterns vs. 1% error total), and (2) to a more robust preprocessing code.


897

3j5 4 4

t \\

..

1

0 0

2I 2

4,\-%-,6 4 6

8 8

10 12 14 16 18 20 22 24 10 12 14 16 18 20 22 24

Rejection (96)

Figure 4: Punting curve. This curve shows the error rates at various rejection rates for plain LeNet (dashed curve) and our simple local algorithm operating on the features computed by LeNet (plain curve).

The 192 features computed by LeNet are used as inputs to three other pattern recognition systems. These systems can be viewed as replacements for the last layer of LeNet. Therefore, results can be directly compared. The best kNN performance, 5.1%raw error, was obtained by using the three closest neighbors only. These few neighbors allow no meaningful rejection strategy. The Parzen system is similar to k".We just replace the square kernel by a gaussian kernel, whose standard deviation is half the distance of the fourth closest pattern. Several variations have been tested; we report the best results only: 4.7% raw error and 10.8% rejection for 1% error. Finally, we have tested the simple local learning algorithm described above, using the 200 closest patterns and a weight decay of 0.01. This weight decay is enough to train our 1920 weights using 200 patterns. At this time, both the 3.3% raw error and the 6.2% rejection rate for 1%error were the best performances reached on this data set. A derivation reported in the appendix shows that this performance improvement is statistically significant; Figure 4 compares the rejection curve of the local system with the rejection curve of "LeNet." The local system performs better for all values of the threshold. At 17% rejection, the single remaining error is a mislabeled pattern.

898


2.5

Decay

Figure 5: Evolutionsof the raw error for the local system around the best values of k and of the weight decay 7 . (In fact, the decay axis displays the product rk.)

With a proper adjustment of its locality and capacity parameters, this simple algorithm outperforms both (1) a nonlocal algorithm (i.e., the last layer of LeNet), and (2) two extremely local algorithms (i.e., K" or Parzen windows). Figure 5 shows how the raw error of the local system changes around the best values of k and of the weight decay 7. Finally, no significant performance changes have been obtained by using smooth kernels or fancy heuristic for controlling the kernel width and the weight decay.

4.3 Recognition Speed. This simple system, however, spends 50 sec for recognizing a single digit. Training a network for each testing pattern is certainly not a practical approach to the optical character recognition problem. In Section 3, however, we have presented two solutions for building local learning algorithms. We have deliberately chosen to implement the simpler one, which leads to very slow recognizers. We could as well design systems based on our second solution, i.e., using a network structure that allows a local control of the capacity of the system. Such a system would be slightly more complex to handle, but would have a much shorter recognition time.


899

5 Conclusion No particular architectural change is responsible for this performance breakthrough. In fact, this system is linear, and replaces a linear decision layer in LeNet. The change concerns the training procedure. The performance improvement simply results from a better control of the basic trade-offs involved in the learning process. Although much remains to be understood about learning, we have some practical and theoretical knowledge of these basic trade-offs. Understanding how these trade-offs affect a specific learning problem often allows us to take profit from their properties for practical applications. Local learning algorithms are just a successful example of this strategy.

Appendix: Confidence Intervals This section presents the derivations establishing the significance of the results presented in this paper. We first derive a nonasymptotical formula for computing confidence when comparing two classifiers on a same test set of N independent examples. Each classifier defines certain decision boundaries in the pattern space. It is enough to compare our classifiers on only those cases where one classifier is right and the other one is wrong. Let us call pl and p 2 the conditional probabilities of error of each classifier, given that one classifier only gives a wrong answer. Similarly, let us define H I and 112 as the numbers of errors that each classifier makes which the other classifier classifies correctly, and n12 as the number of common errors. According to the large number inequality (2.7) in Hoeffding (1963),

Furthermore, if we name v1 and y the measured error rates on our test set, we have n l - n2

=

(nl + n12)

-

(n2

+ n12) = N ( Y I

-

v2)

(A.2)

By solving for f when the right-hand side of inequality (A.1) is 1 - r / , we can compute the minimum difference v1 - 112, which ensures that p1 - p2 is larger than 0 with probability 1 - 11. Since comparing p1 and p2 is enough to decide which classifier is better, the following result is valid: If

then classifier 2 is better than classifier 1 with probability 1 - I / . In our case, all systems achieve less than 5.1% error on a test set of size N = 2007. The quantity nl + n2 is thus smaller than 10.2% of N,

900


probably by a large margin. If we choose 71 = 5%, we get a minimal significative error difference of 1.2%. Measuring the actual value of nl n2, would further reduce this margin. The significance of the results presented in this paper, however, is established without such a refinement.

+

Acknowledgments We wish to thank Larry Jackel's group at Bell Labs for their continuous support and useful comments. We are especially grateful to Yann Le Cun for providing networks and databases, to E. Sackinger and J. Bromley for providing the human performance results.

References Broomhead, D. S., and Lowe, D. 1988. Multivariate functional interpolation and adaptive networks. Complex Syst. 2, 321-355. Denker, J. S., and Le Cun, Y. 1991. Transforming neural-net output levels to probability distributions. In Advances in Neural Information Processing Systems 3 (NIPS*90), Lippmann, R., Moody, J., and Touretzky, D., eds. Morgan Kaufmann, Denver. Guyon, I., Vapnik, V. N., Boser, 8. E., Bottou, L., and Solla, S. A. 1992. Structural risk minimization for character recognition. In Advances in Neural Information Processing Systems, Vol. 4. Morgan Kaufmann, Denver. Hoeffding, W. 1963. Probability inequalities for sums of bounded random variables. J. Am. Statist. Assoc. 58, 13-30. Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. 1991. Adaptive mixture of local experts. Neural Comp. 3(1), 79-87. Le Cun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. 1990. Handwritten digit recognition with a backpropagation network. In Advances in Neural Information Processing Systems, D. Touretzky, ed., Vol. 2. Morgan Kaufmann, Denver. Moody, J., and Darken, C. J. 1989. Fast learning in networks of locally-tuned processing units. Neural Comp. 1(2), 281-294. Vapnik, V. N. 1992. Principles of risk minimization for learning theory. In Advances in Neural Information Processing Systems, Vol. 4. Morgan Kaufmann, Denver. To appear.

Received 19 February 1992; accepted 3 April 1992.

This article has been cited by: 2. Nicola Segata, Enrico Blanzieri, Sarah Jane Delany, Pádraig Cunningham. 2010. Noise reduction for instance-based learning with a local maximal margin approach. Journal of Intelligent Information Systems 35:2, 301-331. [CrossRef] 3. Nicola Segata, Enrico Blanzieri. 2010. Operators for transforming kernels into quasi-local kernels that improve SVM accuracy. Journal of Intelligent Information Systems . [CrossRef] 4. Filip Ponulak, Andrzej Kasiński. 2010. Supervised Learning in Spiking Neural Networks with ReSuMe: Sequence Learning, Classification, and Spike ShiftingSupervised Learning in Spiking Neural Networks with ReSuMe: Sequence Learning, Classification, and Spike Shifting. Neural Computation 22:2, 467-510. [Abstract] [Full Text] [PDF] [PDF Plus] 5. Petr Kadlec, Bogdan Gabrys. 2010. Local learning-based adaptive soft sensor for catalyst activation prediction. AIChE Journal n/a-n/a. [CrossRef] 6. Petr Kadlec, Bogdan Gabrys. 2009. Architecture for development of adaptive on-line prediction models. Memetic Computing 1:4, 241-269. [CrossRef] 7. Isaac Chairez. 2009. Wavelet Differential Neural Network Observer. IEEE Transactions on Neural Networks 20:9, 1439-1449. [CrossRef] 8. G. Polzlbauer, T. Lidy, A. Rauber. 2008. Decision Manifolds—A Supervised Learning Algorithm Based on Self-Organization. IEEE Transactions on Neural Networks 19:9, 1518-1530. [CrossRef] 9. Chen-Chia Chuang. 2007. Fuzzy Weighted Support Vector Regression With a Fuzzy Partition. IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) 37:3, 630-640. [CrossRef] 10. Carlotta Domeniconi, Dimitrios Gunopulos, Sheng Ma, Bojun Yan, Muna Al-Razgan, Dimitris Papadopoulos. 2007. Locally adaptive metrics for clustering high dimensional data. Data Mining and Knowledge Discovery 14:1, 63-97. [CrossRef] 11. J. Cho, J.C. Principe, D. Erdogmus, M.A. Motter. 2006. Modeling and Inverse Controller Design for an Unmanned Aerial Vehicle Based on the Self-Organizing Map. IEEE Transactions on Neural Networks 17:2, 445-460. [CrossRef] 12. R. Kulhavy. 2003. A developer's perspective of a decision support system. IEEE Control Systems Magazine 23:6, 40-49. [CrossRef] 13. C. Domeniconi, Jing Peng, D. Gunopulos. 2002. Locally adaptive metric nearest-neighbor classification. IEEE Transactions on Pattern Analysis and Machine Intelligence 24:9, 1281-1285. [CrossRef] 14. Agnes Schumann, Niels Wessel, Alexander Schirdewan, Karl Josef Osterziel, Andreas Voss. 2002. Potential of feature selection methods in heart rate variability analysis for the classification of different cardiovascular diseases. Statistics in Medicine 21:15, 2225-2242. [CrossRef]

15. N. Giusti, A. Sperduti. 2002. Theoretical and experimental analysis of a two-stage system for classification. IEEE Transactions on Pattern Analysis and Machine Intelligence 24:7, 893-904. [CrossRef] 16. D. Spiegel, T. Sudkamp. 2002. Employing locality in the evolutionary generation of fuzzy rule bases. IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) 32:3, 296-305. [CrossRef] 17. N.K. Kasabov, Qun Song. 2002. DENFIS: dynamic evolving neural-fuzzy inference system and its application for time-series prediction. IEEE Transactions on Fuzzy Systems 10:2, 144-154. [CrossRef] 18. E.A. Rying, G.L. Bilbro, Jye-Chyi Lu. 2002. Focused local learning with wavelet neural networks. IEEE Transactions on Neural Networks 13:2, 304-319. [CrossRef] 19. Patrice Y. Simard, Yann A. Le Cun, John S. Denker, Bernard Victorri. 2000. Transformation invariance in pattern recognition: Tangent distance and propagation. International Journal of Imaging Systems and Technology 11:3, 181-197. [CrossRef] 20. Rudolf Kulhavý, Petya Ivanova. 1999. Quo vadis, Bayesian identification?. International Journal of Adaptive Control and Signal Processing 13:6, 469-485. [CrossRef] 21. J. Manuel Torres Moreno, Mirta B. Gordon. 1998. Efficient Adaptive Learning for Classification Tasks with Binary UnitsEfficient Adaptive Learning for Classification Tasks with Binary Units. Neural Computation 10:4, 1007-1030. [Abstract] [PDF] [PDF Plus] 22. Arnaud Buhot, Juan-Manuel Torres Moreno, Mirta B. Gordon. 1997. Finite size scaling of the Bayesian perceptron. Physical Review E 55:6, 7434-7440. [CrossRef] 23. E. Alpaydin, M.I. Jordan. 1996. Local linear perceptrons for classification. IEEE Transactions on Neural Networks 7:3, 788-794. [CrossRef] 24. R. Rovatti, R. Ragazzoni, Zs. M. Kovàcs, R. Guerrieri. 1995. Adaptive Voting Rules for k-Nearest Neighbors ClassifiersAdaptive Voting Rules for k-Nearest Neighbors Classifiers. Neural Computation 7:3, 594-605. [Abstract] [PDF] [PDF Plus] 25. Federico Girosi , Michael Jones , Tomaso Poggio . 1995. Regularization Theory and Neural Networks ArchitecturesRegularization Theory and Neural Networks Architectures. Neural Computation 7:2, 219-269. [Abstract] [PDF] [PDF Plus] 26. David G. Lowe . 1995. Similarity Metric Learning for a Variable-Kernel ClassifierSimilarity Metric Learning for a Variable-Kernel Classifier. Neural Computation 7:1, 72-85. [Abstract] [PDF] [PDF Plus] 27. V. Vapnik , L. Bottou . 1993. Local Algorithms for Pattern Recognition and Dependencies EstimationLocal Algorithms for Pattern Recognition and Dependencies Estimation. Neural Computation 5:6, 893-909. [Abstract] [PDF] [PDF Plus]

-~

Communicated by Dana Ballard

Object Discrimination Based on Depth-from-Occlusion Leif H. Finkel Paul Sajda Department of Bioengineering orid Znsfitirte of Neiirologicnl Sciences, University of Pennsylvania, Piiil~idelphia,PA 29104-6392 USA We present a model of how objects can be visually discriminated based on the extraction of depth-from-occlusion. Object discrimination requires consideration of both the binding problem and the problem of segmentation. We propose that the visual system binds contours and surfaces by identifying "proto-objects"-compact regions bounded by contours. Proto-objects can then be linked into larger structures. The model is simulated by a system of interconnected neural networks. The networks have biologically motivated architectures and utilize a distributed representation of depth. We present simulations that demonstrate three robust psychophysical properties of the system. The networks are able to stratify multiple occluding objects in a complex scene into separate depth planes. They bind the contours and surfaces of occluded objects (for example, if a tree branch partially occludes the moon, the two "half-moons" are bound into a single object). Finally, the model accounts for human perceptions of illusory contour stimuli. 1 Introduction

___

In order to discriminate objects in the visual world, the nervous system must solve two fundamental problems: binding and segmentation. The binding problem (Barlow 1981) addresses how the attributes of an object-shape, color, motion, depth-are linked to create an individual object. Segmentation deals with the converse problem of how separate objects are distinguished. These two problems have been studied from the perspectives of both computational neuroscience (Marr 1982; Grossberg and Mingolla 1985; T. Poggio et al. 1988; Finkel and Edelman 1989) and machine vision (Guznian 1968; Rosenfeld 1988; Aloimonos and Shulman 1989; Fisher 1989). However, previous studies have not addressed what we consider to be the central issue: how does the visual system define an object-i.e., what constitutes a "thing." Object discrimination occurs at an intermediate stage of the transformation between two-dimensional (2D) image intensity values and visual recognition, and in general, depends on cues from multiple visual modalities. To simplify the problem, we restrict ourselves to discrimiNeltrril Cornpututiorz 4, 901-921 (1992)


Leif H. Finkel and Paul Sajda

902

nation based solely on occlusion relationships. In a typical visual scene, multiple objects may occlude one another. When this occurs, it creates a perceptual dilemma-to which of the two overlapping surfaces does the common border belong? If the border is, in fact, an occlusion border, then it belongs to the occluding object. This identification results in a stratification of the two objects in depth and a de facto discrimination of the objects. Consider the case of a tree branch crossing the face of the moon. We perceive the branch as closer and the moon more distant, but in addition, the two "half-moons" are perceptually linked into one object. The visual system supplies a virtual representation of the occluded contours and surfaces in a process Kanizsa (1979) has called "amodal completion." With this example in mind, we propose that the visual system identifies "proto-objects" and determines which proto-objects, if any, should be linked into objects. For present purposes, a proto-object is defined as a compact region surrounded by a closed, piecewise continuous contour and located at a certain distance from the viewer. The contour can be closed on itself, or more commonly, it can be closed by termination on other contours. We will demonstrate how a system of interconnected, physiologically based neural networks can identify proto-objects, link them into objects, and stratify the objects in depth. The networks operate, largely in parallel, to carry out the following interdependent processes: 0

discriminate edges

0

segment and bind contours

0

identify proto-objects (i.e., bind contours and surfaces)

0

identify possible occlusion boundaries

0

stratify occluding objects into different depth planes

0

attempt to link proto-objects into objects

0

influence earlier steps (e.g., contour binding) by results of later steps (e.g., object linkage).

The constructed networks implement these processes using a relatively small number of neural mechanisms (such as detecting curvature, and determining which surface is inside a closed contour). A few of the mechanisms used are similar to those of previous proposals (Grossberg and Mingolla 1985; Finkel and Edelman 1989; Fisher 1989). But our particular choice of mechanisms is constrained by two considerations. First, we utilize a distributed representation of depth-this is based on the example of how disparity is represented in the visual cortex (G. Poggio et al. 1988; Lehky and Sejnowski 1990). The relative depth of a particular object is represented by the relative activation of corresponding units in a foreground and background map. Second, as indicated above, we make

Object Discrimination

903

extensive use of feedback (reentrant) connections from higher level networks to those at lower levels-this is particularly important in linking proto-objects. For example, once a higher level network has determined an occlusion relationship it can modify the way in which an earlier network binds contours to surfaces. Any model of visual occlusion must be able to explain the perception of illusory (subjective) contours, since these illusions arise from artificially arranged cues to occlusion (Gregory 1972). The proposed model can account for the majority of such illusions. In fact, the ability to link contours in the foreground and background corresponds, respectively, to the processes of modal and amodal completion hypothesized by Kanizsa (1979). The present proposal differs from previous neural models of illusory contour generation (Ullman 1977; Grossberg and Mingolla 1985; von der Heydt et al. 1989; Finkel and Edelman 1989) in that it generates illusory objects-not just the contours. The difference is critical: a network which generates responses to the three sides of the Kanizsa triangle, for example, is not representing a triangle (the object) per se. To represent the triangle it is necessary to link these three contours into a single entity, to know which side of the contour is the inside, to represent the surface of the triangle, to know something about the properties of the surface (its depth, color, texture, etc.), and finally to bind all these attributes into a whole. This is clearly a much more difficult problem. We will describe, however, a simple model for how such a process might be carried out by a set of interconnected neural networks, and present the results of simulations that test the ability of the system on a range of normal and illusory scenes.

2 Implementation

Simulations of the model were conducted using the NEXUS Neural Simulator (Sajda and Finkel 1992). NEXUS is an interactive simulator designed for modeling multiple interconnected neural maps. The simulator allows considerable flexibility in specifying neuronal properties and neural architectures. The present simulations feature an interconnected system composed of 10 different network architectures, each of which contains one or more topographically organized arrays of 64 x 64 units. Two types of neuronal units are used. Standard neuronal units carry out a linear weighted summation of their excitatory and inhibitory inputs, and outputs are determined by a sigmoidal function between voltage and firing rate. NEXUS also allows the use of more complex units called PGN (programmable generalized neural) units that execute arbitrary functions or algorithms. A single PGN unit can emulate the function of a small circuit or assembly of standard units. PGN units are particularly useful in situations in which an intensive computation is being performed but the anatomical and physiological details of how the operation is performed in uiuo are unknown. Alterna-


904

Dlscrlmlnate Edges, Line

I DatermineConHnuily and Closure and DynamicallyBind Contour

I Determbe II Contour is Surrounded

Cumtun

t Determine RelaHve Depth of Object Using Distributed Representstlonin FOREGROUND and BACKGROUND Maps

Figure 1: Major processing stages in the model. Each process is carried out by one or more networks. Following early visual stages, information flows through two largely parallel pathways-one concerned with identifying and linking occlusion boundaries (left side) and another concerned with stratifying objects in depth (right side). Networks are multiply interconnected and note the presence of the two reentrant feedback pathways. tively, PGN units can be used to carry out functions in a time-efficient manner; for example, to implement a one-step winner-take-all algorithm. The PGN units used in the present simulations can all be replaced with circuits composed of standard neuronal units, but this incurs a dramatic increase in processing time and memory allocation with minimal changes in functional behavior at the system level. No learning is involved in the network dynamics. The model is intended to correspond to visual processing during a brief interval (less than 200 msec following stimulus presentation), and the interpretation of even complex scenes requires only a few cycles of network activity. The details of network construction will be described elsewhere; we will focus here on the processes performed and the theoretical issues behind the mechanisms. 3 Construction of the Model

The model consists of a number of stages as indicated in Figure 1. The first stage of early visual processing involves networks specialized for the


905

detection of edges, line orientation, and line terminations (endstopping). As Ramachandran (1987) observed, the visual system must distinguish several different types of edges: we are concerned here with the distinction between edges due to surface discontinuities (transitions between different surfaces) and those due to surface markings (textures, stray lines, etc.). Only the former can be occlusion boundaries. The visual system utilizes several modalities to classify types of edges; we restrict ourselves to a single process carried out by the second processing stage, a network that determines which segments belong to which contours and whether the contours are closed. When two contours cross each other, forming an "X" junction, there are several possible perceptual interpretations of which arms of the "X" should be joined. Our networks carry out the simple rule that discontinuities should be minimized-i.e., lines and curves should continue as straight (or with as much the same curvature) as possible. Similar assumptions underlie previous models (Ullman 1977), and this notion is in accord with psychophysical findings that discontinuities contain more information than continuous segments (Attneave 1954; Resnikoff 1989). We are thus minimizing the amount of self-generated information. We employ a simple sequential process to determine whether a contour is closed-each unit on a closed contour requires that at least two of its nearest neighboring units also be on the contour. It is computationally difficult to determine closure in parallel. We speculate that, iiz uiuo, the process is carried out by a combination of endstopped units and largereceptive field cells arranged in an architecture similar to that described in Area 17 (Rockland and Lund 1982; Mitchison and Crick 1982; Gilbert and Wiesel 1989). Once closure is determined, it is computationally efficient for the units involved to be identified with a "tag." Several of the higher level processes discussed below require that units responding to the same contour be distinguishable from those responding to different contours. There are several possible physiological mechanisms that could subserve such a tag-one possible mechanism is phase-locked firing (Gray and Singer 1989; Eckhorn et al. 1988). We have implemented this contour binding tag through the use of PGN units (Section 2), which are capable of representing several distinct tags. It must be emphasized, however, that the model is compatible with a number of possible physiological mechanisms. Closed contours are a necessary condition to identify a proto-object, but sufficiency requires two additional components. As shown in Figure 1, the remaining determinations are carried out in parallel. One stage is concerned with determining on which side of the contour the figure lies, i.e., distinguishing inside from outside. The problem can be alternatively posed as determining which surface "owns" the contour (Koffka 1935; Nakayama and Shimojo 1990). This is a nontrivial problem that, in general, requires global information about the figure. The classic example is the spiral (Minsky and Papert 1969; Sejnowski and Hinton 1987) in


906

/Op7/ I

1

contour

Figure 2: Neural circuit for determining direction of figure (inside vs. outside). Hypothetical visual stimulus consists of two closed contours (bold curves). The central unit of 3 x 3 array (shown below) determines the local orientation of the contour. Surrounding units represent possible directions (indicated by arrows) of the inside of the figure relative to the contour. All surrounding units are inhibited (black circles) except for the two units located perpendicular to local orientation of the contour. Units receive inputs from the contour binding map via dendrites that spread out in a stellate configuration, as indicated by clustered arrows (dendrites extend over long distances in map). Units inside the figure will receive more inputs than those located outside the figure. The two uninhibited units compete in a winner-take-all interaction. Note that inputs from separate objects are not confused due to the tags generated in the contour binding map. which it is impossible to determine whether a point is inside or outside based on only local information. The mechanism w e employ, as shown in Figure 2, is based on the following simple observation. Suppose a unit projects its dendrites in a stellate configuration and that the dendrites are activated by units responding to a contour. Then units located inside a closed contour will receive more activation than units located outside the contour. A winner-take-all interaction between the two units will


907

concavity

I “‘1;hq I

“I

T

3

I I I

Figure 3: Primary cues for occlusion. Tag junctions (shown in the inset) signal a local discontinuity between occluding and occluded contours. Concave regions and surrounded contours suggest occlusion, but are not as reliable indicators as tag junctions. Additional cues such as accretion/deletion of texture (not considered here) are used i l l 77iuo.

determine which is more strongly activated, and hence which is inside the figure. As shown in Figure 2, it is advantageous to limit this competition to the two units that are located at positions perpendicular to the local orientation of the contour. As will be shown below (see Figs. 5-71, this network is quite efficient at locating the interior of figures. It also demonstrates deficiencies similar to those of human perception-for example, it cannot distinguish the inside from the outside of a spiral. The mechanism depends on the contour binding carried out above. Each unit only considers inputs with the appropriate tag-in this way, inputs from separate contours in the scene are not confused. Identification of a proto-object also requires that the relative depth of the surface be determined. This is carried out chiefly through the use of tag junctions. As shown in Figure 3, a tag junction is formed by the termination of an occluded boundary on an occluding boundary. Tag junctions generally correspond to T-junctions in the image, however, they arive from discontinuities in the binding tags and are therfore associated with surface discontinuities as well. Note that tag junctions are identified at an intermediate stage in the sytem (see Fig. 1) and are not

908


constructed directly from end-stopped units in early vision. This accords with the lack of physiological evidence for "junction" detectors in striate cortex. In this model, tag junctions serve as the major determinant of relative depth. At such junctions, there is a change in the binding (or ownership) of contours, and it is this change which produces the discontinuity in perceived depth. Depth is represented by the relative level of activity in two topographic maps (called foreground and background). The closest object maximally activates foreground units and minimally activates background units; the most distant object has the reverse values, and objects located at intermediate depths display intermediate values. The initial state of the two maps is such that all closed contours lie in the background plane. Depth values are then modified at tag junctions-contours corresponding to the head of the "T" are pushed toward the foreground. Since multiple objects can overlap, a contour can be both occluding and occluded-therefore, the relative depth of a contour is determined in a type of push-pull process in which proto-objects are shuffled in depth. The contour binding tag is critical in this process in that all units with the same tag are pushed forward or backward together. (In the more general case of nonplanar objects, the alteration of depth values would depend on position along the contour.) Tag junctions arise in cases of partial occlusion; however, in some instances, a smaller object may actually lie directly in front of a larger object. In this case, which we call "surround" occlusion, the contour of the occluded object surrounds that of the occluding object. As shown in Figure 1, a separate process determines whether such a surround occlusion is present, and in the same manner as tag junctions, leads to a change in the representation of relative depth. The network mechanism for detecting surround occlusion is almost identical to that discussed above for determining the direction of figure (see Fig. 2). Note that a similar configuration of two concentric contours arises in the case of a "hole." The model is currently being extended to deal with such nonsimply connected objects. These processes-contour binding, determining direction of the figure, and determination of relative depth-define the proto-object. The remainder of the model is concerned with linking proto-objects into objects. The first step in this endeavor is to identify occlusion boundaries. Since occlusion boundaries are concave segments of contours, such segments must be detected (particularly, concave segments bounded by tag junctions). Although many machine vision algorithms exist for determining convexity, we have chosen to use a simple, neurally plausible mechanism: at each point of a contour, the direction of figure is compared to the direction of curvature [which is determined using endstopped units (Dobbins et al. 198711. In convex regions, the two directions are the same; in concave regions, the two directions are opposed. A simple AND mechanism can therefore identify the concave segments of the contours.


a

909

b

C

Figure 4: Linking of occluded contours. Three possible perceptual interpretations (below) of an occlusion configuration (above) are shown. Small arrows indicate direction of figure (inside/outside). Collinearity cannot be the sole criterion for linking occluded edges. Consistency in the direction of figure between linked objects rules out perception c. Once occlusion borders are identified, proto-objects can be linked by trying to extend, complete, or continue occluded segments. Linkage most commonly occurs between proto-objects in the background, i.e., between spatially separated occluded contours. For example, in Figure 3, the occluded contours which terminate at the two tag junctions can be linked to generate a virtual representation of the occluded segment. Since it is impossible to know exactly what the occluded segment looks like, and since it is not actually "perceived," we have chosen not to generate a representation of the occluded segment. Rather, a network link binds together the endpoints of the two tag junctions. In the case where multiple objects are occluded by a single object, the problem of which contours to link can become complex. As shown in Figure 4, one important constraint on this process is that the directions of figure be consistent between the two linked proto-objects. Another condition in which proto-objects can be linked involves the joining of occluding contours, i.e., of proto-objects in the foreground. This phenomenon occurs in our perception of illusory contours, for example, in the Kanizsa triangle (Kanizsa 1979) or when a gray disc is viewed against a background whose luminance changes in a smooth spatial gradient from black to white (Marr 1982; Shapley and Gordon 1987). In this case, a representation of the actual contour is generated. The conditions for linkage are that the two contours must be smoothly joined by a line or curve, and that the direction of figure be consistent (as in the case of occluded contours above).

910


The major difference between these two linking or completion processes is that contours generated in the foreground are perceived while those in the background are not. However, the same mechanisms are used in both cases. We have elected to segregate the foreground and background linking processes into separate networks for computational simplicity-it is possible, however, that in vivo a single population of units carries out both functions. Regardless of the implementation, the interaction between ongoing linking processes in the foreground and background is critical. Since these links are self-generated by the system (they do not exist in the physical world), they must be scrutinized to avoid false conjunctions. The most powerful check on these processes is their mutual consistencyan increased certainty of the occluded contour continuation being correct increases the confidence of the occluding contour continuation, and vice versa. For example, in the case of the Kanizsa triangle, the "pac-man"like figures can be completed to form complete circles by simply continuing the contour of the pac-man. The relative ease of completing the occluded contours, in turn, favors the construction of the illusory contours, which correspond to the continuations of the occluding contours. In fact, we believe that the interaction between these two processes determines the perceptual vividness of the illusion. The final steps in the process involve a recurrent feedback (or reentry, Finkel and Edelman 1989) from the networks that generate these links back to earlier stages so that the completed contours can be treated as real objects. Note that the occluded contours feedback to the contour binding stage, not to the line discrimination stage, since in this case, the link is virtual, and there is no generated line whose orientation, etc., can be determined. The feedback is particularly important for integrating the outputs of the two parallel paths. For example, once an occluding contour is generated, as in the illusory contours generated in the Kanizsa triangle, it creates a new tag junction (with the circular arc as the "tail" and the illusory contour as the "head" of the "T''). On the next iteration through the system, this tag junction is identified by networks in the other parallel path of the system (see Fig. 11, and is used to stratify the illusory contour in depth. 4 Results of Simulations

4.1 Linking Proto-objects. We present the results of three simulations which illustrate the ability of the system to discriminate objects. Figure 5 shows a visual scene that was presented to the system. The early networks discriminate the edges, lines, terminations, and junctions present in the scene. Figure 5A displays the contour binding tags assigned to different scene elements (on the first and fifth cycle of activity). Each box represents active units with a common tag, different boxes rep-


911

resent different tags, and the ordering of the boxes is arbitrary. Note that on the first cycle of activity, discontinuous segments of contours are given separate tags. These tags are changed by the fifth cycle as a result of feedback from the linking processes. Figure 5B shows the output of the directioiz offigure network, for a small portion of the input scene (near the horse’s head). The direction of the arrows indicates the direction of figure determined by the network. The correct direction of figure is determined in all cases: for the horse’s head, and for the horizontal and vertical posts of the fence. Once the direction of figure is identified, occluded contours can be linked (as in Fig. 4), and proto-objects combined into objects. This linkage is what changes the contour binding tags, so that after several cycles (Fig. 5A, right), separate tags are assigned to separate objects-the horse, the gate posts, the house, the sun. The presence of tag junctions (e.g., between the horse’s contour and the fence, between the house and the horse’s back) is used by the system to force various objects into different depth planes. The results of this process are displayed in Figure 5C, which plots the firing rate (percent of maximum) of units in the foreground network. The system has successfully stratified the fence, horse, house, and sun. The actual depth value determined for each object is somewhat arbitrary, and can vary depending on minor changes in the scene-the system is designed only to achieve the correct relative ordering, not absolute depth. Note that the horizontal and vertical posts of the fence are perceived at different depths-this is because of the tag junctions present between them; in fact, the two surfaces do lie at slightly different depths. In addition, there is no way to determine the relative depth of the two objects in the background, the house and the sun, because they bear no occlusion relationship to each other. Again, this conforms to human perceptions, e.g., the sun and the moon appear about the same distance away. The system thus appears to process occlusion information in a manner similar to human perception. 4.2 Gestalt Psychology of a Network. The system also displays a response consistent with human responses to a number of illusory stimuli. Figure 6 shows a stimulus, adapted from an example of Kanizsa (19791, which shows that preservation of local continuity in contours is more powerful than global symmetry in perception (this is contrary to classical Gestalt theory-eg., Koffka 1935). As shown in the middle panels, there are two possible perceptual interpretations of the contours-on the left, the two figures respect local continuity (this is the dominant human perception); on the right, the figures respect global symmetry. Figure 6A shows the contour binding tags assigned by the system to this stimulus, and Figure 6B shows the direction of figure that was determined. Both results indicate that the network makes the same perceptual interpretation as a human observer.

912


4.3 Occlusion Capture. The final simulation shows the ability of the system to generate illusory contours and to use illusory objects in a veridical fashion. The stimulus is, again, adapted from Kanizsa (1979), and shows a perceptually vivid, illusory white square in a field of black discs. The illusory square appears to be closer to the viewer than the background, and, in addition, the four discs that lie inside its borders also appear closer than the background (some viewers perceive the four internal discs to be even closer than the illusory square). This is an example of what we call ”occlusion capture,” an effect related to the capture phenomena involving motion, stereopsis, and other submodalities (Ramachandran and Cavanaugh 1985; Ramachandran 1986). In this case, the illusory square has “captured” the discs within its borders and they are thus pulled into the foreground. Figure 7A shows the contour binding tags after one (left) and three (right) cycles of activity. Each disc receives a separate tag. After the responses to illusory square are generated, the illusory contours are fed back to the contour binding network and given a common tag. Note that the edges of the discs occluded by the illusory square are now given the same tag as the square, not the same tags as the discs. The change in ”ownership” of the occluded edges of the discs is the critical step in defining the illusory square as an object. For example, Figure 7B shows the output of the direction o f f i p r e network after one and three cycles of activity. The large display shows that every disc is identified as an object with the inside of the disc correctly labeled in each case. The two insets focus on a portion of the display near the bottom left edge of the illusory square. At first, the system identifies the “L”shaped angular edge as belonging to the disc, and thus the direction of figure arrows point “inward.“ After three cycles of activity, this same “L”-shaped edge is identified as belonging to the illusory square, and thus the arrows now point toward the inside of the square, rather than the inside of the disc. This change in the ownership of the edge results from the discrimination of occlusion-the edge has been determined to

Figure 5: Facing p g e . Object discrimination and stratification in depth. Top panel shows a 64 x 64 input stimulus presented to the system. (A) Spatial histogram of the contour binding tags (each box shows units with common tag, different boxes represent different tags, and the order of the boxes is arbitrary). Initial tags shown on left; tags after five iterations shown on right. Note that linking of occluded contours has transformed proto-objects into objects. (B) Magnified view of a local section of the direction of figure network corresponding to portion of the image near horse‘s nose and crossing fence posts. Arrows indicate direction of inside of proto-objects as determined by network. (C) Relative depth of objects in scene as determined by the system. Plot of activity (% of maximum) of units in the foreground network after five iterations. Points with higher activity are ”perceived” as being relatively closer to the viewer.


913

be an occlusion border. The interconnected processing of the system then results in a change in the direction of figure and of the continuity tags associated with this edge. The illusory square is perceived as an object. Its four contours are bound together, the contours are bound to the internal surface, and the properties of the surface are identified.

B

C

914


Figure 7C displays the firing rate of units in the foreground map (as in 5C), thus showing the relative depths discriminated by the system. The discs are placed in the background, the illusory square and the four discs within its borders are located in the foreground. In this case, the depth cue which forces the internal discs to the foreground is not due to tag junctions, but rather to surround occlusion (see Figure 3 ) . Once the illusory square is generated, the contours of the discs inside the square are surrounded by that of the square. The fact that the contour is “illusory” is irrelevant; once responses are generated in the networks responsible for linking occluding contours and are then fed back to earlier networks, they are indistinguishable from responses to real contours in the periphery. Thus the system demonstrates occlusion capture corresponding to human perceptions of this stimulus. 5 Discussion

In most visual scenes, the majority of objects are partially occluded. Our seamless perception of the world depends upon an ability to complete or link the spatially separated, non-occluded portions of an object. We have used the idea that the visual system identifies proto-objects (which may or may not be objects) and then attempts to link these proto-objects into larger structures. This linking process is most apparent in the perception of illusory contours, and our model can account for a wide range of these illusions. This model builds upon previous neural, psychological, and machine vision studies. Several models of illusory contour generation (Ullman 1977; Peterhans and von der Heydt 1989; Finkel and Edelman 1989) have used related mechanisms to check for collinearity and to generate the illusory contours. Our model differs at a more fundamental level-we are concerned with objects not just contours. To define an object, surfaces must also be considered. For example, in a simple line drawing, we perceive an interior surface despite the fact that no surface properties are indicated. Thus, the model must be capable of characterizing a surface-and it does so, in a rudimentary manner, by determining the direction of figure and relative depth. Nakayama and Shimojo (1990) have approached the problem of surface representation from a similar viewpoint. They discuss how contours and surfaces become associated, how T-junctions serve to stratify objects in depth, and how occluded surfaces are amodally completed. Nakayama’s analysis concentrates on the external “ecological” constraints on perception. In addition to these Gibsonian constraints, we emphasize the importance of internal constraints imposed by physiological mechanisms and neural architectures. Nakayama has also explored the interactions between occlusion and surface attributes. A more complete model must consider such surface properties such as color, brightness, texture, and surface orientation. The examination of

915


O B A

B

Figure 6: Minimization of ambiguous discontinuities. Upper panel shows an ambiguous stimulus (adapted from Kanizsa 1979), two possible perceptual interpretations of which are shown below. The interpretation on the left is dominant for humans, despite the figural symmetry of the segmentation on the right. Stimulus was presented to the system, results shown after three iterations. (A) Spatial histogram showing the contour binding patterns (as in 5A). The network segments the figures in the same manner as human perception. (B) Determination of direction of figure confirms network interpretation (note at junction points, direction of figure is indeterminate).

916


how surface features might interact with contour boundaries has been pioneered by Grossberg (1987). Finally, in some regards, our model constitutes the first step of a "bottom-up" model of object perception (Kanizsa 1979; Biederman 1987). It is interesting that regardless of one's orientation (bottom-up or top-down) the constraints of the physical problem result in certain similarities of solution as witnessed by the analogies present with A1 based models (Fisher 1989). One of the most speculative aspects of the model is the use of tags to identify elements as belonging to the same object. Tags linking units responding to the same contour are used to determine the direction of figure and to change the perceived depth of the entire contour based on occlusion relationships detected at isolated points (the tag junctions). It is possible to derive alternative mechanisms for these processes that do not depend on the use of tags, but they are conceptually inelegant and computationally unwieldy. Our model offers no insight as to the biophysical basis of such a tag. However, the model does suggest that there should be a relatively small number of tags, on the order of 10, since this number corresponds to the number of objects that can be simultaneously discriminated. This constraint is consistent with several possible mechanisms: tags represented by different oscillation frequencies, tags represented by different phases of firing, or tags represented by firing within discrete time windows (e.g., the first 10 msec of each 50 msec interval). The number of distinct tags generated by these various mechanisms may depend on the integration time of the neuron, or possibly on the time constant of a synaptic switch, such as the NMDA receptor. At the outset, we discussed the importance of both binding and segmentation for visual object discrimination. Our model has largely dealt with the segmentation problem, however, the two problems are not entirely independent. For example, the association of a depth value with the object discriminated is, in essence, an example of the binding of an attribute to an object. Consideration of additional attributes makes the

Figure 7: Facing p u p . Occlusion capture. Upper panel shows stimulus (adapted from Kanizsa 1979) in which we perceive a white illusory square. Note that the four black discs inside the illusory square appear closer than the background. A 64 x 64 discrete version of stimulus was presented to the network. (A) Spatial histogram (as in 5A) of the initial and final (after three iterations) contour binding tags. Note that the illusory square is bound as an object. (B) Direction of figure determined by the system. Insets show a magnified view of the initial (left) and final (right) direction of figure (region of magnification is indicated). Note that the direction of figure of the "mouth of the pac-man flips once the illusory contour is generated. (C) Activity in the foreground network (% of maximum)demonstrates network stratification of objects in relative depth. The illusory square has "captured" the background texture.

917


C

918


problem more complex, but it also aids in the discrimination of separate objects (Damasio 1989; Crick and Koch 1990). For example, we have only considered static visual scenes, but one of the major cues to the linking process is common motion of proto-objects. During development, common motion may, in fact, play the largest role in establishing our concept of what is an object (Termine et al. 1987). Object definition also clearly depends on higher cognitive processes such as attention, context and categorization (Rosch and Lloyd 1978). There is abundant evidence that "top-down'' processes can influence the discrimination of figure/ground as well as the perception of illusory figures (Gregory 1972). The examples considered here (e.g., Figs. 5-7) represent extended visual scenes, and perception of these stimuli would require multiple shifts of gaze and/or attention. The representation of such a scene in intermediate vision is thus a more dynamic entity than portrayed here. The processes we have proposed are rapid (all occur in several cycles of iteration), and thus might be ascribed to preattentive perception. However, such preattentive processing sets the stage for directed attention because it defines segmented objects localized to particular spatial locations. Furthermore, the process of binding contours, surfaces, and surface features may be restricted to one or two limited spatial regions at any one time. Thus, feature binding may be a substrate rather than a result of the attentional process. We have implicitly assumed that object discrimination is a necessary precursor to object recognition. Ullman (1989) has developed a model of recognition that demonstrates that this need not logically be the case. The question of whether you have to know that something is a "thing" before you can recognize what kind of thing it is remains to be determined through psychophysical experiment. It is appealing, however, to view object discrimination as the function of intermediate vision, i.e., those processes carried out by the multiple extrastriate visual areas. In this view, each cortical module develops invariant representations of aspects of the visual scene (motion, color, texture, depth) and the operations of these modules are dynamically linked. The consistent representations developed in intermediate vision then serve as the substrate for higher level cognitive processes. In conclusion, we have shown that one can build a self-contained system for discriminating objects based on occlusion relationships. The model is successful at stratifying simple visual scenes, for linking the representations of occluded objects, and at generating responses to illusory objects in a manner consistent with human perceptual responses. The model uses neural circuits that are biologically based, and conforms to general neural principles, such as the use of a distributed representation for depth. The system can be tested in psychophysical paradigms and the results compared to human and animal results. In this manner, a computational model that is designed based on physiological data and


919

tested in comparison to psychophysical data offers a powerful paradigm for bridging the g a p between neuroscience and perception. Note Added i n Proof The recent findings of dynamic changes in receptive field structure in striate cortical neurons by Gilbert a n d Wiesel (1992) indicates that long-range connections undergo context-dependent changes in efficacy. Such a mechanism may provide the biological basis for the direction of figure a n d linkage mechanisms proposed here. [Gilbert, C. D., a n d Wiesel, T. N. 1992. Receptive field dynamics in adult primary visual cortex. Nnhire 356, 150-152.1 Acknowledgments This work w a s supported by grants from The Office of Naval Research (N00014-90-J-1864), The Whitaker Foundation, a n d The McDonnell-Pew Program in Cognitive Neurocience. References Aloimonos, J., and Shulman, D. 1989. Integration of Visual Modules. New York, Academic Press. Attneave, F. 1954. Some informational aspects of visual perception. Psych. Rev. 61, 183-193. Barlow, H. 8. 1981. Critical limiting factors in the design of the eye and visual cortex. Proc. R. Soc. (London) B212, 1-34. Biederman, I. 1987. Recognition by components: A theory of human image understanding. Psych. Rcz~94, 115-147. Crick, F., and Koch, C. 1990. Towards a neurobiological theory of consciousness. Sernin. Neurosci. 2, 263-275. Damasio, A. R. 1989. The brain binds entities and events by multiregional activation from convergence zones. Neural Comp. 1, 1223-1232. Dobbins, A. S., Zucker, S. W., and Cynader, M. S. 1987. Endstopping in the visual cortex as a neural substrate for calculating curvature. Nntrire (London), 329,438-441. Eckhorn, R., Bauer, R., Jordan, W., Brosch, M., Kruse, W., Munk, M., and Reitboeck, H. 1988. Coherent oscillations: A mechanism of feature linking in the visual cortex? Biol. Cybernet. 60, 121-130. Finkel, L., and Edelman, G. 1989. Integration of distributed cortical systems by reentry: A computer simulation of interactive functionally segregated visual areas. I. Neurosci. 9, 3188-3208. Fisher, R. B. 1989. Froin Objects to Surfaces. John Wiley & Sons, New York. Gilbert, C. D., and Wiesel, T. N. 1989. Columnar specificity of intrinsic connections in cat visual cortex. J. Neurosci. 9, 2432-2442. Gray, C. M., and Singer, W. 1989. Neuronal oscillations in orientation columns of cat visual cortex. Proc. Natl. Acad. Sci. U.S.A. 86, 1698-1702.

920


Gregory, R. L. 1972. Cognitive contours. Nature (London) 238, 51-52. Grossberg, S. 1987. Cortical dynamics of three-dimensional form, color, and brightness perception. I: Monocular theory. Percept. Psyclzophys. 41,87-116. Grossberg, S., and Mingolla, E. 1985. Neural dynamics of form perception: Boundary completion, illusory figures, and neon color spreading. Psychol. Rev. 92, 173-211. Guzman, A. 1968. Decomposition of a visual scene into three-dimensional bodies. Fall Joint Comput. Conf. 1968, 291-304. Kanizsa, G. 1979. Organization in Vision. Praeger, New York. Koffka, K. 1935. Principles of Gestalt Psychology. Harcourt, Brace, New York. Konig, P., and Schillen, T. 1991. Stimulus-dependent assembly formation of oscillatory responses: I. Synchronization. Neural Comp. 3, 155-166. Lehky, S., and Sejnowski, T. 1990. Neural model of stereoacuity and depth interpolation based on distributed representation of stereo disparity. J. Neurosci. 7, 2281-2299. Livingstone, M. S., and Hubel, D. 1988. Segregation of form, color, movement, and depth: Anatomy, physiology, and perception. Science 240, 740-749. Marr, D. 1982. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. W. H. Freeman, San Francisco. Minsky, M., and Papert, S. 1969. Perceptrons. The MIT Press, Cambridge, MA. Mitchison, G., and Crick, F. 1982. Long axons within the striate cortex: Their distribution, orientation, and patterns of connections. Proc. Natl. Acad. Sci. U.S.A. 79,3661-3665. Nakayama, K., and Shimojo, S. 1990. Toward a neural understanding of visual surface representation. Cold Spring Harbor Symp. Quant. Biol. LV, 911-924. Peterhans, E., and von der Heydt, R. 1989. Mechanisms of contour perception in monkey visual cortex. 11. Contours bridging gaps. J. Neurosci. 9, 1749-1763. Poggio, G. F., Gonzalez, F., and Krause, F. 1988. Stereoscopic mechanisms in monkey visual cortex: Binocular correlation and disparity selectivity. J. Neurosci. 8, 45314550. Poggio, T., Gamble, E. B., and Little, J. J. 1988. Parallel integration of vision modules. Science 242, 436440. Ramachandran, V. S. 1987. Visual perception of surfaces: A biological theory. In The Perception oflllusory Contours, S. Petry and G. E. Meyer, eds., pp. 93-108. Springer-Verlag, New York. Ramachandran, V. S. 1986. Capture of stereopsis and apparent motion by illusory contours. Percept. Psychophys. 39, 361-373. Ramachandran, V. S., and Cavanaugh, P. 1985. Subjective contours capture stereopsis. Nature (London) 317, 527-530. Resnikoff, H. L. 1989. The Illusion of Reality. Springer-Verlag, New York. Rockland, K. S., and Lund, J. S. 1982. Widespread periodic intrinsic connections in the tree shrew visual cortex. Science 215, 1532-1534. Rosch, E., and Lloyd, B. B. 1978. Cognitionand Categorization. Lawrence Erlbaum, Hillsdale, NJ. Rosenfeld, A. 1988. Computer vision. Adv. Comput. 27, 265-308. Sajda, P., and Finkel, L. 1990. NEXUS: A neural simulation environment. University of Pennsylvania Tech. Rep.


921

Sajda, P., and Finkel, L. 1992. NEXUS: A simulation environment for large-scale neural systems. Sitnirlntioli, in press. Sejnowski, T., and Hinton, G. 1987. Separating figure from ground with a Boltzmann machine. In Visiotz, Brniti mid C o o p m t i v e Cotriptnfioti, M. Arbib and A. Hanson, eds., pp. 703-724. The MIT Press, Cambridge, MA. Shapley, R., and Gordon, J . 1987. The existence of interpolated illusory contours depends on contrast and spatial separation. In The Percc,ptiotl oflll~rsc~r-!/ Cotitours, S. Petry and C. E. Meyer, eds., pp. 109-115. Springer-Verlag, New York. Termine, N., Hrynick, T., Kestenbaum, T., Gleitman, H., and Spelke, E. S. 1987. Perceptual completion of surfaces in infancy. 1. Exp. Psychol. Hirtrinri Prvccpt. 13, 524-532. Ullman, S. 1989. Aligning pictorial descriptions: An approach to object recognition. Cogtiifion 32, 193-254. Ullman, S. 1977. Filling-in the gaps: The shape of subjective contours and a model for their generation. Biol. Cyberrtef. 25, 1-6. von der Heydt, R., and Peterhans, E. 1989. Mechanisms of contour perception in monkey visual cortex. I. Lines of pattern discontinuity. I. Nclrrosci. 9, 1731-1 748. ~

~~~

~~

~~

~

Received 22 November 1991; accepted 6 April 1992

This article has been cited by: 2. Warren W. Tryon. 2009. Cognitive Processes in Cognitive and Pharmacological Therapies. Cognitive Therapy and Research 33:6, 570-584. [CrossRef] 3. Axel Thielscher, Heiko Neumann. 2008. Globally consistent depth sorting of overlapping 2D surfaces in a model using local recurrent interactions. Biological Cybernetics 98:4, 305-337. [CrossRef] 4. Sheshadri R. Thiruvenkadam, Tony F. Chan, Byung-Woo Hong. 2008. Segmentation under Occlusions Using Selective Shape Prior. SIAM Journal on Imaging Sciences 1:1, 115. [CrossRef] 5. A. Thielscher, H. Neumann. 2007. A computational model to link psychophysics and cortical cell activation patterns in human texture processing. Journal of Computational Neuroscience 22:3, 255-282. [CrossRef] 6. R. Baumann, R. Zwan, E. Peterhans. 1997. Figure-Ground Segregation at Contours: a Neural Mechanism in the Visual Cortex of the Alert Monkey. European Journal of Neuroscience 9:6, 1290-1303. [CrossRef] 7. D.W. Jacobs. 1996. Robust and efficient detection of salient convex groups. IEEE Transactions on Pattern Analysis and Machine Intelligence 18:1, 23-37. [CrossRef] 8. Steven J. Nowlan, Terrence J. Sejnowski. 1994. Filter selection model for motion segmentation and velocity integration. Journal of the Optical Society of America A 11:12, 3177. [CrossRef]

Communicated by John Bridle

An Adaptive Lattice Architecture for Dynamic Multilayer Perceptrons Andrew D. Back Ah Chung Tsoi Department of Electrical Engineering, University of Queensland, St. Lucia 4072, Australia

Time-series modeling is a topic of growing interest in neural network research. Various methods have been proposed for extending the nonlinear approximation capabilities to time-series modeling problems. A multilayer perceptron (MLP) with a global-feedforward local-recurrent structure was recently introduced as a new approach to modeling dynamic systems. The network uses adaptive infinite impulse response (IIR) synapses (it is thus termed an IIR MLP), and was shown to have good modeling performance. One problem with linear IIR filters is that the rate of convergence depends on the covariance matrix of the input data. This extends to the IIR MLP: it learns well for white input signals, but converges more slowly with nonwhite inputs. To solve this problem, the adaptive lattice multilayer perceptron (AL MLP), is introduced. The network structure performs Gram-Schmidt orthogonalization on the input data to each synapse. The method is based on the same principles as the Gram-Schmidt neural net proposed by Orfanidis (1990b),but instead of using a network layer for the orthogonalization, each synapse comprises an adaptive lattice filter. A learning algorithm is derived for the network that minimizes a mean square error criterion. Simulations are presented to show that the network architecture significantly improves the learning rate when correlated input signals are present. 1 Introduction

Neural network models have been successfully applied to many problems requiring static nonlinear mappings and classifications (that is, no time dependence exists between data). Yet to solve many practical problems, it is necessary to model dynamic systems, where the mapping to be learned depends on the present and past states of the system (these may be measurable outputs or inaccessible states). A number of architectures have been used in neural network models to approximate time-dependent systems. These include windows of time-delayed inputs Neural Computation 4, 922-931 (1992)


Dynamic Multilayer Perceptrons

923

(Lapedes and Farber 1987), state inputs (Guez and Selinsky 1988), and recurrent connections (Jordan 1988; Robinson 1989; Williams and Zipser 1989). A network having a local-recurrent global-feedforward structure, was recently introduced (Back and Tsoi 19914, in which the synapses are adaptive infinite impulse resporzse (IIR) filters. The IIR synapse multilayer perceptron (IIR MLP), is a generalization of multilayer perceptrons involving only feedforward time delays [such as the time delay neural network (TDNN')]. This approach differs from other recurrent networks (Robinson 1989; Williams and Zipser 1989; Jordan 1988; Elman 1990) where global feedback is used. An algorithm for an FIR MLP was first presented by Wan (1990). The synaptic equation for an IIR MLP is given by

where y(t) is the synaptic output, b, j = 0.1.. . . . rn are the feedforward A coefficients, a, j = 1 . 2 . . . . .ri are the feedback coefficients, and q - l z ( t ) = z ( t - j ) . It has been shown by simulation (Back and Tsoi 1991a) that this class of networks is a better model than a network with only feedforward time-delay synapses. Since the network overcomes the finite window limitation of the TDNN, it can model long term dynamic behavior. Correlated inputs are known to result in slow learning times for linear adaptive filters (Haykin 1986). This difficulty in learning was also recognized to occur in multilayer perceptrons by Orfanidis (1990b). The IIR MLP is subject to the same problem: when the inputs are not white noise, learning at any particular synapse is slowed by a factor proportional to the eigenvalue spread of the input data to that synapse. In linear filtering theory, the adaptive lattice filter (Friedlander 1982) has been designed to overcome the problem by performing a Gram-Schmidt orthogonalization on the input data. Adaptive lattice filters have additional advantages in terms of reduced sensitivity to finite-precision effects (as would be encountered in VLSI implementations). In this paper, a multilayer perceptron with adaptive lattice synapses is introduced to overcome the above mentioned limitations of the IIR MLP. The outline of the paper is as follows: in Section 2, a lattice network architecture is introduced. In Section 3, a learning rule is derived by extending a previous algorithm for a linear adaptive lattice filter. Simulations of the IIR MLP and AL MLP modeling a nonlinear dynamic 'The synapses in the TDNN can be considered as finite impulse response (FIR) filters, although the overall structure of the TDNN may be quite different (Waibel rt a / . 1988). A more recently introduced algorithm, the autoregressive backpropagation algorithm (Leighton and Conrath 1991), is a special case of the IIR MLP. It is readily seen that the AR neuron is equal to two neurons in our terminology: the first neuron having a single FIR synapse, followed by a second, linear-output neuron with a n all-pole IIR synapse.

Andrew D. Back and Ah Chung Tsoi

924

Figure 1: A two-multiplier lattice synapse.

system with correlated inputs are presented in Section 4. Conclusions are given in Section 5.

2 An Adaptive Lattice MLP Architecture

In this section, a network architecture based on the multilayer perceptron, but with adaptive lattice synapses is introduced. Each synapse is a twomultiplier IIR lattice (Fig. l),although other structures involving various numbers of multipliers could be used (Gray and Markel 1973; Tummala 1989). A detailed review of adaptive lattice filters can be found in (Friedlander 1982). The AL MLP is defined as follows (its structure is shown in Fig. 2). The forward equations for layer I ( I = 0.1.2,. . . .L, i.e., there are L 1 layers in the MLP) are

+

(2.3) (2.4)


925

Figure 2: An adaptive lattice multilayer perceptron (AL MLP).

neuron output, neuron state, ith synaptic output, ith synaptic gain, jth feed forward coefficient in the ith synapse, jth backward residual in the ith synapse, order of each lattice synapse. The flow of information through a synapse is the same as in the linear filter case (Parikh et al. 1980):

b,(t) f,-l(f)

= =

1 ) + k,(t)f,-,(t). fJt) - k,(t)b,-l(t - 1).

b,-l(t

-

j j

1.2.. . . . m = m.rn - 1 . . . . 1 =

(2.5) (2.6)

with initial and boundary conditions:

(2.7) (2.8)

926


Nonlinear Plant ........................................................

c

i

-

*

Lattice Synapse

Reduced Complexity AL MLP

+ Sigmoidal units

Figure 3: A reduced complexity adaptive lattice MLP modeling a dynamic nonlinear plant. With reference to Figure 1, z ( t ) in 2.8 is the output of the neuron in the previous layer 1 - 1. It is often possible to use a network architecture with reduced complexity to model the situation when the unknown system involves only a linear dynamic part followed by a static nonlinear part. In this case, only the first layer ( I = 1)has lattice synapses (equations 2.1-2.8 apply). For layers (2)-(Z.), the network has no dynamic synaptic structure and is defined in the usual way:

(2.10) where

k = 1 , 2 , ...,N,+1 i ~ ,= bias The reduced complexity network is shown in Figure 3. In the next section, a gradient descent learning algorithm is derived for the AL MLP.


927

3 A Learning Algorithm for the AL MLP

In this section, a learning algorithm is developed based on the backpropagation algorithm (Rumelhart et al. 1986). Each coefficient in the lattice synapses is adjusted iteratively, minimizing the mean square error criterion, defined as

where yk(t) is the desired output at time t, and NL is the number of neurons in the output layer (layer L). The update equations for each synapse are similar to the linear case (Parikh rt al. 19801, except that the error term is replaced by a backpropagated delta [ & ( t ) ] , and the synaptic gain terms [ g ( t ) ] are included. The feedforward coefficients for each synapse are updated according to

v ( t + 1)

=

v ( t )+ AZJ(f)

Av(t)

=

-?/-

(3.2)

X(t)

(3.3)

all(t )

(3.4)

(3.5)

The delta term hL(t) for the kth neuron in layer l is computed in a similar manner to the original backpropagation algorithm, taking into account the lattice structure of the synapses. Thus for an L + 1 layer network (3.6)

where vkh,(f) is the jth feedforward coefficient of the synapse between the kth and kth neurons in layers I and 1 1, respectively, and

+

(3.8) (3.9)


928

The reflection coefficients are updated according to

kj(t

+ 1)

=

k,(t)

+ Ak(t)

(3.10)

(3.11) where

The synaptic gain term introduced in 2.2, is updated according to

(3.14)

For the reduced complexity network, the delta terms In:( t ) ] , are calculated the same as in the normal backpropagation algorithm (Rumelhart et al. 1986) by replacing 3.7 with & t ) = f"i:(t)]

c Sl;t'(t)w:'

(3.16)

h=l

The learning algorithm for the AL MLP is an extension of the linear lattice filter derived by Parikh et al. (1980), and offers a method of applying the known advantages of Gram-Schmidt orthogonalization in adaptive lattice filters to nonlinear systems. Consult Parikh et al. (1980) or Friedlander (1982) for further details of the lattice implementation. 4 Simulation Examples The performance of the AL MLP was compared against an IIR MLP by modeling a nonlinear plant described by

yp(t)= sin

0.0154 + 0.0462q1' + 0.0462q-2 + 0.0154qp3 x(t)] 1 - 1.99q-' 1.5729-' - 0.4583qp3

+

(4.1)


929

The networks were trained with correlated input signals generated by a white noise source passed through the filter (4.2) where a1 = -1.6 and a2 = 0 95. In this case, the eigenvalues of the covariance matrix were (3.43.0.35)giving an eigenvalue spread of 3.4310.35 = 9.8. Similarly, for al = -1.8, the corresponding eigenvalues were (8.13, 0.32) giving an eigenvalue spread of 25. The nonlinear system was modeled as shown in Figure 3. Note that a reduced complexity network is used for each network, as introduced in Back and Tsoi (1991b). The order of the IIR synapse was selected as ( m . n ) = (7.7), and the AL synapse order was m = 7. The learning rates chosen were i/dyna,,,,c = 0.0001 and ilStatlc= 0.05, where the rfyiiamic subscript refers to the time-delay synapses, and the static subscript refers to synapses without time delays. The average mean squared error was plotted during learning, using 50 runs with each point being a moving average of the previous 20 points (Fig. 4). Though not shown here, it is observed that as the eigenvalue spread of the input data increases, the time to achieve convergence for each network increases, but the AL MLP has significantly better performance than the IIR MLP. 5 Conclusions

The IIR MLP was presented recently as a means of modeling nonlinear dynamic system systems with good results. A problem exists however, if the input data is correlated (this can occur in many signal processing applications: for example, in speech or sonar signals). The problem of correlated inputs for the standard MLP was also recognized by Orfanidis who proposed the use of a Gram-Schmidt orthogonalization layer in the network structure (Orfanidis 1990b). In this paper, a solution to the problem of correlated input data in the IIR MLP was proposed by introducing the notion of adaptive lattice filters for each synapse. The adaptive lattice network structure was described and a learning algorithm presented. Computer simulations were used to verify that the adaptive lattice multilayer perceptron is capable of significantly improving the time to achieve convergence when the input data are correlated.

Acknowledgments The authors wish to thank the reviewer for helpful comments and suggestions. The first author is supported by a Research Fellowship with the


930

0.75

a

3

E:

0.00 0

100

200

300

400

500

300

400

500

Samples x 100

0.50 b

I

E 0.25

0.00 0

100

200

Samples x 100

Figure 4: Mean square error during learning for IIR MLP and AL MLP (averaged over 50 runs): (a) eigenspread of input = 9.8, (b) eigenspread of input = 25.

Electronics Research Laboratory, DSTO, Australia. The second author acknowledges partial support from the Australian Research Council.


931

References Back, A. D., and Tsoi, A. C. 1991a. FIR and IIR synapses, a new neural network architecture for time series modelling. Neural Cornp. 3(3), 352-362. Back, A. D., and Tsoi, A. C. 1991b. Analysis of hidden layer weights in a dynamic locally recurrent network. Artificial Neural Networks, T. Kohonen, K. Makisara, 0. Simula, and J. Kangas (eds.), pp. 961-966. Elsevier Science Publishers B.V., North-Holland. Elman, J. L. 1990. Finding structure in time. Cog. Sci. 14, 179-211. Friedlander, 8. 1982. Lattice filters for adaptive processing. Proc. ZEEE 70(8), 829-867. Gray, A. H., and Markel, J. D. 1973. Digital lattice and ladder filter synthesis. I E E E Trans. Audio Electroacoust. 21, 491-500. Guez, A., and Selinsky, J. 1988. A neuromorphic controller with a human teacher. Proc. IEEE Int. loitit Conf. Neural Networks 11, 595-602. Haykin, S. 1986. Adaptive Filter Theory. Prentice-Hall, Englewood Cliffs, NJ. Jordan, M. I. 1988. Supervised learning and systems with excess degrees of freedom. COINS Tech. Rep. 88-27, University of Massachusetts, Amherst. Lapedes, A,, and Farber, R. 1987. Nonlinear signal processing using neural networks: Prediction and System modelling. Tech. Rep. LA-UR87-2662, Los Alamos National Laboratory. Leighton, R. R., and Conrath, B. C. 1991. The autoregressive backpropagation algorithm. Proc. l E E E Int. joint Conf. Neural Netzuorks 11, 369. Orfanidis, S. J. 1990a. Optimum Signal Processing, 2nd ed. McGraw-Hill, New York. Orfanidis, S. J. 1990b. Gram-Schmidt neural nets. Neural Comp., 2, 116-126. Parikh, D., Ahmed, N., and Stearns, S. D. 1980. An adaptive lattice algorithm for recursive filters. I E E E Trans. Acous., Speech, Signal Proc. 28, 110-111. Robinson, A. J. 1989. Dynamic error propagation networks. Ph.D. dissertation, Cambridge University Engineering Department. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing, Vol. 1, Ch. 8. The MIT Press, Cambridge, MA. Tummala, M. 1989. One-multiplier adaptive lattice algorithm for recursive filters. Circuits, Systems, and Signal Processing 8(4), 455466. Waibel, A,, Hanazawa, T., Hinton, G., Shikano, K., and Lang, K. 1989. Phoneme recognition using time-delay neural networks. I E E E Trans. Acoust., Speech, Signal Process. ASSP-37, 328-339, March. Wan, E. A. 1990. Temporal backpropagation for FIR neural networks. Proc. I E E E Znt. Joint Conf. Neural Networks I, 575-580. Williams, R. J., and Zipser, D. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural Comp. 1, 268-278.

Received 31 May 1991; accepted 9 April 1992.


Communicated by Christof Koch

A Model Circuit for Cortical Temporal Low-Pass Filtering R. Maex G. A. Orban Laboraforium uoor Neuro- err Psyclrofysiologie, Katholieke Uniuersiteit Leuuen, Campus Gasthuisberg, B-3000 Leuuen, Belgium

We propose that the low-pass characteristics in the temporal and velocity domain of area 17 cells are generated by the abundant excitatory connections between cortical neurons. We have incorporated this anatomical feature in a model circuit in which simple cells' firing is initiated by geniculocortical excitatory synaptic input, with a short time course, and firing is maintained by feedback corticocortical excitatory synapses, which have a longer time course. The low-pass performance of the model is demonstrated by computing the model simple cells' velocity response curves (VRC) elicited by moving bars, and comparing these to those of its LGN (lateral geniculate nucleus) inputs. For the same parameter set, the VRCs of sustained and transient LGN cells are transformed into VRCs typical of central area 17 and central area 18 cells, respectively. 1 Introduction

It has been argued that only cortical circuitry can account for the temporal properties of cortical cells, and particularly for the abundance of velocity low-pass cells in area 17 of cat and monkey (Orban et al. 1981a; Movshon et al. 1978; Lee et al. 1981; Derrington and Fuchs 1979; Orban et al. 1986). Neither the input from the lateral geniculate nucleus (LGN) nor the membrane time constant of cortical neurons can generate the lowpass time constant of 53 to 80 msec (Worgotter and Holt 1991) measured for area 17 simple cells near the area centralis. Functionally, these velocity low-pass simple cells are suited as spatial analyzers during fixation (Duysens et al. 1985b; Orban 1991). Computationally, low-pass filtering is important for creating and preserving direction selectivity. About one-fourth of velocity low-pass cells are direction selective (Orban et al. 1981b; Orban et 01. 1986). This can be achieved by combining inputs that are spatially and temporally separated (Maex and Orban 1991). Due to temporal aliasing, models of direction selectivity begin to prefer the opposite direction of moving gratings at twice their optimal temporal frequency. Since most area 17 cells have an optimal temporal frequency of 1 to 4 Hz, while their geniculate input responds Neural Computation

4, 932-945 (1992)


Cortical Low-Pass Filtering

933

optimally near 10 Hz (Lee ct al. 1981), a low-pass mechanism is needed to eliminate the responses at higher speeds. In this paper, we present a biologically realistic model cortical circuit producing temporal low-pass filtering in simple cells, a n d the model’s responses to bars moving a t different speeds. The contribution of different sources of low-pass filtering in the model is discussed. 2 Model and Simulations 2.1 Model Cortical Circuit. The full model (Fig. 1A) comprises both excitatory (putative pyramidal or spiny stellate cells) a n d inhibitory (pu-

A

B

Figure 1: Full (A) and reduced (8)model for studying cortical low-pass filtering in simple cells. Circles represent neuron somata, > = excitatory synapse, o = inhibitory synapse. (A) The RFs of ON- and OFF-center LGN cells (E and F) totally overlap. Only a single discharge region of the cortical simple cells is modeled: an ON subregion for neurons A and B, an OFF subregion for neurons C and D. Inhibitory neurons (filled circles, B and D) inhibit neurons with RFs of opposite polarity (eg., cell B inhibits cell C and cell D). Excitatory neurons (open circles, A and C) excite neurons the RFs of which have the same polarity (e.g., cell A excites cell A and cell B). (B) Reduced version of the model in A. Only low-pass mechanisms 1 and I1 (see text) are preserved. Neuron A has been decomposed into a pool of 3 cells (a, b, and c). The LGN input into neurons b and c (dashed lines) could be deleted to study the behavior of second-order neurons.

934

R. Maex and G. A. Orban

tative basket or clutch cells) neurons. The excitatory cells excite excitatory as well as inhibitory cortical neurons receiving LGN input of the same polarity, while the inhibitory cells inhibit cortical cells whose major LGN input has the opposite polarity. The receptive fields (RF)of these cortical neurons totally overlap and have but a single discharge region, which has the same polarity as their LGN input. One consequence is that simple cells with a discharge region of opposite polarity mutually inhibit each other in a push-pull way (Palmer and Davis 1981). Modeling a single subregion is acceptable since the low-pass characteristics of simple cells do not critically depend on spatiotemporal interactions within one subregion or between neighboring subregions (Duysens et al. 1985a),and can be predicted from their responses to stationary stimuli (Duysens eta/. 1985b; Baker 1988). The model circuit contains three candidate sources of low-pass filtering (labeled in Fig. 1A): feedback excitation (I), temporal integration by synapses (111, and mutual inhibition (111).

2.2 Single-Neuron Models. A detailed, formal description of cortical neurons and their geniculate input is presented in the appendix. The LGN responses were computed by the following sequence: convolution of the stimulus with the gaussian center and surround weighting functions, separate temporal band-pass filtering for center and surround, subtraction of surround from center responses, addition of spontaneous activity, half-wave rectification, and finally spike generation. Cortical neurons are single passive compartments as in Worgotter and Koch (1991). Their spikes are all-or-nothing events that occur with a high probability above a fixed threshold. In the excitatory neurons, a spike is followed by a fast-decaying and a medium-duration afterhyperpolarization (AHP) (Schwindt et al. 1988). The fast-decaying AHP ensures a regular firing, while accumulation of medium-duration AHPs causes some adaptation, lowering the firing rate (Connors et a/. 1982). Inhibitory neurons have only a short AHP, and hence are fast spiking. Both the synaptic conductances and the AHPs are modeled as conductance changes, the time course of which is an rr-function (Jack e t a / . 1975).

2.3 Parameter Setting. LGN cell centers respond to stimuli of opposite polarity with a latency difference of 32-40 msec (Orban et a/. 1985). The RFs of transient cells are twice the size of those of sustained cells (Orban et al. 1985). This, together with a different value for the parameter governing the degree of transientness of the response to stationary stimuli, creates two types of velocity response curves (VRC), which match the data of Gulyas et al. (1990). Since we were interested mainly in transformations in the temporal domain and also because our stimulus was a narrow bar, we did not model the spatial nonlinearities of transient (or Y) cells.


935

With the parameter values chosen for the AHPs of the excitatory cortical neurons, a steady-state frequency/intensity slope of 24 Hz/nA was measured on current injection, which falls within the range reported by Connors et a / . (1988). The inhibitory synapses have a slow component (time to peak 80 msec, Douglas and Martin 1991) and a fast component (time to peak 2 msec). Since simple cells commonly show a clear hyperpolarization on stimulation with nonoptimal contrast polarity, both components were modeled as conductance changes of the potassium channels, induced by GABABreceptors. The sensitivity of the model’s response to the time course and strength of both geniculocortical and corticocortical excitatory synapses was examined in a systematic way. In the final simulations (Fig. 2C,G), the geniculocortical synapses had a short time course (time to peak 1 msec) (Hagihara et al. 19881, the corticocortical excitatory synapses had a slow time course (time to peak 16 msec) and the ratio of the total conductances induced by these synapses was about 1 over 8 (see Appendix). The width of the discharge region of the sustained-input simple cells, derived from the histograms, is between 0.3 and 0.4 degrees. 2.4 Simulation Methodology. Peristimulus-time histograms were computed for 8 speeds of a narrow light bar by accumulating spikes, generated during at least 80 sweeps, in bins of 8 msec. The VRCs plot the peak responses from the unsmoothed histograms (Orban et a/. 1981a). The membrane potential was recorded as well, to evaluate subthreshold and hyperpolarizing responses. Sometimes, e g , for computing the parameter sensitivity in the feedback excitatory circuit, only a part of the full model was simulated (Fig. 1B). In these simulations, as well as in control simulations of the full model, the excitatory cells were decomposed into a pool of 3 (Fig. 1B) to 7 neurons, in order to prevent a systematic coincidence of the AHPs with the feedback excitation. In that case, the plotted VRCs are the means of the VRCs of the composing neurons. The simulation environment written by the authors in Pascal was run on a microvax. First-order nonlinear differential equations were solved with a central difference method. 2.5 Results. To evaluate the low-pass filtering performed by the model, we compared the VRCs of model sustained (Fig. 2A) and transient (Fig. 2E) LGN cells with model simple cells driven by sustained (Fig. 2C) or transient (Fig. 2G) LGN input, when stimulated with moving narrow light bars. First (Fig. 2B,F) we examined the effect of the time course of the geniculocortical synapse. The rr-functions describing these time courses have been normalized, i.e., their integrals are constant, as in the original formulation by Jack et a / . (1975). For different time-to-peak values, the amplitude of the excitatory postsynaptic potential (EPSP) of an isolated,

936


nonspiking neuron was measured, hence its membrane potential had no components due to feedback excitation, inhibition or AHPs. The only effect (Fig. 2B,F) is an attenuation of the EPSPs at high speeds when time-to-peak values increase. Note that at most speeds the peak EPSP of the sustained-input cell does not reach the threshold (15mV), even if that same neuron generates strong responses in the full model. There are two reasons for this. First, the peak EPSPs plotted are the means over 80 trials in bins of 8 msec, which is, however, half the membrane time constant. Second, the EPSPs received no contribution from the feedback excitation, which is initiated as soon as the threshold is crossed and will maintain firing as long as geniculocorticalexcitation keeps the membrane potential close enough to the threshold. In all VRCs presented infra, the time to peak was set to 1 msec, assuming that receptors of the non-NMDA (Nmethyl-D-aspartate) type are involved in geniculocortical transmission (Hagihara et al. 1988). The responses of the full model (Fig. 2C,G) show a shift of the VRCs to lower speeds, creating velocity low-pass behavior only for sustainedinput cells (Fig. 2C), and not for transient-input cells (Fig. 2G). Notice that the same cortical circuit that transforms the VRCs of sustained inputs into low-pass curves like those recorded in area 17 (Fig. 2D) also transforms VRCs of transient inputs into curves tuned to slow speeds, typical of area 18 (Fig. 2H). The slight attenuation of the response at the lowest speed in Figure 2C is due to the adaptation of the excitatory neuron. The responses of the OFF-simple cells are strongly reduced by the threshold and by mutual inhibition (cell C and cell D in Fig. 2C,G). The ON-simple cell starts inhibiting the OFF-cell as soon as the light bar enters its RF and so prevents the OFF-cell from firing when the bar leaves the RF, except at higher speeds for the transient-input cells. The sensitivity of the shape of these VRCs to the parameters governing the excitatory feedback loop was mainly examined on the reduced model of Figure 1B. [We feel that a decrease of up to 4 times in simulation time justifies this reduction. Indeed, the only change to cell A Figure 2: Facing page. (A, C, E, and G) Simulated, normalized velocity response curves (VRCs, see Simulation Methodology). VRCs are computed for a model sustained LGN cell (A), a model transient LGN cell (El, and model simple cells receiving sustained (C) or transient (G) LGN input. The insets attach each VRC to a neuron of Figure 1A. The vertical bars in A span the standard deviations. (Dand H) Representative examples of VRCs to moving light bars for velocity low-pass (D) and velocity tuned (H) cells, which are the predominant cell types in central area 17 and central area 18, respectively, of the cat (from Orban eta!. 1981a). (B and F) Peak EPSP values (see text) of isolated, nonspiking cortical neurons receiving sustained (8) and transient (F) input for different time-topeak values of the geniculocortical synapse (see insets). (Note that the velocity axes in the simulation graphics are differently scaled for sustained and transient cases.)

937


SUSTAINED

0 0 0 5

TRANSIENT

0

2 0

8 0

32 0

128 0

04 1 0

H

velocity (degrees/ s)

4 0

16.0

64 0

256 0

r e i ~ ~ l i TUNED y

velocity (degrees / S )

938


in Fig. 1A is a loss of its inhibitory input from cell D. Now, cell D did not usually respond at all in the full model (Fig. 2Cl.l The results for sustained input are shown in Figure 3. Increasing the absolute strength of the corticocortical feedback connections, while keeping the geniculocortical weight constant (Fig. 3A, the time to peak of the corticocortical synapses here is 16 msec), amplifies the responses at low speeds until saturation occurs. Increasing time-to-peak values of the corticocortical synapse (Fig. 3B) shifts the velocity at which responses are maximally amplified toward lower speeds. For values close to 16 msec, an optimal velocity low-pass behavior is produced (Fig. 38, compare with Fig. 2D). Changing the absolute strength of the geniculocortical input (Fig. 3C) influences the VRCs in two ways. At low input strengths, the threshold is reached in only a fraction of the sweeps, decreasing the responses at all speeds. At high input strengths, responses at all speeds increase, and discharge regions become wider, hence the upper cutoff velocity (the speed, on the declining part of the VRC, at which the response is half the maximum response) increases until the VRCs become broad-band. Finally (Fig. 3D), we looked at the VRCs of second-order neurons (neurons b and c in Fig. 1B). Since their LGN input must pass an additional neuron and synapse, their upper cutoff velocity decreases, but the responses at low speeds can be amplified as well as in first-order neurons. 3 Discussion

The problem treated in this article is how cortical low-pass filtering might be generated so that the time constants measured by visual stimulation exceed the biophysical membrane time constants of cortical neurons by a factor of u p to eight [53-80 compared to 10-20 msec (Worgotter and Holt 199111. To this end, we have implemented three candidate mechanisms of temporal low-pass filtering in a model circuit for simple cells, and have computed their responses to bars moving at different speeds. The three biologically realistic mechanisms are feedback corticocortical excitatory synapses, mutual inhibition between neurons receiving input of opposite polarity, and temporal integration by synapses with a long time course. Feedback OY self-excitation generates low-pass filtering in a straightforward way: consider an analog neuron with time constant T and activity V , receiving inputs W and V with weights b and u, from the LGN and itself, respectively. For positive values of V, the state-equation can be written as dV 7= - V + a V + bW dt which rescales to r dV -- = - V + - b W 1 - 0 dt 1-a

Figure 3: Parameter sensitivity of the excitatory feedback loop for sustained-input cells. (A, B, and C) All VRCs are the mean VRCs of neurons a, b, and c of Figure 1B. All three neurons are driven by sustained LGN cells. The bold solid VRCs were obtained with the parameter values given in the appendix and used for the simulations of Figure 2C. The parameters changed were the absolute weight of the feedback corticocortical excitatory synapses (A), the time to peak of these corticocortical synapses (B), and the strength of the geniculocortical input (C). The effects of the time-to-peak value of the corticocortical synapses (8) and of the strength of the geniculocortical input (C) on the shape of the VRCs were computed for a range of corticocortical synaptic weights. Each VRC plotted is the one which, for the parameter value tested (see insets), showed the clearest low-pass behavior without exceeding the dynamic range of cortical neurons. The corresponding value of the corticocortical weight is indicated in parentheses in the insets. All these weights have been scaled to the values used for the VRCs in bold solid lines. (D) VRCs of cells a, b, and c (see inset) of Figure 1B. Only neuron a receives first-order LGN input; b and c are second-order neurons. The weights of the geniculocortical and the corticocortical synapses are 1.5 and 1.4 times those used in the bold solid VRCs of A, B, and C.

velocity (dcarees/ I)

D

940


To achieve a 5-times increase in time constant, a must take a value of 0.8. Since the dynamic range of cortical neurons is about half that of LGN cells (Orban et al. 1985), b must be about 0.1 (for steady-state conditions). Thus as a first estimate, the ratio of intra- over subcortical excitatory input conductances should be about 8 in order to produce the desired low-pass filtering. This is precisely the value Douglas and Martin (1991) used, based on ratios of corticocortical over geniculocortical synapses in anatomical studies, in order to simulate their intracellular recordings. In fact, this value of 8 underestimates the low-pass filtering performed by the cortex itself. The time constants measured for cortical cells describe the low-pass filtering between the visual stimulus (not the LGN, which is temporally band-pass) and the cortical response. Moreover, this ratio can hardly be increased because of stability conditions. To ensure stability, Douglas and Martin (1991) recently proposed a microcircuit in which cortical amplification is controlled by inhibitory synapses. We incorporated this feature in our model of Figure 1A (now cell B and cell D inhibit all cells including themselves) and were able to reproduce our VRCs by concomitantly increasing the weight of the feedback excitation, and hence the number of excitatory corticocortical synapses. So our conclusions concerning the transformations in the spatiotemporal domain presented in this paper are not changed by this inhibitory control. Given the above assumption about the dynamic range of cortical and geniculate neurons, the ratio of the corticocortically over geniculocortically induced integrated conductances derived from the simulations of Figure 2C was about 8.1, a value close to the theoretical prediction (see Appendix). Mutual inhibition between neurons receiving input of opposite polarity causes neurons to integrate the difference between their inputs with a time constant that depends on the weight of the inhibition, while common input components are transmitted with a very short time constant (Cannon et al. 1983). This mechanism, proposed for temporal integration of velocity signals in the oculomotor centers, works only as long as the neurons’ input-output function can be linearized around both neurons’ activity values, i.e., it requires a high spontaneous activity or a high common input. Simple cells have little spontaneous activity, but some experiments indicate a high convergence of ON- and OFF- LGN input (Sillito 1975). We were able to exploit this mechanism to shift the upper cutoff velocity of simple cells toward lower speeds in a circuit of analog neurons, but failed to reproduce this result with the more detailed neuron model used in this paper. Temporal integration by synapses with a long time course has been demonstrated to be necessary for normal visual stimulation of the cortex (Miller et al. 1989). These authors found in all cortical layers, after infusion of the cortex with the NMDA-antagonist APV, a high correlation between a neuron’s excitability by visual stimuli and by locally applied NMDA.


941

This temporal integration ( T l ) can be approximated for continuous spike trains of duration T as

This is a sigmoid function with maximum at TI = 1 and values TI(ntr) = 1- [ ( n l)/e"] for integer ti. The assumption of continuous inputs holds, since cortical cells are driven by at least tens of LGN cells (Martin 1988). The greater the time to peak N of the synapse, the longer the duration T of the afferent spike train must be to prevent response attenuation, and the more slowly the EPSP builds up. Synapses with long time courses can at best yield a plateau EPSP of the same amplitude as the EPSP peak in short-time synapses. Temporal integration has been suggested to occur at the level of the geniculocorfical synapse (Miller et al. 1989), however, it will have a stronger low-pass effect if it occurs corticocortically: the time course of the corticocortical synapse determines the range of velocities for which the corticocortically induced EPSP coincides with the geniculocortically induced EPSP. Our measurements show that in the velocity domain, optimal timeto-peaks for corticocortical synapses, for area 17 as well as for area 18 cells, range between 8 and 16 msec. The induced EPSPs peak then between 16 and 32 msec and these values fall within the range of NMDAinduced EPSPs (Jones and Baughman 1988). For shorter time courses, the responses at medium and high velocities are amplified as well, and the limited dynamic range of (cortical) neurons prevents a low-pass behavior. For larger time-to-peak values, the corticocortically generated EPSPs peak when the geniculocortically built up EPSPs are already fading away, except at very low speeds. However, we did not model voltage-dependent conductances and so are not able to draw further conclusions on the role of NMDA receptors in the corticocortical synapses. Since low-pass cells are rare in area 18 of the cat (Orban et a / . 1981b; Movshon et al. 1978), which receives exclusively transient Y-cell input from the LGN, we also modeled transient-input simple cells. Within the parameter range eliciting low-pass behavior in sustained-input simple cells, the VRCs of transient-input simple cells became tuned to a low speed and were not low-pass. It is noteworthy that this difference in velocity properties is exclusively due to the difference in input, and not to a difference in the time constant of cortical model cells. This fits with recent observations of Martin's group who failed to observe a difference in time constants between area 17 and 18 cortical cells (K. Martin, personal communication). Although the distribution of sustained and transient cells is organized differently in the monkey, our model applies equally to the parvocellular and magnocellular systems, and to the generation of velocity low-pass cells in central areas V1 and V2 of the monkey (Orban cf al. 1986). Indeed, the temporal frequency response functions of our modeled sustained and

+


942

transient geniculate cells are close to those of parvo- and magnocellular geniculate cells (Derrington and Lennie 1984; Hicks et al. 1983). Moreover, low-pass cells were observed in area V1 of the monkey mainly outside the magnocellular recipient layers (Orban et al. 1986). We conclude that our current model for temporal low-pass filtering by area 17 simple cells is both biologically plausible and computationally powerful, although contributions from other mechanisms such as spatiotemporally oriented filters (McLean and Palmer 1989), or inhibition by high-pass cells (Duysens et al. 1985b), cannot be excluded.

Appendix 0

Half-wave rectification L

0

n-function g ( t ) with time to peak o

0

Geniculate input

W(t) Wi(t)

=

Si(X.y)

=

Ti(t)

=

=

* = convolution, i = center, surround, sa = spontaneous activity contribution, K = 1, L = 0.6 (sustained) or 0.9 (transient cells), crCenter = 0.3 deg (sustained) or 0.6 deg (transient cells), gsurround = 3 gcenterThe biphasic impulse function T generates rebound responses in both center and surround, and a temporal as well as velocity band-pass behavior. In the n-functions @,,I and g;,z): Qcenter.1 = 8 msec, usurround,l = 16 msec, a1.2= 2 Q , . I . Spikes are generated by a Poisson process (Worgotter and Koch 1991): P(spike, t)

= poAtW(t)

The peak firing frequency of the geniculate input was about 500 spikes/sec for sustained input and about 800 spikes/sec for the transient input. Different strengths of this geniculate input were obtained (Fig. 3C) by changing the value of PO. At = 0.5 or 0.25 msec.

Cortical Low-Pass Filtering 0

943

Cortical cells

r e s ~ > o ~2 ~msec, ~ ~ elseP(spike.t ) P(spike,f ) = ~ ~ ~ t ~ ~ ~ ( t ) - E t ifh f-tsplke = 0. V = membrane potential in mV relative to rest, Ethreshold = 15 mV, E,,, = 100 mV, Elnh = -30 mV, E A H p = -30 mV, T = 16 msec, gleak = 0.125 /is.The following conductances (with time to peak f v and weight factor w between brackets) were used for Figure 2C and G: fast (0 = 2 msec, w = 15) and slow (80 msec, 30) inhibition, fastdecaying (2 msec, 10) and medium-duration (32 msec, 20) AHPs in excitatory neurons, fast-decaying (1 msec, 20) AHP in inhibitory neurons, geniculocortical (1 msec, 0.8) excitation and corticocortical (16 msec, 15) excitation. 0

From these parameter values and from the computed cortical cell response, the ratio of the corticocortical (CC) over the geniculocortical (GC) conductances can be derived as (weight CC synapse) x (number of CC synapses) (weight GC synapse) x (number of GC synapses) with (number of GC synapses) =

(response geniculate afferent pool) (response single LGN afferent)

and, using the results of Orban et al. (1985), (response single LGN afferent) = 2 x (response single cortical cell) For the simulations presented in Figure 2C, this yields 15.0 x 1.0 = 8.08 0.8 x [515/(2 x 111)]

Acknowledgments We are indebted to G. Meulemans for preparing the figures, to M. Van Hulle for his advice concerning numerical problems, and to T. Tollenaere and S. Raiguel for reading the manuscript. This work was supported by Grant RFO/Al/Ol from the Belgian Ministry of Science to G.A.O.

944


References

Baker, C. L, Jr. 1988. Spatial and temporal determinants of directionally selective velocity preference in cat striate cortex neurons. J. Neuropkysiol. 59, 15571574. Cannon, S. C., Robinson, D. A,, and Shamma, S. 1983. A proposed neural network for the integrator of the oculomotor system. Bid. Cybern. 49, 127136. Connors, B. W., Gutnick, M. J., and Prince, D. A. 1982. Electrophysiological properties of neocortical neurons in vitro. J. Neurophysiol. 48, 1302-1320. Connors, B. W., Malenka, R. C., and Silva, L. R. 1988. Two inhibitory postsynaptic potentials, and GABAA and GABABreceptor-mediated responses in neocortex of rat and cat. J. Physiol. (London) 406, 443468. Derrington, A. M., and Fuchs, A. F. 1979. Spatial and temporal properties of X and Y cells in the cat lateral geniculate nucleus. J. Physiol. (London) 293, 347-364. Derrington, A. M., and Lennie, P. 1984. Spatial and temporal contrast sensitivities of neurones in lateral geniculate nucleus of macaque. J. Physiol. (London) 357, 219-240. Douglas, R. D., and Martin, K. A. C. 1991. A functional microcircuit for cat visual cortex. J. Physiol. (London) 440, 735-769. Duysens, J., Orban, G. A., and Cremieux, J. 1985a. Velocity selectivity in the cat visual system. 11. Independence from interactions between different loci. J. Newophysiol. 54, 1050-1067. Duysens, J., Orban, G. A., Cremieux, J., and Maes, H. 1985b. Velocity selectivity in the cat visual system. 111. Contribution of temporal factors. J. Neurophysiol. 54, 1068-1 083. Gulyas, B., Lagae, L., Eysel, U., and Orban, G. A. 1990. Cortifugal feedback influences the responses of geniculate neurons to moving stimuli. Exp. Bruin Res. 79, 441-446. Hagihara, K., Tsumoto, T., Sato, H., and Hata, Y. 1988. Actions of excitatory amino acid antagonists on geniculo-cortical transmission in the cat’s visual cortex. Exp. Bruin Res. 69, 407-416. Hicks, T. P., Lee, B. B., and Vidyasagar, T. R. 1983. The responses of cells in macaque lateral geniculate nucleus to sinusoidal gratings. J. Physiol. (London) 337,183-200. Jack, J. J. B., Noble, D., and Tsien, R. W. 1975. Electrical current flow in excitable cells. Clarendon Press, Oxford. Jones, K. A,, and Baughman, R. W. 1988. NMDA- and non-NMDA-receptor components of excitatory synaptic potentials recorded from cells in layer V of rat visual cortex. J. Neurosci. 8, 3522-3534. Lee, B. B., Elepfandt, A., and Virsu, V. 1981. Phase responses to sinusoidal gratings of simple cells in cat striate cortex. J. Neurophysiol. 45, 818-828. Maex, R., and Orban, G. A. 1991. Subtraction inhibition combined with a spiking threshold accounts for cortical direction selectivity. Proc. Nutl. Acad. Sci. U.S.A. 88, 3549-3553.


945

Martin, K. A. C. 1988. From single cells to simple circuits in the cerebral cortex. Quaf.J. Exp. Physiol. 73, 637-702. McLean, J., and Palmer, L. A. 1989. Contribution of linear spatiotemporal receptive field structure to velocity selectivity of simple cells in area 17 of cat. Vision Res. 29, 675479. Miller, K. D., Chapman, B., and Stryker, M. P. 1989. Visual responses in adult cat visual cortex depend o n N-methyl-D-aspartate receptors. Proc. Nntl. Acnd. Sci. USA 86,5183-5187. Movshon, J. A., Thompson, 1. D., and Tolhurst, D. J. 1978. Spatial and temporal contrast sensitivity of neurones in areas 17 and 18 of the cat's visual cortex. I. Ph!/siol. (London) 283, 101-120. Orban, G. A. 1984. Neuronal operations in the visual cortex. In Sfndirs of Braiii F~inction. Vol. 2 2 , H. B. Barlow, T. H. Bullock, E. Florey, 0. J. Griisser, and A. Peters, eds. Springer-Verlag, Berlin. Orban, G. A. 1991. Quantitative electrophysiology of visual cortical neurones. In Vision and Visual Dysflinctioii. Vol. 4. The Naira/ Basis of Visual Fccnction, J. Cronly-Dillon, Gen. ed., and A. G. Leventhal, ed. Macmillan, London. Orban, G. A., Kennedy, H., and Maes, H. 1981a. Response to movement of neurons in areas 17 and 18 of the cat: velocity sensitivity. J. Neirrophysiol. 45, 1043-1 058. Orban, G. A., Kennedy, H., and Maes, H. 1981b. Response to movement of neurons in areas 17 and 18 of the cat: Direction selectivity. J. Nwrophysiol. 45, 1059-1073. Orban, G. A., Hoffmann, K.-I?, and Duysens, J. 1985. Velocity selectivity in the cat visual system.1.Responses of LGN cells to moving bar stimuli: A comparison with cortical areas 17 and 18. J. Neurophysiol. 54, 1026-1049. Orban, G. A,, Kennedy, H., and Bullier, J. 1986. Velocity sensitivity and direction selectivity of neurons in areas V1 and V2 of the monkey: Influence of eccentricity. J. Neicrophysid. 56, 462480. Palmer, L. A., and Davis, T. L. 1981. Receptive-field structure in cat striate cortex. J. Neurophysiol. 46, 260-276. Schwindt, P. C., Spain, W. J., Foehring, C. E., Stafstrom, C. E., Chubb, M. C., and Crill, W. E. 1988. Multiple potassium conductances and their functions in neurons from cat sensorimotor cortex in vitro. J. Neurophysiol. 59, 424449. Sillito, A. M. 1975. The contribution of inhibitory mechanisms to the receptive field properties of neuroiies in the striate cortex of the cat. J. Physiol. (London) 250, 305-329. Worgotter, F., and Holt, G. 1991. Spatiotemporal mechanisms in receptive fields of visual cortical simple cells: a model. J. Neurophysiol. 65, 494-510. Worgotter, F., and Koch, C. 1991. A detailed model of the primary visual pathway in the cat: Comparison of afferent excitatory and intracortical inhibitory connection schemes for orientation selectivity. J. Neurosci. 11, 1959-1979. -~

Received 18 June 1991; accepted 1 December 1991

This article has been cited by: 2. Paul Mineiro , David Zipser . 1998. Analysis of Direction Selectivity Arising from Recurrent Cortical InteractionsAnalysis of Direction Selectivity Arising from Recurrent Cortical Interactions. Neural Computation 10:2, 353-371. [Abstract] [PDF] [PDF Plus] 3. Stefan Wimbauer, Wulfram Gerstner, J. Leo Hemmen. 1994. Emergence of spatiotemporal receptive fields and its application to motion detection. Biological Cybernetics 72:1, 81-92. [CrossRef]

Communicated by Stephen Gallant

A ”Thermal” Perceptron Learning Rule Marcus Frean Physiological Laboratory, Downing Street, Cambridge CB2 3EG, England

The thermal perceptron is a simple extension to Rosenblatt’s perceptron learning rule for training individual linear threshold units. It finds stable weights for nonseparable problems as well as separable ones. Experiments indicate that if a good initial setting for a temperature parameter, To, has been found, then the thermal perceptron outperforms the Pocket algorithm and methods based on gradient descent. The learning rule stabilizes the weights (learns) over a fixed training period. For separable problems it finds separating weights much more quickly than the usual rules. 1 Introduction

This paper is about a learning rule for a linear threshold unit, often called a perceptron. This is a unit connected by variable weights to a set of inputs across which patterns occur. In the following the ith element of an input pattern (a binary or real number) is denoted by

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

Recommend Documents