Automaton theory and modeling of biological systems

0 Automaton Theory and Modeling of Biological Systems 0 M. L. TSETLIN Translated by Scitran (Scientific Translation S...

Author: M. L. Tsetlin | Mikhail Lʹvovich T︠S︡etlin

83 downloads 1425 Views 5MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

0

Automaton Theory and Modeling of Biological Systems 0

M. L. TSETLIN Translated by Scitran (Scientific Translation Service) Santa Barbara, California

ACADEMIC PRESS New York and London A Subsidiary of Harcourt Brace Jovanovich, Publishers

1973

COPYRIGHT 0 1973, BY ACADEMIC PRESS,WC. ALL RIGHTS RESERVED. NO PART OF THIS PUBLICATION MAY BE REPRODUCED OR TRANSMITTED IN ANY FORM OR BY ANY MEANS, ELECTRONIC OR MECHANICAL, INCLUDING PHOTOCOPY, RECORDING, OR ANY INFORMATION STORAGE AND RETRIEVAL SYSTEM, WITHOUT PERMISSION IN WRITING FROM THE PUBLISHER.

ACADEMIC PRESS, INC.

111 Fifth Avenue, New York, New York 10003

United Kingdom Edition published by ACADEMIC PRESS, INC. (LONDON) LTD. 24/28 Oval Road. London N W l

LIBRARY OF CONGRESS CATALOG CARD NUMBER: 72-11341

PRINTED IN THE UNITED STATES OF AMERICA

Automaton Theory and Modeling of Biological Systems. Translated from the original Russian edition entitled Issledovaniya PO Teorii Avtomatov i Modelirovaniyu Biologicheskikh Sistem, published by “Nauka” Press, Moscow, 1969.

Mikhail L’vovich Tsetlin (1924-1966)

Contents

Foreword . .

xi

Pufau lq the Russian EditWn

xiii

AUTOMATON THEORY Finite Automata and Modeling the Simplest Fonns of Beluorior Introduction . . . . . . . . . . . . . . . . . . . . . .

3

I. BEHAVIOR OF AUTOMATA IN RANDOM MEDIA

1 Behavior of Automata in Stationary Random Media . . . . . . . . . . . 2 Asymptotically Optimal Sequences of Symmetric Automata. The Book Stack Problem

. . . . . . . .

12 16

3 Behavior of Automata in Composite Media . . . . . . . . . . . . . . 4 Behavior of Automata with an Evolving Structure in Random Media . . . Appendix to Part I. Eigenvalues of Markov Chains Describing lhe Behavior of Asymptotically Optimal Automata in Stationary Random Media . . . . . .

2S

3i 34

II. AUTOMATON GAMES. ZERO-SUM GAMES FOR TWO AUTOMATA 1 Automaton

Game~

41

. .

2 Zero-Sum Games for Two Automata . . . . .

46

Ill. HOMOGENEOUS AUTOMATON GAMES

1 Homogeneous Games . . . . . . . . . . . . . . . . . 2 An Example of Simulating a Symmetric Automaton Game 3 Circle Automaton Games . . ." . . . . . . . . . . . . .

SS 62 71

An EXlllllple or Modeling the Beluovior or a Group or Automata with a

Two-Level Organization (The Numerical Method Distributioo Problem) vii

84

viii

Behavior of Automata in Periodic Random Media and the Problem of Synchronization in the Presence of Noise.

93

Organization of the Queuing Discipline in Queuing Systems Using Models of the Collective Behavior of Automata . 102 Mathematical Modeling of the Simplest Forms of Behavior

108

Appendix I. Addr'e$Siess (N onjndividuali:zed) Control , , . , . , , Appendix 2. Languages That Automata Use to Communicate with One Another

124 J2S

ARTICLES ON BIOLOGICAL SYSTEMS AND MATHEMATICAL MODELS IN BIOLOGY Introduction . . . . . . . . . . . . . . . .

129

Mathematical Simulation of the Principles of tbe Functioning of tbe Central Nervous System I On Search Tacues , 4 The Principle of Least Interaction . . . . . . . . . . . . .

Coatinuous Models of Control Systems . . . . • .

.

IS4

Certain Problems in the Investigation of Movements I Synergies and Other Mechanisms Simplifying Motor Control 2 Functioning of Motor Control on the Spinal Level . . . 3 Functioning of a Motoneuron Group and Motor Units . . .

162 16S 169

Computer Simulation of the Functioning of a Motoneuron Group I

Posc~Regimc

of Motor Unit Operation

. . . . . . . . . . . . . . .

172

. . . . . . . . .

174

2 Description or the Model

3 Desynchroni1,ation of Motoncurons; the Influence of the Renshaw CeiJ System on Impulse Transmiss.:ion of Motoneurons

4 Control or Muscle U:ngth . . . S Simulation of Pathological States . . . .

178

182 184

ix Restructuring Prior to a Movement .

187

Bioelectric Control and Diagnostics of States I Usc or Skeletal Muscle Biopotcntial$ for Control . 2 Usc of Cardiac Biocurrents to Control Diagnostic. Devices

198 200

3 Certain Problems Related to Automatic Diagnosis of Acute Pathological States 210

SUPPLEMENTARY ARTICLES Certain Properties of Finite Graphs Related to tbe Transportation Problem . . . . . . . . . . . . . . . . . . . 221 Application of Matrix Calculus to the Synthesis of Relay-Contact 226 Circuits . . . . . . . . . . . . . Bibliography of Papers by M. L. Tsetlin .

232

APPENDICES On tbe Goore Game . . . . . . . . . • . . . . • . . . . . . . . . . 239 A Simplified Description of Games Played by Asymptotically Optimal Automata 247

I Two-Person Games . . . . . . . . . . . . . . . . . . . . 2 A Remark about Games with an Arbitrary Number of Players

251

The Problem of Controlling a Communications Network . . . . . . . 253 The Operation of tbe Apartment Commission . The "Hey" Problem

.

255

. . . . . . . . . . . . . . . . . . . . . . . . 260

Papers on Continuous Exci.table Media 1 One-Dimensional Excitable MeOia . 2 Two-Dimen$ional Excitable Media .

262 264

X

CoiiiOtt.s

The Restructuring

or the

Operation

or tbe Spinal

~vtl

.

• .

267

References .

269

Author lndu

281

Subj~ct

285

Index

Foreword

For an American practitioner in mathematics, reading the work of Soviet scientists frequently arouses disquieting undercurrents of feeling. Despite the fact that mathematics is the most nearly universal of languages, there remains a difference in style, in motivation, and in the attitude of the mathematician to’ward his work. This collection of the papers of the Russian mathematician M. L. Tsetlin has evoked these feelings in me and I believe that other readers may well respond in the same way. M. L. Tsetlin was a mathematician in the tradition of Gel’fand, Fomin, and Pyatetsky-Shapiro, that is to say, of mathematicians who combine a very high order of mathematical skill and rigor with a concern for the extramathematical implications of their theorems. “Automaton Theory” exhibits the qualities we have come to expect of good Russian writing in mathematics. It is expository, the reasoning is explicitly and clearly presented, the proofs are careful and complete, and the theoretical results are applied to real world problems, in this case to aspects of neurology and biological control. The mathematics itself would hardly be termed automaton theory by American and West European automaton theorists. Tsetlin’s work makes relatively little contact with this line of research as we have come to regard it. Rather it is closer in spirit to the issues raised in game theory and to learning machines or perceptrons. Tsetlin considers collections of automata as a model for collective behavior of a group with no a priori information other than the rules of the game, and attempts to derive the structure of systems that exhibit self-organizing behavior. It is true that the papers refer to only the simplest modes of behavior. One wonders whether it is a reflection of cultural or social differences that Tsetlin chose to study cooperative phenomena in choosing “expedient” behavior, while American game theory focuses on competition among the players. In any case, Tsetlin has made a number of digital models, using a rather eclectic mixture of mathematical tools, to begin to understand interacting collections of automata. xi

xii

Fore word

Over half of the text is devoted to Tsetlin’s work, with Gel’fand and coworkers, on biological systems and mathematical models in biology. The readers might well begin reading this collection with Gel’fand’s introduction to this second section. Here and elsewhere in bits and pieces the reader will sense the feeling of personal loss that Tsetlin’s co-workers felt at his early death. Gel’fand writes inter alia “Unfortunately, this stage of our joint work has been interrupted without ever having seriously begun.” I t was undoubtedly this sense of loss that impelled his colleagues to do the job of careful, extensive, and dedicated editing of Tsetlin’s papers. Tsetlin’s work is unfinished; there are insights, starts, and work in progress in a number of directions that will be interesting to neurophysiologists, psychologists, and sociologists, as well as mathematicians. There is as yet little serious impact by mathematics upon the biological and social sciences other than within the special domain of statistics. “Automaton Theory” tells us about one major Russian attempt to formalize the study of living systems. Tsetlin’s work may only be a beginning but I believe we can learn from it when we try our hand at translating biology’s ineffable problems into mathematics. MURRAYEDEN Cambridge, Massachusetts

Preface to the Russian Edition

On May 30, 1966, Mikhail L’vovich Tsetlin died suddenly at the age of 42. He was one of the most outstanding specialists in the area of cybernetics. During the last ten years of his life, Mikhail L’vovich was mainly occupied with finding the general principles underlying the operation of biological systems, and with development of biocontrolled devices. The friends and colleagues of the deceased have collected his principal works, and this is how this book came into being. It contains the published articles and certain materials from Mikhail L’vovich’s archives, prepared for publication by the editors. The basic text of the articles was fully preserved, with the exception of a small number of changes and omissions which were made to avoid repetition. The order of the materials was determined by the desire to produce a self-contained book. For the same reason, the literature quoted in M. L. Tsetlin’s papers and in the appendix has been put together in a single list placed at the end of the book (references to it are in the form of numbers in brackets). M. L. Tsetlin’s works are grouped in three parts. The first part, Automaton Theory, is devoted to a study of the modeling of expedient behavior with the aid of automata. The second part contains the basic papers on the general principles underlying the functioning of biological systems. The basis for the article on bioelectric control was provided by the materials which Mikhail L’vovich prepared together with others for the purpose of writing a separate book. The materials were put in their final form by V. S. Gurfinkel’ after the death of Mikhail L‘vovich. The predominant position in the book is occupied by the lecture, “On the mathematical modeling of the simplest forms of behavior,” which was placed at the end of the first part. This lecture, delivered by Mikhail L’vovich to physiologists, is a simple and clear exposition of the basic ideas discussed in the first part. It is perfectly understandable to persons not having a specialized mathematical training. At the same time, in our opinion, this xiii

xiv


lecture is also of great value to mathematicians who are interested in the problems discussed in the first part. We recommend that the readers become acquainted with the lecture before reading the first part. In addition, the lecture is essentially an introduction to the second part of the book. The third part of the book contains two additional articles, which are not directly related to the subject matter of the first two parts, and a bibliography of papers by M. L. Tsetlin. An appendix, placed at the end of the book, contains the results obtained by Mikhail L’vovich’s students, as well as questions and problems which worried M. L. Tsetlin in the latter years of his life, but were not expressed in his publications. When speaking of the first part of the book, which is devoted to the mathematical problems of a theory of expedient behavior of one individual or a group, it must be noted that in his works M. L. Tsetlin did not strive to model any kind of specific behavior, but instead posed himself the task of explaining the general laws of expedient behavior, in particular, the expedient behavior of groups. Expedient behavior is really an adaptation to the external world. An individual is capable of performing one of a finite number of actions. Whether he wins or loses depends on which action he performed. Should the individual know beforehand for which action he is going to receive the maximum payoff, he would then naturally always perform this most advantageous action, and this type of behavior would then be, of course, most expedient. In reality, however, an individual usually does not have a priori information about which action will be most advantageous to him. In Tsetlin’s works an individual finds himself in “an external world” which is constructed in the following way (for a more precise treatment see pp. 13-14 of the present edition): in response to each action of the individual, he wins with a certain probability and loses with a certain probability. This model of “an external world” is called a stationary random medium. Mikhail L’vovich proposed the construction of an automaton which minimizes the number of unfavorable reactions of the external world. Such an automaton-and this is most remarkable-does not possess any a priori information about the parameters of the stationary random medium with which it is interacting. In the words of Mikhail L’vovich, this problem is the simplest model of “a small living creature in the big world around him.” Mikhail L’vovich’s works, dealing with the expedient behavior of a collective, permit us to approach the expedient behavior of extremely


xv

diverse groups from a single point of view, and to view them as composed of individual cells or nerve centers, or even individual persons. These works gave rise to the theory of automaton games. Contrary to the classical theory of games developed by J. von Neumann, in which players from the very outset know the consequences of various actions (their own and the opponents’), M. L. Tsetlin proposed to investigate a situation in which the players d o not have any a priori information about the numerical values of the parameters of the game. It was found that automata with a linear tactic which successfully dealt with the problem of behavior in a stationary random medium also dealt successfully with a game played under these conditions. This concept made it possible for Mikhail L’vovich to consider not only two-player games, but also games with a larger number of players. Then Mikhail L’vovich applied the principles, developed by him in the theory of automaton games, to various problems. The second part of the book is devoted to the general principles underlying the functioning of biological systems. In contrast with the preceding part which discusses the mathematical theory with sharply delimited boundaries, the articles contained in the second part represent rough sketches of some future theory. They were written by M. L. Tsetlin together with I. M . Gel’fand, V. S. Gurfinkel’, Yu. B. Kotov, M. L. Shik, and others. Basically, these articles deal with the problem of control of movements, and with the principles uilderlying the operation of the nervous system. The approach toward the modeling of biological systems, developed in them, is connected with the theory of automaton games which were discussed in the first part of the book. In addition, this part expounds the principles of bioelectric control. On the basis of the information about electric processes in organisms, it became possible to create a new type of controlled systems in which the role of control signals is played by biopotentials. Isolated attempts of this type were also made earlier, but it was only after the fundamental papers by Soviet authors that the problem of biocontrol became widely investigated. The creation of the first operating models of prostheses and manipulators, based on biocontrol, provided a powerful stimulus, and now this problem is being investigated by dozens of laboratories in various countries. M. L. Tsetlin was one of the creators of the principles of biocontrol, and a direct participant in the construction of the first models. He devoted a great deal of attention to the application of biocontrol principles for the development of new diagnostic instruments. He developed devices providing a synchronization between the switching of diagnostic instruments and the phases of the cardiac cycle, and devices

xvi


for automatic control of the monitoring and registering apparatus used in disturbances of the cardiac rhythm. He also developed principles for a system of automatic observation and diagnosis of the state of patients. The preparation of this book for publication after the death of Mikhail L’vovich was undertaken by M. M. Bongard, I. M. Gel’fand, V. S. Gurfinkel’, I. I. Pyatetskiy-Shapiro, N. N. Chentsov, M. L. Shik. The appendix was prepared by M. B. Berkinblit, A. V. Butrimenko, D. I. Kalinin, V. I. Krinskiy, B. G. Pittel’, V. A. Ponomarev, L. I. Rozonoer, V. L. Stefanyuk, A. L. Toom, I. M. Epshteyn. The editing of the book was done by M. B. Berkinblit and A. L. Toom. The work they have performed went far beyond the usual editorial duties. M. BONGARD I. PYATETSKIY-SHAPIRO N. CHENTSOV

AUTOMATION THEORY

Finite Automata and Modeling the Simplest Forms of Behavior'

Introduction The simplest behavior problem discussed in the present chapter can be formulated as follows: We consider an automaton, i.e., an object capable of receiving a finite number of signals s E (sl, s 2 , . . . ,s , ~ ) ,at every instant of time f = 1, 2 , . . . , and changing its internal state in accordance with these signals. The automaton can carry out a finite number of actions f E (fl,h,.. . ,fJ.The choice of the action is determined by the internal states cp E (cpl, q 2 , .. . ,pm); the number m is called the memory capacity of the automaton. It is assumed that the automaton is situated in a certain environment and that the actions of the automaton cause responses s of the medium C. These responses are, in turn, the input signals of the automaton. It uses them, as it were, to decide its further actions. In this chapter, we confine ourselves to the simplest case in which the automaton treats all the possible reactions of the environment s E (sl,. . . , sN) as belonging to one of two classes: that of favorable reactions (corresponding to a payoff, s = 0), and that of unfavorable reactions (corresponding to a loss, s = 1). Within each of these classes, the responses of the environment are indistinguishable for the automata. The expedient behavior of the automaton in a given environment consists of increasing the number of favorable responses and diminishing the unfavorable ones. The doctoral dissertation of M. L. Tsetlin defended in 1964 at a division of the V. A. Steklov Mathematical Institute (postscript of the compiler, post. comp.). 3

4

Automaton Theory

In what follows, we shall consider deterministic and stochastic automata.2 = F(g;(t)) giving the relationship between an actionf(t) of the automaton at time t and its state p(t), and the stochastic matrix 1) a j j ( s )11, i , j = I , 2,. . . , m. Here aij(s)is equal to the probability of a transition from state v(t)= vi into state v(t+I ) = y j , due to input s ( t I ) . I n the case of deterministic automata, the matrices 1 1 a i j ( s ) 11 consist of zeros and ones. Since we consider automata that are capable of receiving only two signals (s = 0) and (s = I ) , it is sufficient to specify two such matrices I( a i j ( 0 )( 1 and I( a i j ( l )11. Part I deals with behavior of automata in random environments; Parts I1 and 111 deal with various problems related to the collective behavior (games) of the automata. The first section of Part I deals with the simplest problem arising here: the behavior of automata in stationary random environments [154, 15-51. In such environments, the probabilities of payoff or losing are defined for each possible action of the automata. I t is shown that the behavior of an automaton in a stationary random environment can be described by means of a finite Markov chain. Usually, this chain is ergodic. Then there exist final probabilities of states, and this makes it possible to define the mathematical expectation of the automaton payoff, which does not depend on its initial state. This quantity can serve as a measure of the expediency of behavior in a given stationary random environment. The symmetric finite automata, i.e., those automata in whose construction any apriori information about random environments is deliberately omitted, are of special interest. A question arising naturally is that of the existence of asymptotically optimum sequences of symmetrical automata \a,. . . , %,!,. . . , i.e., sequences such that the expected payoff for automata belonging to such a sequence tends, when n-,co, to the maximum possible value in a given random environment. Here n can be interpreted as the number of states of the automaton (the capacity of its memory). When the capacity of its memory is sufficiently large, the automaton carries out, with a probability close to 1, the action that maximizes the probability of winning. Section 2 of Part I presents several constructions of asymptotically optimcm sequences of finite symmetric automata advanced by Krinskiy [102], Krylov [112], Ponomarev [128], and the author [155]. The same section contains a formulation of the necessary condition for the asymptotic optimality of a sequence of finite automata. An automaton is specified by an equation f(t)

+

One can familiarize oneself in more detail with the automaton theory from books by Kobrinskiy and Trakhtenbrot [97] and by Glushkov [70].

Finite Automata and Modeling the Simplest Forms of Behavior

5

The constructions of stochastic automata maximizing the expected payoff with a fixed capacity of the memory were discussed by Milyutin [123], who had indicated such constructions and some of their principal properties. In particular, Milyutin has shown the optimality of automata with linear tactics in some stationary random environments. In this reference Milyutin used a method, suggested by him and Dubovitskiy, of solving extremum problems in the presence of constraints [88, 891. The same section contains a discussion of “the problem of a pile of books,” an example of a system whose behavior largely coincides with the behavior of a finite automaton in a stationary random environment, and which is expedient in a certain sense. The constructions of automata described in Section 2 assure expedient behavior in the simplest case, that of stationary random environments. It seemed natural to us to study their behavior in environments which are not stationary. Section 3 describes the behavior of automata in environments termed composite, i.e., in random environments whose probabilistic properties depend on time as determined by a Markov chain. In composite environments an automaton is forced to continually “evolve,” and the time during which its structure “evolves” greatly influences the expediency of its behavior. For this reason the dependence of the expected payoff to the automaton on the capacity of its memory loses its monotonic character in composite environments. In particular, this section contains a computation of the expected payoff for the case when an automaton with linear tactics, described in Section 2, interacts with a composite environment. It ttirns out that the expected payoff attains a maximum for a fixed capacity of memory, and decreases as this capacity further increases. Owing to the existence of an optimum memory capacity, it is possible to select those automata having a specific design which have the greatest expectation of winning in a given composite environment. Algorithms of optimum behavior in composite environments were studied by Dobrovidov and Stratonovich [87]. Varshavskiy and Vorontsova investigated the behavior of stochastic automata with evolving structures [43, 441. At first, the behavior of these automata is not expedient, but as the responses of the environment are fed into them, these automata change their matrices 11 a i j ( s )11 in such a way that their behavior becomes increasingly expedient during the experiment. The behavior of such automata in stationary and composite random environments was investigated by means of modeling on a digital computer [43]. Methods of structural evolution were found which ensured optimum behavior in stationary random environments. In composite environments, the mean payoffs for automata with evolving structures coincide with those for

6

Automaton Theory

automata with linear tactics and optimum memory capacity. The results obtained by Varshavskiy and Vorontsova are briefly presented in Section 4, Part I. Part I of this chapter deals with the behavior of automata in those random environments whose properties are assumed either to be constant or to change independently of the actions of the automata. Such forms of behavior are sometimes described as “games against nature.” The following two parts of this chapter deal with questions connected with the collective behavior of automata that is produced by their interaction. A convenient way of defining simple forms of such interaction is provided by the terminology of game theory [24, 41, 121, 1221. The use of such terminology narrows down the class of the forms of behavior studied, but at the same time, it leads to the construction of a number of meaningful models. It should be noted that the automaton games are discussed here from a viewpoint that differs from the one accepted in game theory. Indeed, it is normally assumed in the latter that the game is defined by a system of payoff functions previously known to the players. Using this apriori information as well as arbitrarily chosen means of computation, the player selects his strategy. Moreover, each player assumes that his opponents play in the best possible way. The (usually mixed) strategies that are selected in this manner remain constant during the game. The game resembles a game of chess which begins and ends with an analysis of the situation. We thought it interesting to consider games played by finite automata having n o a priori information about the game, and being forced to shape their strategies for each successive replay in the course of the game itself [113, 156-1581.

Automata games are defined in the first section of Part 11. It is assumed that such games are repeatedly played numerous times. A play f of a game G is called a set f = (fj,, f:* ,. . . ,flJ of strategies (actions) selected by the . . , ,u’ playing the game. An outcome s of a playf is the automata %l, ?I2,. set s = (sl,s2,. . . , sy), sj = 0, 1, i = I , 2,. . . , Y. Here, sj = 0 if an automaton %j won the play f, and sj = 1 if this automaton lost the play f. The information about the constructions of the automata playing the game and the probabilities p ( f , s) of the outcomes of the plays defines the game. Thus, the information about the payoff or loss in a play determines the input variable for each of the playing automata, and so determines the choices of actions (strategies) in the plays that follow. The information received by an automaton during the course of the game is made complete by the information about the outcomes of its individual plays. An automaton


7

game defined in this manner is equivalent to a game defined in the usual sense of game theory. Conversely, an automaton game can be constructed on the basis of a given system of payoff functions. For automaton games described as games with independent outcomes, such a construction is unique. The same section proves that an automaton game can be described by a (finite) Markow chain. As a rule, the chain is ergodic. This makes it possible to single out the important class of ergodic games in which there exist welldefined final probabilities (independent of the initial conditions) of winning for each of the automata participating in the game. The section ends with an example of a nonergodic game. The second section of Part I1 is devoted to zero-sum games for two automata. The interest of this case lies in the fact that the equivalent zerosum game for two persons is the subject of von Neumann’s well-known theorem. A game played by an automaton against an opponent who chooses a mixed strategy (in the sense of the game theory) is considered first. It turns out that an automaton belonging to an asymptotically optimum sequence maximizes (if the capacity of its memory is sufficient) its payoff. It obtains a payoff equal to the value of the game in von Neumann’s sense when its opponent chooses an optimum strategy. In other words, such an automaton “plays no worse” than its opponent, choosing an optimum mixed strategy. Behavior that is expedient in a stationary random environment turns out to be expedient in this case also. When choosing the design of playing automata, it is natural to require that in any case their behavior should be expedient in the simplest game, i.e., in a game against nature. The simple automata whose behavior is studied in the present book have no information available about the actions of their opponents, about the strategies that are available to them, or even about the number of opponents. To an automaton, the role of the opponent reduces to forming a more or less complicated environment in which it must behave in an expedient manner. Next, a zero-sum game for two identical automata with linear tactics is discussed in detail in Part 11, Section 2, and the final payoff probabilities for n 00 are computed. It turns out that in the case of an obvious advantage for one of the players (the presence of rows of nonnegative elements in the matrix I mmBI I of the game), it chooses the strategy yielding the maximum guaranteed payoff, so that its behavior is very similar to that which is prescribed by game theory. However, if there is no such obvious advantage, the expected payoff for each player is zero. Automata with linear tactics play, as it were, in a rough way, being unable to take advantage of the subtler

-

I

8

Auromaron Theory

properties of the matrix of the game. These investigations were carried out by the author jointly with Krylov [113, 1581. Part 111 deals with homogeneous games of automata, i.e., games in which all participants have equal rights [32, 591. In the first section of Part 111, a definition of the group G , of the automorphisms g of a game r* is given. Those games are homogeneous for which the automorphism group is transitive over the set of players. In homogeneous games, the sets of strategies of the players are pairwise isomorphic. The invariant sets of game plays are determined with the aid of the automorphism group of a homogeneous game. Here, the arithmetic mean of the payoffs for all players in a given play (value of the play) coincides with the arithmetic mean of the payoffs for any one player over an invariant set of plays (with the value of the invariant set). A Nash play of a game is defined as a play in which it is not convenient for any player to change his strategy, assuming that no other player changes his. In this sense, Nash plays are stable. Games consisting of Nash plays are called Nash games. The invariant sets generated by Nash plays in homogeneous games are called Nash sets. In the study of homogeneous games of automata, it seemed interesting to us to compare, as in the zerosum games for two automata, the behavior of automata with that of peopIe who know the conditions of the game beforehand. In homogeneous games, such behavior is not difficult to predict: people would agree to play sequentially the plays belonging to an invariant set of maximum value. lf the set is a Nash set, then it can be assumed that with a favorable structure of payoff functions the automata will play “no worse.” Games in which the invariant set of maximum value is a Nash set are called Moore games. Section I of Part 111 gives examples of Nash and Moore games. The final part of the section gives a procedure for constructing from a given homogeneous game another game To*which has the same sets of players, their strategies, and the same values of the plays as in the game r*,but is a Moore game. This procedure (we call it the procedure of introducing a “common fund”) is equivalent to an agreement among the players of a homogeneous game about dividing equally their payoffs in the game. The homogeneous games were studied by the author together with Gel’fand and Pyatetskiy-Shapiro. Section 2 of Part I I I discusses homogeneous games for identical automata and an example of modeling a symmetric game of automata (called a game with a distribution [ 6 4 ] )on a digital computer is described in detail. The situation modeled in this game is typical, for example, of the problem in which a predator is selecting his hunting area. Here the number of animals

r*


9

per predator is determined by the supply of game and the number of predators simultaneously present in the area. The use of a certain strategy in the game corresponds to the selection of a hunting area and the various values of the payoff function correspond to the number of prey. The distribution game is defined by x nonnegative numbers, a,, a,, . . . , a,, a, 2 a, 2 . . 2 a, 2 0, called the efficiencies of the strategies. The game is played by the automata % I , . . . , 'u", Y 5 x , each having x strategies fi ,f,,. . . ,f,. An automaton, having chosen a strategyfj in some play of the game, has in this play the expectation of a payoff equal to a j / n j , where nj are the number of automata that have chosen strategyfj in this play. It is not hard to verify that the distribution game is a Nash game. Studying the behavior in this automaton game with a linear tactic, we have found in a number of examples that the payoff to any automaton approaches the value of the Nash set, assuming that the memory capacity n of each increases without a bound; with a probability approaching 1, the automata play those plays that belong to the Nash set. The procedure of introducing a common fund transforms the game of distributions into a Moore game. It turns out that in this case the average payoff of a participant automaton tends to the value of the Moore set (for n + co), i.e., to the maximum average payoff possible in the game. As the memory n of each participant automaton goes to infinity, their collective behavior in the game of distributions does not differ from the behavior of a group of people who know previously the conditions of the game and who have entered into an agreement. In fact, it is obvious that the players who have prior information about the distribution game and agree on joint actions in the game would receive the highest payoff, playing sequentially all the plays in which the strategiesf, , A , .. . ,fy would be used with maximum efficiency. Each one of the strategies is used precisely by one player in each play (such plays form a Moore set). A discussion of the behavior of automata in the distribution game makes it possible to express certain notions about accuracy. The problem is that when a portion of the participating automata leaves the game, those remaining continue to play the most convenient plays (Moore plays for the reduced number of automata). Therefore, independently of which automaton leaves the game, the strategies with highest efficiencies still continue to be used, so that the average payoff to the automata continuing the game increases; this compensates to an extent for the decrease in the total payoff to the group of automata. For finite fixed values n of the memory capacity of the participant automata, the increase of the average payoff to the automata continuing the game is related to the fact that the value of the Moore

-

10

Automaton Theory

set is reached more accurately as the number of the players decreases. The final part of Section 2 in Part 111 contains examples illustrating these ideas. In the course of analyzing games for many automata, it seemed interesting to us to separate out those games whose description does not depend on the number of players. This property is possessed, for example, by games with a limited number of neighbors, where the payoff functions of each player depend only on the strategy chosen by them and on the strategies of a limited number of other players-his “neighbors in the game.” The simplest homogeneous game of this type is a game along a circle with two neighbors in which the payoff function of a player % j depends on his strategy and the strategies of the players %j+l and %j-l, and each of the players has only two strategies, 0 and 1. A description of this game and the results of modeling it on a digital computer form the contents of Section 3. The same section contains a definition of the simplest game on a circle, an analysis of its Nash plays, and a proof of the assertion that a game with an even number of players is always a Nash game. Furthermore, an example is given of modeling on a digital computer a game on a circle that is a Moore game in which the payoff of each of the participating automata reaches the value of the Moore play: for a fixed number of players and an infinite increase of the memory capacity of each of them, the Moore play is played with a probability approaching 1. The value of the Moore play is, however, not reached for all Moore games on a circle. The final part of the section states certain assumptions about the necessary conditions for this to take place. These conditions are imposed on the payoff functions of the game. The assumptions are then verified by means of modeling on a digital computer, and the results of this verification are also given in the same section. The problem of the expedient behavior of a pair of automata was also analyzed in the diploma dissertation of Stefanov [138]. The paper by Borovikov and Bryzgalov [ 2 9 ] is a study of the simplest symmetric automaton game, each having only two actions 0, I , and where the payoff was determined as the fraction of players who chose action 1. Also, the conditions of this game do not depend on the number of players. The behavior of automata in random media and the automaton games to whose study this work is devoted may be viewed as the functioning of self-organizing systems. The theory of the latter (in connection with abstract automaton theory) is investigated in the papers by Glushkov [66,68]. These papers introduce criteria that allow one to judge the capacity of a system for self-organization, as well as quantitative (entropic) character-


I1

istics of self-organization. The actions of automata capable of expedient behavior assure not only a payoff, but also the obtaining of information necessary for the selection of actions. In this sense automata are systems with a dual control according to Fel’dbaum [145, 1461. A paper by Letichevskiy and Dorodnitsyna [I161 describes an interesting model of natural selection which is substantially close to the behavior of automata in random media. A detailed analysis of learning systems different from the ones described here was made in the book by Bush and Mosteller, whose Russian translation is provided with an interesting appendix by Shreyder [ d o ] . It should be noted that the examples of behavior models forming the contents of the chapter refer only to the simplest modes of behavior: It is assumed that the automata are capable of distinguishing between only two possible input signals. The behavior of automata capable of receiving richer and more complex information is considerably more complicated. Numerous studies on the modeling of pattern recognition were devoted to an investigation of certain aspects of such complex behavior. Here we shall mention only the well-known papers of Rozenblatt on perceptrons [130];the papers by Glushkov [67, 691 are devoted to the important aspects of their theory. Recognition algorithms are described by Bongard [25, 261, Ayzerman [ I ] , Braverman “1, Glushkov et al. [71],Kovalevskiy [98],Kharkevich [148], and others. It seems that it would be interesting to study the game behavior of more complex automata, for example, of automata receiving information not only about the outcome of each play, but also about the strategies used in this play by other players. The mathematical models of the simplest forms of behavior described here were used (with Gel’fand) in our joint research in physiology. We have attempted to use them to explain certain peculiarities of the interaction of nerve centers [56]. In their paper Pyatetskiy-Shapiro and Shik [I291 made an attempt to apply these models to the investigation of the spinal mechanisms of motor control. Other attempts at modeling behavior are concerned with the modeling and control of motion in man and higher animals, and are described by Gel’fand and Tsetlin [62],where the basis for the construction of models is provided by the techniques of the automatic search for the extremum of a function of many variables, in particular by the “ravine method” due to Gel’far~d.~ See p. 137 of the present volume (Editor’s note).

12

Automaton Theory

I BEHAVIOR OF AUTOMATA IN RANDOM MEDIA 1 Behavior of Automata in Stationary Random Media

Assume that a deterministic automaton % is described by its canonical equations

+ 1 ) = @(Ql(t),

s(t

f ( t >= F(P(0).

+ 1 I),

(1)

(2)

I n these equations the variable t represents time, and is assumed to take on integer values f = 1, 2,. . . . We shall assume that the input variable s(t) may take on only two values: 0 and I . The value s = 0 will be called the “nonpenalty,” and the value s = I the “penalty” of an automaton (a. We shall further assume that the output variablef(t) of the automaton may take on x different valuesf,, fi,.. . ,f,.The values of the variablef(2) will be called the action of the automaton and we shall say that at the instant t the automaton $11 has performed the uth action if f ( t ) =f,, (1 = . I , 2) . . . , x. I t is also assumed that the variable cp(r) may take on n? different values cp, , q r , .. . , p,,,. These values are called the state of the automaton and the number 111 is said to be its riteniory capacity. Of course ni 2 x. We shall say that the automaton ‘u is, at an instant t, in the jth state, j = I , 2,. . . , in, if y ( t ) = cpj. The action f, is said to correspond to a state qj if F ( y j ) =fa. Equation ( 2 ) describes the relationship between the actions of the automaton and its states, and Eq. ( I ) describes changes in its state due to the action of the input variable s ( t ) . The input variable assumes only two values, so that ( I ) specifies two mappings into itself of the set of states of the automaton; one of the mappings is given for s = 0, and the other for s = I . The mappings will be conveniently written in the form of a special state matrix1 (1 a i j ( s )11, i, j = I , 2 , . . . , in. The state matrix for a deterministic automaton is simple, i.e., each row in this matrix for any fixed value of s contains exactly one element equal to unity, and the remaining elements are zero. The matrix 11 a i j ( s )11 determines the transitions of states for a deterministic automaton in the following manner: if at the instant t the automaton is in state ( p i , then at t 1 it will make a transition to a state vj such that aij(s(t I ) ) = 1.

+

+

For more details, see Tsetlin [150-152], where these matrices are used to construct deterministic automata out of real physical elements. [See also p. 226 (Editor’s note).]


13

The transitions of states for each value of the input variable s can be represented graphically by means of graphs of states. Thus, each state pi of the automaton is juxtaposed with the vertex i of the graph of states, and each nonzero matrix element ajj(s) is juxtaposed with an arrow directed from the vertex i to the vertex j . The simplicity of the matrix aij(s)1 1 implies here that exactly one arrow emanates from each vertex of the graph of states. For automata considered in this study, the transitions of states are described by a pair of such graphs (for s = 0 and s = I). We shall still need the concept of a s/ochastic automaton. A stochastic automaton also has a finite number of states q,, p2,.. . ,pm and a finite number of actionsf, ,,fi2,. . . ,f,. Just as in the case of deterministic automata, described above, we shall assume that the input variable assumes only two values, .F = 0 and s = 1. The actions of a stochastic automaton are uniquely determined by the statef(t) = F ( q ( t ) ) , and the state matrices 1 1 a j j ( s )11, s = 0, are stochastic. Here a j j ( s )signifies the probability of a transition from the ith state to the ,jth state for a given value of the input variable s. Evidently, deterministic automata are a special case of stochastic automata. In what follows we shall simply speak of automata, having in mind both deterministic and stochastic automata. Now we shall begin o u r discussion of the behavior of automata in stationary random media. We shall say that an automaton 21 is in a stationary random medium C = C(a, , a 2 , .. . , a,) if the actions of the automaton and the values of its input variable are related as follows: the action,f,, a = I , 2 , . . . , x performed by the automaton at the instant / generates the value s = I (a penalty) at the instant t I with the probability p, = ( 1 - a,)/2 and the value s = 0 (a nonpenalty) with the probability q, = ( I a,)/2. We assume here that I a, I 5 I . Suppose that at the instant t , the automaton was in statepi, i= I , 2 , . . ., m, to which there corresponds the action f Z l = F(qj). Then the probability p i j of the transition of the automaton from state p i to state pj is given by the formula

+

+

It is not difficult to see that the matrix P = 1 1 p i j 1 1 is stochastic. Thus, the functioning of the automaton in a stationary random medium is described by a Markov chain. In the cases of interest to us, this chain turns out to be ergodic so that final probabilities of automaton states exist in a given medium which are independent of its initial state.

14

Automaton

Theory

Let r i denote the final probability of state y i of the automaton in a stationary random medium C . Let a,, u = I , 2,. . . , x designate the sum of the final probabilities of those states v i , to which there corresponds the action fa. The quantities 0, signify the probabilities of the actionf, of automaton 1’ 1. in the medium C . The mathematical expectation W(%, C ) of a nonpenalty for automaton 3 in the medium C is given by

Obviously,

The expediency of the behavior of an automaton consists in increasing W .

We shall say that an automaton 1 ‘ 1. behaves expediently in the medium C if5

For an automaton performing its actions equiprobably and independently of the reactions of the medium, W = ( I / x ) ( a , a2 . a,). Let us consider the simplest example of an automaton behaving expediently. Consider an automaton L2,2,8possessing two states y1 and y 2 , and two actionsf, = F(pl) andf, = F(y2).The automaton remains in the same state in case of a nonpenalty, and changes its state in case of a penalty. The state matrices have the form

+ + -+

Graphs of states are given in Fig. 1. Suppose that automaton L,,2 is in a medium C ( a i ,a2).Upon constructing, according to (3), the matrix of the transition probabilities

One of the problems in which M. L. Tsetlin was interested in connection with these considerations was: What portion of all stochastic automata with a given number of states, or what portion of some reasonably separated class of such automata, consists of automata capable of expedient behavior? (Editor’s note). Automaton L2,* has the same logic of behavior as a static trigger with a countable input [35].

15


Figure I

we arrive at the following equations for the final probabilities r, and r2:

Using, in addition, the normalization condition rl rl

= PZ/(P,

+ PZ),

r2

= Pl/(Pl

+ r2 = I ,

we have

+ PZ).

We obtain the following expression for the mathematical expectation of a nonpenalty : w 5 2 , 2 C ) = (pza, pla,)/(Pl P 2 ) . 9

+

+

+

It is easy to see that W(L,,,, C ) > (a, a,)/2 for a, # u 2 , i.e., that automaton L2,2behaves expediently in a stationary random medium. Those automata are of particular interest whose structure does not involve any type of information about the type of random medium in which they function. These automata, one might say, do not have the property of the “a priori expediency” of behavior. An automaton % will be called symmetric if the expected value of a nonpenalty, W(%,C ) , in any stationary random medium C(a,, u 2 , .. . , a,) is a symmetric function of a,, a 2 , .. . , a,. Let !lJi be the set of all states of an arbitrary automaton %, and let ma be the set of those states p in which the automaton performs the action fa, i.e., F ( y ) = fa. An automorphism of an automaton % is defined as a one-to-one mapping g of the set !lJi of its states into itself such that the following conditions are satisfied : The partition of the set tu1 into a system of sets !lJmis preserved; in other words, if gp E tD1, for some p E tDlz, then for any p’ E we have gp’ E m,. 2. I I aids) I I = I I agigj(3) I I . 1.

ma

Automorphisms g of automaton % evidently form a group G . Let Godenote a subgroup of G, consisting of automorphisms mapping each of the sets

16

Automaton Theory

!l.Jlm into itself. It is not difficult to see that Go is a normal divisor of the group G . The quotient group G' = G / G ois naturally realized as a group of automorphisms of the set of actions fi,h,.. . ,fx. If G' is transitive on this set, then it is natural to call the automaton homogeneous. For homogeneous automata, the sets Im, are isomorphic to one another. The mathematical expectation of a nonpenalty W(%,C ) of a homogeneouc automaton CU in a stationary random medium C(al , a 2 , .. . , a,) is invariant with respect to transformations from group G'. If group G' is symmetric, then the corresponding automaton is obviously symmetric. Section 2 gives examples of symmetric homogeneous automata for which group G' is cyclic. 2 Asymptotically Optimal Sequences of Symmetric Automata. The Book Stack Problem

In the previous section we gave a definition of the expedient behavior of automata in stationary random media. Obviously, the expectation W ( % ,C ) of a nonpenalty for an automaton in a medium C ( a l ,u 2 , .. . , a,) does not exceed amax= max(al , a 2 , .. . , ax). Naturally, there arises a question of the existence of automata for which W would approach a,,,,,. . . , ,. . . will be called asymptotically A sequence of automata optimal if lim W(%,, C ) = amax. (7)

an

n+m

An automaton belonging to an asymptotically optimal sequence, if n is sufficiently large, performs almost exclusively the action for which the probability of a nonpenalty is maximum. We shall give several examples of asymptotically optimal sequences of symmetric automata.'

I . An automaton L2n,2(an automaton with linear tactic), which is a . . ,~ l 1, , natural generalization of automaton L 2 , 2 ,has 2n states v,l, vZ1,.

v12,q Z 2 ,... ,pn2 and

two different actions A , fi, where

F ( v i U =f,, ) i

i = 1, 2 , . . . , n ,

u = 1, 2.

For a nonpenalty, i.e., when s = 0, states viUmake a transition to q ~ f + ~ , I , 2 , . . . , n - I , and states p,,"make a transition into themselves. For

=

'I The description of automaton L,,,, has been included in the text by us; it is taken from Tsetlin [I551 (Editor's note).

17


a penalty, i.e., when s = 1, states qiapass into & , i = 2, 3,. . . , n ; state q,' becomes ql2,and qI2becomes qI1. Graphs of states of automaton L2n,2are given in Fig. 2. Let us calculate the expectation W(L,,,,, C ) , assuming that a, and a2 f 0, f l.

Figure 2

Computing the transition probabilities from (3), we obtain a system of equations for the final probabilities ria of states qia:

r,,'

=

rLl

=

ril

=

+ qlrLl, plrnl + q 1 r L 2 , + q,r,L,

rz1 = p1r3l

+q2rL p2rn2+ q2ri-2r

rn2 = 42'n2

qlrnl

ri-,

=

9

+

ri2 = w?+,q2&,

+ qlrll,

rz2 = pzr:

+

+ q2r12,

+ + +

+

r,,' = 1. and the normalization condition is rll . . r,,l rI2 * The solution of each of the columns of equations is assumed to have the form ril = u,A;-,, ri2 = We obtain the following characteristic equations for the eigenvalues Al and &: p.12 - Ax qz = 0, CI = l , 2. Calculating from these equations

+

Ail)

=

A 2( 1 )

= 1,

p

=

ql/pl = A,,

a p = q2/p2= A,,

we write the solutions in the forms

Using the equations for r,,l and rn2, we see that B, equations for rll and rI2, we have

=

B2 = 0. From the

18

Automaton Theory

Now let us evaluate the sums

u,

=

C ril = A C n

n

i=l

1=1

=

A,(A,. - l)/(Al - I ) ,

+

The normalization condition u, o2 = 1 and Eq. (8) yield the values for the coefficients A , , A , , and thus the mathematical expectation W ( L 2 , 1 , 2C) , of a nonpenalty can be found, using Eq. (4), to be W(Lzn,2, 3 ' -

a1 Pln - 41n

-(7? PI - 41

02

Pzn

Pzn - 4 2 y ( 21 PIn - 4In I 1 Pzn - q2n) P2 - 42 PI - 41 P2n P2 - q 2

It is important to note that W ( L 2 , 1 ,C 2 ,) is an increasing function of the memory capacity, and that for (a,, a,) 1 0, W = lim W(L,,,, , C ) = max(a, , a 2 ) . n+m

This relation means that automaton L 2 , 1 , 2for , a sufficient memory capacity n, performs almost exclusively the action for which the probability of a penalty is at a minimum. 2. Automata with linear tactic L x , , , x . 8These automata have xn states via,CL = 1, 2,. . . , x, i = 1, 2 , . . . , n. To a state viathere corresponds an action fr. The transitions of states, depending on the value of the input variable s, are effected as follows: for s = 0 (a nonpenalty), state viapasses into state for i = I , 2 , . . . , n - I ; state vnapasses into itself. For s = 1 (a penalty), state cp: passes into state for i = 2, 3,. . . , n. State qla passes into state p;fl, u = I , 2 , . . . , x - I . State qIxpasses into state qll. Graphs of states for an automaton with a linear tactic are shown in Fig. 3. Assuming that an automaton L,,,, is in a stationary random medium C(a1,a 2 , . . . ,a,) and using (3) and (4), we can find W(L,,,,, C ) . We get

* Electronic models of automata with a linear tactic were constructed and experimentally studied in the diploma dissertation of Buylov [35].

Finite Automata and Modeling rhe Simplest Forms of Behavior

19

n

Equation (9) implies that a sequence of automata L,,,,, is asymptotically optimal for stationary random media C(a,, a 2 , .. . , a,) under the condition that a iiiax = max(a,, a 2 , .. . , a,) L 0 . In fact, in those media

In media in which the condition of nonnegativity of a,,,,,does not hold,

w = [(I /x)(a;' + a,' +

* * *

+ a;')]-',

i.e., it coincides with the harmonic mean of a , , a 2 , .. . , a,. 3. Automata D x n , x . 9These automata also have xn states via, rr = 1, 2 , . . . , x , i = 1, 2 , . . . , n. To a state viathere corresponds an action f,. For s = 0 (a nonpenalty), viapasses into a state v,,",i = 1, 2 , . . . , n. For s = 1 (a penalty), transitions of states are effected in the same way as in automata with a linear tactic. Graphs of states for automata D,,,,, are given in Fig. 4. Calculating, as in the previous examples, the expectation of a nonpenalty, assuming that the automata are in a stationary random medium C(a,, a 2 , .. . , a,), we obtain

The construction of the automata D,,,, was proposed by Krinskiy [102].

20

Aufomafon Theory

Figure 4

It is not difficult to check that a sequence of automata D,,,,is asymptotically optimal in an arbitrary stationary random medium. 4. A construction of the automata K,,,, proposed by Krylov [112]. Stochastic automata K,,,, have xn states qim, i = 1 , 2 , . . . , n, a = 1,2,. . . , x ; F(qia) = S,. For s = 0, transitions of states occur in the same way as in the case of automata with a linear tactic, L,,,,,. For s = 1, state p; passes with the same probabilities equal to 4 into states pi“,, and p;-l, i = 2, 3 , . . . , n - 1. State qnupasses with probabilities 4 into states qnaand States 9,“pass, also with the same probabiIities equal to 4, into states qzaand p + l, a = 1, 2,. . . , x - 1 ; state qlxpasses into states qzxand q l l . For automaton K,,,,

It is easy to see that automata K,,,, also form an asymptotically optimal sequence in all stationary random media. 5 . An interesting construction of asymptotically optimal sequences of finite automata was proposed by Ponomarev [128]. These automata were named automata with a comparing tactic V,,7,,. A graph of states for automaton Vz,,z having only two actions is shown in Fig. 5. In this figure the solid circles indicate those circles where the action fi is performed, and the open circles correspond to the action fi. This autr :naton has 2n = 2m 21 states q; (a = 1, 2, i = 1, 2,. . . , n); F ( g L 2 ) = 1, The states qll,. . . ,qll,q I 2 ,.. . ,qt, belonging to the horizontal “comparing” part of the automaton, are related to one another in such a way that a change of actions of the automaton occurs independently of whether the action was followed by a nonpenalty s = 0 or a penalty s = 1 .

+


21

However, in case of a nonpenalty meted out after the action fl (or, correspondingly, in the case of a penalty after the actionjJ the state of the automaton changes to the neighboring left state. I n case of a nonpenalty after the action fl (or, correspondingly, in case of a nonpenalty after the action&) the state of the automaton if replaced by the neighboring right state. In these states automaton V,,,, makes, as it were, the choice of the necessary action. In this the automaton differs from automata with a linear tactic L 2 n , 2 ,where the change of actions occurs only in two states vll and v12.

5

=o

Figure 5

and q$+l ,. . . ,I$,+~ states are related to each other The vi+l,. . . , in the same way as in the automaton with a linear tactic. However, state vI2 (or, respectively, vll) passes, for s = 1 (a penalty), into the extreme state vh+l(or, respectively, I&+~) of the line. This assures a multiple repetition of the corresponding action even in those cases in which the probability of a penalty is considerably greater than t . The above construction of automata can naturally be generalized to the case of an arbitrary number x of actions. Without giving the somewhat cumbersome formula for the mathematical expectation of a nonpenalty, we shall note only that a sequence of symmetric automata V,,,, is asymptotically optimal in all stationary random media. The foregoing examples of asymptotically optimal sequences of automata possess one general property; for each of these sequences, the automaton %, in each of the sets W, of its states has a state vnasuch that it cannot be derived from the W, with a sequence of input signals having length less

22

Automaton Theory

than n. (We remember that here, as before, !Illa denotes the set of all automaton states in which the action L is performed.) Let pl be some state of a deterministic automaton %, and let F ( p ) =A, i.e., pl E ma. The depth d(pl) of a state pl will be defined as the minimum length of a sequence of input signals bringing out of the set !Illa. The depth d(%) of an automaton will be defined as the greatest of the depths of its states. We recall that for homogeneous automata the sets are pairwise isomorphic. In order for a sequence of deterministic automata "Ill ,. . . , a,,. . . to be asymptotically optimal, it is necessary that

Let us prove this assertion. Consider for simplicity the case x = 2. Let a, be the total probability of the states belonging to the set ma. Furthermore, let u k denote the total final probability of a set of states p l Im, ~ having a depth k. Suppose that the automaton is in a stationary random medium C ( a l ,a2). Then it is obvious that we have the following inequalities:

in which ,ul= min(pl, q,), p1 = I - q1 = (1 - a,)/2. These inequalities imply that k = 1, 2,. . . , d,. 0 2 2 plkUk, (14)

Similarly we can show that

The fact that the ratio al/a2 is bounded implies that the expectation of a nonpenalty W ( X n ,C ) = a,al a2u2,a1 a2 = 1, reaches the value a' < a,,,, = max(a, , a2). Thus, a,,, is not attained by automata with a bounded depth. The possibility is not excluded that this property (perhaps, with some additional conditions) is also sufficient for the corresponding automata to form an asymptotically optimal sequence.

+

+


23

We note that all the automata just described admit a natural numbering of the states belonging to the same set !D& (and thus, they are in this sense “linear”). Apparently, this property is characteristic of constructions that are most economical as to the number of states. Milyutin [123] studied the question of optimal constructions of automata, i.e., those constructions of automata that assure the maximum sum of the expectations of a nonpenalty, W , in two stationary random media C,(a,, a2) and C,(a,, a,) for a fixed memory capacity. Using a method (proposed by him and Dubovitskiy [88, 891) of solving extremum problems in the presence of constraints, Milyutin gave an upper estimate for the possibility of a nonpenalty in both media for automata with a fixed memory n. In particular, for automata functioning equally well in the media C,(a, , a,) and C,(a,, a,), we have the following estimate for the medium C , :

I W P

(02/a1>

2 )

((q2/ql)(Pl/Pd)n-l7

(17)

where p 1 > p2,

p i = 1 - qi = (1 - aJ2,

i = 1, 2.

Milyutin exhibited examples of automata for which this estimate is achieved asymptotically. In particular, he proved that automaton L2n,2is close to optimal in the media in which at least one of the numbers a, and a, is close to 1, and described a construction of automata, close to optimal in all media which are largely identical with automata having a comparing tactic, proposed by Ponomarev. In the final part of this section we shall give an example of a system whose functioning is largely similar to the behavior of automata in stationary random media and possesses a naturally defined expediency. Suppose there are n books K , , K2,. . . , K,, lying in a stack on a desk. The books in the stack may be used in various ways-for example, one can choose the desired book and put it back in the same location after use. It is, however, often preferred not to search for the original location of the book chosen, but to simply put it on top of the stack. We shall now show that this method possesses a certain expediency: each book is located, on the average, higher, the more frequently it is used. We shall calculate the probabilities of an arbitrary ordering of books in the stack in terms of the probabilities p k , 1 5 k 5 n, of their being used. We assume that at each instance of time r = 1, 2,. . . , a kth book is taken from the stack with a probability pk (independent of the location of books at preceding instants p,, = 1. of time), and is put on top of the stack. Of course, p1 p2 The location of books in the stack is given by one of the permutations

+ + .- - +

24

Automaton Theory

(il , iz,.. . , i n )of the indices 1, 2,. . . , n. The choice of the kth book changes permutation ( i l , . . . , i2-l,k, i2+l,.. . , in) into permutation (k, i l , . . . , i l - l , it+,,. . . , in). The pile of books may be considered as a finite automaton having n! states sl,s, . . . ,s, , each corresponding to a certain ordering i,, of books. This automaton has n values of the input variable q ~ ~ ~ , ~ ~ , , , , and sP, k = 1, 2,. . . , n, corresponds to the choice of the kth book. The process of redistributing books in the stack is described, obviously, by a finite Markov chain. We shall assume that 0 < Pk < I , k = 1, 2, . . . , n ; then the chain is ergodic. Let ri in denote the final probability of permutation (ii,. . . , in). To determine rLl,.,.,inwe have the following system of n! equations:

and the normalization condition is

Using mathematical induction with respect to n, we can show that the following formula is true:

if

Pi,

4Pi,.

Using Eqs. (18), one can calculate the probabilities rL(Z) that in the stack the book K iis located in the fth position, and thus the mean depth di of the location of this book in the stack is n

di = C hi(/). 2=1

The inequalities (19) imply easily that d j 5 dP for p i 2 p k , i.e., for each pair of books in the stack, that book which is used more often is located higher (on the average). The foregoing considerations explain perhaps the dissatisfaction of a person who finds that the books on his desk were put in order by someone else.

25


3 Behavior of Automata in Composite Media In the preceding sections we discussed the behavior of automata in stationary random media whose probabilistic properties were not assumed to be known beforehand. The a priori knowledge of the constants a , , a 2 , .. . , a,, defining a stationary random medium C(a, , a,, . . . , ax), makes it meaningless to construct an automaton that would be capable of performing, depending on its state, a number of different actions: in this case, one could construct an automaton having only one state and one action (corresponding to a ,) and nevertheless possessing the maximum possible expediency. For stationary random media, the length of the transient process plays no important role, since the time of functioning in a stationary medium is assumed to be infinite. In this section we consider the behavior of an automaton in a medium whose properties change in a random manner. Then, even if the possible probabilistic characteristics of the medium were known, it would be impossible to construct an automaton with a single action which would be capable of expedient behavior. The transient period becomes important here: in such media the automaton must, so to speak, continuously "relearn," and an increase in the "relearning" time lowers the expediency of the automaton's behavior. If in stationary random media the foregoing asymptotically optimal sequences of automata-e.g., of automata with a linear tactic-are typically characterized by a monotonic increase of the expected value of a nonpenalty with an increasing memory capacity, then in nonstationary media this relationship is no longer monotonic. The dependence on time of the probabilistic properties of the medium will be specified in a special way: We shall consider that the medium in which the automaton is located consists of stationary random media whose switching is determined by a Markov chain. A ) , having V states Thus, consider a Markov chain K ( C ( l ) , . . , C(v), P , .. . , 0") and a matrix of transition probabilities d = 11 6,, 11 , a, = 1, 2 , . . . , V. The state C(")corresponds to the stationary random medium 0") = C(alb,a,",. . . , a,"). We shall say that an automaton CU is in a composite medium K if at each instant of time it is located in one of the media C("),a = 1, 2,. . . , V, i.e., if its actions and the values of the input variable are related in the manner described above for stationary random media; if, moreover, at the instant t the automaton is in a medium C("),then at the moment t 1 it will be in a medium C(0)with the probability W .

+

26

Automaton Theory

Let Y/,’@’, = 1,. . . , V, i = 1, 2,. . . , m, denote the state of the system automatonxomposite medium such that the automaton is in state rpi, and the composite medium in state Ccb).Then the probability d?)(,) of a transition of the system from state !Pip’ into state Yj,) can be expressed by the equation n‘iiP’(v’ = [pL:’aij(l)

+ qL:’aij(o)]

where 1 1 a i j ( s )I I is the matrix of states of automaton

+

(20)

-6pv,

a, and p::)

=

( 1 -aai(8’ ) /2

and q::’ = ( I aLf’)/2are the probabilities of a penalty and a nonpenalty, respectively, in the medium C C pfor ) the action f,, = F(pi). The matrix IP’(r’ Z7 = ( 1 zij 11, p, y = 1 , . . . , V , i , j = I , . . . , rn, generates a finite Markov chain. If this chain is ergodic, then the final probabilities r:p) of states Yjp’ of the system do not depend on its initial states, and the expectation of a nonpenalty, W(9l, K ) , for the automaton in the medium K can be found from the equation

In this expression a’; signifies the total probability of those states of the system in which the automaton performs the action f z and the composite medium is in state C(p). We shall limit ourselves to the simplest case in which V = 2, and

where the parameter -6 represents the mean frequency of the switching of states of the composite medium.I0 Let us further assume for the sake of simplicity that the number x of actions of the automaton is 2, and

Under these assumptions we shall indicate the procedure for computing K ) for an automaton with a linear tactic (see Secthe expectation&!.‘(‘+I tion 2). This is easy to see by calculating the mean number of clock intervals during which the state of the medium remains constant:

T

c m8(1 93

=

m-1

- ap-1 =

1/8.


27

For the case x = 2, we shall enumerate the states of a n automaton with a linear tactic in the following way: qil = xi,

q?

=

x ~ + ~ i, = I,

2 , . . . ,n.

Let rja) denote the final probability of state Yja)of the system automatoncomposite medium. We introduce the vector Ri = (rjl), ri2)),i = 1 , . . . , 2n. As before, let p = (1 - a)/2, q = ( 1 a)/2, 1 = p / q . We shall make use of Eq. (20) for the matrix of the transition probabilities of the system, and the definition of an automaton with a linear tactic 1 5 ~ , , ,Then ~. we obtain the following system of equations for the final probabilities

+

R1

=

SR2

+ QRn+19

In this system S and Q are second order matrices:

The solution of this system will be assumed to have the form Rk = R , P ~ - ~ ' , k = 1, 2 , . . . , n, and Rk = R,pk-', k = n + l , . . . , 2n, where R, = (ri", $)') is a constant vector. To determine the eigenvalues p and the eigenvectors, we obtain the characteristic equation

in which E is the unit second order matrix. From the equation det(p2Q - p E

+ S ) = 0,

28

Automaton Theory

we have P4

- P3-

1-6 1-226

4

[

1

+ a2 +--

-+ p 2 21-d I-u2

1-8

-piIjj-

4

Solving this equation we get p l quadratic equation

41

1 1-2261-aZ

+1=0. = ,u2=

1 ; p3 and p4 are the roots of the

Upon finding the eigenvectors from ( 2 5 ) , we write the solution of the system (23) in the form Rk

1

ARL"

+ BRi?' -t CRfj" + DRL",

where R;."= [ I ; I ] ,

where

A

=p/q,

k = I , 2 , . . . , n.

(28)


29

The coefficients A , B, C, D, in ( 2 8 ) are found from the equations for R, and R,,, in the system (23). Then we find the expressions for Rk = (ri*),r i 2 ) ) for k = I , 2,. . . , n:

I n these expressions d is the normalization constant, and x is either root of the quadratic equation (27). The expressions for R l l f k= rj&, rAyk) are found from the equalities

Upon using Eq. (21), we arrive at the following expression for the expectation value of a nonpenalty for an automaton with a linear tactic, L,,,,2,in the composite medium K , specified by formulas ( 2 2 ) and ( 2 2 ' ) : W(L,n,,* K )

=(A

- 1)2

{ 2 n s / ( l - 26)}(A

where cash y

=

+ I),

cash - 1 cosh ny (A

+

( 1 - A ) Z ~1 -s 2A 1 - 26

-

coth y / 2 sinh ny' (32)

I.

Figure 6 presents the plots of M = ( 1 - W ) / 2 versus the memory capacity n of automaton L,,,,2for various values of S and fixed a = 4 ( p = +, q = I). Figure 7 gives the same plots for fixed 6 = 0.01 and various values of p . It is easy to see that the expectation of a nonpenalty is nonnegative, and vanishes at n = 0 and t? + DO; consequently, it attains a maximum rn for some finite value of the memory capacity no. The lowering of W for small values of n is related to the fact that in this case the information about the state of the medium in which the automaton is located is not used to its full extent. When the memory capacity is enormously increased there occurs, one might say, an averaging of the statistical properties of both states of the composite medium (the automaton "fails to relearn"). A decrease in

30

Automaton Theory

--

the switching frequency is equivalent to an increase in the response speed of the automaton. It is natural, then, that with decreasing 6 the values of n o , as well as the maximum value of the nonpenalty W , should both increase. When the mean switching frequency 6 tends to zero, no + 00 and m max(a, -a). With an increase in 6 a reverse process takes place. Thus, for -j

( 1 - 26)/6(1 - 6)

5 (A + l)”A,

the maximum expectation of a nonpenalty is attained at no = 1. Equation MI

a3 az

af -

31


(32) makes it possible to select those automata with a linear tactic that possess the most expedient behavior in a given composite medium. To select the memory capacity for such automata, one can use Table I, where the values of no and m are listed for various a and 6. Each column in the table contains a pair of numbers; the first is no and the second is m. We note that the value of W as a function of n and 6 can serve as a measure of the distinguishability of the random media C") and C2'for automata with a linear tactic. TABLE I 0.001

0.8 0.6 0.5 0.33 0.2 0.1

3; 0.792 5; 0.588 6; 0.488 8;0.306 11; 0,178 15; 0.072

0.010

2; 3; 4; 5; 6; 7;

0.032

0.100

0.744 2; 0.672 1; 0.512 1; 0.532 2; 0.446 2; 0.314 1; 0.424 3; 0.344 2; 0.232 1; 0.250 3; 0.182 2; 0.110 1; 0.112 4; 0.074 2; 0.n40 1; 0.034 4; 0.020 2; 0.010 1;

0.45

0.32

0.230 0,130 0,090 0.040 0.014 0.004

1; 1; 1; 1; 1; 1;

0.064 0.036 0.024 0.012 0.004 0.002

Interesting experiments, in which the behavior of a human being was compared with the behavior of automata in stationary and composite random media, are described in the paper by Alekseyev et al. [ 2 ] . 4

Behavior of Automata with an Evolving Structure in Random Media

In our study of the behavior of finite automata in random media, we have thus far assumed that their structure remains constant. We recall that the structure of an automaton is given by the matrices I I a i j ( s )11, s = 0, 1, specifying the transitions of states p(t) E (pl ,p z ,. . . , pm)of the automaton for various values of the input variables, and by the equationsf(t) = F(p(t)), specifying the actions of the automaton, f(t) E ( fi ,f2,. . . ,fJ,depending on its state. Varshavskiy and Vorontsova [43, 441 studied the behavior of stochastic automata with an evolving structure by modeling them on digital computers. The state matrices for automata of this type also change depending on the values of the input variable.

32

Automaton Theory

The modification of the state matrices occurs in the following fashion. Suppose that at the instant t , under the influence of the input variable s ( t ) , state pi passes into state pj, and then (at the instant t I ) the input variable assumes the value s ( t 1 ) = 0 (a nonpenalty) or s(t 1) = 1 (a penalty). Then the value of the transition probability a i j ( t ,s ( t ) ) increases in the case in which s ( t 1 ) = 0 and decreases when s(t 1 ) = I , and the remaining elements aik(t, s ( t ) ) , k f j , of the row change in such a way as to preserve the stochasticity of the matrix, i.e., to preserve the condition

+ +

+

+

+

2 a& + I , s ( t ) )

=

1=1

I.

The remaining rows of the matrix remain unchanged. We note that at every instant of time only one of the matrices undergoes a change 1 ) a ( t , s ( t ) ) l l , s = 0, I , namely, that one which corresponds to the value of s equal to s ( t ) . We consider the following method of forming the structure of automaton A,,,,, having x actions fi,fz,. . . ,f, and m = n x states p1, pz,. . . p m : 9

u

F(p,,,+,) = f u + l ,

+ I, s(t))= ajk(t + 1, s ( t ) ) = aij(t

Uij(f, s(t)) aik(t,

=

+ (-

0,. . . , x

-

I;

v

=

I,. . . , n ,

I)s(~+l’gaij(f, s(t))(l - O i j ( t , s ( t ) ) ) ,

s ( t ) ) - (-l)sct+l’gaik(t,s(t))aij(t, ~ ( t ) ) , j f k.

(33) In these formulas 0 5 g< 1 ; it is not hard to check that the matrix remains stochastic. The behavior of automata with an evolving structure in random media can be described by a nonhomogeneous Markov chain. For automata whose structures evolve as described by Eqs. (33), there exist stationary values of the transition probabilities, and one can speak of the final probabilities of the states of the system. I n particular, in the discussion of the behavior of automaton in stationary random media it was shown that

i.e., the simplest automaton with an evolving structure having only two states is equivalent to an automaton with a linear tactic having an infinite number of states.


33

An analytic study of the behavior of automata with an evolving structure in composite random media is very involved. The work by Varshavskiy and Vorontsova [43] contains such a study (done by modeling on a digital computer) of the behavior of the system: an automaton with an evolving structure-composite medium. In one of such experiments, a model was made of the behavior of automaton A 8 , z . This automaton has eight states vl,.. . ,vsand two actions fl ,f z . In states yl,. . . , p4the actionf, is performed, and in states v 5 , .. . ,v8, the action fi. The structure of the automata evolves according to (33). At the initial moment t = 0, the state matrices for s = 0 and s = 1 coincide, all of their elements being equal to 8 : aij(O,0) = aij(O, 1 ) = 4, i , j = 1, . . . , 8. Thus, the behavior of the initial automaton is obviously not expedient. The behavior of this automaton was studied in the medium K ( P , C2), A ) , where C") = C(a, - u ) , C2)= C(-a, a), and A has the form (22). A measure of the expediency of behavior is provided by the mean payoff during time T :

FV(A8,2,K, T ) = ( 1 / T )

C (-l)B(t). T

t=1

These experiments have shown that, apparently, lim F(A,,,,K, T ) = m,

T+CC

i.e., for sufficiently large time intervals T, the mean payoff for an automaton with an evolving structure approaches, from below, the value of the expectation of a nonpenalty for automata with a linear tactic and optimal memory capacity. (Here those media were considered.for which no 5 4.)The value of the constant g has little effect on the evolution of the structure. Thus, Fig. 8 shows the plot of the function M ( T ) = &(I - W ( A , , , , K , T ) ) versus T for a = 0.8, 6 = 0.01. For such a medium, the optimal automaton with a linear tactic has memory capacity no = 2 and the value M = i ( 1 - W ) = 0.128. During the evolution of the automaton's structure during its functioning it was noted that the state matrices in the composite media change, approaching the matrices of the finite automata and resembling constructions described in Section 2. It is interesting that the evolution of the matrices I I a i j ( t ,0) 11, corresponding to the value s = 0 (a nonpenalty) proceeds much faster than the evolution of the matrices I I aij(r,1) 11 (s = 1, a penalty). This is due to the fact that automata with an evolving structure acquire

34

Automaton Theory

expedient behavior: they “succeed” more often in winning than in losing, and conversion to the matrix 1 1 u i j ( t ,0) 11 is more frequent than conversion to the matrix I I uij(r, 1) 1 1 . We shall note now that the evolution of the structure of an automaton in a random medium can serve as an example of the automatic synthesis of an automaton according to a given criterion for its operation.

Appendix to Part I. Eigenvalues of Markov Chains Describing the Behavior of Asymptotically Optimal Automata in Stationary Random Media”

In this section we discuss the spectrum (the set of eigenvalues) of Markov chains describing the behavior of some asymptotically optimal automata in stationary random media. To be more precise, we investigate the question of when these Markov chains have a sequence of eigenvalues (those which are not equal to 1) approaching 1 in modulus for n 4 00, and we estimate the rate of convergence.*2In all cases we take x = 2 for simplicity. 1. The easiest to investigate is the spectrum of automaton D 2 n , 2 sug, gested by V. I. Krinskiy. This appendix was prepared on the basis of an unpublished rough draft by M. L. Tsetlin. The preparation for printing and the filling in of any gaps were done by S. G. Gindikin and A. L. Toom (Editor’s note). la As we know, the eigenvalues of a Markov chain (generally speaking, complex) never exceed modulus 1.


35

Let us write out the system of equations for the eigenvalues:

. . . . . . . . . .

. . . . . . . . . .

Here r i , si are the probabilities of being in states vil and q:, respectively (see Section 2). Using all the equations except for the nth and the (n 1)th (relating r i and sj to each other), we express all r i in terms of rl , and all siin terms of s l . Substituting these expressions in the nth and (n 1)th equations, we obtain the system

+

+

For the solution to exist, it is necessary that the determinant of the system be zero, whence

(W - 1) - q , P l n ) ( ~ n Q - 1) - 42P2,) = ptp2(An-1(1 - A)

- q1py1)(A,-1(1 - A)

+

-

q2pp-1).

This polynomial is of degree 2n 2. All the desired eigenvalues are its roots. It has two extraneous roots A = p l , p 2 . This is related to the fact that in the system (35) in the first equation, the fraction can be reduced by A - p l , and in the second equation by A - p 2 ; however, this reduction is not made. We shall rewrite the equation as

Automaton Theory

36 Let us subtract 1 from both sides: 1,-1(1 p12-1(

- 1)(1- p , ) - A ‘ y l - A)(pz - A) . A’&(1 - A) - q2p2n I - A) - qlpl”

Now we see that there is a single root A = 1 (corresponding to the eigenvector of the final probabilities) and the (n - I)-fold root 0. Furthermore,

or

The case in which p1 or p a or both are equal to 1 is excluded from our considerations. It is easy to show that in this case, in general, there are no eigenvalues tending to 1 for n 00. Then we let max(p, , p 2 ) = nz < 1. We intend to find positive E,, + 0 and S,, such that every solution of Eq. (36) satisfies one of the two inequalities: I 1I I m S,,< I or I A - 1 I I E,,. (This will mean that any sequence 1,)is either bounded in modulus by a number less than 1, approaches 1 sufficiently rapidly, or breaks up into two subsequences of this type.) We shall prove this assertion by contradiction. Let us assume that I A - 1 I > E,, and I 1I > m S,, . Then

-

+

+

Then

E,,

and 6, satisfy the relation

+

If we set E,, = 2(1 (S,,/m))’l-l, then the assertion will be proved. We can take S,, = ( 1 - m)/2. Then we obtain either I A I I (I m)/2 or I 1 - 1 I 5 2(1 (1 - m)/2m)”-’.This implies that if we choose A,, in such a way 0. that I 1, I + I , then of necessity I I - A,,I 0, and even I 1 - A,,1”

+

-

+

-


In this case, setting W),

=

A,, = 1

37

- w,,, we easily obtain

+ pzn + P1/91 + P2/42 + O(Pln + P z n Y . Pln

1

Thus we have explained the behavior of I 1, I 1.13 2. Now we shall study the automaton with a linear tactic, L2n,z.The required eigenvalues are solutions of the system ---f

We assume beforehand that p l , q l , p 2 , q2 f 0 , 4, 1. Let us focus our attention on the equations in the first column, except for the first and the last. Their solution rl , r2 ,. . . , rn can be considered as a solution of a linear equation involving finite differences with constant coefficients. The characteristic equation is

(The same can be done for the other column.) We have assumed that p1 # 4 and p z # 4. Then 4p1q1< I and 4pzqz < 1. Since we are interested in A’s whose moduli are close to 1, the case 1 = 4plq1 or 1 = 4p,q2, yielding multiple roots, can be neglected. Then the solution of the equations in the first column, except for the first and last entries, has the form

rk

=

Aplk

+ BpZk,

l 3 Neither here nor in the considerations that follow is the existence of such I I., I --t 1 proved, although in fact this takes place. The problem of the uniqueness of such I., is not considered either (Editor’s note).

38

Automaton Theory

Similarly, we can satisfy all equations of the second column, except for the first and last, if we assume where

Substituting these expressions for rk , sk in the remaining four equations, we obtain a system of four linear equations with four unknowns A, B, C, D . Equating the determinant of the system to zero, we obtain the equation

This equation has one unknown 1 since pl, p 2 , v l , v2 are given functions of 1. Subtracting 1 from both sides of the equation, we can then factor out 1 - 1 in both numerators. By division, we obtain PIOlln - P2") - 41@1-' - pUrt-9 Pln - Pzn

+

P2(V?+'

- YE+') vln

- 42(vln - v2 - v2n

)

= 0.

We see that

Using this result, we shall switch to new variables x, y by setting

Substituting these expressions in the equation, we get {pl sinh nx - (p1p2)1/2 sinh(n - l)x}/sinh nx = {q2 sinh

ny - (p2q2)'I2sinh(n

+ I)y}/sinh ny.

This equation has one unknown 1 since x, y can be expressed in terms of I by formulas 2(p1q1)1/2 cash x = 2(p2q2)1"cash y = I. Making use of the formulas for sinh(a f B), we reduce the equation to the form (p1q1)1'2 sinh x coth nx

+ (p2q2)1'asinh y coth ny

=

1 - (pl

+ p a ).

(39)

39

Finite Automata and Modeling the Simplest F o r m of Behavior

Let us multiply (39) by 2 and introduce the angles a

= arc sin(pl)1/2,

8 = arc sin(p,)”,,

8 < n / 2 , and

a,

8 f. 744.

+ sin 28 sinh y coth ny = cos 2a + cos 28.

(40)

0 < a,

We obtain the system: sin 212 cosh x

= sin

28 cosh y

sin 22 sinh x coth nx

=I

,

The transformations are finished. Now we shall investigate the nature of the I I , I + 1, given by the solutions of the system. We note that if Re x, 2 c > 0, then limn+mcoth nx, = 1, and if Re x, 5 c < 0, then coth nx, = - 1. Therefore we must consider a system which can be obtained from (40) by replacing coth nx and coth ny with 1 or - 1. But first we note that if ( 9 , y ” ) is the solution of this system, yielding some A0, then (-9,yO), (9,-y”), (-9, -y”) are also solutions, yielding the same 1O. Therefore we may seek only those solutions for which Rex20,

Reyq0.

(41)

Then coth nx and coth ny should be replaced only by 1. We obtain a system : sin 2a cosh x sin 2a sinh x

= sin

28 cosh y

= 1,

+ sin 28 sinh y = cos 2a + cos 28.

(42)

The system, considering the condition (41), has only the following solutions: If 0 < a, 8 < n / 4 , then eZ = cot a, eu = cot 8. In this case jZ = 1. ( 2 ) If n / 4 < a, 8 < n / 2 , then eZ = -tan a, eu = -tan 8. In this case I = -1. (1)

For a < n / 4 < 8 or 8 < n / 4 < a, there are no solutions satisfying the condition (41). (The cases in which a or 8 are equal to 0, n/4, n / 2 are not considered.) These solutions will be useful to us later, and now we shall analyze the behavior of Re x and Re y. We set x = x1 ix, , y = y, iy,, where x, , y, 2 0. We intend to find constants c > 0 and 0 < M < 1, such that if x1 5 c or x, 5 c, then I1ILM.

+

+

40

Automaton Theory

It will be noted that

=

sin 2a

ez1

+

e-21

=

2

sin 2a cosh xl.

Similarly, I 1I 5 sin 28 cosh y , . Taking this into consideration, we select c > 0 in such a way that cosh c

< min{ 1 /sin 2%; 1 /sin 2 8 ) .

We can do this since u, # n/4. Then we shall have

I 1 I 5 M = max{sin Since we are only interested in

2.r cosh c, sin 28 cosh c} < 1 .

I 1, I

+

1, the cases in which x1 5 c or

x2 I c, do not, as we have shown, have t o be considered.

Thus, we now consider that

X I , x2

2 c > 0.

(43)

Then

Similarly,

I coth ny - 1 I 5 2/(ezne

1 ).

-

This, as it is easy t o show, implies that any x, y , 1 [solutions of the system (40)], satisfying the condition (43), differ from the solutions of the system (42) by no more than constant x erznc. For the case (43) it is easy to estimate more exactly the difference between 1and 1 and -1. For this purpose, it is sufficient to denote 1by 1 - w o r -1 w , and substitute in the system (40), leaving out the terms involving w in powers greater than one. From the proof of the estimate (01 I constant ~ e - it ~is easy ~ ~to ,generalize this type of calculation. As a result, we obtain:

+

(a) for p l , p z < w =

(41

4, 1 = 1

-

w,

where

+ ( 9 2 - Pz)(P2/42)nI

- P1>(42 - P2)[(41 - P1)(Pl/41)n 41 92 - 1

+ o[(P1/4dn+

+

(P2/42)nl;

41


(b) for p l , p2 > 4,

)3 = -

o = (PI - 41)(P2 - 42"l

1

+ o,where - 41)(41/PJn 42 - 1

41

+

+ (Pa - 42)(42/Pz)nl

+ o[(41/Pl)n + (42/P2)nl.

3. The spectrum of automata Kzn,2 can be studied in a similar fashion. We can show that for p l , q l , p 2 ,q2 f 0, I ilI -+ 1, we have il= 1 - o, where o=-

2q1q2

41

+

.I')*(.')&([.+

[4'(&y+q2(&y]

42

I1 AUTOMATON GAMES. ZERO-SUM GAMES FOR TWO AUTOMATA

1 Automaton Games In this section we shall describe mathematical models for the simplest forms of collective behavior of automata. We shall use some of the notions and results of the von Neumann-Morgenstern theory of matrix games [24,42, Z22, 1221, as well as the definitions and constructions given in Part I of this chapter. We shall consider the collective behavior (game) of automata (E", . . . , 91'. It is assumed that each of these automata is specified by means of its state matrices and Eqs. (1) and (2). Furthermore, let sj(t),f j ( t ) ,@'(t), j = 1,. . . ,v, be the values of the input variable, output variable, and the state, respectively, of automaton ' u j at the instant t. We shall assume, as before, that the input variable sj(t) assumes only two values, d ( t ) = 0 and s j ( t ) = 1, corresponding to the (unit) win and loss of automaton 91j at the instant t. The output variablefj(t) is supposed to take on values belonging to the set f],. . . ,fij.These values will be called strategies of automaton % j , and indicate that at the instant t, automaton ' u j uses its ath strategy iffi(t) =fa'. The values q$,. . . ,vkj of the variable @ ( t ) will be called the states of automaton ' u j , and the number mj the capacity of its memory. Obviously mj2 d. We shall consider the state matrices

1 1 & ( d ( t ) ) 11,

j = 1,. . . , v ; i, k = 1,.

to be given for automata

W,.. . , 'W

. . ,mj;

d ( t ) = 0, 1,

42

Automaton Theory

Now we proceed to describe games played by automata. A play f ( t ) , taking place at the instant t , will be defined as the set f ( t ) = ( f l ) ( t ) , .. . , f ’ ( t ) ) of the strategies used at the instant t by automata W,. . . , %“. The outcome s(r 1) of the playf(t) will be defined as the set

+

s(t

+ 1) = ( s l ( t + I),.

. . ,su(t

+ 1))

of the values of the input variables (unit wins and losses) of these automata 1. at the instant t We shall say that a game f played by automata W,. . . , ‘u“ is given if for each playf(t) we are given the probabilityp(f, s) of its outcome s ( t 1); the equality

+

+

1P ( f ,

s) = 1

(44)

Y

is valid for anyf. Thus a game F for automata ‘E“,. . . , \u‘consists of a sequence of plays f ( t ) , t = 1, 2 , . . . , whose outcomes s ( t 1 ) are determined by the probabilities p ( f ( t ) , s(t 1)). A system of the values of p ( f , s), specifying an automaton game I’, defines a v-person game r understood in the usual sense for game theory. In fact, the payoff functions v ” ( f ) ,j = I , . . . , v, defining a game have the meaning of the expected value of a win for the jth player and his set of strategies f, and are obtained uniquely from the probabilities of the outcomes, using the formula

+

+

r*

r.

r*will be called equivalent to the automaton game Note that specifying the game f* does not determine uniquely the automaton game r. In fact, the game I‘* is specified by Y functions V J ( f ) , and the game f is defined by 2v - 1 probabilities p ( f , s) for each playf. A game played by Y automata f will be referred to as a game with independent outcomes if A v-person game

P(.L s)

P ( f , sl,. . . 9

where Pi(f,

O),

P

G 1)

2 0,

n V

S”) =

j=l

Pj(J s J )

(46)

9

Af, 0 ) + @(f, 1 )

=

1.

43

Finite Automata and Modeling the Simplesi Forms of Behavior

An arbitrary game I'* makes it possible to construct uniquely an automaton game with independent outcomes; here

We shall say that a system of automata % I , . . . , a' participating i n a game I' is in state u ( t ) = (al,. . . , a"),if at the instant t , automaton %j is in state v x j ,aj = 1,. . . , mj,j = 1,. . . , v. We shall show that this system can be described by a finite Markov chain. For this purpose it is sufficient to determine the probabilities p;:;:::;;; of the transition of the system from state u ( t ) = (ul ,. . . , uv), into state

80 + 1)

= (81,. ..3

BY).

Suppose that the following strategies correspond to states q i l , . . . ,yiV of automata W,. . . , W:

Then the probabilities p;:::::;;;of the transition of the system from state u ( t ) into state p ( t 1 ) are given by the formulas

+

It is not hard to verify that

i.e., that the matrix 11 pi:;::::;;I I is stochastic. As a rule, the Markov chain defined in this fashion is ergodic. In this case, the final probabilities of the states of the system exist, and with them also the expectations of the wins of automata which do not depend on the initial states. Automaton games to which ergodic Markov chains correspond will be called ergodic. Let RX1,...,IXv denote the final probability of state (L = ( u l r . . , (1") of the system of automata % I , . . . , %" participating in the game r. States ~ ,.i . ,. , ygV of the playing automata and the strategies ,. . . correspond to this state of the system. Then the expectation WJ of a win for automaton can be expressed as

,fc,

44

Automton Theory

In this formula P(f)is the payoff function for the j t h player in the equivalent game T*for v persons, specified by the expression (45). The quantity Wi will be called the value of the game T for automaton '@. In describing the modeling of automaton games on digital computers in the following, it will be useful to use the final probabilities a ( f ) of the plays f of the game I'. Let f = ,. . . , j z ) and let U ( f ) be the set of all such states of the system of playing automata in which automata 'W,. . . , W use strategies fi:, . . . ,A:, respectively. Then

(fif

It is easy to see that for Y = 1, automaton games reduce to behavior in stationary random media. The models of the collective behavior of automata thus defined utilize the language of game theory. However, the definitions of automaton games and ways of behavior in the games which arise here differ considerably from the point of view accepted in game theory. In fact, in game theory it is assumed that the system of payoff functions, determining the game, is known beforehand by the players. The player is supposed to use this information in order to determine his strategy (usually mixed), which during the game itself would remain unchanged; in choosing the strategy one can decide to use any computing means. Automaton games, however, are defined by specifying not only the systems of payoff functions, but also the structures of the participating automata. The automata taking part in games do not possess any apriori information about the game. The actions of each automaton are determined only by its wins and losses in the course of the game. The role of the payoff functions, defining a game, and that of the opponents of the automaton thus reduce to formation of a more or less complex random medium in which the automaton should be capable of expedient behavior.14 l4 Krinskiy and Ponomarev [I061 discuss the problem of how players should behave who do not have apriori information about the matrix of a game. An algorithm is described which assures that a player will have a guaranteed payoff arbitrarily close to the value of the game (when the game is repeated sufficiently many times). For simplicity, we shall describe this algorithm for the case in which the matrix of the game has a saddle point. Suppose that the first player has n actions; the number of actions of his opponent is not important. For every action, a player remembers the payoff ob-

45

Finite Auiomaia and Modeling ihe Simplesi Forms of Behavior

In selecting the construction of the playing automata, it is therefore natural to require that their behavior be in every case expedient in the simplest game -the game against nature. The absence of any apriori information about the payoff functions leads naturally to the use of symmetric structures for the automata. In the examples of automata games described in the following sections, we shall use structures of automata belonging to the asymptotically optimal sequences described in Section 2 of this part. In concluding this section, we shall give a simple example of an automaton ,game. Even though this game is not ergodic, it is possible to follow certain characteristic features of the behavior of automata in a game. In this game, four automata participate with a linear tactic:

Thus, these automata have, respectively, 2, 3, 4, and 10 strategies each, where

In the case of a win, each of the automata continues to use the previous strategy, and in the case of a loss, each of the automata replaces that strategy by the next (the last strategy is replaced by the first). The conditions of a game consist of the following: for each play f ( t ) = ( f l ( t ) , f 2 ( t ) f, 3 ( t ) ,f 4 ( ( t ) ) we , form the sum o ( t ) =f

Vt)

+f Y t ) +f 3 ( t ) +f 4 ( t > .

Here automaton wins a playf(t) if o ( t ) I 12, and loses in the opposite case. Automata V, \.!win laa play if a ( t ) exceeds the numbers 5, 8, 13, respectively. A typical sequence of playsf(t), values a ( f ) ,and the outcomes s ( t 1) are shown in Table I I . We see that, beginning with the eighth play, all automata except for begin to win and stop changing their strategies; automaton 3' begins to lose and changes its strategies cyclically.

a3,

+

tained the last time this action was executed. At every instant of time, the player performs the action to which the maximum payoff is attached. I n this case, throughout the game a payoff less than the value of the game is obtained n o more than n - 1 times. For matrix games without saddle points, one can construct an algorithm involving mixed strategies which guarantees an average payoff arbitrarily close t o the value of the game if only the game is continued sufficiently long (Editor's note).

46

Automaton Theory

TABLE II t

1 2 3 4 5 6 7 8 9 10

f1

fa

fS

f4

u

1

1

1 2 3 3 3 3 3 3 3 3

1 2 3 4 5 6 7 8 8 8

4 7 9 10 11 12 13 15 14 15

1 1 1 1 1 1 2 1 2

2 2 2 2 2 2 2 2 2

sa

ss

54

0 1 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 1 0 0 1 0 0

1 1 1 1 1

51

I

1 0 0 0

In the following section we shall describe zero-sum games for two automata that are equivalent to zero-sum games for two persons, the wellknown von Neumann theorem holding in this case. In Part 111 we shall discuss homogeneous automaton games, i.e., those automaton games in which all participants are equivalent. 2

Zero-Sum Games for Two Automata

In the foregoing section we have given a general definition of an automaton game. Now we shall consider in more detail two examples of such games. Both examples involve zero-sum games for two automata. The first one involves a game for an automaton playing against an opponent who had chosen a mixed strategy in the sense of game theory, and the second involves a game for two identical automata capable of expedient behavior in random media. For the first example, it will be shown that, for any asymptotically optimal sequence of automata, the win approaches the value of the game if the opponent uses the optimal strategy. In the second example, the value of the automaton game is in a certain sense analogous to the value of a game in the sense of von Neumann's game theory, although it does not coincide with it. Let us first define a zero-sum game for two automata. Consider a game r, in which automata (11' and 912 have, respectively, M and N strategies ( X I = M , x2 = N ) . We assume that in each play f = ( f 1 , f 2 )of this game one of the two automata wins and the other loses. Then the probabilities p ( f , s) = p ( f ' , f 2, sl, s2) of the outcomes (sl, s2) are zero for s1 = s2.


47

The automaton game thus defined will be called a zero-sum game for two automata. The quantity 4flf2 = P ( f ' , f 2, 0, 1) means the probability of a win for the first automaton in a playf= (f',f2), and Pf'f2 = f 2 , 1, 0 ) P(f'9

represents the probability of its winning in this play. According to Eq. (44),

The expectation m f l f 2of a win by automaton X' in the play f = ( f ' , f 2 ) is equal, by virtue of (45), to

Of course, the expectation of the sum of wins of automata X' and a2is zero. The quantities mapform a rectangular matrix 1 1 m,@11, a = 1,. . . , M , = 1,. . . , N , coinciding with the matrix of the equivalent zero-sum twoperson game. If the game is ergodic, then there exist final probabilities of the state of the syytem of the participant automata, and the value of the expectation W(%', W, I') of a win for the first automaton does not depend on their initial states. This value will be called as a convention the value of a game r for automata cU1 and cU2. Let R"1"2denote, as before, the final probability of the state of the system for the two automata participating in a game r. In this state automata Iu" and 912 are in states &I and &, and use strategies hil = P(&) and = F-(&), respectively. According to (49), the value of the expectation W(%', a2,r)of a win for automaton 3' can be calculated using the formula

Now we proceed to consider the examples. Consider first a game for automaton X against opponent U , using some fixed mixed strategy. We shall show that for any asymptotically optimal sequence of the automata, their win approaches the maximum possible value, and if the opponent uses the optimal strategy in the von Neumann sense, then this maximum coincides with the value of the game.

48

Automaton

Theory

In fact, suppose that the opponent U of the automaton 'u in the game realizes some mixed strategy x = (x,,.. . , xLV), i.e., in each play he uses his pth pure strategy, p = 1 , . . . , N , with the probability xp, x1 x N = 1 . By definition of mixed strategy, xp are functions of the matrix I I mup1 I of the game I',and d o not depend on the behavior of the opponent. Then for any pure strategy f,,a = 1,. . . , A4 of automaton 'u, we have the expectation value mu of winning:

+ -

+

Thus, a game with an opponent who has chosen any mixed strategy determines a stationary random medium C ( m , , .. . , m.lf),for any x: W(%, u,

r)= W(%, C ) .

If ?Il,. . . , %,,. . . , is a n asymptotically optimal sequence of automata, then, by definition,

w = lim n+m

w(%,U, I')= max(m,, . . . , mM) = max 2 m,pXB. N

a

8=1

(55)

an,

Thus, automaton for sufficiently large n, maximizes its win. If the mixed strategy x = (x,,.. . , xg) is optimal, then the expectation of a win for automaton 'un (for n 4 co) approaches W = max min a

2 morpXp, N

z B=1

i.e., it coincides with the value of the game according to von Neumann. One can say that such an automaton plays n o worse than its partner who has chosen an optimal strategy, even though it does not have any apriori information about the structure of the matrix 1) map1) for the game I', and receives all the necessary information during the course of the game itself. Behavior which is expedient in a stationary random medium turns out t o be expedient also in the example of the game. The foregoing considerations can easily be generalized to the case of a game for many automata, one of which belongs t o an asymptotically optimal sequence, the rest using mixed strategies. Let us now examine the example of a zero-sum game for two identical, symmetrical automata. For such automata, we shall make use of stochastic automata BMUn,~tt and B N n . N which , will be defined in the following way. Automaton B,,,,


49

has xn states via,a = 1,. . . , x , i = 1,. . . ,n, and in state viait uses strategy L. For s = 0 (a win), the transitions of states occur in the same way as

for an automaton with a linear tactic, namely, state via passes into state i = I , . . . , n - I , and state ynapasses into itself. For s = 1 (a loss), , 2,. . . , n. However, state via, in contrast state viapasses into state q ~ ; - ~i = to automata with a linear tactic, passes with equal probabilities (= I/x) into states vl@,,b = 1,. . . , x . It is not hard to verify that a sequence of automata B,,,, is asymptotically optimal in the same media as the automata with a linear tactic. Assuming that a game is ergodic, we shall show a method of calculating the value of the game W(BMn,M,BNn,N, Then one can show that the limit

r).

W = lim W B M , , M ,B N ~ .,Nr) n+m

exists and possesses the following properties : 1. If the matrix I( map11 of a game T contains even one row consisting of nonnegative elements, then W is the harmonic mean of the elements of the row from that group whose smallest element is largest. In a similar way, if 1 I mapI I contains at least one column all of whose elements are nonpositive, then W is the harmonic mean of the elements of the column from that group whose largest element is smallest. 2. If the conditions of Case I are not satisfied, W = 0.

Case 1 signifies well-known advantages for the first player: The presence of a row containing nonnegative elements means that he has a “loss-free” strategy. In this case the behavior of the first automaton resembles a cautious tactic, prescribed by game theory. One chooses a strategy that yields the maximum guaranteed payoff. The fact that the second player does not minimize the payoff to the first one, and is satisfied only with the harmonic mean, is related to the fact that a sequence of automata B,,,, is not asymptotically optimal in a random media all of whose parameters are negative. Case 2 shows that, in the absence of such clear preference for one of the players, the automata end the game in a draw: W = 0. In this case they play, so to speak, “more roughly,” since they are not able to make use of the subtler properties of the game matrix. However, even in this case, the value of W lies between the upper and the lower values of the game : max min mag5 W 5 min max ma@. (57) I

@

@

a

50

Automaton Theory

Thus, we see that in a game the automata capable of expedient behavior in stationary random media achieve results that are almost the same as those obtained using the methods of game theory and utilizing full a priori information about the game matrix. We shall outline the proofs of assertions 1 and 2. Let R:$ , a = 1,. . . , M ; B = 1,. . . , N , i, k = 1,. . . , n, denote the final probability of the state of the system of the participating automata in which the first one is in state qia, and the second in state vk'. Making use of the definition of automata B M n , M ,BNnSNand game r, we obtain the following system of equations for R:$:

and the normalization condition

2

RY!=

1.

a,B,i.k

We shall show first that R:$=O

for

i+ k # n +

1.

+ +

For this purpose we use induction with respect to s = i k . It will be noted that, according to (65), Ri!n = 0, and we assume that R:,$ = 0 for all i k = s, s > n 2. Then for R:,i, i k = s - 1, we obtain a homogeneous system of linear equations whose determinant is nonzero.

+

+

Finite Automata and Modeling the Simplest Form of Behavior

51

+

Similarly, by virtue of (65), RYfl = 0, and from Ri$ = 0 for i k = s, s < n, it follows that Ri$ = 0 for i k = s - 1. Therefore, the solution of Eqs. ( 5 8 ) can be written as

+

+ c,,A~j',

R:$+l-i = b,,

where

A,,

= qao/pa,.

(67)

From Eqs. (63) and (64), we obtain for b,, and capthe following system of 2MN equations: qaaba,

+ Pa,cap

Pa,ba, iqa&ilcafi

=

(68)

= ra 9

(69)

where

Solving Eqs. (2.25) and (2.26) for b,,, cap,we obtain

Summing Eq. (68) over a, and Eq. (69) over we get M N equalities:

+

B,

and using (70) and (71),

Substituting the value of b,, from (72) into (74) and (75), we obtain a system of M N equations for a,, t,:

c

+

N

ta

6-1

-

l)/(A:ll

c (Ay, - l)/(A;;l M

a,

(4x6

Y-1

- 1)

N

=

-w 3 i;

C 6-1

-

c

l)/(AZ1 -

a = 1, 2 , . . . ,M ,

(76)

M

=

Y-1

-

l)/(A;;l - l b Y , p = 1, 2,. . . ,N .

(77)

Krylov and Tsetlin [IIJ] give the solution of this system for M = N = 2. Here we shall limit ourselves to considering only the limiting case n + 00.

52

Automaton Theory

Consider first case 1. Suppose that the matrix ( 1 map11, a = I , 2,. . . , M , /? = 1,. . . , N contains rows consisting of nonnegative elements. Without

loss of generality, let us assume that the first L rows possess this property. Then A,, > I for a = 1,. . . , L, /? = I , . . . , N . We set

and note that for n -+

and for n

+ 00

and A,,

00

and A,, > 1,

< I,

pxp+ 1 - A,,,

VuS

-

-AZp

+

0.

(80)

Using this notation, we eliminate as from Eqs. (76) and (77). Interchanging the order of summation, we obtain

We note now that the sums vaa over column elements appearing in the denominators of the right-hand side of (81) are finite for n + 00, since by assertion 1, the matrix 1 1 mUp11 does not contain columns all of whose elements would be negative. Among the products v,apu,a for y > L, a 5 L there is at least one that remains finite for n + 00 since the rows with indices y > L contain negative elements and the rows with indices a 5 L consist of positive elements. If, however, y 5 L, then all these products vanish when n + 00. The coefficients o f t , on the left-hand side of Eqs. (81) approach zero for n 00 if a 5 L. Thus, in Eqs. (81) for n + 00, the coefficients of t,,tend to zero for y 5 L, and remain finite for y > L. Therefore, t y ,y > L, have a higher order of smallness than r y , y 5 L. Neglecting these, we shall rewrite (81) as --f

We set paao(a)= max6 pad.Then, neglecting the terms of greater degree of smallness, we obtain

Finite Automata and Modeling the Simplest Forms of Bekvior

53

Let ,uaod0 = min, pad0(a).It is easy to see that the coefficients of ta0 have a higher order of smallness than the coefficients of t,, a # ao. Thus, we have shown that t, + 0, a # ao. Therefore, in Case 1 , we choose the strategy a. ,for which min, maxdpea is attained, or identically, max, mind mad. Now let us calculate the limiting value W for the game T. We return to Eq. (77). Using the notation of (78) and neglecting terms of higher order of smallness, we have M

The coefficients of as on the left-hand sides of Eqs. (84) are finite since any column of the matrix 11 mas11 contains positive elements. Thus, all as have a higher order of smallness than t a o . Let us rewrite Eq. (53) for the value of the game taking into account (67): W(BMn,.u BNn,x 3

r)= c,8 {nbuo + [(A$

- 1)/(%8 - l>Ica~}mas*(85)

According to (74) and (75), we have

c masbas

a.8

= 0.

Next, using the expression (73) for cap, we have

We note that in this expression the coefficients of as, t, remain finite for n + 00 both when Aas > 1 and when A,@ 5 1. Neglecting as, t,, which tend to zero (for a # ao), we obtain

r)= Ntu0.

W = lim W(BMn,M, BNn,N, n+ca

(87)

Taking into account the normalization condition (66), we obtain N

w = N ( C m;oi)-l. 8-1

We have studied the case in which the game matrix contains rows consisting of positive elements. Of course if the matrix contains columns consisting of negative elements, the corresponding analysis is exactly similar.

54

Auromaton Theory

Now let us consider Case 2. In this case any row of the matrix contains negative elements, and any column contains positive ones. Therefore, Eqs. (76) and (77) involve all t a ,ap with finite coefficients. The value of the game W ( B M n , M , B N n , NI ,') will be W(BMn,M B N ~ , N 0 3

9

The first sum in the numerator of this fraction is zero by virtue of (74) and (75); the remaining sums remain finite as n + 00. The presence of a term increasing with n in the denominator means that

w = n+cu lim W ( B i v n , M , B . V n , N ,

= 0.

The foregoing discussion of automaton games permitted us to make conclusions only about the final stationary states. Attempts to analyze the transient processes were undertaken by means of modeling the games described here on digital computers.15

o

mu

10ou

k

Figure 9

0.4 0.8 0.7 0.6 1.0 0.1 0.3 0 . 9 -0.1 -0.3 -0.9 0.1

0.3 0.2 0.9 0.7

'

l5 These examples of simulation were included in the text by us; they were taken from Krylov and Tsetlin [113, p. 9861 (Editor's note).


55

The first two rows of this matrix contain no negative elements, so that the game comes under Case 1. The plots differ in the memory capacities of the automata: for curve 1, n = 19, and for curve 2, n = 5. The solid horizontal line V = 0.47 coincides with the limiting value of the game W, which in this case is the harmonic mean of the elements of the first row. For n = 19 and k > 1O00, the value of V(k) differs very little from W. For n = 5, the value of V ( k )converges to a limit which differs considerably from W.

Figure 10

Figure 10 gives similar plots for the matrix 0.1 -1.0 0.4 0.2 0 . 3 -0.5 0 . 6 -0.3 -0.3 0.2 -0.5 0.3

0.3 1.0

-0.3 -0.4

,

which has no rows consisting of nonnegative elements and no columns consisting of nonpositive elements, i.e., it comes under Case 2. It is clear in the figure that V(k), even at small values of k, differs very little from zero both for n = 19 and for n = 5 (curves 1 and 2, respectively).

III HOMOGENEOUS AUTOMATON GAMES 1 Homogeneous Games

In this part we shall describe homogeneous games, i.e., games in which all participants have equal rights. For this purpose it will be useful to introduce a group of automorphisms of a game. In our study of homogeneous games we shall use such concepts as the value of a play and the invariant set of plays. Among homogeneous games of particular interest

56

Automaton Theory

are those games for which at least one invariant set of plays is stable, i.e., Nash games. If the invariant set of maximum value turns out to be stable, then the corresponding game is called a Moore game. We shall propose a procedure making it possible to construct from a given homogeneous game T* another game To*, which has the same set of players, their strategies, and the same values of the plays as in the game r*,but which is a Moore game. Such a procedure is equivalent to an agreement among the players in a homogeneous game that their winnings in this game will be divided equally. Furthermore, we shall study homogeneous automaton games. In view of the fact that an analytical investigation is rather cumbersome, we made an attempt to analyze such games by means of computer simulation. Examples of such simulation constitute the contents of the last two sections. Before we state the definition of a homogeneous game, it will be useful to introduce definitions of the mapping of a game and of the automorphism of a game. Thus, consider a game r*,played by A’, . . . , A”. Suppose that the player A j has xi strategieshj, f $ , . . . ,f i j , j = 1,. . . , Y available. A play f of the game will be defined as before in the context of an automaton game (see Part 11, Section 1 ) as the set f = (h:,.. . ,fi:), whereh: is the strategy of the j t h player. There are Y functions V(f ), . . . , V”(f ) given on the set of plays defining the game T*. We shall state that a mapping g of a game into itself is defined if we are given:

r*

r*

(I)

a one-to-one mapping g of the set of players into itself:

(2) a one-to-one mapping of the set of strategies of a player A i into the set of strategies of a player M i :

The mapping g defined a mapping into itself of the set {f)of plays of the game In fact, a play f = (fit,. . . , g )is mapped by g into a play

r*.

kl = gj(ij)

if

gj = I,

I = 1,. . . , Y.

Finite Automata and Modeling the Simplest Form of Behavior

57

A mapping g will be called an automorphism of a game F*, if it preserves

the payoff functions, i.e., if for any play f of the game F*, we have the equality Vj(f ) = V q g f ) . (91) Evidently, the set of all automorphisms of the game F* forms a group G p . A game F* is called homogeneous if the group Gr. of the automorphisms of the game is transitive on the set of players, i.e., if for any pair of players A i and A j ther ' 3 an automorphism g such that gAi = A j . Obviously, in a homogeneous game the sets of strategies of all its players are pairwise isomorphic. Suppose now that f is an arbitrary play in a homogeneous game The value U(f ) of a play f will be defined as the arithmetic mean of the expectations of the winnings of all players: I

r*.

where vg is the number of plays of the form gf: In other words, for a homogeneous game F*, the value of a play coincides with the average payoff to any players over the set of plays {gf }, g E G p . The set { g f } will be referred to as the invariant set of plays, generated by a play S, and U( f ) will be called the value of the invariant set {gf }. It will be noted that in a homogeneous game the payoff function for any player is uniquely determined by the payoff function of one of the players and the group Gro of the automorphisms of the game. This circumstance considerably simplifies the description of a homogeneous game. We shall give two examples of homogeneous games. Example 1 Consider a game with two players A' and A2, each of which has N strategies f l a , . . . ,f N a , a = 1, 2. Let V1(f:, f k 2 ) be the winning of player A', using his ith strategy against the kth strategy of his opponent. Suppose also, that

V'(hl,f k 2 )

=

V2(fk1,

This game has a mapping g which acts as follows:

(94)

58

Automaton Theory

By virtue of (94) the mapping g preserves the payoff functions and thus is an automorphism, so that the game under consideration is homogeneous. The plays ( f i 1 , f k 2 and ) ( f k l , f : ) form the invariant set. Example 2 A homogeneous game for v players will be called Z*-symmetric if the group Gr. of its automorphisms coincides with the symmetric group of the permutations of the indices 1, 2,. . . , Y . Lethj,. . . ,fld be the set of strategies of player A j , and let V(f) = P(fi:, . . . ,hi) be the payoffs of this player. Also, let

w-)

=M(h$

a1,.

..

7

a.v-A.

(95)

We shall limit ourselves to symmetric games for which (95) is valid.I6 In this formula, a, is the number of players (other than A j ) who have selected in a playftheir s t h strategy,” s = 1, 2,. . . , N - 1. Let ij = k. Then the invariant set { g f } , generated by the play5 consists of all such plays of the game Z* in which B1 players select the first strategy, B2 players select the second,. . . , BN players select the Nth and

In the study of homogeneous automaton games, it seemed natural to us to separate out those games, in particular, the Nash games and the Moore games, in which the behavior of automata can be compared with the behavior of intelligent beings who know the conditions of the game beforehand. In the foregoing section, such a possibility was assured by virtue of the fact that, for two-person zero-sum games, we have the von Neumann theorem. Now we shall proceed to isolate certain special classes of homogeneous games for which such a comparison is possible. Suppose that in an arbitrary game there is a play f N, in which it is not convenient for any of the players to change their strategies unless the remaining players change theirs. The play f will be called a Nash play.

r*

l6 Equation (95) indicates that it is possible to number the set of player strategies in such a way that the kth strategy of any player will always change into the kth strategy of another player in the case of autornorphisrn. l’ Just as previously, we may state that player A’ selected his sth strategy in the set f = (fil, . . . ,hi) if il = s.

59


The definition of a Nash play f inequalities :

=

(hi,.. . ,A;)

reduces to a system of

mh: . . A; . . ,h:>P mh: . . Aj 9 .

9.

9 .

9. * *

JQ

(96)

for all j = 1, 2,. . . , v and all strategies & a,/2. We shall describe the results of simulating the game of distributions on digital computers.'* In all these examples we used automata with a linear tactic L,,,.,. Example 1 The game was played by five identical automata L7n,7.Each of the automata could choose one of seven possible strategies with the following powers: a, = 0.9, a2 = a3 = . .. = a7 = 0.33. How would people behave in a similar situation? It is clear that for each one it is more convenient to select the first strategy if it is free. Even if somebody had already chosen the first strategy it would still be more Excerpts from Ginzburg ef af. [64] are given later on in this section (Editor's note).

66

Automaton Theory

convenient for another to also choose the first strategy, since the expected value of winning in this case is equal to 0.45, and this is higher than the expected value of winning for any other strategy. If, however, the first strategy were chosen by three players, then each one of them would win 0.3, i.e., less than for any of the remaining strategies. Consequently, the most natural distribution is one in which any two players choose the first strategy, and the remaining players choose one of the remaining strategies each; besides, this distribution is not much different from one in which the first strategy is chosen by three players. Now, what is the behavior of automata in this case? This version was checked for automata with various memory capacities. For n = 10, it turned out that in 78% of the plays the first strategy was used by two automata, in 1274, by three, and in 9%, by one automaton. The strategies in the second-through-seventh group are, practically speaking, chosen simultaneously by one automaton only. Since in a majority of plays the first strategy is selected by two automata, the remaining six strategies are subject to selection by three automata, so that we can naturally expect that each of these strategies will be available in approximately onehalf of the plays, and in half of the plays it will be taken by one automaton. For a memory capacity n = 10, the simulation showed that the second strategy was used by one automaton in 48% of the plays, and in 51 % of the plays it was not used at all; the strategies in the third-through-seventh group are selected in a fashion similar to that of the second. For automata with the memory n = 5, the first strategy is used in the following way: in 34% of the plays it is used by one automaton, in 55% of the plays by two, and in 10% by three automata. The second strategy is available in 49 % of plays, in 47 % of plays it is used by one, and in 4 % by two automata. It is clear that an intelligent tendency is maintained in the behavior of the automata, but the picture is more blurred. Thus, the number (34%) of plays in which the first strategy is selected by only one automaton is still large; on the other hand, the fraction of plays in which the second strategy is simultaneously chosen by two automata is quite noticeable (about 4%). If we consider that the first strategy is, in an overwhelming majority of cases, used simultaneously by two automata, and the remaining ones use one strategy from the second-through-seventh group each, then the expected payoff to every automaton should be close to 3(2 x 0.45 3 x 0.33) = 0.378. The average payoffs obtained as a result of the calculation are given in Table 111 (upper row).

+

67

Fimte Automata and Modeling the Simplest Forms of Behavior

TABLE HI Automaton memory Average payoff without a common fund 0.353 0.368 0.370 0.373

-

Average payoff with a common fund

0.315 0.327 0.335 0.350

Example 2 The game was also played by five identical automata L7*,, , each of which could choose one of seven strategies with the following powers: a, = 0.9, a, = a3 = = a, = 0.15. The Nash set is defined by the collection (5, 0,. . . , 0). By analogy with the behavior of the automata in the previous example, it seemed that all the automata should choose the first strategy, since then the payoff to each one of them would be approximately equal to 0.18, which would be more than the payoff involved in the choice of any other strategy. The calculations have shown, however, that the first strategy is more often chosen by four, three, and even two automata. This is not hard to understand. In fact, if the first strategy is selected by five or four automata simultaneously, then the expectation of a win for each one of them is relatively small, and the probability of a strategy change is quite high. However, the average payoff falls off with an increase in the memory capacity, and apparently for n + 00 it tends to the value of the Nash set. In this example the relationship between the average payoff and the memory capacity is given in Table IV (upper row).

- -

TABLE IV

Average payoff without a common fund

0.25

0.23

0.22

0.21

Average payoff with a common fund

0.23

0.27

0.29

-

68

Aufomafon Theory

Example 3 In this example, as in the two preceding ones, v = 5, x = 7, and all powers of strategies are selected to be identical: a, = a2 = ... = a7 = 0.6. Already for n = 5, in 98% of the plays, each strategy was chosen by no more than one automaton so that p5= 0.6. The game in Example 3 is a Moore game, and the Nash set in this game is the invariant set of maximum value, i.e., a Moore set. With the aid of the “common fund” procedure described in the preceding section, it is not difficult to construct the corresponding Moore game Bo* for the distribution game. We have already said that this procedure is equivalent to an agreement among the players to divide their payoffs equally. For the distribution game with a common fund, the payoff to each player in a play coincides with the value of this play. For this reason all plays of maximum value are Moore plays, in which a change of strategy is equally inconvenient for all players. It is natural, therefore, in the corresponding automaton games to expect that each of the automata achieves (for sufficient memory capacity) the maximum value of the play. We have already mentioned that an automaton game is no different in its results from a game between people who know the conditions of the game beforehand and reach an agreement about joint actions in the game. This statement was confirmed by simulation. Example 4 A simulation was made of a game having a common fund the conditions of which were the same as in Example 1 of this section. The relationship between the average payoff r,:to an automaton in this game and its memory capacity is shown in Table 111 (lower row). For this game the value U ( f M )of a Moore play is &(a,

+ a2 + a3 + + a5) = i(0.9 + 0.33 x 4) = 0.444. U,

The experiment shows that the value of a Moore play is achieved in this case. Figure 11 contains plots for this example of the average payoff to an automaton for a game without a common fund (graph 1) and with a common fund (graph 2) versus the memory capacity n. We see that for n > 6 the procedure of introducing a common fund increases the average payoff to an automaton, and for n 5 6 it decreases it; the introduction of a common fund turns out to be advantageous only for automata with sufficiently high memory capacity. The same figure makes it clear that the average frequency ~ 7 , ~ ( f of ”) Nash plays for the game B* without a common fund tends, for C,(fM).

Finite Automata and Modeling the SimpIest Forms of Behavior

69

to unity faster than the average frequency n + 03 of a Moore play for the game Bo* with a common fund. This can be explained by the fact that in a game with a common fund, a change of strategy by one automaton has relatively little influence on its payoff, and only for fairly high memory capacities is the value of a Moore play achieved. For small memory capacity, however, the selection of the best strategies is not made with sufficient accuracy. Therefore, the common fund procedure turns out to be disadvantageous with small memory capacities of the automata (one might say, “wage leveling damage”). Example 5 In this example a digital computer was used to simulate the distribution game with a common fund, the conditions of which were chosen to be the same as in Example 2. Also in this case for n + 03 the value of a Moore set is achieved which for this example has the value i(0.9 4 ~ 0 . 1 5 )= 0.3. The results of simulation are given in Table IV (lower row). It is easy to see that the remarks made in Example 4 are also valid for this example. The example of the automaton game described in this section can be related to certain considerations involving the reliability of the functioning of an automaton group. In fact, consider a set of distribution games having identical powers of strategies and differing only as to the number of participants. These games have Nash plays in the absence of a common fund, and Moore plays when a common fund is introduced. In both cases, as shown by a computer simulation experiment, the frequencies of Nash plays (or Moore plays)

+

70

Automaton Theory

approach unity as n 400. This predetermines the choice of strategies by the automata participating in the game, independently of their number. In particular, if one of the participant automata fails (according to Glushkov’s ~lassification~~), the remaining automata again for some time choose strategies with the highest powers of the sources. These strategies are selected by the automata independently of which automaton suffered a failure. In particular, in our experiments we periodically failed the automaton that had chosen strategy f l , whose power of source was maximum, and followed the changes in the average payoff to a participating automaton. As the number of the playing automata decreased, the average payoff increased; the number of the strategies used was reduced by eliminating the least advantageous ones.

Example 6 The results of such a simulation for the game of distributions with powers of strategies a, = 0.9, u2 = a3 = = a, = 0.33 (as in Examples 1 and 4) and memory capacity n = 10 are given in Table V.

--

TABLE V

w,, w:,,

It is easy to verify that the values of and are close to the values of Nash and Moore plays, respectively. Moreover, it should be noted that the increase in the average payoff to an automaton with a decrease in their number leads to a situation where the total average payoff to the playing automata falls off relatively slowly with an increase in the number of automata that failed. The increase in the average payoff to the automata with a decrease in their number is related to still another circumstance. For a fixed memory capacity n of the playing automata and with a decrease in their number, the maximum values of plays are more and more reliably achieved. This is connected with the fact that the game is, as it were, simplified, and the I@

See Glushkov [70,p. 3511.

71


previous memory capacity makes it possible to bring the frequency of Moore plays closer to unity. The following example may serve as an illustration of this remark. Example 7 The powers of the sources were selected as follows: a, = 0.9, a, = 0.45, a3 = 0.225, a, = 0.1 12, as = 0.056, a, = 0.028, a, = 0.014. This game with a common fund was first played by five, then by four, and then by three automata. Each time, the automaton that chose the strategy fi,possessing the highest power, was made to fail. In Table VI, U ( f ‘ ) denotes the value of a Moore play. TABLE VI I

I

I

5

I

1

I

I

1I 8:); 1 I 1 1I 0.35 10.28 10.29 10.3 0.32 0.37 0.4 0.43 0.47 0.52

3 Circle Automaton Games In the study of multiautomata games, it is natural to separate out those games for which the description does not depend on the number of players. This property is possessed, in particular, by those games for which the payoff functions of each player depend on the choice of strategies by this player and a limited number of other players-his “neighbors” in the game. In this case the limited number of the arguments of the payoff function considerably simplifies the description of a game. It is convenient to juxtapose games with a limited number of neighbors and special graphs of games. For this purpose a player A k is put in correspondence with a vertex k of the graph. If the payoff function of player A k depends on the choice of a strategy by player Ai, then an arrow is drawn from the vertex i to the vertex k . In games with a limited number of neighbors, each vertex of a graph brings together a limited number of arrows. In this section we shall describe the simplest homogeneous automaton game with a limited number of neighbors as well as the results of its simulation on a digital computer. Consider the following game K * . The game is played by v persons A’, A 2 , . . . , A’, each of whom has x strategies fi ,. . . ,f,. Let f = (fi,,. . . ,&)

72

Automaton Theory

be an arbitrary play of the game. Then the payoff function is given by the formula

W(f)of a player

Players Aj-’ and Aj+’ will be called the left and right neighbors of player A j , respectively. Thus, the payoff to each player in any play of the game is determined by his strategy and the strategies of his two neighbors-on his left and on his right-and the game is defined by a function of three variables, independently of the number of players. A game thus defined will be called a circle game.20 Any automorphism g , , k = 0, I , . . . , v - I of a circle game can be described as follows:

+

+

(a) g,Aj = A j + k if j k 5 v, and g,Aj = A j t k - ” if j k > v; (b) a strategy f a of a player A’ is mapped into the strategy f , on a player g k A j . I t is obvious that the set of automorphisms g , forms a cyclic group of order v, where g,g, = g,+, for k s 5 v and g,g, = gk+s-vfor k s > v. The identity element of this group is the automorphism go for which goAj = A j . The group G K , of automorphisms of the circle game is transitive on the set of players so that the game is homogeneous. In what follows we shall limit ourselves to the simplest case of the circle game in which each of the players has only two strategies: 0 and 1. Then the choice by each of the players A j of a strategy e j , E~ = 0, 1 determines a play f = ( e l , . . . , E ” ) , in which the payoff V j ( f ) to the player A j is determined by the formula

+

+

Evidently, the invariant set generated by a play Plays (El

3 .

. . &A( E 2 9

9

E3,.

..

9

E,

9

E l ) , (83

9

E‘l

9

. . . ,E , ,

El,

( E ~ e2, , .

. . , E ” ) consists of

4,.. . , ( E ”

9

E l , E2

9 .

. . &”-A. 7

2o We have described a game on a circle with two neighbors. It is clear that we may determine a circle game with an arbitrary number of neighbors in a similar way. In this way we may determine a game on a torus, in which the payoff to each participant is determined by his strategy and the strategies of four of his neighbors, etc.


73

A homogeneous game K* may be a Nash game. In this case the sequence cl, E ~ . ., . , E , , determined by the Nash play f = (el, E ~ , .. . , e,), should possess the following property: If the sequence contains a triple E ~ , E ~ + then ~ , it does not have a triple E ~ E, ~ + ~ , such that E~ = E ~ E, ~ # + ~E ~ + ~ , ~ i = + E~ ~ + ~(The . summation over the index is done with respect to the modulus of Y.) In fact, if the sequence sl,. . . , E, contains a triple E ~ E, ~ + E~ ,~ + ~ and determines a Nash play, then V ( E ~E ,~ + E~ ~, + 2 ~ ) V ( e j ,E k , E ~ + ~for ) i?k # E ~ + ~This . sequence may not contain the triple E ~ E, ~ + eifZ, ~ , since in this case it would be advantageous for player Ai+’ to change his strategy. It may be verified that if, for some sequence E ~ . ., . , E , , this condition is observed, then one can construct a system of payoff functions V(E’,E “ , E ” ’ ) such that in the game defined by these payoff functions, f = ( E ~ , .. . , e Y ) will be a Nash play. We shall show that the sequence E ~ e, Z , . . . , E , , which may define a Nash play, belongs to one of the following two categories: (1) All E ~ j, = 1,. . . , Y are equal. (2) In the sequence E ~ , .. . , E , , 0’s occur only once, and 1’s no more than twice in a row, or conversely, 1’s occur only once at a time, and 0’s no more than two at a time. We shall show first that if the sequence E ~ . ,. . , E, contains three identical E’S in a row, then all of the E’S are identical. Suppose for definiteness that the sequence has the form . . . , 0, 0, 0, 1, E , . . .. Then if E = 0, the sequence will contain triples 0, 0, 0 and 0, 1, 0, and consequently cannot define a Nash play. This sequence also cannot define a Nash play when E = 1 since in this case it will contain the triples 0, 0, 1 and 0, 1, 1. We shall show now that if the sequence contains two 0’s in a row, then it cannot contain two 1’s in a row. In fact, if our sequence has the form 0, 0, 1,. . . ,0, 1, 1,. . . , then it contains the triples 0, 0, 1 and 0, 1, 1, which is impossible for a sequence defining a Nash play. We shall proceed now to describe Nash points in mixed strategies.21 Let us recall [122] that a Nash point is defined by a set t = (tl, t 2 , . . . , t,), where t k is the probability with which the kth player executes the action 0, k = 1, 2,. . . , Y. Let E k ( t )denote the expectation of a win by the kth player. The fact that t is a Nash point implies that d E k ( t ) / d t k= 0, whence atk-ltk+l

+ btkel + ctk+l + d = 0,

Excerpts From Bryzgalov et al. [32] will be given later on in the text (Editor’s note).

74

Alrtomaton Theory

where

Using (107) it is not difficult to express tk+l in terms of t k - l . This relationship will be simplified when we use the parameter tk = (1 - t k ) / t k : tk+l

= [rU(l, o ) t k - l

f p(0, 0)1/[-p(19

l b k - 1 - p(0,

(108)

We shall now find a general form for mixed strategies. For this purpose we note that tkfl= A t k p l , where A is the transformation (108), implies that tl = A Y t l . If AY is not an identity mapping, the fixed points of the transformation A” coincide with the fixed points of the transformation A . This is easy to show by direct verification. Hence, if v is odd, then all tk are equal, and if Y is even, all tk with even subscripts and all t k with odd subscripts are equal. Thus, we have shown that if v is odd, then any Nash point has the form ( t , t , . . . , t ) , where t is one of the roots of the equation at2+

(b+c)t+d=O,

(109)

and a, b, c, and d a r e defined in (107). If, however, v is even, then any Nash point has the form ( t , t ‘ , c, t ‘ , . . . , t , t ’ ) , where t and t‘ are the roots of Eq. (109). Equation (109), after t is replaced by t = (1 - t ) / t , becomes

where p ( ~E,’ ) are defined in (107). It is obvious that if p ( 0 , O ) and p(1, 1) have the same sign, then a play composed of 0’s or a play composed of 1’s is a Nash play; here Eq. (1 10) has either two positive roots or none. In this case, in addition to plays composed of 1’s and O’s, there may still be two Nash plays in mixed strategies or none. If, however, p ( 0 , O ) and p ( l , I ) have opposite signs, then Eq. ( I 10) has exactly one positive root. Then either a play of 0’s and a play of 1’s are both Nash plays, or neither of them is a Nash play. In this case there is an additional Nash play in mixed strategies. Moreover, we should note that for circle games with an even number of players, Nash plays always exist. In fact, if V(I,O, 1) < V(1, 1, I), then

Fimte Automata and Modeling the Sinq~lestForms of Behavior

75

the play (1, 1,. . . , 1) is a Nash play. If V(0, 1, 0) < V(0,0, 0), then the play (0, 0,. . . , 0) is a Nash play. If, however, neither of these inequalities is satisfied, then the play (l,O, l , O , . . . , 0) is a Nash play. We have already stated (see p. 42) that from a given play K*, one can construct a game K, i.e., a game with independent outcomes of v automata (u“,. . . , W.Here according to (47), the probability d ( f ) of winning for automaton ‘E[iin a play f = ( E ~ , .. . , EJ is defined by the formula

@(f) = 311

+

V ( ~ j - 1 ,E j , ~ j + l ) l ,

(111)

and the probability p ‘ ( f )of its losing in this play is defined by the formula P i ( f ) = 3[l -

V(Ej-1, E j , E j + l ) l .

(1 12)

So far as the construction of the automata participating in the game is concerned, we use, as before, symmetric automata forming an asymptotically optimal sequence. In particular, in the examples of simulation described in the following, we use automata with a linear tactic, L2n,n.One can assume that the results of simulation would not be considerably different if one used other sufficiently wide classes of symmetric automata belonging to asymptotically optimal sequences. In the simulation, we were, of course, first of all interested in Moore games, i.e., the games for which the invariant set of maximum value plays is a Nash set. Here, there is an opportunity to compare the behavior of a group of the simplest automata with the behavior of persons who have prior knowledge of the conditions of the game, and who therefore can agree as to whether or not to execute only those plays belonging to the Moore set. Let a,(f) denote the probability of an occurrence of a playf. (It is equal to 5 , ( f ) , which is the limit of the ratio of the number of times playfwas executed to the total number of plays made.) In all the examples considered there was a definite tendency for the value of a , ( f ) to change as the number n (automaton memory) increased. This leads us to assume that the limit of a,(f), as n + 00, always exists. This limit is reached with a desired degree of accuracy for relatively small values of n, and the convergence has, a p parently, an exponential character (see Examples 1 and 2). The latter circumstance is important also in connection with the following. Consider a circle game with v players that has a Moore playfM. Then, if the memory of the automata is sufficient, the behavior of the group has only a weak dependence on v, in other words, a n v ( f Mis) almost constant for sufficiently large n and a fairly wide range of v (see Examples 2 and 4). In the intro-

76

Automaton Theory

duction we have already noted that the “reliability” of collective behavior is related to that fact. Simulation data show that, with a favorable structure for the payoff function of the circle game, automata L2,,,,,execute Moore plays with a probability approaching one for n 00, i.e., in these cases --+

lim a n u ( f M=) I

n +m

The automata, as it were, “agree” to perform the actions for which the payoff to each one of them is maximum. I n these cases the expedient behavior of each one of the participant automata assures the expedient behavior of their group. In other cases, the automata are not capable of “agreeing” to take those actions that assure maximum payoff. In describing the examples of simulating the circle automaton games, we shall clarify the importance of constructing payoff functions of the game so that ( I 13) is satisfied. If this relation is satisfied, we shall say that the value of the Moore set is achieved. We shall note finally that in games which d o not contain Nash plays the collective behavior of automata is much more complex. A fairly typical example is the occurrence of a cycle, i.e., a sequence of playsf’, f z , . . . , f 8 with the following property: The automata participating in play f ’ , behave in such a way that play f ’ passes preferentially into play f2,and from it into f 3 , etc., into f 8 , from which again it passes into f ’ (see Example 6). Now we proceed to consider examples of game simulation. Example 1 We simulated a symmetric game for two players with the payoff function V ( E ~E ,~ ) ,el = 0.1, defined as follows: V(0,0) = V(1, 1)

=

0.25,

V(0, I )

= 0.9,

V(1,O)

= -0.1

Here V ( E , ,E * ) denotes the expectation value of a win by a player who performed action c1 at the same time as his opponent performed action E ~ It is easy to verify that playf” = (0,O)is a single Nash play in this game. The values of a , , ( f ) are given in Table VII. The quantity denoted in this table (and subsequent ones) by o l l ( . f )is in reality equal to the ratio of the number of times play f was executed to the total number T of plays executed. In all the examples considered, T was fairly large so that the value of a,,(f ) found by us was sufficiently close to the probability with which the play f was realized. Table VII makes i t clear that on(O,0) approaches unity as n increases.

.


77

TABLE W

Example 2 Consider the circle game in which the payoff function (which is the expectation value of winning) V ( E ~ -E ~~ ,E, ~ + ~is) defined as follows: V(1, 1 , 1) = 0.6, V ( O , O , O ) = 0.43, V(0, 0, l)=V(O, 1 , O)=V(l, 0, O)=V(O, 1, l ) = V ( l , 0, I)=V(l, 1, O)=O. This is a game with two Nash plays: f = (0, 0,. . . , 0), which is a play of O’s, and f = ( I , I , . . . , l ) , which is a play of l’s, the latter obviously being also a Moore play. The players who know the payoff function of the game beforehand find it natural to agree about always performing action 1 : this will guarantee them the maximum possible payoff. It turns out that with sufficient memory capacity, almost all automata throughout the game perform action 1 . The results of simulation are shown in Table VIII, where -depending on the number of automata v and the memory capacity n of each automaton-we list the fraction D of the plays composed of 1’s (as a ratio of the number of plays composed of 1’s to the total number of plays) and the frequency F with which each automaton performs the action 1. Table VIII shows that with sufficient memory the fraction of plays of 1’s is close to unity. It will be noted that, for automata with fixed memory the fraction of plays composed of 1’s falls off, although slowly, as v increases. Apparently, this decrease is on the order of v-= for a suitable a. Approximate curves showing this relationship are given in Fig. 12. The relationship between the fraction of plays composed of 1’s and the memory capacity for a constant number of players is plotted in Fig. 13. We note that D apparently depends exponentially on the memory of the automata. The simulation made it possible to follow the dynamics of the game. In this process it was discovered that the state of the group of automata,

78

Automaton Theory

TABLE Vm 6

f

D

9

F

- -

0.52

-

0.36 0.83 0.80 0.99 0.97 0.99 0.98

F

0.

- -

-

D

0.69 0.84 0.99 0.99

18

0.22 0.78 0.92 0.98

32

-

-

0.67 0.03 0.52 0.87 0.60 0.88 0.95 0.80 0.93 0.99 0.91 0.97

-

0.65 0.88 0.89 0.9

0.00. 0.451 0.72 0.81 1

1

when all of them perform action 1, is stable, i.e., a change by one automaton of its actions did not result in a separation from the others. It is interesting to compare the circle game for automata L,,,, with an automaton game where the automata are not capable of expedient behavior and choose their strategies randomly regardless of the outcomes of the plays executed. For such automata the value of o;(f') is obviously equal to 2-v. In the example described (just as in the ones that follow), the number of states of the Markov chain describing the game is on the order of (2n). lo3,. However, in the case of simulation on a digital computer, when the number of plays executed is on the order of lo5, one can clearly exhibit the states whose final probabilities approach unity when summed. We note that the value of a n v ( f Mfalls ) off as v increases, although relatively slowly. However, a decrease in the memory capacity n has a much

-

369

f3

Figure 12

32

Y

1

3

1

4

56717

Figure 13

Finite Antomata and Modeling the Simplest Forms of Behavior

79

stronger effect on the mean frequency. One can say that in the example considered the individual expediency of the playing automaton plays a more important role than does the number of automata participating in the game. In the simulation of circle automaton games, we noted that the value of the Moore play was not always achieved. To satisfy this relation, it is apparently important that the expectation of winning by the automaton that changes its strategy in the Moore play be not larger than the expectation of winning by its neighbors. In the case with sufficient memory capacity n, the probability of an automaton changing its strategy to return to the previous choice of strategy will be larger than the probability that its neighbors will change their strategies. Consider, for example, the Moore play f M = (1, 1 , . . . , 1) of the game in Example 2. If an automaton rlrj changes its strategy from e j = 1 to ej = 0, then its payoff changes from V(0, 1, 1) = 0 to V(1, 1, 0) = 0, respectively, and thus the expectation of winning by automaton ‘2li when it changes its strategy will not be larger than the expectation of winning by its neighbors.22 This argument apparently explains why relation (1 13) is satisfied for the game in Example 2. One may, however, select the payoff functions for the circle game in such a way that in the Moore play a change of strategy by any of the participant automata will result in a situation in which its expectation of winning turns out to be larger than the expectation of winning of its neighbors. For this case one can assume that the Moore play will, so to speak, be “washed out,” and the value of the Moore play will not be reached. To verify this assumption we shall study Examples 3-5.

Example 3 Consider a symmetric game between two players with the followingpayofffunctions: V(0,O) = 0.43, V(0, 1 ) = 0.43, V(1,O)= -0.1, V(1, 1 ) = 0.5. In this game there are two Nash plays: f = (0, 0) and f = (1, l), where the latter is the Moore play. The results of simulation are given in Table IX, which shows that the automata are in no position to agree about playing the Moore play. This is apparently due to the following: Suppose that both automata perform action 1. If one of them changes its action, then its partner, which continues to perform action 1, turns out

** In the case of a game with a common fund, when there is a deviation from the strategy of the Moore set, the decrease in the mathematical expectation of a payoff is identical for all automata participating in the game, just as in the case discussed for a game on a circle. This apparently is related to the fact that for games with a common fund, Eq. (113) is satisfied in all the modeling cases considered.

80

Automaton Theory

TABLE IX

?:A}

0.05

0.03 0.00

to be in a worse position, and therefore it also changes its action relatively rapidly, and both of them begin to execute action 0. On the other hand, if in play (0,O) one of the automata changes its action, then it would find itself in a worse situation, and play f = (0, 0) would begin again. Thus, the preferential choice of play f = (0,O)is determined by the values of V(0, I ) and V(1,0). This is also confirmed by the results of a simulation of the game with the payoff function V(0,O)= 0.43,

V(0, 1)

=

V(1, 0 ) = 0,

V(1, 1) = 0.5.

Here the results of simulation were the following: for n = 5 and n = 6, the fraction of plays f = (1, 1) is equal to 0.52 and 0.93, respectively, i.e., the automata agree to perform action 1 and obtain the maximum payoff. Example 4 As in Example 2, we consider the circle game with v automata having two strategies each ( E ~= 0.1). The payoff function of the game has the form V(0,0, 0 ) = 0.43,

V ( I , 0, 0) = 0.2,

V(0, 1, 0 ) = 0, V(1, 1,O)

=

-0.2,

V(0,0, 1)

V(0, 1, 1)

=

= 0.2,

-0.2,

V(1, 0, 1) V(1, 1, 1)

= 0,

= 0.6.

As in Example 2, this game has two Nash plays f = (1, 1,. . . , 1) and = (0, 0,. . . , 0), the first of which is also a Moore play. If in the p l a y f M = (1, 1,. . . , I), player %j, j = 1, 2,. . . , v , changes his strategy from 1 to 0, then the expectation of his winning becomes equal to V ( 1 , 0, 1) = 0; the expectation of winning of his two neighbors 9l-l and %j+lwill be equal to V(1, 1, 0) = -0.2 and V(0, 1, 1) = -0.2, respectively. In this case, when one of the players randomly deviates from strategy 1 prescribed by the Moore play, his neighbors in the game will also change their strategies. In this case, (67) will not be satisfied and the expectation of winning of the automata will of course be less than the value of the Moore Play*

f

81


Consider now the Nash playfN = (0, 0, . . . , 0). If any one of the playing automata, for example ‘ u j , changes its strategy from cj = 0 to cj’ = 1, then the expectation of its winning will become equal to -0.2, and that of its neighbors equal to +0.2. Therefore, it can be expected that if in play f = (0, 0,. . . ,0) one o f the players changes his strategy, then this player will return to the previous strategy faster than his neighbors will change their strategies. It is natural, therefore, to assume that for the game in Example 4, we have the relation lim o;(fN)

n+m

where

=

1,

(1 14)

(0,O) . . . , 0 ) .

f N =

The results of a simulation of the game in Example 4 on a digital computer are shown in Table X, in which one can follow the dependence of the mean frequency anv(fN)on the number v of the playing automata and on the memory capacity n of each one of them.

171 1 1 TABLE X

3

6

1 1 8

32

- 0.79 0.42 0.89 0.88 0.69 0.95 0.92 0.82 0.69

The table shows that Eq. (1 14) is satisfied for this game. It is not difficult to note that for Table X the mean frequency of the Nash play depends more on the memory capacity of the playing automata than on their number; we have already noted this circumstance when describing the results of a simulation of the game in Example 2.

Example 5 In this example, the payoff function V ( E ~ c- j~, cj+l), , of the game with Y automata on the circle is given as follows:

cj = 0.1,

V(0,0, 0)

= 0.43,

V(0, 1,0)= 0, 1,.

V(1, 0, 0)

=

-0.2,

V(1, 1, 0) = 0.2,

V(0,0, 1) V(0, 1, 1)

=

-0.2,

= 0.2,

V(1, 0, 1)

V(1, 1, 1)

= 0,

= 0.6.

Just as in the previous examples, plays f = (0, 0, . . . , 0) and f = (1, . . , 1) are Nash and Moore plays, respectively. However, when any one

82

Automaton Theory

of the playing automata deviates from strategy 1 which is prescribed by the Moore play, the expectation of winning of this automaton, V( 1,0, l)=O, will turn out to be less than the payoff to its neighbors, V(1, I , O )

=

V(0, I , I )

= 0.2.

In accordance with the assumption stated above, one can consider that Eq. ( 1 13) will be satisfied for this game. Moreover, it is natural to assume that the mean frequency o , ; ( f " ) for the values of v and n, the same as in Example 2, will assume values approaching 1. When simulating this game, it turned out that 02z(fM) = 0.09 and o"(fM) = 0.8. In Example 2 these values were 0.00 and 0.45, r e s p e c t i ~ e l y . ~ ~ Example 6 Finally, we shall consider at some length an example of a game in which there are no Nash plays. Let us specify in a three-person circle game the following payoff function: V(0,0, 0 ) = -0.9,

V(0, I , I )

=

0.9, V(0,0, 1 )

V(0, 1, 0 ) = 0.7, V(1, 1, 0 ) = 0.7,

=

-0.9,

V(1, 0, 0 ) = 0.7,

V(1, 0, 1 )

V(1, I , 1)

=

0.9,

= -0.9.

It is easy to verify that this game has no Nash plays. In fact, in the plays (0, 0, 0), (0, 0, I), ( I , I , I), it is advantageous to change the action of the middle player, and in the play (0, 1, 1), the action of the one on the right. The remaining plays of the game can be obtained from the preceding ones by a cyclic rotation. The mixed strategies can be described in the following way: The equation for the Nash point in mixed strategies [see ( I l O ) ] has the form 1.6t2- 0 . 4 ~- 1.8 = 0 .

.

Solving this equation, it is easy to find the final distribution of the frequencies of the plays. These frequencies and the results of a simulation of an automaton game with memory n = 8 are listed in Table XI. A typical situation is that a Nash play in mixed strategies is not realized by the automata. This can be explained by the fact that, in contrast with the point of view taken in the von Neumann-Nash theory of games, automata do not select their actions independently at all. Naturally, then, their behavior will not be described by specifying independent frequencies of the actions chosen by the automata. Let us consider in more detail the dynamics of the game. In play (0, I , 0), the first player loses most of the time, whereas the second and third win; as

In Table VIII, these values of 6,"(fM) were designated by an asterisk.

83

Finite Automata and Modeling the Simplest Forms of Behavior TABLE XI

f Frequency of play f at the Nash point

o,o,o

0, 0, 1 0,1,0 1, 0, 0

0, 1, 1

0.1

0.116

0.113

l,O,l

I,],]

1, 1, 0

0.153

therefore, this play is most likely to change to play (1, 1,O). In the latter the second player loses, and the remaining ones win; therefore, play ( I , 1, 0) will change into (1, 0, 0), etc. In plays (0, 0,O) and (1, 1, I), all the players lose, and therefore, the probability of their continuation is negligible. In the final part of our discussion of circle games with automata, it should be noted once again that in our analysis we have studied sets of games that differed from one another only in the number of players. In such games expedient behavior on the part of each automaton participating in a game assures the expedient behavior of the group of automata and its simultaneous increase in a relatively wide range. We have already noted that a change in expedient behavior of an individual automaton (which is related to the memory capacity n ) has a much stronger effect on the behavior of the group than a change in the number of participant automata. These remarks make it possible to speak of the reliability of a group of automata joined by their common participation in a circle game. Suppose that a set { K } of circle games is characterized by the payoff function V ( E ,E ' , E " ) and by the construction of the playing automata, and let, for example, f = ( I , I , . . . , I ) in all these games be a Moore play. The games in this set differ only in the number of players. With a proper choice of payoff function, the value of the Moore play is reached for all games in this set. Therefore, a failure of a number of automata does not result in a breakdown of ( 1 13), and, consequently, it will not change the choice of strategies on the part of the remaining automata. Moreover, as the number of playing automata decreases, the mean frequency of the Moore play will increase, so that the total payoff to the playing automata will decrease relatively slowly. This type of increase in a,;(f) is easy to follow in the table of examples described in this section.

0

An Example of Modeling the Behavior of a Group of Automata with a Two-Level Organization (The Numerical Method Distribution Problem)’ 0

In this chapter we shall describe a game played by several automata that admits various interpretations of its content. Consider the following “the numerical method distribution problem.”2 Suppose that we are given a set of stationary random media C = {Ci}, i = 1, 2 , . . . , N (see p. 13). Each medium C iis characterized by a set of . . numbers a I 2uzZ,. , . . , aQi,representing the expectations of winning for an automaton located in a medium C iand performing actions 1, 2 , . . . , e, respectively. At every instant of time t = I , 2 , . . . , there appears a medium C t with a probability p t . Of course, p 1 p 2 . . . p s = 1 . Suppose also that there are M automata R , , k = 1, 2 , . . . , M , each of which may perform actions I , 2,. . . , 0. We assume that M 5 N . Each automaton R, is at every instant of time located in some medium Ci, where each medium contains at most one automaton. If automaton R, is located in a medium Ci, we shall say that it is tuned to the index i. Suppose that at time t there appears a medium Ci.If some automaton Rk is connected to it at that instant, it will perform an action, will win or lose, and pass into another state. If, however, at that instant no automaton is connected to the medium, then the actions I , 2 , . . . , 0 are selected with equal probability. The win or loss as determined by the medium in both cases (in response

+ +

+

Reprinted with minor changes from the article written jointly with Ginzburg [65] (Editor’s note). * Cf. p. 120 concerning the same problem (Editor’s note). 84

85

The Numerical Method Distribution Problem

to an action of an automaton or a random action) is considered to be a win or a loss for the entire group of automata. The problem is this: Without knowing beforehand the numbers ujiand p i , we are supposed to select the automata REand organize their tuning to the indices in such a way that the expectation of a payoff for the entire group is as large as possible for all combinations of the numbers uji and p i . For the case in which M = N , it would be natural to tune each automaton to its constant index; then the automata would function in a stationary medium. If the automata belonged to asymptotically optimal sequences, one could guarantee (in the limit for infinite memory) the payoff W

=

Cglpimaxjuji.

If, however, the number M of automata is less than the total number of various media, then there is a problem of how to tune the automata to their indices so that the maximum possible payoff will be assured. Sometimes we shall assume that one automaton is tuned to the index 0. It will be turned on when a medium appears to whose index no other automaton is tuned. Consider the simplest example. Suppose there exist two media, C’ = (0.6; 0.9), C2 = (0.45; -0.45), appearing with probability 0.5 each, and suppose we deal with a single automaton. If the automaton is tuned to index 1 and it chooses action 2, its payoff in the case of medium C1 is equal to 0.9. In the case of medium C2, the random decision maker is the system, and the average payoff of the system will be equal to zero. Thus, considering that media C’ and C2appear with probability 0.5, the average payoff of the system will be 0.9 x 0.5 0 x 0.5 = 0.45. Similarly, when the automaton is tuned to index 2, the expectation of the payoff for the system is (0.6 0 . 9 ) ~ 0 . 5 0.45 x 0.5 = 0.6. Suppose now that the automaton is tuned to index 0. In this case it will be turned on when any medium appears. Since the expectation of winning in the case where the first strategy is chosen is 0.6 x 0.5f0.45 xO.5 = 0.525, and in the case where the second strategy is chosen it is 0.9 x 0.5-0.45 x 0.5 = 0.225, our automaton, being set to index 0, will assure a payoff not exceeding 0.525. Thus, in this example it is most advantageous to set the automaton to index 2. “The numerical method distribution problem” may be considered to be the problem of organizing the collective behavior of solving device^.^ We

+

+

+

That is, automata capable of expedient behavior are understood to be the devices which inaccurately solve the problem of choosing the largest number out of the numbers a,’, characterizing a medium C‘ (Editor’s note).

86

Automaton Theory

shall attempt here to organize their interaction in such a way that the expedient behavior of the individual solving devices results in the optimal behavior of the entire system when solving a problem. For this purpose, the type of tuning which will assure the maximum total payoff must be the one that is most advantageous. This is achieved by introducing the common fund. On the other hand, the payoff for each of the solving devices depends not only on which problem it has chosen, but also on how accurately it solves that problem. This implies that, for any fixed tuning of the solving devices to certain indices, a certain amount of time must pass in order that the solving devices be able to make a sufficiently accurate choice of strategies. Therefore, only the average total payoff for the solving devices over the time t,sufficient for them to reach a steady-state operation, can characterize the quality of a given type of tuning. The value of t can characterize the quality of a given type of tuning. The value of t should be selected in such a way that during this time it will be possible to estimate roughly the expediency of a given type of tuning. Of course, there is no need to choose a value of t such that the transient regime will have no effect whatsoever. This characteristic feature of the numerical method distribution problem makes it necessary to use two levels for its solution. Thus, during each cycle r = I , 2,. . . , a problem is chosen from the set C', C2,.. . , C", the corresponding automaton is turned on, a choice of the strategy is made and an evaluation of this choice takes place. At times T, 2 t , . . . , an evaluation of the quality of the choice of indices is made and the tuning to the indices is changed. At every instant f = I , 2,. . . , the actions of each automaton are characterized by two numbers (p, i), namely, the strategy chosen B, B = 1, 2, . . . , e, and the index number i, i = 1, 2,. . . , N . In choosing the number @, the automata act independently of one another. The functioning of each one of them is simply an operation in a stationary random medium. As the solving devices, i.e., automata R , , we shall use pairs of automata with a linear tactic, which will be denoted by ( A i l ) , ,4iz)), k = 1, 2,. . . , M. The actions of automaton Ail) will be the various strategies p = 1, 2,. . . , e; the actions of automaton Aiz) consist of choosing the index i = 0, 1,. . . , N . Here the states of automata ,4iz)and their actions do not change in the interval r = ( r - 1)t I , . . . ,r t . Automaton Ail), k = 1, 2,. . . , M is turned on only when a medium appears whose index i is the same as that to which the automaton Ai') is tuned, and it performs an action @ that corresponds to its state at that time.

+

87


+

Here the automaton wins with probability qoi = $(l usi) and loses with probability I - q;. The win and the loss of automaton Ail) change its state (in accordance with its structure). In this case a medium appears to whose index i none of the automata A i z ) , k = 1,2,. . . , M , is tuned; either some automaton Ail’ is turned on (if Aiz) has index 0) or the random choice maker is turned on. Consequently, in each cycle either one of the automata Ail), k = 1,2,. . . , M , or the random choice maker is turned on; the remaining automata do not change their states. The wins and losses of the automata and the random choice maker are summed over the period t,forming the total payoff of the entire system during that time. This payoff, when divided by the number of cycles t, determines the payoff W for any of automata k = 1,2,. . . , M at the time rt. Automata Ai2), which have N 1 actions 0, 1, 2,. . . , N each, at moments rt, r = 1, 2,. . . , win with probability q = &(1 W ) and lose with probability p = &(I - W ) , which results in a change in their states, and thus, in their actions. The choice of index is made in such a way that each index is selected by at most one automaton. We recall that automata Ai2’ are also automata with linear tactics. We shall describe the operation of the program simulating the system S on a digital computer. A medium C i= Ci(ali,u Z i , . . , uQi) (where usi is the expectation of winning with strategy p for the ith medium, i = 1, 2,. . . , N , /? = 1, 2,. . . , e ) appears with probability p i at the instants t = 1, 2,. . . . The game is played by pairs of automats (Ail), Aiz)),k = 1, 2,. . . , M , and the random choice maker. The states of automaton A i l ) , k = 1, 2,. . . , M , are denoted by q ~ j f i , where /l= 1 , 2 , . . . , e is the number of the strategy chosen by the automaton, and j = I , 2,. . . ,n is the depth of the state. The states of automaton Aiz), k = 1, 2,. . . , M , are denoted by yYi where i = 1, 2,. . . , N is the number4 of the medium (index) to which the automaton is tuned, and y = 1, 2,. . . , m, is also the depth of the state. Suppose that a medium C iappears at time t . Then an attempt is made to find automaton Ai2),which is tuned to the index i. If such an automaton exists, automaton Ail) is turned on; if the state of automaton Ail) at t was q ~ f it, will choose strategy p. Here automaton Ail) wins with probability qoi = $(I ap9. In the case of winning [uk(r)= 11, the automaton Ail) transfers from state qjfl into state q ~ if~ j +# ~n, and remains in state q Q if j = n. In the case of losing [uk(t)= - 11, automaton Ail’ transfers from state pjfi into

+

+

Or i

= 0, 1,

2,. . ., N (Editor’s note).

+

88

Automaton Theory

state pjPl if j f 1, and transfers with probability l / e into one of states pb’, B’ = 1, 2,. . . , e, if j = 1. The quantities u k ( t ) are summed over the time (r - 1)t I , . . . , r t in a counter called the common fund. If at time t none of automata Aiz) has the index i, but there is an automaton Aiz) with zero index, the computation is made as if this automaton had index i. However, it may happen that there is no automaton with index i or 0. In this case the random choice maker is turned on: one of the strategies /? = 1, 2,. . . , p, is chosen with probability l/p, and then the expectation of winning is equal to a:. The payoff W for each automaton ALz) of the system S (the total accumulated payoff to the system during t cycles divided by the number of cycles z) iscomputed during z cycles, and the contents of the common fund is cleared. This is followed by automata Aiz), k = 1, 2,. . . , M , playing for payoffs. Any automaton Aiz) wins with probability q = $(I W ) and loses with probability p = 1 - q.5 In the case of a win, automaton AL? transfers from state yvi into state yv+lif y f m,and remains in state ymi if y = m. In the case of a loss, automaton Ai’) transfers into state y:-l if y # 1. If, however, y = I , then the automaton transfers into state yl’ with equal probability, where I assumes any values except for those occupied already. This type of procedure excludes the case where the same index would be chosen by two or more automata. In addition to accumulating the total payoff to the system S during z cycles, the average payoff to each pair of automata and the random decision maker over the entire computation time T is also calculated. These “personal” payoffs have no effect on subsequent computations, but their sum (payoff to the system S ) is used to judge the expediency of the behavior of automata. If the automata ( A i l ) , ALz)) belong to an asymptotically optimal sequence, one can assume that in the limit (with their memories nil), ni2)-+ co) they choose the indices of strategies in such a way as to obtain the maximum payoff W,;T,,. Let us consider several special cases.

+

+

1. The system S consists of a single pair of automata (A‘”, A ( 2 ) ) and , the zero index is absent, i.e., i = 1 , 2 , . . . , N . Then

w&

=

max

i-1 , Z , .

{pi[max(aIi, a,i,. . . ,

..,N

a:)]

N r#i

The payoffs for automata Ap’ are played for independently (Editor’s note).

89


2. The system S consists of a single pair of automata (A"', A ( 2 ) ) ,but A ( 2 ) may have indices i = 0, 1,. . . , N. In this case W,S,, = max Wi for i = 0, 1, 2,. . . , N , where

w,= wi = pi[max(ali, aZi,.. . ,a,i)I + r#iC N

+ a< + - - - + a;)/el~r,

i = 1, 2,..., N .

3. The system S consists of M pairs of automata (Ail), Ai2)), k = 1, 2,. . . , M. Automata A f ) may only have indices different from zero. In this case, we have the formula

where M

W(il, iz,.. . , iM) = C p i [max(a&. . . , a%)] 6=1

4. The system S consists of M pairs of the automata (Ail), Aiz)),k = 1, 2,. . . , M. The automata Ai2) may have indices i = 0, 1, 2,. . . , N. Then the payoff W of the system for this or other tuning to the indices is determined by the formulas of case 3 if there is no automaton AL? with zero index, and if one of the automata Ai2) is tuned to zero index, then

W(ia= 0, i,, iz,.. . , iM) =

M

C 8-l;6+d

pi6[max(a& aid,. . . , u4)]

In the final analysis: W&,=

i&.

max

...,i M

[W(ia= 0, il, i2,.. . , i M ) , W(il, i z , . . . , iM)].

In the following we state the results of a computer simulation for several examples of the numerical method distribution problem.

90

Automaton Theory

Example 1 C' = (0.9; 0), C2 = (-0.9; 0.9), the media appear with probability 0.5. The game is played by one pair of automata (A"), A',)), with memory n") = d 2 )= 5. The tuning is done every 20 cycles. The payoff W&, , as calculated from the formulas of case 2, is as follows: W, = 0.45, W, = 0.45, W , = 0.675. The results of simulation for the cases i = 1, 2 and for i = 0, 1, 2 are listed in Tables XI1 and XIII. TABLE XII

No. of cycles

'

2000

3000

0.67

0.69

0.66

200

540

540

540

0

460

1460

200

1000

0.44

Total no. of cycles for index 1 Total no. of cycles for index 2

I

Payoff W s

Payoff W s

I

200

I

3000

4000

7

3460

5000

-

0.45

0.66

0.67

0.67

0.69

Total no. of cycles for index 0

0

490

1200

1200

1200

1200


200

510

560

5 60

560

560

0

0

240

1240

2240

3240

-

0.67 540

0.44


4000

- --

No. of cycles

2460

1

4240

These show that automaton A ( , ) in both versions selects index 2 in a sufficient number of cycles. The average payoff Ws to the system in this case is 0.67. Example 2 The media C(l)= (0.6; 0.9), C2 = (0.45; -0.45) appear with probability 0.5 at the input. The game is played by a pair of automata (A"), A',)). The memory of automaton A") is equal to 5 ; the memory of A',) is 7. The index tuning is checked every 20 cycles (t = 20). The index is any integer from 0 to 2.

91


The formulas of case 2 yield the following theoretical payoffs: W, = 0.525, W , = 0.45, W, = 0.6. The results of simulation are as follows: Automaton A',) almost always selects index 2, and the payoff for the system Ws is on the average equal to 0.6. Example 3

One of the five media: C'

=

(0.2; -0.6), C2 = (-0.9; 0.9),

C3= (-0.5; 0.5), C4 = (-0.3; 0.3), C5 = (-0.3; 0.3) appears with equal

probability 0.2. It is not hard to see that for the first medium the strategy p = 1, and for t!x remaining ones the strategy j3 = 2 are advantageous. System S consists of two pairs of automata (Ail), A;z)),k = 1, 2, and of the random choice maker. In this case it is easy to compute the average payoff to the system for any given tuning (see Table XIV). In this table the pair of numbers i, j denotes a tuning of one decision maker to i, and of the other to j . TABLE XIV Tuning

0.1 1 0 . 2 1 0 . 3 1 0 . 4

Payoff

1 1 1 I I I Tuning

1.5

Payoff

0.1

I II II I

2.3

2.4

2.5

0.241 0 . 2 1 0 . 2

3.4

3.5

4.5

0 . 1 2 1 0.121 0.08

Table XIV shows that setting automaton A f ! )to 0 and the other automaton to 1 is most advantageous; the payoff of the system is thus 0.44. The payoff for the system with the tuning to all pairs of indices is on the average 0.18. Therefore, if the payoff for the system is 0.42, it means that the combination (0, 1) occurs approximately 12 times more often than all the remaining ones. The memory of either of automata Ail), k = 1, 2, is 5. Versions were run with various memory capacities niz' of the automata AL"!),k = 1,2, and with various intervals t.Table XV gives the average payoffs and the distribution of the automata Aiz) over the media up to the time of 10,000 intervals t,in terms of the percentages of the total number of intervals t. Table XV shows that for niz)= 5, we obtain a blurred picture of the distribution of the automata over the media. For niz) = 10 the combination

92

Automaton Theory

TABLE XV

1

Distribution of automata over indices Aka)

0

34

1

2

3

4

5

1 0

18 13 12 17 6

0 8 5

0

4

1

2

17 31 24

6 5

93

0

3

4 5 1

10 10 8

0

0

0 7

(0, 1) stands out clearly. How fast this combination begins to assert itself can be seen from Table XVI (niz)= 10, t = 40). Thus, practically speaking, after about 2000 cycles the automata select only the index combination (0, 1). TABLE XVI Number of intervals t

1000 2000 4000 8000 12 000

A:”

A:%)

WS

0

0.14 0.30 0.42 0.42 0.42

0 0 0 0 0

1

2

16 40 24 20 62 10 81 5 87.4 3.3

3

4

0 44 0 32 0 16 0 8 0 5.2

5

0 24 12 6 4.1

0 1 2 3 4 28 0 24 0 8 2 0 9 1 0 9 4 0

0 0 0 0 0

0 0 0 0 0

5

0 72 0 36 0 1 8 0 9 0 6

0

Behavior of Automata in Periodic Random Media and the Problem of Synchronization in the Presence of Noise’ 0

In Section 3 of Chapter 1, Part 1, we studied the behavior of automata in random media whose probabilistic characteristics were governed by a simple Markov chain. It was discovered that there exists an optimal memory capacity for an automaton with linear tactics, and in Section 4 of Part 1 it was learned that a stochastic automaton with a variable structure forms a memory which is close to being optimal. In this chapter we attempt to find out how a priori information concerning the law governing variation of the probability characteristics of a medium can be used to design automata that are optimal for a given medium. To this end we will treat complex automata as structures made up of simple automata, and attempt to study the possibility of proper organization of this collective. To construct such complex automata we will use simple automata with optimal behavior in the simplest situations-such as stationary random media. It is natural to begin with some simple law governing variation of the characteristics of the medium. For simplicity we will consider automata with two outputs, 0 and 1 ; this constraint does not limit the generality of the results. At each instant the automaton output and state of the medium determine whether a “penalty” or “payoff” signal will be applied to the automaton input at the next instant.2 It is assumed that if the output signal of an automaton at time t is 0, the probability of a penalty at the next point in time ispo(t), while if the automaton output at time t is I , the penalty probability at the next point in time is p l ( t ) . Since the probabilities lie in An article written jointly with Varshavskiy and Meleshina [45] (Editor’s note). Input signal s = 1 in case of a penalty and s = 0 in case of a payoff (p. 12) (Editor’s note).

93

94

Automaton Theory

the interval (0, l), the situation reduces, when p o ( t ) and p l ( t ) are monotonic functions of time, to the behavior of an automaton t + 00 in a stationary random medium. Thus, we will be interested only in the case in which p i ( t ) , i = 0, I , are not monotonic functions. We will assume that the p i ( t ) are periodic or asymptotically periodic functions. The simplest example of a periodic function p i ( r ) is a periodic piecewise constant function. We will attempt to relate a model of such a medium to some technical problem so as to give it a practical interpretation. It so happens that the problem of recognizing periodic signals in a noisy channel can be reduced to the problem of behavior in periodic random media. An example of such a situation is the problem of synchronizing the switches of instruments in a system for transmitting telemetry with time division of channels. In connection with the foregoing discussion, we have divided the present paper into two sections. In the first section, we consider the behavior of a particular automaton in periodically changing random media, and, for a medium with a known switching period, we show that this automaton is asymptotically optimal. For a medium with an unknown switching period, we present the results of computer simulation, which show that in this case the automaton solves the problem under investigation. In the second section, we describe the problem of synchronizing the commutating and decommutating circuits of a telemetry system with time division of channels in terms of the behavior in periodically switching random media. We shall say that an automaton operates in a periodically switching ( P , , ~p, , , , , p z , o ,pZ,,,.. . ,p 7 7 , 0 , with period T if the random medium following conditions are satisfied: If the automaton output at time t is 0, the penalty probability at each time t I is p t , o , while if the automaton output at time t is I , the penalty probability at time r 1 isp,,, . We assume that p l + T , O= p t , o ,p l f T , , = p l , l , t = I, 2 , . . . . For simplicity we shall henceforth use the phrase “the behavior of an automaton in a periodic random medium.” It is not hard to show that the possible mathematical expectation of a penalty for any automaton operating in a periodic random medium lies between

+

+

We assume that the period T for the medium is known. Then, if we consider an automaton that operates only at times t , t T,t 2T,. . . , it will function in a stationary random medium C(p,,,,P ~ , ~several ); asymptotically optimal sequential automata are known for this problem. This last consideration determines the structure of an automaton with asymp-

+

+

Behavior of Automata in Periodic Random Media

95

totically optimal behavior in a periodic random medium. A diagram of such an automaton is shown in Fig. 14. The automaton consists of T automata (such as automata with linear tactics) that are asymptotically optimal in a stationary random medium, and two cyclic commutators with switching period T. The first commutator cyclically connects the individual automaton outputs to the output of the device; the second cyclically connects the individual automaton inputs to the input of the device. The automata are connected to the first and second commutators in the same order. The commutators are synchronized with the discrete-time rhythm of the medium, and the second commutator lags behind the first by one cycle, so that at time t 1, the input of the automaton whose output is connected to the output of the device at time f is connected to the input of the device. Thus, if the period of the commutator coincides with, or is a multiple of, the period of the medium, each automaton functions in the same stationary random medium. (Here it is assumed that, when the input of an automaton is disconnected from the input of the device, its states do not change.) Then, if the automata forming the device have a sufficient memory capacity, the expectation of a penalty for each automaton is arbitrarily close to min(p,,,, P,,~), i.e., the expectation of a penalty for the device as a whole is arbitrarily close to Mminand, consequently, in a periodically random medium with a known period, the device as a whole possesses asymptotically optimal behavior. (When automata with linear tactics are used, we must also assume that min(p,,,, p t , J 5 4 for all t . ) We will now attempt to solve the problem for the case in which the period is unknown. We require, however, that the upper limit of possible periods

+

Figure 14

96

Automaton Theory

T,,,, be given. Note that for the majority of practical problems, the last constraint is not excessive. In this case, too, the foregoing discussion provides a natural approach to the construction of an automaton that is asymptotically optimal. i t is sufficient to take the design shown in Fig. 14 and make the switching period equal to the least common multiple of the numbers from I to T,,,,,. Then any possible period of the medium will be a divisor of the switching period. This solution, however, is obviously uneconomical. Of course, there are automaton designs in which the switching period is T,,,and provision is made for optimal inspection of all possible switching periods between 1 and T,,,,,. This inspection can be performed, for example, by an automaton with linear tactics and T,,,,, outputs, the penalty probability for each output being determined by the average number of penalties drawn by a device with switching period defined by the output of this automaton. This solution, however, is also unsatisfactory since the system must have a very large settling time, even though the automaton is asymptotically optimal. The unacceptably long settling time of this automaton necessitates other means of determining the switching period. Note that the problem of determining a period in the interval 1 5 T 5 can be reduced to the simplest symmetric automaton game called a Goore game (the Goore game is discussed on p. 1 13). Indeed, consider a collective of T,,,, automata, each having outputs 0 and 1 ; a Goore game is said to be defined if the payoff probability for each of the automata playing the game depends only on the number of the automata choosing the output 1. It is shown on p. 235 of this book that the number of automata choosing output 1, with the probability tending to unity as the memory of the automata increases, corresponds to the minimum point of the function determining the probability of a penalty as a function of the number of automata choosing output 1. We shall use this result to construct a group of automata, possessing expedient behavior in a periodic random medium with an unknown period. In this case, we must also allow for the fact that the penalty is determined not only by proper selection of the period, but also by proper choice of outputs at each point in time by the automata determining the output state of the device. This, of course, is the essential difference between the system of automata described here and a Goore game. The above considerations were used as a basis for constructing the automaton shown in Fig. 15; it differs from the design of Fig. 14 with respect to the presence of additional automata B, one for each position of the commutator. The number of commutator positions is Tmm. Each automaton B is an automaton with a linear tactic and two outputs, 0 and 1. The commutators can stop only at those positions corresponding to au-


97

tomata B with output 1. After each discrete cycle, the commutator moves from a given position to the next position corresponding to an automaton B with output 1. Thus, the operating period of the commutator is equal to the number of automata B with output 1. After each complete rotation of the commutator, automata B are penalized with a probability equal to the average number of penalties drawn by automata A operating during this period. Analytic investigation of the behavior of such a device is extremely difficult; below we present the results of an experimental investigation by means of computer simulation. The experimental results are shown in Table XVII, where n is the memory capacity of automata with a linear tactic (the same for automata A and B ) , Tmaxisthe number of automaton pairs simulated, T is the actual period of the medium, t is the duration of the transient mode in the number of periods of the medium, and M is the average number of penalties per cycle of automaton operation between termination of the transient mode and the end of the experiment. In order to determine the stability of the device, we ran all the experiments for more than 50,000 cycles. The stability was estimated to three significant figures on the basis of the average number of penalties. The values given for M in the table show that, according to this criterion, the device is stable. It is also clear that this criterion does not detect brief random instabilities. In the experiments, we investigated the behavior in a periodic sequence of

98

Automaton Theory

two stationary random media of the form C , ( p , , 1 - p l ) and C2(1 - p 2 ,p 2 ) . The values of p 1 and p 2 , together with the code determining the sequence of the media in a given period, are shown in the table. It is clear from the table that the duration of the transient mode is very short. It is possible that this is connected with the method of choosing the initial internal state of the automata with linear tactics-they were chosen at the output interface. On the basis of our experiments, we can assert that, with a sufficient degree of accuracy, the design shown in Fig. 15 behaves optimally in periodic random media with period no greater than T,,,. TABLE XWI n

r

may

- 10 10 10

10 10

15 15 15 15 15 15

15

10 10 10 10 10 10

10 10 20 24 24 24 24 10

10 20 24 24 20 20 30 30 30 30

7

Code

M

5 7 6 12

11 8

12 5 7 11 12 6 7 7

10 10 10 10

0. I25 0.125 0.125 0.125 0. I25 0.125 0.125 0. I25 0.125 0.125 0. I25 0. I25 0. I25 0.250 0.333 0. I75 0.250 0.283

I70 147 210 203 208 263 175 200 144 228 I92 250 171 262 250 320 375 382

I 1 100 1 1 1 1 000 llO000 110010001 1 1 1 11001 000111 I 1 001 000 110010001 1 1 1 I I 100

I111OOo I I 001 OOO 1 1 1

110010001 1 1 1 110000 1 llO000 1 110000 1 1 1 1 1lOOOO

1111llOOOO I 1 1 1 llO000 1 1 1 1 llO000

We will now attempt to give the problem of the behavior of automata in periodic random media a practical interpretation by discussing the problem of synchronizing switching circuits in a system for transmitting telemetry with time division of channels over a noisy communications channel. The switching device in the transmitter sequentially interrogates the telemetric pickups in accordance with some program and sequentially transmits data on their states over the communications channel. The switching circuit at the receiving end must extract and identify the signals from the various pickups. Thus, the switching circuit in the receiver must, with the same phase, synchronously execute the same sequence of operations as the switching circuit in the transmitter.


99

The problem of synchronizing switching circuits is of particular value when the received data must be used for on-line control of the object carrying the telemetric pickups. Synchronization of switching circuits includes the problem of synchronizing logic circuits in a receiver with the corresponding logic circuits in a transmitter for proper decoding of received information. We will assume that prior to operation (and possibly during transmission of information), the transmitter, in order to enter the synchronous code, transmits an alignment signal (sync signal) composed of an n-digit binary code K . The time during which one data pickup is connected with the communications channel corresponds to the length of one binary digit in the synchronizing code, and the time at which the pickups are switched corresponds to the time at which the binary digit is changed. We shall assume that the transmitter and receiver operate in some discrete time scale and have identical standards for determining the lengths of the binary digits. Obtaining such a standard is naturally associated with carrier-frequency synchronization. Two more problems can now be stated: (1) synchronization of the times at which binary digits change, i.e., synchronization of the times at which channels change, and (2) synchronous and in-phase restoration of sync signals at the receiver. In order to solve the first problem, we take, for the unit time, one or more periods of the carrier frequency, depending on the accuracy required in determining the point at which the binary digits change. In the second problem, it is natural to take as the unit time the length of a binary transmission, which is either known, for example, in terms of the carrier-frequency period, or is determined by finding the times at which the digits in the binary sync signal change. Since the comrnunications channel is noisy, the initial signal is received with some error. We will assume that at each point in time the errors are independent and the probability of incorrect reception is p , where p < 0.5. In order to eliminate a number of additional difficulties, we will consider synchronizing codes 2k digits long. We start by considering the problem of synchronizing the times at which the digits in the code change. Assume that the probability of incorrect reception of a sync signal in a time interval equal to the period used to determine the switching point is p l . We require that the sum of the ones in the even positions in the code K be less than 0.5k,and we require that the sum of the ones in the odd positions in the code K be larger than 0.5k. Assume, for example, that we are given the synchronizing code 1110101010 ( k = 5). The sum of the ones in the even positions of the code is A,,,, = 1, while the sum of the ones in the odd

100

Automaton Theory

positions is Aodd = 5. Then, at the receiver, the sync signal can be treated as a periodic random sequence with period equal to the length of two digits, so that during the transmission of even digits, the probability of the appearailce of a one in this sequence is Aeven/k(1 - 2p1) p1 < 0.53[in our example, this probability is $(I 3p1)],while during the transmission of odd digits it is Aodd/k(l - 2p1) p1 > 0.5 (in our example, 1 - pl). It is clear that in an investigation of the behavior of the automata just described, the correlation that exists here does not distort results obtained, on the assumption that the sequence is of the Bernoulli type. There are two possibilities: (a) the lengths of the digits are known, and (b) the lengths of the digits are not known. Depending on the circumstances, we choose a device of the type shown in Fig. 14 or of the type shown in Fig. 15. We make the switching period equal to the known (or maximum) length of two digits, compare the output of the device with the input signal, and determine the penalty or payoff, depending on whether or not the signals coincide. It is not difficult to see that in this case the behavior of the devices is equivalent to their behavior in the periodic random medium and, as the foregoing discussion shows, the output signal of the device will be one while odd digits are being transmitted, and zero while even digits are being transmitted, correct to the time interval of the communicators. The solution of the problem of sync code repetition is now obvious. Let p z be the probability of incorrect determination of a signal while a digit is being transmitted. Then, in the digits of code K with value 1, ones will appear with probability p z . As in the preceding case, depending o n the information we have about 2k, we choose either a device of the type shown in Fig. 14, with switching period 2k, or a device of the type shown in Fig. 15, with switching period (2k)n,ax.In order to determine the penalty and payoff, we compare the output of the device with the received signal, using the time standard of the device to determine the point at which the code digits change. It is clear from this that here the device must enter the signal (sync code) repetition mode. Technological realization of this device presents no difficulty, especially when economical (from the viewpoint of technical realization) functional analogs of automata with linear tactics are used. Of course, this does raise its own difficulties. We did not set out to design a sync system. However, we have demon-

+ +

+

s The Russian text uses the subscript “odd” here, but it should correctly be “even” (Translator’s note).


101

strated the possibility of solving synchronization problems and have shown that a particular interpretation of problems concerning the behavior of automata in random media can provide new principles for construction of a number of technical devices; nor have we attempted to compare our method of synchronization with known methods such as special servo systems. It should, however, be noted that our method is extremely reliable. Random failures in the automata have practically no effect on the results of the collective device, and when a device with a variable period is used, the system is also stable with respect to failure of individual automata. This last assertion is an obvious consequence of the operating principle of the device.

0

Organization of the Queuing Discipline in Queuing Systems Using Models of the Collective Behavior of Automata' 0

An analysis of queuing systems with infinite queues shows that the quality of the system operation, determined, for example, by the average length of a queue, can be improved by the introduction of priorities [131]. In this case the highest priority number is given to a subscriber with the shortest average service time. However, in order to organize a system of priorities one needs, first, to know the probabilistic characteristics of the input flows, and, second, a special system of queue control. Often in a number of practically important cases the probabilistic characteristics of inputs are not known beforehand or vary with time. In this report we shall describe a queuing system in which the priorities are worked out directly in the channels without any a priori knowledge of input characteristics. As a result, each channel, in the course of its operation, selects a subscriber with the smallest average service time, and in case the service times are close to one another-a subscriber with the densest flow. The selection of the corresponding subscriber by the channel is done in the following way. At the instant a subscriber is connected to a channel, a control sum N is stored in a special register. The value of N may be chosen to be, for example, equal to the average length of a call, averaged over all subscribers.2 During the call the subscriber is charged per unit time, and the charge is subtracted from the control sum. If the real length of the call, z, is shorter than N , then in the channel register there remains an unspent This chapter was written together with Varshavskiy and Meleshina [46] (Editor's note). It is assumed that the approximate value of N is known; otherwise N is selected when the system is put in operation. 102

103

Organization of the Queuing Discipline in Queuing Systems

sum which entitles this subscriber to use the channel next time without waiting in line. If z > N , the subscriber does not have that privilege. The role of registers is played by automata with linear tactics (see p. 16). The index of the automaton action i (i = 1 , . . . , M ) is determined by the number of the subscriber who at a given instant is selected by the channel. The graphs of transitions within the set of automaton states, corresponding to a single action, are shown in Fig. 16, where the input S, corresponds to the arrival of a claim on the channel with the automaton turned on. The input S, corresponds to the arrival of a claim when the automaton is turned off, and the input S, corresponds to the state in which, at the instant the claim is made, the channel is busy and the automaton is turned on. A change of action may take place only in state number 1.

s =s, s = s, s=s,

8 8

... ... Figure I6

8 8

Thus, when the claim arrives, states with the numbers k ( k = 1,. . . , n - N ) transfer t o the state bearing the number k N , and the states with 1,. . . , n transfer to the state with the number n. numbers n - N Furthermore, when the channel is busy, the number of the state of the automaton is lowered by one during each time cycle. State 1 is here transfered into itself. Thus, the higher the state number of the automaton after a call, the more favored the subscriber is at a given time. In order for the channels to secure the most favored subscribers, a provision is made for competition among the subscribers. For this reason two automata are used to serve each channel-the basic automaton ( A ) and the reserve automaton (B). Let i ( A j ) and i’(Bj,)be the numbers of actions of the basic automaton of channel K j and of the reserve automaton of channel Kjt, respectively. The operation of the system is organized in the following way. The call initiated by the subscriber Ti, is processed without waiting if at the instant of call initiation there is a free channel K j , j = 1,. . . , k , and ( A j ) = i,. In this case automaton A j is turned on. When the call is over the number i ( A j ) remains equal to i, , and the state number is recalculated in accordance with the graph in Fig. 16. If, at the instant of call initiation, channel K j ,

+

+

104

Automaton Theory

i ( A j )= i,, is busy, then the claim will be processed as soon as this channel is freed. If i ( A j ) # i, for all j = 1,. . . , k , then the claim will be processed in the order of the queue. In this case, if there is a reserve automaton Bj., satisfying the condition i(Bjr)= i,, then this automaton is turned on. If the state number of the reserve automaton turns out to be greater than the state number of the basic automaton, then when the service is finished, the automata interchange their places. Finally, if for the subscriber i, for all j = 1,. . . , k , i ( A j ) # i, and i ( B j )# i,, then the claim is processed by any of the free channels, and the corresponding reserve automaton is turned on. If there are several free channels, then that channel is selected for which the state number of the reserve automaton turns out to be the smallest. In those cases when the reserve automaton Bj, is turned on, a state with the number 1 is established, and i(Bjo)is taken as equal to i,. We note that when the channels secure a certain portion of the subscribers, still another characteristic of telephone exchanges is improved- the number of switchings in the system. We shall consider that a switching occurs in a channel if through this channel two claims arrive consecutively with different numbers. Switchings of this type introduce substantial noise and strongly lower the audibility. If the probabilistic characteristics of various inputs are constant, then their operating conditions approach a situation arising in the distribution game (see p. 64). Therefore, if the memory of these automata is sufficiently large, one may assume that the priorities for the subscribers with calls of short length should be established as if the average lengths of calls were known beforehand. We shall give an elementary example. Consider a queuing system consisting of a single channel and two streams of calls with frequencies of occurrence A, = A, = &, and average lengths of calls t i = 12 sec and t2 = 2 sec. For the “first-come-first-served’’ discipline, the average waiting time for the above parameters will be 50 sec. If a priority is given to the second stream of calls, the average waiting time is lowered to 31 sec. This very example was simulated on a digital computer for the case in which the parameters of streams were not known beforehand. The average waiting time also turned out to be equal to 31 sec. Using computer simulation, we investigated the average waiting time for various ratios of flow densities of “short” and “long” callers, and also the amount of switching. The results of simulation are shown in Table XVIII, where W,, is the average waiting time for the system without priorities, W, - W,L1 is the average waiting time for the system with priorities obtained directly by the channels, Wn2is the average waiting time calculated using

105

Organization of the Queuing Discipline in Queuing Systems

03 0

0

00

2

0

0

2

W

m

8

0

0

c!

?

r-

2

?

9

m

I?

?

m m

z ?

m

19m

m N

W-

N

N

r-

d

d

d

m

d

m

--

W

W

-

m

9

.

W

-

2

106

Automaton Theory

formulas and the a priori knowledge of the characteristics of input^,^ n is the memory of the automata A and B, and Z7,, and ZIn are the average numbers of switchings in the systems which are not controlled or controlled by the automata, respectively. Table XVIlI shows that the construction described makes it possible to assure the quality of service which in practice approaches the quality of service of the systems in whose construction the a priori information about the subscribers was used. In practical problems, we are interested in those cases in which the parameters of the individual subscribers vary with time. We have studied experimentally (Tsetlin [155]) the cases in which this time dependence was random and was given by a Markov chain. In this case it is impossible to assign a constant priority in constructing a system. TABLE XIXU

wo

(50.0 150.0

Wi

(31.2

no n,

(50.0

134.4 137.6

1 I I [

0.98

0.82

[

0.98 0.84

1

0.98

0.86

n = 4; N = 4; number of channels = 4. Input characteristics: A, = ?,. = 1/8, t,= T? = 2 (in state 1); A, = I , = l/8, t,= t, = 12 (in state 2).

In the first group of experiments (their results are listed in Tables XIX and XX) it was assumed that the system had two states, where at each instant of time the state number is preserved with probability 1 - 6 and changed with probability 6. In the second group of experiments, the average service time was varied independently for all subscribers: each of the subscribers had two states, at each instant of time the state number was preserved with probability 1 - 6 and was changed with probability 6. The results of this group of experiments are shown in Tables XXI and XXII. WnZwas roughly calculated, using the formula for the average length of a queue for a single channel system having p priorities (see Saati [131, p. 2861).

TABLE XXa 10

12

14

50.0

50.0

50.0

50.0

38.4

39.5

41.3

42.3

n

1

2

3

4

8

Wo

50.0

50.0

50.0

50.0

Wk

48.4

43.1

39.5

36.2

a N = 4 ; S = 1/64; number of channels = 4. Input characteristics: 1, = 1, = 1/8, t I= t, = 2 (in state 1); I, = I., = 1/8, T, = t, = 12 (in state 2). TABLE XXIa

a n = 4; N = 4; number ofchannels = 4. Input characteristics: 1,=&=13=1,=l/8, tl= t2= t,= T, = 2 (in the first state); t , = t 2 = t , = t , = 1 2 (in the second state). At the initial moment t, = t z = 2 , t 3 = t 4 12. =

TABLE XXIIa

a S=1/64; N = 4; number of channels = 4. Input characteristics: A, = 1, = 1, = = 1/8, t l = t2 = T, = t, = 2 (in the first state); tl= t, = t,= t, = 12 (in the second state). At the initial moment tl = t, = 2, t a= T, = 12.

107

0

Mathematical Modeling of the Simplest Forms of Behavior' 0

This chapter will be concerned with certain problems related to the mathematical models of simple forms of behavior. The problems in which I first became interested deal primarily with certain peculiarities of motor control in man and higher animals. Why has the study of movements lead an entire group of mathematicians to pose a number of problems that, one might think, have a purely abstract character and are not particularly related to motor control? Above all, the reason is that by studying movements that can be objectively measured, one can reveal certain characteristic features that govern the mechanism of control of complex systems. Some of these characteristic features, as we shall see, are capable of being modeled. Others still leave a lot to be desired from the point of view of modeling. However, we understand very well that without gaining an understanding of the more complex features of behavior, we shall never be able to obtain any kind of exact and accurate knowledge of control processes. I shall allow myself first to describe a few simple banalities, even anecdotes, that lead us to such problems. This is the first concept I shall discuss later in more detail. Let us imagine the work of a commission that allocates living quarters. The commission will consist of the representatives of various organs of the establishment, but generally speaking, not all of them. Each member of the commission has a list of those in need of living quarters. In addition they know how much living space is available. Note that each member represents first of all the interests of his own branch, and as it is well known to anybody who, as for This lecture was delivered by M. L. Tsetlin at a meeting of a section of the Physiological Society on 23 February, 1965. A tape recording of the lecture remained for our use. The text was prepared for print by V. V. Ivanov, D. I. Kalinin, I. I. PyatetskiyShapiro, and I. M. Epshteyn (Editor's note). 108

Mathematical Modeling of the Simplest Forms of Behavior

109

instance myself, has had to deal with this problem, one can rarely persuade people to see another point of view. I cannot seem to convince any member of the commission that someone from my section is more in need than someone from his section, simply because each member thinks that I am prejudiced as far as the needs of my people are concerned. Nevertheless, the members of the commission come to a unanimous decision, and the list can be compiled. This is of course a paradox. It is easy to imagine that one might never come to any decision by voting. This also applies to the work of a parliament and similar institutions.2 Another paradox in the area of control is even simpler, but very extensive conclusions can nevertheless be drawn from it. Imagine that you come home to your apartment building and find in the hall a notice saying: “75% of residents are required to come tomorrow to a meeting”. It is not hard to imagine that if such a notice were actually hung, nobody would show up. Nevertheless the problem of calling exactly 75% of citizens to a meeting is not in itself uncommon. Exactly the same kind of problem arises in the area of movement generation. Suppose that we have to lift a weight, and it is necessary that 75% of the motor units participate in this undertaking. How a janitor will solve the problem is quite clear. He has a list of the tenants, so he will select the 75%, and will send a note to those that have been selected. In an organism one could, generally speaking, follow this “listtype” solution: one can imagine that somewhere there is a list of motoneurons from which one selects the motoneurons needed, i.e., those motor units which are told to “work,” where the remaining ones will be told not to “work.” However, the procedure involved in this type of selection would be long, complex, and wasteful of nerve cells. Should not one organize the process of control in such a way that it will not be necessary to make up such a list, and that nevertheless one will be able to bring 75% of people to a meeting or to tell 75% of the motor units to work? A similar problem arises in the following situation. A store sells pork and beef, and housewives, it would seem to me, prefer to buy beef. If it is necessary to increase the consumption of pork, then it is possible, of course, to send out agents to people who would explain the advantages of eating pork over beef, and who would tell people why it is in the interest of society to give preference to pork. One could approach each housewife individually and explain the matter to her or one could gather all housewives together and talk to all of them at the same time. One can also, however, Concerning the reason for the “problem of the housing commission” see p. 253 (Editor’s note).

110

Automaton Theory

do this: change the prices. This method will no longer involve listing. It is problems of this kind that I wish to discuss now. I must apologize for a certain artificiality of the title: “Simplest forms of behavior.” I will not, of course, talk about more complex forms of behavior, say, food-oriented or sexual behavior of animals or man. We shall be primarily interested in the simplest forms of collective behavior. Initially, we wanted to reduce collective behavior to the well-investigated behavior of the individual. Since later we shall consider more or less complex groups, we shall define the behavior of a single man, one member of a group (one machine, one animal ; for brevity, these objects will be called automata without going into further details about their structure) in a very simple way. We shall, however, see later that such a definition makes it possible to construct more complicated models. Thus, in order to be able to speak of behavior we must ensure the possibility of an object’s choosing one out of a set of actions. Suppose we have a set of x different actions; they will be denoted by fi, fi, . . . , f,. We can speak of behavior in the sense that our automaton is able at each instant of time to make a definite choice out of the set of possible actions. We shall assume for simplicity that we are interested in the behavior of the automaton only at discrete instants of time 1, 2, . . . . This does not impose any substantial limitations. Furthermore, we shall assume that our automaton is able to observe or receive some external signals. For simplicity’s sake, we shall suppose that all external signals to the automaton can be subdivided into only two groups: favorable signals (we call them payoffs) and unfavorable signals (we call them penalties). In this sense the forms of behavior we shall talk about are simple. We shall also say that an automaton is capable of expedient behavior if it attempts to win more often than it loses. The next assumption is that our automaton (it will be denoted by some letter, say A ) has a memory of definite capacity (I d o not want to state here its precise definition). This can be imagined to mean that the automaton can remember a certain number of elementary facts. For example, suppose that the automaton can remember one of four facts. This means that the automaton may be in one of four states. Our automaton, I must stress, does not receive anything from the external medium except for signals signifying payoffs or penalties, and therefore its states pass into one another only in the presence of win and lose signal^.^ In order to speak of the expedient behavior of such an automaton, we must define some problem in which this expediency will be revealed. As

*

For precise definitions, see p. 12 (Editor’s note).


111

such a simple problem we may take the problem of behavior in a stationary random medium C ( p , , p z , . . . , p J . This means that if the automaton performs actionf,, then it wins with probabilityp, and loses with probability 1 - p l , If the automaton performs action fi , it wins with probability p z ,etc. If, instead of the automaton, a man were told the value of these constants, he would act in an extremely simple fashion. He would find the largest among these probabilities, say p , , and would perform only the action f,. This type of behavior would guarantee him the highest possible payoff. We, however, do not deal with a person who knows all this beforehand, but with an automaton that first does not have that knowledge and second, understands only whether he won or lost at that particular time. The automaton is not able to receive more complex information, for which it is too simple. We can describe its behavior using constants. The automaton itself does not know this. We shall give a simple example of an automaton. Let us imagine that the automaton can perform only two actions-first and second-and since our automaton is simple, let it have only two states, i.e., it can only remember two different facts. The others all merge together for it. We shall prescribe for it this type of behavior: If the automaton performs an action and wins, it will go on performing the same action; it has no reason to change its action. In the case where it loses, the automaton will not behave this way. If it performed some action and lost, it would no longer perform this action and would change to another one. In passing, we note that it is very simple to make such an automaton: this is how an ordinary electronic trigger operate^.^ No cunning of any kind is involved here: one signal throws the trigger over, another leaves it alone. It turns out that such an automaton already possesses a noticeable expediency of behavior. If we compare the payoff that will be received on the average by such an automaton, and the payoff that will be received by an automaton that selects its actions at random, our automaton will have a serious advantage. Now, of course, it is natural to ask: is it possible to build an automaton that would behave no worse than a person who knows the conditions of the problem beforehand, i.e., one that would behave in an optimal way? It turns out that it is possible to construct such an automaton (Fig. 17). I shall give here the simplest design, although, generally speaking, many such designs are known.6 Suppose that an automaton performs actionf, and always wins; then it See Fig. 1 and all the necessary calculations on p. 15 (Editor’s note). See pp. 16-21 (Editor’s note),

I12

Auiomuton Theory

will pass from state 1 into 2, from 2 into 3, and it will remain in state 3 ; i.e., the automaton moves upward in states, continuing to do the action f i . In case of a loss, our automaton will behave, it seems, in a natural manner also: namely, if it performs actionf,, then it will go on transferring to lower and lower states until it finally changes its action. The rules of transition in those states, where the automaton performs actionf,, do not differ at all from the transition rules for the states, where actionf, was performed.

Win

LOSS

Figure 17

If the number of states is sufficiently large, then the automaton, with a sufficient degree of accuracy, will behave in exactly the same way as a person who knows the conditions of the problem beforehand. Naturally, all kinds of mathematical questions arise here : What construction of the automaton is the best? Which automaton is most economical of all those that possess this optimal behavior, etc.? I shall not dwell on these problems here. Thus far we have been talking about individual behavior. In the case of individual behavior it turns out-and this is important for a number of things-that its behavior is better, the more states the automaton has. Generally speaking, such an improvement of memory is not possible in all specific acts of behavior. Let us imagine the following problem. As an automaton we can take, e.g., a car driver. He travels in a country in which in some cities the traffic is on the left side of the streets, and in others on the right side. Then, if the automaton with a large memory capacity changes cities often, it will probably have to pay penalities for major traffic violations since it will not manage to relearn every time those cities having a different system. On the other hand, if it changes cities rarely and drives for a long time in any one city, then with a large memory capacity it will first, pay fines rarely, and second, have the time to learn more exactly and choose


113

the correct direction of traffic with a higher probability. For such a problem, the presence of an optimal memory capacity is characteristic. Without giving offense, one could give the following analogy taken from real life. The mentality of persons living in the city and the country are, as we know from observation, quite different. People who live in the country tend to think more deeply and slowly, and people that live in the city tend to think more superficially and faster. They live at a different rate, and because their environment is rapidly changing, city people can afford to have a smaller memory capacity, and achieve their desired goals by a sufficiently rapid switching of ideas. A large memory capacity that makes it possible in a stationary setting to achieve optimum results may turn out to be directly harmful. The same situation also occurs in games.s In the following we shall discuss how a group of such automata would behave. Above all, we would like to verify to what extent complex forms of collective behavior can be realized this way. First, I would like to give a simple example. Every time we study the behavior of a group, especially that of a group of people, we are confronted with a very important fact which is that people who interact with one another may agree on certain common actions, or joint tactics. Because of our habit of thinking this way about human groups, a concept has arisen that has not been verified to any known extent. The concept is that the most important feature of collective behavior is the possibility of an agreement. Therefore, it is very interesting to ask whether people who cannot see one another, and cannot talk to one another, can nevertheless reach a point where their collective behavior will be expedient. I am not saying at all that it does not make sense to make agreements. Agreements may strongly accelerate and improve collective behavior, but at the same time it is important to analyze whether or not an agreement is a necessary and required feature of expedient collective behavior. The reason is that such an agreement presupposes a considerable complexity in our automata, much greater than the complexity of those automata which understand only whether they won or lost in a given play. Let us imagine the following game (it is called the Goore game'): We have a referee and many players, but the players do not see one another. The referee, though, can see them. The rules of the game are the following: a buzzer sounds from time to time, and each of the players is supposed to raise either one or two fingers. Notice that the players cannot see each other. They can only signal the judge with one or two fingers. The judge counts See p. 30 (Editor's note). the Goore game see p. 239 (Editor's note).

' For

114

Automaton Theory

what percentage of the players raised one finger. If this percentage is zero, then with the probability shown in Fig. 18, a payoff is paid. This probability is not large, and means that a penalty is more likely that will paid by everybody. If it is 20%, then a large payoff is paid. If everybody raised one finger (loo%), then again a small payoff is paid. How will each player behave in this situation? He hears a buzzer, raisis one or two fingers, and either receives a ruble or gives it away. He does not participate in any other interaction with either the judge or other players. Note that the rules of the game are formulated in such a way that our automata can participate in it in a way that they understand: first, either they pay a ruble or they are paid a ruble, and second, they can choose one of two actions: raise one finger or raise two fingers. Nothing else is required of the players in such a game. Let us say that the senses of smell, sight, and touch, and the abilities to understand speech and to speak are not required. For this reason such a game can be played by automata. We can then prove the following theorem: No matter what the number of players in a Goore game may be, with sufficient memory and for the function shown in Fig. 18, exactly 20% of the players will, with probability I-i.e., with certainty-raise one finger, and the remaining 80% will raise two. Of course, if we dealt with people it would not cost them anything to make the following agreement. They could decide amongst themselves that first, one person shows one finger, then two persons one finger, then three, then four, etc. If the players remember when the highest payoff was received, then the judge will be ruined financially, because now exactly 20% will raise one finger every time. For this to happen, it must be necessary for the players to make an agreement, and this is no simple matter. It must be noted that our automata do not have any kind of built-in altruism, and to agree on something for such automata, just as for people, incidentally speaking, is a very dangerous matter. It is dangerous for the following reason: there is no basis for thinking that the person you make an agreement with is not your enemy. Moreover, in game theory one

Figure 18


115

emphasizes and finds most interesting those situations in which it is impossible for the players to communicate. For example, suppose I play the game of 21 with a friend. On what can I agree with him? Every ruble won by me is won at his expense. The only agreement we can enter here is “let us stop playing.” This type of conflict situation is very common. Here, however, it turns out that without any agreement, i.e., without risking anything and without making any commitments, such a group of simple automata achieves the percentage that results in the maximum payoff. The recruitment of the motor units, say a working muscle, can probably be reduced to this type of problem. In this case the individual automaton would be replaced by a motoneuron or a motoneuron with a group of interneurons. The problem is to obtain the required number of working motor units. Here it is not important what the payoff consists of. Imagine that on the axis of abscissas in Fig. 18 we measure off, not the fractions from 0 to 1 but, for example, the pull of 4 kg, or 500 gm. Then the problem becomes exactly the same as the previous one and the motoneurons “agree” to pull with a given force. The only thing that is changed here is the scale. This problem is also directly related to that of gathering the required number of people for a meeting or to the problem of lifting a weight. Consider still another problem which has exactly the same game character, and which was discussed simultaneously with the Goore game- the distribution problem.* Let us imagine the following game. We take certain numbers, e.g., 0.9,0.33,0.33,0.33,0.33,0.33,0.33.Let us imagine mice or some other animals of the kind can receive food from various troughs. The foregoing numbers will represent the amount of food in each trough. Thus, there is one well-supplied trough with 0.9 amount of food and six poorly supplied troughs with 0.33 amount of food. Suppose we have five animals that cannot communicate [they probably can, but we cannot assume that; for our purposes, they manage without]. The game will proceed as follows. A metronome is struck, and using this signal, each of the animals chooses one digit: 1, 2, 3, 4, 5, 6, 7, i.e., creeps to a certain trough. If the animal comes to a given trough by itself, it will eat all the food in the trough. If two animals come to the same trough, they divide the food in half. In the case of three animals, they divide the food in three parts, etc. Let us imagine what we would do in such a Situation. We would imagine that we should distribute ourselves in the following way: two of us would come to the best supplied trough, and each of the remaining ones would come separately to the remaining troughs. Then either of the two at the well-supplied trough See p. 64 (Editor’s note).

116

Automaton Theory

receives 0.45, and the others get 0.33 each. It would seem that this is unjust, but what else is there to do? If either one of those at the rich trough goes to the already occupied poor trough, he will receive 0.16 instead of 0.45. If one of those at the poor troughs goes to the rich trough, he will obtain 0.3 instead of 0.33, thus less again. Consequently, no one has an opportunity to improve his lot. In order to assume our solution, we would have had to inspect these numbers, i.e., we would have had to know them beforehand. We shall not assume that the automata know anything beforehand; we cannot say anything about them since they are not able to perceive numbers. Every time the metronome is struck, the automata utter one of the digits 1,2, . . ., 7 or imitate a movement to some trough, and learn whether they won a ruble or lost it, i.e., the automata are in exactly the same conditions under which they operated when playing the Goore game. Moreover, the automata do not know what game they are playing. It could be any kind of game; perhaps they are playing the Goore game. It turns out that automata with sufficient memory capacity, not necessarily very extensive, say 4-5, distribute themselves in a way that is just as reasonable as for people who would know the contents of each trough. What is also interesting is that they do it more rationally than people would in the same situation. A person would probably behave in the following way and say: “if I came here, then even though 1 get 0.45 and the other players only 0.33, I will not allow myself to be dragged out of here, no matter what.” In automata, by virtue of their probabilistic structure, it turns out that on the average they all get the same amount, i.e., each automaton finds itself at the poor trough for part of the time, and for a greater part of the time at the rich trough. In turn they pass through all the states and are under the same conditions. Of course, in this case it is assumed that the automata have an exactly identical structure. Now let us calculate their payoffs in this case. The total payoff is 0.9

+ 3 x 0.33 = 1.89.

The payoff to each one of them is 1.8915 = 0.378. Thus it turns out that, if the game is played by automata, they will each receive exactly the same payoff. Here, the following fact is of interest: We have assumed that the participants did not promise one another anything or make any commitments. At the same time they behave in the optimal way. They distribute themselves in such a way that it will not be advanta-


117

geous for anybody to change his actions. A stable configuration is formed which will be referred to as the Nash point. But this is not the optimal pattern of behavior. In our case, each gets a payoff of 0.378. Now assume that the participants are people who are unflinchingly convinced that their opponents are honest. Then their configuration will be different. They will say: we shall distribute ourselves in such a way that there will be one person to each trough. This may turn out to be very unjust. This is true: one will get 0.9 and the rest 0.33 each. One of them might say: “What good does that do? I had better come to the rich trough, where I will get 0.45 and not just 0.33.” However, they agreed earlier to play with honesty; thus both those who receive a lot and those who receive little put their money into a common fund and then divide it e q ~ a l l yLet . ~ us calculate what the payoff per person will be in this case: 0.33 x 4 0.9 = 2.22; 2.22 x 5 = 0.444. As we have seen, this type of agreement led to a situation where they received higher payoffs. We point out once again that this kind of agreement is not without its risks. They must count on the one another’s honesty. Otherwise, if a person at the rich trough stops giving money into the common fund, they will not even receive 0.378 each (the payoff guaranteed in the absence of a common fund), but only 0.33 each. What are the answers given by simulating the behavior of finite automata on a digital computer or by analytic calculations? If the memory were large, the automata would receive only 0.44, but in this case they would be required to make the following agreement: that the conditions of the problem be changed so that all their payoffs are deposited in a common fund, and then divided equally. However, this kind of joint action, accomplished by means of a common fund in physiology, for example, is something that is fully justified. For this reason it is natural to presuppose kind of homogeneity of the payoff for physiological applications. What is now the situation with the automata? If their memory is large, they receive 0.44; if their memory is small, they receive less. Depending on the memory, the payoffs can be plotted on a graph [see Fig. 11 on p. 69 (Editor’s note)]. For sufficiently large memory without a common fund, the automata receive 0.378. With small memory capacities, the use of a common fund lowers the total payoff, and for large memory capacities, it increases the payoff. For some intermediate value of the memory, the common fund yields neither a gain nor a loss. A banal sociological analogy occurs at this point: How is it that the common fund can be disadvantageous? The reason is this: Imagine that two automata found themselves at a poor trough. What

+

See the definition of a game with a common fund on p. 61 (Editor’s note).

118

Automaton Theory

they did was obviously silly, and if it were not for the fact that they received payoffs from the common fund, their personal payoff would drop from 0.33 to 0.16 which would be quite noticeable and would make them change their actions. If, however, they receive payoffs through the common fund, the consequences of such unreasonable acts become for them much less acute. They receive a compensation at the expense of their more successful partners. This is exactly what is commonly called the damage of equalization. If the memory is insufficient, if relatively small payoff decreases with such a memory are not noticed, then the common fund is simply damaging. It obliterates the worth of individual acts. If, however, the memory capacity is sufficiently large, making it possible to notice even very small changes in the total payoff, then the common fund brings a considerable improvement. This, incidentally, shows why one should not organize overly large collectives, functioning with a common fund, and receiving identical payoffs-let us say, the same amount of food. It is a well-known fact that production quotas are set up within a single brigade because within one brigade there is enough memory capacity, or awareness if you will, to understand that low productivity lowers not only the wages of others, but also those of the person involved. Within a factory, production quotas are never set up, since at that level one needs a higher degree of awareness to feel any possible losses. Thus as a factory worker, if I smoke a cigarette for half an hour, I would not lose a single kopeck from my wages. On the factory level this is not noticeable, but if I were fulfilling individual production quotas, this act could cost me a fortune. In connection with this problem, I would like to mention still another characteristic of control systems, in particular, systems related to motor control. I have in mind the idea of compensation. There are various forms of reliability, which can have different definitions in technology and in physiology. Specifically, the following undesirable property of all our technical products is their nonuniform reliability. I refer to unreasonable actions such as the fact that a shirt is thrown away when its collar is worn out, even though its remaining portions are in a good condition. Machines, even large and heavy, are replaced even when the wear and tear is insignificant -literally perhaps only a couple of grams of metal are missing. If, however, the foundation is beyond repair, this cannot be replaced. Wise people who were making shirts 30 or 40 years ago sold extra collars with the shirts. The collars were attached to the shirts with clips, and thus they could be replaced. Incidentally, as far as technical parts are concerned, the moving parts are made replaceable as much as possible. I t would, of course, be much more convenient if shirts wore out uniformly. This is, however, not the case.


119

If shirts had extra material and enough of it, then a single shirt could probably be worn ten times longer than it is worn today. This, of course, applies to shoes to an equal extent: shoes are thrown away when they are generally still quite new. It would be more convenient if such a compensation were accepted as a matter of course. Using the example of the distribution game, it will be easy to see how this is done.1° I would like to use these numbers to explain why such a system of playing automata behaves in exactly the same way as a shirt which is being shortened to make a new collar out of the extra material rather than being thrown away. In every case, the automata transfer to the procedure involving a common fund (Fig. 19). Now let us imagine that these automata are mortal; each automaton works for some time, then dies and is excluded from the game. Irrespective of the number of automata, even with just one remaining, the following configuration will develop each time due to their optimaljty: the first n places will be occupied, where n is the number of automata, and the remaining places will remain free.

Number o f phys executed

Figure 19

The same type of reliability picture occurs also in the Goore game which was described before this example. In fact, there the payoff will depend only on the fraction, and the failures of some of participants obviously lower the payoff, but do not disrupt the automata from their optimal mode. This reliability of behavior seems to be typical of automaton groups. I am attempting to formulate here certain general assertions. The only thing I have in mind is the fact that, since the automata have equal rights and their properties are universal to the extent that they can play this game the Goore game, and dozens of other games that I could quote here, these are then identical automata and their properties are indistinguishable. Therefore, the death of any of these automata, a failure to operate, is immediately comlo

See p. 69 (Editor’s note).

120

Automaton Theory

pensated for, and since they are capable of optimal behavior, the optimal distribution of the automata over the actions available is maintained to the last automaton; as the saying goes, they “fight to the last soldier.” Generally speaking, game theory was proposed by the brilliant American mathematician J. von Neumann to describe the behavior of people, and not of automata. It was found that automata, even the simplest, behave no worse than humans in a number of cases. It will be noted that the homogeneous games described above were not analyzed in von Neumann’s game theory. Homogeneous games seemed interesting to me, especially in application to automata. Suppose we have a universal computer. Then we are required to set up a timetable listing tasks to be done by various shifts. We must see to it that electric power is available, that the cleaning woman will wash the floor, that the spare parts will be supplied on time, and that maintenance will be done systematically, etc. As the computer manager, our job is to take care of the computer, and not to solve problems. Furthermore, we sell the computer time for money, of course. A mathematician comes and solves his problem. Another mathematician comes and solves his problem; these problems are completely different. A third mathematician comes and solves his problem, and so on. A universal computer is exceptional in the sense that it can be used to solve a variety of problems. When speaking about physiology, naturally I will often be naive, but I hardly feel the need to apologize for that because so many physiologists still perpetuate these naive notions themselves. What is the weight of a human brain? The answer is about 2 kg, so we shall assume approximately that there is some 2 kg of brain matter. [Incidentally, to me it is not important whether it is 2 or 1 kg.] The question is: There is a certain amount of brain substance; how can it be distributed? The most absurd thing one could do would be to proceed something like this: Here is the right hand, 500 gm of brain substance will be allotted to it (the right hand being very important); here is the left hand: give it 300 gm. For each leg, 150 grams will be enough. The tongue will take something too. How much shall we give to the eyes, the ears, food preparation, emotions, social feelings, remembering that 15 gm must still be left for playing chess? We finally come to a point where there is no more brain substance to give away. I t would seem that one might act like this, but it is hardly feasible that this is what actually has been done. The reason could be that when playing chess, I am not at that moment thinking about my professional work or jumping a


121

rope [which I can only d o with some difficulty]. This means, it would seem, that when I am playing chess it would be more advantageous to devote to it not 15 gm but, let us say, 1 & kg out of the 2 kg of brain substance available. Tomorrow it might happen that I will become involved in a fight, and during that fight when I need to move as fast as possible and make fast motor decisions, it would be natural to take the gray matter away from the eyes, ears, chess-playing, cooking, and emotions and give it completely to the fight. Instinctively we feel that this is indeed the case: when completely involved in a fight, you forget about everything else. One can ask, couldn’t the control centers be distributed according to the tasks? For all the naiveti of such a viewpoint, and at the risk of leaving myselfopen to ridicule, I do think that this is actually the way the brain works. There are certain computing means which I conventionally measure in grams [I think that it is more natural to use grams], and those means can be redistributed. In order for brain substance to be able to be redistributed, it must have certain universal characteristics so that when it is brought to bear upon one problem after another it will still be able to deal with this new problem. Our automata, by virtue of the universality of their properties as postulated and developed, but not based on physiology, do possess that characteristic. I can make the same automaton play the Goore game, the distribution game, a zero-sum game according to von Neumann, and virtually anything else I please. The universality of the automaton is sufficient for that purpose. Having this kind of automata available, it would obviously be interesting to see if it would be possible to construct a game in which there would be some choice of problems, so that the automata would choose the most important problems and solve those. And if the priority of problems were to change, than they would drop the problem that became unimportant, and would begin to solve the problem that would be of highest priority at any given moment. Together with S . L. Ginzburg, in 1964 we made certain numerical experiments which were run on a computer.ll Such games are easy to invent, easy to interpret, and what is most important, consume very little machine time. The problem is very simple for automata. However, I am far from being convinced about so absolute a universality for brain matter. I think that a very large fraction of these 2 kg is irreversibly specialized to perform various functions, for example, vegetative functions. But in any case such a universal functioning of the basic brain substance must probably take place, since otherwise it would be impossible to understand l1

See p. 84 (Editor’s note).

122

Automaton Theory

the occurrence of compensation. One can refer to the well-known experiments by Laszlo, which involved removal of the frontal brain lobes, or to other similar experiments, but it seems to me that this is unnecessary. Numerous examples of functional compensation indicate that brain matter does indeed possess such a universal capacity. Finally, as mentioned at the beginning of the chapter, I would like briefly to clarify the difficulty concerning the distribution of living space and what relation it has to automata. Suppose we have a number of apartments, the number being rather small, and certainly much smaller than the number of those in need. [If the number of the needy were not very great, then there would be no problem, and there would be no need to set up a commission, since the commission would not have anything to do.] All apartments will be assumed identical and to consist of two rooms each. [Should the apartments be different, there would simply be several different problems: how to distribute two-room apartments, how to distribute single-room apartments, etc.] There are N persons needing apartments, and there are m members of the commission, where m is not very large. Now let us imagine how the commission actually operates. Each member takes a list of the applicants, and tries t o ascertain which one of them is most in need, who is second most in need, etc., i.e., each member puts the applicants in a certain order. Thus, for example, the first member will write something like this (persons will be denoted by letters for convenience) : all a 2 9 a3,

I

014,

-

9

OLN

3

and at some place he will make a mark, say after a s , meaning that there are no more apartments. Note that while making up the list, he will very carefully choose those that fall on the left of the mark. The second commission member will do the same thing, and so will the third, etc. These opinions are then announced to everybody else. For example, one can suppose that they will write them out on the blackboard. One cannot decide anything by voting, the reason being that the number of apartments is smaller than the number of applicants, and the lists will usually not be the same. But if I made up a list and one of the people I was representing was not on the final list, then I would not approve such a decision. Therefore, I would only approve my own list. So far, if I were not convinced that all of my people would appear on the final list, I would not approve it. And if I were convinced, then there was no need to select me for the commission in the first place. Let us look at what happens. The blackboard has the opinions of all commission members. If in the majority of cases these opinions are the same, then one can decide the matter by voting. Now, is that likely to happen?


123

No, this is completely unlikely, since there are N ! various opinions, where N is the number of applicants, and the probability of the coincidence of opinions is very small. Therefore, the first thing that the commission members are going to see is that it is impossible to come to a common opinion. Incidentally, in any reasonably constituted commission, decisions are not arrived at by voting: they begin to vote only when there is a conviction that a decision is unanimous. It is clear why this is done thay way. If I were to stick to my own opinion, and the other members could not convince me that the decision was right, then the commission’s work would have been in vain. I would then go to the local committee, and the proceedings would start again. As a rule, any decision of the commission should be approved by the local committee, and if one of the commission members demurs the decision, then the local committee would not be able to approve anything. Instead it would send the commission members back until a unanimous decision was reached. Thus, voting cannot be used to make the decision. Perhaps this is an appropriate time to say when voting can be used to make decisions. If we were to choose a chairman from among three possible candidates, then this is a problem that could be resolved by a vote, since the number of possible views is here much smaller than the number of those who vote. It is only in such cases that decisions can be arrived at by a vote. In our case, the decision cannot be made this way. Thus, the members. of the commission must come to some reasonable compromise by making agreements among themselves without voting. How can they resolve their disagreements? First, no one is denied the right to change his mind. Second, and we always think about this seriously, we can try to persuade each other [although I think this is not very likely]. In fact, any reasonable person can be persuaded somewhat, as far as most questions are concerned. But in considering the housing problem, this is a completely hopeless task. The reason is very simple: each member of the commission represents some department, supports its interests, and will defend its interests no matter what, and this is indeed the case. How can we convince one member except by saying: “Look, you are defending this person while everybody else is against him; you will not be able to succeed, and upon can’t help him.” Here the point is not that the member is shouted down, but that he begins to change his mind. It turns out that this problem may be formulated in terms of automata games, and from many examples we see that automata come to the decision that would also be arrived at by humans. This example was given because I wanted to show that even fairly complex forms of behavior (the behavior of the commission members is a complex form of making collective decisions) can be simulated by means of automata.

124

Automaton Theory

Appendix 1 Addressless (Nonindividualized) ControP

If one assumes that all control proceeds from the top down to a specific address, then the system becomes very complex. For example, motor control involving addresses means that one has to write a schedule of the actions of all muscles, to inform each receptor where it is supposed to direct its pulse. On the other hand, this can be done differently. We shall show this using an example of an automaton game. If the conditions of a game are given, then the automata find the required actions by themselves. In this case they do not need individual commands. We think that the higher levels of the nervous system-e.g., the supraspinal level relative to the spinal level-need only a general type of information about the states of the lower levels. In other words, only very general instructions are needed. Using only these most general instructions, the automata can then select the optimal level by themselves. On the basis of numerical analysis, the military have often made this assertion: in fact, the army is urgently in need of computerization since now the infantry does not walk. They ride on personnel carriers, and can cover 100 or 150 km in the same time as 30 km were traveled before. Now, in order to write a marching order for an infantry division, two days are needed. However, this does not mean at all that computers are needed, because the bureaucracy that expressed this point of view had in mind that the orders should specify exactly what must be done by all subunits, all special units, all staffs, all camps, all magazines, all cooking personnel, medical-sanitary battallions, each person in fact. But the orders are not written that way. A division commander makes only one decision: From point A, move to point B. Then he says to the staff officer: “Now write the orders.” The staff officer writes the orders for the commanders of infantry regiments: “You have orders to reach that point,” and says to the rear staff officer: “Look, we are leaving for town B, so you have to move your troops.” Thus, there is no one person who could give this kind of allencompassing order. In my opinion the nervous system is in the same situation. Here it is important that all subordinate nerve centers be able to understand the language that we use. From the point of view of control, it is precisely here that address-free control has an advantage, i.e., it is not necessary to tell each one what to do. All that one has to do is to hang out a notice saying la The text in the appendix is made up of the answers to the questions posed after the lecture (Editor’s note).

Mathetktical Modeling of the Simplest Fo,-ms of Behavior

125

“to whom it may concern.” [I found with astonishment that, for example, the work of prisoners is more expensive than that of free men, even though the former are much worse fed and clad, and they work no less. The point is not only that the efficiency of prisoners is lower, but that a prisoner must be fed, clad, and watched by someone else. With a free person the matter is different: e.g., “I get paid twice a month, I give the money to my wife, and then my manager knows that I am not hungry, that I have my shoes in order, that I get my meals on time, that I will not leave, etc. He doesn’t have to think when to change my shoes or linen or what to do with my children, and so forth.”] Automata are, strictly speaking, good to use for simulation, because their properties are universal. If a payoff system is given (in terms of wins and losses), then the automata can already act on their own, attempting to be punished as little as possible. Appendix 2 Languages That Automata Use to Communicate with One Another

We have discussed very simple forms of behavior, and for this reason we limited ourselves to the simplest types of automata. The exchange of information among these automata takes place in the language of penalties and rewards. Although this language seems universal enough, it would, however, be interesting to also look at more complicated automata that possess some specialized language to communicate with other automata. Such automata are needed to describe more complex forms of behavior. These more complex behavioral forms necessitate the use of much more diverse information. Thus, for example, a player participating in a sports game, in addition to a general estimate of the situation (whether he acts correctly or not, he probably is not always aware), sees a lot with his eyes and hears with his ears, i.e., he receives many different kinds of information. When we deal with, say, a card game [and card games can be very complex, so complex that it is simply inconceivable to me how people can learn to play them], then we know beforehand exactly what information a card player has at his disposal. He can see his pack, he saw the cards that were put away before a given moment, and one can always say what he knows and what he does not know. In the case of an athlete, however, I think it is impossible to describe exactly what information he is using in his actions.

126

Automaton Theory

For this reason, I think that sports games are some of the most complex forms of behavior that can be described at all. I think that the game played by a soccer player is much more complex than the game of the participants in, as another example, an international conference. Automata are necessary to describe more complex forms of behavior, and this is the reason why: Imagine that I am playing chess with somebody, let us say with V. B. Malkin [it so happens that we did play chess together]. Anybody who has played chess at least once can understand what kind of moves can be expected from a given opponent. The player has some internal representation of how his opponent thinks. In my case, my opponent also thinks of me. This means that when I think about my opponent I should simultaneously include in my model of the opponent the model of myself that the opponent has. But then I have a model of the opponent, and thus one obtains m at r e~ h k a s 'contained ~ inside one another which are impossible to untangle in any reasonable way. This is a paradox which is natural in mathematical logic. Automata are to a large extent free of that paradox. They cannot have any model of the opponent. In the case of more complex forms of behavior, models with richer languages that can be used for communication are needed. It is, however, worthwhile to introduce a special language for communication, since it will be necessary for us to consider in this language the languages used by the opponent, and thus matreshkas come into play again. Nevertheless, they d o not appear because the model of a chess opponent is generally replaced by chess theory, which is applied not specifically to a given opponent, but to any person proficient in chess.

lS Wooden dolls of successively smaller sizes contained in one another (Translator's note).

ARTICLES ON BIOLOGICAL SYSTEMS AND MATHEMATICAL MODELS IN BIOLOGY

0

Introduction’ 0

The research on the problems related to physiology and other “complex” systems that M. L. Tsetlin and I have been pursuing may be divided into three periods : 1. The first period was devoted to an analysis of the properties of continuous media. Our interest in those questions arose in connection with the problems of the heart, and continuous media appeared as a proper method of description. This period saw a development of the axiomatics of continuous media and a paper about Benkebach cycles. At that time we formulated the principle of least interaction, using the example of spontaneously active elements. The principle was later of great heuristic value to us. 2. The second period was characterized by an enthusiasm about the ideology that resulted from the ravine method. We were interested in ways of overcoming the complexity of the problem, using a hypothesis about the organization of the world. The questions and ideas that originated as a result turned out to be very useful later. 3. Finally, after the appearance of M. L. Tsetlin’s papers on automata, an attempt was made to apply the principle of nonindividualized control to various biological problems. In the application to the problems of spinal motor control, the work resulted in specific predictions. The situation was similar so far as the problem of pretuning was concerned.

The present stage is only in its initial state. This is the stage in which postulates (“precepts”) are formulated that are typical of a living system. Unfortunately, this problem was not covered in the papers written. It seemed to us that this gradual change of our viewpoint from stage to This introduction was written by I. Gel’fand. 129

130

Biological Systems and Mathematical Models in Biology

stage should have resulted over a period of years in the construction of some “language” (a set of “postulates”) in which one could speak of living systems. Unfortunately, this stage of our joint work has been interrupted without ever having seriously begun2 It can be stated that the papers that were completed and are published in this collection are in a sense only a prelude to the basic theme-which is to understand the principles underlying living systems, and making them so much different from inanimate systems. Perhaps it is precisely the difficulty of this problem that was the cause of so many different attempts. The only common trait in these attempts was the anxiety not to let this “feeling for life” slide through our fingers. We have nevertheless hoped that each model, which has been inevitably more or less f o r m a l i ~ e dwould ,~ leave out at least a tiny part which would make it possible to come a little closer to the understanding of “life.” We always remembered the words of the great physicist N. Bohr who said that, in his opinion, the leading branch of knowledge will not be physics, but biology. Now a final remark. What should be the degree of formalization in biology in the study of living systems? Considering quantum mechanics, one can distinguish two stages in its formation. The first stage took place when Bohr created the philosophy of quantum mechanics. At that time, the formulas did not yet exist, and even if they did, they were not quite as they should be or were completely wrong. The second stage was a period of a rapid growth, and quantum mechanics became an exact branch of physics with a large number of precise formulas. But this stage was possible only after the first stage had taken place. By comparison, in biology the first stage has not yet occurred.

Rather, it was the undercurrent all that time, and all the preceding papers have dealt with it. Maybe the problem “Ah” is proof that the undercurrent was always there. Unfortunately, often less formalized than we would wish.

0

Mathematical Simulation of the Principles of the Functioning of the Central Nervous System’ 0

This chapter is not a survey o f the methods of mathematical modeling of the mechanisms underlying the central nervous system; it is devoted to some mathematical models related to the physiology of the central nervous system. We shall limit ourselves to an exposition of the basic ideas. Readers who are interested in the details of the mathematical apparatus and the details of physiological applications will be able to find additional information in the bibliography. We present three models. The first is concerned with the modeling of expedient behavior based on the methods of searching the extremum of a nonstationary function of many variables. The second model is devoted to simulating the behavior of collectives consisting of automata capable of expedient behavior. Both of these models were generated by studying behavior which we understand as a single process of studying the environmentmaking and carrying out a decision. The question of the specific physiological mechanisms performing the behavior will not be discussed here. The third model is an attempt to describe mathematically the functioning of the simplest excitable tissues. These mocels are quite diverse, but we would like to see in them examples realizing a certain general principle. The last part of the chapter is an attempt to formulate such a principle (the principle of least interaction). The problems that result in the necessity of studying complex control systems are extremely diverse and are generated by various branches of contemporary science and technology. The peculiarity of these systems forces us to rethink the very word “learnJointly with Gel’fand [62a]. The text is slightly changed: it includes the material from a previous work [62] (Editor’s note). 131

132


ing.” The problem is that a completely isomorphic description, making it possible to take into consideration all the characteristic features of a phenomenon, is inadequate for complex systems, precisely by virtue of their complexity. Numerous examples are known of the inadequacy of descriptions of that type. Thus, using a system of differential equations to describe the motion of gas particles and their initial coordinates and velocities does not add anything substantial to our knowledge of the macroscopic properties of a gas. For complex systems, it is typical to find that the method of description is dependent on the problem that is to be solved by means of that description. It will be noted that for complex systems used in solving a problem, it makes sense to introduce the notion of the quality of the solution, i.e., the degree to which it is adequate. We shall consider this type of situation, using as an example a problem involving the calculation of a minimum of a multivariable function. This problem will naturally involve such notions as: the complexity of a problem, organization, search, tactics, hypothesis, and others. The most striking examples of complex control systems occur when one studies the behavior of a “little animal in the big world.” Here especially one can see the inadvisability and even the practical impossibility of an isomorphic description. This perhaps explains why physiological statements often bear a distinctly model-like character. Among many aspects of behavior, we shall touch upon only one, and this is the problem of the generation of motion. This problem, important in itself in physiology, attracts us by its ‘‘physicality’’-many motion parameters can be measured and described quantitatively. This also implies the necessity of organization and the use of tactics that, in view of the continuity of the process, refer both to the analysis of afferentation and to a generation of the motions themselves. 1 On Search Tactics We shall consider the functioning of a complex control system which is designed to reach a certain definite objective. It is assumed that the system is capable of determining how close it is to reaching its objective. The informatic .i necessary for the successful functioning of the system is received by the system in the activity directed toward the objective.2 The complexity In this sense the systems studied here are systems with dual control according to Fel’dbaum [145, 1461. Generally speaking, the case is more natural when a somewhat

PrincipIes of the Functioning of the Central Nervous System

133

of the system is defined by the number of parameters necessary to specify its state. Problems of this type are encountered when studying control processes governing numerous physiological mechanisms and complex technological systems. Examples of such systems are the movements of animals and of humans. Numerous problems in the area of complex systems would seem to have been quite completely analyzed by means of classical mathematics and functional analysis. However, it often happens that the algorithms proposed there, which permit a formal solution of a problem, turn out to be practically inapplicable. Thus, for example, to obtain an extremum of a multivariable function, classical mathematics proposes the following technique : differentiate the function in turn with respect to each of its arguments and set the derivatives obtained to zero. Thus, the problem reduces to solving a system of equations whose number is equal to the number of arguments. In practical numerical problems it turns out, however, that a solution of such a system is by no means simpler than to obtain an extremum directly. For a larger number of arguments both problems are in a general case extremely involved, and their solution is beyond the capabilities of today’s numerical analysis. Often the direct use of other algorithmic methods is also practically i m p ~ s s i b l e . ~ One can state other problems for which, even if it is possible to construct an algorithm suitable for all cases, its realization turns out to be impossible in view of the limitations inherent in contemporary computer technology and the restrictions on the time during which a problem should be solved. The restrictions on the time spent on solving a problem are particularly important. The difficulty is that practical problems (for example, those ocmore precise knowledge of the objective is also developed when the problem is solved. The problem of developing the objective (and the corresponding estimator or a system of such estimators) is probably even more important. However, this problem is more difficult, and we shall limit ourselves here to only those problems in which the objective and the estimator are known. An example of such a problem is the game of chess; to solve it, it is very important to develop an estimator. a Ulam [194], noting that finding a minimum of a function of more than four or five unknowns on a computer is often practically impossible, suggests that to solve such problems one should construct a cooperative link between the man and the machine. In this case, the machine displays on the screen reliefs of given two-dimensional sections of a function, and the man using these reliefs makes a decision about further action (i.e., what sections should be taken, what area should be examined in magnified scale, etc.). In this very wise proposition, one can clearly see the understanding of the imperfection of exact isomorphic algorithms. However, the proposed division of labor between the man and the machine is based upon existing machines and methods of their use.

I34


curring in physiology) typically involve situations that vary with time, so that a delayed solution may turn out to be outright erroneous. In this sense even a relatively rough approximate solution that is obtained rapidly may be preferable to a more exact, but delayed s ~ l u t i o n In . ~ such situations, the acceptable solution can be achieved only by means of the organization which is to a larger or lesser degree possessed by problems occurring in practical human activity or perhaps in physiology. It is a very complex task to try to give a more complete definition of the notion of organization. In essence, the term “organization” is understood to mean those characteristic features of a problem or a situation which may facilitate obtaining a solution. These characteristic features are not known beforehand exactly, but are only more or less probable. Therefore, we make use of organization by advancing certain hypotheses and constructing tactics based on these hypotheses. Usually i t is impossible to verify these hypotheses directly. They are, one might say, tested in practice: the criterion used to judge the correctness of a hypothesis is provided by the “goodness” of a solution. Let us consider a simple example. The solution, using difference methods, of a fairly complex system of partial differential equations (for instance, of those arising in hydrodynamics) is based on an unverified hypothesis; after all, no one attempts to prove that a solution of a difference system for a step 6 chosen by the problem-solver differs from the exact solution by less than a number E specified by the problem-solver. The use of hypotheses implies a deliberate refusal to consider all possible situations such as those random cases that are the most probable in the formally mathematical sense. In this section we attempt to construct a tactic for finding a minimum of a multivariable function. The tactic will take advantage of the proposed organization of the problem. The type of organization of functions which we assume here (this will be described somewhat later) is very typical for many cases, and is probably one of the most frequently used types. Extremum problems are also interesting in themselves, comprising an important chapter of modern automatic control theory (automatic optimization [142-144]). Let F(x,, . . . , x,, y , , . . . , y,) be a function whose minimum values are sought. It is assumed that a system is able to measure the values of F. It is also assumed that the system can measure the values of the variables “It is possible to predict weather exactly for tomorrow, but one needs a month to do this”-according to Richardson.

Principles of the Functioning of the Central Nervous System

135

x,, . . . , x I 1 ,which will be called its working parameters. The arguments y , , . . . , y , are the hidden parameters of the system that are dependent on time, and perhaps also on the variables x1, . . . , x , ~ the : system is not able to measure or change the values of the hidden parameters. The function F ( s , , . . . , x,,, y , , . . . , y,,) will be written as (sl, . . . , s,, r ) ; the function @ will be referred to as the estimator of the system. It should be noted that the function @ is not assumed to be given analytically or in any other way, so that the selection of the required values of the working parameters should be done experimentally. The time-dependence of the estimator (this dependence is by no means assumed to be known) implies the necessity of a continuous search for the required values of the arg~ments.~ The important property of this type of search is its speed. Only those search tactics can be acceptable for which, roughly speaking, satisfactory values of the estimator are obtained within a time interval during which the estimator will not change appreciably. Thus, the search speed turns out to be related to the rate of change of the estimator. We should note that the time-dependence of @ loses its meaning if one is concerned with finding the absolute minimum. Therefore, we shall limit ourselves only to the problem of finding the region in which the estimator has relatively small values. (The region is equivalent to a level which must be maintained.) The automatic search for values of the working arguments which assure sufficiently small values of @ can be conducted in many different ways. Those various methods may be divided conveniently into three groups. The first group includes the so-called blind search methods. For them, it is characteristic that all points of the space of working parameters are either scanned in a certain order or selected randomly (the principle of homeostasis [132, 1631). When sufficiently small values of @ are reached, the search is stopped until those. values go beyond the admissible limits. Blind search methods make very little use of the characteristic features of the estimator (its organization). The results of a separate experiment are not used in the search that follows, so that the information about the estimator which was gained when measuring the values of @ is lost. Therefore, the values of @ are not improved from experiment to experiment, and on Moreover, if @ did not depend upon time (and the time devoted to search were infinite), one could be satisfied with complete scanning of its values. Using only one memory cell containing the minimum of the previous values, one could obtain the value of the absolute minimum.

136


the average turn out to be relatively high. Blind search methods use only one value of @-namely, the one that is given at any given time. In this sense systems that work using blind search d o not have a memory. The second group comprises local search methods. They are quite numerous : they include such methods as the gradient, relaxation, steepest descent techniques, and others. Their common feature is localness: the working point continuously moves through the space of working parameters. To prepare the next experiment, one uses the values of @ in the small neighborhood of the preceding experiment. Tactics of this type make it possible to achieve a systematic lowering of the values of @ during the search process, and thus give a considerable advantage to local methods as compared with blind search techniques. Local search systems (as applied to the problems of automatic optimization) are described in detail in the papers by Fel’dbaum [142-144], which also indicate the possible circuit designs of the electronic systems in question. The use of any specific local method necessitates an experimental determination of certain constants which define the search (e.g., the magnitude of the step in the gradient method of steepest descent). The values of the constants that assure the most rapid search are important characteristics of the minimizing function. However, inasmuch as the local search methods use only the local features, the constants are different in various regions and d o not characterize @ completely. The use of only the local properties of @ limits the effectiveness of the local search methods, and creates a constant danger that the search will “cycle” in one place in any “small second-degree dip.” When the values of the gradient are small, the search becomes a blind wandering, and its effectiveness then is insignificant. The third group of automatic search methods comprises methods which we call nonlocal. These methods are characterized by the fact that the trajectories along which the working point moves in the space of parameters are not continuous. For this reason, the volume scanned per unit time becomes considerably greater, and the search itself is significantly faster, thus comparing favorably with local methods. The simplest of the nonlocal search methods combines the principle of homeostasis with any local method. Methods of this type (which, incidentally, are often used in computing practice) can be described as follows. Upon choosing an arbitrary point, one conducts a search (descent) using a certain local method. When the search “cycles” in one place, i.e., when changes in


137

the value of the estimator become small during subsequent motion, a new arbitrary point is chosen, and the process is repeated. This is the way, for example, the simplest nonlocal gradient method works. Methods of this type also use only the local properties of the minimizing function; the information obtained during the local descent is not used further and is lost. Therefore, at the beginning of each descent a start is inescapably made in a region of large values of @, which means that the local descent is long. We shall now describe a method of nonlocal search which in our work [61] was referred to as “the ravine method.” This method permits us to use properties of function organization that are more extensive than the local behavior of the function. The ravine method is effective in those cases in which the working parameters xl, . . . , x, may be subdivided into two groups. The first group, which includes almost all of these parameters, consists of those arguments whose variation results in a significant change of the value of @. Thus, the selection of the values of these parameters (we shall call them nonessential) can be achieved in a relatively simple and rapid fashion.. The second group of parameters includes a small number (for instance, one, two, or three) of variables. These variables may themselves be the working parameters from among sl, . . . , s,, , but more often are functions of them. A variation of the variables in the second group (essential variables) results in a relatively small change of the estimator. The number of the essential variables will be called the dimensionality of a ravine. Of course, such a subdivision of parameters is impossible for every function that could be defined by a mathematician. However, for functions occurring in man’s practical activities (here we include reasonable problems in physics, engineering, and physiology) this type of breakdown is apparently possible in a considerable number of cases (probably even in a majority of cases). Keeping in mind the difficulties associated with an exact definition of these concepts, we shall nevertheless allow ourselves to call functions well-organized which admit of such a parameter decomposition. The hypothesis that an estimator @ is well-organized lies at the basis of the ravine method. The decomposition of parameters into essential and nonessential should, of course, be done automatically in the course of the search. Here it is important to note that the decomposition of the parameters into groups depends, generally speaking, on the time and on the point X = (.yl, . . . , x , ~ ) in the space of working parameters.

138


Now we shall describe the search itself. At first an arbitrary point X , is selected. From this point a descent is made along the gradient (or using any other local method). The descent is continued until the relative decrease A@/@ exceeds some present value A , called the gradient test. When the ravine method is used, the local descent should be made roughly, selecting the value of A to be relatively large (e.g., d = 20%). The problem is that since only the local search ceases to decrease the values of @ to any significant degree, we fall into a region where the variables in the first and second groups become equivalent, i.e., the function ceases to be well organized. Therefore, if we continue local search, not moving significantly along the essential variables, we shall wander randomly moving only along the nonessential variables. Properly speaking, this is indeed the reason for the low effectiveness of local search methods. Thus, suppose that a descent along a gradient has brought us to a point A , . Then a point X , is selected in a neighborhood of the point X , at a distance considerably larger than a step of the gradient descent (for example, in some direction perpendicular to the gradient). From the point XI a local descent is made to a point A , . After the points A , and A , are found, the point X , is found by means of the so-called “step in the ravine.” The points A , and A , are connected by a straight line on which the point X , is found at a distance L from A , , called the length of the ravine step. For well-organized functions, this length is chosen much greater than the length of the gradient step. The ravine step is chosen experimentally. Its value will, to a large extent, determine the efficiency of the automatic search: for a fixed value of L,we “roll over small ridges” and “climb tall mountains.” These scales of magnitude are determined by the value of the ravine step. When the point X , is chosen, a gradient descent is made to a point A,; a point X , is selected using the points A , and A , just as the point X , was found using the points A , and A , , and then the process is repeated. The points Xiare thus found in places, where there is an expectation of small values of the estimator or close to them,6 so that the entire search is essentially conducted in regions where the estimator has small values. Another important thing should be noted. When the length of the ravine step is correctly chosen, as one moves along the ravine an adaptation to its direction occurs, so that the lengths of the gradient descents become much smaller than the lengths of the ravine step. This adaptation is related to the The value of @ at points Xi(on the “slopes of the ravine”) does not have to be small, because of the effect of the nonessential variables.

139


fact that, in the course of moving along the ravine, the direction of the movement is made more accurate. Thus, because of a more accurate separation of the essential variables, the fraction of time spent on descents along gradients is lowered. This results both in an acceleration of the search process itself and in a considerable lowering of the values of @ thus computed. This adaptation of tactics possesses features which may be related to such terms as learning or expedient behavior. The efficiency of the search to a large extent depends on the choice of the ravine step length and the value of the gradient test A . The values of these parameters that assure the most effective search are important characteristics of the function @. Thus, the use of the ravine method makes it possible to obtain considerably more information about the structure of this function than is possible using local methods. We have described the ravine method in its simplest form, i.e., when the ravine is one dimensional. In the case of a multidimensional ravine this method, particularly at the beginning of the search, is inadequate. In those cases it is advantageous to take several points at the outset (e.g., two for each of the working parameters). Then, having obtained the points resulting from descent, one can find the correct direction of the ravine from them. It is often expedient to start the search from a “pencil of ravines” originating in the same region. We have described here the tactics of ravines of first rank in the sense that the variables are divided into two groups. It is not difficult to devise tactics of higher ranks. Thus, for example, in a tactic of second rank the variables are divided into nonessential, essential of first rank, and essential of second rank, and the lengths L1 and L, of the ravine steps along these variables are chosen correspondingly. The gradient and steepest descent methods require the computation of gradients, i.e., the computation of the values of @ at the (n 1)st point, which for large n involves a considerable number of operations. Here it may be useful to use finite automata capable of expedient behavior [154, 15.51. One can use the automata Lzkn,n (k is the number of working parameters, n is the complexity of an automaton). “Penalties” and “rewards” are determined from a given value of the gradient test, and to each of the rays in the diagram of states of an automaton there corresponds a motion in a definite direction along one of the working parameters. It is important to note that (for functions which vary rapidly with time) in the course of motion along a ravine, the correlation between the values of the function at points lying at a distance of a ravine step becomes low. Therefore, the effectiveness of the ravine method begins to approach the

+

140


effectiveness of the simplest nonlocal gradient method. A further enhancement of the time dependence leads also to a decrease of the local correlation (the correlation between the values of the function at points lying at a distance of one gradient step from each other) so that the tactics begin to approach a blind search. A similar situation also arises when the degree of organization of the function is lowered. In essence, the ravine method includes both the nonlocal gradient method and blind search, thus yielding a significant payoff in those cases in which the estimator is well organized, and the speed of search exceeds the rate of variation with time. In the remaining cases, the ravine method is not inferior to other methods. In using various methods of automatic extremum search, the question of their relative worth arises. Here it is natural to use functionals of the type (I,T)

ST’

Y p q X I , . . , x,, t ) ) d t .

0

(1)

The function Y may be chosen differently depending on the choice of the criterion for judging the efficiency of the search. Thus, for example, for ‘P(@)= @ the value of this functional corresponds to the so-called “cost of the search” defined for simple systems of automatic minimization [159]. I n problem where the objective is to reach the values of @ which do not exceed a certain level C (“minimization with respect to the level C”), it is convenient to use the functional which can be obtained from (1) upon setting

When the ravine method is used, the value of these functions is lower than in the case of other methods since, due to the process of adaptation, the entire search is conducted in the region of small values of @. The application of the ravine method to the problems of the phase scattering of particles and to the analysis of the structure of crystals was described in the literature [42, 51, 52, 581. The corresponding physical phenomena are also described [115, 1241.

2 Simulation of Expedient Behavior of a Group of Automata Before proceeding to the next group of mathematical models that arise in our preoccupation with the physiology of the central nervous system, it seems to us that it will be of some use to emphasize certain characteristic

PrincipJes of the Functioning of the CentYaJ Nervous System

141

features of the structure of complex control systems. Knowledge of these systems is essential for simulation of: (1) complexity of the systems: presence of a large number of relatively autonomous subsystems, experimental difficulties involved in studying and describing the interactions among such subsystems; (2) reliability of functioning assuring the expedient behavior of the entire system also in the case when some subsystems suffer a failure; and (3) diversity of problems handled by complex control systems, and the impossibility of separating specialized systems for solving each problem.

When simulating the behavior of complex control systems, the necessity naturally arises of separating the simplest forms of such behavior, of searching for structures that possess expedient behavior in the simplest forms, and of constructing a language useful for describing the mutual interaction among the slmplest structures whose collective behavior would allow us to convey the essential features of the behavior of complex control systems. We would like to mention one attempt at describing complex control systems by separating out elementary problems and structure^.^ In choosing the structures of the simple systems capable of expedient behavior, we have made use of finite automata. Here an automaton is defined as an object capable at each instant of time t = 1, 2, . . . of receiving a finite number of signals S , , . . . , S, and of changing its internal state accordingly. An automaton is capable of a finite number of actions f,, . . . , fk. The choice of an action is determined by the internal state of the automaton; it is assumed that an automaton has m internal statesv,, . . . , pm; the number m will be called the capacity of the automaton memory. If an automaton is in some medium, then its actions I cause the responsive reactions of the medium S, which are in turn the input signals for the automaton. The automaton, one might say, uses them to make a decision about subsequent actions. In the simplest case we shall assume that all possible reactions of a medium are received by the automaton as reactions belonging to one of the two classes-the class of favorable reactions or the class of unfavorable reactions; these classes will be called rewards and penalties. Inside each of these classes, the reactions of the medium are indistinguishable. The expedient behavior consists in increasing the number of favorable reactions and decreasing the number of unfavorable ones. A more detailed exposition is given on pp. 12-83. In what follows, we give an exposition for a nonmathematical reader (Editor’s note).

142


The role of the medium is to establish the relation between the actions of the automaton and the signals received at its input. This relationship may, generally speaking, be very complex, especially when a given automaton interacts with other auiomata. On the other hand, the information obtained by the automaton consists merely of whether a reward or a penalty followed the last action, so that the character of the medium is not known to the automaton beforehand. It is therefore natural to select the construction of the automaton in such a way that its behavior will possess the maximum expediency in the simplest cases, and then study the behavior of the automaton and groups of automata in more complex media. The simplest of the problem arising here involves the behavior of an automaton in a stationary random medium. In such media, for each of the actions fi(i = 1, . . ., k) of the automaton, we are given the expectations ai of its reward, so that the set a i , . . . , uk specifies a stationary random medium. Here the probabilities of a reward and of a penalty are given by

The functioning of a finite automaton A in a stationary random medium is described by a finite Markov chain, and for ergodic chains one can speak of final probabilities of states and of the final (independent of the initial state) value W ( A ) of the average payoff received by the automaton in a stationary random medium. It is natural to compare the average payoff obtained by such an automaton with the average payoff which could be obtained by a person who (as distinguished from the automaton) would know beforehand the parameters a i , . . . , uk of the medium. This person would obviously perform only the action that yields the maximum payoff, and the average payoff for this person would be equal to the greatest of the numbers a i , . . . , u k . It turns out that the payoff to any finite automaton is less than ( a i , . . . , ah), but one could construct sequences of finite automata A i ,. . . , A , , . . . , such that lim W(A,)

n+m

=

max(a, ,. . . , ak).

Such sequences are called asymptotically optimal. A description of a number of such constructions can be found on pp. 18-21 and in the papers [102, 112, 128, 154, 1571, which are devoted to a simulation of the simple forms of behavior. In this case, the number n for an automaton in an asymptotically optimal sequence may be interpreted as the capacity of its memory. We shall not


143

dwell here upon questions related to the behavior of automata in media whose properties are not stationary (see p. 25). We shall first of all be interested in the problems related to the collective behavior of automata. The collective behavior of automata is generated by their interaction. We have agreed to consider only the simple automata in whose construction there is no a priori information about the medium or other automata; the information obtained by a simple automaton is limited to just information about a reward or a penalty incurred by a given action. For this reason, we shall only consider those forms of interaction which can be realized in the collective behavior of such simple automata. The language of game theory provides a convenient tool that can be used to describe such forms of interaction. However, the models of collective automaton behavior differ considerably from the models accepted in game theory. Thus, in game theory we assume that the system of payoff functions which defines a game is revealed to the players before the game. Using this a priori information and making use of any computing means, the player makes a choice of a strategy. The strategies thus chosen are not changed in the course of the game, so that the game is similar to a play of chess that begins and ends with a home analysis. Models of collective automaton behavior (an automaton game) [I571do not assume that any a priori information is present so that the strategies are selected in the course of the game itself. It is assumed that an automaton game consists of a sequence of plays. Here a playf(t) of a game r, played at time t, is defined as a set f(t) = ( f l ( t ) , . . . ,f x ( t ) ) of actions (strategies) selected at that moment by the 1) automata A', . . . , A x participating in the game. The outcome S(t of a playf(t) is defined as a set S(t 1) = (S1(t l), . . . , SaV(t l)), 1) = 0 if the automaton Aj wins in this play, and Sj(t 1)= 1 where Sj(t if the automaton loses in the playf(t). By giving the structures of the participant automata and the probabilities P ( f , s) of the play outcomes, we define a game played by automata. Using these probabilities, one can determine the expected value W(f) of the payoff to an automaton Aj in a playf. The system of payoff functions thus constructed defines a game in the sense of game theory, a game which is equivalent to an automaton game. Thus, the information concerning wins and losses as a result of a given play specifies the values of the input variables for the participant automata, determining the choice of strategies in the subsequent plays of the game. In this case, the automata do not receive any information about either the actions of their partners or the strategies which are at the disposal of their

+

+

+

+ + +

144


partners, of even about the number of opponents. For a given automaton the role of the remaining players is reduced to the formation of a more or less complex medium in which the automaton should be capable of expedient behavior. Therefore, when selecting the structures of the participant automata it is natural to require that these constructions assure us of expedient behavior in the simplest game-a game with one player (“games against nature”), i.e., in a stationary random medium. It turns out that for quite a number of automaton games such simple structures make for expedient behavior. Consider as an example a zero-sum two-person game. Suppose that the game matrix is given to one of the opponents before the game, and he chooses the optimal strategy in the gametheoretical sense, and his opponent is an automaton belonging to an asymptotically optimal sequence. Then the automaton (with sufficient memory capacity) achieves a payoff equal to the value of the game according to von Neurnanr8 If both players in the game are automata, then their payoff is a!so close in some sense to the value of the game. Of course, the most interesting are games played by many automata. We are interested here in the simplest games, namely, those in which all the players are equivalent. The simplest example of such a game is the so-called Goore game. This game is played by N persons each of whom is capable of only two actions. Here the probability of winning for any of the players is determined only by the fraction of the players using the first strategy. It is obvious how people would behave in this game if they knew the conditions of the game beforehand-they would make an agreement that the first strategy should be used by that number of players which assures the maximum payoff. The simple automata playing the games do not know beforehand the payoff function of the game or the actions of or the payoffs to the opponents in its separate plays. Each of the automata receives only the information about its own win or loss in a given play. Nevertheless, with sufficient memory capacity they also maximize their payoff. The expedient behavior of every automaton in the simplest problem results in the expediency of their collective behavior, replacing such difficult to formalize phenomena as “agreement on common actions.” It is interesting to note that for a fixed memory capacity, if the number of It will be noted that the opponent of an automaton with a linear tactic, who does not use von Neumann’s strategy, may win from a linear automaton more than the value of the game (Editor’s note).


145

automata playing the Goore game increases, the expediency of their collective behavior decreases, in the limit not differing from random behavior. Conversely, for any fixed number of players an increase in the memory of each automaton results in a heightened expediency, and the average payoff in this case tends to the maximum possible value. Let us give another example of an automaton game which we called the distribution game. The situation simulated in this game is typical of a problem facing predatory animals when they must choose their hunting grounds. Here the number of prey per predator is determined by the supply of prey in the area chosen by him and the number of predators hunting there at the same time. A choice of a certain strategy corresponds to a choice of the hunting grounds in the game, and a certain value of the payoff function corresponds to the number of prey. The distribution game is specified by k nonnegative numbers a, 2 2 ak 2 0 called powers of the strategies. The game is played by automata A ’ , . . . , Ax,N < k, each of whom has k strategies fi,. . . ,fk. The expectation of winning for an automaton that chooses strategy Fj in some play of the game is equal to ajlmj, where mi is the number of automata using strategyfi in this play. The distribution game as played by simple automata was studied by us using a computer simulation. We found that the behavior of automata with sufficiently large memories did not differ from the behavior of people knowing the conditions of the game beforehand and making agreements about their actions-the automata (with probability close to unity) were choosing their actions in the optimal manner. Thus, for example, in the distribution = a, = 0.33 game played by five automata with a, = 0.9; a, = a, = it turned out that in 99% of the plays the first strategy was chosen by two automata, and the remaining ones by one. In this case the average payoff to each automaton amounted to 0.38. It is not hard to verify that in this situation it is not advantageous for any automaton to change its strategy. This type of behavior of the automata coincides with the behavior of people knowing the powers of the strategies beforehand. However, by making an agreement about sharing the payoffs, people could achieve higher payoffs. In fact, if the total payoff is divided equally among all the players, it would be useful to use the first five strategies taking one of each: then the payoff to each player would amount to 0.44. If the payoff to the automata in the distribution game is totaled in each play and shared equally (“the game with a common fund”), then the behavior of the automata will also change: in each play the first N strategies are chosen, where each strategy is chosen by one automaton. In this case

-

146


the average payoff changes, and with an increasing memory capacity of the participant automata it approaches the maximum possible value (in our example, 0.44). It is interesting to note that the growth of the average payoff when the common fund is introduced is achieved only with a sufficiently high memory capacity; if the memory is small, the introduction of a common fund lowers the average payoff. In other words, if the individual expediency of behavior of each player is low, then the equalizing distribution of payoffs is not advantageous. The behavior of automata in the distribution game possesses the characteristic features of reliability. In fact, let us assume that the automata playing the game with a common fund may suffer a failure. However, the remaining automata will, as before, play in the most advantageous way, that is, independently of whether any automaton fails; the strategies with maximum powers will be used as previously. The increase in the average payoff to each of the automata continuing the game will partially compensate for the lowering of the total payoff. When new automata are included in the game, they will also be distributed in the most optimal manner. The behavior of such a group of automata is similar to a perfectly reliable machine in which the wear and tear to the most important parts is automatically compensated for at the expense of the less important parts. For the sake of the following, it is important to note that both the Goore game and the distribution game are examples of automaton groups that are “easy to control” in the sense that to control them it is sufficient merely to specify the pay-off functions. T o achieve the optimal operating mode, it is not necessary to control the behavior of each individual automaton. In these examples the optimal operating mode is selected by automata that do not have any information about the actions of the other automata and are not capable of directly changing these actions, so that the interaction of the automata is limited to participation in a common game. We note, in addition, that a slightly more complicated distribution game may serve as a model for the naturally arising problem concerning the most convenient distribution of the computing means when it is necessary to simultaneously solve a number of problems. Those automaton games are interesting in which the payoff functions of each automaton depend only on the strategy chosen by them and the strategies of a limited number of other players, their “neighbors” in the game. The simplest example of such a game is provided by the so-called “circle game” in which the payoff to a player Aj depends only on his strategy and the strategies of the players Ai-’ and Ai+‘ that are his neighbors.


147

For games with a limited number of neighbors, it is characteristic that the plays assuring the maximum payoff are selected relatively rapidly. Here the behavior of the automata possesses the features of reliability that were referred to when describing the distributioii game.

3 A Mathematical Description of Excitable Tissues The first two sections of the present work were devoted to the simulation of certain features of the behavior of complex systems. In constructing these models, we have not attempted to consider the specific physiological structures underlying behavior. In this section we shall (of necessity) briefly describe certain attempts at simulating the simplest physiological structures whose properties are close to the simplest properties of excitable tissues, which will perhaps make it possible to explain certain features of their functioning. Here, in contrast to the traditional models of the type of nerve networks given by W. S. McCulloch and W. H. Pitts, we shall not consider a system composed of a large number of separate elements with a complex system of connections among them. Instead, we shall consider continuous media, assuming that those points are neighbors of a given point which lie in its immediate geometric neighborhood. We note that in a physiological experiment, the separation of an individual element is sometimes difficult and is not always meaningful. Let us consider the simplest example of such a continuous medium [60].O An active tissue will be defined as a medium that possesses the following properties : Each point of the medium is capable of instantaneous excitation. During a time R after the instant of excitation, the point cannot be excited. The value of the interval R is called the time of refractivity. The phase t ( x , t) of a point x at time t is defined as the time that has passed from the time of the last excitation. If t ( x , t ) is less than R, we shall say that the point is refractive. 2. The excitation may propagate in the medium. The velocity c(x, t) of the excitation propagation at the point x at time t depends on the phase of this point: c(x, t) = q ( t ( x , t)). The excitation cannot propagate in refractive media. 1.

*

This paper is included in the book (see p. 154) (Editor’s note).

148


3. A point is capable of spontaneous activity. This means that during a time T (the period of spontaneous activity) after the last excitation, the point may again become spontaneously excited. (It should be noted that property (3) is not obligatory, and we shall also consider media without spontaneous activity.)

The propagation of an excitation in active tissues possesses a number of interesting properties. Thus, for example, the process of propagating excitation impulses in a homogeneous ring from the active tissue is autosynchronizing: no matter what the initial phases and the initial distribution of the impulses in a ring may be, a regime is established such that the impulses are set up at equal distances in the ring and they propagate with a constant velocity. In the case of periodic excitation of the end of a segment, the impulses are also propagated, in the limit, from the active tissue with a constant velocity that does not depend on the initial distribution of the phases. (See also the literature [5, 6, 63, 1.391.) Now we shall give an example of the functioning of a plane excitable tissue. We assume, for simplicity, that the velocity c of the excitation propagation is constant, and that the initial phase is identical for all points and equal to zero. It is clear that all points of such a tissue will be excited simultaneously and with the same period T. Let us now imagine that at time f, 2 R the point x, will be excited from the outside. Then, obviously, the excitation will spread from this point with velocity c. At a time t < T, the set of the excited points forms a ring of radius c ( t - t o ) with the center at x o , and at a time T a l l points will be excited that lie outside of the ring of radius c(T - f,). At a time to T, the point x, is again excited spontaneously, and the process will be repeated periodically. In this case, each point of the medium will be excited with period T ; one can say that there is no interaction in the system here. It is not difficult to see that for an arbitrary initial distribution of phases and external excitations, a regime is established in the system in which each point will be excited spontaneously earlier than the excitation is received from its neighbors, i.e., a regime without interaction. For an active tissue whose points have different periods of spontaneous activity, in steady-state conditions the period of excitation of any point of the medium will be equal to the minimum period of spontaneous activity -the medium will be synchronized by its most active point. This very synchronization mechanism is realized in certain physiological objects. In the paper by Gel’fand el al. [57] it was, for example, shown that the sinus

+


149

portion of the heart is automatically synchronized by the active cell working with the highest frequency. The concept of an active tissue was used in constructing a number of models of physiological mechanisms. Thus, for example, in the work by Lukashevich [ZZ7], the author describes a computer simulation of the process of excitation propagation in the heart. This model made it possible to study the peculiarities of a number of pathologies in the functioning of the heart. Keder-Stepanova and Rikko [94J used nonhomogeneous active tissues to construct models of the salvo-type activity of the breathing center. Arshavskiy and co-workers [9, 15, 1351 have studied the “cable nets,” i.e., the homogeneous lattices whose ribs possess the properties of active tissues, as well as the characteristics of excitation propagation in such structures using the example of the myocardium and the dendrites of nerve cells. 4 The Principle of Least Interaction

We have described here various mathematical models connected in one way or another with our preoccupation with physiology. We understand very well the extreme diversity of these models, which are united only by the generality of their approach. Nevertheless, it seems to us that the models also possess a certain internal common feature which is perhaps also present in those physiological mechanisms which led to these models. We have in mind the principle of least interaction. We are still very far from the definitive formulation of this principle, and we actually only associate with these words our hope that it will be possible to construct a mathematical theory of complex control systems in which this principle would play a role analogous to that played by variational principles in analytical mechanics.’O We would like to think that in this future theory a uniform description will also be made of those preliminary mathematical models which were referred to in the present work. However, if we attempt here to talk about certain notions related to the principle of least interaction, this is only because these notions were useful in our activities. lo At the present time, we are still not able to determine for sufficiently diverse systems a function which, on one hand, could be readily interpreted as the interaction of parts of the system among themselves or parts of the system with the surrounding world, and on the other hand, such that when the system minimizes this function, the expedient behavior of the entire system will follow (Editor’s note).

150


A system will be said to function expediently in some external medium if the system strives to minimize its interaction with this medium. Here, as a rule, a natural definition of the interaction function results from the properties and the purpose of the system itself. Thus, for example, a measure of the interaction of an organism with a medium is provided by the deviations of the parameters of the organism’s internal medium from their optimal values. For models about which we talked before, these functions are diverse. Thus, for example, for a system used to search for the minimum of a multivariable function (Section l), a measure of interaction may be provided by the average value (over a certain time interval) of the function minimized. For automata capable of expedient behavior (Section 2), the interaction is measured by the average value of the penalty. For models of active tissues (Section 3), a measure of the interaction for an eiement of the tissue may be provided by a monotonic function of the deviation of the mean interval between two excitations from the period of spontaneous activity. Thus, all our models are examples of expedient systems. For such systems, it is typical that the most stable are states with minimum interaction. In this sense, expedient systems are, so to speak, inertial-they tend to enter a state of small interaction so that they will not have to change states again. It should be noted that similar notions lie at the basis of the principle of a homeostat as proposed by Ashby. For complex control systems, a structure is typical which allows a separation of the individual, relatively autonomous subsystems. For each of such subsystems, all remaining subsystems belong to the external medium, and the expediency of the subsystems is revealed in the minimization of the interaction among them, so that in stable states these subsystems function, as it were, independently or autonomously. In the last analysis, the functioning of each subsystem is determined by the external medium and the functioning of the entire complex system, the subsystem being part of the latter. However, at each moment the subsystem solves its own “particular,” “personal” problem--namely, it minimizes its interaction with the medium; therefore, the complexity of the subsystem does not depend on the complexity of the entire system. The expediency of the entire system is revealed in the minimization of the total interaction of the system with the medium. When the external medium changes, the previous mode of operation of the control system no longer assures the minimum interaction. The interaction both of the entire system with the medium, as well as of the individual subsystems among themselves, increases. The expediency of the system leads to another stable mode assuring the minimum interaction in the new medium.


151

Let us clarify the above with some examples. First consider the simplest homogeneous systems all of whose subsystems are equivalent. A model of an active tissue is such a system. As its elementary subsystems one can take the points of the medium forming the tissue. We have already seen that in such a tissue, no matter what the initial phase distributions and points of the initial external excitations are, a regime is established in which all points are excited spontaneously, i.e., they d o not interact among themselves. The propagation of the impulses in a ring from the active tissue proceeds at a constant velocity, so that the interaction of all points is identical and with a good choice of the interaction function it is also minimal. Similar terms can also be used to describe such important physiological phenomena as the synchronization of the operation of the individual elements, and to explain the necessity of the appearance of special desynchronizing mechanisms (for example, the type of Renshaw cells in the spinal cord). Another example of homogeneous expedient systems is the homogeneous automaton game, for example, the Goore game or the distribution game described in Section 2. Here we also see that the expedient behavior of the individual automata assures us of the expedient behavior of the entire system of the participant automata. For the case in which the external medium (in this model, the values of the functions defining the game) changes, the behavior of the automata changes, lowering the interaction. When discussing the distribution game, we have already mentioned the reliability of a group of playing automata. This property is characteristic in general of complex systems in which the expedient behavior consists in the interaction of subsystems which themselves possess expediency. For these systems, another property is also apparently typical. This is the fact that the expedient group behavior of the individual subsystems may also be achieved without the presence of direct links between them, so to speak. In a number of problems, to achieve this expediency it is sufficient to have the simpler type of interaction which occurs in automaton games. Lack of the necessity for direct links is also important because it enhances the reliability of control systems, and because it makes it possible to construct complex systems from simple subsystems. Otherwise, each subsystem would have a system of links growing with the increasing complexity of the entire system, and its structure, assuring expedient behavior, would also become complex. We must note that the principle of least interaction is also important, because it permits us to consider any system (or its component), if it is capable of expedient behavior, as a single whole, which, of course, considerably simplifies the analysis.

152


An example of an inhomogeneous control system may be provided by a system designed to search for the extremum of a multivariable function using the ravine method. In fact, this system consists, roughly speaking, of two subsystems-the level forming the local search and the level performing “the step along the ravine.” The first of these levels minimizes the value of the function using, for example, the gradient method. The second one minimizes the time spent on the local search, so that the medium for the second level is provided by the first level. In those regions where the properties of the function (i.e., the subdivision of the variables into essential and nonessential ones) are stationary, the interaction of both subsystems is minimized, i.e., both the values of the function (on the average) and the lengths of the gradient descents become small. Where the function changes (e.g., at the bend of the ravine) the interaction increases. The expediency of the functioning of ‘our subsystems has the effect of imparting expediency to the entire system-it lowers the average values of the function in the course of the search. It is possible that the above notions can be used to explain certain general features of the interaction of nerve centers. The central nervous system contains a large number of separate nerve centers serving to control the effectors, so that each act of behavior (e.g., motion) is a result of their joint activity. Here a change in the functioning of one of the centers should of necessity result in a change of the activity of the remaining ones. If we assume that each nerve center represents an expediently functioning system, then our mathematical models allow us (to a certain degree) to imagine the interaction of the nerve centers without considering the complex system of links and the coordination of their activity. Here one may consider that, for the nerve centers at some level, the stimuli of the medium are replaced by afferentation “coming from below,” and the stimuli from the higher centers determine their particular problems (one can say, determine the organization of this afferentation into a “system of payoff functions”). Here, of course, great importance is attributed to the question of what “particular, personal” problem is solved by each nerve center, which for it is a measure of interaction. We assume that this role is played for each center by the incoming afferentation, and the expediency of its operation consists of lowering the stream of afferentation arriving at the nerve center, which is the measure of interaction for the nerve center. We take the risk of assuming that this role of afferentation is universal, i.e., it takes place in one way or another for all nerve centers, and the nerve system as a whole is organized according to the principle of least interaction. For each external situation, the problem facing the nervous system consists in finding a mode


153

of operation in which afferentation is minimal. An important feature of the model described here is the relative simplicity of control: After the payoff functions are given, there is no need to control in detail the centers of a given level. By virtue of the expediency of their behavior, they will themselves choose the best distribution of responsibilities (compare with the distribution game described in Section 2). On the other hand, this method of control is also distinguished by its great flexibility-the corrections to the structure of the “payoff functions” can be made by a large number of nerve centers which have some relation with the problem and are not acting in coordination.

0

Continuous Models of Control Systems’ 0

A number of papers have been devoted t o the construction of mathematical models of control systems imitating the functions of the nervous system. These papers and the theory of automata (logical networks) that arose in this connection turned out to be useful in studying the principles of the construction and functioning of the logical circuits of computers. However, their underlying discreteness with respect t o the elements and time makes them of little use when it comes t o describing the functioning of somewhat more complicated systems, like biological systems. It seems reasonable t o us that, instead of considering a large number of individual elements with a complex structure of links among them, one should investigate continuous models.2 Concepts of this kind were used in the important and interesting work of Wiener and Rosenblueth [I971 in describing the mechanism of the fibrillation of the heart muscle. Such a continuous medium will be constructed phenomenologically by considering at the same time certain “natural” assumptions about its properties. It will be noted also that in physiological experiments it is difficult, and not always meaningful, t o separate the individual elements so that a description of the medium not in terms of separate elements but directly, is advantageous. We shall describe here the simplest possible model with the three properties t o be given in what follows. In this sense the article is preliminary, and we hope t o construct This chapter was written jointly with I. M. Gel’fand and previously published [60, p. 12421 (Editor’s note). In this chapter, we shall limit ourselves to continuous models in the simpler sense of the word-namely, in a continuous medium closeness will be understood as geometrical closeness. Generally speaking, however, closeness could also mean closeness in “phase space.” (For example, we can understand the distance between the points A and E as the time that it takes for a signal to travel from A t o E.) 154

155

Continuous Models of Control System

continuous models later which are better approximations of the physiological prototypes. An active tissue will be defined as a medium possessing the following properties3: ( I ) Each point of the medium is capable of instantaneous excitation. During a time R after the time of excitation, the point cannot be excited. The quantity R is called the time of refractivity. The phase t ( x , t ) of a point x at time t will be defined as the time that has passed from the last excitation of this point. For example, if a point x was last excited at time t = 0, then ~ ( xt ,) = t . If t ( x , t ) < R, then we shall say that the point is in the refractive phase. ( 2 ) The excitation can propagate in the medium. The velocity c(x, t ) of the excitation propagation at a point x at a time t depends on the phase of the point c(x, t ) = q [ s ( x , t ) ] .The function q ( t )is defined for all t 2 R. The propagation of an excitation is impossible in regions in the refractive phase. Thus, the state of each point of the system is determined by whether it is excited at a given time or not, and by its phase t. The propagation of an excitation is understood as a propagation of a “discontinuity” in the state of the system whose front moves with the velocity c along the normal to the “discontinuity.” One could also consider a model in which the front of the excitation would have a finite extension. The propagation of an excitation could then be compared to the spreading of flames in a burning m e d i ~ m . ~ (3) The point is capable of spontaneous activity. This means that during a time T after the last excitation, the point again becomes excited spontaneously (if, of course, the point did not become excited due to the influence of the neighboring points). The quantity T is called the period of spontaneous activity.

Let us consider some examples of the processes that may take place in such a tissue 1. We shall investigate the operational mode called the ring rhythm. Properties (1) and (2) were essentially considered by Wiener and Rosenblueth [197]. These properties together with (3) are convenient, not only in the investigation of phenomena in the myocardium, but also in the construction of control systems. A model of flame propagation could probably be useful in studying the propagation of an excitation along an axon or a muscular fiber. A model could be provided by the equation [XI] azu/at2 = A a w x ’ F(~)

+

or by systems of equations of this type.

156


I

i

I

Figure 20

We have a thin thread of length f closed to form a ring. We assume that spontaneous activity is absent, and the velocity c of propagation of an excitation is constant. As the initial conditions, we take the phase distribution ~ ( x 0), , shown in Fig. 20 with a solid line. Thus, initially the point 8, is excited, and its neighbors on the right (the segment 8,O’) are in the refractive phase. Therefore, the propagation of the excitation will proceed to the left from the point O,, and the points located to the right of 8, will gradually leave the refractive phase (their phases will grow). By a time t the phase distribution t(x, t ) will take the form shown in Fig. 20 with a dotted line. The length of the segment 8,8, is obviously equal to cr. Since the beginning and the end of the segment are identified, the excited point during time t = f / cwill return to the position B0 and the phase distribution will coincide with the initial one. From this example, we can see the role of refractivity that makes it possible for the excitation to move in one direction. In this sense, the refractivity that makes it possible for a point A to act on a point B, where the point B does not act back on the point A , appears to be the opposite of Huyghens’ p r i n ~ i p l e .In ~ the following, we shall return again to operating modes of this type. 2. Consider now an active tissue, and let T be the period of its spontaneous activity. If all points of the tissue were excited simultaneously at the time t =;- 0, then later they would simultaneously become excited at times t = T, 2T, . . .. Thus, on the phase diagram distribution at the initial instant coincides with the axis of abscissas, and then moves upward with a velocity equal to unity. When the time t = T is reached, this straight line again coincides with the axis. Since, however, the phases 0, T , 2T are indistinguishable, it will be more convenient for us to imagine the phase distribution as continually rising, and to define the phase as the distance of the point in question on the curve of the phase distribution to the straight line t

= nT,

nT 5 t ( x , t ) < (n

+ l)T.

An important property of Huyghens’ principle is its mutuality.

157

Continuous Models of Control Systems

We shall say that there is no interaction in the system if any point of the tissue is excited spontaneously and earlier than it could become excited under the influence of the neighboring excited points. In the absence of interaction, the curve of the phase distribution moves upward parallel to itself. It is not hard to see that interaction will be absent for all initial distributions for which the slope of the tangent is less than c-l. For a system of a large number of measurements, the condition for the absence of interaction assumes the form grad t ( x , t ) 5 c-l, where x = (xl,x2,x3) are the coordinates of the point of the tissue. We shall see that, when this condition is violated, including the presence of discontinuities, after some time a phase distribution will be reached that will satisfy this condition. The medium in question possesses reliability in a certain sense. In fact, if for some set of points of the medium xl,. . . ,xn the periods of spontaneous activity increase randomly, then this is not going to result in any noticeable change in the phase distribution, since these points become excited under the influence of the neighboring points with an infinitesimal lag. On the other hand, a random single decrease AT in the period of spontaneous activity will result in a change of the phase distribution only in a region of radius c AT. In this respect, the active tissue discussed here differs from the discrete models of logical networks in which a change of the properties of an individual element distorts the functioning of the entire network. We shall show now that such an active tissue may serve in the role of a memory. Consider, for the purpose of simplicity, the initial phase distribution to be z(x, 0) = 0,and let the point x,, become excited at a time t = R E. The E will, as agreed before, be repphases of the point x, at time t = R resented as equal to T (curve 1, Fig. 21). The same figure shows the changes

+

+

I

ff

A

XU

Figure 22

X

158


in the phase distribution at various moments of time (curves 2-4). It is not hard to see that, beginning with the time t = T, the phase distribution curve ceases to change. Thus, the excitation of a point x was followed by a change in the phase distribution within a cone (which is characteristic) of the base radius c ( T - R - E ) , the cone being preserved so that both the point of excitation and its phase are “remembered.” Figure 22 shows how the phase distribution t ( x , 0) = 0 is established: the point t = R E is . base radius of the excited consecutively at times xo and t = 2R 2 ~ The characteristic cone is here equal to 2c(T - R - E ) . This example shows that each point of the medium may serve as a counter of the number of elementary excitations. We shall note in addition that in this case, in a steady state, at any point in time there are excited points, but interaction is absent.6 3. Consider now a thin slice of tissue with length I, devoid of any spontaneous activity, in which the propagation velocity C ( T ( X , r ) ) of an excitation is given for the values t 2 R and increases monotonically with increasing t. Let us assume now that the point x = 0 is periodically excited with a period T, and consider the process of propagating excitation impulses. Let y,+(x), k = I , 2, . . . , denote the time that it takes for the kth impulse to travel from point 0 to point x. Then the passage of impulses will be described by a sequence of differential equations y k l ( x ) = c-’(T y k - Y , + - ~with ) boundary conditions y,+(O) = 0 and initial condition y o ( x ) = p?(x). It can be shown that for an arbitrary function ~ ( xwe ) have the formula limk,,yk(x) = xc-’(T), i.e., that the propagation of an excitation in the presence of a periodic excitation occurs in the limit with a constant velocity. This discussion also indicates a method of experimentally determining the function ~ ( t )which , thus reduces to changing the established time interval between the excitation of the beginning and the end of the thread.

+

+

+

(I One could introduce a measure of interaction in the system in the following way. A t , such a measure For a point x , becoming excited at a time lying between t and t could be provided by the quantity T - t ( x , I ) . A measure of interaction for the entire system at a time t could be provided by the expression

+

s,

lim ( I / d t ) At+o

( T - t ( x , I ) ) do,

+

where E is the set of points becoming excited during the time from t to t A t . In Example 2 the interaction tends to 0 with increasing t . Conversely, one could construct examples (see, e.g., Wiener and Rosenblueth [197]), where the interaction tends to a constant value for t --f 03. The states in which the interaction tends to zero will be called normal, and the remaining ones will be called particular. (Particular states arising in the heart could naturally be called fibrillational.)

Continuous Models of Control Systems

159

Similarly, one can also investigate the propagation of an excitation in a ring. Let t ( x , 0) = ~ ( x )and , suppose that from x = 0 an impulse started propagating in a certain direction. Furthermore, let zk(x) be the time that it takes for the kth impulse to travel from 0 to x. Then the process will be described by a sequence of differential equations

Also, in this case one can show that for an arbitrary function p(x) limk+mzk(x)= xf0Zk1, where to can be found from the equation r, = /c-l(t,) [for 1 4 R c ( R ) ] . Analogous results can also be obtained when studying the circulation of a group of impulses along a ring. Discrete electronic models of this type were investigated by Ivanov and Telesnin [91]. We express our gratitude to the participants of the seminar, the physiologists I. S . Balakhovskiy, V. S . Gurfinkel', V. B. Malki,n, and M. L. Shik for their valcable participation in the discussion of the problems presented here.

0

Certain Problems in the Investigation of Movements’ 0

This chapter presents certain general concepts that seemed useful t o us in the study of the physiology of motor activity. Many of these views are strictly connected with the theory about the construction of movements developed in the outstanding papers by Bernshteyn [I7-221. Motor control is one of the most important functions of the nervous system. The structure and function of the nervous system are undoubtedly t o a large extent determined by this task. The physiology of movements is basically a study of a goal-oriented activity of the nervous system as a whole. Therefore, motor control seems t o us to be one of the most natural objects for studying those integral functions of the nervous system which are related, so t o speak, to “operative control.” The ultimate result of the functioning of the nervous system, so far as its motor control is concerned, is sending impulses t o the muscles, and the basic question of the physiology of movement is a study of the mechanism governing the development of expedient combinations and sequences of such signals. The simplest point of view here is the notion that there exists some higher nerve center (lying, for example, in the cortex), where generation of such commands takes place. The commands would completely determine the motion, so that the role of the remaining nerve mechanisms consistes only of a transmission of those commands. The lack of evidence for this point of view is obvious, and papers on the physiology of movement (beginning with the discovery by Fritz and Gittzig, and then the classic works of Sharington, Magnus, Pavlov, Ukhtomskiy, and the publications of contemporary researchers) all deal with the central question of the interaction among various nerve mechanisms in the process of performing a movement (see the literature [ I 7-22]). In natural movement dozens of various muscles, Jointly with Gel’fand et at. [62a] (Editor’s note). 160

Certain Problems in the Investigation of Movements

161

working in coordination, participate. A system of commands necessary for performing motion cannot help but be very complex. In developing this system, it is necessary to take into consideration a rich and diverse afferentation, including that arising in the course of the movement itself. Specific performance of motion to a large extent depends on the initial conditions-the initial position, and so forth. It is beyond any doubt that in the development of commands for muscles and the afferentation processing for that purpose a great number of nerve centers is needed. A study of their interaction brought us to an attempt to describe the peculiar features of complex control systems from a single point of view that will be described below. However, numerous and sometimes essential concepts of the theory proposed in this chapter are still in need of experimental verification, and are considered by us as working hypotheses to help in further search. This point of view (we call it the principle of least interaction; see p. 149 and Gel’fand et a/. [56]) is that a complex multilevel control system is considered to be a set of subsystems possessing relative autonomy. Each of these subsystems has its own “personal” task consisting of lowering the interaction with the “external medium” : the latter for a given subsystem consists of the medium, external with respect to the entire system, and of the remaining subsystems. Complex control systems may consist of several levels, each of which includes a number of such subsystems. For subsystems of a certain level, the actions of the medium (which is external with respect to that level) include the afferentation coming from below, and the organization of their interaction is determined by the interaction of the higher levels. For the lowest level, the afferentation is exclusively receptive in character; the subsystems of this level have outputs into the effectors. Section 2 will dwell on these concepts in more detail. In the organization of motor control, an important role is played by utilization of those features of the motor task which can simplify control: lower the number of independently controlled effector parameters and simplify the processing of the advancing afferentation. The problems that possess those features are, so to speak, organized. In motor control, organization is revealed primarily in the fact that for each motor act one can separate out a relatively small number of the leading effector parameters and determine the basic afferentation necessary for performing a given movement. The striving toward such a simplification is also a case of the principle of least interaction. The lowering of the number of controlled parameters lowers the general level of the impulsion needed for control.

162


This chapter consists of three sections. The first one is devoted to synergies and certain other mechanisms simplifying motor control. In the second section we shall present our general concepts relating to the mechanism of movement generation at the spinal level. In the last section we describe modeling considerations connected with the functioning of a group of motoneurons. 1 Synergies and Other Mechanisms Simplifying Motor Control

In order for the higher levels of the central nervous system to effectively solve the tasks of organizing motor acts within a required time, it is necessary that the number of controlled parameters be not too large, and the afferentation, requiring analysis, not too complex. An important role in creating such working conditions is played by the so-called synergies. Synergies are those classes of motions which have similar kinematic characteristics, coinciding active muscle groups, and leading types of afferentation. Each synergy is associated with certain peculiar connections imposed on some muscle groups, a subdivision of all the participant muscles into a small number of related groups. Due to this fact, to perform motion it is sufficient to control a small number of independent parameters, even though the number of muscles participating in the motion may be large. Although synergies are few in number, they make it possible to encompass almost all the diversity of arbitrary motions. One can separate relatively simple synergies of pose control (synergy of stabilization), cyclic locomotive synergies (walking, running, swimming, etc.), synergies of throwing, striking, jumping, and a certain (small) number of others. The synergies enumerated here are for a fully developed adult; the biomechanical side of the majority of them has been completely investigated. A more detailed study of one special synergy-the breathing synergy of standing-can be found in the paper by Gurfinkel’ af al. [79]. Here we shall describe it very briefly. In connection with breathing, noticeable displacements occur in various parts of the body. However, these displacements have very little effect on the location of the general center of gravity. The reason, as was explained, is that synchronously with the bending of the trunk to the back (during inhalation) the pelvis is bent forward; during exhalation the coupled displacements occur in the opposite directions. This synergy is specific, and does not occur, for example, during an external disturbance (a light pat on the spine). The synergy breaks down in some neurologic ailments, and then the center of gravity of the body moves in accordance with the phase of breathing. One might think that the above breathing synergy

Certain Problem in the Investigation of Movements

163

of the vertical pose is not an exception, but an example of a typical mechanism occurring in the most diverse natural motions. It is natural to assume that the learning of motion consists of acquiring the corresponding synergy and lowering the number of the parameters requiring independent control. Such a new synergy is produced each time not in an empty place, of course, but on the foundation of a small number of basic synergies and the inborn neurophysioIogical mechanisms lowering the number of the independent parameters of the control system. Some of those mechanisms, even though they were not studied by researchers from this point of view, have been known for a long time. We can mention here such well-researched examples of functional organization as the system of interaction of the muscle motoneurons; these are antagonists acting on the same joint, a system of postural reactions using a fixed system of the interaction of receptions of various types (labyrinth, otolithic, proprioceptive involved in receptions in the neck and the limbs), as well as an important mechanism of the development of time relations. The basic synergies and the simplest neurophysiological mechanisms enumerated here form, one might say, “the vocabulary of motions.” Developing this analogy, one can say that the letters of the language of motions are the stresses of the individual muscles, and the synergies combine these letters into words whose number is much smaller than simply the number of letter combinations. In this case the richness of the vocabulary makes for the diversity of the allowable motions. A majority of the motor tasks faced by an organism lies within the limits of this vocabulary, and it is only in exceptional cases that there arises a need to enrich it. Thus far, we have been talking primarily about the effector side of synergies. It is clear, however, that each synergy involves afferent streams which contain the leading signals and the addresses typical for them. The language of synergies is in this sense not only external (motions), but also the internal language of the nervous system used in motor control. Synergies make it possible to simplify the development of afferentation, organizing it in accordance with the motor task. From this point of view in a problem such as pattern recognition, one must first of all decide what synergy this problem involves, which in turn determines the subsequent course of recognition. We can consider that learning a new motion consists of developing a simple method of motor control, and reduces to a search for and a correction of a convenient synergy or group of synergies, including also identifying the leading afferentation. Krinskiy and Shik [ I I O ] have studied the role played by this last factor under conditions in which the motor task should

164


have been performed under the control of a purposely faulty visual afferentation. It turned out that, under certain transformations of the visual field, the system making use of visual afferentation in the problem of pose maintenance is stable. We can consider that such imperfections of the visual field still permit the use of the available synergy. More significant impairments of the visual field made the completion of the task impossible, and required a certain training time in order that the tas, might be performed. It is interesting that under certain transformations of the visual signal involving deviations from a given pose, the relearning, even though it made the completion of the task possible, did not result in regaining the accuracy of its solution which was achieved by a tested person using unimpaired visual afferentation. The use of the synergy mechanism is, of course, only one of the methods simplifying the problem of motor control. Another possible approach is connected with the similarity between the problem of motor control and the mathematical problem of the search for the minimum of a multivariable function. The language, natural for this mathematical problem [6Z], is convenient for describing the construction of motion. The combination of local improvements with extrapolation, which is characteristic of nonlocal methods of extremum search, is apparently also typical of the process of decision-making in motor problems, and certain features of that search can be revealed experimentally. Such an approach, as applied to the problem of maintaining an orthogradic posture, was used in a paper by Gel’fand et a/. [55] (see also the literature [56, 62, 74-77]) and stimulated further research devoted to the problem of tremor. Krinskiy and Shik [ZZZ] made an attempt to study a method for solving a simple motor problem imposing, however, high requirements on the accuracy with which the task is performed. The person tested was asked to find a position of the two joints of the upper limbs, such that the galvanometer needle whose position was a certain function of the angles in the joints, would achieve a zero position. The experimenter could simultaneously register the trajectory of the ray on the oscilloscope screen in the coordinate system of the joint angles. A change in one of the joint angles caused the ray to move along the horizontal, and a change in the other caused the ray to move along the vertical (the person tested did not see the screen). The experimenter could also change the coefficients, so that in the successive test runs (each took 10-60 sec) the desired position of the joints would be different. The function used had the form of a “boat” with one lowest point


165

Initially, all the subjects used only one method: they made the successive changes in the angles of the joints, moving the location of the ray step by step along the horizontal and the vertical closer to a given point. (A change in the position of the joint was made when the arrow, having passed through a minimum, again indicated that the required pose was not being maintained.) But later, along with this method of solution, another method was used that employed the organization of the function, even though for the subject it remained unknown (independently of the level of theoretical preparation). One could see on the screen how the subject, coming to the “bottom” would follow it toward the lowest point, without climbing the edge (this is possible only with a coordinated change in the angles at both joints). Only in the immediate neighborhood of the deepest point would he make the required pose more accurate by consecutively changing the angles at the joints. The tactics involved in the behavior of the subject were in this case close to the so-called ravine tactics [62]. If one agrees with the concepts presented here-namely, that the higher levels of the nervous system, developing the leading afferentation, control the operation of only a small number of muscle groups determining a given motion (or a synergy)-and if, in addition, one considers that this control is not affected directly but by retuning the interactions at lower levels, then it becomes clear that the basic “chores” involved in performing motion are done by these lower levels. The higher levels only form the functional synergies and retune the system of interactions among the elements in lower levels. Numerous papers on the simulation of pattern recognition state that the purpose of a model is to make possible the formation of a general picture which would reflect only the most essential features of a real object. In motor control we solve, one might say, the opposite problem: from an abstract representation we must construct a real motion with all its necessary details. The realization of a pattern in the form of a real motion requires its translation from the language using spatial and kinematic notions into the language of muscle dynamics : the motor composition, the number of motor units, the order of their spatial and temporal recruitment, and so on.

2 Functioning of Motor Control on the Spinal Level For further discussion, it will be convenient to distinguish the intermediate nerve structures, so that we can dwell in more detail on the functioning of the last effector link-the group of motoneurons and the interneuron structures related to it.

166


The principles underlying the operation of the intermediate nerve structures will be analyzed using the example of the spinal level of motor organization. The spinal level is characterized by an extremely large volume of developed afferentation and a large number of efferent outputs. Here the afferentation is received by the spinal level “first hand,” and the efferent commands are rapidly carried out. All the activity of the spinal level is influenced by the supraspinal actions organizing its operation with the objective of performing a given motion. The essential feature of the structure of this level is the presence of relatively autonomous subsystems which are also spatially separated. The presence of a number of inborn (or built-in) interactions is typical for these subsystems. The autonomy (even if it is relative) of the individual subsystems of the spinal level permits effective control within short time intervals-the decision does not have to go through a complex and long process of securing agreements. Obviously, the organization of a slightly more complex motion by no means reduces to just one such subsystem or several “linked” subsystems. For this reason, in our discussion of the organization of motion on the spinal level, the central role is played by interaction among the individual subsystems. When we speak of the possibility of the autonomous operation of individual subsystems, we come across an inescapable question : What particular “personal” goal is realized by any given subsystem in a specific motor problem? It seems probable to us that the goal, to a first approximation, is to lower the total impulsation received by the subsystem both from the periphery and from other subsystems and higher levels of the nervous system (“the principle of least interaction”). The total impulsation which serves as the estimate of the work done by a given subsystem is apparently developed by special mechanisms whose functioning depends on a given system of interaction. I t is natural to assume that the “habitual’ or “expected” afferentation makes a relatively small contribution to the total afferentation compared with the “nonhabitual” or “unexpected” afferentation. Perhaps a special role is played also by the “unbalanced” afferentation which contributes a great deal to the total afferentation. In other words, one could assume that the goal of a subsystem is to minimize the external stimuli which tend to disturb its present state and bring it into a new one. The tendency toward the minimization of the interaction leads to a coordinated functioning of the individual subsystems, subjecting the autonomous activity of each one of them to the interests of solving the total problem defined by the supraspinal afferentation. In this case the essential role played by the supraspinal influences consists


167

of a suitable reorganization of the interaction of individual subsystems of the spinal level [129].* Such a reorganization may be expressed in a motor effect. However, this is not obligatory. The reorganization may produce “readiness for movement,” but to perform the actual movement an additional stimulus is needed. It is possible that certain supraspinal influences (for example, related to the operation of the labyrinth system) usually use only this method to affect spinal activity. The activity of each relatively autonomous subsystem is, as we assumed, directed toward minimization of the total afferentation consisting of the proprioceptive afferentation and that afferentation which is external relative to a given subsystem from the neighboring subsystems of the same level and the supraspinal afferentation. A change in the functioning of a subsystem results first of all in a change of its afferentation and is directed toward minimization of the total afferentation. If the contribution of the subsystem’s afferentation to the total afferentation is large, then the role played by the remaining stimuli including the supraspinal afferentation stimuli, is decreased. This can be seen under the conditions created in special experiments [47, 133, 175, 1761, or in some situations arising during sports activities. Probably the phenomenon of a dominant elemcnt, discovered by Ukhtomskiy, can also be clearly understood from this point of view. The transfer of control over habitual movements to the lower levels (automatization of motor habits according to N. A. Bernshteyn) from this point of view is a result of the attempt by the higher levels to minimize their interaction with the lower levels. The individual details of motion are created and developed in the interaction among the subsystems forming the spinal level of motor control. If we simplify Wells’ classification a little [195], then in typical movements of man and higher animals one can distinguish muscles performing the basic active part of the motor task (they are relatively few in number), and muscles stabilizing the location of the preponderant mass of the kinetic links of the body (they are in the majority). One and the same muscle may in one motion be included in one group, and in another motion in another group.

* A natural mathematical model for research into control methods by specifying interactions is provided by games played by automata [158]. A computer simulation was made [33] of the behavior of a system consisting of a large number of automata, and the interactions between them were corrected by some upper limit. The problem of the upper limit was to develop an interaction that would result in a given behavior of the automata. The computer simulation showed that control of interaction can be achieved without obtaining complete correspondence between the behavior of the automata group and the interaction specified beforehand.

168


The functioning of the corresponding control nerve mechanisms is essentially autonomous, and the leading afferentation is peripheral ; the interaction of these mechanisms with the others is relatively small. This implies that such “stabilizing” modes of operation are typical in motion. The relatively minor role played by interaction under those conditions results in the “freedom” of motions, making it possible for the higher levels of the central nervous system not to worry about control over the corresponding spinal subsystems. Apparently, in certain of the simplest cases this type of stabilization is already realized on the segmentary level [80].If, however, the afferentation is not minimized at this lowest level, the higher levels come into play. The functioning of a large number of the stabilizing mechanisms assures the smoothness of motion without which the latter would become atactic. Of course, the regulation of active motion by no means reduces to the operation of stabilizing mechanisms. The interaction system determined by the higher levels should provide unbalanced functioning of the muscles responsible for the active part of motion, and almost always, a suitable variation of the angles at the joints. Although arbitrary motions have been little investigated, one can nevertheless state certain views which result from the above concepts. In particular, if the role of the higher levels of the nervous system is not to send direct commands, but rather to reorganize the interaction system (retune it) that governs the interaction among the nerve mechanisms of the spinal level, then, naturally, such a reorganization should take more time than is needed for the usual transfer of commands. It is well known that the latent time of a simple motor reaction is practically constant, and even systematic training does not lower it by more than 10-15 msec. Here out of 120-180 msec of the latent time, the communication time constitutes no more than 50-80 msec. Since arbitrary active motions for the majority of the nerve structures at the spinal level imply that stabilization mechanisms, required by a given task, must be included, such a reorganization should be roughly the same for a majority of the spinal mechanisms. Of course, a reorganization may change, and even substantially, during the course of the motion. However, a study of the retuning is simplest when the motion has not yet started. The phenomena related to preparation of the spinal nerve mechanisms for the motion will be called the pretuning, and we shall dwell upon them in more detail. One of the first observations which, as is now clear, indicates the presence of the pretuning was made by Hufschmidt [181]. He discovered that even up to 60 msec before the beginning of an arbitrary contraction of a muscle, a braking of the activity of the antagonist muscle takes place.


169

Our experiment went as follows. The subject was asked to perform a certain motion on a signal e.g., to bend his foot. For some time after the signal (but before the motion) a sinew reflex was produced and the amplitude of the electromiographic response was measured. It was found that the magnitude of the reflex depended substantially on how much time remained before the start of the motion. This dependence bears the same character for sinew reflexes and the monosynaptic H-reflex [I801on the corresponding muscle. It is interesting to note that for relatively large intervals before the onset of motio., ,70-50 msec), the variations in the amplitude of the spinal reflexes are approximately the same both for those muscles which must participate in the motion and for those which remain inactive. As the beginning of motion is approached, these variations are expressed much more sharply in exactly those muscles that will participate in the motion. Thus, apparently, the retuning of the spinal mechanisms has initially a diffused character, and as the start of motion is approached, the restructuring of the interaction system on the spinal level becomes localized. The possibility is not excluded, of course, that diffused changes in the state of the spinal control mechanisms are related to an orientating reaction occurring in the experiments in question. The experiments are described in more detail later (see p. 189). 3 Functioning of a Motoneuron Group and Motor Units

In this section we would like to discuss certain concepts related to the operation of the last effector link of the motor control system: the motoneuron group. Here we shall be interested in the mode of operation of the individual motoric units, so that attention will be focused on the functioning characteristics of the group, related to its atomicity, the latter being a general property of all nervous structures. The nerve cells themselves do not possess and cannot possess any kind of complex behavior: the information received by individual neurons is immeasurably smaller than that obtained by the entire organism, and their reactions are very stereotyped. Therefore, the central problem in neurophysiology is to study how the desirable behavior of the organism, interacting with a changing external medium, results from the interaction of various nerve structures, and ultimately from the behavior of individual neurons. In this connection, it is of great importance to search for those principles of neuron interaction which would assure that the entire physiological act is completed. The striving to minimize the interaction, which we already mentioned above, creates the possibility of a nonindividualized control over neurons in

I70


a given nerve structure by affecting their own interaction system. Of course, the influences on the system of neuron interaction may also include direct influences on the neurons. A nonindividualized influence on the neurons in a given center makes it possible to describe their functioning simply. The determining role of the autonomous collective operation means, in particular, that an important place in the solution of any physiological problem is occupied by the so-called “horizontal” interaction of neurons, which have been studied experimentally up to the present on only the simplest examples. These concepts are probably applicable to the organization of the functioning of both the motor and sensor systems. Suppose that there is a uniform medium made up of neurons connected to one another in such a way that each neuron, after excitation, has a relaxing influence on its neighbors. Then the system will be stable when all working neurons function synchronously. Their interaction will then be at a minimum. The applicability of this assertion to real nerve structures is difficult to verify directly, first, because of methodological difficulties, and second, because there is no basis for thinking that the interaction of neurons will only be positive and ~ymmetric.~ Among neurons in a homogeneous system, such as neurons in a single muscle, there must be some interaction simply because of the proximity of their electric fields, the overlapping of the regions of dendrite branching, and also in connection with the fact that each muscular receptor is projected not on one motoneuron, but on several, and each motoneuron receives impulsions from several muscle receptors. Various receptors exert an influence on the motoneurons either directly or through the interneurons. The impulsion of an individual muscle receptor depends on the activity of many motor units of the muscle. But once the mutual interaction of motoneurons exists, then if it is relaxing, the motoneurons should become excited synchronously. If, however, they function independently, this means that there is a special mechanism which prevents synchronization. An experimental investigation of the activity of individual motor units in a human muscle developing a moderate stress in the pose-mode showed that they function practically independently [54, 781. This is even more surprising, because the average frequency of the impulsion of all active motor units is approximately the same, about 7-1 1 /sec (operation at another A mathematical model of such a structure was discussed on p. 155. A physiological model may be provided by the sinus portion of the heart, all of whose elements function synchronously. Here, the leading element is one whose impulse transmission frequency is highest [57].


171

frequency is obviously unstable), and because the impulsion of an individual motor unit during several dozen cycles is very stable. (The ratio of the standard deviation to the average duration of the interval between the impulses is about 0.2-0.3.) The independent impulsion of various motor units of one muscle creates the principal possibility of individual control over the units. Under special conditions of artificial visual and auditory control over the activity of motor units, a person can in fact “turn on” or “turn off’ at will a given motor unit (strictly speaking, a group of motor units which contains the given unit) without changing the impulsion of another randomly chosen motor unit in the same muscle [54,1651. The independent activity of motor units makes it possible to understand the genesis of the physiological tremor, and predict the relationship between its amplitude and the stress produced, and its spectral content [53,1471. The actual mechanism producing asynchronous activity and which compares active motor units is unknown. However, there is information available about the properties of certain elements of the segmentary apparatus in the spinal chord and about the character of the connections between them [73, 90,99, 160, 162, 179, 182, 184, 186, 189, 193, 196). This information makes it possible to propose a model of this mechanism. The principal functioning of the models is discussed by Gel’fand el al. [54],a more detailed exposition is given by Kotov and Tsetlin [IOO]and on p. 172 of this book. Here we shall only note that the principal role in creating normal operating conditions in the muscular motor units is played, according to these model representations, by reverse braking, in particular, the dependence of the pulsation of the Renshaw cell on the frequency of its activation, and the hypothesis about the coordinated activation of the corresponding alpha- and gamma-motoneurons, all these in addition to the known properties of the motoneurons themselves. To explain certain pathological modes of operation of muscles (e.g., Parkinsonism and postpoliomyelitic paresis) in which the activity of motor units significantly deviates from the normal, it is sufficient within the framework of this model to assume certain quite specific and minor variations in the properties of the elements of the model. The investigation of the behavior of the model made it possible to distinguish certain essential parameters of the architecture of the segmentary apparatus and the properties of its elements, whose variation has a pronounced effect on the functioning of the system, and other parameters whose variation has little effect on the system. Such conclusions would probably be much harder to obtain directly from experiments.

0

Computer Simulation of the Functioning of a Motoneuron Group’ 0

Gel’fand ef af. [54] discuss the characteristic features of the active functioning of a muscle when maintaining a pose-the pose-oriented operational mode of the motor units with respect to their norm and pathology-and it formulates a number of hypotheses about the nerve mechanisms providing the asynchronous impulsion of the motor units with a relatively constant frequency. The data on the functioning of individual elements of a motoneuron group and muscles, which are available in the physiological literature [169, 170 178, 179, 182, 187, 188, 1901, allow us to formulate a number of hypotheses about the mutual relations and connections among these elements [54, 801. One of the natural methods of verifying the completeness and lack of contradiction in the accepted system of hypotheses is computer simulation of the group activity. In this chapter we present the results of such simulation. We also want to discuss the role of the individual parameters in explaining the features of normal and pathological operational modes of the motor units, and to estimate how critical certain parameters of the muscle and the motoneuron group are, the parameters being taken from the group of those that are either not measured in experiments, or are known only roughly. 1 Pose-Regime of Motor Unit Operation Let us recall the most essential information concerning the properties of a muscle and the motoneuron group that controls it. We know that a The article was written jointly with Yu. B. Kotov. Its final version was finished [I001 after M. L. Tsetlin’s death (Editor’s note).

I 72

Computer Simulation of the Functioning of a Motoneuron Group

173

muscle consists of motor units which are groups of muscle fibers innervated by a single motoneuron each. Motoneurons are relatively large nerve cells located in the front parts of the spinal cord. The impulse produced when a motoneuron is excited stimulates not only the motor unit of the corresponding muscle, but also the specific interneurons, the Renshaw cells. When excited, a Renshaw cell generates a series of impulses with gradually increasing intervals between the impulses (from 0.7 msec at the beginning to 3050 msec at the end of the series). The duration of the series depends on the rest time of a Renshaw cell after the preceding excitation. The impulses from the Renshaw cell have a braking effect on motoneurons. The probability of motoneuron excitation depends on the stimuli received by it from a number of brain sections and from the muscle receptors (spindles and sinews). The spindles (length receptors) have a monosynaptic relaxing effect on a motoneuron. Sinew Golgi2 receptors (receptors of muscle tension) disynaptically depress a motoneuron (lower the probability of excitation). In the pose-maintenance mode, each motor unit of a healthy muscle (and, consequently, the motoneuron innervating it) is capable of pulsating for a long time with the intervals between the impulses amounting to 70140 msec, where the possible range of the intervals between the impulses is 20-200 msec. Any two motor units transmit impulses practically independently of one another. For a moderate stress, only part of the motor units is transmitting impulses. An increase in the stress is achieved by recruiting new motor units and a slight increase in the frequency of individual motor units. As we know [54, 991, in the tremor form of Parkinson’s disease, the motor units transmit impulses in groups (2-4 impulses to a group) with the frequency of group follow-up amounting to 3-5/sec. The groups of various motor units are synchronized with one another, and are usually associated with a definite phase of the oscillations of the joint angle. Instead of the irregular tremor, invisible to the eye, which is characteristic of a healthy person, one observes oscillations of the limbs of great amplitude and close to sinusoidal. When a person who has had poliomelitis tries to contract his muscles, the first motor units are rapidly followed by the remaining ones. Almost all motor units transmit impulses synchronously with an increased frequency (up to 40/sec). Any attempt at longer maintenance of the stress in the damaged muscle fails.

a

C. Golgi, Italian histologist (Translator’s note).

I74


2 Description of the Model

We shall describe a formalization of the properties of a muscle and a ganglion that we have chosen for the model. 1. We assume that a muscle possesses elasticity k and the viscosity ,LA, and sets in motion a mass m (reduced mass of the movable link of the joint). Therefore, the dynamics of a loaded muscle are described by the differential equation mx + p i kx = F ( t ) - mg. (1)

+

Here x ( t ) is the length of the muscle (subtracting the minimum length), F ( t ) is the stress of the entire muscle, the coefficients m, p, k are chosen in accordance with data of physiological measurements of elasticity and the viscous properties of the muscles, and mg is the load on the muscle. The force F ( t ) produced by the entire muscle is a sum of the forces of the individual motor units. 2. The behavior of a motor unit is completely described by a single function of time, namely, the force f(t) produced in response t o the arrival of a n impulse from a motoneuron. Suppose that at time t* a motor unit receives a stimulating impulse from a motoneuron. Then the stressf(t) for the time t > t* is given by a positive function (Fig. 23a, curve 2), having one maximum and approaching zero for t + 00. Figure 23a (curve 1) also shows a curve obtained in a physiological experiment [ZOI].The function f ( t ) was selected in the following way:

In this formulaf(t*) is the stress of a motor unit at the time of impulse arrival and t , is a constant determining the position of the stress maximum. We also assume that the stress produced by a motor unit is limited from above by a constantf,. Therefore, the coefficient a should satisfy the additional condition at, < 1. The form of the curvef(t) was studied in a number of physiological experiments [IOI]and the values of the constantsf,, a, t , , and the time constant t in the model were chosen so that they would fit the curve closely (fo= 16 gm, a = 1/64, t , = 16 msec, t = 25 msec). Thus, the stress of a motor unit increases linearly; during ( t * , t* t,) it reaches a maximumf, = f ( t * ) at,(f, - f ( t * ) ) .

+

+

175


t" t " + f ,

-0./6 -O./

I

0

fol

$

40

2

20

F

c3

\

2

0

200

8!,msec

400

fcl Figure 23

3. In our model, a motoneuron is completely described by two functions ) the state function q ~ ~ ( t ) . ~ of time: the threshold P A t f ( tand The external stimuli change the function of state vM(t). The external stimulus acting on a motoneuron is called a facilitating stimulus if it increases vM(f), and inhibitory if it lowers it. In the absence of external stimuli ejLM(t) decreases exponentially in absolute magnitude with time constant t P M i.e., , it decreases as exp[- t/t,,]. We say that a motoneuron is excited at a time to if ejLM(to) 2 PM(to). At a time immediately following the time of excitation (it will be denoted by to E , with the understanding that E > 0 and is arbitrarily small), the threshold of the motoneuron receives a positive addition P*w+,and the state function becomes zero. Later, the threshold decreases exponentially with time constant t P M approaching , a constant PM,,i.e., for t > t o ,

+

The difference p ? ~ ( t ) &(t) of the state function and the threshold of motoneurons corresponds, within a constant, to a potential of the cell membrane. Thus, our description of the state of motoneurons corresponds to that accepted by electrophysiologists.

176


Upon excitation, the motoneuron generates an impulse which stimulates a motor unit with a lag I , . The values of t P M tp , M1,, are taken from physiological experiments, and P M + , Phl0 are selected in such a way as to obtain the best fit to the experimental curves of threshold variation. In our experiment, tI,M = 35 msec, teM = 4 msec, PAW+ = 1.5, PA4,,= 2, and t, = 6 msec. The number of motoneurons was chosen as 60. 4. It is assumed that to the state function of each motoneuron a constant y o is added, which is the same for all motoneurons. In addition, the state functions qAl(t)receive “white noise” [ ( t ) with zero mean value and deviation at2. The constants y o and ut2 were chosen in such a way that the probability of negative values of the sum ~ l , [ ( t ) would be small (in a majority of the versions y o = 0.5, at = 0.125). The values of [ ( t ) are independent for various motoneurons. In the computer program the time was considered discrete (1 cycle = 2 msec). Therefore, the “white noise” & t ) was simulated by adding to v.vl(f) in each cycle a random term with a normal distribution. 5. For each of the receptors for muscle length (spindles) a threshold is specified, i.e., the smallest length at which it begins to exert a facilitating influence on the motoneurons connected to it. With a further increase in the length, the positive addition to the state functions of the motoneurons increases proportionally to the difference between the length of the muscle and the given spindle threshold (the proportionality coefficient is 4 cm-I). The spindle thresholds were chosen to be different. The threshold distribution determines the dependence of the active stress of a muscle on its extension. We have selected a distribution (Fig. 23b) that results in an increase of rigidity when the muscle is extended. In some experiments with the model, the spindle thresholds were given an identical additive term which varied with time (a model of the y influence). The model contains 20 spindles. Each spindle stimulates 15 motoneurons, i.e., each motoneuron receives stimulation from five spindles. This system of connections makes it possible to slightly lower the effect of the instantaneous differences in the spindle thresholds. A similar smoothing process apparently takes place in the spinal cord with the participation of the afferents of Group I1 from the spindles. 6. For the tension receptors in the muscle (sinew Golgi receptors) we are given a threshold F, > 0, which is the tension in the muscle at which the sinew receptors begin to exert an inhibitory effect on all motoneurons. The inhibitory effects are considered to be proportional to the difference between the tension F ( t ) and the threshold F,. The proportionality coeffi-

+

I77


cient al and the threshold tension F, in a majority of experiments are equal to a1 = (1 / 128) gm-' ; Fo = 16 gm, respectively. 7. In our model, the state of a Renshaw cell is completely described by three functions of time: the state function Q ? R ( t ) , the threshold P H ( t ) ,and the time GR(t) passed from the last excitation. We say that a Renshaw cell becomes excited at a time to, if qR(to0) 2 PR(too).At a time to, E immediately after the time of excitation, the threshold increases its value by a positive number P R + , and & becomes zero. Then the threshold decreases exponentially with time constant t P Rap, proaching a positive constant P R O i.e., , for t > t,,,

+

P R ( f ) = [PR(tOO)

- PRO

+

PR+l

exp[-(t - fOO)/SPR1

+

(4)

A Renshaw cell receives impulses from 14 motoneurons. Each impulse from a motoneuron increases rpR(t) by one. In the intervals between the impulses from the motoneurons, F R ( f ) decreases exponentially with time constant zpRapproaching zero. At an excitation time, a Renshaw cell begins to generate a series of inhibitory impulses. In the program, the series from a Renshaw cell is simulated by means of a negative (inhibitory) additive term to the state functions rpM(t) corresponding to the motoneurons. The absolute value of the term at the time of the excitation of a Renshaw cell is equal to a constant g o , and then decreases linearly to zero during the series. The duration of the series f T is a function of the rest time 8Rof a Renshaw cell after the preceding excitation. The form of the function t T ( 0 R ) is shown in Fig. 23c. On the axis of abscissas we plot the time GR that has passed from the time of the previous excitation, and on the ordinate axis, the duration tT of the inhibitory series of a Renshaw cell. In the program, the function tT(&) for aR5 500 msec was approximated by a cubic polynomial. For & > 500 msec, tR(&) = f T (500 msec). The curve (Fig. 23c) does not contradict the physiological data available [169, 170, 1791. In a majority of experiments with the model, tpR = 25 msec, tpR = 25 msec, P R O = 17/16, PR+ = 3, and go = The number of Renshaw cells was chosen to be equal to 26. The connections of the Renshaw cells with the motoneurons were random, but such that each Renshaw cell, upon excitation, inhibits all the motoneurons that could excite it. In addition, the inhibitory series from a Renshaw cell is also received by 9 motoneurons which cannot excite it. With such a system of connections, each motoneuron is inhibited by a group of Renshaw cells that were excited by its impulses, and from some Renshaw cells which it cannot stimulate.

a.

178


3 Desynchronization of Motoneurons; the Influence of the Renshaw Cell System on Impulsion of Motoneurons

The synchronous impulsion of two motoneurons is defined as a n impulsion mode in which the probability of excitation of the second motoneuron attains a maximum after some constant time from the excitation of the first. This time is called the time lag. A natural measure of the synchronization of two motoneurons (with indices i, j) would be the deviation of the distribution of the intervals 6, between the times of excitation of these motoneurons. Using this characteristic, one could apparently combine all the motoneurons in a ganglion into several independent synchronous groups. The number of such groups and the number of their elements are characteristics of the ganglion-muscle system, which are analogous t o the number of the degrees of freedom in mechanics. However, the use of the deviation of the distribution of Gij as a measure of the synchronization requires a measurement of a considerable number of intervals, i.e., it requires very long computer experiments. The most interesting for our model is the case in which the time lags between the synchronized motoneurons are small compared with the intervals between the excitations of the same motoneuron, and when all motoneurons in a ganglion form one more o r less synchronous group. In our model, the synchronizing stimulus for a motoneuron is provided by a signal from a sinew receptor which is identical for all motoneurons and received by them simultaneously. The lag is absent, and all motoneurons enter one synchronous group; thus the impulses from various motoneurons should come in “bunches” with large intervals between them. The intervals between the successive times of excitation of various motoneurons can be divided in two categories; short intervals between the impulses from various motoneurons inside the bunch and long intervals between the bunches. Therefore, the density of the interval distribution has a large value only in regions of very short and very long intervals. This type of distribution possesses [48] a significant excess

which was also chosen as a measure of synchronization. In this formula p i is the central moment of ith order of the interval distribution. In describing the results of computer experiments, we shall also state in each of the examples what is the largest deviation a of the length of the muscle from the mean in a stationary mode. This characteristic also describes the degree of synchronization of the motoneurons in a ganglion. However, it is less sen-


179

sitive to low synchronization. For a majority of simulation experiments, values of e < 3, 2a < 0.1 cm correspond to the normal asynchronous impulsion. The synchronizing stimuli on the motoneurons in the spinal cord are, apparently, very numerous. Therefore, to obtain normal asynchronous impulsion of motoneurons, we need a special desynchronizing mechanism. This mechanism, according to our ideas, is provided by the system of Renshaw cells. Suppose that a ganglion contains a group of synchronized motoneurons. The white noise, acting on pM (see point 4) causes a nonsimultaneous excitation of motoneurons in this group, other conditions being equal. As we already mentioned, each Renshaw cell, upon excitation, sends an inhibitory series to approximately one third of all motoneurons in the ganglion. The excitation of any motoneuron may cause excitation of several Renshaw cells that are in states sufficiently close to the threshold, and excitation of the majority of motoneurons immediately after a given one (and during the entire series from a Renshaw cell) is very likely. The inhibitory impulses from a Renshaw cell, upon becoming excited by the first impulses from the motoneurons in a synchronous group, make the excitation of the remaining motoneurons more difficult (the remaining ones constitute a major part of a synchronous group). Thus, the number of motoneurons belonging to a synchronous group decreases, i.e., a desynchronization takes place. With the parameters used in our model, a breakdown of the group of synchronous impulse transmitting motoneurons occurs during a time corresponding to 2-3 impulses (Fig. 24a). In this figure on the axis of abscissas we plot the time t, and on the axis of ordinates, the number N,,(t) of the motoneurons excited during 20 msec beginning with a time 1. The point R corresponds to the time when the Renshaw cells, which were turned off until that instant, begin to function. Figure 24b also illustrates the sharp damping of the oscillations in the length of the muscle x ( t ) when the Renshaw cells become activated. If, without changing the system of connections and parameters of Renshaw cells, one deactivates a certain portion of randomly chosen Renshaw cells, then certain motoneurons will receive less inhibitory impulses from Renshaw cells, and may be capable of combining into a synchronous group. The number of motoneurons in this group and the degree of their synchronization are greater, the fewer Renshaw cells remain. The results of the corresponding computer experiment are given in Table XXIII. It can be seen that halving the number of Renshaw cells

180


R

-0.5Figure 24

results in an increase of the amplitude a of the muscle length oscillations by approximately three times. The degree of synchronization e also increases. The lowering of the number of motoneurons sending impulses to one Renshaw cell lowers the effectiveness of desynchronization, since this lowers the probability of excitation of even one Renshaw cell by an impulse from some motoneuron. The reduction of the number of motoneurons inhibited by one Renshaw cell also leads to impairment of desynchronization except that the impairment is less pronounced due to a significant overlapping of the outputs from Renshaw cells on the motoneurons. The results of the corresponding computer experiment are shown in Table XXIV. The TABLE XXm No. of Renshaw cells

26

13

7

0

e

0.18

7.9

32

48

2a

0.045

0.15

0.5

1.9


I81

data of Table XXIV show that the effectiveness of desynchronization, when the number of inputs to a Renshaw cell is halved, is reduced to a larger extent than when the number of outputs from a Renshaw cell is halved. A simultaneous lowering of the number of inputs and outputs leads to an even sharper impairment of desynchronization. TABLE XXIV Connections=

1 M N + 6 RC 8 RC --f 1 MN

1

1 MN+3RC 4 RC + 1 MN a

I

e

I

0.18

1 1 23

2a 0.045

0.5

MN: motoneutrons, RC: Renshaw cel

Renshaw cells apparently participate in the stabilization of the frequency of motoneuron impulse transmission. This assertion is based on the fact that the portion of time during which a motoneuron experiences an inhibitory influence of a Renshaw cell, ~ ( 8=~fT(8R)/8R, ) increases sharply for intervals between the excitations of Renshaw cells lasting less than 100 msec (Fig. 25a). In addition, Renshaw cells inhibit not only a group of active motoneurons, but also those 6eldom active, rarely impulsing motoneurons, which action becomes even less frequent. Histograms of the intervals for various active motoneurons4 are very similar to each other. For this reason, to lessen the machine time, we used their sum as an indicator of the interval stability. A change in the form of the characteristic tr(8R) of a Renshaw cell results in a change in the position of the maximum and in the deviation of the inThe active motoneurons are those motoneurons which transmit impulses at great frequency and account en masse for one-half of all impulses.

182

2


0.2

F*

40

\

i -

\

ssp

‘4

q

20

\

2-

0

o./ 0

100 200 300 400 S,, msec

100 200 t, msec

f 6)

fa) Figure 25

terval distribution (see Fig. 25). Figure 25 shows two different characteristics tT1(&) and t772(OR),and their corresponding histograms of intervals n l ( t ) and nz(t).For comparison, we show a histogram n,(t) constructed from the data of Gel’fand et al. [MI. In the majority of computer experiments, we used the characteristic tT, (see Fig. 25). The deviation of the interval duration in computer experiments, even for the most satisfactory versions of the characteristics of Renshaw cells, is greater than the interval deviation, as measured in a physiological experiment. Apparently, there are still other mechanisms of the interval length stabilization which we have not considered.

4 Control of Muscle Length Using the program just described, we simulated a simple motor problem -that of changing the length of a muscle. The length of a muscle, as already mentioned, depends on its load, its elastic and viscous properties, and the active tension it develops, i.e., ultimately on tue facilitating stimuli received by motoneurons. One could, for example, control the length of a muscle by changing the stimuli received by motoneurons from other sections of the spinal cord. In this type of control situation, the length of the muscle will depend strongly on the load, and in a number of cases may be obtained only within a large error. To us it seems more natural to assume another method of control-that related to changing spindle thresholds. This method makes it possible to achieve a more accurate fit between the actual length of the muscle and the desired length. To control the length of a muscle through spindles, we introduced an additional link which produces a term


183

d ( t ) that is added to the thresholds of all spindles, and depends on the difference between the actual length of the muscle x(r) and the desired length

x,(t):

Here the coefficients a,, y , the time lag t , , and the time constant t5 are constants whose values are selected empirically (aZ= Q, y = 1/64, t , = 20 msec, t 5= 256 msec). The required length of the muscle x,(t) was given in the program. A stepwise change of the required length xo(t)caused a subsequent change in the length of the muscle x(t). The transient time for the new length x ( t ) and the character of the transient mode are practically identical when xo(t) is either increased or decreased. In the figure, on the abscissa axis we plot the time measured from the time of the stepwise change in -\-,(I) from the starting value of xo = 0.5 cm to $0.25 cm (xol)or to -0.25 cm (xO2).On the axis of the ordinates, we plot the values of x(t), the length of the muscle, and the value of the corrective term 6(t). The curves s, and 6, show the variation of the coordinate and the correction term for x o ( t ) = x,; the curves xz,6, show the same quantities for xo(t) = xO2. An instantaneous change in the load on a muscle also causes a deflection of s ( t ) from xo(t).Adding a load and subtracting a load on the muscle cause different transitional processes; both differ .in character and duration (Fig. 26b). In this figure, on the abscissa axis, we plotted the time measured from the time of adding (subtracting) a load. On the ordinate axis, we plotted the length of the muscle x(t) and the correction constant 6 ( t ) . The curves x, and 6, correspond to a variation of the muscle length and the correction term after adding a load (an increase of m from 30 to 60 gm); x2 and 6, correspond to the same quantities after subtracting a load (a decrease in m from 30 to 15 gm). The plot corresponding to the subtraction of the load exhibits a characteristic feature. A large surge at the origin and a “step” afterward result from the action of a reflex on the extension. The subsequent drop in the curve is caused by an additional feedback mechanism. It is interesting to note that similar curves of the transient process were obtained in the experiments of G.A. Arutyunyan. With a sharp decrease in the weight of a pistol (unloading) in the hand of a marksman, the aim point follows a trajectory coincident with the curve of Fig. 26b. Certain angles at the joints vary in a similar fashion. When adding to a load, the action of the corrective mechanism is masked by the slowing of the motion because of the larger mass.

184


0.2

xoz

--

- 0.8

0

'. -

\.-*.----. - -0.2 ----_ , ---

100 200 300 t, msec

fa/

0.3

- 0.2 0

100 ZOO 300

t, msec fbl

Figure 26

5 Simulation of Pathological States In simulating the impulsion mode of a motoneuron in a person having Parkinson's disease (tremor form), we started with the assumption [54] that the thresholds of Renshaw cells in this person are increased. The results of the experiment with the model are shown in Table XXV. The maximum amplitude of oscillations of the muscle length a and the maximum synchronization correspond to the case PI a 2 , then

for any initial states p(n) and y ( n ) of automata A , . It is this basic property that allows us to determine the payoff in a zero-sum game. First, let us consider an example. Suppose that a game is played by identical automata, and the matrix of this zero-sum game is

In the play (1, 1) the payoff to the first automaton, (0.3), is greater than that to the second automaton (-0.3). With this type of payoff in a stationary random medium, the mean time before a change of action by the second automaton is considerably less than the mean time before a change of action by the first automaton. Therefore, it is natural also to expect in a game with this type of payoff that the second automaton will change its action sooner than the first, and the automata will go from play (1, 1) to play (1, 2). In play (1, 2), the payoff to the first automaton is already less than that to the second. Now the first automaton will change its action sooner, and the automata will change to play (2, 2). Subsequently, the change of actions will take place in the following manner: 0.3 + -0.1

t

-0.4

t

1

-0.2.

The mean time t i j ,during which automata participate in a play (i, j ) , is determined by the automaton that receives a smaller payoff in the play Therefore, t,,, TA(-O, 3), t1,2 TA(--O.1), tZ,, T,(-0.4), t 2 , 2'T (-0.2). Then ( I ) implies that play (1, 2) is played much longer than the remaining ones, and thus this play makes the greatest contribution to the payoff. Naturally, we can expect that the payoff to the first automaton in such a game is close to -0.1. (This is the matrix element of the payoff matrix that is closest to zero.) The above qualitative picture of the behavior of automata turns out to be not far from the truth. Krinskiy [ Z W ] describes a class S of automata for which it is possible to determine the payoff in zero-sum games. These automata satisfy condition (1) and two additional requirements. The limiting behavior of those automata in a game can be determined in terms of the

-

-

-

-

249

Games Played by Asymptotically Optimal Automata

automaton value d. The automaton value for a pair of asymptotically optimal sequences of automata A , , . . . , A , , . . . and B,, . . . , B,, . . . from this class is determined as follows. The mean time TB,(a) for an automaton B for sufficiently large n is a monotonically increasing function of a, and Ts,(- a ) is a monotonically decreasing function of a. Suppose that for a = d(n), we have the following equality:

The limit d(n) for n-00 (if it exists) is called the automaton value d. The automaton value d is greater, the smaller the “inertia” of the design of A, i.e., the smaller the time TA(a) and the more inertial its opponent B. For identical automata, d = 0. We have the following theorem: Suppose we are given two sequences of automata A , , . . . , A , , . . . , and B,, . . . , B,, . . . , belonging to class S and asymptotically optimal in all stationary random media, and suppose we are given a zero-sum game between two automata A, and B,. Then the limiting payoff W to automaton A always lies between the upper and lower values of the game V and u. Strictly speaking:

r

r,

(a) If a game T is such that the automaton value d 2 V , then W = V. (b) If u < d < V , then the limiting payoff W is close to d. The closeness is understood in the following sense:

ailjl 5 W I ai,aj,, where

ailjl = max{aij}, aij

Automaton theory and modeling of biological systems

Cellular Automaton Modeling of Biological Pattern Formation

Modeling Biological Systems:: Principles and Applications

Modeling biological systems: Principles and applications

Thriving Systems Theory and Metaphor-Driven Modeling

Biological Database Modeling

Biological Database Modeling

Mathematical Modeling of Complex Biological Systems: A Kinetic Theory Approach (Modeling and Simulation in Science, Engineering and Technology)

Computational Modeling of Biological Systems: From Molecules to Pathways

Mathematical Modeling of Complex Biological Systems: A Kinetic Theory Approach (Modeling and Simulation in Science, Engineering and Technology)

Mathematical Modeling of Complex Biological Systems: A Kinetic Theory Approach (Modeling and Simulation in Science, Engineering and Technology)

Modeling of dynamic systems

Petri Net theory and the modeling of systems

Information Processing and Biological Systems

Electrochemical studies of biological systems

Dynamic Models and Control of Biological Systems

Biological invasions: theory and practice

Modeling, Control and Coordination of Helicopter Systems

Manufacturing Systems Modeling and Analysis

NMR IN BIOLOGICAL SYSTEMS

Multiscale Modeling of Developmental Systems

Modeling and Simulation of Capsules and Biological Cells

Towards a Mathematical Theory of Complex Biological Systems

Modeling and Verification of Real-time Systems

Modeling and Control of Engineering Systems

Fundamentals of Basin and Petroleum Systems Modeling

Modeling and Simulation of Dynamic Systems

Modeling and Simulation of Capsules and Biological Cells

Automated Modeling of Physical Systems

Dynamic Modeling and Control of Engineering Systems

Fundamentals of basin and petroleum systems modeling

Automaton theory and modeling of biological systems

Cellular Automaton Modeling of Biological Pattern Formation

Modeling Biological Systems:: Principles and Applications

Modeling biological systems: Principles and applications

Thriving Systems Theory and Metaphor-Driven Modeling

Biological Database Modeling

Biological Database Modeling

Mathematical Modeling of Complex Biological Systems: A Kinetic Theory Approach (Modeling and Simulation in Science, Engineering and Technology)

Computational Modeling of Biological Systems: From Molecules to Pathways

Mathematical Modeling of Complex Biological Systems: A Kinetic Theory Approach (Modeling and Simulation in Science, Engineering and Technology)

Mathematical Modeling of Complex Biological Systems: A Kinetic Theory Approach (Modeling and Simulation in Science, Engineering and Technology)

Modeling of dynamic systems

Petri Net theory and the modeling of systems

Information Processing and Biological Systems

Electrochemical studies of biological systems

Dynamic Models and Control of Biological Systems

Biological invasions: theory and practice

Modeling, Control and Coordination of Helicopter Systems

Manufacturing Systems Modeling and Analysis

NMR IN BIOLOGICAL SYSTEMS

Multiscale Modeling of Developmental Systems

Modeling and Simulation of Capsules and Biological Cells

Towards a Mathematical Theory of Complex Biological Systems

Modeling and Verification of Real-time Systems

Modeling and Control of Engineering Systems

Fundamentals of Basin and Petroleum Systems Modeling

Modeling and Simulation of Dynamic Systems

Modeling and Simulation of Capsules and Biological Cells

Automated Modeling of Physical Systems

Dynamic Modeling and Control of Engineering Systems

Fundamentals of basin and petroleum systems modeling

Recommend Documents