Multidimensional Neural Networks Unified Theory

This page intentionally left blank Copyright © 2008, New Age International (P) Ltd., Publishers Published by New Age ...

Author: Rama Murthy | G

239 downloads 2336 Views 1MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

This page intentionally left blank

Copyright © 2008, New Age International (P) Ltd., Publishers Published by New Age International (P) Ltd., Publishers All rights reserved. No part of this ebook may be reproduced in any form, by photostat, microfilm, xerography, or any other means, or incorporated into any information retrieval system, electronic or mechanical, without the written permission of the publisher. All inquiries should be emailed to [email protected]

ISBN (13) : 978-81-224-2629-8

PUBLISHING FOR ONE WORLD

NEW AGE INTERNATIONAL (P) LIMITED, PUBLISHERS 4835/24, Ansari Road, Daryaganj, New Delhi - 110002 Visit us at www.newagepublishers.com

Dedicated to the memory of JESUS CHRIST and SARASWATI


Preface This book deals with a novel paradigm of neural networks, called multidimensional neural networks. It also provides comprehensive description of a certain unified theory of control, communication and computation. This book can serve as a textbook for an advanced course on neural networks or computational intelligence/cybernetics. Both senior undergraduate and graduate students can get benefit from such a course. It can also serve as a reference book for practicising engineers utilizing neural networks. Further more, the book can be used as a research monograph by neural network researchers. In the field of electrical engineering, researchers have innovated sub-fields such as control theory, communication theory and computation theory. Concepts such as logic gates, error correcting codes and optimal control vectors arise in the computation, communication and control theories respectively. In one dimensional systems, the concept of error correcting codes, logic gates are related to neural networks. The author, in his research efforts showed that the optimal control vectors (associated with a one dimensional linear system) constitute the stable states of a neural network. Thus unified theory is discovered and formalized in one dimensional systems. Questioning the possibility of logic gates operating on higher dimensional arrays resulted in the discovery as well as formalisation of the research area of multi/infinite dimensional logic theory. The author has generalised the known relationship between one dimensional logic theory and one dimensional neural networks to multiple dimensions. He has also generalised the relationship between one dimensional neural networks and error correcting codes to multidimensions (using generator tensor). On the way to unification in multidimensional systems the author has discovered and formalised the concept of tensor state space representation of certain multidimensional linear systems. It is well accepted that the area of complex valued neural networks is a very promising research area. The author has proposed a novel activation function called the complex signum function. This function has enabled proposing a complex valued neural associative memory on the complex hypercube. He also proposed novel models of neuron (such as linear filter model of synapse). This book contains 10 chapters. The first chapter provides an introduction to the unified theory of control, communication and computation. Chapter 2 introduces a mathematical

(viii)

Preface

model of multidimensional neural networks and the associated convergence theorem. In Chapter 3, the concepts of multidimensional error correcting codes, multidimensional neural networks and optimization of multi-variate polynomials (associated with a tensor) over various subsets of multidimensional lattice, are related from different view points. In Chapter 4, Tensor State Space Representation (TSSR) of certain multidimensional linear systems is discussed. In Chapter 5, Unified Theory of Control, Communication and Computation in multidimensional linear systems is summarized. In chapter 6, the author proposes a novel complex signum function. In Chapter 7, a novel optimal filtering problem associated with a one dimensional linear system is formulated and solved. In Chapter 8, a linear filter model of synapse is proposed. Also a novel continuous time associative memory and the associated convergence theorem are discussed. In Chapter 9, a novel model of neuron and associated real/complex neural networks are proposed. Finally in Chapter 10, advanced theory of evolution based on the unified theory is briefly discussed. The Chapters in this book are organised in such a way that there is considerable flexibility in its use by its readers. For instance, Chapters 1 to 5 can form the basis for a graduate course on multidimensional neural networks and unified theory. This course is a compulsory course for students interested in doing research on computational intelligence (cybernetics). The students/researchers interested in doing research on complex valued neural networks will find interesting material in Chapters 6 and 9. Further, the students/ researchers interested in exploring interrelationship between signal processing and neural networks will enjoy understanding the material in Chapters 7 and 8. Finally, Chapter 10 will provide counter-intuitive insights into the theory of organic evolution. This writing project would not be possible without the cooperation of my brother Dr. G.V.S.R. Prasad and my beloved mother. I thank many colleagues at IIIT and those around the world who believe that this book is my first masterpiece. I specially thank Sri Damodaran and other employees of New Age International (P) Ltd. for making my dream of publishing this book a reality. G. Rama Murthy

Contents PREFACE 1.

INTRODUCTION

L OGICAL BASIS FOR COMPUTATION L OGICAL BASIS FOR CONTROL L OGICAL BASIS OF COMMUNICATION A DVANCED THEORY OF EVOLUTION 2.

MULTI/INFINITE DIMENSIONAL NEURAL NETWORKS, MULTI/INFINITE DIMENSIONAL LOGIC THEORY

2.1. 2.2. 2.3. 2.4. 2.5. 2.6. 2.7. 3.

INTRODUCTION MATHEMATICAL MODEL OF MULTIDIMENSIONAL NEURAL NETWORKS CONVERGENCE THEOREM FOR MULTIDIMENSIONAL NEURAL NETWORKS MULTIDIMENSIONAL LOGIC THEORY, LOGIC SYNTHESIS INFINITE DIMENSIONAL LOGIC THEORY: INFINITE DIMENSIONAL LOGIC SYNTHESIS N EURAL NETWORKS, LOGIC THEORIES, CONSTRAINED STATIC OPTIMIZATION CONCLUSIONS

MULTI/INFINITE DIMENSIONAL CODING THEORY: MULTI/INFINITE DIMENSIONAL NEURAL NETWORKS—CONSTRAINED STATIC OPTIMIZATION

3.1. 3.2. 3.3. 3.4.

INTRODUCTION MULTIDIMENSIONAL NEURAL NETWORKS: MINIMUM CUT COMPUTATION IN THE CONNECTION STRUCTURE: GRAPHOID CODES MULTIDIMENSIONAL ERROR CORRECTING CODES: ASSOCIATED ENERGY FUNCTIONS—GENERALIZED NEURAL NETWORKS MULTIDIMENSIONAL ERROR CORRECTING CODES: RELATIONSHIP TO STABLE STATES OF ENERGY FUNCTIONS

3.5. 3.6. 3.7. 3.8.

N ON-BINARY LINEAR CODES N ON-LINEAR CODES CONSTRAINED STATIC OPTIMIZATION CONCLUSIONS

(vii) 1

3 3 4 6

9

9 11 14 17 20 23 25

27

27 29 34 39 42 45 53 59

(x) 4.

Contents TENSOR STATE SPACE REPRESENTATION: MULTIDIMENSIONAL SYSTEMS

61

4.1. 4.2.

61

4.3. 4.4.

INTRODUCTION STATE OF THE ART IN MULTI/ INFINITE DIMENSIONAL STATIC/ DYNAMIC SYSTEM THEORY: REPRESENTATION BY TENSOR LINEAR OPERATOR STATE SPACE REPRESENTATION OF CERTAIN MULTI/ INFINITE DIMENSIONAL DYNAMICAL SYSTEMS: TENSOR LINEAR OPERATOR MULTI/ INFINITE DIMENSIONAL SYSTEM THEORY: LINEAR DYNAMICAL SYSTEMS – STATE SPACE REPRESENTATION BY TENSOR LINEAR OPERATORS

4.5. 4.6. 4.7. 5.

STOCHASTIC DYNAMICAL SYSTEMS DISTRIBUTED DYNAMICAL SYSTEMS CONCLUSIONS

UNIFIED THEORY OF CONTROL, COMMUNICATION AND COMPUTATION: MULTIDIMENSIONAL NEURAL NETWORKS

5.1. 5.2. 5.3. 5.4.

INTRODUCTION ONE DIMENSIONAL LOGIC FUNCTIONS, CODEWORD VECTORS, OPTIMAL CONTROL VECTORS: ONE DIMENSIONAL NEURAL NETWORKS OPTIMAL CONTROL TENSORS: MULTIDIMENSIONAL NEURAL NETWORKS MULTIDIMENSIONAL SYSTEMS: OPTIMAL CONTROL TENSORS, CODEWORD TENSORS AND SWITCHING FUNCTION TENSORS

5.5 6.

COMPLEX VALUED NEURAL ASSOCIATIVE MEMORY ON THE COMPLEX HYPERCUBE

6.1. 6.2. 6.3 6.4. 7.

8.

CONCLUSIONS

INTRODUCTION FEATURES OF THE PROPOSED MODEL CONVERGENCE THEOREMS CONCLUSIONS

63 65 69 70 73 76

79

79 80 82 90 92 95

95 96 97 105

OPTIMAL BINARY FILTERS: NEURAL NETWORKS

107

7.1. 7.2. 7.3. 7.4.

107 107 113 114

INTRODUCTION OPTIMAL SIGNAL DESIGN PROBLEM: SOLUTION OPTIMAL FILTER DESIGN PROBLEM: SOLUTION (DUAL OF SIGNAL DESIGN PROBLEM) CONCLUSIONS

LINEAR FILTER MODEL OF A SYNAPSE: ASSOCIATED NOVEL REAL/COMPLEX VALUED NEURAL NETWORKS

117

8.1. 8.2. 8.3 8.4.

117 118 120 121

INTRODUCTION C ONTINUOUS TIME PERCEPTRON AND GENERALIZATIONS A BSTRACT MATHEMATICAL STRUCTURE OF NEURONAL MODELS FINITE IMPULSE RESPONSE MODEL OF SYNAPSES: NEURAL NETWORKS

Contents

8.5. 8.6. 8.7. 8.8. 9.

10.

(xi)

N OVEL CONTINUOUS TIME ASSOCIATIVE MEMORY MULTIDIMENSIONAL GENERALIZATIONS G ENERALIZATION TO COMPLEX VALUED NEURAL NETWORKS (CVNNS) CONCLUSIONS

122 125 125 126

NOVEL COMPLEX VALUED NEURAL NETWORKS

129

9.1. 9.2 9.3. 9.4. 9.5. 9.6. 9.7. 9.8.

129 130 133 133 134 135 135 136

INTRODUCTION DISCRETE FOURIER TRANSFORM: SOME COMPLEX VALUED NEURAL NETWORKS COMPLEX VALUED PERCEPTRON N OVEL MODEL OF A NEURON: ASSOCIATED NEURAL NETWORKS CONTINUOUS TIME PERCEPTRON LEARNING LAW SOME IMPORTANT GENERALIZATIONS SOME OPEN QUESTIONS CONCLUSIONS

ADVANCED THEORY OF EVOLUTION OF LIVING SYSTEMS

137

10.1. 10.2. 10.3. 10.4.

137 137 138 139

UNIFIED THEORY: CYBERNETICS ORGANIC EVOLUTION EVOLUTION OF LIVING SYSTEMS: INNOVATIVE PRINCIPLES CONCLUSIONS

INDEX

141


CHAPTER

1

Introduction

Ever since the dawn of civilization, the homo-sapien animal unlike other lower level animals was constantly creating tools that enabled the community to not only take advantage of the physical universe but also develop a better understanding of the physical reality through the discovery of underlying physical laws. The homo-sapien, like other lower level animals had two primary necessities: metabolism and reproduction. But, more important was the obsession with other developed necessities such as art, painting, music and sculpture. These necessities naturally lead to the habit of concentration. This most important habit enabled him to develop abstract tools utilized to study nature in most advanced civilizations. Thus the homo-sapien animal achieved the distinction of being a higher animal compared to the other animals in nature. In ancient Greece, the homo-sapien civilization was highly advanced in many matters compared to all other civilizations. Such a lead was symbolized by the development of mathematics subject in various important stages. The most significant indication of such development is left to posterity in the form of 13 books called, Euclid’s Elements. These books provide the first documented effort of axiomatic development of a mathematical structure such as the Euclidean geometry. Also, Greek, Babylonian civilizations made important strides in algebra: solving linear, quadratic equations and studying the quadratic homogeneous forms in two variables (for conic sections). Algebra was revived during the Renaissance in Italy. In algebra, solution of cubic, quartic equations was carried out by the Italian algebraists. This constituted the intellectual heritage, cultural heritage along with religious, social traditions. To satisfy the curiosity of observing the heavens, various star constellations, astronomical objects were classified. In navigating the ships for battle purposes as well as trade, astronomical observations were made. These provided the first curious data related to the natural world. In an effort to understand the non-living material universe, homosapiens have devised various tools: measuring equipment, experimental equipment, mathematical procedures, mathematical tools etc.

2

Multidimensional Neural Networks: Unified Theory

With the discovery that Sun is the center of our relative motion system by Copernicus, Ptolemaic theory was permanently forsaken. It gave Galileo, the curious motivation for deriving the empirical laws of far flung significance in natural philosophy/natural science/physics. Kepler after strenuous efforts derived the laws of planetary motion leading to some of the laws of Newton. Issac Newton formalized the laws of Galileo by developing calculus. He also developed a theory of gravitation based on the empirical laws of Kepler. Michael Faraday derived the empirical laws of electric and magnetic phenomena. Though Newton’s mechanical laws were successfully utilized to explain heat phenomenon, kinetic theory of gases as being due to mechanical motion of molecules, atoms, they were inadequate for electrical phenomena. Maxwell formalized Faraday’s laws of electro-magnetic induction leading to his field equations. Later physics developed at a feverish pace. These results in physics were paralleled by developments in other related areas such as chemistry, biology etc. Thus, the early efforts of homo-sapiens matured into a clearer view of the non-living world. The above description summarizes the pre 20th century development of this progress on homo-sapien contributions to understanding the nonliving material universe. In making conclusive statements on the origin and evolution of physical reality, the developments of the 20th century are more important. In that endeavor, Einstein’s general theory of relativity was one of the most important cornerstones of 20th century physics. It enabled him to develop a general, more correct theory of gravitation, outdating the Newtonian theory. It showed that gravitation is due to curvature of spacetime continuum. The general theory of relativity also showed that all natural physical laws are invariant under non-linear transformations. This result was a significant improvement over special theory of relativity, where he showed that all natural physical laws are invariant under linear Lorentz transformations. This result (in special theory of relativity) was achieved when Einstein realized that due to finiteness of velocity of light, one must discard the notions of absolute space and time. They must be replaced by the notions of space-time continuum i.e. space and time are not independent of one another, but are dependent. Thus, special and general theories of relativity constrained the form of natural physical law. In the 20th century, along with the Theory of Relativity, Quantum Mechanics was developed due to the efforts of M.Planck, E. Schrodinger and W. Heisenberg. This theory showed that the electromagnetic field at the quantum level was quantized. This, along with, wave-particle duality of light was considered irreconcilable with the general theory of relativity. To reconcile general theory of relativity with various quantum theories, Y. Nambu proposed a string model for fundamental particles and formalized the dynamics of light string. Utilizing the experimentally verified quantum theories of chromodynamics, electrodynamics, supersymmetry of fundamental particles (unifying Bosons and Fermions), it was possible to supersymmetrize the string model of fundamental particles, resulting in the so-called superstring (supersymmetric string)

Introduction

3

theory. Currently, to explain the non-living universe, string model hopes to be experimentally verifiable, theoretically viable model. But the material universe consists of living universe as well as non-living universe. All efforts in science probed the non-living universe using experimental as well as theoretical methodology. The efforts of all scientists enabled them to see farther by “standing on the shoulders of earlier giants”. The homo-sapien animals by devising various tools discovered and formalized various laws and theories related to non-living physical reality based dynamical systems. The homo-sapien animal learned to build machines to facilitate his life and that of the community surrounding him. By understanding the mechanism of various functional units in living system such as ear, eye, various machines such as telephone, television, loud-speaker were built. Also, in the research area of artificial intelligence in Electrical Engineering, various functions of human brain are simulated in machines called robots. In the case of living universe, the scene was entirely different. The author made various pioneering innovations on living systems unlike the extended, stretched over effort of non-living systems by various eminent scientists. The objective/goal of this is to provide artificial/manufacturable models of living systems i.e. robots which resemble in every respect living systems. In arriving at artificial models, the effort of various eminent mathematicians, scientists culminating in those of N. Wiener (who coined the word CYBERNETICS) were helpful. The important discovery and the associated formalization belonged to the pioneering efforts of the author.

LOGICAL BASIS FOR COMPUTATION George Boole developed the algebra when the variables assume “true” or “false” values. This algebra is called the Boolean algebra. Certain elementary Boolean algebraic expressions are realized in equipment called “logic gates”. When the logic gates are combined/coordinated, arbitrary Boolean algebraic expression can be computed. The combination of Boolean logic gates ( an assemblage with some minimum configuration of gates) and memory elements forms an arithmetic unit. When such a unit is coupled with a control unit the Central Processor Unit (CPU) in a computer is realized. The CPU in association with a memory, input and output units forms a computational unit without intelligence. This is just a machine which can be utilized to perform computational tasks in a fast manner. Various thought provoking modifications make it operate on data in an efficient manner and provide computational results related to various problems.

LOGICAL BASIS FOR CONTROL Faraday conducted the experiments related to electrical and magnetic phenomenon. He discovered the laws of electro-magnetic induction. Based on his investigations, Fleming

4


discovered that a time varying electric field leads to magnetic field which can be capitalized for the motion of a neutral body. He also discovered that a time varying magnetic field leads to electric field inside a neutral conductor and flow of current takes place. These formed the Fleming’s left hand and right hand rules relating the relativistic effects between the electric field, magnetic field and conductor. These investigations of Faraday and other scientists naturally paved the way for electric circuits consisting of resistors, inductors and capacitors. Such initial efforts led to canonical circuits such as RL circuit, RLC circuit, RC circuit etc. The systems of differential equations and their responses were computed utilizing the analytical techniques. The ability to control the motion of an arbitrary neutral object led to applications of electrical circuits and their modifications for control of trajectories of aircrafts. Thus, the automata which can perform CONTROL tasks was generated. These control automata were primarily based on electrical circuits and operate in continuous time with the ability to make synchronization at discrete instants. Later utilizing the Sampling Theorem, sample-data control systems operating in discrete time were developed.

LOGICAL BASIS OF COMMUNICATION The problem of communication is to convey message from one point in space to another point in space as reliably as possible. The message on being transmitted through the channel, by being subject to various forms of disturbance (noise) is changed/garbled. By coding the message (through addition of redundancy), it is possible to retrieve the original message from the received message. Thus, the three problems: control, communication and computation can be described through the illustration in Figure 1.1. From the illustration, the message that is generated may be in continuous time or discrete time. Utilizing the Sampling Theorem, if the original signal is band-limited, then the message can be sampled. The sampled signal forms the message in discrete time. The message is then encoded through an encoder. It is then transmitted through a channel. If the channel is a waveform channel, various digital modulation schemes are utilized in encoding. The signal, on reaching the receiver is demodulated through the demodulator and then it is decoded. This whole assembly of hardware equipment forms the COMMUNICATION equipment. The above summary provided the efforts of engineers, scientists and mathematicians to synthesize the automata which serve the purpose of CONTROL, COMMUNICATION AND COMPUTATION. These functions are the basis of automata that stimulate living systems. These automata model the living systems. In other words, control, communication and computation automata when properly assembled and co-ordinated lead to robots which simulate some functions of various living systems. In the above effort at simulating the functions of living systems in machines, traditionally the control, communication and computation automata led to sophisticated robots (which served the purpose pretty well). Thus, the utilitarian viewpoint was partially satisfied. But, the author took a more FUNDAMENTAL approach to the problem of simulating a


6

are extended to multi/infinite dimensional linear systems. Also, the results developed in one dimension for computation of optimal control are immediately extended to certain multi/infinite dimensional linear systems. This result in association with the formalization of multi/infinite dimensional logic theory, multi/infinite dimensional coding theory (as an extension of one dimensional linear and non-linear codes) provided the formal UNIFIED THEORY in multi/infinite dimensional linear systems. The formal mathematical detail on models of living system functions are provided in Chapters 2 to 5. These chapters provide the details on control, communication and computation automata in multiple dimensions. Several generalized models of neural networks are discussed in Chapters 5 to 9. Also relationship between neural networks and optimal filters is discussed in Chapter 7. In Chapter 10 advanced theory of evolution is discussed.

ADVANCED THEORY OF EVOLUTION Mathematical models of living system functions motivated us to take a closer look at the functions of natural living systems observed in physical reality. In physical reality, we observe homo-sapiens as well as lower level animals such as tigers, lions, snakes etc. It is reasoned that some of the functions of natural living systems are misunderstood or un-understood. Biological living systems such as homo-sapiens lead to a biological culture. In a biological culture that originated during the ice age in oceans, various living species were living in the oceans. Through some process, the two necessities of metabolism and reproduction were developed by all living species. The homo-sapien species was responsible for our current understanding of various activities, functions of observed living systems. The author hypothesizes that the homo-sapien interpretations are totally wrong. For instance, • Metabolism which leads to killing of one species by another is unnecessary to sustain life. • The belief (like many superstitions) that death and aging are inevitable is only partially true. To be more precise, it should be possible to take non-decayed organs of a living species and by recharging the dead cells, make it living. Many such innovative ideas on living systems are discussed in Chapter 10. The only necessities of natural living systems that are observed are ‘metabolism’ and ‘reproduction’. By and large the only organization and community formation that we see in other (than homo-sapiens) natural sustems are of the following form • • • • •

Migratory pattern of birds Sharing the information on the place of food Forming a group of families to satisfy the reproductive needs Occasional bird songs of mutual courtship Occsional rituals related to protecting the members of their group etc.

Introduction

7

The organization, culture observed in other biological systems and other natural living systems is nowhere comparable to those observed in the homo-sapien species. But the author hypothesizes that this marginal/poor organization is primarily due to lack of coordination which is achieved through the language. Thus, major effort in organizing the lower level species of living systems is through teaching a language. Thus, organization of living systems other than the homo-sapiens (for homo-sapien and other purposes) should be possible. An important part of organizing the homo-sapiens was the educational system through an associated language. In the same spirit, by teaching some lower level animals to speak certain language, they could be organized/educated to understand as well as develop science and technology. When the lower level animals are organized in a zoo through various methods, they could lead to a culture and a civilization. Various natural living machines have developed organs/functional units due to evolutionary needs. These functional units essentially include sensors to collect video, audio information or more generally sensors to collect data on the surrounding environment in the universe. The data gathered by the living machine from the surrounding environment in physical reality is utilized to perform some primary functions such as metabolism, reproduction etc. The data is processed by various functional sub-units inside the brain of a living machine. Thus the understanding of the operation of various functional sub-units in the brain of natural living machines leads to building artificial living machines which are far superior in functional capabilities.


CHAPTER

2

Multi/Infinite Dimensional Neural Networks, Multi/Infinite Dimensional Logic Theory

2.1 INTRODUCTION One dimensional logic theory is concerned with the study of static/dynamic transformations on one dimensional arrays of zeroes and ones to arrive at arrays of zeroes and ones. Various standard logic gates such as AND, OR, NOT, NAND, XOR, NOR are defined on one dimensional arrays/vectors. The logic synthesis of digital integrated circuits, consisting of the interconnection of logic gates which transit through a set of states, is performed through the utilization of the associated state transition diagram. The set of allowed transitions in the state space lead to various classes of digital circuits such as shift registers, counters, flip flops etc. In one dimensional logic theory various theorems on the decomposition, synthesis of Boolean functions are proved and are utilized in the logic synthesis of complex digital integrated circuits. In the practical implementation of such digital integrated circuits, semiconductor technology with devices such as diodes, transistors, field effect transistors was effectively utilized. The design and implementation of complex digital integrated circuits led to the development of highly sophisticated computers, computer systems serving various practical applications. Some practical applications such as those in medical imaging, remote sensing, pattern recognition led to the design and implementation of various types of parallel computers. These computers operate on two dimensional arrays of zeroes and ones. But the processing units in these computers treat the two dimensional array elements as those from one dimensional arrays. Thus, the two dimensional nature of an array with dependency structure is never capitalized. This limitation led the author to innovate information processing units which operate on two/multidimensional arrays. Such information processing units should necessarily be based on sub-units which operate on arrays of binary data and produce binary arrays. These sub-units constitute the two/multidimensional logic circuits. A more

10


general class of information processing sub-units and thus the units operate on arrays whose entries are allowed to assume multiple (not necessarily binary) values. Automata which operate on multidimensional arrays to perform desired operation can be defined heuristically in many ways. In some applications such as in 3-d array/ image processing, the information processing operation can only be defined heuristically based on the required function. But, a more organized approach to define multidimensional logic functions is discovered and formalized by the author. In this chapter, the author describes the mathematical formalization for multidimensional logic units. The relationship between multidimensional logic units and multidimensional neural networks is also discussed. The generalization of the results to infinite dimensions is also briefly described. Two dimensional neural networks were utilized by various researchers working in the area of neural networks. The application of two dimensional neural networks to various real world problems was also extensively studied. But, an effective mathematical abstraction for modeling two/multi/infinite dimensional neural networks was lacking. The author in this chapter demonstrates that tensors provide a mathematical abstraction to model multi/ infinite dimensional neural networks. The contents of this chapter are summarized as follows: A mathematical model of an arbitrary multidimensional neural network is developed. A convergence theorem for an arbitrary multidimensional neural network represented by a fully symmetric tensor is stated and proved. The input and output signal states of a multidimensional logic gate/neural network are related through an energy function, defined over the fully symmetric tensor representing the multidimensional logic gate, such that the minimum/maximum energy states correspond to the output states of the logic gate realizing a logic function. Similarly, a logic circuit consisting of the interconnection of logic gates, represented by a symmetric tensor, is associated with a quadratic/higher degree energy function. Multidimensional logic synthesis is described. Infinite dimensional logic theory, logic synthesis are briefly discussed through the utilization of infinite dimension/ order tensors. This chapter is organized as follows. In section 2, a mathematical model of an arbitrary multidimensional neural network and associated terminology is developed. In section 3, a convergence theorem for an arbitrary multidimensional neural network is proved. In section 4, the input/stable states of a multidimensional neural network are associated with the input/output signal states of a multidimensional logic gate. A mathematical model of an arbitrary multidimensional logic gate/circuit is described. Thus, multidimensional logic theory, logic synthesis is formalized. In section 5, infinite dimensional logic theory, logic synthesis are described. In section 6, the relationship between multidimensional neural networks, multidimensional logic theories, various constrained static optimization problems is elaborated. Various constrained optimization problems that commonly arise in various problems are listed. Various innovative ideas in multidimensional neural networks are briefly described. The chapter concludes with a set of conclusions.


11

2.2 MATHEMATICAL MODEL OF MULTIDIMENSIONAL NEURAL NETWORKS A discrete time multidimensional neural network paradigm is a dynamical system evolving in discrete time. It can be represented by a weighted connectionist structure in multidimensions. Thus, there is a weight attached to each edge of the connectionist structure in multidimensions and a threshold value attached to each node. At each node of the connectionist structure, a certain algebraic threshold function is computed. It is well known in the theory of one dimensional neural networks that a symmetric matrix can be utilized to represent a one dimensional neural network. With the motivation, applications of one dimensional neural networks, two dimensional neural networks were heuristically designed and utilized for various applications. But, the author for the first time realized that tensor is the most natural mathematical abstraction that can be utilized to represent two/multidimensional neural networks.

Multidimensional Neural Networks: Tensors Before describing the mathematical model of multidimensional neural networks, the following discussion on tensors and associated concepts is very relevant. It is important to realize that given n independent variables, the expression n

∑C X i =1

i

(2.1)

1

is called a homogeneous linear form in the variables, the expression n

n

∑∑ C i =1 j =1

ij

Xi X j

(2.2)

is called a homogeneous quadratic form, the expression n

n

n

∑∑∑ C i =1 j =1 k =1

ijk

Xi X j K k

(2.3)

is called a homogeneous form (BoT) of degree three and so on. Given the components of a tensor of order n, of dimension m , it is possible to define a homogeneous form of degree n. The connection structure of a one dimensional neural network, the symmetric matrix, is naturally associated with a homogeneous quadratic form as the energy function, which is optimized over the one dimensional hypercube. Thus, in one dimension, to utilize a homogeneous form of degree n as the energy function, a generalized neural network is employed, in which, at each neuron, an arbitrary algebraic threshold function is computed. But, in multidimensions, to describe the connection structure of a neural network, a tensor is necessarily utilized.


12

With the above description of necessity of tensors to represent generalized/ multidimensional neural networks, some notation related to tensors is provided to facilitate the description of mathematical model of an arbitrary multidimensional neural network.

Tensors, Tensor Products Matrices are utilized to represent quadratic forms, whereas tensors are necessary to represent a homogeneous form of degree n. i.e.

Suppose, one second order tensor is a linear function of another second order tensor Aik = λiklm Bim

(2.4)

where λiklm is a set of k 4 coefficients. It is easy to see that λiklm is a tensor of dimension k and order 4. This is illustrative of linear transformation of tensors. Now, we discuss some concepts in the multiplication of tensors. Let A i k and Bi k be the components of two second order tensors. Consider all possible products of the form Ciklm = Aik Bim

(2.5)

Then, the numbers C iklm are the components of a fourth-order tensor, called the outer product of tensors with components A ik and Bi k. Multiplication of any number of tensors of arbitrary order is defined similarly (BoT), i.e. the product of two or more tensors is the product of the components of the tensors, which are factors. The order of a tensor product is clearly the sum of the orders of the factors. Contraction of Tensors: The operation of summing a tensor of order n (n >2) over two of its indices is called contraction. It is clear that contraction of a tensor of order n leads to a tensor of order n-2. This tensor can be repeatedly contracted to arrive at a tensor of order 2 or a scalar depending on whether n is even or odd. The result of multiplying two or more tensors and then contracting the product with respect to indices belonging to different factors is often called an Inner Product of the given tensors. Thus, based on the notation associated with the indices, it is understood from the context whether inner product or outer product of tensors is utilized. With the above requisite notation from tensor algebra summarized, before describing a mathematical model of an arbitrary multidimensional neural network, the following intuitive discussion is provided to facilitate easier understanding. The state of a neuron at the discrete time instant n+1 is computed by summing the contributions from other neurons connected to it through synaptic weights which are the components of a fully symmetric tensor S, representing the connection structure and the


13

state tensor of neuronal states at the time instant n. Thus, we first compute the outer product of connection tensor and the state tensor of neurons at the time instant n and perform the contraction over all the indices (representing the neurons) connected to a chosen neuron. Thus, this inner product operation followed by determining its sign/parity/polarity (positive or negative value) gives us the state tensor at time instant n+1. This procedure is repeated at all the neurons where the state is updated. Remark Throughout the research article, the notation “multidimensional neural network” is utilized. The standard notation associated with tensors utilizes the term, “dimension” to represent the number of values an independent variable can assume and the term, “order” to represent the number of independent variables. Thus, the state tensor order represents the number of independent dimensions in the multidimensional neural network, MN. The notational confusion between the usage of terms “order”, “dimension” should be resolved from the context.

Mathematical Model Description Let MN be a multidimensional neural network of dimension m and order n, then MN is uniquely specified by ( S, T ) where ( the number of neurons in each independent variable/ dimension/ order index is m ) S is a fully symmetric tensor of order 2n and dimension m . S, the connection structure of multidimensional neural network, is a fully symmetric tensor in the following sense

Si 1, i 2,..., in ; j 1, j 2,..., jn = Sj 1, j 2,..., jn ; i 1, i 2,...,in

(2.6)

for all {i1,i2,...,in}, {j1,j2,...,jn}. This captures the intuitive notion that the multidimensional neural network has nodes which correspond to the multidimensional neurons. The connectionist structure of the network, in the fully connected case, has a synaptic connection from every neuron to every other neuron and thus specifies the number of order indices/ dimensions/variables of the fully symmetric tensor. Furthermore, it is fully symmetric since there is a link between any two nodes and the weight attached to the link is the same in both directions. T is a tensor compatible with S such that each component is the threshold at the node (i1, i2,...,in) of the multidimensional neural network. Every node ( multidimensional neuron ) can be in one of the two possible states, either +1 or –1. The state of node (i1, i2,...., in) at time t is denoted by Xi1, i2,..., in (t). The state of MN at time t is the tensor Xi1, i2,..., in (t), where X is tensor of dimension m and order n. The state evolution at node (i1, i2,...,in) is computed by

Xi 1, i 2,..., in (t + 1) = Sign ( Hi 1, i 2,..., in (t)),

(2.7)


14

where, m

m

j1= 1

jn = 1

Hi 1, i 2,..., in (t) = ∑ ... ∑ Si 1,..., in ; j 1,..., jn X j 1,..., jn (t) − Ti 1,..., in (t)

(2.8)

The next state of the network Xi1,...,in (t +1) is computed from the current state by performing the evaluation (2.7) at a subset of the nodes of the multidimensional neural network, to be denoted by G. The modes of operation of the network are determined by the method by which the subset G is selected in each time interval. If the computation is performed at a single node in any time interval, i.e.|G| = 1, then we will say that the network is operating in a serial mode, and if |G| = m n, then we will say that the network is operating in a fully parallel mode. All other cases, i.e. 1 < |G| < m n, will be called parallel modes of operation. Unlike a one dimensional neural network, multidimensional neural network lends itself for various parallel modes of operation. It is possible to choose G to be the set of neurons placed in each independent dimension or a union of such sets. The set G can be chosen at random or according to some deterministic rule. A state of the network is called stable if and only if

Xi 1,..., in (t) = Sign (S ⊗ Xi 1,..., in (t) − Ti 1,...iin )

(2.9)

where ⊗ denotes inner product i.e. outer product followed by contraction over the appropriate indices. Once the network reaches such a state, there is no further change in the state of the network no matter what the mode of operation is.

2.3 CONVERGENCE THEOREM FOR MULTIDIMENSIONAL NEURAL NETWORKS Utilizing a fully symmetric tensor to represent the connection structure of a multidimensional neural network, utilizing the notation of tensor products, in the following, convergence theorem for an arbitrary multidimensional neural network is stated and proved. Theorem 2.1: Let MN = (S, T) be a multidimensional neural network of dimension m and order n. S is a fully symmetric tensor of order 2n and dimension m with Si 1,..., in ; i 1,...,in ≥ 0 . The network MN always converges to a stable state while operating in the serial mode (i.e. there are no cycles in the state space) and to a cycle of length utmost 2 while operating in a fully parallel mode (i.e. the cycles in the state space are of length ≤ 2 ). Proof: Serial mode of operation of the multidimensional neural network is first considered. In this mode of operation, during each time step of the operation of the neural network, the state of only one neuron is updated. In other words, the state of each neuron is only updated serially. At each multidimensional neuron in the network MN, the total synaptic


15

contribution from all neurons is first determined and its sign is determined to arrive at the updated state of the neuron. Mathematically, this is achieved by computing the outer product of the fully symmetric tensor S and the {+1, –1} state tensor of the multidimensional neural network. In tensor notation, this is specified by

Ci 1,..., in ; j 1,..., jn = Si 1,..., in ; j 1,..., jn X j 1,..., jn .

(2.10)

The total synaptic contribution at any neuron located at the location (i1, i2,..., in) is determined by contracting the above outer product over all the indices {j1, j2,..., jn} i.e. over all the neurons connected to it through the synaptic weights determined by the components of the fully symmetric tensor S. The resultant scalar synaptic contribution at any neuron (i1, i2,..., in) is thus determined by the inner product operation. The sign of the resulting scalar constitutes the updated state of neuron. Thus, the state of any neuron (i1, i2,..., in) in the multidimensional neural network in the serial mode of operation is given by m

m

j1= 1

jn = 1

Xi 1, i 2,..., in ( k + 1) = Sign ( ∑ ... ∑ Ci 1,..., in ; j 1,..., jn ( k ) − Ti 1,..., in )

= Sign (S ⊗ X(k ) − T )

(2.11) (2.12)

where ⊗ is utilized as the symbol to denote the inner product between compatible tensors. This symbol is sometimes suppressed and it should be understood from the context whether inner product/outer product between the tensors is meant. With the state updating scheme in the tensor notation specified, the energy function that is optimized in the network MN is described. It is given by m

m

m

m

E = < X( k ), S ⊗ ( k ) > = ∑ .. ∑ ∑ .. ∑ Si 1,..., in ; j 1,..., jn X i 1,..., in ( k ) X j 1,..., jn ( k ) i 1 = 1 in = 1 j 1 = 1

jn = 1

(2.13)

where < > denotes the inner product operator between the compatible tensors. It is assumed in the above specification of the energy function of the neural network MN that the threshold at each neuron is zero. This is no loss of generality, since by augmenting the tensor S and the state tensor, the threshold values can be forced to be zero. It is easy to see that such a thing can always be done by considering a one dimensional neural network in which the threshold at each neuron is non-zero and arriving at a network in which the threshold at each neuron can be made zero by augmenting the state vector as well as the connection matrix. Utilizing the definition of the above energy function of the network, let ∆E = E1 ( t + 1) − E1 ( t) , (discrete time index t instead of k is used) be the difference in the energy associated with two consecutive states (transited in the serial mode of operation of the multidimensional neural network ), and let ∆X i 1,....in denote the difference between the next state and the current state of the node at location (i1, i2,..., in) at some arbitrary time t. Clearly,


16

0, if, Xi1,..., in (t) = Sign (Hi1,...,in (t)) ∆Xi 1,..., in = {−2, if, Xi 1,..., in (t) =1, and, Sign( Hi 1,...,in (t)) =− 1

(2.14)

+2, if, X i 1,..., in (t ) = −1, and, Sign ( H i 1,..., in (t ) = + 1 By assumption, the computation (2.14) is performed only at a single node at any given time. Suppose this computation is performed at any arbitrary node at location (i1, i2,..., in) ; then the difference in energy resulting from updating the network state is given by

..∑ Si 1,..., in ; j 1,..., jn X j 1,..., jn + ∑ ..∑ Si 1,...,in ; j 1,..., jn Xi 1,...,in ) ∆E = ∆Xi 1,..., in (∑ j1 jn i1 in

+ Si 1,..., in ; i 1,...,in ∆Xi 1,..., in − 2∆Xi 1,..., in Ti 1,..., in

(2.15)

Utilizing the fact that S is fully symmetric and the definition H i1,..., in (t), it follows that

∆E = 2 ∆Xi 1,..., in Hi 1,...,in + Si 1,...,in ;i 1+ ,...in

∆Xi 1,...., in

(2.16)

Hence, since ∆Xi 1,..., in Hi 1,..., in ≥ 0 and Si 1,..., in i 1,..., in ≥ 0 , it follows that at every time instant, ∆E ≥ 0 . Thus, since the energy E is bounded from above by the appropriate norm of S, the

value of energy will converge. Now, it is proved in the following that convergence of energy implies convergence to a stable state. Once the energy in the network has converged, it is clear from the following facts that the network will reach a stable state after utmost m 2n time intervals. (a) if ∆X = 0 then it follows that ∆E = 0; (b) if ∆X ≠ 0 , then ∆E = 0, only if the change in Xi 1,..., in (t) is from –1 to +1, with Hi 1,..., in = 0. In the fully parallel mode of operation of the network MN, the state updating scheme for the state tensor of MN is given by

Xi 1,..., in (t + 1) = Sign (S ⊗ Xi 1,..., in (t) − Ti 1,..., in )

(2.17)

where ⊗ denotes the inner product between compatible tensors. Since, the serial mode proof shows that a stable state is always reached with the above stated updating scheme, it is immediate that by pairwise flipping of the values of any two dimension variables in the state tensor, the same energy function value is attained. This, in turn implies that in the parallel mode of operation of a multidimensional neural network, either a stable state is reached or a cycle of length utmost 2 is reached (The two state tensors lead to the same value of the energy function). This approach to the proof for the parallel mode of operation follows the one provided in reference. [Br G] Q. E. D


17

2.4 MULTIDIMENSIONAL LOGIC THEORY, LOGIC SYNTHESIS One dimensional logic theory as well as logic synthesis deal with information processing logic gates, logic circuits which operate on one dimensional arrays of zeroes and ones (or more generally one dimensional arrays containing finitely many symbols ). The operations performed by AND, OR , NOR, NAND, XOR gates have appropriate intuitive interpretation in terms of the entries of the one dimensional arrays i.e. vectors. Any effort to generalize the one dimensional logic operations to multidimensions leads to various heuristic possibilities and requires considerable ingenuity in formalizing a definition. But, in the following, utilizing the multidimensional neural network model described above, a formal/ mathematical procedure to multidimensional logic theory is described. The input and output signal states of a multidimensional logic gate are related through an energy function. Equivalently, the multidimensional logic functions are associated with the local optimum of various energy functions defined over the set of input m-d arrays. In view of the mathematical model of a multidimensional neural network described in section 3, it is most logical to define the minimum/ maximum energy states of a multidimensional neural network (optimizing an energy function over the multidimensional hypercube ) to correspond to the multidimensional logic gate functions operating on the input arrays. Definition 2.1 A multidimensional logic function realized through a multidimensional logic gate (with inputs and outputs) is defined to be the local minimum/maximum of the energy function of an associated multidimensional neural network. Equivalently, the local optima of the energy function of a multidimensional neural network correspond to the logic functions that are realized through various logic gates. The following detailed description is provided to consolidate the above definition vital to multidimensional logic theory. The logic functions which operate on the input array are identified to be the stable states of a multidimensional neural network ( in multiple independent variables i.e. time, space etc.). These are the transformations between a set of input states of a multidimensional neural network which converge to a stable state on iteration of a multidimensional neural network. In other words, in multiple independent variables, the mapping between the input states and the stable states to which the network converges on iteration are defined to be the logic functions realized by a multidimensional logic gate. By the proof of the convergence theorem, the logic functions are invariants of a tensor on the multidimensional hypercube. The definition of multidimensional logic function is illustrated in Figure 2.1. In the case of one dimensional logic theory, it has been shown that the set of stable states of a neural network correspond to various one dimensional logic functions (CAB). With the definition of multidimensional logic function stated and clarified in many redundant ways above, multidimensional logic synthesis is described in the following.

18


Multidimensional Logic Synthesis A multidimensional logic circuit consists of an arbitrary interconnection of multidimensional logic gates. Multidimensional logic synthesis, as in one dimension, involves synthesizing logic circuits for different purposes. In view of the above definition of multidimensional logic functions defined through the local optima of energy functions (realized through multidimensional neural networks), it is natural to see if it is possible to associate energy functions with multidimensional logic circuits. When such a scalar valued energy function can be associated with logic circuits, the problem of multidimensional logic synthesis, is reduced to realizing such energy functions. In the following, this important idea is developed. A multidimensional logic circuit consists of interconnection of multidimensional logic gates. But, the interconnection structure of a multidimensional logic gate is represented by a fully symmetric tensor. Since, every two gates in a logic circuit need not necessarily be connected to one another, a multidimensional logic circuit connection structure is represented by a tensor of necessary/compatible order which is not necessarily fully symmetric but it is required to be minimally symmetric. Thus, this block symmetric tensor which is fully symmetric within the blocks (representing the connection structure of a multidimensional neural network corresponding to a component logic gate) provides a representation of multidimensional logic circuit. This tensor is utilized to associate quadratic/higher degree energy functions with the multidimensional logic circuit. The set of local optima of the energy functions constitute the stable states of one or more interconnected logic gates. Thus, the set of input states (input pins) and output states (output pins) of an entire multidimensional logic circuit are related through an energy function, defined over the connection structure of a very high dimensional neural network. The set of local optima of the energy function relating the input and output pins of a multidimensional logic circuit realize various multidimensional logic functions. From the above description, it is evident that the multidimensional logic synthesis depends on how the multidimensional logic gates are connected to one another. The structure of interconnection determines the structure of symmetric tensor representing the multidimensional logic circuit. The essential result in multidimensional logic synthesis is summarized through the following theorem. Theorem 2.2: Given a multidimensional logic circuit, there exists a block symmetric tensor S, representing the inter-connection structure of multidimensional neural networks (modeling the multidimensional logic gates). The mapping between the input and output states of a multidimensional logic circuit corresponds to that between input tensors, local optima of energy function (quadratic/higher degree) represented by the block symmetric tensor. The stable states of interconnected multidimensional neural networks represent the multidimensional logic functions synthesized by the logic circuit.


19

The proof of the above theorem follows from the convergence theorem and is avoided for brevity. The classification of multidimensional logic circuits is based on the type of transitions allowed between the states in the multidimensional state space. The type of state transitions fall into the following form: (a) whether the next state reached depends on the past state only or not, as in one dimensional logic synthesis, (b) the type of neighbourhood of states about the current state on which the next state reached depends. The type of neighbourhoods about the current state are classified into few classes. These classes are similar to those utilized in the theory of random fields, multidimensional image processing, (c) the classification of trajectories transited by the multidimensional neural network or a local optimum computing circuit/scheme. In the above discussion, we considered quadratic forms as the energy functions (motivated by the simplest possible neural network model) optimized by the logic gates, which when connected together lead to logic circuits. This approach toward multidimensional logic theory motivates the definition of more ‘general‘ switching/logic functions as the local optimum of higher degree forms over the various subsets of multidimensional lattice (hypercube, bounded lattice etc.). Definition 2.2 A generalized logic function (representing a generalized logic gate or generalized logic circuit) is defined as a mapping between an m -dimensional input array and the local optimum of a tensor based form of degree greater than or equal to two, over various subsets of multidimensional lattice (the multidimensional hypercube, multidimensional bounded lattice). These local optimum of higher degree form (based on a tensor) are realized through the stable states of a generalized multidimensional neural network. In (Rama 3) , it is shown that the strictly generalized logic function defined above has better properties than the ordinary logic function described in Definition 4.1. The generalized logic function is related to a multidimensional encoder utilized for communication through multidimensional channels. Now, with the generalized multidimensional logic gate defined above, logic synthesis with these types of logic gates involves interconnection of them in certain topology. This ordinary and generalized approach to multidimensional logic gate definition and logic synthesis is depicted in Figures 2.1 to 2.3. Detailed documentation on logic synthesis and design of future information processing machines is being pursued.


21

Proof: One dimensional neural network with state vector size infinity is uniquely defined by (S, T) where S is an infinite dimensional (rows as well as columns) symmetric matrix and T is an infinite dimensional vector of thresholds at all the neurons. The state of the neural network at time t is a vector whose components are +1 and –1. The next state of a node is computed by

Xi (t + 1) = Sign(Hi (t)) = + 1, if, Hi (t) ≥ 0

(2.18)

–1, otherwise where, ∞

Hi (t ) = ∑ Sji X j (t ) − Ti

(2.19)

j =1

The entries of S are such that the infinite sum in the above expression converges. The next state of the network i.e. X ( t+1 ), is computed from the current state by performing the evaluation (2.18) at a subset of the nodes of the network, to be denoted by K. The mode of operation of the network is determined by the method by which the set K is selected at each time interval i.e. if |K| = 1, then we will say that the network is operating in a serial mode. Without loss of generality T = 0. In the following, we consider the serial mode of operation. We argue that with the above stated updating scheme at an arbitrary chosen neuron, the energy function (quadratic) increases. ∞

∞

E( k ) = ∑∑ Sij Xi ( k ) X j ( k )

(2.20)

i =1 j =1

Without loss of generality, consider the case where all the thresholds are set to zero. It is easy to see (set the last component of state vector to –1 and appropriately augmented entries of S) that for any finite L, we have L

L

∑∑ S i =1 j =1

ij

Xi ( k ) X j ( k ) ≤

L

L

∑∑ S i =1 j =1

ij

Xi ( k + 1) X j ( k + 1)

(2.21)

by the convergence theorem for one dimensional neural networks of order L, for any arbitrary L. Now let L tend to infinity. Hence ∞

∞

∑∑ Sij Xi (k) X j (k ) ≤ i =1 j =1

∞

∞

∑∑ S i =1 j =1

ij

Xi ( k + 1) X j ( k + 1)

(2.22)

Thus, in the serial mode, the network converges to a stable state. By the Convergence Theorem for one dimensional neural network (with the state vector size finite) in the parallel mode of operation, if any finite set of nodes is state updated, there is either convergence or existence of a cycle of length 2. Thus, when an infinite

22


dimensional vector is state updated in the parallel mode, for every finite segment of it, either there is convergence or a cycle of length 2 (utmost two vectors for which the energy values are the same) exists. Since, the energy function associated with the infinite dimensional vector is the limit of those associated with the finite segments, it is evident that the scalar energy values converge or a cycle of length utmost two exists. Q.E.D. Now, we discuss briefly, the other infinite dimensional neural networks of dimension infinity and order finite/infinite ( modeling tensor variables). The following lemma is well known from the set theory. Lemma 2.1: Countable union of countable sets is countable. The above lemma implies that the convergence theorem proved above in association with the convergence theorem for multidimensional neural networks (its proof argument in section 3) provides us with the convergence proof for a large class of infinite dimensional neural networks (dimension and/or order of tensors utilized in modeling is infinity). Details on the convergence theorem for infinite dimensional neural networks are provided below. Tensors utilized to represent the connection structure, state of neurons of infinite dimensional neural network are such that the either the dimension or the order is finite/infinite with not both of them being finite (either the dimension or the order or both are infinite ). In one dimension, when the number of neurons is infinite and a quadratic energy function is optimized through a neural network scheme, by a straightforward extension of the results in (Rama 3), the stable states of the neural network constitute a graph-theoretic code (with the length of the codeword being infinite). The set over which optimization is carried out is the unbounded unit hypercube (countable number of entries in the infinite dimensional state vector), a subset of the lattice ( based on one independent variable ). The following theorem is concerned with the points on the lattice in multi/infinite dimensions. This theorem is the infinite dimensional extension of the result proved in section 3. Theorem 2.4: Let MN = (S, T) be an infinite dimensional neural network of order n/ infinite and dimension infinity (number of neurons in each dimension). S is a fully symmetric tensor of dimension infinity and order 2n/infinity with Si 1,..., in ; i 1,...,in ≥ 0 . The network MN always converges to a stable state while operating in a serial mode (i.e., there are no cycles in the state space), while in the parallel mode, the network will always converge to a stable state or to a cycle of length 2 (i.e., the cycles in the state space are of length ≤ 2). Proof: For a multidimensional neural network modeled by a tensor of dimension and order finite, in the serial mode of operation, the network always converges to a stable state. Since, the quadratic energy function is a scalar value defined over the connection tensor (whose order, dimension are finite ), by letting the dimension and/or order tend to infinity in (2.13), it is immediate that the energy function value increases in the serial mode until a stable state is reached starting in a certain initial state. Thus, for various infinite


23

dimensional neural networks considered, convergence to a stable state in the serial mode of operation is ensured ( i.e. there are no cycles in the state space ). In the parallel mode of operation of the infinite dimensional neural network, by the same reasoning as in Theorem (2.1), the network will always converge to a stable state or to a cycle of length 2 depending on the order of the network ( i.e. the cycles in the state space are of length less than or equal to 2). Q.E.D As in the case of multidimensional logic theory, the above convergence theorem is utilized as the basis to describe infinite dimensional logic theory as well as logic synthesis. It should be noted that the infinite dimensional logic synthesis only has theoretical importance. Brief discussion on infinite dimensional versions is provided for the sake of completeness. Definition 2.3 An infinite dimensional logic function realized through an infinitedimensional logic gate (with inputs and outputs) is defined to be the local minimum/maximum of the energy function of an associated infinitedimensional neural network. Equivalently, the local optima of the energy function of an Infinitedimensional neural network correspond to the logic functions that are realized through various logic gates. With the above definition of infinite dimensional logic function, detailed results in infinite dimensional logic synthesis are being developed along the lines of those in multidimensional logic synthesis. Brief description is provided in the following for the sake of completeness. An infinitedimensional logic circuit consists of an arbitrary interconnection of infinitedimensional logic gates. Infinitedimensional logic synthesis, as in one dimension involves synthesizing logic circuits for different purposes. These infinite dimensional logic circuits only have theoretical implementations. Infinitedimensional logic synthesis depends on how the infinitedimensional logic gates are connected to one another. The structure of interconnection determines the structure of symmetric tensor (order and/or dimension is infinity) representing the infinitedimensional logic circuit.

2.6 NEURAL NETWORKS, LOGIC THEORIES, CONSTRAINED STATIC OPTIMIZATION Multidimensional neural networks provide a computational paradigm to determine the local optima of quadratic as well as higher degree forms defined in terms of tensors (including matrices) over various subsets of the multidimensional lattice. These units which map a multidimensional array/tensor to a local optimum (stable state of the multidimensional neural network), thus constitute the multidimensional logic gates. Interconnection of such multidimensional logic gates constitutes a multidimensional logic circuit. Thus, multidimensional logic circuits are interconnected multidimensional neural

24


networks. The interconnection structure weights are represented through a symmetric tensor. Thus, multidimensional logic theory/logic synthesis are associated with the theory of multidimensional neural networks. These theories are in turn related to static optimization of various forms (quadratic as well as higher degree) over different subsets of lattice and other sets. Various constrained static optimization problems that are of interest in different applications (neural networks, logic theories etc.) are summarized below: (1) Optimization of a quadratic form in finitely many variables over the one dimensional hypercube (one independent variable), (2) Optimization of a higher degree form in finitely many variables over the one dimensional hypercube (one independent variable), (3) Optimization of a quadratic form over the infinite dimensional (size of the state vector) hypercube in one dimension, (4) Optimization of a higher degree form over the infinite dimensional (size of the state vector) hypercube in one dimension, (5) Optimization of a quadratic form over the finite/infinite dimensional hypercube in finitely/infinitely many dimensions, (6) Optimization of a higher degree form over the finite/infinite dimensional hypercube in finitely/infinitely many dimensions, (7) Optimization of a quadratic form over a bounded lattice in finitely/infinitely many dimensions, (8) Optimization of a higher degree form over a bounded lattice in finitely/infinitely many dimensions, (9) Optimization of a quadratic form over the unbounded lattice in finitely/infinitely many dimensions, (10) Optimization of a higher degree form over the unbounded lattice in finitely/ infinitely many dimensions. When the constraint set is the lattice (unbounded lattice) in finitely/infinitely many dimensions and the number of state variables is not finite but countable, the objective function is a power series each of whose terms is a quadratic/higher degree form. It is proved in (Rama 3) that some of the constrained optimization problems arise in the design of multi/infinite dimensional codes. In (Rama 4), various optimization problems described above are utilized in dynamic optimization setting. In the following , various innovative themes in multi/infinite dimensional neural networks are briefly discussed.


25

Continuous Time Neural Networks The well known model of a neural network is a discrete time system in one or multiple dimensions. A signal design problem for optical/magnetic recording channels modeled as linear systems, led to the discovery of continuous time neural networks ( Rama 5). The state updating scheme of the continuous time neural network takes the following form T

X(t) = Sign ( ∫ R(t , s) X( s) ds) 0

(2.23)

In this technical memorandum, the author for the first time associates energy functions with the state updating scheme. The multidimensional versions of these continuous time neural networks are discussed in (Rama 4).

Complex Neural Networks Neural networks in which the entries of the connection structure as well as state variables (indicating the binary states of the neuronal networks) are complex valued are already studied in one dimension. These results have the corresponding multidimensional versions. These results parallel the results for real neural networks. These results are aided by the fact that the quadratic form associated with a Hermitian symmetric matrix is always real and thus the eigenvalues of the Hermitian symmetric matrix are always real.

Adaptive Neural Networks These are neural networks in which the connection structure of the one/multidimensional neural network is varying with discrete/continuous time index. More explicitly, the connection tensor whose elements constitute the synaptic weights between the neurons that are located in one/two/multiple dimensions is varying with the time index in some orderly ( or random ) manner. The analysis of such one/multidimensional neural networks is being studied.

2.7 CONCLUSIONS A mathematical model of an arbitrary multidimensional neural network is described. This model is utilized to prove the convergence theorem for multidimensional neural networks. Utilizing the convergence theorem, multidimensional logic functions are defined and multidimensional logic synthesis is discussed. Infinite dimensional logic synthesis is briefly described. Various constrained static optimization problems of utility in control, communication, computation and other applications are summarized. Several innovative themes on one/multidimensional neural networks are summarized.

26


REFERENCES (BoT) A. I. Borisenko and I. E. Tarapov, “Vector and Tensor Analysis with Applications,“ Dover Publications Inc., New York, (BrG) J.Bruck and J.W. Goodman, “A Generalized Convergence Theorem for Neural Networks”, IEEE Transactions on Information Theory, Vol. 34, No. 5, Sept 88. (CAB) S.T. Chakradhar, V.D. Aggarwal and M.L. Bushnell, “Neural Models and Algorithms for Digital Testing”, Kluwer Academic Publishers. (HoT) J. J. Hopfield and D. W. Tank, “Neural Computations of Decisions in Optimization Problems,“ Biological Cybernetics., Vol. 52, pp. 41-52, 1985. (Rama 1) Garimella Rama Murthy, “Multi/Infinite Dimensional Logic Synthesis,“ Manuscript in Preparation. (Rama 2) Garimella Rama Murthy, “Unified Theory of Control, Communication and Computation-Part 1,” Manuscript to be submitted to IEEE Proceedings. (Rama 3) Garimella Rama Murthy, “Multi/Infinite Dimensional Coding Theory: Multi/Infinite Dimensional Neural Networks: Constrained Static Optimization,” Proceedings of 2002 IEEE Information Theory Workshop, October 2002. (Rama 4) Garimella Rama Murthy, “Optimal Control, Codeword, Logic Function Tensors— Multidimensional Neural Networks, International Journal of Systemics, Cybernetics and Informatics, October 2006, pages 9-17. (Rama 5) Garimella Rama Murthy, “Signal Design for Magnetic and Optical Recording Channels: Spectra of Bounded Functions, “ Bellcore Technical Memorandum, TM-NWT-018026.

Multi/Infinite Dimensional Coding Theory: Multi/Infinite Dimensional Neural Networks

27

CHAPTER

3

Multi/Infinite Dimensional Coding Theory: Multi/Infinite Dimensional Neural Networks —Constrained Static Networks— Optimization

3.1. INTRODUCTION In the recent years, technological developments in parallel data transfer mechanisms led to HIPPI (high performance parallel interface), SMMDS (switched multi-megabit data service), FDDI (fiber distributed data interface). To match these high speed parallel data transfer mechanisms, multidimensional coding theory has been originated and some ad hoc procedures were developed for designing linear as well as non-linear codes. Multidimensional codes are utilized to encode arrays of symbols for transmission over a multidimensional communication channel. Thus, the central objective in multidimensional coding theory is to design codes that can correct many errors and whose encoding/decoding procedures are computationally efficient. A multidimensional error correcting code can be described by an energy landscape, with the peaks of the landscape being the codewords. The decoding of a corrupted codeword (array) which is a point in the energy landscape that is not a peak is equivalent to looking for the closest peak in the energy landscape. An alternative way to describe the problem is to design a constellation which consists of a set of points on a multidimensional lattice that are enclosed within a finite region, in such a way that a certain optimization constraint is satisfied. Neural network model, simulated annealing, relaxation techniques are some of the various computation models (based on optimization) that have been attracting much interest because they seem to have properties similar to those of biological and physical systems. The standard computation performed in a neural network is the optimization of the energy function. The state space of a neuro-dynamical system can be described by the topography defined by the energy function associated with the network. The connection structure of a neural network can either be distributed on a plane or in multidimensions (Rama 2). Thus, the field of multidimensional neural network theory and the field of multidimensional coding theory are linked through the common thread of optimization of

28


multivariate polynomials (tensor based) over various subsets of the multidimensional lattice. In a nut shell, multidimensional error correcting codes and multidimensional neural networks can be associated with such polynomials. In contrast to the traditional ad hoc attempts to design multidimensional codes by a generation of researchers, the author for the first time discovered and formalized the idea of utilizing the theory of tensor spaces to represent and study multidimensional error correcting codes. The theory of tensor spaces enables the design of codes in one dimension (encoding as well as decoding techniques) to be translated to multi/infinite dimensions. Utilizing this representation, the author took a significant step forward in formally demonstrating the relationship between multidimensional neural networks, multidimensional codes and optimization of multivariate polynomials/monomials over various subsets of multidimensional lattice. This relationship provides new insights into the design of multidimensional encoders as well as decoders. Also, the relationships between concepts such as minimum distance, correctable errors of multidimensional codes can be derived through new proof arguments. Furthermore, the relationship enables the utilization of multidimensional decoding techniques for the solution of optimization of multivariate polynomials over the multidimensional hypercube ( other subsets of multidimensional lattice), a difficult problem that arises in various applied fields such as operations research, theoretical computer science etc. Also, utilizing the powerful techniques developed in these applied areas for such problems, new algorithms for maximum likelihood decoding of multidimensional error correcting codes can be designed. Thus, the results in this chapter are summarized in the following three paragraphs. The concepts of multidimensional neural networks, multidimensional error correcting codes, optimization of quadratic/higher degree forms based on components of a tensor (tensor component based multivariate polynomials), over various subsets of multidimensional lattice, are related from different viewpoints. It is proved that given a multidimensional linear block code, a neural network (generalized neural network) can be constructed in such a way that every local maximum of the energy function corresponds to a codeword tensor and every codeword tensor corresponds to a local maximum. It is shown that determining the global maximum of the energy function of a multidimensional neural network/generalized neural network is equivalent to performing the maximum likelihood decoding in a linear block multidimensional code. The results are generalized to multidimensional non-linear as well as non-binary codes. Theorems related to optimization of tensor based multivariate polynomials (terms/ monomials are based on the components of tensors) over arbitrary open/closed sets are proved. Infinite dimensional extension of the results is briefly discussed. This chapter is organized as follows. In section 2, after briefly reviewing the theory of multidimensional neural networks, it is proved that finding the global optimum of the energy function of the network is equivalent to finding a minimum cut in a certain


29

graphoid, the connection structure of a multidimensional neural network. In section 2, a connection between the multidimensional neural network model and graphoid based codes is established. It is shown that maximum likelihood decoding in a graphoid based code is equivalent to finding a minimum cut in a certain graphoid. Thus, it is shown that maximum likelihood decoding in a graphoid based code is equivalent to finding a maximum of the energy function in a multidimensional neural network. In section 3, the results are extended to general multidimensional linear block codes. A general energy function, not necessarily quadratic, is defined based on the generator tensor of a given linear block code. It is proved that finding the global maximum of the energy function is equivalent to maximum likelihood decoding in the code. In section 3, it is briefly discussed how the infinite dimensional codes are represented through the infinite order/dimension (either the order or the dimension or both is infinite) generator tensor (the entries of which satisfy some regularity conditions) and thus enable the infinite dimensional versions of the results to be derived. In section 4, the energy function associated with the parity check tensor of the multidimensional linear block code is described. When the tensor is written in the systematic form, it is shown that each codeword tensor corresponds to a local maximum of the multivariate polynomial associated with the parity check tensor and that each local maximum corresponds to a codeword tensor. The results are interpreted as the dual to the ones in the previous section for defining the Maximum Likelihood Decoding (MLD) problem. In section 5, the results are generalized to nonbinary codes. Further, in section 6, the results are generalized to non-linear multidimensional codes. In section 7, by means of a decomposition principle, theorems related to optimization of tensor based (based on the components of a tensor) multivariate polynomials over arbitrary open/closed sets are proved. Also, various innovative ideas on the utilization of results in previous sections, to derive very general results in static optimization are described. The chapter concludes with a summary of results derived. The results in this chapter are exactly the multidimensional versions of those in (BrB).

3.2 MULTIDIMENSIONAL NEURAL NETWORKS: MINIMUM CUT COMPUTATION IN THE CONNECTION STRUCTURE: GRAPHOID CODES A discrete time multidimensional neural network is a discrete time dynamical system represented by a weighted undirected connectionist structure in multidimensions. At each multidimensional neuronal element, there is a threshold value which will fire each neuron on crossing it. Each neuronal element computes an algebraic threshold function in the input variables. Let MN be a multidimensional neural network of dimension m and order n; then MN is uniquely specified by (S, T) where ( the number of neurons in each dimension is m i.e. the number of values assumed by each independent dimension variable) S is fully symmetric tensor of dimension m and order 2n, and T is a tensor of thresholds attached to neuronal elements with compatible order ( n ) and dimension ( m ). Every node can be in one of two


30

possible states +1 and –1. The state of node ( i1, i 2,..., in ) at time t is denoted by Xi 1, i 2..., in (t) . The state of MN at time t is the tensor X i1, i2,...1. in (t)of dimension m and order n. The state evolution at node ( i1, i 2,..., in ) is computed by

Xi 1, i 2,..., in (t + 1) = Sign (Hi 1, i 2,..., in (t ))

(3.1)

where

Hi 1,..., in (t ) =

m

m

j1= 1

jn = 1

∑ ... ∑ Si1,..., in; j1,..., jn X j1, j 2,..., jn (t) − Ti1,..., in (t)

The next state of the network i.e. X i1, i2,..., in (t + 1), is computed from the current state by performing the evaluation (3.1) at a subset of nodes of the multidimensional neural network, to be denoted by G. The modes of operation are determined by the method by which the subset G is selected in each time interval. If the computation (3.1) is performed at a single node in any time interval i.e. G| = 1 , then we will say that the network is operating in the serial mode, and if G|= mn , then we will say that the network is operating in the fully parallel mode. A state is called stable if and only if

Xi 1, i 2,..., in (t) = Sign (S ⊗ Xi 1,..., in (t) − Ti 1,..., in )

(3.2)

where ⊗ denotes inner product (the symbol is sometimes suppressed for notational brevity). Once a neural network reached such a state there is no change in the state of the network no matter what the mode of operation is. An important feature of the network MN is the convergence theorem stated below. Theorem 3.1: Let MN = (S, T) be a multidimensional neural network of dimension m and order n. S is a fully symmetric tensor of order 2n and dimension m . The network MN always converges to a stable state while operating in the serial mode (i.e. there are no cycles in the state space) and to a cycle of length utmost 2 while operating in the fully parallel mode.( i.e. cycles in the state space are of length utmost 2 ). This theorem is proved in (Rama 2). This theorem suggests the utilization of MN as a device for performing a local search of the optimum of an energy function. In the following, we formulate a problem that is equivalent to determining the global maximum of an energy function and how to map it onto a multidimensional neural network. Definition 3.1 Let G = (V, E) be a weighted and undirected non-planar graph in multidimensions where V denotes the set of nodes of G and E denotes the set of edges of G. Let K be the fully symmetric tensor whose components are the weights of the edges of G. Let V1 be a subset of V, and let V–1 = V–V1. The set of edges each of which is incident at a node inV1 and at a node in V–1 is called a cut in G.


31

Definition 3.2 The weight of a cut is the sum of its edge weights. A minimum cut (MC) of a non-planar graph/graphoid is a cut with minimum weight. In the following, we show the equivalence between the minimum cut problem in a graphoid (from now onwards, we call the connection structure of a multidimensional neural network also as a graphoid ) and the problem of maximizing the quadratic form as the energy function of a multidimensional neural network. Every non-planar graph including the connection structure of a multidimensional neural network is a Graphoid (by definition). Theorem 3.2: Let MN = (S, T) be a multidimensional neural network with all the thresholds being zero i.e. T = 0. The problem of finding a state V for which the quadratic energy function E is maximum is equivalent to finding a minimum cut in the graphoid corresponding to MN. Proof: Since T = 0, the energy function is given by

E=

m

m

m

m

∑ ... ∑ ∑ ... ∑ S

i1= 1

in = 1 j 1 = 1

jn = 1

i 1,..., in ; j 1,..., in

Xi 1,..., in

X j 1,..., jn

(3.3)

Let (i1,i2,...,in) = i,(j1,...,jn) = j. Let S + + denote the sum of weights of edges in MN with both the end points being in the same vertex set of the cut i.e. i=j =1, and let S − − ; S + − denote the corresponding sums of the other two cases. It follows that E = 2 ( S++ + S− − − S+− ) which can also be written as E = 2 ( S++ + S− − + S+− ) – 4 S+ –

(3.4)

Since, the first term in the above equation is constant (it is the sum of weights of the edges), it follows that the maximization of E is equivalent to the minimization of S+–. Clearly, S+– is the weight of the cut in MN with V1 being the nodes of MN with a state equal to 1. Q. E. D.

Graphoid Based Codes In this sub section, relationship between multidimensional neural networks and error correcting codes based on graphoids is investigated. The ‘multidimensional error correcting codes’ associated with graphoids (connection structure of a multidimensional neural network ), are called “graphoid-theoretic” codes. The family of graphoid codes are defined based on the tensors naturally associated with the connection structure of a multidimensional neural network with nodes as well as edges. Let G = (V, E) be an undirected connectionist structure of a multidimensional neural network with weights on the edges. Like a graph in the plane, this is a representation for a


32

non-planar graph type structure called graphoid (not necessarily the connection structure of a multidimensional neural network). Consider a fully symmetric tensor of dimension m and order 2n; which is utilized to describe the connection structure of a multidimensional neural network. A subset of the set of edges of G can be represented by a characteristic tensor of order 2n with the edge between two nodes Vi 1, i 2,..., in , Vj 1, j 2..., jn , leading to an entry of +1 at those locations in the tensor. Thus, an edge characteristic tensor of a graphoid E is defined such that

Eˆ i1,..., in; j1,..., jn

 1 if an edge is incident between  =  nodes(i1,...., in )and ( j1,...., jn ).  0 otherwise. 

(3.5)

Definition 3.3 The incidence tensor of a graphoid G = (V , E ) is a block tensor of the form

TVˆ1    TVˆ2  DGˆ = .  ..  (3.6)   TVˆ   n   where TVˆ represents the tensor of the set of edges incident upon the node Vi . It should be noted that the incidence tensor is a blocked tensor and the above illustration is shown to aid the imagination of the reader. Various concepts associated with planar graphs are utilized as the basis to define the following concepts associated with a graphoid (non-planar). They provide the notation associated with graphoid theoretic codes. The following lemmas are very easy to verify. Lemma 3.1: The set of characteristic tensors that correspond to the cuts in a connection i

structure G = (V , E ) of a multidimensional neural network form a linear tensor/m-d vector (depending on the notational convenience) space over GF(2) in multidimensions of dimension

( V − 1)

.

The linear tensor/m-d vector space that corresponds to the cuts of a graphoid Gˆ will be called the only cut space of Gˆ . Furthermore, the circuits in a graphoid also constitute a linear tensor/vector space. Lemma 3.2: Given a connected graphoid G = (V , E ) ; the incidence tensor of Gˆ has rank

Multi/Infinite Dimensional Coding Theory: Multi/Infinite Dimensional Neural Networks ( V − 1)

33

. Every block tensor in DGˆ associated with a node is a characteristic tensor of a cut

(

)

and every Vˆ − 1 block tensors of DGˆ corresponding to different vertices/nodes of the graphoid form a basis for the cut space of Gˆ .

Hence, given a connection structure Gˆ , the cut space of the graphoid is a

(

)

multidimensional linear block code of dimension Vˆ − 1 . For the sake of brevity, in the following, we only consider ‘cut codes’. Given a graphoid, Gˆ , an interesting question is how to formulate the maximum likelihood decoding (MLD) problem of the code CGˆ in a graphoid-theoretic language.

(

)

That is, given a graphoid Gˆ = Vˆ , Eˆ and a (0, 1) tensor Y of dimension m and order 2n; what is the codeword in C ˆ closest to Y in Hamming distance? G The following lemmas will answer the questions. Hamming Distance: Given two (0,1) tensors, X, Y; the Hamming distance between m dimensional tensors of order 2n is the number of places where they differ. This definition is motivated by transmitting a binary tensor X through a noisy multidimensional channel, observing the output Y and counting the number of errors that have occurred.

(

)

Lemma 3.3: Let Gˆ = Vˆ , Eˆ be a graphoid. Let CGˆ be the multidimensional code associated with Gˆ . Let Y be a (0,1) tensor of order 2n (dimension m). Construct a new graphoid, to be defined/denoted by Gˆ Y; by assigning weights to the edges of Gˆ as follows: Wi1, i2,..., in; j1,..., jn = (–1) Yi1,..., in; j1,..., jn ((−1)Power…) (3.7)

Wi 1, i 2,..., in ; j 1,..., jn is the weight associated with the edge (i1,..., in ; j1,..., jn ) in Gˆ . Then the maximum likelihood decoding of the tensor Y with respect to CGˆ is equivalent to finding the minimum cut in Gˆ Y . Proof: Assume the number of ones in Y is b. Let P be an arbitrary codeword in CG. Let L i,j denote the number of positions in which P contains an i ∈ {0, 1} and Y contains a j ∈ {0,1}. Clearly,

b = L0,1 + L1,1

(3.8)

−L1,1 + L1, 0 = L0,1 − b + L1,0

(3.9)

Thus,

= L0,1 + L1,0 − b

(3.10)


34

Minimizing the right hand side of the above expression over all P ∈ C G is equivalent to finding a codeword which is the closest to Y. On the other hand, minimizing the left hand Q.E.D side is equivalent to finding the minimum cut in G Y. From the above lemma, the following theorem follows.

(

)

Theorem 3.3: Let Gˆ = Vˆ , Eˆ be a graphoid. Then, maximum likelihood decoding of a tensor word Y with respect to CGˆ is equivalent to finding the maximum of the quadratic energy function E of the multidimensional neural network defined by the graphoid Gˆ Y with all its threshold values equal to zero. Proof: By Lemma 3.3, maximum likelihood decoding of Y with respect to C Gˆ is equivalent to finding the minimum cut in Gˆ Y . By Theorem 3.2, finding the minimum cut in a graphoid is equivalent to finding the global maximum of the energy function (quadratic) of a multidimensional neural network defined by a graphoid with all the thresholds at each neuronal element set to zero. Q.E.D. Graphoid based error correcting codes are very limited since the connection structure of a multidimensional neural network is represented by a fully symmetric tensor. This imposes restrictions on the minimum distance of multidimensional codes. Thus, a natural question that arises is whether the equivalence stated above in the Theorem 3.3 can be generalized to arbitrary multidimensional linear block codes. Graphoid codes arose naturally out of the topological properties of the connection structure of a multidimensional neural network. The connection structure required a fully symmetric tensor to represent it. The neural network model enabled the association of a quadratic energy function with the fully symmetric tensor and its optimization over the multidimensional hypercube. Thus, the encoders and decoders of graphoid codes are defined through topological structure and optimization of multivariate polynomials. Since, an arbitrary tensor like the fully symmetric tensor constitutes a linear operator, unlike graphoid codes, arbitrary multidimensional linear codes are first defined through their algebraic structure in the next section. Then the maximum likelihood decoding problem of such codes is discussed.

3.3 MULTIDIMENSIONAL ERROR CORRECTING CODES: ASSOCIATED ENERGY FUNCTIONS—GENERALIZED NEURAL NETWORKS Recent advances in high speed parallel data transfer mechanisms based on light wave/ optical networks motivated the design and analysis of multidimensional codes. Several researchers utilized ad hoc techniques (sometimes pseudo-mathematical techniques) to design and analyze multidimensional codes based on the extensions of the ideas in one dimensional error control coding theory. The author for the first time developed the idea of utilizing ‘tensor linear operator’ for the design and analysis of multi/infinite dimensional linear as well as non-linear codes conceived as sub-spaces over tensor spaces.


35

A multidimensional linear codeword constellation is defined as the subspace of a tensor linear space defined through the generator tensor. The encoding operation of a multidimensional (m, n ; m,l) linear code, defined by an m dimensional (n+l) order generator tensor Gi 1, i 2,..., in ; j 1, j 2,..., jl is performed in the following manner: An m dimensional information tensor of order n, Bi1, i2,..., in (with 0, 1 symbols) is encoded into the m-dimensional “codeword tensor”, C j 1, j 2,..., jl (constellation member) of order ‘l’ by the following tensor inner product (outer product followed by contraction over appropriate indices) scheme:

Bi 1, i 2,...., in ⊗ Gi 1, i 2,...., in ; j 1, j 2,..., jl

= C j 1, j 2,..., jl

(3.11)

where ⊗ denotes the inner product operation (between tensors defined over a finite field) by means of exclusive or operation between the components of outer product of tensors (contraction over appropriate indices of the sum of products of binary variables). The above procedure of generating the codeword tensor from an information tensor leads to the following interesting considerations which are inherent to multidimensional code design. In one dimension, a binary information vector of length k is encoded into a codeword vector of length n by padding the parity bits to it. The parity check equations obtained through the parity check matrix determine these bits. In the case of two/multidimensional array of information bits, there are many ways to encode the array into a codeword array. Even in the simplest two dimensional array case, by padding a border of parity bits along the row wise as well as column wise directions, the codeword array can be generated. In the following, this degree of freedom in multidimensional coding is formally described. A multidimensional information array (information tensor) is mapped into a codeword array in the following ways: (1) An m-dimensional information tensor of order n is mapped into an m-dimensional codeword tensor of order l (l > n), (2) An m -dimensional information tensor of order n is mapped into k -dimensional codeword tensor (k > m ) of order n, (3) An m-dimensional information tensor of order n is mapped into a k -dimensional (k > m) codeword tensor of order l (l > n). For the purpose of notational convenience, in the following encoding through the operation (1) is only utilized. It is easy to realize that by transposing the information as well as generator tensors, the operation (2) in encoding is achieved. But to encode an information tensor into a generator tensor through the operation (3), a second generator type tensor is utilized. Various ideas familiar in one dimensional coding theory (parity check matrices, primitive polynomials, basis, cosets etc.) have corresponding parallels in multi/infinite dimensional coding theory based on the tensor linear operator defined over a finite field. The detailed translation from one dimensional encoding/decoding algorithms to

36


multidimensional encoding/decoding algorithms is done by utilizing tensor algebra concepts with parallel linear algebra concepts.

Infinite Dimensional Codes Now let us consider infinite dimensional codes. An infinite dimensional tensor can be of the following types: (a) the dimension of the tensor is finite, whereas the order is infinite, (b) the dimension of the tensor is infinite, whereas the order is finite, (c) the dimension as well as order of the tensor are infinite. An infinite dimensional code can be generated in the following manner. It is assumed that the generator tensor of the code is such that either the dimension or the order or both are infinite. Also, it is assumed that the entries of the generator/information tensor satisfy the regularity conditions necessary to ensure that the inner product makes sense (convergence of the partial sums of outer product to a limit etc.). (i) An information tensor of finite dimension/order is mapped into a codeword tensor of infinite dimension/order. This type of encoding can happen in practical multidimensional communication systems, (ii) An infinite dimension/order ( or both are infinite) tensor is mapped into a codeword tensor with either the dimension or the order or both being infinity. In the above encoding schemes, the generator/parity check tensors are of compatible dimension/order (with the information tensor being encoded) to ensure that a proper infinite dimensional codeword is generated. Infinite dimensional extensions of the results in sections 3, 4, 5, 6, 7 (to be described in the following paragraphs) follow from the immediate extensions of the formal arguments to infinite dimensional tensors that satisfy the regularity conditions. They are not explicitly repeated. In the following, a very brief summary of multidimensional information theory is provided as it is based on the tensor linear space structure idea necessary to model multidimensional arrays.

Multi/Infinite Dimensional Information Theory In one dimension, a mathematical theory of communication is developed utilizing the concepts of information/entropy associated with a random variable, conditional entropy, joint entropy etc. These concepts are the vital tools to prove the noiseless channel coding theorem. Various channel models are developed. The concepts of mutual information, capacity of a discrete memoryless channel are utilized to prove the second channel coding theorem. One dimensional information theory then led to rate distortion theory. In multidimensions, a source generates multidimensional arrays of information which pass through a multidimensional channel. A multidimensional independent, identically distributed information array of symbols is associated with the concept of entropy H (Xi1, i2,..., in) in the following manner:


H (Xi 1, i 2,..., in ) =

m

m

i1= 1

in = 1

∑ ...∑ Pi1, i 2,..., in log (1 Pi1, i 2,..., in )

37

(3.12)

Given the basic idea of the above definition, results from one dimension are generalized to multidimensions utilizing the principles described in (Rama 3). Complex sources such as a Markovian Source require some sophistication in defining the entropy/uncertainty of the source. The interesting channel model in multidimensions is the discrete memoryless channel represented through a stochastic tensor whose elements are conditional probabilities Pj 1,..., jl , i 1,..., ik . This corresponds to a Markov random field. Detailed theorems are derived utilizing the principles described in (Rama 3). With the multidimensional encoding scheme formally described, it is proved in the following that the maximum likelihood decoding problem of a multidimensional linear block code is equivalent to the maximization of multivariate polynomial (whose terms/ monomials are described in terms of the entries of received, generator tensors) associated with the generator/received tensors over the multidimensional hypercube. The essential idea in the derivation of the desired result is (generalization of Theorem 3.3 to arbitrary multidimensional linear codes) to represent the symbols of the additive group as symbols in the multiplicative group through the following transformation: (3.13)

a → ( − 1) a i .e . 0 → 1, 1 → − 1 .

Thus, the information tensor Bi 1,..., in is represented by the tensor Xi 1,..., in , where the component Xi 1,..., in =

(–1)

Bi 1,...in

. The encoded codeword C j 1,..., jl is thus represented by the

tensor Yj 1,..., jl . Hence, a component of the tensor Y is given by C j1 ,...., jl

Yj1, j2 ,.... Jl = ( −1)

m

m

i1 = 1

in = 1

i1,.....,in = ∏ ...∏ X i 1,...., in

G

; j1,...... jl

(3.14)

Definition 3.4 In the {1, –1} representation of a multidimensional linear code, instead of a generator tensor, given an information tensor Xi 1,..., in , an encoding procedure X → Y is utilized, where the tensor Y j1,..., jl is such that Y j1,..., jl component is a monomial that consists of a subset of the

X i1,..., in . An encoding procedure is systematic if and only if Y j1,..., js = X j1,..., js for 1 < s < n. Definition 3.5 Let Gi1, i 2,..., in ; j1, j 2,...., jl be a generator tensor of ones and zeroes. The polynomial representation of generator tensor G with respect to a {+1, –1} received tensor of dimension m and order l, W denoted by E is,


38 m

m

i1 = 1

in =1

EW ( X ) =W ⊗ ∏...∏ X i1,...,in = W ⊗Y (X )

(3.15) (3.16)

where ⊗ denotes inner product between the tensors (i.e. outer product of the tensors followed by contraction over appropriate indices). Consider the linear multidimensional block code defined by the generator tensor G (or equivalently by the encoding procedure associated with G ). The polynomial representation of G i.e. EW ( X ), will be called the energy function of W with respect to the encoding procedure X → Y . To establish the connection between the energy functions (optimized by neural/ generalized neural networks over various subsets of the multidimensional lattice) and linear multidimensional block codes, we will prove that finding the global maximum of EW (X) is equivalent to maximum likelihood decoding of a tensor W with respect to the code C. Theorem 3.4: Given an ( m, l ; m, n ) multidimensional linear block code C defined by an encoding procedure X → Y , and a tensor W of ones and minus ones i.e. a {+1, –1} tensor, the closest codeword (in Hamming distance) to W in C corresponds to an information tensor B if and only if EW (B) = Maximum overall tensors X of {EW(X)}. (3.17) Proof: For an { +1, –1 } information tensor,X the scalar energy function is given by EW (X) = W ⊗ Y(X)

(3.18)

= {( j1, j 2,..., jl ): W j1,..., jl = Y j1,..., jl ( X )} − {( j1,..., jl ): W j1,...., jl ≠ Y j1,...., jl ( X )}

(3.19)

= ml − 2 {( j 1,... jl ): Wj 1,..., jl ≠ Yj 1,..., jl (X )} = ml − 2 dH (W , Y)

(3.20)

where d H denotes the Hamming distance between the multidimensional codewords W, Y . From the above expression, EW ( B ) will achieve a maximum if and only if d H (W , Y) achieves a minimum. Q. E. D.

Minimum Distance of Linear Multidimensional Block Codes Given an encoding procedure, we can use the same argument as in the above theorem, to express the minimum distance of the code. Consider the encoding procedure:

X = (Xi 1,..., in ) → Y = (Yj 1,..., jl ) and the energy function with W, a tensor with all the components equal to one.

(3.21)


EW (X ) =

m

m

j1= 1

jl = 1

∑ ...∑ Yj1 ,..., jl

39

(3.22)

i.e. sum of all the components of tensor Y. As in Theorem 3.4,

EW ( X ) = m l − 2d H ((all ones tensor), Y )

(3.23)

and minimum over all X tensors ≠ (all ones tensor) of d H ((all ones tensor), Y) occurs at M =Maximum overall tensors other than all ones tensor of EW (X )

(3.24)

Thus, d * ( the minimum distance of the code is given by ) d* =

ml – M 2

(3.25)

The above results are being generalized to infinite dimensional codes utilizing infinite dimension/order tensors . Game-Theoretic Codes: Optimal Codes In the theory of error correcting codes, minimum distance of a linear code provides a measure of the number of errors that can be corrected. From ( 3.25), it is evident that the maximization of minimum distance of a multidimensional linear block code requires minimizing M. Thus, we have the following Lemma. Lemma 3.4: The multidimensional (m, n ; m, l) linear block code which minimizes M in (3.24) enables the correction of maximum number of errors among all possible such error correcting codes: Proof: From (3.25), maximization of minimum distance of an (m, n ; m , l) linear code is equivalent to minimizing M, i.e. minimizing the maximum value of the energy function over the m-d hypercube ( excluding the all ones tensor). Such a code design problem fits in the game-theoretic framework. It is well known that maximization of minimum distance also maximizes the number of errors that can be corrected. Q. E. D.

3.4 MULTIDIMENSIONAL ERROR CORRECTING CODES: RELATIONSHIP TO STABLE STATES OF ENERGY FUNCTIONS Let C be a linear multidimensional block code (over GF(2)) defined by the generator tensor G. Let EC be a polynomial over the components of {+1, –1} tensors (energy function) with the property that every local maximum in EC corresponds to a codeword in C and every codeword in C corresponds to a local maximum in EC. An interesting problem is,


40

given a code C defined by G, the generator tensor, is there an efficient algorithm to construct to EC ? In the following, the above problem is solved by considering the parity check tensor of a multidimensional linear block code. Consider an (m, l ; m , n) linear multidimensional block code. Without loss of generality, let us consider the generator tensor G given in the systematic form i.e.

Gj 1, j 2,..., jn = I j 1,..., jl Pjl + 1,..., jn

(3.26)

where I j 1,..., jl is an identity tensor of compatible order. The parity check tensor of C is denoted by H and is given by

P  HT = H j ( n − 1),..., ,..., j 1 =  I  (3.27)   i.e. a blocked tensor with sub-tensors of compatible dimension and order. From the definition of a parity check tensor of a multidimensional linear block code, C j 1,..., jn ⊗ H j ( n −1),..., ,..., j 1 = 0

(3.28)

where the multidimensional tensor codeword on appropriate/compatible inner product (outer product followed by contraction over the appropriate indices) with the parity check tensor gives the zero tensor. The above equation can be rewritten using the polynomial representation of generator tensor devised in the previous section (with the tensor of coefficients being the all-ones tensor. It should be noted that the all-ones tensor in the {1, –1} representation corresponds to all-zero tensor in the {0,1} representation). Lemma 3.5: Let E (X) be the polynomial representation of parity check tensor HT with respect to the all ones tensor. Then, X ∈ C , the multidimensional linear block code if and only if

E (X) = m(n–l). Proof: E , the polynomial representation of parity check tensor has m (n–l) terms, and all the coefficients are equal to one. Hence, E = m(n–l). if and only if all the terms are equal to one. Q. E. D. The above Lemma ensures that in the polynomial representation, E (X), every codeword corresponds to a global maximum (stable state). An interesting question is, does every local maximum correspond to a codeword. This question is answered by the following theorem. Theorem 3.5: Let C be a linear multidimensional block code, with G, H, EC, and E as defined above. Then E is a polynomial with the properties of EC. That is, X corresponds to ∈C. a local maximum in E if and only if X∈


41

Proof: From the above Lemma, the global maximum of E is m (n–l) ; thus every codeword is a global ( and thus a local ) maximum. The converse follows from the fact that the tensor H has a systematic form. Specifically, the last m (n–l) variables in E ; i.e., xi1,i 2 ,...., in – l +1 ,...., in ; where the order indices iˆn −l +1 ,..., iˆn (each of them) assume m values, each appear only in

one term. That is, since I is an identity tensor in the parity tensor H; x i 1,..., iˆn – l +1 ,...., iˆn appears only in first term, and so on. Now, assume that a tensor V exists that corresponds to a local

(n−l ) maximum (which is not global maximum). That is E (V) = L, where L < m . Hence, at least one term exists in E (V ) that is not one. However, this can be made one by flipping the value of the index variables that appear in this term. This contradicts the fact that V is a local maximum. Q. E. D. To summarize, given a linear code C, the algorithm for constructing a polynomial is as follows:

(1) Construct the systematic generator tensor of C by the standard techniques in tensor algebra, (2) Construct the systematic parity check tensor of C in accordance with (3.27) (3) Construct E , which is the polynomial representation of H with respect to the allones tensor. By the above Theorem 3.5, EC = E . In the following, generalizations of the above results are discussed. Also, some important comments, remarks are provided. (A) The construction just described also works for cosets of linear multidimensional block codes. Let W be a tensor of dimension m and order (n – l) of the coefficients of E. In the construction described above, the all-ones coefficient tensor was chosen and it was concluded that EC = E . It corresponds to the all-zero syndrome tensor. Let C be a coset of C, and let T be the syndrome which corresponds to C. Utilizing the proof argument of Theorem 3.5, it can be proven that a one-toone correspondence exists between the local maxima of polynomial representation of the parity check tensor H with W = T and the tensors in the coset C. Clearly, the syndrome that corresponds to the code C is the all-ones tensor (by noting that in the transformation in section 3, 0 goes to 1). (B) The construction described in this section is a dual way of defining the maximum likelihood problem (MLD) (with respect to the one suggested in section (3)). Consider a linear multidimensional block code defined by the parity check tensor H. Given a tensor V, the maximum likelihood decoding (MLD) problem can be defined as finding the local maximum in EC closest to V or, equivalently, finding a local maximum of the energy function associated with the syndrome (corresponding to V) that is achieved by a tensor of minimum weight. The above results are generalized to some infinite dimension/order tensors in a straightforward manner. In the following section, the above results are generalized to non-binary codes.


42

3.5. NON-BINARY LINEAR CODES In the following, results on maximum likelihood decoding of non-binary linear codes are discussed. Consider a linear multidimensional block code over a finite field GF(p) with p being a prime. For the sake of notational convenience, we first consider an (m, k ; m, n) linear multidimensional block code which maps a transmitted input tensor of dimension m, order k into a codeword tensor of dimension m and order n. Let G denote the generator tensor of the code which maps the ( m, k ) input tensor into a (m, n ) codeword tensor. Then, m k symbols of the input tensor B in Zp are encoded into the codeword V by the procedure:

Vi 1, i 2,..., in = (Bi 1, i 2,..., ik ⊗ Gi 1,..., ik ; j 1,..., jn ) mod p

(3.29)

The essential idea is once again to utilize the multiplicative representation. Let u be the p th root of unity i.e. µ = e ( j 2 Π )/ p (3.30) The additive Zp group can be represented as a multiplicative group of p th roots of unity through the transformation: a → u a In the multiplicative representation, the information symbols in information tensor are represented as X i 1,..., ik = u Bi 1,....,ik

(3.31)

Thus, the encoded codeword tensor V, is represented by a new tensor Y, where

Yi 1,..., in = u

Vi1 ,..., in

= u

m m .. . ∑ ( B i 1 ,. .. , ik ∑ i 1 = 1 ik = 1

m

m

i1= 1

ik = 1

m

m

i1= 1

ik = 1

= ∏ ...∏ u

G i 1 ,. .. , ik ; j 1 ,. .. , j n ) m o d p

(3.32)

Bi1 ,..., ik Gi1 ,...,ik ; j1 ,..., jn

= ∏ ...∏ Xi1,...,i 1,...,ik ik ; j 1,..., jn G

Hence, as in the case of a binary linear multidimensional code, we can represent a multidimensional code over a field (finite) with p elements (p is a prime) by an encoding procedure. The elements are now p th roots of unity. Thus, given an information tensor

X = (Xi 1,..., ik ) , we have the one-to-one assignment X = (Xi1,.....,i k)

–→

Y = (Yi1,....in)

(3.33)

where Y = (yi 1,..., in ), is a monomial. We discuss the maximum likelihood decoding problem with respect to two different distance measures. In the first generalization, we consider solving the maximum likelihood


43

decoding (MLD) problem with the metric being the Hamming distance between the tensors while in the second case, we consider the Lee distance. The generalization for the case where the Hamming distance is utilized in the maximum likelihood decoding (MLD) problem is based on the following well known Lemma. Lemma 3.6: Let p be a prime, and let ( j 2 π

p ). Assume KE{0,1,2,...,(p − 1)}µ=e then

1, if k = 0, = m =0 0, othe rw ise The generalization is stated through the following theorem. (1 p )

( p −1)

∑u

km

(3.34)

Theorem 3.6: Consider an (m, k ; m, n) multidimensional linear block code over GF(p), with p

p being a prime. Let X → Y be the corresponding encoding procedure. Let EW be the following multivariate polynomial representation of the generator tensor G with respect to an arbitrary received tensor W : p

EW (Y ) =

( p −1)

∑ (W l=0

• i 1,..., in

⊗ Yi 1,..., in )

(3.35)

where W• denotes the complex conjugate of W and ⊗ denotes the inner product between the tensors. Then, the maximum likelihood decoding of W i1 ,......, i n is equivalent to finding p

the maximum of EW (Y ) . Proof: It follows by the same argument as Theorem (3.4) adopted to the variables appearing p

in the polynomial EW (Y) and the application of above Lemma.

Q.E.D.

The essence of the above theorem stated in more explicit language leads to the following conclusion. Given a received tensor Wi 1,..., in , the closest codeword tensor (in Hamming distance) to W in C (the code utilized at the input to the multidimensional channel) corresponds to a tensor B if and only if ( p −1) Max l EW (B ) = All tensors EW (Y ) = ∑ (W i 1,..., in ⊗ Yi 1,..., in ) l =0

(3.36)

Next, we consider the maximum likelihood decoding problem with respect to the Lee distance. We first consider the cases where p = 3 or 5. In these cases, there are easy expressions for the energy function. It is convenient to redefine the energy function in the following manner: Given an encoding procedure for a transmitted tensor X = (Xi1,..., i k), into a codeword tensor Y = (Yi1,..., in), by the following procedure i.e.

X = (Xi 1,..., ik ), → Y = (Yi 1,..., in )

(3.37)


44

and W = (W i1,..., in), a tensor whose entries are the pth roots of unity, we redefine the energy function as follows: i EW ( X ) = Re(W i1 ,....,in ⊗Yi1 ,..., in )  (3.38) where Re (x) denotes the real part of the complex number,  x  denotes the integral part of the number x and xi denotes the complex conjugate of x. It should be noted that the energy function coincides with the one for p = 2 (in the case u = –1). The definition of Lee distance is provided to facilitate the easier understanding of further discussion. Definition 3.6 p The Lee weight of an m-dimensional tensor of order k , X = (Xi 1,..., ik ), (Xi 1,..., ik ) ∈ Z , p is a

prime, is defined as m

m

i1=1

ik =1

WL = ∑ ...∑ Xi 1,..., ik

(3.39)

where 0 < X i 1, i 2 ,..., ik ≤ ( p 2)  X i 1, i 2,..., ik , X i 1, i 2,..., ik =   p − X i 1, i 2 ,..., ik , ( p 2) < X 1i , i 2,..., ik < ( p − 1) The Lee distance between any two compatible tensors is defined as the Lee weight, W L of their difference.

With the above definition, we study the cases where p = 3, p = 5. From now, in the following discussion, X → Y denotes the encoding procedure that defines a code (multidimensional), and X , Y are tensors of dimension m and order k , n respectively, of third or fifth roots of unity. In the following, two new theorems are proved. The first one is equivalent to the Theorem (3.4). It states that maximum likelihood decoding (MLD) in a ternary code is equivalent to the maximization of the energy function in (3.39). The Theorem is formally stated below: Theorem 3.7: Let p = 3, A → B; then B is the closest multidimensional codeword (in the Hamming distance) to a received tensor word W if and only if

EW ( A ) = Max EW (X ). X

(3.40)

Proof: The proof is similar to that of Theorem (3.4) and is avoided for brevity Q.E.D. The proof of Theorem (3.7) as well as Theorem (3.8) requires the utilization of Lemma (3.6) and a clear understanding of when the energy function is maximized. The new


45

energy function provided in (3.39) is a convenient expression. Its definition is once again based on understanding when the energy function is maximized and how utilizing Re ( ), the real part of a complex number does not alter the end effect in decoding a received word. Now, we consider the problem of maximum likelihood decoding (MLD) with respect to the Lee distance: Theorem 3.8: Let p = 5, A → B ; then B is the closest multidimensional codeword (in Lee distance) to a received tensor word W if and only if

EW ( A ) = Max EW (X ). X

(3.41)

Proof: From the definition of the energy function in i EW (A ) = (i 1, i 2,..., in ) : W ( i 1,..., in ) ⊗ B i 1,..., in = 1

{

}

{

i − (i 1, i 2,..., in ) : W( i 1,..., in ) ⊗ B i 1,..., in = u 2 or u 3

}

(3.42)

i = m n − (i 1, i 2,..., in ) : W ( i 1,..., in ) ⊗ B i 1,...., in = u or u 4

{

{

− 2 (i 1, i 2,..., in ) : W i ( i 2,..., in ) ⊗ Bi 1,..., in = u 2 or u 3 = mn − dL (W,B)

}

} (3.43)

Where W• denotes the complex conjugate of all components of W and d L denotes the Lee distance. Hence, EW (A) reaches a maximum if and only if d L (W, B) reaches the minimum. Q.E.D. The above results are generalized to infinite dimension/order tensors in a straightforward manner.

3.6 NON-LINEAR CODES In the theory of error control codes in one dimension, linear block codes are first extensively studied and various problems including the sphere packing problem was subjected to intense theoretical investigations. The research and development led to various theoretical as well as practical encoding/ decoding algorithms. Then, because it was thought that linear codes are limited from the point of view of various code parameters such as the number of (Ara) correctable errors/minimum distance, non-linear block codes were studied. The research in this direction culminated in the discovery of codes from algebraic geometry based techniques. The encoding algorithm was generally easy from the point of view of theory as well as physical hardware. It is the decoding algorithm which was considered difficult and was the subject of intense investigations resulting in several decoders. The maximum likelihood decoding (MLD) problem of linear codes and the relationship to energy functions (discussed


46

in the previous sections) naturally suggests a search for similar techniques to non-linear codes. In the following, non-linear multidimensional codes are investigated. The essential idea in generalizing the results in previous section to non-linear multidimensional codes is to consider the representation of Boolean functions as polynomials over the field of real numbers. In the context of one dimensional non-linear codes, part of the discussion is known (BrB) and is repeated here for the sake of completeness. Also, utilization of some subtle ideas associated with tensor products make the presentation essential aid for realizing that non-linear multidimensional codes share various features with linear codes. Definition 3.7 A Boolean function f on n variables, is a mapping

f : {0,1}n → {0,1}

(3.44)

For the present discussion, it is useful to define Boolean functions using the symbols 1 and –1 instead of the symbols 0 and 1, respectively. Definition 3.8 A Hadamard matrix of order m, denoted by H , is an m × m matrix of +1’s and –1’s such that

Hm HmT = mI m ,

(3.45)

where Im is the m × m identity matrix. The above definition is equivalent to the assertion that any two rows of H are orthogonal. Hadamard matrices of order 2k exist for all k > 0. The construction is as follows: H1 = [1]

1 1  H2 = 1 – 1   

 H 2n H 2n  . H 2n +1 =   H 2n − H 2n 

(3.46)

Definition 3.9 Given a Boolean function f of order n, P is a polynomial (with the coefficients over the field of real numbers) equivalent to f if and only if for all vectors X ∈ {1, – 1}

n

f (X ) = Pf (X ).

(3.47)

An important problem that is relevant to the investigation of non-linear multidimensional codes is the following: Given a Boolean function f of order n, compute Pf , polynomial which is equivalent to f. From the results in section 3, it is evident that the components of the codeword tensor (of a linear code), in the {1, – 1} representation are Boolean functions (monomials) in the


47

components/elements of the information tensor through the mapping described in Definition 3.4. Thus, the idea once again is to represent the corresponding Boolean functions of non-linear codes as polynomials/monomials over the field of real numbers. In the context of vector variable, the following inferences from Theorem (3.4) are well known in switching theory textbooks. But, in the case of vector variables, in (BrB), an alternative proof is given. In the following, it is shown that given the Boolean functions which are the components of a codeword tensor, there exist polynomials (with coefficients over the field of real numbers) equivalent to them over the multidimensional hypercube. Theorem 3.9: Let f be a Boolean function of order either strictly less than or equal to m n (in the components of a tensor X of dimension m and order n). Let Pf be a polynomial equivalent n to f. Let B denote the tensor of coefficients of Pf . Let P denote the tensor of utmost 2 m values of Pf (corresponding to m n{+1, – 1} components of tensor X ). Then, (1) the polynomial Pf always exists and is unique, (2) the following relationship is satisfied P = G ⊗ B, where ⊗ denotes the inner product of tensors. Proof: The proof is constructive in nature. The essential idea is to determine the coefficients of the polynomial by solving a system of linear equations, possibly imbedded in tensors. First, let us consider a Boolean function f of one variable and let us determine the coefficients of the polynomial Pf . Pf (x) = b 0 +b 1 x

(3.48)

Evaluating the polynomial on the domain of the Boolean function, we have

Thus,

Pf (1) = b 0 + b 1

(3.49)

Pf (–1) = b 0 – b 1

(3.50)

P = G ⊗ B,

 +1 + 1 where G =  +1 – 1   

(3.51)

G is a Hadamard matrix and B as defined before is the vector of coefficients of

Pf (X1 , X2 ,..., Xn + 1 ) . Remark Before proceeding with the proof, the following comparison/discussion on the similarities and differences between tensor products, matrix products is very relevant. Consider a G matrix and a column vector B. The tensor product, when the variables (matrix, column vector) are treated as tensors is given by


48

CONTRACTION → Pi G ⊗ B = Gi , j Bk = Pijk 

G11 B1 G21 B1

G11 B2 G12 B1 G21 B2 G22 B1

G12 B2 G22 B2

(3.52)

Now, we perform contraction on certain indices of the tensors. The resulting tensor is a first order tensor. Specifically, suppose we do the contraction over the indices j, k. Then, we have

G11 B1 + G12 B2 G21 B1 + G22 B2

(3.53)

Thus, the tensor product, in contrast to the matrix product allows more freedom in summing the components over different indices (contraction over different indices in the language of tensor algebra) of the tensor. Now, we return to the original proof. The above argument is now generalized to less than or equal to m n variables ( or arbitrary finite/countable number of variables which are possibly the components of a tensor ) by the method of mathematical induction. The case m = 1, n = 1 is proved at the beginning of the proof. Since m n is still a large number (finite), say l, it is sufficient as well as necessary to prove the result for a finite number l ( in the case considered, the binary variables are imbedded inside a tensor. Also, the polynomial representing the Boolean function is expressed through inner product operation over appropriate tensors ). Now, as an induction hypothesis, assume that the claim is true for l P = G2n B

(3.54)

variables. Since, every polynomial of (l + 1) variables can be written as a combination of two polynomials each of l variables, we have

Pf (X1 , X2 ,..., Xl + 1 ) = Pf 1 (X1 , X2 ,..., Xl ) + Xl + 1 Pf 2 (X1 , X2 ,..., Xl )

(3.55)

There are two possibilities, either Xl + 1 = + 1 or Xl + 1 = − 1. Hence, by the induction hypothesis (3.55), the system of linear equations in (l + 1) variables, becomes G2n G2n  P = G – G   2n 2n  From the recursive definition of Hadamard matrices,

we have

G2n G2n  1 1 G 1 = [ 1 ] ; G 2 = 1 – 1 ; G2n +1 = G – G    2n   2n P = G 2n + 1 B

(3.56)

(3.57)

Hadamard matrices are non-singular; thus, for any given f, a unique Pf exists (defined by a vector of coefficients).


49

In the language of tensor algebra, the same argument holds true except that the tensor can have ( the tensor utilized to couple the coefficients of the polynomial representing a Boolean function to the values of the polynomial) ‘0’ (zero) entries in addition to +1, –1 entries ( when contraction is performed over the appropriate indices). Uniqueness of such a polynomial is ensured by the uniqueness as a representation of Boolean function ( from the discussion/proof above ). Thus, in the tensor algebra notation, we have P = G ⊗B

(3.58)

where ⊗ denotes the inner product of two tensors.

Q. E. D.

It should be clear that the above representation theory has relevance to the minimum sum of products representation of a Boolean function. The above theory, as is easily seen holds true, if one is interested in finding the equivalent polynomial of a Boolean function which assumes {0,1} values. One way to see the result is by the following claim. CLAIM: Every monomial over {1, –1} can be written as a polynomial over {0,1} by the change of variable (BrB), x = 1 - 2 u, as follows: k

k

i =1

i =1

∏ Xi = 1+ ∑ (−2)i

∑ ∏U Si j ∈Si

j

(3.59)

with Si , a subset of {1, 2, ..., k } with i elements. For example, X1 X 2 = (1 − 2U1 ) (1 − 2U 2 ) = 1 − 2 (U1 + U 2 ) + 4U1U 2

The variables can be the components of a tensor X = (Xi 1,..., ik ), The representation theory developed above is now utilized for representing the multidimensional error correcting codes in a way that generalizes the representation described in section 3. Consider the linear multidimensional (m, k ; m, n) code C. The code can be represented by viewing each component of the codeword tensor as a Boolean function of utmost m k variables. A tensor V ∈ C , if and only if there exists an m-dimensional tensor of order k ( binary entries ) such that, with {1, – 1} tensor, X = (Xi 1,..., ik ),

Vi 1, i 2,..., in = fi 1, i 2,..., in (X ) .

(3.60)

The Boolean functions associated with the components of a linear multidimensional codeword tensor are determined by the generator tensor entries through which the code is represented. For linear multidimensional codes, every component of the codeword tensor fi 1,i 2,..., in (X ) correspond to an XOR operation of some variables of the information tensor ( determined by the corresponding entries of the generator tensor ). Thus, for every component (i1, i2,..., in), the Boolean function fi 1,i 2,..., in (X ) can be transformed by the method

50


described in Theorem (3.9) to an equivalent polynomial over {1, – 1} one monomial only.

mk

which consists of

Now, by the same argument as in Theorem (3.4), the maximum likelihood decoding (MLD) of a given received tensor word is equivalent to solving the following maximization problem:

Max (Wi 1,..., in ⊗ fi 1,...., in (X )).

(3.61)

By the procedure/reasoning through which we arrived at the above conclusion, tells us that the MLD problem as defined above also holds ( (3.61) holds) for non-linear multidimensional codes. For non-linear multidimensional codes, a component of the codeword tensor fi 1,i 2,..., in (X ) can consist of more than one monomial. Other than that, each component satisfies all the conditions to arrive at the above conclusion. From the above generalization, it follows that, for both linear as well as non-linear multidimensional codes, the maximum likelihood decoding problem is equivalent to the maximization of a multi-variate polynomial defined over the components of {1, –1} tensor i.e. over a tensor X with entries, of dimension m and order k . Hence, the following interesting theorem follows: Theorem 3.10: The following three problems are equivalent: (1) Maximization of multivariate polynomials with rational coefficients over the multidimensional hypercube, (2) Maximum likelihood decoding (MLD) problem of an (m, k ; m, n) multidimensional linear code, (3) Maximum likelihood decoding (MLD) problem of a not necessarily linear (possibly non-linear) multidimensional code, each of whose codewords are tensors of dimension m and order k . In view of the results in section 5 for non-binary codes which parallel those in section 3 for binary codes, maximum likelihood decoding of non-binary, nonlinear multidimensional codes is again equivalent to maximization of multivariate (variables being the components of a tensor) polynomial over a subset of multidimensional lattice. Various results (theorems, concepts, designs etc.) on optimization of multi-variate polynomials over various subsets of lattice were developed in various scientific fields such as electrical engineering, mathematics, computer science, operations research etc. These results are being translated to multidimensions and also the repercussions which follow immediately from the tensor linear operator are being documented. For instance, in one dimensional logic theory, various theorems including the representation of a Boolean function in the minimum sum of products (MSOP) form are well studied. In view of the results in (Rama 3), utilizing the fact that matrix linear operator is a special case of tensor linear operator, various theorems on


51

multivariate polynomials ( imbedded in matrices ) are translated to the case where the polynomials/monomials are imbedded inside tensors. In as much as the linear space structure is utilized in deriving the results/theorems, the translation of the results from one dimensional logic theory to multidimensions is done with the generic principles described in (Rama 3), (Chapter 4). The results in sections 3, 4, 5, 6 effectively demonstrated the relationship between multidimensional codes, the energy functions optimized by multidimensional neural networks over various subsets of the lattice, optimization of multivariate polynomials (the terms/monomials of which are based on the generator, other tensors) over the various subsets of the multidimensional lattice. Thus, these local optima of the multivariate polynomials have the structure parallel to various linear transformation groups and basis of a certain linear spaces. Utilizing a natural leap of imagination, the author considers univariate as well as multivariate polynomials, power series in tensor variables with tensor coefficients. Specifically, an interesting problem that arises in structured Markov random fields is the problem of determination of tensor zeroes of the following univariate tensor polynomial, power series equations (Rama 6).

X 2 ⊗ A 2 + X ⊗ A1 + A0 = 0 m

∑X j =1

j

⊗ Aj =0

∞

∑ X j ⊗ Aj = 0

(3.62)

j =1

where X, {A} are tensors of compatible dimension, order such that the inner/outer product operations are well defined. The solution techniques developed in (Rama 11) when the linear operators are matrices are extended to the tensor linear operator case in (Rama 6). Also, various results that are well documented in the books such as (Gol) for matrix polynomials based on the properties of matrix linear operator are extended to tensor linear operator. Furthermore, in one dimensional system theory, various results are developed for systems of matrix polynomial equations utilizing only linear operator properties of a matrix. These results are extended to systems of tensor polynomial equations (Rama 3). In (Rama 6), the author formulates as well as solves the problem of determination of tensor variate zeroes of multi-tensor variate polynomial, power series equations L

L

i1= 0

m= 0

∞

∞

∑ ...∑ X ∑ ...∑ X

i1= 0

m= 0

i1 1

i1

1

⊗ X i22 ... X mim ⊗ Ai 1,...., in = 0 ⊗ X i22 ⊗ ... X mim ⊗ Ai 1,...., in = 0

(3.63)


52

Various other associated results are documented in (Rama 6). It is well known that the zeroes of a uni-variate scalar polynomial constitute a group. By utilizing the set of zeroes of a determinental polynomial associated with the uni-variate/multi-variate (tensor variables) polynomial, the set of tensor zeroes are divided into certain set of equivalence classes. Thus, a group structure is imbedded onto the linear subspace of tensor zeroes of uni-variate/multi-variate polynomial equations. Unlike the multivariate polynomials (whose terms/monomials are based on the components of tensors) optimized in sections 3, 4, 5, 6; in view of the above results, a natural question that arises is whether the local optimum of multi-tensor variate polynomials over various subsets of multidimensional (very high dimensional) lattice lead to (each variable is a tensor) codeword sets with better properties. When the information tensor, generator tensor, codeword tensors are blocked into sub-tensors and the objective function for the optimization problem over a subset of multidimensional lattice is rewritten, it is evident that a multi-tensor variate polynomial appears. Thus, such polynomials are subsumed in the ones considered in sections 3, 4, 5, 6. Integer Programming Problems: Solutions Using Decoding Techniques In computer science, operations research and other fields, problems of the following form arise very often:  n Maximize  ∑ Wi  i =1

∏X j ∈Si

j

  

(3.64)

where Sj is a subset of {1,2,...,n} and X j ∈{0,1} . Thus, the problem is concerned with optimizing a multivariate polynomial, whose variables assume integer values. By the discussion, in this section, every polynomial over {1, –1} can be transformed to an equivalent one over {0,1} by a change of variable. It is shown in section 2, that a special case of the above problem i.e. maximization of a quadratic form in {1, –1} variables arises in connection with the determination of global optimum stable state of a neural network and is equivalent to the minimum cut problem. This problem is known to be an NP hard problem. The problem in (3.64) was studied extensively by various researchers and the main effort concentrated in identifying the special cases which are solvable in polynomial time and in devising approximation techniques. The most common technique for solving the unconstrained {0, 1} program of the form in (3.64) is by transforming them to the problem of finding the maximum weight independent set in a graph, which is an NP-hard problem. The problem in (3.64) is transformed to the problem of finding the maximum weight independent set by using the concept of a conflict graph of a 0-1 polynomial. In (BrB), it is shown how decoding techniques can be utilized to maximize 0-1 nonlinear programs. The multidimensional version of the 0-1 nonlinear programming problem in (3.64) is given by


 n  Maximize  ∑ Wi ⊗ X ii   i =1 

53

(3.65)

where W, X are tensors containing the known coefficients w ‘s in W and the monomials in the variable components of the unknown tensor X. The inner product between these two tensors provides the scalar objective function whose variables are allowed to assume only {0, 1} or more generally finitely many values. It is shown in (Rama 6) that such an integer programming problem can be solved utilizing the multidimensional decoding techniques for linear block multidimensional codes. These results in operations research are avoided here and relegated to (Rama 6).

3.7 CONSTRAINED STATIC OPTIMIZATION In one/two independent dimensions, various static optimization problems are solved under the sub-fields of optimization theory such as (a) linear programming, (b) non-linear programming, (c) calculus of variations, (d) combinatorial optimization etc. With the innovative idea of formulating and solving the parallel problems in multidimensions (Rama 3) through the utilization of tensor linear operator (motivated by practical applications), vast literature in multidimensional optimization theory is generated. Various consequences of this innovative idea of the author are fully explained in the companion research article (Rama 3) on dynamic optimization. In the following, some innovative ideas of generic consequence in static optimization are described. In view of the results in section 5, the constraint set over which a multivariate polynomial (terms of the polynomial expressed in terms of the components of a generator tensor, received tensor in the case of MLD) is optimized is a subset of the multidimensional lattice (or bounded lattice, say, in multidimensions) and subsumes the multidimensional hypercube as its subset. These results naturally lead to a question as to whether it is possible to utilize the results in sections 3, 4, 5, 6 for optimizing multivariate polynomials over more general constraint sets in multidimensions. In the following theorems, constrained optimization over more general constraint sets utilizing the results of sections 3, 4, 5, 6 is discussed. Theorem 3.11: Consider a compact set in a multidimensional metric space. The local optimum of a multivariate polynomial (with the terms/monomials expressed in terms of the components of tensors and assuming binary/finitely many integer values) whose variables are allowed to assume finitely many values, over the compact set, occurs at the union of codewords of finitely many multidimensional non-binary/binary codes. Proof: From real/complex analysis (also topology), we have the Heine-Borel Theorem, which states that every open covering of the compact set ( in the space described by multiple independent variables ) has a finite sub-covering. The covering generally consists of open

54


balls ( hyperspheres in multidimensional space) in every metric space ( although other sets could be utilized for covering ). But, it can be chosen to be a collection of convex hulls of bounded lattices in multidimensions. This (possibly countable) collection covers the compact set and thus has a finite sub covering. This implies that the constraint set chosen for optimization can be covered by finitely many bounded lattices (convex hulls of bounded lattices in multiple independent dimensions ). But, by the results of sections 3, 4, 5, 6 the local optimum of multivariate polynomials (terms/monomials expressed in terms of the components of tensors) over multidimensional bounded lattice (subsets of multidimensional lattice) constitute a linear/non-linear multidimensional codewords. Hence, the local optimum is achieved at the set of codewords of finitely many linear/non-linear codewords (tensors). Q. E. D. It should be noted that the determination of global/local optimum of the multivariate polynomial over the compact set is reduced to determining the global/local optimum of the energy functions of finitely many neural/generalized neural networks. It should be understood that some codewords may not be in the feasible region i.e. strictly inside the compact set. Also, when specific compact sets are chosen, further detailed information can be obtained on the local optima. Theorem 3.12: Optimization of a multivariate polynomial (with the terms/monomials expressed in terms of the components of tensors) over an arbitrary open set in a multidimensional space (metric space) is equivalent to the optimization over the union of codewords of countably many multidimensional linear codes or an infinite dimensional code. Proof: Let us consider an arbitrary open set in a multidimensional space (metric space). By the Lindeloff’s covering lemma, the open set can always be covered by utmost a countable collection of open balls or other sets. It is evident that the covering can be chosen to be a countable collection of convex hulls of bounded lattices (in multidimensional space). But, by the results in section 4, the local optima of the multivariate polynomial (the monomials/ terms being expressed in terms of the components of tensors) over a bounded lattice constitute multidimensional codeword set. Thus, the local optima of a multivariate polynomial occurs at the union of codewords of countably many multidimensional codes. Q. E. D. Remark Suppose, the compact set/open set (in multidimensions) is covered by finitely/countably many hyperspheres (multidimensional) and a quadratic/higher degree form is optimized. By the spectral representation theorem, the local optima of quadratic/higher degree form occur at the eigentensors with the eigenvalues being the corresponding values. This corresponds to L 2 norm based optimization. The above theorems illustrate two essential ideas of generic utility in static optimization: (a) optimization over more general constraint sets, (b) decomposition principle.


55

In the following, the decomposition principle is explained as it is of generic utility in static optimization. Decomposition Principle Consider an arbitrary constraint set over which an objective function is optimized in one or more independent dimension variables. The constraint set is decomposed into the union of finitely many special sets with interesting structure. Optimization of various objective functions over the special sets has various interesting features: (a) various results are well known, (b) the local optima have interesting structure, (c) it is thoroughly studied etc. Utilizing these features, optimization of any objective function over the original set is decomposed into simpler problems. The above two theorems are only illustrative. The discovery and application of the above decomposition principle to multidimensional constrained optimization problems naturally led the author to investigate various other innovative ideas in static optimization. (I) Approximation of Objective Function by Polynomials, Power Series ( other Special Classes of Functions) Polynomials and power series (uni/multi-variate) are very important classes of functions. The optimization results (unconstrained as well as constrained) associated with these functions enable one to derive the local optimum of some classes of functions over various constraint sets invoking standard theorems (from approximation theory). For instance, the following theorem enables deriving results on continuous objective functions utilizing polynomials: Theorem 3.6 is utilized in association with the following theorem. Theorem 3.13: Every continuous function over a compact set always attains its maximum/ minimum over the set. Every continuous function can be arbitrarily closely approximated by polynomials ( multi-variate/univariate). Also, invoking the standard theorems from approximation theory, various classes of functions are arbitrarily closely approximated by polynomials: uni-variate/multi-variate. Thus, when these functions are utilized as objective functions, results associated with polynomials (derived in sections 3-6 ) are invoked. (II) Discovery of new local/global optimization techniques This requires utilizing either new classes of functions or new constraint sets. The constraint set structure renders the local optima of some functions with interesting structure and also the properties satisfied by the objective functions enables discovering efficient techniques. NP-Hard Problems: In computer science, operations research and other applied/theoretical research fields, various NP-hard problems are well identified and studied. It is well known that one NPhard problem is as complex ( in the terminology of complexity theory in theoretical computer


56

science) as any other NP-hard problem. Finding algorithms which are efficient (in terms of complexity) for an NP-hard problem is well recognized as a difficult problem. The following is a difficult open problem in theoretical computer science: Problem: Does a polynomial time algorithm exist for an NP-hard problem? In other words, is the class of problems in NP, the same as the class of problems in P? i.e. is P = NP? In the following, an innovative algorithm/approach to solve various NP-hard problems in one dimension is described. The multidimensional generalization of this algorithm/ approach to any NP-hard problem (in multidimensions) is being formalized. It is an extension of the following results to multidimensions. In section 2, the problem of computation of minimum cut in a graph is shown to be equivalent to the problem of determining the global optimum of the energy function of a neural network i.e. maximizing a quadratic form over the hypercube. It is well known that this is an NP-hard problem. In the following, an attack on this problem is described. Positive Definite Synaptic Weight Matrix: Determination of Global Optimum Stable State of a Neural Network: Consider a neural network whose synaptic weight matrix is symmetric as well as positive definite. In the following, an algorithm to determine the global optimum stable state of such a neural network is described. (a) Utilizing the well known theorem in linear algebra, every positive definite symmetric matrix, S can be decomposed into the following form by means of Cholesky Decomposition. S = N NT (3.66) where N is a lower triangular matrix. (b) The quadratic form being optimized by the neural network over the hypercube can be expressed into the following form: (3.67) X T S X = XT N N TX = YT Y , where Y = N TX . Since S is positive definite, XT S X > 0. Thus, YT Y > 0. The scalar expression for the quadratic form n

in terms of Y is given by

∑Y j =1

2

j

. Thus, it is evident that the value of the quadratic form is either

maximized/minimized when the value of each term i.e. Yj = ∑ N kj X k is either maximized or k

minimized. But, since, N in the Cholesky decomposition is a lower/upper triangular matrix, each of the terms is a linear form. Thus, to maximize the quadratic form, each linear form maximum/minimum value (whichever is larger) is determined over the constraint set, hypercube. Thus, the original NP-hard problem (of maximization of a quadratic form over the hypercube) is reduced to several linear programming problems i.e. optimization of several linear forms over the hypercube. In this novel algorithm/approach for various classes of NP-hard problems (minimum


57

cut computation in an undirected graph, knapsack problem etc.), the complexity of the algorithm is determined by (a) Complexity of determination of Cholesky decomposition of a positive definite symmetric matrix. Since there are various polynomial time procedures for the spectral decomposition, computationally well studied efficient algorithms are available, (b) Solving the linear programming problems related to optimization of linear forms ( maximization or minimization whichever leads to a larger value for the term) over the hypercube. It is well known that there are polynomial time algorithms for linear programming problems. In some problems that arise in operations research, communication theory etc., constraint set is a convex polygon/polytope (convex hull of various finite structures leading to convex sets bounded by hyperplanes) etc. and a quadratic/higher degree form is optimized over the constraint set. Then, by means of Spectral/Cholesky type decomposition of the positive definite symmetric linear operator (in one as well as multidimensions), various linear programming problems are solved through efficient polynomial time procedures. The computation of complexity of such procedures, efficient algorithms for NP-hard problems in one and multi-dimension, are being documented. When the connection matrix has other special structure efficient algorithms are found.

Linear Programming Problems: Decomposition Principle In the general framework of a linear programming problem, the constraint set is a convex polytope. By means of the decomposition principle, utilizing the hyperplanes bounding the feasible set, the convex polytope is expressed as the union of finitely many rotated, translated hypercubes. The linear objective function is converted into a quadratic objective function (reversing the technique utilized in the above algorithm) and the results from neural network theory are invoked to determine the local optima of the objective function (union of stable states of various neural networks). Thus, unlike the simplex algorithm, only a subset of the vertices of the feasible region that constitute the stable states of neural networks is searched for determining the global optimum for the linear program. In the following, an alternative algorithm for any NP-hard problem is formally described.

Hop-Skip Algorithm: Maximum Likelihood Decoding – NP-hard Problems This algorithm is designed/analyzed by the author for maximum likelihood decoding of linear block codes (Rama 1). From (BrB), and Theorem (3.3), maximum likelihood decoding of a received word , Y with respect to a graph-theoretic code is equivalent to


58

finding the maximum of the energy function E of a neural network defined by the graph G (the weights on the edges of G are given by W = (–1) yi with all its threshold values equal to zero. But, it is well known that the local optimum of a quadratic form over the hypersphere occurs at the eigenvectors (eigentensors of the symmetric second order tensor) of the symmetric matrix (associated with the symmetric matrix) with the value of the quadratic form being the eigenvalue. Thus, maximum eigenvector of the symmetric matrix maximizes the quadratic form over the hypersphere. Thus, the sign structure (sign of the components of the vector) of the maximum eigenvector is utilized as the initial condition to run a neural network i.e. Mathematically, let X 0 be the vector given by X 0 = Sign ( X max ), where X max is the normalized maximum eigenvector and X 0 is the initial state in which the neural network starts. A is the symmetric matrix. The analysis of hop-and-skip algorithm is provided below. X T A X = (X – X0 + X0)T A(X – X0 + X0)

(3.68)

= (X – X0)T A( X – Xmax) + X0 A X0 + 2 XT0 A( X – X0 ) = λmax + (X – X0)T A(X – X0) + 2 X0T A( X – X0) = λmax + (X – X0)T A(X – X0) + 2 λmax X0T(X – X0) = n –λmax + (X – X0) T A(X – X0) + 2 λmax X0T X

(3.69)

The above manipulations enable one to compare the value of the quadratic form on the hypercube at any discrete time instant against the maximum value on the unit hypersphere. The particular choice of initial condition, minimizes the Hamming distance between the maximum eigenvector and the initial condition vector to run the neural network. The set of eigenvectors of the connection matrix of neural network span the entire space or a subspace of it. Similarly, the set of stable states/ stable vectors span the space or a sub-space. To determine the maximum stable state, the essential idea of the above approach is to find the vector closest to the maximum stable state and utilize it as the initial condition to run the neural network. Detailed analysis of the algorithm is being investigated. Dynamic Optimization In (Rama 3), certain multidimensional system, in discrete/ continuous time is described by the following state space representation through tensors: Discrete Time:

X(n + 1) = A(n) ⊗ X(n) + B(n) ⊗ U(n), Y(n) = C(n) ⊗ X(n) + D(n) ⊗ U( n).

(3.70)


59

Continuous Time:

X(t + 1) = A(t) ⊗ X(t) + B(t) ⊗ U(t), Y(t) = C(t) ⊗ X( t) + D(t) ⊗ U(t).

(3.71)

where ⊗ denotes the inner product between compatible tensors in the system description in continuous/discrete time. Utilizing this state space representation, the author formalized a unified theory of control, communication and computation in multi/infinite dimensional systems, first discovered in (Rama1) for one dimensional systems. This theory enabled the author to develop a highly advanced version of the theory of evolution of life from organic matter. In this theory the author reasons that various body organs, functions of living systems have evolved over time and that bilogical systems are organic/inorganic matter based dynamical systems.

3.8 CONCLUSIONS Tensor linear spaces over finite fields are utilized to describe and study the structure/ properties of multi/infinite dimensional linear codes. The three concepts: multidimensional neural/generalized neural networks, multidimensional codes, multivariate polynomial (terms/monomials being expressed in terms of the components of generator, other tensors) optimization over various subsets of lattice, are related. It is shown that (a) the problem of maximum likelihood decoding of error correcting codes (multidimensional), (b) finding the global maximum of the energy function of neural/ generalized neural networks, and (c) solving integer/non-linear programming problems in multidimensions are related. The equivalence is proved for binary as well as non-binary cases. This equivalence naturally suggests utilizing the solvable cases of one problem to the equivalent problem and vice versa. Full capitalization of equivalence leads to various new results (Rama 6). The programming problem of multidimensional neural networks is solved. Several new heuristic procedures for NP-hard problems in multidimensions are suggested from the equivalence. The decoding techniques of various (multidimensional extensions of one dimensional codes) codes are utilized to find approximate solutions of NP-hard problems. Various innovative results in static optimization are described. Infinite dimensional generalization of the results is briefly described.

REFERENCES (Ara) B. Arazi, “Common Sense Approach to the Theory of Error Correcting Codes, “ MIT Press book. (BoT) A.I. Borisenko and I.E. Tarapov, “Vector and Tensor Analysis with Applications, “ Dover Publications Inc., New York, 1968.

60


(BrB) J. Bruck and M. Blaum, “Neural Networks, Error Correcting Codes and Polynomials Over the Binary Hypercube, “ IEEE Transactions on Information Theory, Vol. 35, No. 5, September 1989. (Gaal) Gaal, “ Group Theory, “ Academic Press, 1982, (Gol) I. Goldberg, “Matrix Polynomials, “ Academic Press, 1972. (Rama 1) Garimella Rama Murthy, “Unified Theory of Control, Communication and Computation—Part-1, “ Manuscript to be submitted to the IEEE Proceedings. (Rama 2) Garimella Rama Murthy, “Multi/Infinite Dimensional Neural Networks, Multi/Infinite Dimensional Logic Theory, Logic Synthesis, “ Published in International Journal of Neural Systems, Vol. 15, No. 3, pp 223-235, 2005. (Rama 3) G. Rama Murthy, “Optimal Control, Codeword, Logic Function Tensors: Multidimensional Neural Networks,” International Journal of Systemics, Cybernetics and Informatics, October 2006, pages 9-17. See also Chapter 4. (Rama 4) Garimella Rama Murthy, “Multi/Infinite Dimensional Logic Synthesis, “ Manuscript to be submitted to the IEEE Transactions on Computers. (Rama 5) Garimella Rama Murthy, “Signal Design for Magnetic and Optical Recording Channels, “ Bellcore Technical Memorandum, TM-NWT-018026. (Rama 6) Garimella Rama Murthy, “Tensor Variate Polynomials/Power Series, Tensor based Functions, Tensor Algebraic Geometry: Optimization, “ Manuscript to be submitted to the Transactions of American Mathematical Society. (Rama 10) Garimella Rama Murthy, “Unified Theory of Control, Communication and Computation: Dynamical Systems, “ Manuscript in Preparation. (Rama 11) Garimella Rama Murthy, “Transient and Equilibrium Analysis of Computer Networks: Finite Memory and Matrix Geometric Recursions, “ Ph. D. Thesis, Purdue University, West Lafayette, Indiana. (Rama 12) Garimella Rama Murthy, “Origin of Universe: Living/Non-Living: Grand-unification Theory of Universe, “ Manuscript in preparation.

Tensor State Space Representation: Multidimensional Systems

61

CHAPTER

4


4.1 INTRODUCTION With the efforts of researchers in electrical engineering, linear system theory started with abstract models of arbitrary linear systems through forced/unforced nth order difference equations in discrete time and differential equations in continuous time. Such representations are called the input-output representations of the linear system. These arbitrary system (electrical, mechanical, chemical, hybrid systems) evolution equations were then converted into first order differential/difference equations in state, control, input, output vectors through state, input, output coupling matrices. Such a representation is called the state space representation. The state space equations take the following form (Gop) Discrete Time Systems:

X(n + 1) = A( n) X(n) + B(n) U(n), Y(n) = C(n) X(n) + D(n) U(n), Continous Time Systems : X(t) = A(t ) X( t) + B( t) U( t)

Y(t ) = C(t ) X (t ) + D(t ) U (t )

(4.1)

where {A(n), B(n), C(n), D(n)} as well as {A(t), B(t), C(t), D(t)} are matrices of compatible dimensions. Thus, in the design, analysis and synthesis of linear systems, linear algebra techniques were extensively utilized. Various, input-output representation related concepts such as impulse response, systems function were shown to be derivable from the state space description. Also new concepts such as controllability and observability are studied in terms of state space representation. Thus, the state space representation of linear systems proved to be a far better description of arbitrary systems.


62

Motivated by various practical applications, multidimensional systems were then studied. Various system theorists tried to extend one dimensional system theory to multidimensions utilizing the ideas of local state and local control. For instance, consider a typical discrete time, two dimensional system. The evolution of a prototype linear model is described by the state updating equation

X(h + 1, k + 1) = A1X(h, k + 1) + A2 X(h + 1, k) + B1U(h, k + 1) + B2U(h + 1, k) where

X( h, k )∈Rn and U( h, k )∈Rm are the local state and local input value at (h,k) and

A1,A

2

, B1 ,B

2

are real matrices of suitable dimensions. This type of approach based

on local state and local control was utilized in association with partial differential equation based continuous time linear multidimensional systems. These representations of continuous time as well as discrete time multidimensional systems required considerable amount of ingenuity, careful tracking of the indices, in designing and analyzing such systems. To a certain degree, this notation impeded further progress in multidimensional system theory. With this type of approach/notation, modeling, design and analysis of certain linear/nonlinear, multi/infinite dimensional systems was a complicated task. The author for the first time realized that, for the evolution of CERTAIN multidimensional linear systems, tensor linear operator based state space description is necessary as well as helpful. This mathematically formal tensor state space representation was an important contribution for further progress in multi/infinitedimensional system theory (linear/non-linear dynamical systems). Also, the author after carefully observing various multi/infinite dimensional systems (explicitly stated as a static or dynamical system or when a proper abstraction is made the multidimensional nature of problem/ phenomenon becomes apparent) such as those that arise in multi/infinite dimensional neural networks (Rama 2), databases ( utilizing multiple attribute tree etc. ), multi/infinite dimensional coding theory (Rama 3), proposed the utilization of tensors ( of order, dimension finite/infinite ) as the linear operators in the design, analysis and synthesis of such systems. This idea is already utilized in some applications. It should be noted that in the analysis of some systems defined over finite fields and other discrete structures, utilization of tensors considerably simplifies the analysis. In the case of multidimensional systems, there is no natural notion of causality. Various types of causality ( quarter-plane causality, half-plane causality) are artificially imposed by different choices of neighbourhood sets. With such an approach (for all multidimensional systems), it is very difficult to study controllability, observability and stability. The author realized that for certain multidimensional systems, utilization of tensor linear operators to represent the state, control, input, output variables, is very convenient (from the point of view of design and analysis of such systems) (Rama 1).


63

This chapter is organized as follows. In section 2, the conventional approaches to two/ multidimensional system theory are summarized. It is also described how the utilization of tensor linear operator associated with multidimensional linear spaces provides a new approach for formulating as well as solving the problems related to static as well as dynamical systems (defined over multidimensional linear space). In section 3, state space representation of certain multi/infinite dimensional linear systems utilizing the tensor linear operator is formally described. In section 4, it is illustrated how the utilization of tensor based state space representation enables one to translate the results from one dimensional systems to certain multidimensional systems. Various generic principles of how to translate the results from one dimensional system theory to multi/infinite dimensional system theory are provided. In section 5, multi/infinite dimensional time series analysis models are described. In section 6, utilizing the concepts of local state, local input, local control in the multi/infinite dimensional state space, various state space representations for multi/infinite dimensional distributed systems are formally described. These state space representations enable one to translate the results developed for conventional multi/infinite dimensional systems to those described through the tensor state space representation. The chapter concludes with some conclusions.

4.2 STATE OF THE ART IN MULTI/INFINITE DIMENSIONAL STATIC/DYNAMIC SYSTEM THEORY: REPRESENTATION BY TENSOR LINEAR OPERATOR One of the main tools in the design and analysis of one dimensional linear dynamic systems as well as static systems is linear algebra. Motivated by practical applications in image processing and other fields, system theorists proposed various input-output models for two/multidimensional systems. Models which exhibit quarter plane causality have been initially investigated from the input-output point of view (BiF) in the framework of two dimensional filter theory, where two dimensional filters are represented by proper rational functions in two indeterminates of the following type:

W (Z1 , Z2 ) =

∑nZ

i + j ≥1

1+

ij

1

i

Z2 j

∑dZ

i + j≥1

ij

1

i

Z2 j

(4.2)

The idea of associating two dimensional state space models with two dimensional filters was originated very naturally. However, since the beginning it appeared that the canonical technique based on the Nerode equivalence leads to an infinite dimensional state space. The reason was to utilize a matrix as the linear operator to describe the state dynamics. So, following some heuristic procedures, several finite dimensional models have been (BiF) introduced, where two notions of state play different roles: 1. local states: X(h,k) belong to a finite dimensional vector space. They enter in the state updating equation and determine the value of the output.


64

2. global states Xn = {X (i + h , – i ) , i ∈ z} provide the initial conditions on a separation set of Z × Z. These belong to an infinite dimensional vector space (in one independent dimension), which provides an extension of the space of Nerode equivalence classes. The most common state space model with quarter plane causality is represented by the following equation.

X(h + 1, k + 1) = A 1X (h, k +1) + A2 X(h + 1, k ) + B1 u (h, k + 1) + B2 u (h + 1, k ) Y(h , k ) = CX(h , k )

(4.3)

where x(h , k )∈ R n , u( h, k )∈R m , y(h , k )∈R p , are the values of the local state, the input and the output at (h, k)∈Z × Z . Since the local state at (h+1, k +1) is computed by solving a first order difference equation, the system (4.3) denoted by Σ 1 = (A 1, A 2, B1, B2, C) is named a first order system. The above model has been extensively studied in its general form and under some conditions/constraints on the system matrices. The most popular particularized version of (4.3) is Roesser’s model, where the local state space X is assumed as the direct sum of two vector spaces Xh and Xv , and the matrices of the model are constrained to have the following form (partitioned) 0   A111 A121   0 B11   0  = = A1 =  B B , ,  , A2 =  2     2  1 2 (4.4) 2 0   0  A21 A22  0   B2  Second order models are less frequently used: the typical structure of their equation is given by

X ( h + 1, k + 1) = A1X ( h , k + 1) + A2 X ( h + 1, k ) + A0 X ( h , k ) + BU ( h , k ) Y (h , k ) = C X(h , k )

(4.5)

In Attasi’s model A 1 and A 2 are commutative matrices. Also, A 1A 2= –A 0. It realizes separable filters only and constitutes an interesting second order model, as the underlying theory is very close to the one dimensional theory (BiF). Recently, the behavior approach has been extended to two dimensional systems. Following this theory, a two dimensional system is defined by a family of β admissible functions (behavior), defined over the discrete plane. These functions are characterized by the property of belonging to the Kernel of a polynomial matrix M (Z1, Z2) in two variables

β = {ω = ∑ wij z1i z2j M ω = 0} i , j∈Z

(4.6)

Associated with the external description provided by the behavior different internal representations can be given by introducing the so called latent variable models. State variable models constitute a particular type of latent variables, that hold the memory of the system with respect to the notion of past introduced on Z × Z. When a state description is possible,


65

i.e. when the notion of past, present and future are allowed by the structure of β , the behavior is called Markovian. Since there is not any natural direction for the evolution in Z × Z , the Markovian property appears more general than the familiar quarter plane causality and has been exploited in the analysis of non-causal two dimensional dynamics. Also, various static systems that involve simple linear transformations in the multidimensional space were previously abstracted utilizing the matrix linear operator. Such systems arise in practical applications such as databases (modeling storage of multiple attribute trees), computerized topography etc. The techniques developed for design and analysis of such systems were thus very elementary. The above efforts in two/multidimensional system theory were primarily utilizing the matrix linear operator on an n-dimensional ( in one independent variable) vector space. System theorists did not realize that utilization of tensor linear operator (in multidimensions) could lead to design and analysis of a large class of multidimensional systems. In the following areas, utilization of tensor linear operator to describe the multi/infinite dimensional state space enables one to formulate new problems , introduce new concepts, derive new results/theorems. Some of the areas of interest where such an idea could be utilized are (1) Multi/Infinite dimensional computation theory, (2) Multi/Infinite dimensional information/communication/coding theory, (3) Multi/Infinite dimensional rate distortion theory, (4) Multi/Infinite dimensional stochastic systems—Theory of Markov random fields, (5) Multi/Infinite dimensional time series analysis, (6) Multi/Infinite dimensional digital signal processing, (7) Theory of Multi/Infinite dimensional connectionist structures—graphoids, (8) Theory of databases utilizing multidimensional storage, (9) Matroid theory, (10) Multi/Infinite dimensional Game theory. By the utilization of the idea of capturing a multidimensional state space through a tensor linear operator, new research problems can be formulated and solved.

4.3 STATE SPACE REPRESENTATION OF CERTAIN MULTI/INFINITE DIMENSIONAL DYNAMICAL SYSTEMS: TENSOR LINEAR OPERATOR A multidimensional system transforms an m -dimensional tensor (array) of order r into a k dimensional tensor of order s. In the following, some confusion that arises in the terms utilized is cleared. Remark: Notation In the tensor notation, the word “dimension of a tensor” stands for the number of values each independent variable assumes, whereas the word, “order” represents the


66

number of independent variables. In that sense, usage of the term “multidimensional” system seems incorrect compared to “multi-order” system. But, this is a matter of notation. To stick with familiar jargon, in the following the author utilizes the term “multidimensional systems”. From the context, the reader should be able to ascertain the usage of words, “order”, “dimension”. Infinite dimensional systems lead to further confusion. If each independent variable assumes infinitely many values ( in contrast to finitely many values assumed by each independent variable in a multidimensional system ) and there are only finitely many independent variables, the system description utilizes infinite dimensional tensors of finite order for state, input, output variables. But, if the number of independent variables is also infinite, the dimension as well as order of tensors utilized in the representation of variables is infinite. It should be noted that, in the case of discrete time systems, each component of the tensor input, output, state variables is a function of a discrete time index. But, in the case of continuous time systems, each component of the tensor input, output, state variables is a function of the continuous time index. Also, in the case of time varying systems, the transformation is a function of the index (discrete or continuous), whereas in the case of time-invariant systems, the transformation is independent of the index. Definition A dynamical system is linear if and only if, given any two points (scalar, vector, tensor variables) in the input space, say U1 and U2, and given any two scalar ( real or complex ) constants, the following property is satisfied by the transformation L, describing the dynamical system: L (C 1U 1 + C 2U 2 ) = C 1L (U 1 ) = C 1L (U 1 ) + C 2L (U 2 ); C 1 , C 2 ∈C or R or any field

(4.7)

If the above property is violated by the dynamical system, we call it a non-linear system. Conventionally, in multidimensional ( multi-order may be more appropriate, but is not utilized by the author ) system theory, in the case of discrete time dynamical system (an example is provided in section 2), the evolution is described by means of local state, local control, local input and local output variables. This is very cumbersome. In the case of certain multidimensional systems, the state space representation by means of tensors (described below) enables one to compactly capture a higher order difference equation through TENSOR notation. In order to describe the tensor state space representation, the following concepts/ideas from tensor analysis are explained.

Concepts from Tensor Analysis Tensor Function of a Scalar Argument It is a rule assigning a unique value of a tensor to each admissible value of a scalar t (BoT). The variable t can be a discrete index assuming countably many values or a


67

continuous index assuming uncountably many values. To indicate such a function, we write Ai 1,i 2 ,...,in = Ai 1, i 2 ,..., in ( t ) (4.8) For instance, the state of stress of an elastic medium varies in time. Then, the stress tensor becomes a function of time i.e. Pik = Pik (t) (4.9) By the derivative of the function (4.8) with respect to time/index, we mean the tensor with the components,

d Ai 1, i 2,..., in (t) dt

=

Lt Ai 1, i 2,..., in (t + ∆t ) − Ai 1,..., in (t ) ∆t → 0 ∆t

(4.10)

calculated in a coordinate system which does not vary in time. The derivative is clearly of the same order as the tensor itself. With the above notation from tensor analysis, certain multi/infinite dimensional discrete time/index dynamical system can be described by means of a state space description of the following form:

Tensor State Space Representation of Certain Discrete Time Systems Discrete Time Systems:

X( i 1,..., ir ) (n + 1) = A( i 1,..., ir ; j1,..., jr ) (n) X( j 1,..., jr ) (n) + B( i 1,..., ir ; j1,..., jp ) (n)U( j 1,..., jp) (n), Y(l 1,...,ls ) (n) = C(l 1,..., ls ; j 1,..., jr (n) X( j 1,..., jr ) (n) + D(l 1,..., ls ; j 1,..., jp ) (n)U( j 1,..., jp ) (n).

(4.11)

where A(n) is an m dimensional tensor of order 2r (called the state coupling tensor ), X(n) is the state of the dynamical system at the discrete time index n, whereas X(n+1) is the state of the system at the discrete time index n+1. Furthermore B(n) is an m dimensional tensor of order r+p ( called the input coupling tensor ), Y(n) is an output tensor of dimension m and order s. U (n) is an m dimensional input tensor (varying with the discrete time index of order p) and C(n) (called the state coupling tensor to the output dynamics) is an m dimensional tensor of order (s + r), D(n) is the input coupling tensor to the output dynamics of dimension m and order s+p. In the above state space description of certain type of multidimensional discrete time dynamical system, there are r dimension variables which are inherently discrete. The evolution of the system (changes in the system parameters) occur at discrete time instants. The notation for index set in the state equations requires some explanation. Since the state tensor is an m -dimensional tensor of order r, it will have m components. When the system evolves, it transits through tensors in the state space. With the summary of tensor functions of scalar argument provided above, the dynamics of certain type of multi/infinite dimensional continuous time/index systems is described by the following state space description:

68


Tensor State Space Representation of Certain Continuous Time Systems i X( i 1,..., ir ) (t) = A( i 1,..., ir ; j1,..., jr ) (t) X( j 1,..., jr ) (t) + B( i1,..., ir ; j 1,..., jp ) (t) U( j1,..., jp ) (t), Y(l 1,..., ls ) (t) = C(l 1,..., ls ; j 1,..., jr ) (t) X( j 1,..., jr ) (t ) + D(l 1,..., ls ; j 1,..., jp ) (t)U( j 1,..., jp ) (t ).

(4.12)

where A(t) is an m dimensional tensor of order 2r (called the state coupling tensor to the state dynamics), X (t) is the state of the dynamical system at the continuous time/ . index t, whereas X (t) is derivative of the state of the system. Furthermore, B (t) is the m dimensional tensor of order r+ p (called the input coupling tensor to the state dynamics), Y (t) is the output tensor of dimension m and order s. Also, U (t) is an m dimensional input tensor of dimension m and order p , and C (t) is an m -dimensional tensor of order (s + r), D (t) is the input coupling tensor to the output of dimension m and order s + p . It should be noted that the state space description provided above for certain continuous/discrete index systems hold true even for certain infinite dimensional systems. In the case of infinite dimensional systems, in the state space descriptions, the tensors utilized are of dimension/order infinity ( either or both of them). Now, the above tensor state space representations are contrasted with the conventional approaches in the representation of certain multidimensional systems. It is reasoned that the Tensor State Space Representation is an important leap in multi/ infinite dimensional system theory. Also, another objective is to remove the confusion in the mind of the reader who read the classical literature in multi/infinite dimensional system theory with matrix linear operator notation. The primary source of confusion is not so much in the discrete time/index multidimensional systems, but in the case of continuous time /index multidimensional systems. Conventional Multidimensional System State Space Representation versus Modern Tensor State Space Representation: In section 2 as well as section 3, the limitations of the way system theorists tried to represent and analyze the two/multidimensional discrete time/index systems is discussed. Also, the advantages of tensor state space representation (of certain large class of multi/ infinite dimensional systems) discovered and formalized by the author are described. The transition from the conventional mode of thinking where the system is represented by means of multiple independent variables, local state/local control are coupled to the system dynamics by means of matrices to the modern version where tensor notation is utilized, requires the realization that the linear space utilized in multidimensions is captured through the tensor and the system dynamics when done in discrete time requires a discrete variable. The continuous index case requires more imagination to understand the transition from conventional approaches to the modern approaches. In the conventional multidimensional system representation, partial differential equations are utilized to describe the input-output behavior as well as the state (internal description) dynamics. In the conventional approaches, multiple independent variables are tracked through separate indices, leading to partial differential equations. But, the utilization of tensor linear operator


69

and the tensor function of scalar argument enables one to describe the dynamics of tensor state variable as a function of one continuous time/index variable. Thus, the discrete as well continuous multi/infinite dimensional system state space representation utilizing tensors resembles the familiar one dimensional system state space description. The above tensor state space description reduces to the one dimensional case when the order of the tensors is one. Thus, various results developed on one dimensional linear spaces for one dimensional linear systems are readily translated to certain multi/ infinitedimensional systems described through tensor linear spaces (with some care taken in pathological cases as well as when the problem being solved depends heavily on the neighborhood set).

4.4 MULTI/INFINITE DIMENSIONAL SYSTEM THEORY: LINEAR DYNAMICAL SYSTEMS – STATE SPACE REPRESENTATION BY TENSOR LINEAR OPERATORS The state space representation of one dimensional linear systems resembles that in (4.11), ( 4.12). In fact, one dimensional linear systems are a very special case of certain multi/ infinite dimensional systems described through (4.11), (4.12). A natural question that arises is whether it is possible to transfer the results from one dimensional systems to certain multidimensional systems described through (4.11), (4.12). It is explained in the following that it is possible to do such a translation provided some care is taken in deriving the results for certain class of multi/infinite dimensional systems. Some principles which can be utilized as a guideline in deriving the results for multi/infinite dimensional systems are provided below: (1) In the case of one dimensional systems utilizing the state space representation of a linear system, if a result is derived on the system response (invoking the standard theorems in the theory of ordinary difference/differential equations), that result has a corresponding version for multi/infinite dimensional systems when the inner product and outer product between state vector/input vector/ output vector, matrices appearing in the state space descriptions are replaced by those between compatible tensors in multi/infinite dimensions. One must exercise care in making sure that the tensor products make sense. (2) The tensor state space representation (rather than vectors and matrices in one dimensional case) enables one to translate the results on controllability, observability, stability from one dimensional linear space based dynamical systems to certain multidimensional linear space based dynamical systems. The tensor state space representation enables one to translate various problems for one dimensional systems, in a one to one manner to certain multi/infinite dimensional systems. These problems are defined utilizing the state space structure to be linear (linear spaces in one/multi/infinite dimensions). In translating the solution

70


of the problem from one dimension to multidimensions, inner product/outer product between vector-matrix variables are replaced by those between tensortensor variables. Care should be taken to ensure that the problem statement in multidimensions doesn’t utilize the neighbourhood structure. (3) The multi/infinite dimensional state space structure is such that there is no notion of causality. From 1970s, system theorists, electrical engineers, computer scientists developed various notions such as quarter plane causality, half plane causality, other types of causality ( to introduce some form of ordering on the two/multidimensional state space ) for providing an input-output description. But the state space representation through tensors (of certain multidimensional systems) enables one to get the associated input-output description as a special case ( for such systems ). Thus, various problems in image processing, database theory, theory of random fields are reformulated utilizing the tensor state space description and solved in this context. When these problems have multi/infinite dimensional state space structure (implicitly or explicitly specified) imbedded into the statement, utilizing the tensor linear operator (or the theory of tensor linear spaces) and the results in this chapter, they are considered to be solved. The systems in which problems are formulated can be static or dynamic. It should be reminded that various problems in different scientific disciplines (as listed in section 2) which are based on multi/infinite dimensional description are effected by the tensor state space description for linear dynamical systems. Even static systems where the state space structure is a multidimensional linear space, utilization of tensor linear operator, tensor algebra techniques provide convenient tools for formulation as well as solution of them. The above generic principles are easily illustrated with the typical problem of response determination of certain multi/infinite dimensional linear systems (whose dynamics are captured through the Tensor State Space Representation ). Details are avoided for brevity.

4.5 STOCHASTIC DYNAMICAL SYSTEMS In the following, multi/infinite dimensional versions of time-series models are discussed. They are the multi/infinite dimensional versions of Auto-Regressive (AR), Auto-Regressive Moving Average (ARMA) models. The models are formally described utilizing the tensor linear operator for the variables. The discrete time, multi/infinite dimensional versions of AR, ARMA models are given by

Yi 1,..., ir (n + 1) = Ai 1,..., ir ; j 1,..., jr (n) ⊗ Yj 1,..., jr (n) + Wi 1,..., ir (n),

(4.13)

Yi 1,..., ir (n + 1) = Bi 1,..., ir ; j 1,..., jr ⊗ Yj 1,..., jr (n) + Vi 1,..., ir (n) + Ci 1,..., ir ⊗Vj 1,..., jr (n − 1) + Di 1,..., jr ⊗ Vj 1,..., jr (n − 2)

(4.14)

where ⊗ denotes the inner product and the variables such as⊗Yj 1,..., jr (n) are tensors. The


71

noise models Wi 1,..., ir (n), Vi 1,...,ir (n) are multidimensional versions of white noise. As in one dimension, the continuous time versions of these models are based on utilizing a continuous time index t, in the place of discrete time index n and replacing the noise models in (4.13 and 4.14) by the continuous time white noise or colored noise models. The formal description is avoided for brevity. The above models (which effectively reduce to the one dimensional models in the one dimensional case) enable one to derive various important details related to such stochastic processes in multi/infinite dimensions. For instance, the autocorrelation tensors, the power spectrum are derived based on the well known techniques for one dimensional systems. It should be noted that the multi/infinite dimensional power spectrum estimation problem (formulated using local state etc.) was well known to be very difficult. Thus, the utilization of tensor linear operators in certain multidimensional systems enabled one to invoke the results from one dimensional systems to be extended to certain multidimensional systems. Various interesting identities arise in the actual analysis. The details are avoided. In the following, state space representations for arbitrary stochastic linear systems are described. In one dimension, it is well known that the widely utilized Markov chains constitute the one dimensional stochastic linear systems. Thus, there has been research effort to extend the idea, approach to multi/infinite dimensions. Like the deterministic multi/infinite dimensional linear systems, conventionally various models based on the local state approach were developed. These are traditionally called the random field models. With the Tensor State Space Representation (TSSR) (of certain multidimensional systems) provided in section 3, stochastic multi/infinite dimensional linear systems, called structured Markov random fields, are based on the tensor linear operator. In the spirit of the one dimensional approach, the multi/infinite dimensional structured Markov random fields are homogeneous stochastic linear systems, described by difference equation of the following form in the discrete time/index

∏

( n + 1)

= ∏ ( n ) ⊗ P( n )

(4.15)

where Π(n) is the tensor of probabilities of the states in the state space, P (n) is the state transition tensor of the discrete time structured Markov random field. When the structured Markov random field is homogeneous, then P(n) = P . Both P(n), P are stochastic tensors. In the continuous time, the multi/infinite dimensional structured Markov random field is described by means of a generator tensor. It is given by •

d

∏ (t ) = dt π (t) = π (t) ⊗ Q(t)

(4.16)

where Π(t) is the tensor of probabilities of states in the state space at time t, Q (t) is the generator tensor of the continuous time strucured Markov random field. Q(t) satisfies the properties of a generator tensor. The equilibrium distribution of states in the discrete as well as continuous time/index

72


structured Markov random field are derived through the utilization of the spectral representation theorem of the linear operator (tensor) utilizing the eigenvalues and eigentensors of the linear operator. When the state transition tensor as well as generator tensor have the G/M/1-type structure, M/G/1-type structure (Neu), the invariant distribution of the random field has the tensor geometric form. The derivation of the form of invariant distribution and efficient recursions for the invariant distribution follow from a generalization of the results in one dimension. In the following, state space representations for various types of multidimensional stochastic dynamical systems that are commonly utilized in electrical engineering are discussed. In the discrete time, the multi/infinite dimensional dynamical system is described by the difference equation of the following form:

X(n + 1) = A(n) ⊗ X(n) + B(n) ⊗U(n) + W (n) Y(n) = C(n) ⊗ X(n) + V (n) + D(n) ⊗ U(n)

(4.17)

The tensors A(n), B(n), C(n), D(n) and the state, input, output tensors are of compatible dimension and order. The noise terms are multi/infinite dimensional extensions of the independent, identically distributed noise model in one dimension. It is based on the following tensor based random variable/random process (like vector random variables, vector random processes) specification. Generally, they are zero mean tensors (each component random variable has zero mean) and as a sequence constitute independent tensor random variables. This model is the simplest model that is commonly utilized in stochastic control theory (ZoP), (SaW). Utilizing Tensor State Space Representation (TSSR), Unified Theory of Control, Communication and Computation is formalized in (Rama 4).

Co var iance tensor {W (m), W (n)} = Q(m) δ (m − n), Co var iance tensor {V (m), V (n)} = R(m) δ (m − n), Co var iance tensor {W (m), V (n)} = 0

(4.18)

These plant noise and measurement noise models are assumed to be independent of the normal random initial state tensor, X( ). The continuous time multi/infinite dimensional stochastic models utilize continuous time I.I.D. noise (as in one dimension). The state space model description has an additive I.I.D. noise term to those described in section 3. With the above state model, theorems in one dimensional stochastic control are extended to multi/infinite dimensions, since the matrix linear operator is replaced by the tensor linear operator. In translating the results inner/outer product between vectors/matrices are replaced by those between the tensors/tensors. Now, we consider a noise model which describes processes which are more complicated than the ones considered previously. The colored noise model considered in ARMA time series model is a special case version of the following noise model. In this model, the noise processes constitute a structured Markov random field in multi/infinite dimensions. The


73

plant noise model and measurement noise are uncorrelated/independent. The noise models satisfy the following equations.

X(n + 1) = A(n) ⊗ X(n) + B(n) ⊗U(n) + L(n) Y (n) = C(n) ⊗ X(n) + M(n) + D(n) ⊗ U(n)

(4.19)

L(n), M(n) are discrete time structured Markov random fields. The fact that Markov random field is a stochastic linear system enables one to apply the stochastic dynamic programming. In the above noise model, the plant and measurement noise are made to be the most general models that are conceivable, while at the same time they are tractable. The continuous time version of the state space model has an additive term added to those in section 3. With the above state space representation, various results developed in one dimensional stochastic control theory (SaW) are extended to multi/infinite dimensional systems utilizing the generic principles described in section 3. Thus, various recursive forms for state estimation, filtering and prediction are translated from one dimensional systems to multidimensional systems, particularly with the I.I.D. form of noise. The time series model discussed at the beginning of the section with tensor state space representation, led the author to provide very detailed linear prediction type results in multi/ infinite dimensions when the noise process is white as well as colored. Thus, the linear prediction theory, which was so successful in theoretical as well as practical applications is successfully (in mathematical completeness) advanced to multi/infinite dimensions by the author with the tensor state space representation. The mathematical equations look familiar with tensor products being utilized in the equations. It should be noted that using the signal and noise models described in this section, multidimensional versions of Wiener and Kalman filters can easily be derived. Various results on estimation, prediction and control are translated from one dimension to multidimension (Rama 4) (when the multidimensional system has Tensor State Space Representation i.e. TSSR). In summary various results developed in one dimensional stochastic control theory, theory of one dimensional random processes are extended to multi/infinite dimensions through the Tensor State Space Representation.

4.6 DISTRIBUTED DYNAMICAL SYSTEMS Distributed dynamical systems are a class of systems which are more general than the dynamical systems considered above in some sense. They arise in various practical applications such as the electrical transmission lines (distributed inductance, capacitance, resistance along the line), image models, models of tomographic images of brain etc. One/multi/infinite dimensional systems in which the tensors which appear in the system dynamics that vary with time are one of the simple illustrations of distributed dynamical systems. These systems illustrate a form of non-homogeneity in the evolution


74

of the system in the state space i.e. a dependence on the discrete/continuous time index of the manner in which the state coupling, input coupling, output coupling tensors vary with time, resulting in a distributed nature of the manner of state transitions depending on the location i.e. discrete/continuous time index. This naturally motivates considering systems, based on practical applications, in which the state transitions in multi/infinite dimensions depend on the location. This is once again reminiscent of the conventional models of two/ multidimensional signal processing. To formally provide models of distributed dynamical systems in multi/infinite dimensions, the following notation from tensor algebra/analysis is introduced.

Tensor Functions of Multiple Arguments It is a rule assigning a value of a tensor B to each admissible value of a set of variables (t1, t2 ,..., ts ). To indicate such a function, we write

Ci 1,..., in = Ci 1,..., in (t1 ,..., ts )

(4.20)

In the models of distributed systems described in the following, utilizing tensor linear operators, the state, input, output variables are functions of multiple discrete time/index or continuous time/index. The following concept from tensor analysis is also extremely helpful. Tensor Field: By a tensor field, we mean a rule assigning a unique value of a tensor to each point of a certain volume V ( V may be all of space). Let r be the radius vector of a variable point of V with respect to the origin of some coordinate system. Then, a tensor field is indicated by writing

Ai 1,..., in = Ai 1,..., in (r )

(4.21)

if the tensor is of order n. A special class of tensor fields are nonstationary fields, which are functions of both space and time i.e. of both the vector r and the scalar t:

ϕ = ϕ (r , t ), A = A(r , t )

(4.22)

A tensor field is said to be homogeneous if it has no spatial dependence. In this case, the above reduces to A = A(t)

(4.23)

Tensor fields which are continuous are of utility in physical applications and in modeling various real life dynamical systems. Non-stationary fields are of utility in modeling distributed dynamical systems. It will be evident to an intelligent reader, how the above concepts are utilized in the following models of distributed dynamical systems. Particularly, tensor fields enable one to define dynamical systems over regions in the higher dimensional space which are not


75

bounded by hyperplanes. Such systems are of importance in various practical applications.

Quarter Plane Causal Distributed Dynamical Systems Motivated by the quarter plane causal model familiar from conventional two dimensional system theory, the author defines the following model of a linear system distributed in the plane or two dimensional distributed system. It is given by (1)

(2)

Xi 1,..., ir ( h + 1, k + 1) = A i 1,..., ir ; j 1,..., jr ( h, k + 1) ⊗ X j 1,.., jr (h , k + 1) + A i 1,..., ir ; j 1,..., js ( h + 1, k ) ⊗ X j 1,..., js ( h + 1, k ) + B 1i 1,..., ir ; j 1,..., jt ( h + 1, k ) ⊗ U j 1,..., jt ( h + 1, k )

Yi 1,..., ip (h , k ) = Ci 1,..., ip ; j 1,..., jr ( h, k ) ⊗ X J 1,..., jr ( h, k )

(4.24) (4.25)

where X(h, k ), U(h,k), Y(h,k) are the values of the local state tensor, input tensor and output tensor at (h, k) ∈ Z × Z. The multidimensional extension of this model is described based on the same spirit in the sense that the nearest/farthest neighbourhood set is partitioned into causal/non-causal parts and utilizing it in writing the multidimensional difference equation describing the system dynamics. For instance, the half plane causal model familiar in two dimensional signal processing is written utilizing the tensor linear operator in the same spirit as the above quarter plane causal model. The spirit in which various notions of causality is introduced into the system evolution in the state space is by means of natural/artificially induced decomposition of the state space. The state space is partitioned into neighborhoods and the dynamical system is described by means of a difference equation (multi/infinite dimensional) of the following form

X{( i 1,..., in )( k )∈( N + 1)} = A(i 1,..., in ; j 1,..., jn )( k ) ⊗ X {( j 1,...., jn )(k )∈ N} + B( i 1,..., in ; j 1,..., jn )( k) ⊗ U{( j 1,..., jn )( k )∈N}

(4.26)

Y{(i 1,..., in )(k )∈N} = C( i 1,..., in ; j 1,..., jn) (k ) ⊗ X {( j 1,..., jn) (k )∈N } + D( i 1,..., in ; j 1,..., jn) (k ) ⊗ U{( j 1,..., jn )( k )∈N }

(4.27)

where N, N+1 are neighbourhood sets in the multi/infinite dimensional state space which are not necessarily bounded by hyperplanes (captured by a structure like tensors/matrices). The above state space description of a dynamical system in discrete index variables is in the most general format conceivable. The advantages of such a model is the ability to make an arbitrary choice of the neighbourhood. If the neighborhood is chosen to be one among those in the set utilized for embedding causality structure onto the state space, various models result. The continuous time version of the above model utilizes, non-stationary tensor fields. The typical system evolution equations are given by i X{( i 1,..., in )( t)∈( N + 1)} = A(i 1,..., in ; j 1,..., jn )(t ) ⊗ X {( j 1,..., jn )(t )∈ N } +


76

B( i 1,..., in ; j 1,..., jn )(t) ⊗ U{( j 1,..., jn )(t)∈N}

(4.28)

Y{(i 1,..., in )(t )∈N } = C( i 1,..., in ; j 1,..., jn) (t ) ⊗ X {( j 1,..., jn) (t)∈N} + D( i 1,..., in ; j 1,..., jn) (t) ⊗ U{( j 1,..., jn )(t)∈ N}

(4.29)

i where, Xi 1,..., in (t) is the tensor of partial derivatives (like the Jacobian matrix, we can call it a Jacobian tensor) of Xi 1,..., in (t) . Once again this is the most general model conceivable. If the neighbourhood set is represented by a tensor, we have a very important special case. If one has understood carefully, the notions of local state, local control and the essential ideas of the theory of ordinary/partial difference/differential equations, many results developed in those fields have been adopted to the case where vector-matrix variables are replaced by the tensor-tensor variables. The outcome of this mathematically formal approach is: (i) Results developed by the differential/partial differential equations community are adopted to the tensor-tensor based equations. Once again, the translation is done with relative ease, (ii) Distributed dynamical systems are modeled by using the half plane, quarter plane causal type neighbourhood models. In these models, the matrices/vectors are replaced by tensors. Various other models based on local state, local control, various types of decompositions of the state space that arise in fields such as image processing, tomography etc. are translated to the multidimensional case by replacing the vectors/ matrices by tensors. Various types of problems formulated and solved in conventional two/ multidimensional system theory are adopted to the tensor based difference/differential equations by utilizing tensor products and tensor algebra/analysis. Some illustrations of design, analysis of distributed systems are reported utilizing the tensor linear operators for local state, local control, local input, local output variables and replacing the vector/ matrix products by means of tensor-tensor products. They are avoided here for brevity.

4.7 CONCLUSIONS Utilization of tensor linear operator associated with dynamic as well as static linear systems enables one to formulate as well as solve various known as well as new problems utilizing the powerful tools of tensor algebra (Rama1). This important representation invoked by the author is hoped to have useful effect on various scientific/mathematical fields. State space representation by tensor linear operators is discovered and formalized (Rama1). It is formally demonstrated how the theory of certain multidimensional systems is developed utilizing the tensor state space representation and translations of the results from one dimensional system theory. Approaches to translate one dimensional stochastic control theory to multi/infinite dimensional systems are briefly described. New state space representations for distributed dynamical systems are developed which enable translating the results from conventional state space models of multidimensional systems. Thus, in


77

essence the tensor linear operator based representation of static as well as dynamic systems has important impact on various fields of scientific endeavour.

REFERENCES (BiF) M. Bisiacco and E. Fornasini, “Optimal Control of Two Dimensional Systems,” SIAM Journal of Control and Optimization, Vol. 28, pp. 582-601, May 1990. (BoT) A. I. Borisenko and I. E. Tarapov, “Vector and Tensor Analysis with Applications,” Dover Publications Inc., New York, 1968. (Gop) M. Gopal, “Modern Control System Theory“, John Wiley and Sons, New York. (Neu) M.F. Neuts, “Matrix Geometric Solutions in Stochastic Models”, Marcel-Dekker, Baltimore. (Rama 1) Garimella Rama Murthy, “Tensor State Space Representation: Multidimensional Systems, International Journal of Systemics, Cybernetics and Informatics (IJSCI), January 2007, page 16-23 (Rama 2) Garimella Rama Murthy, “Multi/Infinite Dimensional Neural Networks, Multi/Infinite Dimensional Logic Theory,” International Journal of Neural Systems, Vol. 15, No. 3, June 2005. (Rama 3) Garimella Rama Murthy, “Multidimensional Neural Networks: Multidimensional Coding Theory:Constrained Static Optimization” Proceedings of 2002 IEEE International Workshop on Information Theory. (Rama 4) “Optimal Control, Codeword, Logic Function Tensors: Multidimensional Neural Networks, IJSCI, October 2006, Pages 9-17. (SaW) Sage and White, “Optimal Control Theory,” Academic Press. (Zop) R. Zoppoli and T. Parisini, “Learning Techniques and Neural Networks for the Solution of N-stage Non-linear No n-quadratic Optimal Control Problems,” Topics in 2-d System Theory, 1992.


CHAPTER

5

Unified Theory of Control, Communication and Computation: Multidimensional Neural Networks

5.1 INTRODUCTION In the mid 1940s, Norbert Wiener coined the word Cybernetics for the research field dedicated to understand the control, communication, computation and other such functions of living systems. It is well agreed that these functions of living systems are controlled by various functional sub-assemblies in the brain synthesized through bio-chemical circuits. Research work on this field was pursued by several researchers in diverse fields. The multidisciplinary effort resulted in progressing the literature on the subject. But no formally precise discoveries were made. Also, starting in 1950s, the research efforts in electrical engineering discipline led to the isolated theories of control, communication and computation. The central goal of these three fields is summarized in the following: • The problem of communication is to convey a message from one point in space and time to another point in space and time as reliably as possible. • The problem of control is to move a system from one point in state space to another point in state space such that a certain objective function is minimized • The problem of computation is to process a set of input symbols and produce another set of output symbols based on some information processing operation. These three problems, on the surface seem to be unrelated to one another. Also, in the mid 1960s, several researchers became interested in the mathematical model of the nervous system. This effort was meant to complement the research in cybernetics. Hopfield/ Amari succeeded in providing an abstract model of associative memory. Based on this abstract model, researchers are led to the following question which remained unanswered. Question: Is it true that the functional units responsible for control, communication and computation are synthesized through a network of homogeneous neurons?

80


Occasionally research efforts led to establishing some relationship between the three fields. But, in this chapter it is shown (with mathematical clarity and preciseness) that in the sense of optimization ( consolidating the earlier efforts of other authors) of some objective function, these three problems are related to one another leading to one form of unification. From a practical point of view, this unification leads to design of brain of powerful robots. With the efforts of the author, Boolean Logic theory was generalized to multi/infinite dimensions using an optimization approach (Rama 1). This approach led to the area of multidimensional neural networks (Rama 1). Also using the generalization of results in (BrB) in one dimension, multidimensional linear as well as non-linear codes are related to multidimensional neural networks. Thus using these results the research fields: Computation and Communication are related through the common thread of neural networks. In this paper, the main achievement of the author is to show that optimal control tensors of certain multidimensional systems are synthesized as the stable states of neural networks. Thus utilizing the results summarized in this paragraph, Unified Theory of Control, Communication and Computation is generalized to multidimensional systems. This chapter is organized in the following manner. In section 2, unification of control, communication and computation in one dimensional systems is summarized. In Section 3, the discovery and formalization of Tensor State Space Representation of certain multidimensional systems is briefly discussed. Using this representation, optimal control tensors (in a well known criteria of optimality) are shown to constitute the stable states of a multidimensional Hopfield neural network. In Section 4, utilizing the results in (Rama 1), (Rama 2), Unified Theory of Control, Communication and Computation in multidimensional systems is formally described. Conclusions are reported in Section 5.

5.2 ONE DIMENSIONAL LOGIC FUNCTIONS, CODEWORD VECTORS, OPTIMAL CONTROL VECTORS: ONE DIMENSIONAL NEURAL NETWORKS Researchers such as Hopfield realized that an associative memory is associated with optimizing a quadratic form over the hypercube. Other authors also realized that the concept of a logic gate (CAB) (in one dimension), concept of error correcting code (BrB) could be related to one dimensional neural networks (optimizing a quadratic/higher degree form). These efforts are summarized in the following paragraphs. The essential goal of this section is to summarize unification of control, communication and computation functions (in one dimensional systems) through the common thread of one dimensional neural networks.

One Dimensional Logic Theory: One Dimensional Neural Networks One dimensional logic theory as well as logic synthesis deal with information processing logic gates and logic circuits which operate on one dimensional arrays of zeroes and ones (or more generally one dimensional arrays containing finitely many symbols). The


81

operations performed by AND, OR, NOR, NAND, XOR gates have appropriate intuitive interpretation in terms of the entries of the one dimensional arrays i.e. vectors. Research in the area of artifical neural networks led to the problem whether all one dimensional logic gates can be synthesized using a single layer neural network. Chakradhar et al. provided an answer to the problem. They showed that the set of stable states of a Hopfield neural network correspond to one dimensional logic functions (CAB). Equivalently, the input and output signal states of a logic gate are related through an energy function. The outputs correspond to the stable states of neural network (which constitute the local optima of the energy function). Thus, in a well defined sense, one dimensional neural networks and logic theory are related.

One Dimensional Error Correcting Codes: One Dimensional Neural Networks In (BrB), several ways of relating the concept of neural networks and the concept of error correcting codes are presented. Specifically it is shown that, given a linear block code, a neural network can be constructed in such a way that every local maximum of the energy function corresponds to a codeword and every codeword correspond to a local maximum. Also it is shown that performing maximum likelihood decoding in a linear block error correcting code is shown to be equivalent to finding a global maximum of the energy function of certain neural network. Thus, one dimensional neural networks and error correcting codes are related.

One Dimensional Optimal Control Vectors: One Dimensional Neural Networks In dealing with the problem of storage of data in magnetic and optical recording systems, Wyner formulated an important open research problem (GoC). The problem is “Consider a Single Input, Single Output, (SISO) linear time invariant continous time system. Consider the input which is constrained to assume values between +1 and –1. Determine the optimal input signals which maximize the total output energy over a finite horizon”. This problem was solved by the author in (Rama 5) and independently by Honig et al. (HoS). In (RKB), the author formulated and solved the problem in the case of SISO discrete time, linear time invariant systems. The result in the case of discrete time SISO systems shows that the optimal control vectors over a finite horizon constitute the stable states of a Hopfield Neural Network. Thus optimal control vectors are synthesized as the local optima of energy function associated with a Hopfield neural network. The associated derivation is provided in Chapter 7. Thus, the research work summarized in the previous paragraphs shows that optimal control vectors, optimal codeword vectors and optimal logic gate outputs are synthesized as the stable states of a one dimensional neural network (not necessarily same). Hence the three research areas of control, communication and computation are unified using the common thread of neural networks. One should note that the unification is done in one dimension (one independent variable). In the following, we extend the unification to multidimensions. Particularly in the following section, the main achievement of the author (in this chapter) is discussed.


82

5.3 OPTIMAL CONTROL TENSORS: MULTIDIMENSIONAL NEURAL NETWORKS In the case of one dimensional linear systems, it was shown that the state space representation of the dynamics is much better than input-output description. Specifically, state space representation naturally leads to concepts such as controllability, observability associated with the system. Unfortunately, in the case of multidimensional systems, there is no natural notion of causality. Thus system theorists introduced notions such as quarter-plane causality, halfplane causality etc by partitioning the index set for state variables. In contrast to these approaches, the author discovered and formalized (Rama 3), Tensor State Space Representation (TSSR) of CERTAIN multidimensional systems. It is discussed in (Rama 3) that this particular representation enables transferring results from one dimensional systems (with vector-matrix state space representation) to certain multidimensional systems. In summary, CERTAIN multi/infinite dimensional discrete time/index dynamical systems can be described by means of a state space description of the following form:

Tensor State Space Representation of Certain Discrete Time Systems Discrete Time Systems X( i 1,..., ir ) (n + 1) = A( i 1,..., ir ; j1,..., jr ) (n) ⊗ X( j 1,..., jr ) (n) + B( i 1,..., ir ; j 1,..., jp ) (n) ⊗ U ( j 1,..., jp ) (n), Y( l1,..., ls ) (n) = C( l1,..., ls ; j 1,..., jr ) (n) ⊗ X( j 1,..., jr ) (n) + D ( l1,..., ls ; j 1,..., jp ) (n) ⊗ U( j 1,..., jp ) (n).

(5.1)

Where ⊗ denotes inner product operation between compatible tensors (BoT). Also in (5.1), A(n) is an m dimensional tensor of order 2r (called the state coupling tensor ), X(n) is the state of the dynamical system at the discrete time index n, whereas X(n+1) is the state of the system at the discrete time index n+1. Furthermore B(n) is an m dimensional tensor of order r+p ( called the input coupling tensor ), Y(n) is an output tensor of dimension m and order s. U(n) is an m dimensional input tensor of order p (varying with the discrete time index of order p) and C(n) (called the state coupling tensor to the output dynamics) is an m-dimensional tensor of order (s + r), D(n) is the input coupling tensor to the output dynamics of dimension m and order s + p. With the above important representation of certain multidimensional systems, we formulate and solve an important problem in optimal control of certain multidimensional systems. The solution of the problem shows that the optimal control tensors are synthesized as the stable states of a multidimensional Hopfield neural network (The connection structute of m -d Hopfield neural network is a fully symmetric tensor).


83

Problem Definition Find an admissible sequence of (realizable) input signal tensors, U(k ) for k ∈ { 0, 1, 2, ....} (with each component of the tensor being bounded in amplitude by unity (one) or without loss of generality be a fixed constant) i.e. Ui 1, i 2,..., ir ( k ) ≤ 1 in order to minimize the criterion  −1  kf J = 2  ∑ Yin ,..., i 1 ( k ) ⊗ Yi1,..., in ( k )   k=0 

(5.2)

subject to X (n +1) = A(n ) ⊗ X (n) + B(n ) ⊗ U (n)

(5.3)

Y (n ) = C ( n ) ⊗ X ( n ) (5.4) where A(n), B(n), C(n), D(n) are tensors arising in the system dynamics of the discrete time multi/infinite dimensional system. Furthermore, X(n) is the state tensor of the system. These tensors which arise in the system dynamics are of compatible dimensions. Without loss of generality, a multi-input, multi-output multidimensional linear system is considered. Let the impulse response tensor of the system be denoted by h(k, l). This is the discrete time version of the problem given in (GoC) for CERTAIN discrete time multidimensional systems. The open problem given in (GoC) is solved in (Rama 5).

Problem Definition The optimality condition is derived through the application of the maximum principle or equivalently, the dynamic programming principle. The application of dynamic programming enables us to derive the necessary as well as sufficient condition through the principle of optimality in some cases. Discrete time, Time Varying Linear Systems: Let U(k) , k = 0, 1, 2, ..... k f − 1 be the optimal control tensor sequence, and let X (k) , k = 0,1,2 ,..., be the state response of the linear system due to the input tensors U(k), uniquely specified by (5.3), (5.4) and the initial condition of the linear dynamical system. Then, under reasonable assumptions, discussed in the application of the discrete maximum principle (SaW), it is shown that there exists a non-trivial function satisfying δ H(Xk , Uk , λk + 1 , k ) λk = (5.5) δ Xk where the Pontryagin function/Hamiltonian is given by

H(X k , U k , λk + 1 , k ) =

−1 (C(k ) ⊗ X(k ))in ,..., i1 ⊗ (C(k ) ⊗ X(k ))i1,..., in + 2

λil ,..., i 1 (k + 1) ⊗ [ A(k ) ⊗ X (k ) + B(k ) ⊗ U (k )]

(5.6)


84

Thus, the adjoint tensor equation for the problem is given by

λ ( k ) = − C jm , jm −1,..., j 1 ( k ) ⊗ Yi 1,..., ip ( k ) + Ais ,..., i 1 ( k ) ⊗ λ ( k + 1)

(5.7)

Since, the terminal state is unspecified, we have

λ ( k f ) = − C jm ,..., j 1 ( k f ) ⊗ Yi 1,..., ip ( k f )

(5.8)

This will provide the terminal condition for solving (5.7). Since the input tensor sequence is constrained, it must necessarily satisfy

H (X k ,U k , λk + 1 , k ) = Min H ( Xk ,V , λk + 1 , k ) for all k = 0,1,..., k f − 1, V ∈T Where

T is the constraint set.

Thus, U ( k ) = − Sign (Bsl ,..., S1 (k ) ⊗ λ t 1 ,...., tn ( k + 1))

(5.9)

Solving (5.7) for λ(k + 1) and substituting in (5.9), we arrive at the optimal control sequence. When the constraint set is other than a hypercube, various well known techniques from mathematical programming for different constraint sets such as a convex polytope, convex polyhedra are invoked in the context of quadratic programming. The cost function is quadratic and it is optimized over various types of constraint sets such as the one described previously. With the terminal state specified, the equation (5.7) is recursed backwards to arrive at the optimal control tensor in the case of multi/infinite dimensional systems. Thus, an efficient computational form for solving the two point boundary value problem is derived in the following. It should be noted that, we derive the expression for λk +1 in the case of certain linear time varying multi/infinite dimensional dynamical systems

λ ( k ) = − C jm ,..., j 1 ( k ) ⊗ Yi 1,..., ip ( k ) + Ais ,..., i 1 ( k ) ⊗ λ ( k + 1)

(5.10)

starting with the terminal condition, recursing backwards. Remark Before we proceed further, it should be reminded that the indices for tensor describing the order of the tensor are given values by the symbols that came to mind. The tensors in the above state space representation are of compatible order to ensure that inner and outer products make sense. Now, we return to the derivation. In the following, the notation ⊗ is utilized to denote the inner product (BoT) between the tensors of compatible order.

λt1,..., tl (k f ) = – C jm ,..., j 1 (k f ) ⊗ Yi 1,..., ip (k f )

(5.11)

λt1,..., tl (k f −1 ) = − C jm ,..., j 1 (k f −1 ) ⊗ Yi 1,..., ip (k f −1 ) − Ais ,..., i 1 (k f −1 ) ⊗ C jm ,..., j 1 (k f ) ⊗ Yi 1,..., ip (k f ) (5.12)


85

λt1,..., tl (k f − 2 ) = − C jm ,..., j 1 (k f − 2 ) ⊗ Yi 1,..., ip (k f − 2 ) − Ais ,..., i 1 (k f − 2 ) ⊗ C jm,..., j 1 (k f −1 ) ⊗ Yi 1,..., ip (k f − 1 ) − Ais ,..., i 1 (k f − 2 ) ⊗ Ais ,..., i 1 (k f − 1 ) ⊗ C jm,..., j 1 (k f ) ⊗ Yi 1,..., ip (k f )

(5.13)

λt1,..., tl ( k f − 3 ) = − Cjm ,..., j 1 ( k f − 3 ) ⊗ Yi 1,..., ip ( k f − 3 ) − Ais ,..., i 1 ( k f − 3 ) ⊗ Cjm,..., j 1 ( k f − 2 ) ⊗ Yi 1,..., ip (k f − 2 ) − Ais ,..., i 1 (k f − 3 ) ⊗ Ais ,..., i 1 ( k f − 2 ) ⊗ C jm ,..., j 1 (k f − 1 ) ⊗ Yi 1,..., ip ( k f − 1 ) – A is ,..., i 1( k f – 3 ) ⊗ Ais ,..., i 1 ( k f – 2 ) ⊗ Ais ,..., i 1 ( k f –1 ) ⊗ C jm ,..., j 1 ( k f ) ⊗ Yi 1,...,ip ( k f )

(5.14)

Thus, continuing the solution of the difference equation backwards, we have

λt1,..., tl ( k f − l ) = − Cjm ,..., j 1 ( k f − l ) ⊗ Yi 1,..., ip ( k f − l ) − Ais ,..., i 1 ( k f − l ) ⊗ Cjm,..., j 1 ( k f − l + 1 ) ⊗ Yi 1,..., ip (k f – l + l ) − Ais ,..., i 1 (k f – l ) ⊗ Ais ,..., i 1 (k f – l + 1 ) ⊗ C jm ,..., j 1 (k f – l + 2 ) ⊗Yi 1,..., ip (k f – l + 2 ) –... – A is ,...,i 1 ( k f – l ) ⊗ A is ,..., i 1 ( k f – l + 1 ) ⊗ ... ⊗ A is ,..., i 1 ( k f – 1 ) ⊗ C jm ,..., j 1 ( k f ) ⊗ Yi 1,..., (k f )

(5.15)

Let l = k f − k −1 . This implies k f − l = k +1 . Hence, by the substitution,

λtl ,..., tl ( k + 1) = − C jm ,..., j 1 ( k + 1) ⊗ Yi 1,..., ip ( k + 1) − Ais ,..., i 1 ( k + 1) ⊗ Cjm,..., j 1 ( k + 2) ⊗ Yi 1,..., ip (k + 2) − Ais ,..., i 1 (k + 1) ⊗ Ais ,..., i 1 ( k + 2) ⊗ C jm,..., j 1 (k + 3) ⊗ Yi 1,..., ip (k + 3) −... − Ais ,..., i 1 ( k + 1) ⊗ Ais ,..., i 1 (k + 2) ⊗ ... ⊗ Ais ,..., i 1 ( k + l) ⊗ C jm ,..., j 1 (k + l + 1) ⊗ Yi 1,..., ip ( k + l + 1) (5.16) Thus we have the optimal control solution for the problem given by (utilizing (5.9))

Uv 1,..., vr ( k ) = Sign (Bsl ,..., s1 (k ) ⊗ C jm ,..., j 1 (k + 1) ⊗ Y( k + 1) + l

∑B i =1

sl ,..., s1

( k ) ⊗ Ais ,..., i 1 (k + 1) ⊗ .... ⊗ Ais ,..., i 1 ( k + i) ⊗ C jm ,..., j 1 ( k + i + 1) ⊗ Yi 1,..., ip (k + i + 1) )

(5.17)

Now, utilizing the definition of the impulse response tensor of the time varying linear system, we have

Uv 1,..., vr ( k ) = Sign (Bsl ,..., s1 ( k ) ⊗ C jm ,..., j 1 ( k + 1) ⊗ Y( k + 1) + l

∑ h (k + i + 1, k ) ⊗ Y(k + i + 1)) i =1

l

= Sign (∑ h (k + i + 1, k ) ⊗ Y(k + i + 1) ) i=0

(5.18)


86

h(.,.) is the transposed tensor of the impulse response tensor. The term in the parenthesis is given by l

l

k +i +1

i=0

i=0

j=0

∑ h (k + i + 1, k) ⊗ Y(k + i + 1) = ∑ h (k + i + 1, k ) ⊗ ∑ h (k + i + 1, j) ⊗ u ( j)

(5.19)

Exchanging the order of summation, (with the help of associated index grid), we have kf

kf − k − 1

j =0

i = max imum {0, j − k − 1}

∑

∑

( h ( k + i + 1, k ) ⊗ h ( k + i + 1, j) ⊗ u( j)

(5.20)

Rewriting the above expression for optimal control, we have kf − k − 1  kf  U ∗ (k ) = Sign  ∑ (h (k + i + 1, k ) ⊗ h (k + i + 1, j )) ⊗ U ( j )  ∑  j = 0 i = max{1, j − k − 1} 

(5.21)

Let us define kf − k − 1

∑

R(k , j ) =

i = max{1, j − k − 1}

(h (k + i + 1, k ) ⊗ h (k + i + 1, j))

(5.22)

Thus, we have for the optimal control, kf

U ∗ ( k ) = Sign ( ∑ R( k , j) ⊗ U( j)) j=0

(5.23)

Now, for the time invariant linear systems, we have

R( k , j) =

kf − k − 1

∑ (h(i + 1) ⊗ h(k + i + 1 − j)

i = max {1, j − k − 1}

(5.24)

This is the energy density tensor of time invariant linear system obtained from the impulse response tensor. Thus the optimal control tensor is the stable state of a multidimensional Hopfield neural network. Continuous Time Dynamical Systems Now, we formulate and solve the continuous time versions of the problems. The continuous time versions of the problem provides us with the structure of the local optimum of a quadratic form over the continuous time multi/infinite dimensional hypercube. This is the problem where the L∞ norm of the control tensors is constrained in amplitude by unity. In the derivation of the optimal control, the following definition is necessary. Integral of Tensor Function of a Scalar Argument: By the integral of a tensor of a scalar continuous argument, we mean the tensor with the components,


∫A

i 1,..., i

(t) dt or

∫A

i 1,..., in

(t1 ,..., tn ) dt1 dt 2 ... dm

87

(5.25)

Optimal Control Problem Formulation Consider a multi/infinite dimensional linear system with continuous index/argument. The system dynamics are given by . X i 1,..., ir ( t ) = Ai 1,..., ir ; j 1,..., jr (t) ⊗ X j 1,..., jr (t ) + Bi 1,..., ir ; j 1,..., jp (t ) ⊗ U j 1,..., jp (t)

Yl 1,..., ls (t) = Cl1,..., ls ; j 1,..., jr (t) ⊗ X j 1,..., jr (t)

(5.26)

The objective function being minimized in the optimal control problem is given by tf

tf

−1 J = ∫ Yls ,..., l 1 (t) ⊗ Yl 1,..., ls (t) dt = ∫ φ (X , U , t ) dt 2 to to

(5.27)

subject to the constraint given in (5.26) and the input tensors are constrained to be on the continuous time multi/infinite dimensional hypercube. Solution: Form the Pontryagin function ( or Hamiltonian) of the problem. It is given by H (X , U , λ , t) =

−1 (C(t) ⊗ X(t))ls ,..., l 1 ⊗ (C(t) ⊗ X(t))l 1,..., ls + 2 λir ,..., i 1 (t) ⊗ ( A(t) ⊗ X(t) + B(t) ⊗ U(t))

(5.28)

Minimize the Pontryagin function H ( X , U , λ , t ) with respect to all the admissible control tensors i.e. control tensors whose components are constrained in amplitude by unity. Thus,

U*

j 1,..., jp

(t) = − Sign {Bjp ,..., j 1;ir ,..., i 1 (t) ⊗ λi 1,..., ir (t)}

Thus, the optimal control tensors for the problem is obtained from the above equation. To explicitly determine the optimal control, the adjoint equations and associated boundary conditions are given by

. δ H(X , U , λ , t) δφ ( X , U , t) δ T − λi 1,..., ir (t) = = + [ A(t) X(t) + B(t)U(t)] ⊗ λ(t), δX δX δX where

δ is a partial derivative operator, δX

(5.29)

λi 1,..., ir (t f ) = 0 The above equations (5.28), (5.29) alongwith the system dynamics described through (5.26) are solved for determining λi 1,..., ir (t)


88

i −λi 1,..., ir (t) = − Cls ,..., l1 (t) ⊗ Cl1,..., ls (t) ⊗ Xi 1,..., ir (t)  + Ajr ,..., j 1; ir ,..., i 1 (t) ⊗ λi 1,..., ir (t) with λi 1,..., ir (t f ) = 0

(5.30)

λi 1,..., ir (t) = − Ajr ,..., j 1; ir ,..., i 1 (t) ⊗ λi 1,..., ir (t) + Cl 1,..., ls (t) ⊗ Yl 1,..., ls (t) with λi 1,..., ir (t f ) = 0 .

(5.31)

The above differential equation is solved, like the state equations for the linear dynamical system, to arrive at t

a a λi 1,..., ir (t) = φ (t , t f ) ⊗ λ (t f ) + ∫ φ (t , τ ) ⊗ Cls ,..., l 1 (τ ) ⊗ Yl 1,..., ls (τ ) dτ , tf

d a φ (t ,τ ) = − Ajr ,..., j 1; ir ,..., i 1 (t) ⊗ φ a (t , τ ) dt

(5.32)

(5.33)

φ a ( t ,τ ) = I ; φ a ( t , t f ) = φ ( t f , t ) a where φ (t f , t ) is the state transition tensor. Thus, we have

t

a λi 1,..., ir (t) = ∫ φ (t ,τ ) ⊗ Cls ,..., l1 (τ ) ⊗ Yl1,..., ls (τ )dτ tf

t

=

∫ φ (τ , t) ⊗ C

ls ,..., l1

(τ ) ⊗ Yl1,..., ls (τ )dτ

tf

(5.34)

tf

= − ∫ φ (τ , t) ⊗ Cls ,..., l1 (τ ) ⊗ Yl1,..., ls (τ )dτ t

Hence, we have

−Bjp ,..., j 1; ir ,..., i 1 (t) ⊗ λi 1,..., ir (t) = tf

∫B

jr ,..., j1; ir ,..., j1

(t) ⊗ φ (τ , t) ⊗ Cls ,..., l 1 (τ ) ⊗ Yl 1,..., ls (τ )dτ

(5.35)

t

where τ

Yl1,..., ls (τ ) = ∫ Cl1,..., ls (τ ) ⊗ φ (τ , s) ⊗ B( s) ⊗ U ( s)ds to

(5.36)


89

Thus, we have − B jp , ..., j 1 ; ir , ..., i 1 (t) ⊗ λ i1, ..., ir (t) tf

=

∫

τ

B j p , ..., j1 ; ir , ..., i1 (t ) ⊗ φ (τ , t ) ⊗ C ls, ..., l1 (τ ) ⊗ [

t

∫

C(τ ) ⊗ φ (τ , s) ⊗ B(s ) ⊗ U ( s ) ds] dτ

(5.37)

t0

Exchanging the order of integration (with associated index diagram)

−Bjp ,..., j 1; ir ,..., i 1 t ⊗ λi 1,..., ir (t) = tf tf

∫ ∫B

jp ,..., i 1

(t) ⊗ φ (τ , t) ⊗ Cls ,..., l 1 (τ ) ⊗ Cl1,..., ls (τ ) ⊗ φ (τ , s) ⊗ Bi 1,..., jp ( s) dsdτ

(5.38)

0 s

But, the impulse response tensor is given by

 H j1 ,..., jp (t , τ ) = Cl 1,..., ls (t) ⊗ φ (t , τ ) ⊗ Bi 1,..., jp (τ )  t

Multidimensional Neural Networks Unified Theory

Neural Networks Theory

Neural networks

Neural Networks

Handbook Of Brain Theory And Neural Networks

Artificial Neural Networks and Information Theory

Handbook of brain theory and neural networks

Handbook Of Brain Theory And Neural Networks

Process Neural Networks: Theory and Applications

Recurrent Neural Networks

Neural Networks (Grassroots)

Neural Networks and Micromechanics

Neural Networks and Micromechanics

39.Neural Networks

Fundamentals of Neural Networks

Artificial Neural Networks

Neural Networks and Micromechanics

Pulsed Neural Networks

Pulsed Neural Networks

Invertebrate Neural Networks

Recurrent Neural Networks

Neural Networks 001

Complex-Valued Neural Networks

Neural Networks in Finance

Unified Signal Theory

Pattern recognition and neural networks

Data mining with neural networks

An Introduction to Neural Networks

Industrial Applications of Neural Networks

Bayesian nonparametrics via neural networks

Neural Networks: A Systematic Introduction

Multidimensional Neural Networks Unified Theory

Neural Networks Theory

Neural networks

Neural Networks

Handbook Of Brain Theory And Neural Networks

Artificial Neural Networks and Information Theory

Handbook of brain theory and neural networks

Handbook Of Brain Theory And Neural Networks

Process Neural Networks: Theory and Applications

Recurrent Neural Networks

Neural Networks (Grassroots)

Neural Networks and Micromechanics

Neural Networks and Micromechanics

39.Neural Networks

Fundamentals of Neural Networks

Artificial Neural Networks

Neural Networks and Micromechanics

Pulsed Neural Networks

Pulsed Neural Networks

Invertebrate Neural Networks

Recurrent Neural Networks

Neural Networks 001

Complex-Valued Neural Networks

Neural Networks in Finance

Unified Signal Theory

Pattern recognition and neural networks

Data mining with neural networks

An Introduction to Neural Networks

Industrial Applications of Neural Networks

Bayesian nonparametrics via neural networks

Neural Networks: A Systematic Introduction

Recommend Documents