Radu Dogaru Systematic Design for Emergence in Cellular Nonlinear Networks
Studies in Computational Intelligence, Volume 95 Editor-in-chief Prof. Janusz Kacprzyk Systems Research Institute Polish Academy of Sciences ul. Newelska 6 01-447 Warsaw Poland E-mail:
[email protected] Further volumes of this series can be found on our homepage: springer.com Vol. 70. Javaan Singh Chahl, Lakhmi C. Jain, Akiko Mizutani and Mika Sato-Ilic (Eds.) Innovations in Intelligent Machines-1, 2007 ISBN 978-3-540-72695-1 Vol. 71. Norio Baba, Lakhmi C. Jain and Hisashi Handa (Eds.) Advanced Intelligent Paradigms in Computer Games, 2007 ISBN 978-3-540-72704-0 Vol. 72. Raymond S.T. Lee and Vincenzo Loia (Eds.) Computation Intelligence for Agent-based Systems, 2007 ISBN 978-3-540-73175-7 Vol. 73. Petra Perner (Ed.) Case-Based Reasoning on Images and Signals, 2008 ISBN 978-3-540-73178-8 Vol. 74. Robert Schaefer Foundation of Global Genetic Optimization, 2007 ISBN 978-3-540-73191-7 Vol. 75. Crina Grosan, Ajith Abraham and Hisao Ishibuchi (Eds.) Hybrid Evolutionary Algorithms, 2007 ISBN 978-3-540-73296-9 Vol. 76. Subhas Chandra Mukhopadhyay and Gourab Sen Gupta (Eds.) Autonomous Robots and Agents, 2007 ISBN 978-3-540-73423-9 Vol. 77. Barbara Hammer and Pascal Hitzler (Eds.) Perspectives of Neural-Symbolic Integration, 2007 ISBN 978-3-540-73953-1 Vol. 78. Costin Badica and Marcin Paprzycki (Eds.) Intelligent and Distributed Computing, 2008 ISBN 978-3-540-74929-5 Vol. 79. Xing Cai and T.-C. Jim Yeh (Eds.) Quantitative Information Fusion for Hydrological Sciences, 2008 ISBN 978-3-540-75383-4
Vol. 83. Bhanu Prasad and S.R.M. Prasanna (Eds.) Speech, Audio, Image and Biomedical Signal Processing using Neural Networks, 2008 ISBN 978-3-540-75397-1 Vol. 84. Marek R. Ogiela and Ryszard Tadeusiewicz Modern Computational Intelligence Methods for the Interpretation of Medical Images, 2008 ISBN 978-3-540-75399-5 Vol. 85. Arpad Kelemen, Ajith Abraham and Yulan Liang (Eds.) Computational Intelligence in Medical Informatics, 2008 ISBN 978-3-540-75766-5 Vol. 86. Zbigniew Les and Mogdalena Les Shape Understanding Systems, 2008 ISBN 978-3-540-75768-9 Vol. 87. Yuri Avramenko and Andrzej Kraslawski Case Based Design, 2008 ISBN 978-3-540-75705-4 Vol. 88. Tina Yu, David Davis, Cem Baydar and Rajkumar Roy (Eds.) Evolutionary Computation in Practice, 2008 ISBN 978-3-540-75770-2 Vol. 89. Ito Takayuki, Hattori Hiromitsu, Zhang Minjie and Matsuo Tokuro (Eds.) Rational, Robust, Secure, 2008 ISBN 978-3-540-76281-2 Vol. 90. Simone Marinai and Hiromichi Fujisawa (Eds.) Machine Learning in Document Analysis and Recognition, 2008 ISBN 978-3-540-76279-9 Vol. 91. Horst Bunke, Kandel Abraham and Last Mark (Eds.) Applied Pattern Recognition, 2008 ISBN 978-3-540-76830-2 Vol. 92. Ang Yang, Yin Shan and Lam Thu Bui (Eds.) Success in Evolutionary Computation, 2008 ISBN 978-3-540-76285-0
Vol. 80. Joachim Diederich Rule Extraction from Support Vector Machines, 2008 ISBN 978-3-540-75389-6
Vol. 93. Manolis Wallace, Marios Angelides and Phivos Mylonas (Eds.) Advances in Semantic Media Adaptation and Personalization, 2008 ISBN 978-3-540-76359-8
Vol. 81. K. Sridharan Robotic Exploration and Landmark Determination, 2008 ISBN 978-3-540-75393-3
Vol. 94. Arpad Kelemen, Ajith Abraham and Yuehui Chen (Eds.) Computational Intelligence in Bioinformatics, 2008 ISBN 978-3-540-76802-9
Vol. 82. Ajith Abraham, Crina Grosan and Witold Pedrycz (Eds.) Engineering Evolutionary Intelligent Systems, 2008 ISBN 978-3-540-75395-7
Vol. 95. Radu Dogaru Systematic Design for Emergence in Cellular Nonlinear Networks, 2008 ISBN 978-3-540-76800-5
Radu Dogaru
Systematic Design for Emergence in Cellular Nonlinear Networks With Applications in Natural Computing and Signal Processing
With 80 Figures and 10 Tables
Dr. Radu Dogaru Polytechnic University of Bucharest Department of Applied Electronics and Information Engineering "Leu" building (B), Bvd. Iuliu Maniu 1-3, Room B232 061071 Bucharest, Romania
[email protected] ISBN 978-3-540-76800-5
e-ISBN 978-3-540-76801-2
Studies in Computational Intelligence ISSN 1860-949X Library of Congress Control Number: 2007939778 c 2008 Springer-Verlag Berlin Heidelberg This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Cover design: Deblik, Berlin, Germany Printed on acid-free paper 9 8 7 6 5 4 3 2 1 springer.com
Preface
The main problem addressed in this book came out during a Fulbright research fellowship stage at U.C. Berkeley (California, USA, 1996–1998). Then, I had the opportunity to work in the research group of Leon Chua on a subject called CNN (cellular neural/nonlinear network). The CNN, developed in the end of the 1980s was an important step ahead in getting cellular computing closer to practical applications. The CNN is actually a cellular array made of many identical elements interconnected in a neighborhood. But unlike the cellular automata (CA) the CNN was designed as a circuit-oriented architecture being developed up to the point where it serves as a low-power smart sensor or visual microprocessor capable to acquire and process images or multi-dimensional signals in general with processing speeds of the order of 1012 (Tera) operations per second. Today, there are several academic and industrial groups providing CNN-based solutions for various practical problems, particularly when high speed processing at low power is needed. Also, researchers in the area of nano-technology recognize cellular computing as a suitable computing paradigm to the specific of these technologies where many identical active elements are available on a mass proportion. Researchers in biology also recognize the cellular paradigm as a well-suited paradigm for models of natural systems. Indeed there is a similarity, as biological systems essentially are functionally meaningful aggregates of mostly local interconnected cells. A brief review of the cellular computing paradigm, its relation to natural computing in general and the state of the art of its use in applications are given in Chaps. 1 and 2 of this book. At the moment of my arrival at Berkeley a well-established theory of the standard CNN architecture was already developed, and many researchers worked on finding novel applications. Still several problems were recognized as needed to be solved. A first one was to define universal cells, i.e. to find a general nonlinear system describing any arbitrary local interaction (e.g. a Boolean function) within a family of similar functions. A set of parameters of the nonlinear system, called a gene, is supposed to prescribe any particular “individual” from the family of functions. The parallelism with biology is obvious: Here, the DNA string represents the gene while the unfolding mechanism producing and multiplying biological cells may be associated to the cellular nonlinear system and its dynamics. A theory of piecewise-linear representation of cells was then developed and briefly recalled in this book in Chap. 3 since later it turned out to be relevant in a consistent framework for predicting emergence, or as we will call it often throughout this book, for the “design for emergence” of cellular systems. Indeed, a second problem posed in the mid 1990s for the CNN researchers was to develop a consistent theory of emergence, or in other words to find a way to
vi
Preface
anticipate how the cellular array (CA) will behave without extensively running the entire CA system. It is a crucial design problem since the space of possibilities in choosing the cells’ gene is huge and looking exhaustively to cellular systems behaviors associated with each and all of them is impossible. Researchers in biology face a similar problem in the attempt to “decode” the genome i.e. to understand how different biological behaviors (disorders, diseases, etc.) emerge from a known genome. A great step towards a theory of emergence is Leon Chua’s theory of local activity (published in 1997) as it is able to predict, to a certain degree, the nature of the emergent phenomena within the cellular array, while investigating only the simple nonlinear system associated to the isolated cell. A series of papers were published on this topic providing clues and practical methods for locating the “edge of chaos” as a narrow region within the cells’ parameter (or gene) space. Such widely known nonlinear systems as Fitz-Hugh Nagumo model of excitable nervous membranes, the Brusselator and the Gierer-Meinhard’s model for morphogenesis were considered, and it was verified that indeed local activity is powerful enough to locate previously reported emergent behaviors but also to discover new ones. Still, the theory of local activity has some limitations in that it relies on some circuit theory theorems and therefore assumes that the model to be investigated shall be cast into a circuit specific framework. Therefore, the theory can be applied and was done so far solely for Reaction–Diffusion systems. However, there are many other cellular or network-based computing paradigms, as briefly recalled in Chap. 3, that may benefit from a general theory of emergent computation. Here comes the novelty of this book, which reports recent results aiming to establish a global theory and practice for a “design for emergence” framework. The first step in designing for emergence was to recognize that one needs to measure emergence. This is the aim of Chap. 4, where several scalar measures were introduced and defined as algorithms operating during the running of a cellular system. Some hints on how to associate various values of these measures to various categories of emergent behaviors were also given. Chapter 5 comes with an additional measure which later proved to be very interesting in that an equivalent form of it could be determined without running the CA but solely based on the cells’ structure. With accurately defined measures of emergence the next step towards a “design for emergence” framework was to recognize and define tools to select among the entire space of possibilities only those genes related to desired emergent behaviors, described often in a fuzzy natural language. Their definition and some practical aspects of their use are the subjects of Chap. 6. The most interesting result comes in Chap. 7 where the grounds for a general theory of emergence are provided, using tools from the information theory. Here, we provide analytic formulae to determine a measure of emergence solely based on cell’s structure (its gene) and its neighborhood. Comparisons with the experimentally determined (during the CA running) similar measure confirmed the validity of this new theory called a theory of the probabilistic exponents of growth, for reasons to be detailed within the chapter.
Preface vii
As its name anticipates, probabilities and other notions from information theory play a crucial role in its development. Unlike the theory of local activity, this new theory is rather general and it does not depend too much on the specific cellular system. Although we exemplify it for discrete-time binary-state cellular automata, hints are provided on how to expand it to any type of system. Also, the neighborhood and its parameters are included in the theory, improving its predictive value, when compared with theories such as local activity where emergence is predicted with some degree of uncertainty left on the behalf of the interconnectivity. It can also explain many situations that were previously observed empirically in cellular systems. While doing several research stages at Institute of Microelectronic Systems at T.U. Darmstadt, Germany in a research group oriented towards solving many practical problems, it turned out that there is a huge demand from the industry to build low power intelligent systems. It later turned out that natural computing approaches like the use of cellular automata systems for various signal processing tasks may dramatically reduce the implementation complexity when compared to traditional solutions. What was needed, were the CA design tools. Once they have been introduced in the previously mentioned chapters, the last chapter of this book presents three different innovative applications of cellular automata in signal processing. As cellular systems are part of the natural computing systems and the focus of many interdisciplinary research groups, and since myself was formed as an electrical and computer science engineer I wrote the book in an easily to understand style, without too much mathematical formalism. Some programs are also provided within the text to facilitate a faster access of the interested reader to the design for emergence tools. Although mathematicians may not like the lack of formal proofs I encourage readers from this area to browse the book, and I hope that they may extract some useful ideas for a more formal treatment. Much of the work reported here was done with the support of an Alexander von Humboldt research fellowship, during a stage at T.U. Darmstadt between 2005 and 2006. Some of the chapters in the book were taught as lectures in a course on Natural Computing systems I gave at Darmstadt during this stage. I am deeply indebted to Professor Manfred Glesner, the director of the research group and to many members of his team for their continuous support and hospitality during this stage, as well as for the opening they gave me to seek not only theoretical sides of cellular systems but also to look for convincing practical applications of them. I acknowledge the steady support of Professor Leon Chua who sparkled and stimulated my interest for the fascinating research area of cellular computing systems and shaped deeply my way of thinking. Other sponsors such as the Volkswagen Stiftung are also acknowledged here, particularly for their role in shaping my interest for solving some practical signal processing problems. I am also indebted to colleagues and my former professors from the Applied Electronics and Information Engineering as well as to many friends and colleagues, anonymous reviewers and other persons familiar to my work who gave me some critic feedback in various stages.
viii Preface
Last but not the least I thank all the members of my family for their patience and continuous support. Special thanks goes to my wife Ioana who, in addition to her understanding and patience, gave me a lot of feedback and made a lot of useful comments on the manuscript.
Radu Dogaru Bucharest, September 2007
Contents
Preface …………………………………………………………………………
v
1 Natural computing paradigms and emergent computation …………………. 1.1 Principles of natural computing ................……………………………… 1.1.1 Natural computing structures as hierarchies of interconnected cells .. 1.1.2 The principle of optimal number of entities (Ockham’s razor) …… 1.1.3 Natural computing systems are dissipative systems ………………. 1.1.4 Transient nature of the behavioral complexity of natural systems .. 1.1.5 Natural systems and recurrence ..…………………………………. 1.1.6 Emergence, complexity, and local activity of cells ……………….. 1.2 Open problems and book description ...………………………………….
1 1 1 2 3 3 3 3 4
2 Cellular nonlinear networks: state of the art and applications ………………. 7 2.1 Introduction ………….…………………………………………………. 7 2.2 Typical applications of cellular computers ……………………………... 9 2.3 Hardware platforms for implementing cellular computers …………….. 12 3 Cellular and natural computing models and software simulation ...………... 3.1 Cellular systems: cells, neighborhoods, states and dynamics .………….. 3.1.1 Genes ……………………………………………………………... 3.1.2 Discrete and continuous states and outputs ………………………. 3.1.3 Boundary conditions ……………………………………………… 3.2 Major cellular systems paradigms ……………………………………… 3.2.1 The Cellular Neural Network (CNN) model …………………… ... 3.2.2 The Generalized Cellular Automata ……………………………… 3.2.3 Reaction-Diffusion Cellular Nonlinear Networks ………………... 3.3 Matlab simulation of Generalized cellular automata and cellular neural networks …................................................................. 3.3.1 Uncoupled GCAs ………………………………………………….. 3.3.2 Coupled GCAs …….………………………………………………. 3.3.3 Simulation of standard cellular neural networks ………………….. 3.4 Nonlinear representation of cells ……………………………………… .. 3.4.1 Piecewise-linear representation and implementation …………… ... 3.4.2 Extended families of cells ……………………………….……… ... 3.4.3 Structured universes of cells …………………………….….…… .. 3.5 Modeling and simulation of semitotalistic cellular automata ………….... 3.6 Modeling and simulation of “Small-Worlds” systems ………………….. 3.7 A wider variety of cells, taxonomy and family labels ………………….. 3.8 A CA simulator for all kind of cells ……………………………………..
15 15 17 17 17 18 18 19 20 21 21 23 25 25 26 36 37 38 40 41 43
x
Contents
4 Emergence, locating and measuring it .……………………………………... 4.1 Emergence: the software engineering of cellular computing systems ….. 4.2 Visual interpretation of emergent phenomena – classes of behaviors ….. 4.3 Semitotalistic cells: a first step in narrowing the search space …..……. . 4.4 Clustering and transients and as measures of emergence ………………. 4.4.1 Visualizing complexity for an entire family of cellular systems – Wolfram classes revisited …….…................….
47 47 48 57 60 66
4.5 Simulation examples, properties, more accurate complexity measures ... 69 4.5.1 Properties of complexity measures ……………………………… .. 71 4.5.2 A composite complexity measure ………………………………… 74 5 Exponents of growth ………………………………………………….……. 5.1 Exponents of growth and their relationship to various emergent behaviors …….......................................................................................... 5.2 Mutations for continuous state cellular systems ……………………….. 5.3 Distribution of exponents of growth among families of genes …………. 5.4 Influences of the “Small World” model ………………………………… 5.4.1 The “small worlds” model allows fine tuning of the “edge of chaos” …............................................................................ 5.5 On the independence between various measures of complexities ………
77 77 85 87 89 90 92
6 Sieves for selection of genes according to desired behaviors ………………. 6.1 Introduction …………………………………………………………….. 6.2 Defining sieves ……………………………………………..................... 6.3 Examples and applications of the double sieve method …………........... 6.3.1 Intelligent life behaviors and uncertainty in using sieves ...………. 6.3.2 Classic “Life”: using sieves to locate similar behaviors …….......... 6.3.3 Other interesting emergent behaviors .……………………………. 6.3.4 Sieves to locate feature extractors ………………………………... 6.3.5 Pink-noise generators …………………………………………….. 6.3.6 Sieving and intelligence – evolving a list of interesting genes .…..
95 95 96 99 99 101 104 107 109 110
7 Predicting emergence from cell’s structure .………………………………… 7.1 Introduction .…………………………………………………………….. 7.2 Relationships between CA behavior and the Boolean description of the cell ….......................................................................…. 7.3 Parametrizations as tools to locate similar behaviors ………………….. 7.4 The theory of probabilistic exponents of growth ……………………….. 7.4.1 Uncertainty index of a cell, active and expansion areas ………….. 7.4.2 Minimal set of cells to compute the probabilistic exponent of growth ……………………………………………….. 7.4.3 Computing the cells function of probability ……………………… 7.4.4 Computing the probabilistic exponent of growth ………………… 7.5 Exponents of growth and their significance ……………………………. 7.5.1 Predicting behaviors from ID, an example ……………………… .. 7.5.2 Other properties of the probabilistic exponents of growth ……… .. 7.5.3 Comparison with the experimental exponent of growth ………….. 7.6 Conclusions …………………………………………………………… ..
11 3 113 114 116 118 120 121 123 124 126 127 127 129 130
Contents
xi
8 Applications of emergent phenomena …..………………………………… . . 8.1 Introduction ......…………………………………………………………. 8.2 Smart sensor for character recognition …..……………………………… 8.2.1 Motivation and general description …..…………………………… 8.2.2 Architecture and functionality of the CA-based sensor ………… .. 8.2.3 Feature extraction and classification ................................................ 8.2.4 Experimental results ………………………………………………. 8.3 Excitable membranes, for temporal sequence classification …………… 8.3.1 Mapping variable-length signals into terminal states ...………… .. 8.3.2 Detecting genes to build EMTMs ….…………………………… .. 8.3.3 Experimental results in sound classification ……………………… 8.4 Image compression using CA-based vector quantization .....…………… 8.4.1 Coding and decoding principle …………………………………… 8.4.2 Detecting the useful genes ....…………………………………… .. 8.4.3 Results and comparison with other compression standards ......…... 8.4.4 Aspects on hardware implementation .....…………………………
133 133 134 134 136 138 140 141 142 144 145 147 147 151 153 156
References ......……………………. ......………………………………… …. … 159 Index …………………………………………………………………………... 165
1 Natural Computing Paradigms and Emergent Computation
Natural computing was recently defined as a novel paradigm for computation where nature is taken as an example to define computational architectures and algorithms capable to solve problems efficiently and while being based on a low complexity description of its structure. A living being can be considered as performing various natural computation tasks. While exhibiting complexity in performing various functional tasks (pattern recognition, decision, orientation, optimization, planning, creative thinking, self-repair and self-reproduction, to name just a few) it is assumed that the entire development of the being is encoded within its genome. Due to recent progress in mapping the entire human genome, it is widely accepted that such a genome contains no more than several megabytes of information stored in the strings of DNA, being even simpler for more primitive species. Thus, in a simplified approach, the entire behavioral complexity of a living being can be regarded as being encoded in a relatively compact information storage structure, the DNA. This is what we will call structural information, i.e. the minimal information required to construct a computational architecture.
1.1 Principles of Natural Computing The goal of computational intelligence is to understand and exploit some basic principles of the natural computing to construct various artificial systems capable to mimic some of the functions of living beings. Figure 1.1 gives a sketch of the main ingredients and principles to be considered in analyzing and understanding a naturally inspired computing systems. Some of these principles are detailed next. 1.1.1 Natural Computing Structures as Hierarchies of Interconnected Cells As seen in the above figure, in order to achieve functionality, the gene information is unfolded in creating many similar cells in a basic hierarchy and these cells are naturally distributed so that each cell typically communicates with its immediate neighbors. The local connectivity is the essence of the cellular computing model, to be developed further. In nature, besides nearest neighbors, also long-range connections are present allowing a small fraction of distant cells to communicate. R. Dogaru: Natural Computing Paradigms and Emergent Computation, Studies in Computational Intelligence (SCI) 95, 1–6 (2008) www.springerlink.com © Springer-Verlag Berlin Heidelberg 2008
2
1 Natural Computing Paradigms and Emergent Computation
This otherwise natural fact determined Watts recently to propose and investigate the “Small Worlds” model [1]. He established that a small fraction of long-range connection is capable to dramatically change the nature of the emergent functionality. This small fraction of distant connections determines a dramatic reduction of the average path between two basic cells. Such a phenomena was previously recognized in social network, often being associated with the exclamation “What a small world!” when remote people find common acquaintances. It turns out that on average, six social links can be established between any two people in the world. The short communicating path is the result of adding only several long-range connections to the model.
Fig. 1.1. A schematic view of a natural computing system
1.1.2 The Principle of Optimal Number of Entities (Ockham’s Razor) While cell genes are interpreted to create the architecture of a natural computing system, an optimal number of basic cells forming a functional entity (called a cell from the higher level of the hierarchy, in Fig. 1.1) is established and here we may recall another principle of organization dating back from 1300s in the time of Ockham [2,3]. He stated that “…entities must not multiply without necessity” and indeed this is the case in most natural systems. When entities multiply without necessity natural systems often come to destruction like in the case of tumor growing
1.1 Principles of Natural Computing
3
or fire expansion. To limit the phenomena of multiplying without necessity, new cells will form other functional entities (i.e. a cell on a higher level of the hierarchy) and so the process unfolds until a certain number of levels of hierarchies is reached (also obeying Ockham’s principle) like in the case of a mature living being. 1.1.3 Natural Computing Systems are Dissipative Systems According to Prigogine [4] a natural system is a dissipative system, which has achieved its functionality as a “far from equilibrium” state, i.e. a state of apparent order and stability but which is in fact a complicated dynamic process. Therefore such a system requires a source of energy and since it is a dynamic process rather than a static one, it should evolve. 1.1.4 Transient Nature of the Behavioral Complexity of Natural Systems A natural computing system is not likely to have an “eternal life”. Instead it evolves through maturity to aging until it eventually “dies”, i.e. enters another (stable) dynamical regime. In a recent paper [5] it was suggested that complex and far from equilibrium dynamics (i.e. functional life) has to be always associated with a long transient process. The longer the transient, the more complex is the systems functionality. This principle inspired the introduction of the transient length measure of emergence in Chap. 4. 1.1.5 Natural Systems and Recurrence A system is recurrent if there is at least one closed loop (i.e. feedback loop) in the cell interconnection graph, i.e. a cell has always a feedback about its “actions” via its neighboring cells. 1.1.6 Emergence, Complexity, and Local Activity of Cells Quite recently, in addition to the principle of nonlinearity [4] regarded by Prigogine as a precondition for a far-from-equilibrium dynamics, the principle of local activity was introduced by Chua [7,8] to narrow the search for an ideal structure of a cell. Previously it was assumed that emergence and complexity are solely the effects of cell’s nonlinearity. Chua’s local activity theory demonstrates that in addition a cell must be locally active, i.e. capable to amplify small fluctuations received from its neighbors. In a series of papers [9–11] we demonstrated how this principle may be applied to various models of natural computation to identify proper cells (i.e. cells for which emergence is likely to occur) without being necessary to simulate the cellular system but only performing an analytical investigation of
4
1 Natural Computing Paradigms and Emergent Computation
the nonlinear ordinary differential equation describing the cell. Subdomains of the cell parameters were found (called “edge of chaos” domains) where emergent behavior and complexity is likely to occur. The principle of local activity is a first step towards the establishment of a new science of designing for emergence.
1.2 Open Problems and Book Description Looking around in nature it is clear that above principles apply for various systems ranging from living beings, economic and social networks, international relationships, computer networks, etc. Moreover, at a critical observation all functional architectures designed so far by information engineers are natural computing architectures. Indeed, any computer is built upon an hierarchy of functional modules where at the basic level one finds the basic cell to be the locally active and nonlinear transistor (or any other device exhibiting local activity). Various functions were discovered through a process of mental evolution, which gives the proper interconnection between such basic cells. A further level of hierarchy was the creation of logic gates, then the creation of memory cells, arithmetic units etc, up to the level where some flexibility was added to this construction through what is today called software, i.e. a flexible description of how different cells should be connected to perform a certain task. Algorithms then are also designed as interconnections of basic “software” cells called instructions in the computer science parlance. However, designing such systems is based on description rather than emergence. Description can be regarded as a process of transferring human knowledge into a model of existing cells. In other words we have the cells (i.e. the transistors or the logic gates or more sophisticated modules, etc.) and we describe how to interconnect them such that the resulting system will manifest a desired functionality. The typical design process starts with a specification of the end function and proceeds with a given set of rules or procedures (e.g. methods for drawing schematics or programming in software) to interconnect the basic cells available in a given technology to produce the expected result. Although description is itself an emergent process whenever novel functionality has to be described, it turns out that several challenging problems in information technology are not enough satisfactory solved using the “classic design” method. A list of problems includes but is not limited to such functions as: x x x x x x x
Self-repair and self-reproduction of complex functional architectures Voice recognition (speaker identification and message identification) Gesture recognition Visual field abstraction (i.e. generating a brief description from a visual scene) NP-complete problems optimization and decision making Orientation in a complex environment Creative thinking and problem solving.
1.2 Open Problems and Book Description
5
Actual solutions to these problems are often long and complicated descriptions resulting in messy hardware architectures or software modules and requiring many resources. On the other hand, the actual social demand is to develop a wide range of intelligent but portable products (see recent tendencies described as ubiquitous computing [12], ambient intelligence [13], organic computing [14], etc.), integrated with various sensors that are available at decreasing cost prices. Since the most important critical issue for these systems is power consumption, messy algorithms and architectures are certainly not a solution. Besides, such devices are supposed to behave naturally, i.e. perform such functional tasks which are similar in nature to functions listed above as major challenges. A better solution for designing such systems is to take the approach to be developed in this book, i.e. the design for emergence approach (first expressed in [15]). Novel technologies such as nanotechnology or molecular electronics are also better physical supports for the ideas to be developed further within this book. In the next we will describe what is the definition of emergence and how design for emergence can be achieved. Then the simplest model of a natural computing system to perform emergent computation is introduced in the form of cellular nonlinear network (CNN). While being a generalization of the widely known cellular automata (CA) model, we will see that a simple altering of this model is possible, making it compatible with the more realistic “small world” model [1]. Regarded as a potential model of universe by Zuse [16], the cellular computing model was recently proposed by Wolfram [17] as the main ingredient of a “new kind of science”. The praxis of this science may also benefit from the “design for emergence” methods [15] to be further introduced. The philosophy of designing for emergence can simply be stated as follows: Instead of giving a description for a function to be performed, one rather defines and determines the preconditions for emergence, i.e. a set of conditions to select among many possible genes associated with basic cells. Then, one simply picks genes within this reduced pool and observes (visually or automatically) details of the emergent behaviors and further selects only those genes that are best fit to the application in mind. This book passes throughout all steps of a design for emergence process and provides in the end (Chap. 8) several application examples where these principles were successfully applied. In order to understand the relevance and the potential of cellular computing models (also called cellular nonlinear networks or CNNs, cellular arrays, etc.) the next chapter will provide and overview of the field and its state of the art. Then, Chap. 3 introduces several of the most widely used cellular models. Matlab simulators are provided for each of them. Moreover, in order to cover in a unified framework a large spectrum of cellular computing arrays (CA), a convenient taxonomy of cells is introduced in the same chapter. This taxonomy relies on a piecewise-linear representation of cells as nonlinear functions rather than tables. It also allows to define families of cells as a narrowed search pool for detecting emergence. So far emergence was subjectively defined, but in order to design for emergence one needs to quantify it. A first set of measures for
6
1 Natural Computing Paradigms and Emergent Computation
emergence will be introduced in Chap. 4. Their common feature is that they are experimental, i.e. they require a simulation of the CA with some conveniently chosen initial state and number of cells during a convenient number of iterations. Although the choices of these “convenient” CA parameters induces a degree of uncertainty in these measures, they are extremely useful for a better understanding of emergence and its relationship with the cell structure. They also provide useful hints on how to tune these measures in order to select only CA genes giving a desired behavior. In fact this process of tuning and selection is further detailed in Chap. 6 where it is referred as a sieving process. Sieving with experimental measures of emergence require that a table with this measures is available and computed previously for all cells within the family. Chapter 5 is devoted to one additional measure, the exponent of growth, which provides novel information in addition to those defined previously in Chap. 4. The exponent of growth turns out to have a good predictive power for emergent properties, and although it is experimentally defined in Chap. 5, it turns out later in Chap. 7 that is the only among all measures for emergence that can be also analytically determined solely based on cells structure and its neighborhood. Therefore, the theory of probabilistic exponent of growth in Chap. 7 represents a step ahead towards the “holly grail” of emergence, i.e. predicting the behavior without a need to simulate the system. Information theory and the view of cells as nonlinear functions plays essential role in structuring this theory. Several examples of applying this theory, and comparisons with the experimental counterpart of the same measure confirm its validity and predictive power. It is particularly worth applicable for huge families of cells where the time to determine the measures experimentally for the entire family o cells is prohibitive. Instead, the hints given by the theory of probabilistic exponent of grow allows a clever design instead, where the result is simply a set of inequalities with variables drawn from the bits defining the cells gene. The last chapter provides three innovative applications of cellular computing systems in problems such as pattern recognition and lossy image compression to exemplify the potential of the design for emergence tools described in this book.
2 Cellular Nonlinear Networks: State of the Art and Applications
2.1 Introduction Solving some of the open problems using the principles of natural computing exposed in the previous chapter led to the idea of developing a computing paradigm called cellular computing. The structure of such a computing system is defined by a grid (often two-dimensional) of locally interconnected cells. Each cell may be in a number of states (ranging from 2 to infinity) and the state of a cell depends by its own previous state and the previous states of its neighbors through a nonlinear functional, which may be defined in different ways. This functional is associated with a practical implementation of the cell and includes a set of tunable parameters grouped as a gene vector [7]. By tuning the gene parameters one can achieve programmability, i.e. different emergent behaviors within the same basic cellular architecture. The cell assumes an initial state and may have one or more external inputs. In a cellular system, computation can be considered any form of meaningful emergent global phenomenon resulted from a proper design of the cell. Usually the initial state and the inputs code the problem to be solved while the answer to this problem is coded in an equilibrium state. For instance, in the character segmentation example provided in Chap. 8, the initial state contains a visual field with black/white pixels (e.g. handwritten figures) while the steady state result of the emergent dynamics is a collection of rectangles, each enclosing a compact handwritten character. More complicated dynamics (e.g. oscillatory or chaotic) can also encode a solution to the problem posed as initial state. This is the case when cellular systems are used to generate pseudo-random sequences, a widely known application of the cellular systems. The first cellular computers were theoretical constructs introduced by Stanislaw , M. Ulam in the 1950 s [18]. He then suggested John von Neumann to use what he called “cellular spaces” to build his self-reproductive machine [19]. Konrad Zuse (who built the first programmable computers between 1935–1941) was the first to suggest that the entire universe is being computed on a computer, possibly a cellular automaton (CA) [16]. Many years later similar ideas were also published by Edward Fredkin [20,21] and recently (2002) by Stephen Wolfram [17]. In 1982 Fredkin and Toffoli published a paper [22] where cellular computation based on R. Dogaru: Cellular Nonlinear Networks: State of the Art and Applications, Studies in Computational Intelligence (SCI) 95, 7–13 (2008) www.springerlink.com © Springer-Verlag Berlin Heidelberg 2008
8
2 Cellular Nonlinear Networks: State of the Art and Applications
conservative logic was proposed as a nondissipative cellular system. Their approach is interesting while we recall that in general any natural computing system is regarded as a dissipative system. Initially cellular automata (CA) were developed to explain various natural phenomena. Choosing the proper genes in the form of local rules defining the behavior of each cell was the equivalent of programming in serial computers. The well known “Game of Life” rule introduced by Conway in the 1970s [23] gained popularity due to the complex and diverse patterns emerging in such a simply defined system. Designing a proper set of local rules was then a matter of intuition and educated guess rather than the outcome of a well-defined procedure. It was proved that such a simple machine (this is a 2 state per cell cellular automata with a very simple local rule) is capable of universal computation (i.e. it is a universal Turing machine [24]). Following the line of von Neumann, a lot of research has been devoted to the study of cellular automata and local rules leading to emergent properties such as self-reproduction and artificial life. An overview of these nonconventional computers can be found in [25]. Simulation software for a wide palette of cellular paradigms including von Neumann’s self-reproducing machine can be found at [26]. Recently, starting with the work of Chua and Yang [27] a novel cellular computing paradigm called cellular neural network was developed. It inherits the basic ideas of cellular computing and in addition bore some interesting ideas from the field of neural computation. While most of the previously described cellular computing paradigms were conceptual, the CNN was from the beginning circuit oriented, i.e. intended for practical applications as an integrated circuit. Moreover, in 1993 Roska and Chua [28], proposed a revolutionary framework called a CNN Universal Machine, in fact a specialized programmable cellular computer which is capable to execute complex image processing tasks and which found numerous applications in vision, robotics and remote sensing [29]. Nowadays this paradigm is successfully exploited in various applications dealing mainly with extremely fast nonlinear signal processing and intelligent sensors. In [7] it is demonstrated that the CNN paradigm includes cellular automata as a special case. Therefore many of the research in the area of cellular automata can be easily mapped into the CNN formalism with the advantage of exploiting a range of powerful chip implementations that have been developed over the years [30,31]. To date several types of emergent computation were identified as meaningful and useful either for computing applications (e.g. in the area of vision and image processing) or for modeling purposed (e.g. models of the cell membrane) by what I would generically call “evolutionary strategies”. An interesting example is the development of a relatively large library of CNN genes (templates) over the last decade [32]. Many of these genes were discovered by chance, studying the dynamic evolution of the CNN and identifying certain dynamic behaviors with meaningful computational primitives such as edge or corner detection, hole filling, motion detection, and so on. Although some theoretical approaches, mainly inspired from the techniques of filter design were successfully employed to design new chromosomes, there is still much to do for a systematic design of the cells and genes. This book offers several novel approaches and solutions to this problem.
2.2 Typical Applications of Cellular Computers
9
2.2 Typical Applications of Cellular Computers A search done several years ago on the IEEE Xplore database reveals the following distribution of applicative areas for cellular automata and cellular neural network architectures (Fig. 2.1).
Fig. 2.1. Paper distribution on various applicative subjects; Based on IEEE publications between 1988 and 2003. The number within each category is the number of papers found to deal specifically with a certain item (e.g. VLSI) while having the words “cellular automata” or “cellular neural network” also in the article title
The above figure reveals that most of the applications of cellular computers are related to VLSI implementations. They are either digital implementations (custom or reconfigurable) or mixed-signal. A notable example from the mixed-signal category is the CNN-UM (CNN universal machine) mentioned before. The massive parallelism of the cellular computing architecture is the more appealing feature for a VLSI implementation. The result is a fast signal-processing engine, outperforming a conventional signal processor several orders of magnitudes. This is particularly effective for multi-dimensional signal processing, i.e. image processing. Another popular application of cellular computers is that of pattern recognition. Several papers proposed so far the use of cellular computers as classifiers, working on a different principle than classic feed-forward neural networks. Here the classes are associated with a finite number of attractors and the initial state with the pattern to be recognized. Convergence towards an attractor or another indicates the membership of the initial state pattern to a certain category. While
10
2 Cellular Nonlinear Networks: State of the Art and Applications
cellular computers may have hundreds of thousands of cells they can process large databases such as images or other multi-dimensional signals. For instance cellular computing systems can extract edges, corners and other features of interest from an image regardless the size of that image. Besides pattern recognition, cellular systems are capable of various linear and nonlinear filtering tasks. Ciphering is another popular application of cellular computing. Indeed several patents have been filled for such applications where cells are designed such that a complex, chaotic dynamics, emerge in the array of cells. Unlike other methods for random number generation (e.g. the linear feedback shift registers) the CA-based method is scalable, i.e. one can add more cells without changing the essence of the dynamics behavior. By employing a larger number of cells, the probability of deciphering decreases making such systems extremely reliable in terms of security. In a recent paper [33] the use of cellular automata as ciphering systems is carefully investigated and several benchmarks are computed showing that highly reliable random number generators can be obtained using relatively simple cells (e.g. four input Boolean cells) arranged in one-dimensional arrays of several hundreds of cells. The pseudo-random and complex dynamics of cellular systems is also exploited for built in self test (BIST) systems. Such systems are required to perform functional analysis of complex circuits and detect functional failures. In doing so, a convenient solution is to embed a cellular system acting as a pattern generator. The CA is designed such that a large sequence of patterns is generated as a result of the CA dynamics. The length of the sequence is optimized as a tradeoff between a reasonable testing time (demanding thus not a very long sequence) and enough information in the sequence to detect certain failures. In the parlance of emergence ideal BIST sequence generators are operated in the “edge of chaos” regime, i.e. they are neither random signal generators with very long cycles neither orderly systems with very small length limit cycles. Signal compression is another interesting application of cellular automata. Several solutions were reported so far. For instance [34] proposes a solution called a “CA transform” where a signal (image or sound) is decomposed as a binary weighted sum of basis signals. The basis signals are generated by properly tuned cellular automata with certain genes (cell parameters). In order to reconstruct the signal one needs only the set of binary weights and the (relatively) short description of the CA cells generating the basis signals. In order to demonstrate the idea of image compression using cellular automata we proposed recently a novel approach where generalized cellular automata (GCA) can be used. Using the following simple Matlab programs, a wide palette of images can be obtained (one pixel corresponds to one cell in the image), part of which are depicted in Fig. 2.2. Each image in Fig. 2.2 displays above the set of 17 generating parameters. Only a few parameters (17 parameters, underlined above) control the diversity of the obtained images. Such images or part of them can be combined to approximate
2.2 Typical Applications of Cellular Computers
11
% 1 function implementing the cell. It also contains the gene function y=gca_u_cell(u1,u2,u3,u4,u5,u6,u7,u8,u9) Z=[0.9, 0.3, 0.3, 2, 0.3, .6, 0.5]; B=0.5*[1.1 1 1 1 0 1 1 1 1]; sigm=B(1)*u1+B(2)*u2+B(3)*u3+B(4)*u4+B(5)*u5+B(6)*u6+B(7)*u7+B(8)*u8+B(9)*u9; w=Z(1)+Z(2)*u5+Z(3)*sigm-abs(Z(4)+Z(5)*u5+Z(6)*sigm)+Z(7)*abs(sigm); y=w;
% 2 main program running the GCA for a given number of steps function y=gca_u(steps) % e.g. steps=100 (runs the GCA for 100 iterations) x0=-ones(199,199); x0(100,100)=1; [m n]=size(x0); i=1:m; j=1:n; left_j=[n,1:n-1]; right_j=[2:n,1]; up_i=[m,1:m-1]; low_i=[2:m,1]; y=x0; for s=1:steps u9=y(up_i,left_j); u8=y(up_i,j); u7=y(up_i,right_j); u6=y(i,left_j); u5=y; u4=y(i,right_j); u3=y(low_i,left_j); u2=y(low_i,j); u1=y(low_i,right_j); y=gca_u_cell(u1,u2,u3,u4,u5,u6,u7,u8,u9); end set(1,‘Position‘,[291 180 505 540]); image(20*y+32); axis image; colormap gray
any part of a real image. The extreme case is when one of the resulting images has to be compressed. Then since it is the result of running the above programs (acting as decompressing engines) an extremely high compression rate is achieved. Assuming that each of the 17 parameters as well as the pixels from the image are represented with 8 bits, for a 200 × 200 pixels image a compression rate of 200 × 200 can be obtained. This is a rate far beyond any of the actual com2353 17
pression scheme. A challenging task remains to find families of cellular systems capable to approximate quite well larger blocks from real natural images. The larger the blocks the higher the compression rate, which in the limits can reach values as high as thousands or tens of thousands. The advantage of a very simple decompression scheme (the above programs) shall be exploited in making compressed documents such as journals, books or compact encyclopedias.
12
2 Cellular Nonlinear Networks: State of the Art and Applications
Fig. 2.2. Several different images obtained with the same “decompression” software starting from different sets of cell parameters. The compression rate achieved is of the order of thousands since each image is defined by only 17 parameters (depicted above)
2.3 Hardware Platforms for Implementing Cellular Computers Cellular computers are particularly attractive for hardware implementation because is only in hardware where their full computational power come to a life. Moreover, since they are interconnected arrays of regular cells they are simple to implement starting from the design of cell, which is multiplied as needed. The designer of a cellular system focuses on cell-design while interconnection often
2.3 Hardware Platforms for Implementing Cellular Computers 13
comes naturally by arranging the cells given the technology constraint. Often the arrays are two-dimensional with each cell connected to the immediate neighbors. In digital technologies cellular computers can be realized either as dedicated VLSI products but in this case they are less flexible and have large development costs or better using the FPGA (field-programmable-gate-array) technology which offers a high degree of flexibility and programmability as a results of allowing random interconnection between a number of identical yet reconfigurable digital cells. Several FPGA implementations of cellular systems were reported so far [35], including a CNN prototyping board. Digital technologies have the disadvantage of a low density of cells. This is why an alternative solution is the use of mixed-signal cells, where compact cells based on nonlinear computation in analog devices are used. The CNN paradigm was from the beginning oriented to such a technology. So far a series of chips (today called visual microprocessors or imagers) was developed. Among the latest implementation solution of a CNN universal machine is the ACE16k chip [36] which has optical input and can implement the standard CNN model. It has a number of about 16,000 cells, each cell being associated with an image pixel. Certain high-speed processing tasks were demonstrated such as the task of classifying objects (medical pills) presented to the optical input with a speed of 20,000 objects per second. Several other mixed-signal cell architectures were proposed, for instance the architecture described in [37], which is based on a nested (recurrent) utilization of a nonmonotone nonlinear function to reduce the complexity and implement arbitrary Boolean functions. Arbitrary Boolean functions with n inputs can be implemented with a linear complexity in n. A schematic of this function is presented in Chap. 3. A novel implementation medium, although yet experimental, is represented by the use of nano-technologies [38]. Browsing the literature of the last 2 years, we found an increased interest in the “quantum dot” cellular computing paradigm [39], abbreviated QCA (from quantum cellular automata). Recently [40] the paradigm evolved into molecular QCA. More details and java simulations of QCA systems can be found at http://www.nd.edu/~qcahome/. A quantum dot is a nanometer scale active cell. Quantum dots are interconnected by proximity so there is no need for additional metal layers as in standard electronic technologies. A state change in a QCA cell propagates to its neighbors and this is exactly the basis for cellular computation. When arranged properly such cells are capable to do various computational tasks.
3 Cellular and Natural Computing Models and Software Simulation
3.1 Cellular Systems: Cells, Neighborhoods, States and Dynamics The main features of cellular systems are regularity and homogeneity. In fact a cellular system can be defined as a structured collection of identical elements called cells. The structure of a cellular computing system is given by the choice of a lattice. Such lattices are one-dimensional, two-dimensional and, less used, three or more dimensional. The following are examples of commonly used twodimensional lattices:
Fig. 3.1. Cells, lattices, neighborhoods and indexes
In the above pictures, the cells in the neighborhood are depicted in lighter gray shade, with the exception of the central cell. The neighborhood represents the set .RDogaru: Cellular and Natural Computing Models and Software Simulation, Studies in Computational Intelligence (SCI) 95, 15 –64 (2008 ) www.springerlink.com © Springer-V erlag eBrlin eHidelberg 2008
16
3 Cellular and Natural Computing Models and Software Simulation
of cells that are directly interacting with the central cell. The neighborhood is represented by the set N and a unique index k is chosen to identify the neighboring cell in a particular neighborhood. The assignment of different integer values to k is conventional and some examples are provided in Fig. 3.1 where indexes are depicted in white for each type of neighborhood. The cells are in fact nonlinear dynamic systems. The cell dynamics can be continuous in time and mathematically described by ordinary differential equations ODEs or discrete in time and then their dynamics is described by difference equations. Example 1:
y1 (t )
¦ ak yk t 1 defines the dynamics of the output y1 t of a
k N
discrete time autonomous cell defined by a weighted summation of all neighboring cell outputs at the previous time clock. The discrete time variable t is here an integer. The above example represents one of the simplest possible cells. It is autonomous since there is no external input to drive the dynamics of the cell. In the general case a cell is described by the following variables: Inputs – usually denoted by variable u (scalar) or u (vector of inputs); States – usually denoted by variable x (scalar) or x (vector of states); Initial states – a particular state variable at the initial moment, t=0. Outputs – usually denoted by variable y (scalar) or y (vector of outputs). In terms of a computing machine, inputs and initial states are used to supply the cellular system with the input data (to be processed) while the result or output data is available in the form of a pattern (spatial, temporal or spatio-temporal) of the outputs. For simpler cells sometimes outputs and states may coincide, otherwise the outputs represent a nonlinear function applied to states. Example 2: The “standard cellular neural network” cell The ODE defining the standard cellular neural network [27] is: x1
where yk
1 2
x1 ¦ ak yk ¦ bk uk z k N
(3.1)
k N
xk 1 xk 1 ,
and the initial states xk 0 are specified for all cells in the neighborhood N. Observe that this is a more complex, continuous time, dynamic system with a nonlinearity induced by the relationship between states and outputs. In general (nonstandard CNN cell) (3.1) is replaced with: x1 x1 f G , yk , uk , where f is a nonlinear function and G is a gene as defined next.
(3.1’)
3.1 Cellular Systems: Cells, Neighborhoods, States and Dynamics
17
3.1.1 Genes The cell dynamics, and its local functionality are prescribed by an unique gene [7] vector G >a1, a2 ,..an1,.., b1, b2 ,..bn 2 ,...@ containing all parameters of the nonlinear system modeling the cell. The analogy with biology is obvious, and the gene vector may be regarded as the equivalent of the information contained in the DNA string. 3.1.2 Discrete and Continuous States and Outputs In defining a cellular system one has to prescribe the variation domain of the state and output variables. For example, one can use continuous state cells where the outputs are defined within a bounded interval or one can also use discrete state cells where the states or/and the outputs belong to a finite set of possible values. A binary output cell implements Boolean functions, i.e. it provides a logical “TRUE” or “FALSE” output for each of the 2n possible combinations of inputs. Each input also represents a truth value. The cell can be again specified as a discrete-time dynamical system but it can be also specified using a transition table or a set of local rules. The last two modes of specifying a cell are specific to the cellular automata formalism [41]. As we will show in the next chapter, a compact piecewise linear description (i.e. a discrete dynamical system) can be found for any Boolean or other type of input function: Example 3: A Boolean cell y1 (t )
§ · sign¨¨ z0 ¦ bk uk t ¸¸ kN © ¹
(3.2)
Assuming that the set ^1,1` is used to code the truth set ^FALSE, TRUE` , the cell equation (3.2) above defines a family of Boolean functions. For a particular Boolean function one should specify the gene parameters. For example, the AND function with three inputs is associated with the following gene: G >z0 , b1, b2 , b3 @ > 2,1,1,1@ . If the gene is replaced with G >z0 , b1, b2 , b3 @ >2,1,1,1@ , the OR function with three inputs is implemented. 3.1.3 Boundary Conditions In cellular automata the cells located on the boundary of a lattice have a special in interconnectivity. One should define precisely the way in which these cells interact with their neighbors. This is called a boundary condition. The choice of a particular boundary condition may have a great influence on the dynamics of the cellular system. The following two are among the mostly used boundary conditions: Periodic boundary conditions: Opposite borders of the lattice are connected. A one-dimensional “line” becomes following that way a circle, a two-dimensional lattice becomes a torus.
18
3 Cellular and Natural Computing Models and Software Simulation
Reflective boundary conditions: The border cells are mirrored, i.e. the consequence is having symmetric border properties. In the remainder of this chapter exclusively square lattices and periodic boundary conditions are used.
3.2 Major Cellular Systems Paradigms
3.2.1 The Cellular Neural Network (CNN) Model The cellular neural network (CNN) model was proposed by Chua [27] as a practical circuit alternative to oHpfield and ot her type of recurrent networks. The CNN cell is a continuous time and continuous state dynamical system with some saturated nonlinearity (see (3.1)) which is well suited for implementation using analog circuits. nUlike CA s which are mostly used to prove various theories or to model physical processes the CNN was intended from the beginning to be also a useful signal processing paradigm. A n important step towards making this paradigm an application oriented one was the introduction in 19 93 of the concept of CNN universal machine (CNNU M ) [28]. W ithin the framework of the CNN-U M , a CNN kernel is employed to perform different parallel information processing tasks. Each task is associated with a gene (i.e. a set of parameters of the synaptic interconnections) which may be selected from a continuously growing library of more than 200 different primitive genes (and tasks). Therefore one may combine various such primitives, which are stored in an analog memory much like the instruction-code memory of digital microprocessor. V arious high-level comp utations emerge, making the CNN-U M implementation a suitable medium for computing at TerraOps computing speed. R ecent electronic implementations of the CNN-U M are in fact sensor computers [42], having the capability to sense and to process an image on the same chip. Several generations of microelectronic chips were reported so far [31], as well as development tools, allowing users to program the CNN as visual microprocessor. There is a wide range of applications, mostly in the area of image processing. Such applications include image segmentation, image compression, fast halftoning, contour tracking, image fusion, pattern recognition, to name uj st a few. A lthough initially the equilibrium dynamics of CNNs was mostly exploited for applications, recently the nonequilibrium dynamics is employed for certain interesting applications in what is currently called “computing with waves” [43,44]. There is an increasing research interest around the world for the field of cellular neural networks, most of which is reported in the rPoceedings of the IEEE CNNA workshops (cellular neural networks and their applications), ISCA S or IJCNN
3.2 Major Cellular Systems Paradigms
19
conferences as well as in numerous journal papers or books. Some recent tutorials about CNNs are [7,29,31,45]. Several CNN simulators are freely available and the reader may check [4 6], and [47] for an easy to use, computing platform independent CNN simulator. Awide range of simulators as well as news about the progress in the CNN research can be accessed from [4 8]. The SCNN simulator from the pAplied hPysics Department of the oGethe nUiversity in Frankfurt am M ain can be used on U nix platforms [49]. 3.2.2 The Generalized Cellular Automata In [7], the idea of a generalized cellular automata (G CA ) was introduced as an extension of the CNN so that a C GAinclude s CA s as a special case. The main idea of the C GAis to use a CNN in a discrete-time loop. In other words, for each period of the clock the CNN system evolves until it eventually reaches a steady output (for some CNN genes it is possible to have complex oscillatory dynamics) starting from a given initial condition and from inputs variables that are copies of the C GAneighboring cells outputs at the end of the previous time cycle. The additional CNN loop is thus given by the discrete time equation uk (t ) yk (t 1) . There are two cases of interest: x W hen the CNN cell is uncoupled (i.e. all coefficients ak 0 for k t 2 in (1)) it was proved that the nonlinear dynamic system (3.1’) converges towards a steady state output solution and therefore aft enough period of time T the cell can be described as a simple nonlinear equation of the following form: y1 t F G , uk t where F is a nonlinear function and G is a gene (i.e. tunable parameters). In this case the C GAcan emulate any CA , provided that there exist a method to map the local rules or transition table into the nonlinear function F. A s it can be easily observed such G CAcan also implement discrete-time but continuous state cellular automata. x W hen the CNN cell is coupled, one should first determine a set of proper genes such that the same steady state behavior occurs during the clock time T. This is often not a trivial task. Then, the behavior of the resulting C GAis more complex than that of a normal CA . The reason is that an emergent computation already takes place during the period of time T in the CNN, therefore at the discrete time moment when the output of a cell is sampled, it does not represent only the contribution of the neighboring cells as in the case of a “classic” cellular automata but rather the contribution of all CNN cells. In some sense one can say that employing a continuous time CNN during the clock time is equivalent to artificially increasing the neighborhood to the whole cellular space. This behavior may imply interesting computational consequences.
20
3 Cellular and Natural Computing Models and Software Simulation
3.2.3 Reaction-Diffusion Cellular Nonlinear Networks R eaction-diffusion CNNs (R D-CNNs) [8] were proposed as a particular case of continuous-time autonomous1 CNN and are discrete-space models of partial differential equations describing the reaction-diffusion physical processes. From a circuit perspective a D R -CNN can be modeled as a collection of multiport nonlinear cells. These cells are coupled with their neighboring cells via linear resistive networks (Fig. 3.2).
Fig. 3.2. The topology of a reaction diffusion cellular neural network. Acell is a m-port described by a nonlinear ODE which models a physical reaction. The coupling with neighboring ing cells is done via resistive grids modeling the physical process of diffusion
Example :4 The case of a reactiondiffusion cell with von Neumann neighborhood. The cell equation is of the following form: x&11
.. x&1j
..
x&1m 1
f1 ( x11, x12 ,..., x1m , G1 ) D1 x12 x13 x14 x15 4 x11 ,
f2(x11, x12 ,..., x1m , G j ) D j x2 j x3 j x4j x5j 4 x1j ,
fm( x11 , x12 ,..., x1m , G m ) Dm x2m x3m x4m x5 m 41x m ,
nA autonomous system has no external inputs. Information processing in such systems consists in prescribing the initial state of all cells with some pattern to be processed followed by a dynamic transformation of this pattern until a stopping criterion is met.
3.3 Matlab Simulation of Generalized Cellular uAtomata and Cellular Neural Network
21
where xkj is the state variable of cell k (using the neighborhood index as in Fig. 3.1 with the central cell indexed as k=1) in layer j. The gene is also distributed among layers as G 1 ,...G m and in addition each layer is characterized by the scalar diffusion coefficient D j corresponding physically to the conductance of the resistors in the resistive grid in Fig. 3.2. R D-CNNs have the interesting property that emergent behaviors can be predicted by carrying out a local activity analysis of the isolated cell (ignoring the coupling). Adetailed analysis based on the local activity [7,8] reveals regions of emergence in the cell parameter space [9 –11,15 ] called an “edge of chaos”. So far, an analytic method to draw the boundaries of the “edge of chaos” regions in the cell parameter space exists only for reaction–sifussion cellular systems. Though, within this chapter certain algorithmic methods for detecting emergence regions within the cell parameter space of arbitrary cellular systems are given.
3.3 Matlab Simulation of Generalized Cellular Automata and Cellular Neural Networks It is convenient to simulate various types of cellular system using the programming environment aMtlab (http:/w / ww.math works.com/products/matlab/) produced by aMthworks. eWwill assume here the reader has some basic knowledge of aMtlab. For readers unfamiliar with this language, a good starting point might be the primers [50] or [51]. 3.3.1 Uncoupled GCAs In order to simulate an uncoupled C GAone should first write the function G CA _U CEL L .M which implements the G CAcell. It is in fact a simple description of the nonlinear function y1 t F G , uk t :
function y=gca_u_cell(u1,u2,u3,u4,u5,u6,u7,u8,u9) Z=[-1,-2,-4,-8,-7]; B=[1 1 1 1 1 1 1 1 1]; s=-1; sigm=B(1)*u1+B(2)*u2+B(3)*u3+B(4)*u4+B(5)*u5+B(6)*u6+B(7)*u7+B(8)*u8+B(9)*u9; w=Z(1)+abs(Z(2)+abs(Z(3)+abs(Z(4)+abs(Z(5)+sigm)))); y=s*sign(w); %y=0.5*(abs(w+1)-abs(w-1)); %y=w;
The nine inputs of this function can be a scalar, a vector or an array. The above program implements the aPrity9local logic function. oHwever, by simply changing
22 3 Cellular and Natural Computing Models and Software Simulation
the gene parameters (in our case the values of the ,Z,Bs) many other local oBolean functions can be implemented. In [5 2] an d [15] a detailed procedure is given to represent any arbitrary oBolean function us ing a piecewise-linear representation as above. M oreover, one can simply change the cell structure and use any other nonlinear function. For example, as seen in the last lines, while a sign function is employed, the result is a C GAemulatin g a binary cellular automaton. One may also use continuous valued functions with or without saturation, resulting in a G CAwith continuous states (the last lines of the code). L et us now have a look at the code C GA _U .M to implement the generalized cellular automata. It is implemented in the form of a M atlab function with two input parameters. The first is a scalar and represents the number of iteration steps until stop (a faster stop can be achieved by pressing the keys CTR Land C). The second may be absent but if not it represents the name (a string) of a file containing the initial state. The initial state gives also information about the size (number of cells) of the C GA . Aperiodic boundary condition is implemented, the lattice is square with a M oore neighborhood indexed as explained in the source code: function gca_u(steps,init_cond) % e.g. steps=100 (runs the GCA for 100 iterations) % e.g. init_cond='cross199' % The inital state should be previously saved as matrix x0. For example: % x0=ones(199,199); x0(90:110,90:110)=1 % save one199 x0 % The neighborhood index is chosen as follows: % 9 8 7 % 6 5 4 % 3 2 1 if narginG1, T1 , G1, T2 ,...Gm , Tm @ will define an overall processing function, i.e. a “program”. Note that unlike in the case of traditional computing systems where programs are defined by algorithms, here we need to search for those emergent genes G1,..Gm and associate the underlying processing with certain useful information processing functions. The above is the subject of “designing for
^
`
26 3 Cellular and Natural Computing Models and Software Simulation
emergence”, the topic of this book. Note also that since CAgenes are usually described by a small number of bits, the CA“program length” is usually quite small, at least compared to traditional programs. Though, even for a “small” program length of 152 bits, the number of possible programs is huge and therefore adequate tools and methods have to be defined for an effective way to locate those programs. In one discrete time step a synchronous update of all cells takes place, based on the status of their neighborhoods at the previous time step. For a semitotalistic CA , the update depends only on the values of D 2y t 3y t y4 t y5 t and Bolean function (gene) can be defined usE y1 t . Therefore the transition local o ing Table 3.1 where the gene is a 10-dimensional binary vector G >g9 , g8 ,..., g 0 @ : Table 3.1. Acell definition using a table
G D E
g9 4 1
g7
g8
3
2
1
g6
1
1
g5
0
1
1
g3
g4
4
3
0
g2
2
0
g1
1
0
g0
0
0
0
The state of the neighborhood is entirely described by two integers: D , E W hen each of the binary components g j of the gene G are defined, the above table precisely indicates a local rule, i.e. how to update the state (output) of the central cell for any possible configurations of the neighborhood given by D , E y1 t 1 G D t , E t
(3.3)
The gene G >g9 , g8 ,..., g 0 @ is a binary string. In practice is often more convenient to use its decimal representation, denoted ID (cell identifier). For a given constraint of the cellular model (e.g. specifying that semitotalistic cells with nine inputs are used, or the label of a taxonomy such as “2s9” – see Sect. 3.7) an ID clearly individualizes a cell within a fami ly composed of all possible cells given the constraints. 3.4.1 Piecewise-Linear Representation and Implementation The function in (3.3) can be implemented directly using a look-up table (a 1,024 bits R A M or R OM ) or via compact mixedsignal cells [37]. This later method is the most convenient in terms of density of cells. In order to define a circuit as in [37], but also for other reasons, a nonlinear representation of (3.3) is more convenient that a table. For instance, as seen later, the use of such a nonlinear description in aMtlab allows the use of the intrinsic matrix operations, therefore leading to an important speedup compared to the case when tables would be used. In order to understand the principles of nonlinear representations [5 2] let us consider the case of implementing the function ID= 103 (or 0001100111 binary).
3.4 Nonlinear Representation of Cells
27
L et introduce a unique variable V describing the excitation of the neighborhood (it should encode all possibilities for the collective states of the neighborhood), and note than in the case of our semitotalistic cell it is described by V 5E D . In what follows we will also call V a projection because it represents a projection from the n-dimensional input space to a one-dimensional (scalar) space while preserving all information contained in the n-dimensional vector defining the collective state of the neighborhood. In general, for any semitotalistic cell with n inputs the following formula stands: (3.4)
bE D ,
V
where D , E are defined as above, and b (called an orientation [52]) is an integer b ^0,1,..n` . Implicitly b n but one can optimize its value (minimize b) such that for the same number n of inputs the gene will become smaller than n+1 (needs less bits). Adiscussion on the significance of b follows later within this section. Now Table 3.1 can be rewritten as Table 3.2. eHre we used the convention w=–1 to denote an output equal to logic “0”, and we also specified the transition points, i.e. those values of V such that the w function switches from –1 to + 1 or viceversa. Note that any nonlinear function with the same roots as the transition points can be selected as a nonlinear representation of the cell. In particular the simplest solution is a polynomial function, i.e. w
s V t1
(3.5)
V t 2 ... V t m .
Table 3.2. Acell definition based on the function al dependence between the projection V and the output y (H ere exemplified for ID= 103) G= y( V ) w
0
0
0
–1
–1
–1
0
1 1
1 1
5E D
9
8
7
t 3 6. 5
6
5
V
0 t 2 4. 5
0 –1
0 –1
0
1 1
1 1
1 1
4
3
t1 2. 5
2
1
0
Note than any scalar function which passes through the transition points marked within the table can implement the table.
The output value is defined by y sgnw 1 /2.
(3.6)
In (5) the s (sign) variable is computed as: s 2 g01, (3.7) i.e. s 1 for even ID and s 1 for odd ID. Note than in the above are possible at most m N 1 transition points where N is the number of bits defining the gene G.
28 3 Cellular and Natural Computing Models and Software Simulation
Instead of (3.5) a piecewise linear (P W L ) canonical formula could be used for more convenient analysis and hardware implementation [15 ]. For a set of m transition points t1, t2 ,.., tm the canonical P W Lrepresentation corresponds to:
>
@
m1
w z 0. 5 1 m 1 sV s ¦ 1 k V zk
with
(3.8)
k 1
zk 0. 5tk tk 1 , and
z
>
@
m 1
1 m 1 st1 s ¦ 1 k t1 zk k 1
Examined to a closer look, either formulae (3.5 ) or (3.8) are fully defined if for any given ID (string of N bits) one knows the following set of parameters: b, t1,...tm . U sing a simple method an optimal 2 implementation via (3.8 ) for each of the 1,024 possible genes can be determined. In order to determine a realization with a minimal number of m (i.e. minimum number of transitions), successive integer values for b are considered (until b becomes equal to n) and one keeps only the value of b giving the minimum number of transitions. The successive values t1 ,...t m are then recorded. For the case of all 1,024semitotalistic func tions with five inputs the next tables give the corresponding realizations ( b, m ). The sign value s can be simply computed with (3.7) and therefore is not given in the table. lAso the exact transition points t1 ,...t m are missing from the table since they can be easily computed, as explained above (Table 3.3). Table 3.3. Alist of all semitotalistic genes indicating their optimal realization. In each row, the numbers indicate, ID, b, and m (the number m of transitions, also indicating the complexity of the cell. Those cells with b= 0 are totalistic (i.e. their output depends exclusively by the sum of the neighbor cells) 0
0
0
14
4
2
28
5
2
42
3
4
1
1
1
15
4
1
29
5
3
43
3
3
2
2
2
16
5
2
30
5
2
44
3
2
3
2
1
17
5
3
31
5
1
45
3
3
4
3
2
18
5
4
32
5
2
46
3
2
5
3
3
19
5
3
33
0
1
47
3
1
6
3
2
20
5
4
34
1
2
48
4
2
7
3
1
21
5
5
35
1
1
49
4
3
8
4
2
22
5
4
36
2
2
50
4
4 3
9
4
3
23
5
3
37
2
3
51
4
10
4
4
24
5
2
38
2
2
52
4
4
11
4
3
25
5
3
39
2
1
53
4
5
12
4
2
26
5
4
40
3
2
54
4
4
13
4
3
27
5
3
41
3
3
55
4
3
2
In the sense of minimizing the number of transition points.
3.4 Nonlinear Representation of Cells
29
56
4
2
96
5
2
136
1
2
176
4
57
4
3
97
5
3
137
1
3
177
4
5
58
4
4
98
5
4
138
4
6
178
4
6
59
4
3
99
0
1
139
4
5
179
4
5
60
4
2
100
5
4
140
4
4
180
2
4
61
4
3
101
5
5
141
4
5
181
2
5
62
4
2
102
1
2
142
4
4
182
2
4
63
4
1
103
1
1
143
4
3
183
2
3
64
4
2
104
5
4
144
2
2
184
4
4
65
4
3
105
5
5
145
2
3
185
4
5
66
0
2
106
5
6
146
2
4
186
4
6
67
4
3
107
5
5
147
2
3
187
4
5
68
1
2
108
2
2
148
5
6
188
4
4
69
1
3
109
2
3
149
5
7
189
4
5
70
4
4
110
2
2
150
5
6
190
4
4
71
4
3
111
2
1
151
5
5
191
4
3
72
2
2
112
4
2
152
5
4
192
4
2
73
2
3
113
4
3
153
5
5
193
4
3
74
2
4
114
4
4
154
5
6
194
4
4
75
2
3
115
4
3
155
5
5
195
4
3
76
4
4
116
4
4
156
5
4
196
4
4
77
4
5
117
4
5
157
5
5
197
4
5
78
4
4
118
4
4
158
5
4
198
0
2
79
4
3
119
4
3
159
5
3
199
4
3
80
3
2
120
3
2
160
5
4
200
4
4
81
3
3
121
3
3
161
5
5
201
4
5
82
3
4
122
3
4
162
5
6
202
4
6
83
3
3
123
3
3
163
5
5
203
4
5
84
3
4
124
3
2
164
5
6
204
1
2
85
3
5
125
3
3
165
0
3
205
1
3
86
3
4
126
3
2
166
5
6
206
4
4
87
3
3
127
3
1
167
5
5
207
4
3
88
5
4
128
3
2
168
3
4
208
3
2
89
5
5
129
3
3
169
3
5
209
3
3
90
5
6
130
3
4
170
1
4
210
3
4
91
5
5
131
3
3
171
1
3
211
3
3
92
5
4
132
0
2
172
3
4
212
3
4
93
5
5
133
3
5
173
3
5
213
3
5
94
5
4
134
3
4
174
3
4
214
3
4
95
5
3
135
3
3
175
3
3
215
3
3
4
(Continued)
30
3 Cellular and Natural Computing Models and Software Simulation
216
2
2
256
2
2
296
3
4
336
3
4
217
2
3
257
2
3
297
0
3
337
3
5
218
2
4
258
2
4
298
3
6
338
3
6
219
2
3
259
2
3
299
3
5
339
3
5
220
5
4
260
3
4
300
3
4
340
1
4
221
5
5
261
3
5
301
3
5
341
1
5
222
5
4
262
3
4
302
3
4
342
3
6
223
5
3
263
3
3
303
3
3
343
3
5
224
5
2
264
0
2
304
4
4
344
5
6
225
5
3
265
4
5
305
4
5
345
5
7
226
5
4
266
4
6
306
1
4
346
5
8
227
5
3
267
4
5
307
1
3
347
5
7
228
5
4
268
4
4
308
4
6
348
5
6
229
5
5
269
4
5
309
4
7
349
5
7
230
5
4
270
4
4
310
4
6
350
5
6
231
0
1
271
4
3
311
4
5
351
5
5
232
5
4
272
1
2
312
4
4
352
5
4
233
5
5
273
1
3
313
4
5
353
5
5
234
5
6
274
5
6
314
4
6
354
5
6
235
5
5
275
5
5
315
4
5
355
5
5
236
5
4
276
5
6
316
4
4
356
5
6
237
5
5
277
5
7
317
4
5
357
5
7
238
1
2
278
5
6
318
4
4
358
5
6
239
1
1
279
5
5
319
4
3
359
5
5
240
4
2
280
5
4
320
4
4
360
5
6
241
4
3
281
5
5
321
4
5
361
5
7
242
4
4
282
5
6
322
4
6
362
5
8
243
4
3
283
5
5
323
4
5
363
0
3
244
4
4
284
5
4
324
4
6
364
2
4
245
4
5
285
5
5
325
4
7
365
2
5
246
4
4
286
5
4
326
4
6
366
2
4
247
4
3
287
5
3
327
4
5
367
2
3
248
3
2
288
5
4
328
2
4
368
4
4
249
3
3
289
5
5
329
2
5
369
4
5
250
3
4
290
5
6
330
0
4
370
4
6
251
3
3
291
5
5
331
2
5
371
4
5
252
2
2
292
2
4
332
4
6
372
4
6
253
2
3
293
2
5
333
4
7
373
4
7
254
2
2
294
2
4
334
4
6
374
1
4
255
2
1
295
2
3
335
4
5
375
1
3
3.4 Nonlinear Representation of Cells
31
376
3
4
416
5
4
456
4
4
496
4
2
377
3
5
417
5
5
457
4
5
497
4
3
378
3
6
418
5
6
458
4
6
498
4
4
379
3
5
419
5
5
459
4
5
499
4
3
380
3
4
420
5
6
460
4
4
500
4
4
381
3
5
421
5
7
461
4
5
501
4
5
382
3
4
422
5
6
462
0
2
502
4
4
383
3
3
423
5
5
463
4
3
503
4
3
384
3
2
424
3
4
464
3
2
504
3
2
385
3
3
425
3
5
465
3
3
505
3
3
386
3
4
426
3
6
466
3
4
506
3
4
387
3
3
427
3
5
467
3
3
507
3
3
388
3
4
428
3
4
468
3
4
508
2
2
389
3
5
429
0
3
469
3
5
509
2
3
390
3
4
430
3
4
470
3
4
510
1
2
391
3
3
431
3
3
471
3
3
511
1
1
392
4
4
432
4
4
472
2
2
512
1
1
393
4
5
433
4
5
473
2
3
513
1
2
394
4
6
434
4
6
474
2
4
514
2
3
395
4
5
435
4
5
475
2
3
515
2
2
396
0
2
436
2
4
476
1
2
516
3
3
397
4
5
437
2
5
477
1
3
517
3
4
398
4
4
438
2
4
478
5
4
518
3
3
399
4
3
439
2
3
479
5
3
519
3
2
400
2
2
440
4
4
480
5
2
520
4
3
401
2
3
441
4
5
481
5
3
521
4
4
402
2
4
442
1
4
482
5
4
522
4
5
403
2
3
443
1
3
483
5
3
523
4
4
404
5
6
444
4
4
484
5
4
524
4
3
405
5
7
445
4
5
485
5
5
525
4
4
406
5
6
446
4
4
486
5
4
526
4
3
407
5
5
447
4
3
487
5
3
527
4
2
408
1
2
448
4
2
488
5
4
528
0
1
409
1
3
449
4
3
489
5
5
529
5
4
410
5
6
450
4
4
490
5
6
530
5
5
411
5
5
451
4
3
491
5
5
531
5
4
412
5
4
452
4
4
492
5
4
532
5
5
413
5
5
453
4
5
493
5
5
533
5
6
414
5
4
454
4
4
494
5
4
534
5
5
415
5
3
455
4
3
495
0
1
535
5
4
(Continued)
32 3 Cellular and Natural Computing Models and Software Simulation
536
5
3
576
4
3
616
5
5
656
2
3
537
5
4
577
4
4
617
5
6
657
2
4
538
5
5
578
4
5
618
5
7
658
2
5
539
5
4
579
4
4
619
5
6
659
2
4
540
5
3
580
1
3
620
2
3
660
0
3
541
5
4
581
1
4
621
2
4
661
5
8
542
5
3
582
4
5
622
2
3
662
5
7
543
5
2
583
4
4
623
2
2
663
5
6
544
5
3
584
2
3
624
4
3
664
5
5
545
5
4
585
2
4
625
4
4
665
5
6
546
1
3
586
2
5
626
4
5
666
5
7
547
1
2
587
2
4
627
0
2
667
5
6
548
2
3
588
4
5
628
4
5
668
5
5
549
2
4
589
4
6
629
4
6
669
5
6
550
2
3
590
4
5
630
4
5
670
5
5
551
2
2
591
4
4
631
4
4
671
5
4
552
3
3
592
3
3
632
3
3
672
5
5
553
3
4
593
3
4
633
3
4
673
5
6
554
3
5
594
0
3
634
3
5
674
5
7
555
3
4
595
3
4
635
3
4
675
5
6
556
3
3
596
3
5
636
3
3
676
5
7
557
3
4
597
3
6
637
3
4
677
5
8
558
3
3
598
3
5
638
3
3
678
5
7
559
3
2
599
3
4
639
3
2
679
5
6
560
4
3
600
5
5
640
3
3
680
3
5
561
0
2
601
5
6
641
3
4
681
3
6
562
4
5
602
5
7
642
3
5
682
1
5
563
4
4
603
5
6
643
3
4
683
1
4
564
4
5
604
5
5
644
3
5
684
3
5
565
4
6
605
5
6
645
3
6
685
3
6
566
4
5
606
5
5
646
3
5
686
3
5
567
4
4
607
5
4
647
3
4
687
3
4
568
4
3
608
5
3
648
1
3
688
4
5
569
4
4
609
5
4
649
1
4
689
4
6
570
4
5
610
5
5
650
4
7
690
4
7
571
4
4
611
5
4
651
4
6
691
4
6
572
4
3
612
5
5
652
4
5
692
2
5
573
4
4
613
5
6
653
4
6
693
0
4
574
4
3
614
1
3
654
4
5
694
2
5
575
4
2
615
1
2
655
4
4
695
2
4
3.4 Nonlinear Representation of Cells
33
696
4
5
736
5
3
776
4
3
816
4
3
697
4
6
737
5
4
777
4
4
817
4
4
698
4
7
738
5
5
778
4
5
818
1
3
699
4
6
739
5
4
779
4
4
819
1
2
700
4
5
740
5
5
780
4
3
820
4
5
701
4
6
741
5
6
781
4
4
821
4
6
702
4
5
742
5
5
782
4
3
822
4
5
703
4
4
743
5
4
783
4
2
823
4
4
704
4
3
744
5
5
784
1
1
824
4
3
705
4
4
745
5
6
785
1
2
825
0
2
706
4
5
746
5
7
786
5
5
826
4
5
707
4
4
747
5
6
787
5
4
827
4
4
708
4
5
748
5
5
788
5
5
828
4
3
709
4
6
749
5
6
789
5
6
829
4
4
710
4
5
750
1
3
790
5
5
830
4
3
711
4
4
751
1
2
791
5
4
831
4
2
712
4
5
752
4
3
792
0
1
832
4
3
713
4
6
753
4
4
793
5
4
833
4
4
714
4
7
754
4
5
794
5
5
834
4
5
715
4
6
755
4
4
795
5
4
835
4
4
716
1
3
756
4
5
796
5
3
836
4
5
717
1
4
757
4
6
797
5
4
837
4
6
718
4
5
758
4
5
798
5
3
838
4
5
719
4
4
759
0
2
799
5
2
839
4
4
720
3
3
760
3
3
800
5
3
840
2
3
721
3
4
761
3
4
801
5
4
841
2
4
722
3
5
762
3
5
802
5
5
842
2
5
723
3
4
763
3
4
803
5
4
843
2
4
724
3
5
764
2
3
804
2
3
844
4
5
725
3
6
765
2
4
805
2
4
845
4
6
726
0
3
766
2
3
806
2
3
846
4
5
727
3
4
767
2
2
807
2
2
847
4
4
728
2
3
768
2
1
808
3
3
848
3
3
729
2
4
769
2
2
809
3
4
849
3
4
730
2
5
770
2
3
810
3
5
850
3
5
731
2
4
771
2
2
811
3
4
851
3
4
732
5
5
772
3
3
812
3
3
852
1
3
733
5
6
773
3
4
813
3
4
853
1
4
734
5
5
774
3
3
814
3
3
854
3
5
735
5
4
775
3
2
815
3
2
855
3
4
(Continued)
34
3 Cellular and Natural Computing Models and Software Simulation
856
5
5
896
3
1
936
3
3
976
3
857
5
6
897
3
2
937
3
4
977
3
2
858
0
3
898
3
3
938
3
5
978
3
3
859
5
6
899
3
2
939
3
4
979
3
2
860
5
5
900
3
3
940
3
3
980
3
3
861
5
6
901
3
4
941
3
4
981
3
4
862
5
5
902
3
3
942
3
3
982
3
3
863
5
4
903
3
2
943
3
2
983
3
2
864
5
3
904
4
3
944
4
3
984
2
1
865
5
4
905
4
4
945
4
4
985
2
2
866
5
5
906
4
5
946
4
5
986
2
3
867
5
4
907
4
4
947
4
4
987
2
2
868
5
5
908
4
3
948
2
3
988
1
1
869
5
6
909
4
4
949
2
4
989
1
2
870
5
5
910
4
3
950
2
3
990
0
1
871
5
4
911
4
2
951
2
2
991
5
2
872
5
5
912
2
1
952
4
3
992
5
1
873
5
6
913
2
2
953
4
4
993
5
2
874
5
7
914
2
3
954
1
3
994
5
3
875
5
6
915
2
2
955
1
2
995
5
2
876
2
3
916
5
5
956
4
3
996
5
3
877
2
4
917
5
6
957
0
2
997
5
4
878
2
3
918
5
5
958
4
3
998
5
3
879
2
2
919
5
4
959
4
2
999
5
2
880
4
3
920
1
1
960
4
1
1000
5
3
881
4
4
921
1
2
961
4
2
1001
5
4
882
4
5
922
5
5
962
4
3
1002
5
5
883
4
4
923
5
4
963
4
2
1003
5
4
884
4
5
924
0
1
964
4
3
1004
5
3
885
4
6
925
5
4
965
4
4
1005
5
4
886
1
3
926
5
3
966
4
3
1006
5
3
887
1
2
927
5
2
967
4
2
1007
5
2
888
3
3
928
5
3
968
4
3
1008
4
1
889
3
4
929
5
4
969
4
4
1009
4
2
890
3
5
930
5
5
970
4
5
1010
4
3
891
0
2
931
5
4
971
4
4
1011
4
2
892
3
3
932
5
5
972
4
3
1012
4
3
893
3
4
933
5
6
973
4
4
1013
4
4
894
3
3
934
5
5
974
4
3
1014
4
3
895
3
2
935
5
4
975
4
2
1015
4
2
1
3.4 Nonlinear Representation of Cells
1016
3
1
1017
3
2
1018
3
3
1019
3
2
1020
2
1
1021
2
2
1022
1
1
1023
0
35
0
A nother nonlinear representation called amultinested representation was proposed in [52] and can be used to find the parameters of a cell circuit [37] (a circuit realization of this concept is depicted in Fig. 3.3) capable to implement compactly piecewise-linear cells. Note however, than in this case:
w s z0 z1 ... zk V
(3.9)
Fig. 3.3. nA example of mixed-signal circuit implem entation of a cellular automata cell of any complexity. The above example is given for the case with three inputs. For any additional input one needs only a cascade of 2 CM OS transistors (such as T5and T6). For different numbers of transitions (i.e. different m complexities) one may use one ore more “nesting” units (each composed of 6 CM OS transistors)
36
3 Cellular and Natural Computing Models and Software Simulation
hWere the z parameters cannot be analytically determined. Instead, a computationally intensive global optimization algorithm is used to find all gene values. Compared to (3.8 ), the realization of (3.9 ), which is also piecewise-linea r, is much more compact in terms of absolute value functions (usually k | log 2 m ) and therefore better suited for physical implementations via circuits such as the one depicted in Fig. 3.3. It is interesting to note that all (3.5 ) (3.8 ) and (3.9 ) are defined around a certain optimal number m of transitions points thus revealing a certain local structural complexity of the cell, with simpler cells being the linearly separable ones (i.e. m=1). R ecent results [53] indicate that at least m=2 is required for computational universality in a cellular automata, although interesting computational effects are possible even for m=1 as seen for the gene ID= 768, performing the function of character segmentation (see Chap. )8. The following aMtlab function returns the set of transition points t1 ,...t m and the sign s for any gene given as the decimal ID of length N bits. function [s, T]=id2t(ID,N) % generate the s T parametrs for a given ID % from the corresponding string of N bits %- this is a preliminary step for representing any % Boolean function using a canonical PWL "polynomial" % Copyright Radu Dogaru, July 2006. %
[email protected] %------------------------------------------------------------------y=dec2bin(ID,N); if y(N)=='0' s=-1; else s=1; end ks=y(N); off=0.5; T=[]; for i=1:(N-1) if y(N-i)~=ks T=[T off]; ks=y(N-i); end off=off+1; end
3.4.2 Extended Families of Cells One may consider expanded families of semitotalistic functions. For instance if one replaces E with 1 E (i.e. the negate version of the central cell is considered instead of the original) another set of 1,024cells is obtained, with different emergent behaviors, among which certain computationally meaningful genes could be found. B y convention in the next we w ill denote with ID*the genes belonging to such a family. Note that either ID and ID*are defined by the same unique set
3.4 Nonlinear Representation of Cells 37
of s, b, t1,...tm parameters. nA example is 63*belonging also to the case m= 1 (linearly separable) which leads to the useful emergent function of edge detection. Other families of cells can be obtained as well by replacing 5
t 3y t y4 t y5 t with D ¦ >ai 1 yi yi 1 ai @ and taking various i 2 binary values for A >2a ,3a , a4 , a5 @ others than A >0,0,0,0@ (i.e. inverting some of the neighboring inputs). For instance one may take A >1,0,0,0@ . D
2y
U sing the above transformations it follows that there are possible 32 different families of 1,024functions each. lAl these families have the same common set of s, b, t1,...tm parameters.
3.4.3 Structured Universes of Cells Figure 3.4depicts the whole set of 1,024 functions from the basic family of semitotalistic functions with five inputs grouped into complexity (m) classes. Note than lower complexity classes (m=1) display “low frequency” features, i.e. less
Fig. 3.4. The structured universe of semitotalistic oBolean functions. oMst of the 1,024 functions are in the categories m 4 and m 5 . Only 38of them are linearly separable ( m 1 ) and only two functions have the highest possible level of complexity m 8
38
3 Cellular and Natural Computing Models and Software Simulation
than lower complexity classes (m=1) display “low frequency” features, i.e. less transitions from 0 to 1, while highest complexity cells (m= 8) display “high frequency” features (many transitions from 0 to 1 and viceversa). The most numerous are the functions with an intermediate degree of structural complexity. These functions exhibit a wide-spectrum of frequencies (assuming that a certain spectral transform is applied to the binary strings representing the functions). It is interesting to note that there is a relation between the nature of the emergent phenomena in the global array and the local structural complexity of the cells.
3.5 Modeling and Simulation of Semitotalistic Cellular Automata In the following we will expose the concepts of emergence and some practical tools for locating emergence using cellular automata based on semitotalistic cells. The piecewise-linear representation discussed above represents not only a convenient solution for the hardware implementation of such systems but also a solution for building fast simulators using the interpreter language aMtlab. hWile defining the cell as a table will require a call to the table for each particular cell location (thus requiring many slow M atlab accesses to the table function, the nonlinear (piecewise-linear) description in (3.4 ) gives directly a vectorial aMtlab function which computes the outputs for all cells internally in the precompiled aMtlab functions. Of course, for a C/C+or other compiler-based language implementation of the cellular system is more convenient to consider the table-definition of the cell. The aMtlab program for implementin g the semitotalistic cellular automata is presented next: function [x0]=semit_ca(name,steps,ID,dig,eps,vis,f); % name - character string conatining the initial state % steps - number of discrete time steps used in simulation % ID - the decimal representation of the ID % dig - =1 implements Boolean cells; =0 implements continuous state cells % eps - a perturbation value to investigate emergence % in the case of continuous state cells (normally 0) % vis - =1 succesive steps of simulation are visualized; =0 not visualized % f>0 fraction of randomized links (for the Small-World model) % f=0 the simple cellular model %-----------------------------------------------------------------% x0 - the 2D array of cells in the end %-----------------------------------------------------------------% Load the set of parameters (b,t0,t1,..) for the given ID load rezult; a=Btab(ID+1); ntr=Ntrtab(ID+1); mm=ntr-1; tr=Ttab(ID+1,:); % Mutate tr with eps tr(1)=tr(1)+eps; % Compute the z offset in (8)
3.5 Modeling and Simulation of Semitotalistic Cellular uAtomata
z=0.5*(tr(2:ntr)+tr(1:ntr-1)); s=2*mod(ID,2)-1; w0=0; if mod(ntr,2)==1 w0=w0+s*tr(1); end for i=1:mm w0=w0+s*((-1)^i)*abs(tr(1)-z(i)); end %-------------------------------------------------------------------eval(['load ',name]); x0=(x0+1)/2; %load the initial state [m n]=size(x0); i=1:m; j=1:n; % define the array of cells %-------------------------------------------------------------------% generate a list of random locations for the Small World model nloc=round(f*m*n); % number of cells to be connected with distant cells i_loc=round(1+(m-1)*rand(2,nloc)); j_loc=round(1+(n-1)*rand(2,nloc)); %-------------------------------------------------------------------% START the iterations (st is the discrete time here) for st=1:steps % perform the rewiring for the random locations selected % in the case of the Small Worlds model (see text) for p=1:nloc temp=x0(i_loc(1,p),j_loc(1,p)); x0(i_loc(1,p),j_loc(1,p))=x0(i_loc(2,p),j_loc(2,p)); x0(i_loc(2,p),j_loc(2,p))=temp; end %--------------- compute the output of all cells (Matlab implementation of % the piecewise linear model in (5) sigm=a*(x0); sigm=sigm+(x0(i,[2:n,1])+x0(i,[n,1:n-1])+x0([2:m,1],j)+x0([m,1:m-1],j)); w=w0; if mod(ntr,2)==1 w=w0-s*sigm; end for k=1:mm w=w-s*((-1)^k)*abs(sigm-z(k)); end % Consider the case of Boolean or continuous cells %------------------------------------------------------------------if dig==1 x0=(sign(w)+1)/2; % implementeaza forma canonica a lui ID else x0=(0.5*(abs(w+1)-abs(w-1))+1)/2; end %----Consider the case of visualizing or not the cell array ----------if vis==1 clf image(64*x0); axis image; title(['Iteration ',num2str(st)]) waitforbuttonpress; end %--------------------------------------------------------------------end
39
04
3 Cellular and Natural Computing Models and Software Simulation
Note that before using this program, a file called “result” has to be stored in the running directory. This file contains a list with the b, t1,...tm 1 parameters for all ID from 1 to 1,022 (the ID= 0 and ID= 1,023 are trivial constant functions with no relevance for cellular computation). The file is organized around three matrix variables all having the line as index for a given ID (line 1 corresponds to ID= 0, etc.). The matrix tBab contains the value of b, the matrix Ntrtab conatins the value of the number of transients and finally the matrix Ttab contains the effective values of the transients t1 ,...t m 1 . The information to be processed shall be previously loaded into a matrix x0 and saved with a name, which is an input parameter of the above function.
3.6 Modeling and Simulation of “Small-Worlds” Systems Note that in the above aMtlab function semit_ca.m we introduced a variable called f, allowing to randomize part of the local connections such that a fraction f of cells is not connected locally anymore. Instead of connecting the output to one of the neighbors, the output of such a cell is rerouted randomly to another cell in the array (Fig. 3.5 ). W atts [1] introduced this model as a more natural model of an interconnection network. Further research revealed that many systems in nature have this property; most “cells” are locally connected but some large-distance connections are also present. B y taking two arbitrary cells in a network aW tts defined the average degree of separation “d” as the number of connections connecting a cell to another. A small distance is desirable because it means fast computation. Indeed for small d, changes are rapidly transmitted to all cells in a network. For instance it turns out that in a social network this average distance is around 6, a fact which may appear surprising given the number of several billion of humans on Earth. uBt this is the effect of the “small-world” model where each person knows a few important or distant fellows besides her very close friends. In a cellular array, due to the fact that interconnectivity is purely local, the average separation distance d is large, i.e. the information cannot propagate faster than 1 unit of discrete space per iteration. At the opposite extreme, a fully connected network has d=1 because each cell is connected to any other possible cell. But fully connected networks are not natural and they require a large amount 2 ( N ) of interconnecting devices. Much more than mN devices required in the case of cellular systems with m cells in the neighborhood. The most important discovery of Watts is that relatively small separation lengths d, much smaller than in the case of the cellular model, can be obtained by altering a small fraction f of cells from a cellular model such that their neighboring connections are replaced with randomly chosen connections with distant cells.
3.6 Modeling and Simulation of “Small-W orlds” Systems
14
Fig. 3.5. The “Small oWrld” model vs. the cellular computing model. Achange in topology for a small fraction of the cells can lead to dramatic effects, in particular the shortening of the average separation distance d between random cells in the network
In the listing of semit_ca.m we show a simple solution to model a SmallW orlds model starting from a cellular automata. The same can be applied to other cellular systems models discussed previously. sA we will see later, the SmallW orlds effects are dramatic in terms of em ergence and they can lead to certain interesting applications. In particular it turns out that the f parameter allows a fine tuning of the model into an emergent regime, much finer than using one of the cells parameter, as it is usually the case for a cellular model.
3.7 A Wider Variety of Cells, Taxonomy and Family Labels In the above, we focused only on semitotalistic cells. However there are many other possibilities to define cells. What differentiates among the types of cells is the way of computing the projection parameter V . Their genes and nonlinear implementation are the same as described in Sect. 3.4 while their design starts always with a table similar to Table 3.2. In the following we will consider cells with n inputs and will differentiate among three relevant types of cells, as summarized in Table 3.4.
24
3 Cellular and Natural Computing Models and Software Simulation Table 3.4. Three common types of cells and their definition (in terms of computing V )
Type of cell (symbol)
P rojection variable
n
¦ uk
V
Totalistic (t)
N
k 1
Semitotalistic (s)
V
U niversal (a)
V
aMximal number of bits N defining the gene n 1
n
bu1 ¦ uk k 2
n
¦2
k 1
k 1
uk
N
2n
N
2n
Comments
The most compact kind of cells
Discussed in extenso in the previous sections H uge number of possible cells
Note that there is a more general formula for the universal cell, where V
n
¦ bk uk
k 1
[52] with an orientation vector b >b1,.., bn @ optimized to reduce the number of transitions as explained above for the semitotalistic cells. oHwever, unlike in the case of the semitotalistic cell where the optimization process is as simple as trying all values from 0 to n for the single orientation parameter b, in the case of universal cells this process is computationally intensive and therefore in the next we will consider the default orientation b 20 ,21,..,2n 1 which is not necessary optimal but gives a valid representation of the cell. A ssuming rectangular grids in D dimensions, it would be useful to define a taxonomy formed as a three-letter string (also called a family label) containing information about: the size of the grid (o ne-dimensional, two-dimensional, threedimensional, etc.), the type of cell (as indicated in the above table), and the number n of inputs. The taxonomy thus defines a family of cells but also a grid model. For example:
>
@
x The family of cells “1t3” includes totalistic cells on a one-dimensional grid, with a neighborhood formed by 3 cells (for instance the left, center and right cells). Note that many other neighborhoods with 3 cells could be defined, denoted by adding some additional information to the family type. For instance “1t3a” would define a cell like the above but where the neighborhood is defined as (2 cells to the left, mid cell, 2 cells to the right), etc. 3
x The family of cells “1a3” has 2 2 = 256 members and was extensively studied by Wolfram. As reported in [17] such cells with ID=110 give a cellular automation capable of universal computation.
3.8 A CA Simulator for lAl iKnd of Cells 34
x The family “2s5” corresponds to a case discussed in extenso in Chaps. 4and 5, i.e. that of semitotalistic cells supposed to operate on a 2D grid and having five neighbors (in a typical von Neumann neighborhood, as indicated in Fig. 3.1). Note that the same family of cells (as functions and ID) can be labeled “1s5” when the cells are supposed to operate on a 1D grid. sAexpected, identical IDs will produce different emergent effects when used in different grids or neighborhoods. x The family “2s9” has 218 members (i.e. almost a quarter of a million) and it includes a “famous” individual, the cell for the “G ame of iLfe” [23], having ID= 6,152. It was also proved to generate a cellular automata acting as an universal Turing machine. The same family of cells can be embedded into a 1D CAgrid and then it is labeled “1s9” instead. One may also imagine a 3D CA grid for the same family of cells and consequently the family is labeled “3s9”.
3.8 A CA Simulator for All Kind of Cells In what follows, a M atlab program for simulating all kind of cells, according to the taxonomy presented above, is presented. function [x0]=ca_sim(name,typ,steps,ID,dig,vis,f) % name=file name (e.g. 'ran12') containing the initial state (and the CA array dimensions) % typ= string with three characters e.g. 2s9 (2- 2D, s=semitot, 9 neighborhood) % steps = number of iterations to run the CA % ID = specification of cell % dig =1 binary cell , 0 continuously valued cell % vis =1 visualization of the evolution, 0 not visualized % f fraction of cells (according to Small World model) f=0 for the cellular model % Copyright Radu Dogaru, March 2007,
[email protected] %--------------------------------------------------------------------------------------------------------dim=str2num(typ(1)); tip=typ(2); neigh=str2num(typ(3)); if tip=='t' N=neigh+1; elseif tip=='s'
4
3 Cellular and Natural Computing Models and Software Simulation
N=neigh*2; elseif tip=='a' N=2^neigh; end % compute values needed for canonical PWL representation of cells [s, tr]=id2t(ID,N); % determine the z offset in (8) for the canonical PWL implementation offset_cell; % load the initial state (saved previously with -1/+1 values ) eval(['load ',name]); x0=(x0+1)/2; % move to 0 and 1 form % compute the size of the array [m n]=size(x0); % depending on typ determine the neighborhood indexes i=1:m; j=1:n; % generate indexes of the neighbor cells to be used by the proj.. file % be sure that you previously defined % a file corresponding to your size of the grid (neigh1.m or neigh2.m) eval(['neigh',typ(1)]); % initialize the Small Worlds model with f=fraction sworld1; % MAIN LOOP ================================ if dim==1 X=[]; end for st=1:steps if dim==1 X=[X; x0]; end; sworld2; % apply the Small Worlds model % compute projection % be sure that you previously defined a file corresponding to your "typ" eval(['proj',typ]); % compute the cells output out_cell; % visualize the dynamics if (vis==1 & dim==2) clf image(64*x0); axis image; title(['Ca type: ',typ,' ID=',num2str(ID),' Iteration ',num2str(st)]) waitforbuttonpress; end end if (vis==1 & dim==1) clf image(64*X); axis image; title(['Ca type: ',typ,' ID=',num2str(ID),' Iterations ',num2str(steps)]) end
3.8 A CA Simulator for lAl iKnd of Cells
54
A s seen, the program requires several additional .Mfiles. The neigh1.m or neigh2.m files specify indexes of the cells in the neighborhood to be later used by files like proj2s5.m which evaluates the projection according to the specified family label (note the syntax proj< typ> .m of the projection file). The files specifying the neighborhoods are presented bellow as well as two examples of projection files: %neigh1.m jr=[2:n,1]; jrr=[3:n,1:2]; jrrr=[4:n,1:3]; jrrrr=[5:n,1:4];
% right cells
%neigh2.m jr=[2:n,1]; jl=[n,1:n-1]; id=[2:m,1]; iu=[m,1:m-1];
% right cells % left cells % upper cells % lower cells
jl=[n,1:n-1]; % left cells jll=[n-1,n,1:n-2]; jlll=[n-2,n-1,n,1:n-3]; jllll=[n-3,n-2,n-1,n,1:n-4];
% proj1s5 sigm=5*x0(i,j);
sigm=sigm+x0(i,jll); sigm=sigm+x0(i,jl); sigm=sigm+x0(i,jr); sigm=sigm+x0(i,jrr);
% proj2s5 sigm=5*(x0);
sigm=sigm+x0(i,jr); sigm=sigm+x0(i,jl); sigm=sigm+x0(id,j); sigm=sigm+x0(iu,j);
Other two files are in relation with the canonical W L Pimplementation of the relationship between output and projection (enc oded in Table 3.2). The first, entitled offset_cell.m computes an offset z according to (3.8) and is external to the main loop, while the second (out_cell.m) updates the cell outputs according to (3.8) and it is included in the main loop. Choosing a different form of cell implementation requires changing only of these two files. %offset_cell.m ntr=size(tr,2); mm=ntr-1; z=0.5*(tr(2:ntr)+tr(1:ntr-1));
w0=0; if mod(ntr,2)==1 w0=w0+s*tr(1); end for i=1:mm w0=w0+s*((-1)^i)*abs(tr(1)-z(i)); end
% out_cell.m w=w0; if mod(ntr,2)==1 w=w0-s*sigm; end for k=1:mm w=w-s*((-1)^k)*abs(sigm-z(k)); end if dig==1 x0=(sign(w)+1)/2; else x0=(0.5*(abs(w+1)-abs(w-1))+1)/2; end
64
3 Cellular and Natural Computing Models and Software Simulation
The file id2t.m which calculates the transition points was presented in Sect. 3.4. Other files called by the program are the sworld1.m to initialize a fraction of random cells where the local connectivity is replaced with long range connectivity and sworld2.m to effectively implement the connectivity within the main loop: % sworld1.m % generate a list of random locations % for the Small World model nloc=round(f*m*n); i_loc=round(1+(m-1)*rand(2,nloc)); j_loc=round(1+(n-1)*rand(2,nloc));
% sworld2.m for p=1:nloc temp=x0(i_loc(1,p),j_loc(1,p)); x0(i_loc(1,p),j_loc(1,p))=x0(i_loc(2,p),j_loc (2,p)); x0(i_loc(2,p),j_loc(2,p))=temp; end
4 Emergence, Locating and Measuring It
4.1 Emergence: The Software Engineering of Cellular Computing Systems Cellular computing systems hold the potential of modeling and understanding of complex aspects of our world. So far the CNN paradigm [7], which integrates the cellular automata (CA) as a special case, offers a unified framework to understand complex emergent behaviors and to apply them in real world applications. The issue of emergence [15] in the form of a relationship between the structure of the cell and the global behavior was identified as a key factor in understanding cellular systems. To date emergence is vaguely defined and the most recent definitions perceive it as a “surprise effect” [54], i.e. a global behavior which cannot be predicted by simply knowing the local structure of the cell. Among emergent behaviors, there are some which may be effectively exploited for various applications. Since our final goal is to use cellular automata for such applications, a first step is to precisely quantify emergent behaviors and give hints for using the measures of emergence to select (filter) only those CAs with meaningful computational properties. This chapter and the next two focuses on these issues. The immediate conclusion of the “surprise effect” definition for emergence is that one in principle cannot know whether there is something interesting in the dynamics of a cellular system without running a simulation of the system and observing the global behavior. Though, recent results based on the “local activity theory” [8] suggest that, at least for reaction-diffusion cellular systems, knowing the structure of the local cell may give a good prediction for the likelihood of emergent behaviors [15], without running the entire system. Still, the exact nature of the behavior and its relevance for computational applications has to be established only after effectively running the cellular system and observing its behavior. Therefore it is reasonable to investigate what are the relationship between emergence and the local structure of the cell, at least for reducing the search space through all possible cells to a much manageable subset. But the first step in starting to do such analysis is to quantify emergence. This is the subject of this and the next two chapters. As mentioned previously, emergence is defined usually based on observation. The main problem is how is this observation done? For instance, the method of R. Dogaru: Emergence, Locating and Measuring It, Studies in Computational Intelligence (SCI) 95, 47–75 (2008) www.springerlink.com © Springer-Verlag Berlin Heidelberg 2008
48
4 Emergence, Locating and Measuring It
visual observation is the main instrument based on which Wolfram recently proposed his “new kind of science” [17,55]. Since the visual observation is done by humans, besides a certain subjectivity, if one would like to analyze a large population of cells in terms of the emergent behaviors, such a visual detection of the “surprise” effect would be boring or even impossible in a reasonable interval of time. Another questionable issue is the classification of global behaviors. So far the taxonomy proposed by Wolfram [56] with four main classes, is quite largely accepted although several others were recently proposed [57]. But one issue observed in many works is that for some special cases it would be quite difficult to choose among classes, therefore the boundaries between classes are rather fuzzy than crisp [58]. The aim of the research described herein is to introduce and exemplify a set of tools to evaluate numerically the global behavior of a cellular system. Since these tools are computer programs, they eliminate the human factor in observing emergence. The result is that an exhaustive analysis of the relationship between local cells and global behaviors can be done automatically, considering very large populations of cells. Moreover, the tools introduced herein allow one to develop design for emergence programs where a desired global behavior can be now quantified numerically as an objective function of an optimization algorithm.
4.2 Visual Interpretation of Emergent Phenomena – Classes of Behaviors According to Wolfram [56] behaviors in a cellular automata (starting from a random initial state) shall fall into one of the following four classes: x Class I – Dull behavior, the cellular automata converges towards a pattern where all states of the cell are the same. This class may also correspond to the “passivity” concept introduced in [7,8]. x Class II – Simple dynamic behavior, the cellular automata converges to a finite (alternating) number of states (fixed points with states that may differ from one cell to another or low periodicity, as spoken in nonlinear dynamics parlance). x Class III – Is defined by Wolfram as “random” behaviors (corresponding to chaotic or aperiodic regimes in nonlinear dynamics parlance). x Class IV – This is the most interesting according to Wolfram and it is vaguely defined as a “mixture of order and randomness” with “interacting structures”. One interesting idea comes from this definition, namely the “interacting structures” also called gliders in a simpler parlance. Indeed it turned out that gliders are crucial for demonstrating universal computation and emergence in cellular systems and therefore we would like to insist more on them.
4.2 Visual Interpretation of Emergent Phenomena – Classes of Behaviors 49
Now let’s use the program ca_sim.m or any other convenient simulator presented in the previous chapter to simulate the entire family of 1,024 CAs corresponding to the label 1s5. In order to do so, simply type the following line in the Matlab console: >> for ID=1:1024; [x0]=ca_sim('mid101','1s5',150,ID,1,1,0); axis image; waitforbuttonpress; end
The CA evolution is shown as a spatio-temporal figure with 150 iterations on the vertical axis, while the states of all 101 cells composing the cellular automata are displayed horizontally. The initial state contains all “black” cells except some 11 cells in the middle assigned with random states (and probability ½). Let now consider different CA behaviors from each of the classes mentioned above. First let’s start with examples belonging to Class I, as displayed in Fig. 4.1. Although they all fit Wolfram’s definition (the CA converges towards an equilibrium with all cells in the same state), at a closer look they exhibit different complexities and even emergence.
(a)
(b)
(d)
(c)
(e)
Fig. 4.1. Different behaviors of CAs belonging to Wolfram’s Class I
50
4 Emergence, Locating and Measuring It
Let first observe that there is a transient length (Tr), to be defined systematically later in Sect. 4.4, that differs among simulations. It can be measured as the number of iterations until the equilibrium CA state (all cells “white”) is reached. While the CA in Fig. 4.1a) (ID = 855) requires only a few iterations to reach the equilibrium, the longest transient (apparently associated with the most complex behavior from this class) is observed for the CA in Fig. 4.1c) (ID = 852), which needs more than 130 iterations to reach an equilibrium. The CA with ID = 852 in Fig. 4.1c) exhibits two species of gliders (traveling configurations of bits) emerging from a complex interaction process after some 20 iterations. One of these gliders “travels” downwards and the other “rightwards” but since the CA boundaries are connected it eventually collides with the first glider producing an annihilation effect around iteration 110. The existence of such gliders is a clear sign of emergent complexity. Metaphorically speaking gliders may be regarded as stable formations emerging from a low level structure (that of the coupled cells) much like stable words may emerge from a finite alphabet of letters. Moreover, gliders do interact, eventually producing higher effects much like words can interact and form phrases in a sentence. In our particular case the two gliders annihilate each other but in general, one may assume that the same CA using a different initial state may produce quite different behaviors. This is just an example of ambiguity in Wolfram’s definition, showing that a measurable numerical parameter such as Tr would be useful for a finer and more precise characterization of emergence in cellular systems. In terms of the equilibrium reached, as we will see later in Sect. 4.4, a numerical measure (called a clustering coefficient) tells us precisely what kind of stable (or unstable pattern) is reached after the transient. For Class I behaviors this measure is always at its maximum value, i.e. C = 1. Another interesting difference among the above five examples, and somehow in relation with the transient time, is given by the “speed” of a hypothetical wave, which propagates from the block of 11 initially random bits. In Fig. 4.1d the speed is slower while in Fig. 4.1e the speed is the greatest, both cases indicating an “exploding” kind of phenomena (after the first iterations more than 11 cells are engaged in forming a coherent spatial pattern). Figure 4.1c (glider producing CA) also displays an explosion with a “speed” similar to that of the CA in Fig. 4.1d). In fact at a closer look, the CA in Fig. 4.1d) also produces two gliders (one moving to the right and one to the left) which eventually collide around iteration 90 (with less collateral effects) into a stable equilibrium of the entire CA. For the CA in Fig. 4.1a there is a “shrinking” or “implosion” effect, i.e. after the first iterations the number of cells engaged in forming a coherent spatial pattern decreases, while in Fig. 4.1b there is a delicate balance between some initial growth followed suddenly by an “implosion” after iteration 30. These observations on “growing” and “imploding” behaviors led to the notion of an exponent of growth (U), as detailed in Chap. 5. Imploding effects correspond to U1. Edge phenomena (such as the delicate balance in Fig. 4.1b) correspond to values of U close to 1. From the examples above it turns out that complex behaviors are expected when
4.2 Visual Interpretation of Emergent Phenomena – Classes of Behaviors
51
U>1 but not very large (the “speed of light” limit1), as is the case in Fig. 4.1e). The “speed of light” in this case is 2 cells/iteration, as follows from the observation of a “wave front” starting from the middle (position 50) and reaching the left side of the CA after 25 iterations. As seen in Fig. 4.1c, an initial growth which is rather slow (U near 1 but much smaller than the “speed of light”) is a precondition for the birth of gliders. Why are these gliders so important? For the simple reason that by their interaction one may establish a correspondence between basic computational elements of classic computers (such as AND, NOT and OR gates) and properly designed and positioned gliders in a CA. Such behaviors may allow to translate any complex logic diagram (including that of a universal Turing machine) into a spatial initial distribution of gliders as an initial state. Consequently the resulting cellular automata can be proved to behave as a universal computer. This is basically the idea used in many proofs aiming to demonstrate that various CA can perform universal computation [71]. Among the most investigated (with a rich repertoire of gliders) is the “Game of Life” [23]. Of course not all gliders can be combined in such a way as to demonstrate universal computation, therefore the presence of gliders per se is not a necessary and sufficient condition for universal computation. Still, their presence is a signature for complex emergent behaviors. Let now consider in Fig. 4.2 other examples, chosen from what Wolfram proposes as Class II. In all these cases the CAs are evolving towards a nonhomogeneous equilibrium state (as in Fig. 4.2a or d, where some cells are in a stable state, different from that of the majority of cells) or towards a low periodic state as in Fig. 4.2b or c (where a homogeneous state for all cells alternates with period 2), or towards a nonhomogeneous periodic state as in Fig. 4.2e,f. Again it is clear that there are many different types of complexities, which may be better discriminated using the measures just mentioned before. As in the case of Class I, long transients are often associated with gliders and complexity and also with moderate speeds (or exponents of growth) of the wave front generated by the coherent pattern of the 11 initially random bits (with U ranging from just above 1, as in Fig. 4.2b, to larger values as in Fig. 4.2d and even larger as in Fig. 4.2c,e but not as larger as the “speed of light” as seen in Fig. 4.2f ). The gliders in Fig. 4.2e are among the most interesting since the result of their collision is not an annihilation process but rather the birth process of two new other gliders. As seen later, the clustering coefficient C which measures the degree of order in the final state ranges from close to 1 (indicating a “low frequency” order) as in most cases displayed in Fig. 4.2, to 0 (indicating a “high frequency” order) as in Fig. 4.2f ). 1
The “speed of light” in a cellular universe is measured in cells/iterations and its value is determined by the neighborhood radius. For instance, in a “1s9” CA, the radius, and consequently the “speed of light” is 3 because there are 3 cells to the left and therefore, there may be circumstances when in the next iteration a certain pattern may propagate 3 cells. to the left (or right). In a “2s9” CA, the speed of light is only 1 because the neighborhood is formed only by a “sheet” of 1 cell size around the central cell.
52
4 Emergence, Locating and Measuring It
a)
b)
c)
d)
e)
f)
Fig. 4.2. Different behaviors of CAs belonging to Wolfram’s Class II
Several examples of Class III cellular automata are provided in Fig. 4.3. While all examples in Fig. 4.3 fit within the definition of Class III, it is clear that some differences exist, and again the transient time (until reaching a “stable” chaotic dynamics) and the exponent of growth U allow to differentiate among the different behaviors belonging to the same class. The example in Fig. 4.3a is a typical Class III behavior, with a fast growing (with the “speed of light”) towards a chaotic dynamics. The visual observation of the pattern barely shows any sign of spatial correlation (among cells at the same time step), and the clustering coefficient C is almost equal to 0.5 indicating a perfect spatial disorder. Note that starting with the examples in Fig. 4.3b and ending with that in Fig. 4.3f ) there is an increase in the exponent of growth, which is greater than 1 in all cases. Particularly for the cases with longer transients and lower exponents of growth, there are signs of more spatial correlation which may result in a “pink noise” rather than the “white noise” specific for the CA in Fig. 4.3a. If clustering coefficients would be calculated in the “stationary” chaotic regime they will deviate
4.2 Visual Interpretation of Emergent Phenomena – Classes of Behaviors
53
from 0.5, usually towards 1, indicating a shift from complete disorder (C=0.5 or white noise) to perfect order (C=1, or C=0). The presence of gliders in the above examples is detectable, particularly in Fig. 4.3b–d, i.e. exactly in those cases with long transients and where the exponent of growth is larger than one but not reaching the largest possible value.
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 4.3. Different behaviors of CAs belonging to Wolfram’s Class III
Note the example in Fig. 4.3b, run for 150 iterations, visually representing a Class III behavior. Further simulation for 4,000 iterations indicate that after a very long transient of 2,739 iterations the system enters a low periodic state, specific for Class II. The lesson learned is that for those behaviors susceptible of high complexity, longer simulations are needed to detect their exact nature. But, looking at the exponent of growth complex behaviors may be detectable even after a shorter time simulation. This is the case for to ID=133 depicted in Fig. 4.3b. The scenario for emergence of gliders is similar with what we found for the previous examples belonging to different Wolfram classes.
54
4 Emergence, Locating and Measuring It
So far we were able to show that complexity can be characterized in many different ways, other than using the four classes proposed by Wolfram. The previous examples also suggest the introduction of three measurable quantities (the transient length, the clustering coefficient and the exponent of growth) that can give a synthetic description of the dynamic behavior. It turns out that all “interesting” phenomena, other than “chaotic” (often used for random number generators [67,72]) are related to the presence of interacting gliders. These gliders are acting as basic computational units with predictable behavior and therefore could be organized into hierarchical structures much like simple logic gates are used to generate higher levels of complex computing machines. For instance, Fig. 4.4 depicts interaction between gliders emerging from a random initial state in a CA from the 1s5 family (ID=179) where two basic logic functions (XOR and AND) are implemented as an effect of collision between gliders. More complex functions can be constructed based on such “spatiotemporal” gates. Note that some gliders (such as the horizontal one in the picture) act as reflective and diffractive mediums producing “billiard ball” effects [20] to be further exploited in defining logic functions. By properly selecting the spatial distribution of gliders in the initial state and a spatio-temporal location of an output, various logic functions can be implemented, using the same cellular medium (CA cells with ID=179). Such cellular mediums may be eventually proved to be universal computers, where the “program” is now encoded in the initial state distribution of gliders, the
Fig. 4.4. Emergence of gliders in CA with ID=179 and their use to compute basic logic functions. Within this figure, time is on the horizontal axis, while a vertical column at a specified iteration represent the collection of states for the CA cells
4.2 Visual Interpretation of Emergent Phenomena – Classes of Behaviors
55
computation is the dynamic evolution characterized by glider interaction, and there is a specified way to interpret the result of the computation as a terminal state (obtained after a certain number of iterations). There are also emergent phenomena which do not rely on gliders but still may have some meaningful computational properties. Such phenomena are better related to natural computation of the type seen in the brain, in social collectivities, etc. As shown in the applicative chapters of this book, such emergent behaviors can successfully be used to certain perceptual-related tasks such as feature extraction, image compression or time-sequence classification with the advantage of reducing the implementation complexity when compared to traditional, algorithmic solutions for similar tasks. Detecting the presence of interacting gliders or other emergent phenomena by visual observation of a CA simulation is simple in principle. But for families of CA with large numbers of individuals (e.g. “1s9”, “2s9” or “1a9”, “2a9”) visual detection becomes completely unpractical. Therefore we need to develop measures of emergence to evaluate and quantify the variety of phenomena mentioned above, including the likelihood of emergent gliders. Wolfram defines Class IV as the one containing the most interesting behaviors, mostly based on gliders. Based on a visual observation of all 1,024 CAs from the 1s5 family, we selected the next three examples shown in Fig. 4.5 as representative.
(a)
(b)
(c)
Fig. 4.5. Different behaviors of CAs belonging to Wolfram’s Class IV
Again, like in the case of ID = 133 (Fig. 4.3b), if the CA with ID=684, displayed above in Fig. 4.5b), is simulated for a longer period, it turns out that the behavior actually belongs to Class II, i.e. ending into a low periodic pattern with a high degree of order. This is clearly visible in Fig. 4.6, and confirms that there is a strong connection between the presence of interacting gliders, long transients (even if the simulation takes a small number of iterations) and a “halting” process (described by the low periodic pattern reached after a large number of it erations) specific for any computational process.
56
4 Emergence, Locating and Measuring It
Fig. 4.6. A longer simulation of CA with ID = 684 indicates that it has a behavior which can be actually classified in Class II. Nevertheless, the long transient introduced here as an emergence measure clearly indicates a complex dynamics having all features of a computational process: interaction of gliders, and halting after 2,355 iterations in this case. The lower graph indicates the evolution of the clustering coefficient used to detect the transient time
Note that for the other two examples in Class IV, we were not able to find the halting moment but this does not mean it does not exist. It is only the limit we can afford to do the simulation which impedes on finding large values of halting moments. So far, a conclusion is that simple visual classification schemes as the one proposed by Wolfram, although useful as a fist attempt to quantify emergence, are not suitable for an in-depth analysis, particularly when it should be done automatically by a computer program. Instead, numerical values (also called measure of emergence) should be defined, to capture more precisely the quality of the nonlinear dynamics in a cellular system. These measures were briefly introduced above but their accurate definitions and computing algorithms will be detailed next. Later we will show that Wolfram classes do exist and still have their relevance but as partially overlapping sub-domains in a space defined by the newly introduced measures of emergence.
4.2 Visual Interpretation of Emergent Phenomena – Classes of Behaviors 57
4.3 Semitotalistic Cells: A First Step in Narrowing The Search Space Let us consider a network of N interconnected cells. In a typical cellular architecture the cells are locally connected within a Moore or Von Neumann neighborhood. Let us denote k the number of links of a cell with other neighbors. Assuming k
that each cell has only two states (Boolean cells), there are 2 2 possible choices for the cell (all Boolean functions with k inputs). This is a huge number even for reasonably small values of k. On the other hand experiments and observations prove that there is an ordering principle governing the real world, such that not any randomly chosen Boolean cell will lead to an emergent behavior. We are thus interested to narrow the huge gene search space (of all possible Boolean functions) by revealing some structural constraints in relationship to the assumed ordering principle. Are there any structural constrains in choosing k so that emergence becomes likely? The answer is positive and it is motivated by several independent results. 1. According to the theory of “small worlds” [1] the average degree of separation2 “d” in a large network of nodes is a small number. For instance, real world webs were found to have from d 3 (for networks of molecules in a cell) to d 19 (for the World Wide Web) [59,63]. This postulate proved true for many natural systems indicating that in nature networks of interconnectivity are organized such that they are neither sparse (i.e. cells connecting only with one ore two other cells) nor dense (i.e. fully interconnected networks). For instance, in a social network d | 6 [59] representing the average number of acquaintances (shaking hands) that can be established between two arbitrary persons in the world. Note than in a fully connected network d=1 (minimum) because any node is connected to any other one. However, the network is not economic at all, requiring a large number N 2 of links.
2. In a 1D cellular system with links only to the left, the value of d reaches its maximum (i.e. d=N, where N is the total number of cells) corresponding to a sparse connectivity. Here the number of links is O(N) but the average degree of separation is too small to produce emergent phenomena (the only phenomena possible then is a propagation one, with a speed limited at one cell/iteration). Increasing the grid dimension of a cellular system (e.g. from two-dimensional grids to two-dimensional grids) will decrease d while maintaining a reasonable number of local links. For instance, d | N in the case of 2D topologies, therefore if the number of cells is less than 400, a CA with 2D topology will fit the natural models, characterized by values of d ranging from 3 to 19. It is thus 2
Average number of connections between two randomly selected nodes in the graph of the network. This number is related to k (the degree of connectivity for each node) and the interconnection topology.
58
4 Emergence, Locating and Measuring It
clear that a natural d is achieved for neither sparse nor fully connected networks. A cellular model is thus a good example of a network topology where a natural connectivity (and consequently degree of separation d) occurs by properly choosing the number of cells N and the interconnection topology. As shown in the previous chapter and implemented in the CA simulator with Small Worlds effect, by altering a small fraction f 1 of the local connections within a cellular model, a dramatic change in d can be achieved in the sense of reducing it (as compared to the pure cellular model, where f=0) without changing the number of cells. Therefore, within the cellular model, we have an instrument to finely tune d without changing the overall architecture. 3. Kauffman proposed random Boolean networks as models of biological processes, showing that emergence (in the sense of behaviors comparable with some natural phenomena to be modeled) occurs neither for a high connectivity nor for a very low one but for a certain “optimal” degree of connectivity, i.e. in a “small worlds” parlance, an optimal d [60]. Since our aim is to search for complex emergent processes in cellular systems, in the light of the above we are interested to see what are the implications of the typical structural constraints for the degree of connectivity and therefore for emergence. In a network of N cells, if a cell is connected on average with k other cells (neighbors, but also distant neighbors for the altered fraction f), the following relationship holds: N | k d . Since in a cellular system k is usually constrained to 5 or 9, the optimal number of cells (in order to achieve a “natural” separation distance 3 d 8 , as confirmed by “Small Worlds” models for various natural systems) is given in Table 4.1: Table 4.1. Number of cells for various neighborhoods and degrees of separations k k
5 9
d 3 125 (11 x 11) 729 (27 x 27)
d 4 625 (25 x 25) 6561 (81 x 81)
d 6 15625 (125 x 125) 531441 (729 x 729)
d 8 390625 (625 x 625) (6561x 6561)
The resulting N values correspond to typical cellular systems sizes used in various models and applications. It turns out that in order to increase the chances for emergence, a larger number N of cells in needed in the case of nine-cells neighborhoods than for the cellular models with 5-cells neighborhoods. Therefore it would be computationally more convenient to investigate first such small neighborhood models. Let us now consider the structural constraints of the cell and how they influence the connectivity of a cell k. From the above discussion we will conclude that among all possible Boolean cells, the sub-family of semitotalistic cells concentrate most of the potentially emergent behaviors. The behavior of a cellular network is defined by the local rule (i.e. the cells ID). As we already mentioned, there is a huge number of Boolean functions even for
4.2 Visual Interpretation of Emergent Phenomena – Classes of Behaviors 59
the case of small local connectivity. Without any structural restriction, i.e. accepting k
all local rules as potentially emergent we have to investigate 22 232 | 4 109 local rules, for k 5 . It is quite impractical, if not impossible, to test all of them for emergence. The situation is getting even worse for larger neighborhoods. But are all these functions equally important? In the following I will propose a framework which differentiates among these Boolean functions and reveals how to address only a smaller fraction of them, while maximizing the chance to get emergent behaviors. As shown in [52] and in the previous chapter any Boolean function can be represented as a piecewise linear (but nonlinear) function: y
s sgn w V ,
(4.1)
k (4.2) ¦ bi u i , i 1 where w() is a nonlinear function with up to p roots, and V is the onedimensional projection of the k-dimensional input vector u >u1 , u2 ,...uk @ through the projection3 (or coupling) vector b >b1 , b2 ,.., bk @ . In the CA framework, the binary output y of any cell in the network is updated according to (4.1) and (4.2) with an input vector u >u1 , u2 ,...uk @ formed by the binary outputs of the cells in the neighborhood at the previous discrete time moment. The importance of describing an arbitrary Boolean function as (4.1) and (4.2) is twofold: V
x First, the coupling equation (4.2) indicates the coupling degree of a cell with its k
neighbors. The coupling degree can be defined as c
¦ bi
i 1
max bi
and it is not
revealed unless a Boolean cell is described as a nonlinear function. The maximum degree of coupling c k is obtained for totalistic cells (defined in previous works [56] as cells where all bi are equal in their absolute value). As shown above, in a network of cells there is a critical range for the cell coupling so that emergence may occur (postulating that emergence is a characteristic of a natural system characterized by 3 d 8 ). For any other values of bi , the degree of coupling will decrease resulting in a departure from the optimal critical coupling k responsible for emergence (as shown in Table 4.1). Thus many of the Boolean functions characterized by a low degree of coupling may be discarded. Correspondingly, it would be more reasonable to focus first on the group of 2k k totalistic functions characterized by the maximal degree of k coupling k for a given neighborhood. Compared to all 2 2 Boolean cells it 3
Here we will assume optimal projections i.e. b is optimized such that using the representation (4.1) and (4.2) of a Boolean function it is achieved with a minimum of p roots of the w() function. The linearly separable function correspond to p=1.
60
4 Emergence, Locating and Measuring It
represents a much smaller and manageable fraction. Since the optimal degree of coupling (for a given CNN size) is not a crisp value we would expect to find also some emergent phenomena for less coupled cells although we expect them to be more sparse than from the set of totalistic functions. Therefore it would be reasonable to consider a second, extended set of local rules, i.e the class of semitotalistic functions (note that the set of totalistic functions is included in the set of all semitotalistic functions). Here, one of the coupling coefficients bi is allowed to have a different absolute value. It follows that in this case, the coupling degree ranges as 2 1k d c d k . Note than in his work, Kauffman [60] found that a degree of coupling of 2 as minimal for producing emergent computation. It follows from the above that d max log 2 ( N ) for a coupling degree of 2. For N 15625 cells (i.e. a 125 ×125 two-dimensional CNN) this gives d max | 14 , i.e. at the most superior limit of the known natural systems. On these grounds we may postulate that most of the emergent behaviors have to be found within the families of semitotalistic local Boolean functions. It is thus no surprise that looking at most of the useful CNN templates [32] we find that they obey the above conjecture.
x Second, the nonlinear representation of the Boolean function allows to define a a local structural complexity, N defined [53] as the optimal (minimum) value of p (the roots of the discriminant function w in (4.1) for the realization of a given Boolean function. This coefficient is actually the same as m used in the piecewise linear definition (3.8) from the previous chapter. The linearly separable functions correspond to N d 1 , while all the other functions, with different levels of complexities correspond to the linearly not separable functions. The notion of local structural complexity was first introduced in [53], where it was conjectured that a necessary condition for emergence and complexity in an array of Boolean cells is that they are linearly not separable. This conjecture was then proved for one-dimensional CAs with 3 cells neighborhoods (the 1a3 family). The tools for detecting emergence presented in the next sections confirm this conjecture for many other types of 1D and 2D semitotalistic and totalistic cellular systems with various neighborhoods.
4.4 Clustering and Transients and as Measures of Emergence Emergence is observed as a global effect of the local interconnection of cells. Wolfram employs a visually based classification of emergent phenomena using four qualitative classes. Although his classification proved useful to derive certain conclusions, it is quite unpractical to evaluate large numbers of rules in such a way. In [61] we introduced several tools to detect emergence as a long transient
4.2 Visual Interpretation of Emergent Phenomena – Classes of Behaviors
61
dynamics in the evolution of a global parameter called there a magnitude of cellular disorder. In [62] we refined the above analysis defining a dynamics signature as a vector of finite size (usually log 2 (T ) , where T is the number of CA iterations) and we were able to prove that a clustering neural network is able to detect six common prototypes of these signatures, which may correspond to the same number of distinct qualitative behaviors. For any particular dynamic behavior a degree of membership to one of the prototypes can be easily computed thus providing an automatic tool for detection of certain types of emergent behaviors. For instance such tools allow genetic or other types of global optimization algorithms to search in a huge space of possible cells only for those cells producing a certain type of emergent behavior. This would be something impossible to do using visual categorization of global phenomena (as proposed by Wolfram) since a human decision would be required in each objective function evaluation of the optimization algorithm. Here we will go further and propose a much simpler and easier to use measure of complexity. Compared to [62] we eliminate the need to use an adaptive vector quantization scheme and for each global state of the cellular automata at the terminal state T we evaluate a clustering coefficient “C(T)” and a transient length “Tr”. Both can be easily computed from the time series formed of consecutive clustering coefficients C (t ) t 1,..,T . Other measures of complexities were proposed in the literature (e.g. Wuensche’s input entropy [73]). As detailed later it was found however that the clustering coefficient is computationally less complex than the input entropy. The clustering coefficient is exclusively a feature of the array of cells and it can be computed for either binary or multi-state cells. The use of a clustering coefficient was suggested by the observation that in a cellular system with a complex emergent behavior an initially chaotic state pattern evolves towards a sequence of patterns characterized by a steady state value of the clustering index. The “degree of clustering” was proposed by several authors as a measure of complexity in complex networks [63]. Moreover, in biological systems, life can be regarded as long transient processes from a certain degree of clustering among an array of cells (the initial or birth state) to another relatively stable clustering degree formed at the end of the life cycle. The clustering coefficient may vary between 0 (indicating a “high frequency order: in the form of a chessboard) to 1 (indicating a “low frequency order”, i.e. patterns where all cells have their neighbors in the same state). For a random uniformly distributed array of cells the clustering coefficient is 0.5, thus indicating a “perfect disorder”. From the examples discussed in Sect. 4.2, several strategies in locating various behaviors follow: If after running a CA for T time steps the final clustering coefficient C(T) is around 0.5 the behavior can be classified as a “chaotic” or Class III one, and its immediate application may be in building random number generators. Note for instance the example in Fig. 4.7b. It is interesting to observe that in the case of chaotic dynamics the time series C(t) has a relatively large variance. As we will see later, in calculating the transient we will define the admissible variation ' as the
62
4 Emergence, Locating and Measuring It
variation of C(t) for the last 20% steps of the run. To precisely locate the transient we will look back in the past to detect the transient when the value of C(t) goes out of the boundaries imposed by ' . This procedure makes possible to avoid false results in terms of transition time, particularly for the case of chaotic behaviors. Let us also define E C T 0.5 . The closest this value is to 0, the better is the quality of the random number generator. Experimental results indicate that if E ! 0.05 , the transient length should be additionally considered in order to have a better description of the emergent behavior. Large values of Tr clearly indicates the possibility of gliders and other complex phenomena, with many potential applications. Two such examples are given in Fig. 4.7a, c. In the first, cell ID = 133 from the 1s5 family is used and the final clustering coefficient C(T) has a value far from pure disorder (note the variance which is always smaller when E is large). The transient length in this case is 173, for a cellular system with 31 cells. In the following we will accept that a transient is large if it is larger than the number of cells in the CA. This indicates that in order to detect large transients we shall run the CA for a large number of iterations. In practice we found convenient to choose T 10 N . In the second case (Fig. 4.7c), the cell ID = 372 from 1s5 gives a behavior with a moderate to low transient of 92 (comparable with N=100 in this case), a signature for less complexity. Indeed, the final clustering coefficient is now closer to the 0.5 indicating “pure disorder” and less interacting structures (gliders) are observed. As E 0.04 and approaches 0, the variance of the clustering coefficient is larger. Finally, there are many low complexity behaviors characterized by a short transient (smaller or much smaller in comparison to N ) towards as near “pure order” state, characterized by a large deviation from disorder E . Such an example, plotted in Fig. 4.7d) is characterized by C(T)=1, after a short transient of Tr 15 . Note again the small admissible variance ' , as E reaches its maximum. We need now some systematic procedures to compute the above parameters: At a given time moment t the global state of a cellular automata is ^xi, j t `. It is assumed that each cell output xi, j ^0,1` . For a given array X of cells the following formula is used to compute the clustering coefficient C : X t
C
ª § ·º « abs¨¨ kX ¦ Xl ¸¸ » l1 ¹ » © mean «1 « » k « » «¬ »¼
(4.3)
where mean Y is the average value of the whole set of elements of matrix Y, 1 is a neighborhood formed by the immediately adjacent cells (excluding diagonal cells and the central cell), k is the number of cells in the neighborhood 1 and Xl is the array of cells outputs relative to the center cell according to neighbor index l . The abs operator applied to a matrix returns another matrix where each
4.2 Visual Interpretation of Emergent Phenomena – Classes of Behaviors
(a)
(b)
(c)
(d)
63
Fig. 4.7. Evolution of cells and clustering coefficients for several types of behaviors: (a) Class II with very long transients; (b) Class III; (c) Class IV; (d) Class I (using Wolfram’s taxonomy). Note that the clustering coefficient (plotted on vertical axis in the lower graphs) converges towards a “steady state” in the sense of a constant variance. This variance is larger for chaotic dynamics and is considered for the last 20% of the iterations, to detect the transient phenomena much accurately
element is the absolute value of the corresponding element in the input matrix. For instance, in the case of one-dimensional arrays with N cells, the above formula becomes: C
where X
xi 1 xi 1 2 xi º 1 Nª ¦ «1 », 2 N i 1«¬ ¼»
(4.4)
>x1 ,...xi ,..x N @ .
It is interesting to note that the clustering coefficient defined as above is based on a diffusive filter applied to the matrix of cell states. It is also related to a quantity, the current entering a multiport-cell in a Reaction-Difussion CNN, essential for the local activity theory [8]. Before computing the transient length, it is necessary to have the whole timesequence ^C t `t 1,..T computed. Let us now consider the following two Matlab
64
4 Emergence, Locating and Measuring It
modules implementing the above concept, together with the CA simulator ca_sim.m presented in the previous chapter: function y=dif_filter(x0) % implements a diffusive filter assuming that x0 is in the % range 0 1 [m, n]=size(x0); i=1:m; j=1:n; jl=[n, 1:n-1]; jr=[2:n,1]; iu=[m, 1:m-1]; id=[2:m,1]; if m==1 y=1-0.5*abs(x0(jl)+x0(jr)-2*x0); else y=1-0.25*abs(x0(i,jl)+x0(i,jr)+x0(iu,j)+x0(id,j)4*x0); end
% compute_clus.m % compute the clustering coefficient during % each CA iteration df=dif_filter(x0); if dim==1 C=[C mean(df)]; elseif dim==2 C=[C mean(mean(df))]; end
The function dif_filter.m is a diffusive filter implementing (4.3) for either onedimensional and two-dimensional CAs. This function is called from compute_clus.m, which has to be included into the main loop of the CA simulator. Before the loop one should include an initialization of C to a void vector. At the end of the main loop the entire time sequence of clustering coefficients is available as a vector C with a size T. To calculate the transient length Tr the following are applied to the time sequence ^C t `t 1,..T :
x The sequence is smoothed, for instance using a FIR filter defined as: tzf (t )
1 10 ¦ C (t k ) 10 k 1
x The admissible variance ' is computed based on the last 20% of the samples of tzf(t) : '
T
2
¦ tzf (t ) mean(tzf )
t 0.8T
x Tr is decremented from T to 1 and its value is outputted when tzf (Tr ) tzf (T ) ! '
x An average clustering coefficient Clus is computed as an average over the time steps of the C t sequence after the transient period: Clus
C (f )
1
T
¦ C (t ) T trans t trans 1
In addition to the defined measures of emergence (Tr, and Clus) another measure, the variance Var is determined. It is inspired from other works [73–75] where
4.2 Visual Interpretation of Emergent Phenomena – Classes of Behaviors 65
variances of entropies (called input entropy [73]) were determined to classify automatically the behavior of cellular automata. Our variance measure is simply determined as Var ' , i.e. it is actually the admissible variance computed as above. As we will see later, the two measures Tr and Var ' complement each other and can be used to compose a synthetic and powerful indicator of complexity Cplx capable to predict the occurrence of gliders and complex behaviors with a high probability. In comparison to Wuensche’s input entropy and its variance, the measures proposed above have the advantage of a smaller computational complexity. Note that for each CA iteration, the largest computational time is allocate to the diffusive filtering in (4.3). The complexity of this equation is of the same order of magnitude as the complexity of updating all cells in the CA. Therefore, an overall computational complexity of no more than 2–3 times larger the time used to run the CA without estimating the measures for emergence is expected. Moreover, (4.3), which may be regarded as a one-step simulation of a derivate CA, may be easily executed on the same fast parallel hardware used to simulate the CA. On the opposite, the algorithm to compute the input entropy is sequential in nature and would require more computational resources for the same number of cells. Also, long transients that are often mentioned in literature as a sign of complexity are not exactly measured using the input entropy but rather postulated to be in a direct dependence to large variances of it. As we will see later, the two measures Var and Tr are rather orthogonal and a better measure of complexity can be defined as a linear combination of them. function [trans, clus, tzf, delta]=compute_complex(C) % Usage: [Tr, clus, tzf, var]=compute_complex(C); % input data is the temporal sequence of % the clustering index % tzf is the filtered version of it, % Tr, clus, and var are parameters used to estimate % complexity %--------------------------------------------------steps=size(C,2); tzf=filter(ones(1,10),1,C)/10; % in the above, the C sequence is filtered delta=0.001+8*sqrt(var(tzf(round(0.8*steps):steps))); % in the above the variance is determined % for the last 20% time of CA run en=tzf(steps); % end value to be compared with previous % in order to determine the transient trans=steps; while (abs(en-tzf(trans))<delta) & (trans>1) trans=trans-1; end clus=mean(tzf(trans:steps));
66
4 Emergence, Locating and Measuring It
The compute_complex Matlab function implements the above concepts and it can be called from the main CA simulator program at the end of the main loop, assuming that the sequence of clustering coefficients is computed as indicated above. In evaluating the above measures of emergence a certain degree of approximation has to be accepted. The following are some reasons for this approximation:
x It is unpractical to simulate a cellular system to infinity. Therefore we will define in advance a stopping moment T . Empirically it was found that in most of the cases a “steady state”4 of C t is achieved if T DN with typical values D 2 y 10 . Therefore instead of a theoretical C f we use the Clus value estimated as above. x As seen in Fig. 4.7b C t may have large fluctuations, impeding on finding the exact length of the transient. However in practice the exact length of the transient does not matter too much, and in fact it was found that its dependence on the number of cells N is more relevant for characterizing the global dynamic behavior. Therefore, we introduced the admissible variance ' and measure its estimate as indicated above. This estimate is less accurate for Class III behaviors where the fluctuation can be quite large. For a better accuracy of the transient time Tr, it is also useful to average over several CA runs, each with a different random initial state. Though, note that improving accuracy comes for the price of an increase in the computational complexity (proportional with the number of additional CA runs). 4.4.1 Visualizing Complexity for an Entire Family of Cellular Systems – Wolfram Classes Revisited
Now we can characterize each individual of a CA family (e.g. one of the 256 members of the 1a3 family) as a point in a plane determined by two of the measures for emergence defined above. Let us first consider, the plane formed by the pair of variables Tr, Clus . Since the above family of CAs was used by Wolfram [56] to define its four classes, such plots will give a visualization of the class concept when measured as proposed above. For visualization purposes is convenient to replace Tr with its logarithm Ltr ln Tr . The plot for the 1a3 family is depicted in Fig. 4.8. Each point (circle in the graph) is associated to a specific ID, while its position within the plane is correlated to its associated class (within the four proposed by Wolfram). Such a visual representation gives now a better perspective on the relationship between our measures and the 4-class system.
4
Here by “steady state” we accept small period oscillations or even chaotic oscillation with an amplitude within the admissible variance ' defined as above.
4.2 Visual Interpretation of Emergent Phenomena – Classes of Behaviors
67
Fig. 4.8. The >Ltr, C @ plane as a tool to rapidly identify different types of global behavior. Note than the boundaries between domains are rather fuzzy and not sharp as suggested by the use of word “classes of behavior”. The dynamics of several 1D cellular automata are used to exemplify the location of various behaviors on the plane. The ID coding is the one proposed in this book and it coincides with the one proposed by Wolfram in [55,56]
The plane may be divided into several sections although the borders are not as sharp as indicated in the picture. On the middle ( Clus 0.5 0.05 ) there is a strip corresponding to the Class III of Wolfram. Here the length of the transient is less interesting (due to large fluctuations it is imprecisely measured), but all CA within this strip display the randomness associated with Class III. The CA with ID = 30 (reported to be used for the random number generator in Mathematica) belongs to this strip. Above and below the strip, the length of the transient differentiates among the left side (short transients, or Class I, and II) and the right side (long transients, complex behaviors, i.e. Class IV). Where exactly should be that border placed? There is no precise answer but experiments suggest that taking the boundary on the Ltr axis at lnN may be a good choice, consistent with experimental observations showing that transient times comparable with the number of cells are often associated with the presence of gliders. In our case, for N 100 cells, the boundary is at Ltr # 4.6 . Our emergence measures allow to draw a portrait for an entire family of cells while not discarding entirely the classification proposed by Wolfram. They rather allow to see the subtle differences between genes otherwise considered as belonging to the same class, thus resolving the uncertainty in the proposed classification system, as observed by many authors.
68
4 Emergence, Locating and Measuring It
Fig. 4.9. Projection on the Ltr, Clus for the 1,024 members of the 2t9 CA family. Note that long transient behaviors in this case, correspond also to large values (close to 1) of the clustering coefficient
As expected, the famous “110” CA (proved in [17] to be the simplest CA capable of universal computation and its 3 “relatives” (according to [76, 77]) with ID = 124, 137 and 193 are located within the Class IV region of the plane, but quite closely to the Class III strip. Indeed, gliders are present in such CAs but within a rather “noisy” dynamics, typical for Class III. As seen later, the profile for different families of CA is such that some individuals will be placed in the Class IV region but closer to the Clus=1 boundary, indicating the presence of neat gliders, not affected by noise. For instance, Fig. 4.9 presents the profile obtained in the Ltr, Clus plane for the 2t9 CA family (totalistic, two-dimensional with 9 cells in the neighborhood). In addition to clarifying the origin of uncertainties in the classification scheme proposed by Wolfram, the above measures of emergence bring several other important advantages:
x One may easily do a statistics on a large family of cells, inferring relationships such as the probability that a certain kind of behavior will occur in a family or another (e.g. 1s5 or 1s9).
4.2 Visual Interpretation of Emergent Phenomena – Classes of Behaviors 69
x Using the projections on Ltr, Clus planes allows to draw relationships between measures of the local complexity (computed at the cell level) and the global complexity measured as described above. For instance, if one selects all ID-s with large transients (Tr>200) in the case of “1s5” CA family, it turns out that gliders are present only in the case of cells with a local structural complexity defined by m ^4,5` . Also, “chaotic” patterns are usually obtained for those cells with higher local structural complexity m ^6,7,8` , with m 8 providing the best random number generators.
x One may employ optimization algorithms acting on an objective function defined as a function of the above emergence measures. Such objective functions may be used to locate only those genes leading to a type of desired behavior. This idea will be further exploited in Chap. 6 where “sieves” are defined in association to such objective functions. For instance if one wants to locate chaotic (or Class III) behaviors, a proper objective function to minimize is: Clus 0.5 . Such a search would be impossible using the visually based classification system of Wolfram.
4.5 Simulation Examples, Properties, More Accurate Complexity Measures Let now consider several families of CA, and apply them the above measures of emergence. Projections in the three-dimensional space (Tr,Clus,Var), such as the one depicted in Fig. 4.10, will give an overall characterization of each CA family. First we will consider the CA family “1s5” (semitotalistic cells with five neighbors, arranged on a one-dimensional grid topology). In our simulations a CA with 100 cells is considered with a simulation time T=400, considered large enough to provide information about long transients. The following simple Matlab program is invoked to obtain a database (here the array Tab) necessary for a further characterization of the CAs in the family. >> Tab=[]; for id=1:1024 [x0, trans, clus, var]=ca_sim('mid100','1s5',400,id,1,0); Tab=[Tab; [id, trans, clus, var]]; id, end;
The information in “Tab” can be used to draw a projection of the entire family within the (Tr,Clus, Var) space (see Fig. 4.10) or as a database for applying certain “sieves” (detailed later in Chap. 6) to filter out cells associated with certain desired behaviors that may be of interest in practical applications. For instance, the next Matlab command provides the list of IDs associated with good random number generators ( Clus 0.5 0.002 and Var ! 0.1 ): >> ix=find(abs(Tab(:,3)-0.5)0.1)
70
4 Emergence, Locating and Measuring It
Fig. 4.10. Projection of the “1s5” CA family within the (Tr,Clus,Var) plane. Observe that complex behaviors (gliders) are associated with either long transients or large variances. Note also that Tr and Var are orthogonal, i.e. there are little CAs having both large variances and transient times
Such a list contains the following IDs: 378, 634, 645, 650 but it can be expanded if a larger range for Clus is accepted. All these cells are associated with large values of the local structural complexities m as seen from Table 3.3. The following “sieve” reveals the complex behaviors, with long transients (Tr>200): >>ix=find(Tab(:,2)>200),
returning the following list of IDs: 21, 212, 294, 475, 507, 684, 851, 866, 870. Note the presence of ID = 684, already described in detail in Fig. 4.4. Also when checked with Table 3.3. it is found that local structural complexities m ^4,5` are lower than in the case of cells leading to “chaotic” behaviors, but still large. A closer look to Fig. 4.10 reveals that complex behaviors are rather seldom, most of the CA individuals (IDs) being located near the axis Tr=0 with low variance and low transient lengths and with a wide range of clustering coefficients. Indeed, as many authors already reported, complex emergent behaviors in cellular automata are rare, and consequently difficult to locate. For mid values of the clustering coefficients, there is a large concentration of IDs with 0.1