New developments in formal languages and applications

Gemma Bel-Enguix, M. Dolores Jiménez-López and Carlos Mart´ın-Vide (Eds.) New Developments in Formal Languages and App...

Author: Gemma Bel-Enguix | M. Dolores Jiménez-López | Carlos Martin-Vide

40 downloads 1044 Views 13MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Gemma Bel-Enguix, M. Dolores Jiménez-López and Carlos Mart´ın-Vide (Eds.) New Developments in Formal Languages and Applications

Studies in Computational Intelligence, Volume 113 Editor-in-chief Prof. Janusz Kacprzyk Systems Research Institute Polish Academy of Sciences ul. Newelska 6 01-447 Warsaw Poland E-mail: [email protected] Further volumes of this series can be found on our homepage: springer.com Vol. 92. Ang Yang, Yin Shan and Lam Thu Bui (Eds.) Success in Evolutionary Computation, 2008 ISBN 978-3-540-76285-0 Vol. 93. Manolis Wallace, Marios Angelides and Phivos Mylonas (Eds.) Advances in Semantic Media Adaptation and Personalization, 2008 ISBN 978-3-540-76359-8 Vol. 94. Arpad Kelemen, Ajith Abraham and Yuehui Chen (Eds.) Computational Intelligence in Bioinformatics, 2008 ISBN 978-3-540-76802-9 Vol. 95. Radu Dogaru Systematic Design for Emergence in Cellular Nonlinear Networks, 2008 ISBN 978-3-540-76800-5 Vol. 96. Aboul-Ella Hassanien, Ajith Abraham and Janusz Kacprzyk (Eds.) Computational Intelligence in Multimedia Processing: Recent Advances, 2008 ISBN 978-3-540-76826-5 Vol. 97. Gloria Phillips-Wren, Nikhil Ichalkaranje and Lakhmi C. Jain (Eds.) Intelligent Decision Making: An AI-Based Approach, 2008 ISBN 978-3-540-76829-9 Vol. 98. Ashish Ghosh, Satchidananda Dehuri and Susmita Ghosh (Eds.) Multi-Objective Evolutionary Algorithms for Knowledge Discovery from Databases, 2008 ISBN 978-3-540-77466-2 Vol. 99. George Meghabghab and Abraham Kandel Search Engines, Link Analysis, and User’s Web Behavior, 2008 ISBN 978-3-540-77468-6 Vol. 100. Anthony Brabazon and Michael O’Neill (Eds.) Natural Computing in Computational Finance, 2008 ISBN 978-3-540-77476-1 Vol. 101. Michael Granitzer, Mathias Lux and Marc Spaniol (Eds.) Multimedia Semantics - The Role of Metadata, 2008 ISBN 978-3-540-77472-3 Vol. 102. Carlos Cotta, Simeon Reich, Robert Schaefer and Antoni Ligeza (Eds.) Knowledge-Driven Computing, 2008 ISBN 978-3-540-77474-7

Vol. 103. Devendra K. Chaturvedi Soft Computing Techniques and its Applications in Electrical Engineering, 2008 ISBN 978-3-540-77480-8 Vol. 104. Maria Virvou and Lakhmi C. Jain (Eds.) Intelligent Interactive Systems in Knowledge-Based Environment, 2008 ISBN 978-3-540-77470-9 Vol. 105. Wolfgang Guenthner Enhancing Cognitive Assistance Systems with Inertial Measurement Units, 2008 ISBN 978-3-540-76996-5 Vol. 106. Jacqueline Jarvis, Dennis Jarvis, Ralph Rönnquist and Lakhmi C. Jain (Eds.) Holonic Execution: A BDI Approach, 2008 ISBN 978-3-540-77478-5 Vol. 107. Margarita Sordo, Sachin Vaidya and Lakhmi C. Jain (Eds.) Advanced Computational Intelligence Paradigms in Healthcare - 3, 2008 ISBN 978-3-540-77661-1 Vol. 108. Vito Trianni Evolutionary Swarm Robotics, 2008 ISBN 978-3-540-77611-6 Vol. 109. Panagiotis Chountas, Ilias Petrounias and Janusz Kacprzyk (Eds.) Intelligent Techniques and Tools for Novel System Architectures, 2008 ISBN 978-3-540-77621-5 Vol. 110. Makoto Yokoo, Takayuki Ito, Minjie Zhang, Juhnyoung Lee and Tokuro Matsuo (Eds.) Electronic Commerce, 2008 ISBN 978-3-540-77808-0 Vol. 111. David Elmakias (Ed.) New Computational Methods in Power System Reliability, 2008 ISBN 978-3-540-77810-3 Vol. 112. Edgar N. Sanchez, Alma Y. Alan´ıs and Alexander G. Loukianov Discrete-Time High Order Neural Control: Trained with Kalman Filtering, 2008 ISBN 978-3-540-78288-9 Vol. 113. Gemma Bel-Enguix, M. Dolores Jiménez-López and Carlos Mart´ın-Vide (Eds.) New Developments in Formal Languages and Applications, 2008 ISBN 978-3-540-78290-2

Gemma Bel-Enguix M. Dolores Jiménez-López Carlos Mart´ın-Vide (Eds.)

New Developments in Formal Languages and Applications

123

Gemma Bel-Enguix M. Dolores Jiménez-López Carlos Mart´ın-Vide Rovira i Virgili University Research Group on Mathematical Linguistics Plaza Imperial Tàrraco, 1 43005 Tarragona Spain

ISBN 978-3-540-78290-2

e-ISBN 978-3-540-78291-9

Studies in Computational Intelligence ISSN 1860-949X Library of Congress Control Number: 2008921711 c 2008 Springer-Verlag Berlin Heidelberg ° This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Cover design: Deblik, Berlin, Germany Printed on acid-free paper 9 8 7 6 5 4 3 2 1 springer.com

Preface

The theory of formal languages is widely accepted as the backbone of theoretical computer science. It mainly originated from mathematics (combinatorics, algebra, mathematical logic) and generative linguistics. Later, new specializations emerged from areas of either computer science (concurrent and distributed systems, computer graphics, artificial life), biology (plant development, molecular genetics), linguistics (parsing, text searching), or mathematics (cryptography). All human problem solving capabilities can be considered, in a certain sense, as a manipulation of symbols and structures composed by symbols, which is actually the stem of formal language theory. Language – in its two basic forms, natural and artificial – is a particular case of a symbol system. This wide range of motivations and inspirations explains the diverse applicability of formal language theory Ű and all these together explain the very large number of monographs and collective volumes dealing with formal language theory. In 2004 Springer-Verlag published the volume Formal Languages and Applications, edited by C. Martín-Vide, V. Mitrana and G. Păun in the series Studies in Fuzziness and Soft Computing 148, which was aimed at serving as an overall course-aid and self-study material especially for PhD students in formal language theory and applications. Actually, the volume emerged in such a context: it contains the core information from many of the lectures delivered to the students of the International PhD School in Formal Languages and Applications organized since 2002 by the Research Group on Mathematical Linguistics from Rovira i Virgili University, Tarragona, Spain. During the editing process of the aforementioned volume, two situations appeared: Some important aspects, mostly extensions and applications of classical formal language theory to different scientific areas, could not be covered, by different reasons. New courses were promoted in the next editions of the PhD School mentioned above.

VI

Preface

To intend to fill up this gap, the volume Recent Advances in Formal Languages and Applications, edited by Z. Ésik, C. Martín-Vide and V. Mitrana, was published in 2006 by Springer-Verlag in the series Studies in Computational Intelligence 25. The present volume is a continuation of this comprehensive publication effort. We believe that, besides accomplishing its main goal of complementing the previous volumes in representing a gate to formal language theory and its applications, it will be also useful as a general source of information in computation theory, both at the undergraduate and research level. For the sake of uniformity, the introductory chapter of the first volume that presents the mathematical prerequisites as well as most common concepts and notations used throughout all chapters appears in the present volume as well. However, it may happen that terms other than those in the introductory chapter have different meanings in different chapters or different terms have the same meaning. In each chapter, the subject is treated relatively independent of the other chapters, even if several chapters are related. This way, the reader gets in touch with diverse points of view on an aspect common to two or more chapters. We are convinced of the usefulness of such an opportunity to a young researcher. Acknowledgements Our deep gratitude is due to all the contributors, for their professional and friendly cooperation, as well as to Springer-Verlag, for the efficient and pleasant collaboration. Tarragona, October 2007

Gemma Bel-Enguix M. Dolores Jiménez-López Carlos Martín-Vide

Contents

1 Basic Notation and Terminology ................................................................

1

2 Open Problems on Partial Words Francine Blanchet-Sadri . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3 Alignments and Approximate String Matching Maxime Crochemore and Thierry Lecroq . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4 An Introductory Course on Communication Complexity Carsten Damm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 5 Formal Languages and Concurrent Behaviours Jetty Kleijn and Maciej Koutny . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 6 Cellular Automata – A Computational Point of View Martin Kutrib . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 7 Probabilistic Parsing Mark-Jan Nederhof and Giorgio Satta . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 8 DNA-Based Memories: A Survey Andrew J. Neel and Max H. Garzon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259

1 Basic Notation and Terminology

This chapter presents the basic mathematical and formal language theory notations and terminology used throughout the book.

1.1 General Mathematical Notations The notations are those provided by standard Latex and customary in mathematics. Set theory: ∈ denotes the membership (of an element to a set), ⊆ denotes the inclusion (not necessarily proper) and ⊂ denotes the strict inclusion; the union, intersection, and difference of two sets are denoted by ∪, ∩, −, respectively. (We do not use \ for the difference, because \ denotes the left quotient of languages.) The empty set is denoted by ∅, the power set of a set X is denoted by 2X , while the cardinality of a set X is denoted by card(X). A singleton set is often identified with its single element, and hence we also write a for {a}. Two sets X and Y are said to be incomparable if both X − Y and Y − X are non-empty. Sets of numbers: the set of natural numbers (zero included) is denoted by N, while the sets of integer, rational, and real numbers are denoted by Z, Q, R, respectively. The subsets of these sets consisting of strictly positive numbers are denoted by N+ , Z+ , Q+ , R+ , respectively.

1.2 Basic String Notions and Notation An alphabet is a finite nonempty set of abstract symbols. For an alphabet V we denote by V ∗ the set of all strings (we also say words) of symbols from V . The empty string is denoted by λ. The set of nonempty strings over V , that is V ∗ − {λ}, is denoted by V + . Each subset of V ∗ is called a language over V . A language which does not contain the empty string (hence being a subset of V + ) is said to be λ-free.

2

Basic Notation and Terminology

If x = x1 x2 , for some x1 , x2 ∈ V ∗ , then x1 is called a prefix of x and x2 is called a suffix of x; if x = x1 x2 x3 for some x1 , x2 , x3 ∈ V ∗ , then x2 is called a substring of x. The sets of all prefixes, suffixes, and substrings of a string x are denoted by Pref(x), Suf(x), and Sub(x), respectively. The sets of proper (that is, different from λ and from the string itself) prefixes, suffixes, and subwords of x are denoted by PPref(x), PSuf(x), and PSub(x), respectively. The length of a string x ∈ V ∗ (the number of occurrences of symbols from V in x) is denoted by |x|. The number of occurrences of a given symbol a ∈ V in x ∈ V ∗ is denoted by |x|a . If x ∈ V ∗ and U ⊆ V, then by |x|U we denote the length of the string obtained by erasing from x all symbols not in U, that is, |x|a . |x|U = a∈U ∗

For a language L ⊆ V , the set length(L) = {|x| | x ∈ L} is called the length set of L. The set of symbols occurring in a string x is denoted by alph(x). For a language L ⊆ V ∗ , we denote alph(L) = x∈L alph(x). The Parikh vector associated with a string x ∈ V ∗ with respect to the alphabet V = {a1 , . . . , an } is ΨV (x) = (|x|a1 , |x|a2 , . . . , |x|an ) (note that the ordering of the symbols from V is relevant). For L ⊆ V ∗ we define ΨV (L) = {ΨV (x) | x ∈ L}; the mapping ΨV : V ∗ −→ Nn is called the Parikh mapping associated with V . A set M of vectors in Nn , for some n ≥ 1, is said to be linear if there are m ≥ 0 and the vectors vi ∈ Nn , 0 ≤ i ≤ m, such that M = {v0 +

m

αi vi | α1 , . . . , αm ∈ N}.

i=1

A finite union of linear sets is said to be semilinear. A language L ⊆ V ∗ is semilinear if ΨV (L) is a semilinear set. The family of semilinear languages is denoted by SLIN .

1.3 Operations with Strings and Languages The operations of union, intersection, difference, and complement are defined for languages in the standard set-theoretical way. The concatenation of two languages L1 , L2 is L1 L2 = {xy | x ∈ L1 , y ∈ L2 }. We define further:


3

L0 = {λ}, Li+1 = LLi , i ≥ 0, ∞ Li (the Kleene ∗ -closure), L∗ = L+ =

i=0 ∞

Li (the Kleene + -closure).

i=1 ∗

∗

A mapping s : V −→ 2U , extended to s : V ∗ −→ 2U by s(λ) = {λ} and s(x1 x2 ) = s(x1 )s(x2 ), for x1 , x2 ∈ V ∗ , is called a substitution. If for all a ∈ V we have λ ∈ / s(a), then h is a λ-free substitution. If card(s(a)) = 1 for all a ∈ V , then s is called a morphism (we also say homomorphism). A morphism h : V ∗ −→ U ∗ is called a coding if h(a) ∈ U for each a ∈ V and a weak coding if h(a) ∈ U ∪ {λ} for each a ∈ V . If h : (V1 ∪ V2 )∗ −→ V1∗ is the morphism defined by h(a) = a for a ∈ V1 , and h(a) = λ otherwise, then we say that h is a projection (associated with V1 ) and we denote it by prV1 . ∗ For a morphism h : V ∗ −→ U ∗ , we define a mapping h−1 : U ∗ −→ 2V (and we call it an inverse morphism) by h−1 (w) = {x ∈ V ∗ | h(x) = w}, w ∈ U ∗ . The substitutions (hence also the morphisms and inverse morphisms) are extended to languages in the natural way. For x, y ∈ V ∗ we define their shuffle by x

y = {x1 y1 . . . xn yn | x = x1 . . . xn , y = y1 . . . yn , xi , yi ∈ V ∗ , 1 ≤ i ≤ n, n ≥ 1}.

The left quotient of a language L1 ⊆ V ∗ with respect to L2 ⊆ V ∗ is L2 \L1 = {w ∈ V ∗ | there is x ∈ L2 such that xw ∈ L1 }. The left derivative of a language L ⊆ V ∗ with respect to a string x ∈ V ∗ is

∂xl (L) = {w ∈ V ∗ | xw ∈ L}.

The right quotient and the right derivative are defined in a symmetric manner: L1 /L2 = {w ∈ V ∗ | there is x ∈ L2 such that wx ∈ L1 }, ∂xr (L) = {w ∈ V ∗ | wx ∈ L}. Let F be a family of languages and ◦ be an n-ary operation with languages from F . The family F is closed under ◦ if ◦(L1 , L2 , . . . , Ln ) ∈ F for any choice of the languages Li ∈ F, 1 ≤ i ≤ n. The family F is closed under substitution with languages from the family C if for any language L ⊆ V ∗ , L ∈ F, and ∗ any substitution s : V ∗ −→ 2U such that s(a) ∈ C for all a ∈ V , the language s(L) = x∈L s(x) still lies in F. If C = F , we simply say that F is closed under substitution.

4


A family of languages closed under (arbitrary) λ-free morphisms, inverse morphisms and intersection with regular languages is called (full) trio - known also as (cone) faithful cone. If a (full) trio is further closed under union, then it is called (full) semi-AFL. The abbreviation AFL comes from Abstract Family of Languages. A (full) semi-AFL closed under concatenation and Kleene (*-) +-closure is called a (full) AFL.

1.4 Chomsky Grammars A Chomsky grammar is a quadruple G = (N, T, S, P ), where N, T are disjoint alphabets, S ∈ N , and P is a finite subset of (N ∪ T )∗ N (N ∪ T )∗ × (N ∪ T )∗ . The alphabet N is called the nonterminal alphabet, T is the terminal alphabet, S is the axiom (start symbol), and P is the set of production rules of G. The rules (we also say productions) (u, v) of P are written in the form u → v. Note that |u|N ≥ 1. Sometimes, one uses to denote by VG the total alphabet of G, that is, VG = N ∪ T . For x, y ∈ (N ∪ T )∗ we write x =⇒G y iff x = x1 ux2 , y = x1 vx2 , for some x1 , x2 ∈ (N ∪ T )∗ and u → v ∈ P. One says that x directly derives y (with respect to G). When G is understood we write =⇒ instead of =⇒G . The reflexive closure of the relation =⇒ is denoted by =⇒+ , and the reflexive and transitive closure by =⇒∗ . Each string w ∈ (N ∪ T )∗ such that S =⇒∗G w is called a sentential form. The language generated by G, denoted by L(G), is defined by L(G) = {x ∈ T ∗ | S =⇒∗ x}. Two grammars G1 , G2 are called equivalent if L(G1 ) − {λ} = L(G2 ) − {λ} (the two languages coincide modulo the empty string). According to the form of their rules, the Chomsky grammars are classified as follows. A grammar G = (N, T, S, P ) is called: – –

– – – – –

length-increasing (one also says monotonous), if for all u → v ∈ P we have |u| ≤ |v|. context-sensitive, if each u → v ∈ P has u = u1 Au2 , v = u1 xu2 , for u1 , u2 ∈ (N ∪ T )∗ , A ∈ N, and x ∈ (N ∪ T )+ . (In length-increasing and context-sensitive grammars the production S → λ is allowed, provided that S does not appear in the right-hand members of rules in P .) context-free, if each production u → v ∈ P has u ∈ N . linear, if each rule u → v ∈ P has u ∈ N and v ∈ T ∗ ∪ T ∗ N T ∗ . right-linear, if each rule u → v ∈ P has u ∈ N and v ∈ T ∗ ∪ T ∗ N . left-linear, if each rule u → v ∈ P has u ∈ N and v ∈ T ∗ ∪ N T ∗ . regular, if each rule u → v ∈ P has u ∈ N and v ∈ T ∪ T N ∪ {λ}.


5

The arbitrary, length-increasing, context-free, and regular grammars are also said to be of type 0, type 1, type 2, and type 3, respectively. We denote by RE, LI, CS, CF, LIN, RLIN, LLIN, and REG the families of languages generated by arbitrary, length-increasing, context-sensitive, context-free, linear, right-linear, left-linear, and regular grammars, respectively (RE stands for recursively enumerable). By F IN we denote the family of finite languages, and by ARB the family of arbitrary languages. The following equalities and strict inclusions hold: F IN ⊂ REG = RLIN = LLIN ⊂ LIN ⊂ CF ⊂ CS = LI ⊂ RE ⊂ ARB. We call this the Chomsky hierarchy.

1.5 Decision Problems The goal of this section is to give an informal description of a decision problem and to mention the most common decision problems in formal language theory. Roughly speaking, a decision problem requires an output YES/NO to any of its instances. For example, “Is the natural number n prime?” is a decision problem; further, “Is 3 prime?” is an instance of the problem which is true while “Is 4 prime?” is a false instance of the same problem. A decision problem is (algorithmically/recursively) decidable if there exists an algorithm, which for any instance of the problem given as input, outputs YES or NO, provided that the input is true or not, respectively. The most common decision problems in formal language theory are: – – – – –

Emptiness: Is a given language empty? Finiteness: Is a given language a finite set? Membership: Does w ∈ L hold for a given word w and a language L? Inclusion: Does L1 ⊆ L2 hold for two given languages L1 and L2 ? Equivalence: Does L1 = L2 hold for two given languages L1 and L2 ?

Clearly, a decision problem is proved to be decidable if one provides an algorithm as above. Generally, a decision problem is proved to be undecidable by reducing it to a problem known to be undecidable. The following combinatorial problem, known as the Post Correspondence Problem (PCP), is undecidable. An instance of the PCP consists of an alphabet V with at least two letters and two lists of words over V u = (u1 , u2 , . . . , un )

and

v = (v1 , v2 , . . . , vn ).

The problem asks whether or not a sequence i1 , i2 , . . . , ik of positive integers exists, each between 1 and n, such that ui1 ui2 . . . uik = vi1 vi2 . . . vik . We do not give here further elements of formal language theory. They will be elaborated in the subsequent chapters. For the reader’s convenience, we end this section with a list of monographs and collective volumes directly or partially related to formal language theory.

6


1.6 Books on Formal Language Theory 1. A.V. Aho, J.D. Ullman, The Theory of Parsing, Translation, and Compiling, Prentice Hall, Englewood Cliffs, N.J., vol. I: 1971, vol. II: 1973. 2. A.V. Aho, J.D. Ullman, Principles of Compiler Design, Addison-Wesley, Reading, Mass., 1977. 3. I. Alexander, F.K. Hanna, Automata Theory: An Engineering Approach, Crane Russak, 1975. 4. J. Berstel, Transductions and Context-Free Languages, Teubner, Stuttgart, 1979. 5. R.V. Book, ed., Formal Language Theory. Perspectives and Open Problems, Academic Press, New York, 1980. 6. W. Brauer, Automatentheorie, B.G. Teubner, Stuttgart, 1984. 7. C. Choffrut, ed., Automata Networks, Lecture Notes in Computer Science, Springer-Verlag, Berlin, 1988. 8. D.I.A. Cohen, Computer Theory, 2nd edition, John Wiley, 1997. 9. E. Csuhaj-Varjú, J. Dassow, J. Kelemen, Gh. Păun, Grammar Systems. A Grammatical Approach to Distribution and Cooperation, Gordon and Breach, London, 1994. 10. J. Dassow, Gh. Păun, Regulated Rewriting in Formal Language Theory, Springer-Verlag, Berlin, Heidelberg, 1989. 11. J. Dassow, G. Rozenberg, A. Salomaa, eds., Developments in Language Theory, World Scientific, Singapore, 1995. 12. M.D. Davis, E.J. Weyuker, Computability, Complexity, and Languages, Academic Press, New York, 1983. 13. P.J. Denning, J.B. Dennis, J.E. Qualitz, Machines, Languages, and Computation, Prentice-Hall, Englewood Cliffs, N.J., 1978. 14. D.-Z. Du, K.-I Ko, Problem Solving in Automata, Languages and Complexity, John Wiley, 2001. 15. H. Ehrig, G. Engels, H-J. Kreowski, G. Rozenberg, eds., Handbook of Graph Grammars and Computing by Graph Transformation, World Scientific, Singapore, 1999. 16. S. Eilenberg, Automata, Languages, and Machines, Academic Press, New York, vol. A: 1974, vol. B: 1976. 17. E. Engeler, Formal Languages, Markham, Chicago, 1968. 18. Z. Ésik, C. Martín-Vide, V. Mitrana, eds., Recent Advances in Formal Languages and Applications, Springer-Verlag, Berlin, 2006. 19. K.S. Fu, Syntactic Pettern Recognition. Applications, Springer-Verlag, Heidelberg, 1977. 20. M.R. Garey, D.S. Johnson, Computers and Intractability. A Guide to the Theory of NP-completeness, W.H. Freeman, San Francisco, 1979. 21. F. Gécseg, Products of Automata, Springer-Verlag, Berlin, 1986. 22. F. Gécseg, I. Peak, Algebraic Theory of Automata, Akademiai Kiado, Budapest, 1972. 23. F. Gécseg, M. Steinby, Tree Automata, Akademiai Kiado, Budapest, 1984.


7

24. S. Ginsburg, The Mathematical Theory of Context-Free Languages, McGraw-Hill Book Comp., New York, 1966. 25. S. Ginsburg, Algebraic and Automata-Theoretic Properties of Formal Languages, North-Holland, Amsterdam, 1975. 26. A. Ginzburg, Algebraic Theory of Automata, Academic Press, New York, 1968. 27. M. Gross, A. Lentin, Notions sur les grammaires formelles, GauthierVillars, Paris, 1967. 28. M. Harrison, Introduction to Formal Language Theory, Addison-Wesley, Reading, Mass., 1978. 29. G.T. Herman, G. Rozenberg, Developmental Systems and Languages, North-Holland, Amsterdam, 1975. 30. J.E. Hopcroft, J.D. Ullman, Formal Languages and Their Relations to Automata, Addison-Wesley, Reading, Mass., 1969. 31. J.E. Hopcroft, J.D. Ullman, Introduction to Automata Theory, Languages, and Computing, Addison-Wesley, Reading, Mass., 1979. 32. J.E. Hopcroft, R. Motwani, J.D. Ullman, Introduction to Automata Theory, Languages, and Computation, Addison-Wesley, Boston, 2001. 33. M. Ito, ed., Words, Languages, and Combinatorics, World Scientific, Singapore, 1992. 34. M. Ito, Gh. Păun, S. Yu, eds., Words, Semigroups, and Transductions, World Scientific, Singapore, 2001. 35. J. Karhumäki, H.A. Maurer, Gh. Păun, G. Rozenberg, eds., Jewels are Forever, Springer-Verlag, Berlin, 1999. 36. D. Kelley, Automata and Formal Languages. An Introduction. PrenticeHall, New Jersey, 1995. 37. Z. Kohavi, Switching and Finite Automata Theory, McGraw-Hill Book Comp., New York, 1978. 38. D.C. Kozen, Automata and Computability, Springer-Verlag, New York, 1997. 39. W. Kuich, A. Salomaa, Semirings, Automata, Languages, Springer-Verlag, Berlin, Heidelberg, New York, 1986. 40. P. Linz, An Introduction to Formal Languages and Automata, D.C. Heath and Co., Lexington, Mass., 1990. 41. M. Lothaire, Combinatorics on Words, Addison-Wesley, Reading, Mass., 1983. 42. M. Lothaire, Algebraic Combinatorics on Words, Cambridge University Press, 1997. 43. C. Martín-Vide, V. Mitrana, eds., Where Mathematics, Computer Science, Linguistics, and Biology Meet, Kluwer, Dordrecht, 2000. 44. C. Martín-Vide, V. Mitrana, eds., Grammars and Automata for String Processing: From Mathematics and Computer Science to Biology, and Back, Taylor and Francis, London, 2002. 45. C. Martín-Vide, Gh. Păun, eds., Recent Topics in Mathematical and Computational Linguistics, Ed. Academiei, Bucureşti, 2000.

8


46. C. Martín-Vide, V. Mitrana, Gh. Păun, eds., Formal Languages and Applications, Springer-Verlag, Berlin, 2004. 47. H. Maurer, Theoretische Grundlagen der Programmiersprachen, Hochschultaschenbücher 404, Bibliographisches Inst., 1969. 48. A. Meduna, Automata and Languages, Springer-Verlag, London, 2000. 49. M. Minsky, Computation: Finite and Infinite Machines, Prentice-Hall, Englewood Cliffs, NJ, 1967. 50. J.N. Mordenson, D.S. Malik, Fuzzy Automata and Languages, Chapman & Hall/CRC, London, 2002. 51. A. Paz, Introduction to Probabilistic Automata, Academic Press, New York, 1971. 52. Gh. Păun, Recent Results and Problems in Formal Language Theory, The Scientific and Encyclopaedic Publ. House, Bucharest, 1984 (in Romanian). 53. Gh. Păun, ed., Mathematical Aspects of Natural and Formal Languages, World Scientific, Singapore, 1994. 54. Gh. Păun, ed., Mathematical Linguistics and Related Topics, The Publ. House of the Romanian Academy, Bucharest, 1995. 55. Gh. Păun, Marcus Contextual Grammars, Kluwer, Dordrecht, 1997. 56. Gh. Păun, Membrane Computing. An Introduction, Springer-Verlag, Berlin, 2002. 57. Gh. Păun, G. Rozenberg, A. Salomaa, DNA Computing. New Computing Paradigms, Springer-Verlag, Berlin, 1998. 58. Gh. Păun, G. Rozenberg, A. Salomaa, eds., Current Trends in Theoretical Computer Science. Entering the 21st Century, World Scientific, Singapore, 2001. 59. Gh. Păun, A. Salomaa, eds., New Trends in Formal Languages: Control, Cooperation, Combinatorics, Lecture Notes in Computer Science 1218, Springer-Verlag, Berlin, 1997. 60. Gh. Păun, A. Salomaa, eds., Grammatical Models of Multi-Agent Systems, Gordon and Breach, London, 1999. 61. J.E. Pin, Varieties of Formal Languages, Plenum Press, Oxford, 1986. 62. G.E. Revesz, Introduction to Formal Languages, McGraw-Hill Book Comp., New York, 1983. 63. G. Rozenberg, A. Salomaa, The Mathematical Theory of L Systems, Academic Press, New York, 1980. 64. G. Rozenberg, A. Salomaa, Cornerstones of Undecidability, Prentice Hall, New York, 1994. 65. G. Rozenberg, A. Salomaa, eds., Developments in Language Theory, World Scientific, Singapore, 1994. 66. G. Rozenberg, A. Salomaa, eds., Handbook of Formal Languages, SpringerVerlag, Berlin, 3 volumes, 1997. 67. A. Salomaa, Theory of Automata, Pergamon, Oxford, 1969. 68. A. Salomaa, Formal Languages, Academic Press, New York, London, 1973. 69. A. Salomaa, Jewels of Formal Language Theory, Computer Science Press, Rockville, 1981.


9

70. A. Salomaa, Computation and Automata, Cambridge Univ. Press, Cambridge, 1985. 71. A. Salomaa, M. Soittola, Automata-Theoretic Aspects of Formal Power Series, Springer-Verlag, Berlin, New York, 1978. 72. A. Salomaa, D. Wood, S. Yu, eds., A Half-Century of Automata Theory, World Scientific, Singapore, 2001. 73. D. Simovici, R.L. Tenney, Theory of Formal Languages With Applications, World Scientific, Singapore, 1999. 74. S. Sippu, E. Soisalon-Soininen, Parsing Theory. Vol. I: Languages and Parsing, Springer-Verlag, Berlin, Heidelberg, 1988. 75. M. Sipser, Introduction to the Theory of Computation, PWS Publishing Company, Boston, 1997. 76. H.J. Shyr, Free Monoids and Languages, Hon Min Book Comp., Taichung, 1991. 77. R.G. Taylor, Models of Computation and Formal Languages, Oxford University Press, 1998. 78. D. Wood, Grammar and L Forms. An Introduction, Springer-Verlag, Berlin, 1980 (Lecture Notes in Computer Science, 91). 79. D. Wood, Theory of Computation, Harper and Row, New York, 1987.

2 Open Problems on Partial Words∗ Francine Blanchet-Sadri Department of Computer Science, University of North Carolina P.O. Box 26170, Greensboro, NC 27402–6170, USA [email protected]

2.1 Introduction Combinatorics on words, or sequences or strings of symbols over a finite alphabet, is a rather new field although the first papers were published at the beginning of the 20th century [120, 121]. The interest in the study of combinatorics on words has been increasing since it finds applications in various research areas of mathematics, computer science, and biology where the data can be easily represented as words over some alphabet. Such areas may be concerned with algorithms on strings [38, 48, 50, 51, 52, 69, 72, 84, 102, 118], semigroups, automata and languages [2, 45, 55, 75, 82, 92, 93], molecular genetics [78], or codes [5, 73, 79]. Motivated by molecular biology of nucleic acids, Berstel and Boasson introduced in 1999 the notion of partial words which are sequences that may contain a number of “do not know” symbols or “holes” [4]. DNA molecules are the carriers of the genetic information in almost all organisms. Let us look into the structure of such a molecule. A single stranded DNA molecule or a DNA strand may be viewed as a sequence over the alphabet consisting of the four nucleotides: a (adenine), c (cytosine), g (guanine), and t (thymine). Each strand has two different ends: the 3 end, and the 5 end. The familiar double helix of DNA, which was discovered by Watson and Crick, arises by the bonding of a strand in the 5 − 3 direction with another strand in the 3 − 5 direction with the restriction that adenine bonds with thymine, and cytosine bonds with guanine. Such a bonding gives rise to a double stranded DNA molecule as in the figure 5 - c c a c c t c g a c c c t c - 3 3 - g g t g g a g c t g g g a g - 5 Because of Watson-Crick’s complementarity (a bonds to only t, and c bonds only to g), we can view double stranded DNA sequences as single stranded ∗ This material is based upon work supported by the National Science Foundation under Grant Nos. CCF–0207673 and DMS–0452020.

F. Blanchet-Sadri: Open Problems on Partial Words, Studies in Computational Intelligence (SCI) 113, 11–58 (2008) c Springer-Verlag Berlin Heidelberg 2008 www.springerlink.com

12

Francine Blanchet-Sadri

strings by keeping the strand in the 5 − 3 direction. The molecule in the example above can be viewed as 5 - ccacctcgaccctc - 3 or simply as ccacctcgaccctc, a word over the alphabet {a, c, g, t}. However bonding is not always perfect in nature as in the figure 5 - c c a c c t c g a c c c t c - 3 3 - g g t t g a g c c g g g a g - 5 where there is an occurrence of c paired with t, and an occurrence of a paired with c. In such a case, we can view the molecule as 5 − cca ctcg ccctc − 3 or as cca ctcg ccctc, where the ’s stand for “do not know” symbols also called “holes”. Thus, the latter example gives rise to a partial word with two holes over the alphabet {a, c, g, t}. Processes in molecular biology can be seen as operations on DNA sequences [72, 112]. If a set of DNA molecules fulfilling a certain property has changed a little bit after some time or under some influence, it is important to know whether the desired property still holds [91]. Several interesting combinatorial properties of partial words have been investigated and connections have been made with problems in graph theory and number theory, in particular, with problems concerning primitive sets of integers [23, 24], lattices [23, 24], partitions of integers and their generalizations [14], chromatic polynomials [23], Sudoku games [107], vertex connectivity in graphs [12, 29], etc. Partial words are useful in a new generation of pattern matching algorithms that search for local similarities between sequences. In this area, they are called “spaced seeds” and a lot of work has been dedicated to their influence on the algorithms’ performance [40, 66, 83, 97, 103, 104]. Partial words have the potential for impacts in bio-inspired computing where they have been considered, in particular, for finding good encodings for DNA computations [90]. We provide here a few bibliographic remarks. Lothaire’s first book Combinatorics on Words appeared in 1983 [92], while recent developments culminated in a second book Algebraic Combinatorics on Words which appeared in 2002 [93] and in a third book which appeared in 2005 [94]. Several books have appeared quite recently that emphasize connections of combinatorics on words to several research areas. We mention the book of Allouche and Shallit where the emphasis is on automata theory [2], the book of Crochemore and Rytter where the emphasis is on string algorithms [52], the book of Gusfield where the emphasis is on algorithms related to biology [72], the book of de Luca and Varrichio where the emphasis is on algebra [55], and finally the book of Blanchet-Sadri where the emphasis is on partial words [10]. Research in combinatorics on partial words is underway where there are several open problems that lay unexplored. After reviewing basic concepts on words and partial words in Section 2.2, we will discuss some of these open

2 Open Problems on Partial Words

13

problems which we have divided into sections: 2.3–2.5 study extensions to partial words of three basic classical results on periodicity of words: The theorem of Fine and Wilf which considers the simultaneous occurrence of different periods in one word [67], the critical factorization theorem which relates local and global periodicity of words [43], and a theorem of Guibas and Odlyzko which gives the structure of the set of periods of a word [71]. Section 2.6 deals with the two word properties of primitiveness and borderedness and is concerned, in particular, with the counting of primitive and unbordered partial words. Section 2.7 solves some equations on partial words. Here the notion of “equality” is replaced by that of “compatibility ”. Section 2.8 studies the concept of unavoidable set of partial words, while Section 2.9 develops square- and overlap-freeness of partial words. Finally, Section 2.10 discusses some other open problems related to codes of partial words, punctured languages, and tiling periodicity.

2.2 Preliminaries This section is devoted to reviewing basic concepts on words and partial words. 2.2.1 Words Let A be a nonempty finite set of symbols called an alphabet. Symbols in A are called letters and any finite sequence of letters is called a word over A. The empty word, that is, the word containing no letter, is denoted by ε. For any word u over A, |u| denotes the number of letters occurring in u and is called the length of u. In particular, |ε| = 0. The set of all words over A is denoted by A∗ . If we define the operation of two words u and v of A∗ by juxtaposition (or concatenation), then A∗ is a monoid with identity ε. We call A+ = A∗ \ {ε} the free semigroup generated by A and A∗ the free monoid generated by A. The set A∗ can also be viewed as n≥0 An where A0 = {ε} and An is the set of all words of length n over A. A word of length n over A can be defined by a total function u : {0, . . . , n− 1} → A and is usually represented as u = a0 a1 . . . an−1 with ai ∈ A. A period of u is a positive integer p such that ai = ai+p for 0 ≤ i < n − p. For a word u, the powers of u are defined inductively by u0 = ε and, for any i ≥ 1, ui = uui−1 . The set of symbols occurring in a word u is denoted by α(u). The reversal of u, denoted by rev(u), is defined as follows: If u = ε, then rev(ε) = ε, and if u = a0 a1 . . . an−1 , then rev(u) = an−1 . . . a1 a0 . A word u is a factor of the word v if there exist words x, y such that v = xuy. The factor u is called proper if u = ε and u = v. The word u is a prefix (respectively, suffix) of v if x = ε (respectively, y = ε). A nonempty word u is primitive if there exists no word v such that u = v i with i ≥ 2. Note the fact that the empty word is not primitive. If u is a nonempty word, then there exist a unique primitive word v and a unique positive integer i such that u = v i .

14


2.2.2 Partial Words A partial word u of length n over A is a partial function u : {0, . . . , n−1} → A. For 0 ≤ i < n, if u(i) is defined, then i belongs to the domain of u, denoted by i ∈ D(u), otherwise i belongs to the set of holes of u, denoted by i ∈ H(u). A word over A is a partial word over A with an empty set of holes (we sometimes refer to words as full words). The length of u or n is denoted by |u|. If u is a partial word of length n over A, then the companion of u, denoted by u , is the total function u : {0, . . . , n − 1} → A ∪ { } defined by u (i) =

u(i) if i ∈ D(u)

otherwise

The bijectivity of the map u → u allows us to define for partial words concepts such as concatenation, powers, etc in a trivial way. The word u = abb bbcb is the companion of the partial word u of length |u| = 8 where D(u) = {0, 1, 2, 4, 5, 6, 7} and H(u) = {3}. For convenience, we will refer to a partial word over A as a word over the enlarged alphabet A = A ∪ { }, where the additional symbol plays the special role of a “do not know” symbol or “hole”. This allows us to say for example “the partial word aba aa ” instead of “the partial word with companion aba aa ”. The set of all partial words over A with an arbitrary number of holes is denoted by A∗ which is a monoid under the operation of concatenation where ε serves as the identity. A (strong) period of a partial word u over A is a positive integer p such that u(i) = u(j) whenever i, j ∈ D(u) and i ≡ j mod p. In such a case, we call u (strongly) p-periodic. Similarly, a weak period of u is a positive integer p such that u(i) = u(i + p) whenever i, i + p ∈ D(u). In such a case, we call u weakly p-periodic. The partial word abb bbcbb is weakly 3-periodic but is not strongly 3-periodic. The latter shows a difference between partial words and full words since every weakly p-periodic full word is strongly p-periodic. Another difference worth noting is the fact that even if the length of a partial word u is a multiple of a weak period of u, then u is not necessarily a power of a shorter partial word. The minimum period of u is denoted by p(u), and the minimum weak period by p (u). The set of all periods (respectively, weak periods) of u is denoted by P(u) (respectively, P (u)). For a partial word u, positive integer p and integer 0 ≤ i < p, define ui,p = u(i)u(i + p)u(i + 2p) . . . u(i + jp) where j is the largest nonnegative integer such that i + jp < |u|. Then u is (strongly) p-periodic if and only if ui,p is (strongly) 1-periodic for all 0 ≤ i < p, and u is weakly p-periodic if and only if ui,p is weakly 1-periodic for all 0 ≤ i < p. Strongly 1-periodic partial words as well as the full factors of weakly 1-periodic partial words are over a singleton alphabet. If u and v are two partial words of equal length, then u is said to be contained in v, denoted by u ⊂ v, if all elements in D(u) are in D(v) and


15

u(i) = v(i) for all i ∈ D(u). The order u ⊂ v on partial words is obtained when we let < a and a ≤ a for all a ∈ A. For example, a b ⊂ a

b and a b ⊂ a ab, while a b ⊂ a bb. A partial word u is primitive if there exists no word v such that u ⊂ v i with i ≥ 2. Note that if v is primitive and v ⊂ u, then u is primitive as well. It was shown in [9] that if u is a nonempty partial word, then there exist a primitive word v and a positive integer i such that u ⊂ v i . However uniqueness does not hold as seen with the partial word u = a where u ⊂ a2 and u ⊂ ba for distinct letters a, b. Partial words u and v are compatible, denoted by u ↑ v, if there exists a partial word w such that u ⊂ w and v ⊂ w. In other words, u(i) = v(i) for every i ∈ D(u) ∩ D(v). Note that for full words, the notion of compatibility is simply that of equality. For example, a b a ↑ a

cbb but a bbc ↑ bb c . In the rest of this section, we discuss commutativity and conjugacy in the context of partial words. Let us start with commutativity. The case of full words is well known and is stated in the following theorem. Theorem 1. Let x and y be nonempty words. Then xy = yx if and only if there exists a word z such that x = z m and y = z n for some integers m, n. For nonempty partial words x and y, if there exist a word z and integers m, n such that x ⊂ z m and y ⊂ z n , then xy ⊂ z m+n , yx ⊂ z m+n , and xy ↑ yx. The converse is not true in general: if x = bb and y = abb , then xy = bbabb ↑ abb

bb = yx but no desired z exists. Let us first examine the case of one hole. Theorem 2. [4] Let x, y be nonempty partial words such that xy has at most one hole. If xy ↑ yx, then there exists a word z such that x ⊂ z m and y ⊂ z n for some integers m, n. Now, for the case of an arbitrary number of holes, let k, l be positive integers satisfying k ≤ l. For 0 ≤ i < k + l, define seqk,l (i) = (i0 , i1 , i2 , . . . , in , in+1 ) where i0 = i = in+1 ; for 1 ≤ j ≤ n, ij = i; and for 1 ≤ j ≤ n + 1, ij =

ij−1 + k if ij−1 < l ij−1 − l otherwise

For example, seq6,8 (0) = (0, 6, 12, 4, 10, 2, 8, 0).

16


Definition 1. [11] Let k, l be positive integers satisfying k ≤ l and let z be a partial word of length k + l. We say that z is (k, l)-special if there exists 0 ≤ i < k such that seqk,l (i) = (i0 , i1 , i2 , . . . , in , in+1 ) contains (at least) two positions that are holes of z while z(i0 )z(i1 )z(i2 ) . . . z(in+1 ) is not 1-periodic. Example 1. Let z = cbca

cbc caca, and let k = 6 and l = 8 so |z| = k + l. We wish to determine if z is (6, 8)-special. We already calculated seq6,8 (0) and z(0) z(6) z(12) z(4) z(10) z(2) z(8) z(0) c c c

c c c c This sequence does not satisfy the definition, and so we must continue with calculating seq6,8 (1) = (1, 7, 13, 5, 11, 3, 9, 1). The corresponding letter sequence is z(1) z(7) z(13) z(5) z(11) z(3) z(9) z(1) b b a

a a

b Here we have two positions in the sequence which are holes, and the sequence is not 1-periodic. Hence, z is (6, 8)-special. Under the extra condition that xy is not (|x|, |y|)-special, an extension of Theorem 2 holds when xy has an arbitrary number of holes. Theorem 3. [11] Let x, y be nonempty partial words such that |x| ≤ |y|. If xy ↑ yx and xy is not (|x|, |y|)-special, then there exists a word z such that x ⊂ z m and y ⊂ z n for some integers m, n. Now, let us discuss conjugacy. Again, the case of full words is well known. Theorem 4. Let x, y, z (x = ε and y = ε) be words such that xz = zy. Then x = uv, y = vu, and z = (uv)n u for some words u, v and integer n ≥ 0. For example, if x = abcda, y = daabc, and z = abc, then xz = zy because (abcda)(abc) = (abc)(daabc). Here u = abc, v = da, and n = 0. The case of partial words is more subtle. Theorem 5. [28] Let x, y, z be partial words with x, y nonempty. If xz ↑ zy and xz ∨ zy is |x|-periodic, then there exist words u, v such that x ⊂ uv, y ⊂ vu, and z ⊂ (uv)n u for some integer n ≥ 0. To illustrate Theorem 5, let x = ba, y = b , and z = b ab

. Then we have xz = b a b a b zy = b a b b xz ∨ zy = b b a b a b b It is clear that xz ↑ zy and xz ∨ zy is |x|-periodic. Putting u = bb and v = a, we can verify that the conclusion does indeed hold.


17

Corollary 1. [28] Let x, y be nonempty partial words, and let z be a full word. If xz ↑ zy, then there exist words u, v such that x ⊂ uv, y ⊂ vu, and z ⊂ (uv)n u for some integer n ≥ 0. Note that the above Corollary does not necessarily hold if z is not full even if x, y are full. The partial words x = a, y = b, and z = bb provide a counterexample. Two conjugacy theorems follow without any restriction on z. Theorem 6. [13] Let x, y and z be partial words such that |x| = |y| > 0. Then xz ↑ zy if and only if xzy is weakly |x|-periodic. Theorem 7. [13] Let x, y and z be partial words such that |x| = |y| > 0. Then the following hold: 1. If xz ↑ zy, then xz and zy are weakly |x|-periodic. |z| > 0, then xz ↑ zy. 2. If xz and zy are weakly |x|-periodic and |x| |z| The assumption |x| > 0 is necessary. To see this, consider x = aa, y = ba and z = a. Here, xz and zy are weakly |x|-periodic, but xz ↑ zy.

2.3 Periods in Partial Words Notions and techniques related to periodic structures in words find important applications in virtually every area of theoretical and applied computer science, notably in text processing [51, 52], data compression [49, 119, 123], coding [5], computational biology [39, 72, 100, 112], string searching and pattern matching algorithms [38, 50, 51, 52, 69, 72, 84, 102]. Repeated patterns and related phenomena in words have played over the years a central role in the development of combinatorics on words, and have been highly valuable tools for the design and analysis of algorithms [45, 92, 93, 94]. In many practical applications, such as DNA sequence analysis, repetitions admit a certain variation between copies of the repeated pattern because of errors due to mutation, experiments, etc. Approximate repeated patterns, or repetitions where errors are allowed, are playing a central role in different variants of string searching and pattern matching problems [85, 86, 87, 88, 111]. Partial words have acquired great importance in this context. The notion of period is central in combinatorics on words and there are many fundamental results on periods of words. Among them is the well known periodicity result of Fine and Wilf [67] which intuitively determines how far two periodic events have to match in order to guarantee a common period. More precisely, any word u having two periods p, q and length at least p + q − gcd(p, q) has also the greatest common divisor of p and q, gcd(p, q), as a period. The bound p + q − gcd(p, q) is optimal since counterexamples can be

18


provided for shorter lengths, that is, there exists an optimal word of length p + q − gcd(p, q) − 1 having p and q as periods but not having gcd(p, q) as period [45]. Extensions of Fine and Wilf’s result to more than two periods are given in [42, 47, 80, 122]. For instance, in [47], Constantinescu and Ilie give an extension for an arbitrary number of periods and prove that their bounds are optimal. Fine and Wilf’s result has been generalized to partial words in two ways: •

First, any partial word u with h holes and having two weak periods p, q and length at least the so-denoted l(h, p, q) has also strong period gcd(p, q) provided u satisfies the condition of not being (h, p, q)-special (this concept will be defined below). This extension was done for one hole by Berstel and Boasson where the class of (1, p, q)-special partial words is empty [4]; for two or three holes by Blanchet-Sadri and Hegstrom [25]; and for an arbitrary number of holes by Blanchet-Sadri [8]. Elegant closed formulas for the bounds l(h, p, q) were given and shown to be optimal. • Second, any partial word u with h holes and having two strong periods p, q and length at least the so-denoted L(h, p, q) has also strong period gcd(p, q). The study of the bounds L(h, p, q) was initiated by Shur and Gamzova [114]. In particular, they gave a closed formula for L(h, p, q) in the case where h = 2 (the cases where h = 0 or h = 1 are implied by the above mentioned results). In [12], Blanchet-Sadri, Bal and Sisodia gave closed formulas for the optimal bounds L(h, p, q) in the case where p = 2 and also in the case where q is “large”. In addition, they gave upper bounds when q is “small” and h = 3, 4, 5 or 6. Their proofs are based on connectivity in a graph G(p,q) (u) associated with a given p- and q-periodic partial word u. More recently, in [29], Blanchet-Sadri, Mandel and Sisodia pursue by studying two types of vertex connectivity on G(p,q) (u): the so-called modified degree connectivity and r-set connectivity where r = q mod p. As a result, they give an efficient algorithm for computing L(h, p, q), and manage to give closed formulas in several cases including the h = 3 and h = 4 cases.

In this section, we discuss in details the two ways Fine and Wilf’s periodicity result has been extended to partial words: Section 2.3.1 discusses the weak periodicity generalizations and Section 2.3.2 the strong periodicity generalizations. For easy reference, we recall Fine and Wilf’s result. Theorem 8. [67] Let p and q be positive integers. If a full word u is p-periodic and q-periodic and |u| ≥ p + q − gcd(p, q), then u is gcd(p, q)-periodic. 2.3.1 Weak Periodicity In this section, we review the generalizations related to weak periodicity. We first recall Berstel and Boasson’s result for partial words with exactly one hole where the bound p + q is optimal.


19

Theorem 9. [4] Let p and q be positive integers satisfying p < q. Let u be a partial word with one hole. If u is weakly p-periodic and weakly q-periodic and |u| ≥ l(1, p, q) = p + q, then u is strongly gcd(p, q)-periodic. When we discuss partial words with h ≥ 2 holes, we need the extra assumption of u not being (h, p, q)-special for a similar result to hold true. Indeed, if p and q are positive integers satisfying p < q and gcd(p, q) = 1, then the infinite sequence (abp−1 bq−p−1 bn )n>0 consists of (2, p, q)-special partial words with two holes that are weakly p-periodic and weakly q-periodic but not gcd(p, q)-periodic. In order to define the concept of (h, p, q)-speciality, note that a partial word u that is weakly p-periodic and weakly q-periodic can be represented as a 2-dimensional structure. Consider for example the partial word w = ababa

bab bb bbbbbbbbb where p = 2 and q = 5. The array looks like:

u(0) u(5) u(10) u(15) u(20) u(2) u(7) u(12) u(17) u(22) u(4) u(9) u(14) u(19) u(1) u(6) u(11) u(16) u(21) u(3) u(8) u(13) u(18) u(23) and its corresponding array of symbols looks like:

a a a b bb

bbb

bbb a b

bb bbb

In general, if gcd(p, q) = d, we get d arrays. Each of these arrays is associated with a subgraph G = (V, E) of G(p,q) (u) as follows: V is the subset of D(u) comprising the defined positions of u within the array, and E = Ep ∪ Eq where Ep = {{i, i − p} | i, i − p ∈ V } and Eq = {{i, i − q} | i, i − q ∈ V }. For 0 ≤ j < gcd(p, q), the subgraph of G(p,q) (u) corresponding to D(u) ∩ {i | i ≥ 0 and i ≡ j mod gcd(p, q)} will be denoted by Gj(p,q) (u). Whenever gcd(p, q) = 1, G0(p,q) (u) is just G(p,q) (u). Referring to the partial word w above, the graph G(2,5) (w) is disconnected (w is (5, 2, 5)-special). Here, the ’s isolate the a’s from the b’s. We now define the concept of speciality.

20


Definition 2. [8] Let p and q be positive integers satisfying p < q, and let h be a nonnegative integer. Let h ( 2 + 1)(p + q) − gcd(p, q) if h is even l(h, p, q) = ( h2 + 1)(p + q) otherwise Let u be a partial word with h holes of length at least l(h, p, q). Then u is (h, p, q)-special if Gj(p,q) (u) is disconnected for some 0 ≤ j < gcd(p, q). It turns out that the bound l(h, p, q) is optimal for a number of holes h. Theorem 10. [8] Let p and q be positive integers satisfying p < q, and let u be a non (h, p, q)special partial word with h holes. If u is weakly p-periodic and weakly q-periodic and |u| ≥ l(h, p, q), then u is strongly gcd(p, q)-periodic. In [33], progress was made towards allowing an arbitrary number of holes and an arbitrary number of weak periods. There, the authors proved that any partial word u with h holes and having weak periods p1 , . . . , pn and length at least the so-denoted l(h, p1 , . . . , pn ) has also strong period gcd(p1 , . . . , pn ) provided u satisfies some criteria. In addition to speciality, they discovered that the concepts of intractable period sets and interference between periods play a role. Open problem 1 Give an algorithm which given a number of holes h and weak periods p1 , . . . , pn , computes the optimal bound l(h, p1 , . . . , pn ) and an optimal partial word for that bound (a partial word u with h holes of length l(h, p1 , . . . , pn ) − 1 is optimal for the bound l(h, p1 , . . . , pn ) if p1 , . . . , pn are weak periods of u but gcd(p1 , . . . , pn ) is not a strong period of u). Open problem 2 Give closed formulas for the bounds l(h, p1 , . . . , pn ). The optimality proof will probably be based on results of graphs associated with bounds and tuples of weak periods. 2.3.2 Strong Periodicity In this section, we review the generalizations related to strong periodicity. There exists an integer L such that if a partial word u with h holes has strong periods p, q satisfying p < q and |u| ≥ L, then u has strong period gcd(p, q) (L(h, p, q) is the smallest such integer L) [115]. Recall that L(0, p, q) = p + q − gcd(p, q). The following result is a direct consequence of Berstel and Boasson’s result. Theorem 11. [4] The equality L(1, p, q) = p + q holds. For h = 2, 3 or 4, we have the following results.


21

Theorem 12. [114, 115] The equality L(2, p, q) = 2p + q − gcd(p, q) holds. Theorem 13. [29] The following equality holds: ⎧ p ⎪ ⎨2q + p if q − p < 2 L(3, p, q) = 4p if p2 < q − p < p ⎪ ⎩ 2p + q if p < q − p Theorem 14. [29] The following equality holds: ⎧ p ⎪ ⎨q + 3p − gcd(p, q) if q − p < 2 L(4, p, q) = q + 3p if p2 < q − p < p ⎪ ⎩ q + 3p − gcd(p, q) if p < q − p Other results follow. Theorem 15. [12, 113, 114, 115] The equality L(h, 2, q) = (2 hq + 1)q + h mod q + 1 holds. Setting h = nq + r where 0 ≤ r < q, L(h, 2, q) = (2n + 1)q + r + 1. Consider the word u = r w( q w)n where w is the unique full word of length q having periods 2 and q but not having period 1. Note that u is an optimal word for the bound L(h, 2, q). Indeed, |u| = (2n + 1)q + r, u has h holes, and since w is not 1-periodic, we also have that u is not strongly 1-periodic. It is easy to show that u is strongly 2- and q-periodic. In [114], the authors proved that if q > p ≥ 3 and gcd(p, q) = 1 and h is large enough, then pq p+q−2 (h

+ 1) ≤ L(h, p, q)
4. Any partial word with h holes and having n strong periods p1 , . . . , pn and length at least the so-denoted L(h, p1 , . . . , pn ) has also gcd(p1 , . . . , pn ) as a strong period. Open problem 4 Give an algorithm which given a number of holes h and strong periods p1 , . . . , pn , computes the optimal bound L(h, p1 , . . . , pn ) and an optimal partial word for that bound (a partial word u with h holes of length L(h, p1 , . . . , pn ) − 1 is optimal for the bound L(h, p1 , . . . , pn ) if p1 , . . . , pn are strong periods of u but gcd(p1 , . . . , pn ) is not a strong period of u). Open problem 5 Give closed formulas for the bounds L(h, p1 , . . . , pn ).

22


2.4 Critical Factorizations of Partial words Results concerning periodicity include the well known and fundamental critical factorization theorem, of which several versions exist [43, 45, 60, 61, 59, 92, 93]. It intuitively states that the minimal period (or global period) of a word of length at least two is always locally detectable in at least one position of the word resulting in a corresponding critical factorization. More specifically, given a word w and nonempty words u, v satisfying w = uv, the minimal local period associated to the factorization (u, v) is the length of the shortest square at position |u| − 1. It is easy to see that no minimal local period is longer than the global period of the word. The critical factorization theorem shows that critical factorizations are unavoidable. Indeed, for any word, there is always a factorization whose minimal local period is equal to the global period of the word. More precisely, we consider a word a0 a1 . . . an−1 and, for any integer i (0 ≤ i < n−1), we look at the shortest repetition (a square) centered in this position, that is, we look at the shortest (virtual) suffix of a0 a1 . . . ai which is also a (virtual) prefix of ai+1 ai+2 . . . an−1 . The minimal local period at position i is defined as the length of this shortest square. The critical factorization theorem states, roughly speaking, that the global period of a0 a1 . . . an−1 is simply the maximum among all minimal local periods. As an example, consider the word w = babbaab with global period 6. The minimal local periods of w are 2, 3, 1, 6, 1 and 3 which means that the factorization (babb, aab) is critical. Crochemore and Perrin showed that a critical factorization can be found very efficiently from the computation of the maximal suffixes of the word with respect to two total orderings on words: the lexicographic ordering related to a fixed total ordering on the alphabet l , and the lexicographic ordering obtained by reversing the order of letters in the alphabet r [50]. If v denotes the maximal suffix of w with respect to l and v the maximal suffix of w with respect to r , then let u, u be such that w = uv = u v . The factorization (u, v) turns out to be critical when |v| ≤ |v |, and the factorization (u , v ) is critical when |v| > |v |. There exist linear time (in the length of w) algorithms for such computations [50, 51, 101] (the latter two use the suffix tree construction). Returning to the example above, order the letters of the alphabet by a ≺ b. Then the maximal suffix with respect to l is v = bbaab and with respect to r is v = aab. Since |v| > |v |, the factorization (u , v ) = (babb, aab) of w is critical. In [22], Blanchet-Sadri and Duncan extended the critical factorization theorem to partial words with one hole. In this case, the concept of local period, which characterizes a local periodic structure at each position of the word, is defined as follows. Definition 3. [22] Let w be a nonempty partial word. A positive integer p is called a local period of w at position i if there exist partial words u, v, x, y such that u, v = ε, w = uv, |u| = i + 1, |x| = p, x ↑ y, and such that one of the following conditions holds for some partial words r, s:


1. u = rx 2. x = ru 3. u = rx 4. x = ru

and and and and

v v y y

= ys = ys = vs = vs

23

(internal square), (left-external square if r = ε), (right-external square if s = ε), (left- and right-external square if r, s = ε).

In this context, a factorization is called critical if its minimal local period is equal to the minimal weak period of the partial word. As an example, consider the partial word with one hole w = ba baab with minimal weak period 3. The minimal local periods of w are 2 (left-external square), 1 (internal square), 1 (internal square), 3 (internal square), 1 (internal square) and 3 (right-external square), and both (ba b, aab) and (ba baa, b) are critical. It turns out that for partial words, critical factorizations may be avoidable. Indeed, the partial word babdaab has no critical factorization. The class of the so-called special partial words with one hole has been described that possibly avoid critical factorizations. Refining the method based on the maximal suffixes with respect to the lexicographic/ reverse lexicographic orderings leads to a version of the critical factorization theorem for the nonspecial partial words with one hole whose proof provides an efficient algorithm which, given a partial word with one hole, outputs a critical factorization when one exists or outputs “no such factorization exists”. In [35], Blanchet-Sadri and Wetzler further investigated the relationship between local and global periodicity of partial words: (1) They extended the critical factorization theorem to partial words with an arbitrary number of holes; (2) They characterized precisely the class of partial words that do not admit critical factorizations; and (3) They developed an efficient algorithm which computes a critical factorization when one exists. Some open problems related to the critical factorization theorem follow. Open problem 6 Discover some good criterion for the existence of a critical factorization of an unbordered partial word defined as follows: A nonempty partial word u is unbordered if no nonempty partial words x, v, w exist such that u ⊂ xv and u ⊂ wx. Open problem 7 In the framework of partial words, study the periodicity theorem on words, which has strong analogies with the critical factorization theorem, that was derived in [102]. In [62], the authors present an O(n) time algorithm for computing all local periods of a given word of length n, assuming a constant-size alphabet. This subsumes (but is substantially more powerful than) the computation of the global period of the word and the computation of a critical factorization. Their method consists of two parts: (1) They show how to compute all internal minimal squares; and (2) They show how to compute left- and right-external minimal squares, in particular for those positions for which no internal square has been found.

24


Open problem 8 Find the time complexity for the computation of all the local periods of a given partial word. Now, consider the language CF = {w | w is a partial word over {a, b} that has a critical factorization} What is the position of CF in the Chomsky hierarchy? It has been proved that CF is a context sensitive language that is not regular. The question whether or not CF is context-free remains open. Theorem 16. [36] The language CF is not regular. Theorem 17. [21] The language CF is context sensitive. Open problem 9 Is the language CF context-free?

2.5 Correlations of Partial Words In [71], Guibas and Odlyzko consider the period sets of words of length n over a finite alphabet, and specific representations of them, (auto)correlations, which are binary vectors of length n indicating the periods. Among the possible 2n bit vectors, only a small subset are valid correlations. There, they provide characterizations of correlations, asymptotic bounds on their number, and a recurrence for the population size of a correlation, that is, the number of words sharing a given correlation. In [108], Rivals and Rahmann show that there is redundancy in period sets and introduce the notion of an irreducible period set. They prove that Γn , the set of all correlations of length n, is a lattice under set inclusion and does not satisfy the Jordan-Dedekind condition. They propose the first efficient enumeration algorithm for Γn and improve upon the previously known asymptotic lower bounds on the cardinality of Γn . Finally, they provide a new recurrence to compute the number of words sharing a given period set, and exhibit an algorithm to sample uniformly period sets through irreducible period sets. In [24], the combinatorics of possible sets of periods and weak periods of partial words were studied in a similar way. There, the notions of binary and ternary correlations were introduced, which are binary and ternary vectors indicating the periods and weak periods of partial words. Extending the result of Guibas and Odlyzko, Blanchet-Sadri, Gafni and Wilson characterized precisely which binary and ternary vectors represent the period and weak period sets of partial words and proved that all valid correlations may be taken over the binary alphabet (the one-hole case was proved earlier in [16]). They showed that the sets of all such vectors of a given length form distributive lattices under suitably defined partial orderings extending results of Rivals and Rahmann. They also showed that there is a well defined minimal set of generators for any binary correlation of length n, called an irreducible period


25

set, and demonstrated that these generating sets are the primitive subsets of {1, 2, . . . , n − 1}. These primitive sets of integers have been extensively studied by many researchers including Erdös [65]. Finally, they investigated the number of partial word correlations of length n. More recently, recurrences for computing the size of populations of partial word correlations were obtained as well as random sampling of period and weak period sets [23]. We first define the greatest lower bound of two given partial words u and v of equal length as the partial word u ∧ v, where (u ∧ v) ⊂ u and (u ∧ v) ⊂ v, and if w ⊂ u and w ⊂ v, then w ⊂ (u ∧ v). The following example illustrates this new concept which plays a role in this section: u =ab caab

aa v =acbcaab bba u∧v = a c a a b a The contents of Section 2.5 is as follows: In Section 2.5.1, we give characterizations of correlations. In Section 2.5.2, we provide structural properties of correlations. And in Section 2.5.3, we consider the problem of counting correlations. 2.5.1 Characterizations of Correlations Full word correlations are vectors representing sets of periods as stated in the following definition. Definition 4. Let u be a (full) word. Let v be the binary vector of length |u| for which v0 = 1 and 1 if i ∈ P(u) vi = 0 otherwise We call v the correlation of u. For instance, the word abbababbab has periods 5 and 8 (and 10) and thus has correlation 1000010010. Binary vectors may satisfy some propagation rules. Definition 5. 1. A binary vector v of length n is said to satisfy the forward propagation rule if for all 0 ≤ p < q < n such that vp = vq = 1 we have that vp+i(q−p) = 1 for all 2 ≤ i < n−p q−p . 2. A binary vector v of length n is said to satisfy the backward propagation rule if for all 0 ≤ p < q < min(n, 2p) such that vp = vq = 1 and v2p−q = 0 p , n−p we have that vp−i(q−p) = 0 for all 2 ≤ i ≤ min( q−p q−p ). Note that a binary vector v of length 12 satisfying v7 = v9 = 1 and the forward propagation rule also satisfies v7+2(9−7) = v11 = 1. Note also that setting p = 0 in the forward propagation rule implies that viq = 1 for all i whenever vq = 1.

26


Fundamental results on periodicity of words include the following unexpected result of Guibas and Odlyzko which gives a characterization of full word correlations. Theorem 18. [71] For correlation v of length n the following are equivalent: 1. There exists a word over the binary alphabet with correlation v. 2. There exists a word over some alphabet with correlation v. 3. The correlation v satisfies the forward and backward propagation rules. Corollary 2. [71] For any word u over an alphabet A, there exists a binary word v of length |u| such that P(v) = P(u). Now, partial word correlations are defined according to the following definition. Definition 6. [24] 1. The binary correlation of a partial word u satisfying P(u) = P (u) is the binary vector of length |u| such that v0 = 1 and 1 if i ∈ P(u) vi = 0 otherwise 2. The ternary correlation of a partial word u is the ternary vector of length |u| such that v0 = 1 and ⎧ ⎪ ⎨1 if i ∈ P(u) vi = 2 if i ∈ P (u) \ P(u) ⎪ ⎩ 0 otherwise Considering the partial word abaca

acaba which has periods 9 and 11 (and 12) and strictly weak period 5, its ternary correlation vector is 100002000101. A characterization of binary correlations follows. Theorem 19. [24] Let n be a nonnegative integer. Then for any finite collection u1 , u2 , . . . , uk of full words of length n over an alphabet A, there exists a partial word w of length n over the binary alphabet with P(w) = P (w) = P(u1 ) ∪ P(u2 ) ∪ · · · ∪ P(uk ). Corollary 3. [24] The set of valid binary correlations over an alphabet A with A ≥ 2 is the same as the set of valid binary correlations over the binary alphabet. Phrased differently, if u is a partial word over an alphabet A, then there exists a binary partial word v of length |u| such that P(v) = P(u). Follows is a characterization of valid ternary correlations.


27

Theorem 20. [24] A ternary vector v of length n is the ternary correlation of a partial word of length n over an alphabet A if and only if v0 = 1 and 1. If vp = 1, then for all 0 ≤ i < np we have that vip = 1. 2. If vp = 2, then there exists some 2 ≤ i < np such that vip = 0. The proof is based on the following construction: For n ≥ 3 and 0 < p < n, let n = kp + r where 0 ≤ r < p. Then define if r = 0 (abp−1 )k ωp = p−1 k r−1 if r > 0 (ab ) ab ψp = abp−1 bn−p−1 Then given a valid ternary correlation v of length n, the partial word

∧ ω ψ p>0|vp =1 p p|vp =2 p has ternary correlation v. For example, given v = 100002000101, then abbbb bbb b has correlation v as computed in the following figure: ω9 = a b b b b b b b b a b b ω11 = a b b b b b b b b b b a ψ5 = a b b b b b b b b b b abbbb bbb b The following corollary implies that every partial word has a “binary equivalent”. Corollary 4. [24] The set of valid ternary correlations over an alphabet A with A ≥ 2 is the same as the set of valid ternary correlations over the binary alphabet. Phrased differently, if u is a partial word over an alphabet A, then there exists a binary partial word v such that 1. |v| = |u|

2. P(v) = P(u)

3. P (v) = P (u)

In [74], Halava, Harju and Ilie gave a simple constructive proof of Theorem 18 which computes v in linear time. This result was later proved for partial words with one hole by extending Halava et al.’s approach [16]. More specifically, given a partial word u with one hole over an alphabet A, a partial word v over the binary alphabet exists such that Conditions 1–3 hold as well as the following condition 4. H(v) ⊂ H(u) However, Conditions 1–4 cannot be satisfied simultaneously in the two-hole case. For the partial word abaca

acaba can be checked by brute force to have no such binary equivalent (although it has the binary equivalent abbbb bbb b as discussed above).

28


Open problem 10 Characterize the partial words that have an equivalent over the binary alphabet {a, b} satisfying Conditions 1–4. Open problem 11 Design an efficient algorithm for computing a binary equivalent satisfying Conditions 1–4 when such equivalent exists. Open problem 12 Can we always find an equivalent over the ternary alphabet {a, b, c} that satisfies Conditions 1–4? 2.5.2 Structural Properties of Correlations A result of Rivals and Rahmann [108] states that Γn , the set of full word correlations of length n, is a lattice under set inclusion which does not satisfy the Jordan-Dedekind condition, a criterion which stipulates that all maximal chains between two elements of a poset are of equal length. Violating the Jordan-Dedekind condition implies that Γn is not distributive. We now discuss corresponding results for partial words. Theorem 21. [24] The set ∆n of partial word binary correlations of length n is a distributive lattice under ⊂ where for u, v ∈ ∆n , u ⊂ v if P(u) ⊂ P(v), and thus satisfies the Jordan-Dedekind condition. Here 1. The meet of u and v, u∩v, is the unique vector in ∆n such that P(u∩v) = P(u) ∩ P(v). 2. The join of u and v, u ∪ v, is the unique vector in ∆n such that P(u ∪ v) = P(u) ∪ P(v). 3. The null element is 10n−1 . 4. The universal element is 1n . The union of u and v, u ∪ v, is the vector in ∆n defined as (u ∪ v)i = 0 if ui = vi = 0, 1 if either ui = 1 or vi = 1, and 2 otherwise. However, ∆n is not closed under union. Considering the example u =102000101 v =100010001 (u ∪ v) = 1 0 2 0 1 0 1 0 1 there is no i ≥ 2 such that (u ∪ v)i2 = 0, and therefore (u ∪ v) is not a valid ternary correlation. However 101010101 is valid. Theorem 22. [24] The set ∆n of partial word ternary correlations of length n is a distributive lattice under ⊂ where for u, v ∈ ∆n , u ⊂ v if ui = 1 implies vi = 1 and ui = 2 implies vi = 1 or vi = 2. Here 1. The meet of u and v, u∧v, is the vector (u∩v) in ∆n defined by P(u∧v) = P(u) ∩ P(v) and P (u ∧ v) = P (u) ∩ P (v). 2. The join of u and v, u ∨ v, is the vector in ∆n defined by


29

P (u ∨ v) = P (u) ∪ P (v) P(u ∨ v) = P(u) ∪ P(v) ∪ B(u ∪ v) where B(u ∪ v) is the set of all 0 < p < n such that (u ∪ v)p = 2 and there exists no i ≥ 2 satisfying (u ∪ v)ip = 0. In the case of full words, some periods are implied by other periods because of the forward propagation rule. If a twelve-letter full word has periods 7 and 9 then it must also have period 11 since 11 = 7 + 2(9 − 7), so {7, 9, 11} corresponds to the irreducible period set {7, 9}. Another result of Rivals and Rahmann shows that the set Λn of these irreducible period sets is not a lattice but does satisfy the Jordan-Dedekind condition as a poset [108]. However, forward propagation does not hold in the case of partial words as can be seen with the partial word abbbbbb b bb which has periods 7 and 9 but does not have period 11. The set {7, 9, 11} is irreducible in the sense of partial words, but not in the sense of full words. This leads us to the definition of generating sets. Definition 7. [24] A set P ⊂ {1, 2, . . . , n − 1} generates the correlation v ∈ ∆n provided that for each 0 < i < n we have that vi = 1 if and only if there exists p ∈ P and 0 < k < np such that i = kp. For instance, if v = 1001001101, then {3, 6, 7, 9}, {3, 6, 7}, {3, 7, 9}, and {3, 7} generate v. However, the set {3, 7} is the minimal generating set of v. For every v ∈ ∆n there is a minimal generating set R(v) for v which we call the irreducible period set of v. Namely, this is the set of p ∈ P(v) such that for all q ∈ P(v) with q = p we have that q does not divide p. Denoting by Φn the set of irreducible period sets of partial words of length n, we see that there is an obvious bijective correspondence between Φn and ∆n given by R : ∆n → Φn v → R(v) E : Φn → ∆n P → p∈P pn For n ≥ 3, we see immediately that the poset (Φn , ⊂) is not a join semilattice since the sets {1} and {2} will never have a join because {1} is always maximal. On the other hand, the following holds. Proposition 1. [24] The set Φn of irreducible period sets of partial words of length n is a meet semilattice under set inclusion which satisfies the JordanDedekind condition. Here the null element is ∅, and the meet of two elements is simply their intersection. Open problem 13 Is there an efficient enumeration algorithm for ∆n ?

30


2.5.3 Counting Correlations In this section, we look at the number of valid correlations of a given length. In the case of binary correlations, we give bounds and link the problem to one in number theory. In the case of ternary correlations, we give an exact count. A primitive set of integers is a subset S ⊂ {1, 2, . . .} such that for any two distinct elements s, s ∈ S we have that neither s divides s nor s divides s. The irreducible period sets of correlations v ∈ ∆n are precisely the finite primitive subsets of {1, 2, . . . , n − 1}. A result of Erdös can be stated as follows. Theorem 23. [65] Let S be a finite primitive set of size k with elements less than n. Then k ≤ n2 . Moreover, this bound is sharp. This bound shows that the number of binary correlations of length n is at most the number of subsets of {1, 2, . . . , n − 1} of size at most n2 . Moreover, the sharpness of the bound gives us that ∆n ≥ 2 2 n

Thus

ln ∆n ln 2 ≤ ≤ ln 2 2 n

Open problem 14 Refine this bound on the cardinality of ∆n , the set of all partial word binary correlations of length n. Guibas and Odlyzko [71] showed that as n → ∞ 1 2 ln 2

+ o(1) ≤

ln Γn

(ln n)2

≤

1 2 ln( 32 )

+ o(1)

and Rivals and Rahmann [108] improved the lower bound to

ln Γn

1 ln ln n 2 0.4139 1.47123 ln ln n 1 1 − ≥ + − + O 2 2 2 (ln n) 2 ln 2 ln n ln n (ln n) (ln n) where Γn is the set of all full word correlations of length n. Thus the bounds we give, which show explicitly that ln ∆n = Θ(n), demonstrate that the number of partial word binary correlations is much greater than the number of full word correlations. Lemma 1. [24] 1. Let u be a partial word of length n. Then p ∈ P(u) if and only if ip ∈ P (u) for all 0 < i ≤ np . 2. If S ⊂ {1, 2, . . . , n− 1}, then there exists a unique correlation v ∈ ∆n such that P (v) \ {n} = S. Consequently, the cardinality of ∆n , the set of valid ternary correlations of length n, is the same as the cardinality of the power set of {1, 2, . . . , n − 1}, and thus


31

∆n = 2n−1 We end this section with the following open problem. Open problem 15 Exhibit an algorithm to sample uniformly (weak) period sets through irreducible (weak) period sets.

2.6 Primitive and Unbordered Partial Words The two fundamental concepts of primitiveness and borderedness play an important role in several research areas including coding theory [5, 6, 117], combinatorics on words [45, 92, 93, 94, 96], computational biology [39, 100], data communication [41], data compression [49, 119, 123], formal language theory [57, 58], and text algorithms [38, 50, 51, 52, 69, 72, 84, 102, 118]. A primitive word is one that cannot be written as a power of another word, while an unbordered word is a primitive word such that none of its proper prefixes is one of its suffixes. For example, abaab is bordered with border ab while abaabb is unbordered. The number of primitive and unbordered words of a fixed length over an alphabet of a fixed size is well known, the number of primitive words being related to the Möbius function [92]. In this section, we discuss, in particular, the problems of counting primitive and unbordered partial words. 2.6.1 Primitiveness A word u is primitive if there exists no word v such that u = v i with i ≥ 2. A natural algorithmic problem is how can we decide efficiently whether a given word is primitive. The problem has a brute force quadratic solution: divide the input word into two parts and check whether the right part is a power of the left part. But how can we obtain a faster solution to the problem? Fast algorithms for testing primitivity of words can be based on the combinatorial result that a word u is primitive if and only if u is not an inside factor of its square uu, that is, uu = xuy implies x = ε or y = ε [45]. Indeed, any linear time string matching algorithm can be used to test whether the string u is a proper factor of uu. If the answer is no, then the primitiveness of u has been verified [51]. So testing whether or not a word is primitive can be done in linear time in the length of the word. Primitive partial words were defined in [9]: A partial word u is primitive if there exists no word v such that u ⊂ v i with i ≥ 2. It turns out that a partial word u with one hole is primitive if and only if uu ↑ xuy for some partial words x, y implies x = ε or y = ε [9]. A linear time algorithm for testing primitivity of partial words with one hole can be based on this combinatorial result. As an application, the existence of a binary equivalent for any partial word with one hole satisfying Conditions 1–4 discussed in Section 2.5 was obtained [16]. In [11], a linear time algorithm was described to test primitivity on

32


partial words with more than one hole. Here the concept of speciality related to commutativity on partial words, which was discussed in Section 2.2, is foundational in the design of the algorithm. More precisely, it was shown that if u is a primitive partial word with more than one hole satisfying uu ↑ xuy for some nonempty partial words x and y such that |x| < |y|, then u is (|x|, |y|)special. The partial words u = ab bbb b, x = a , and y = c bbcb illustrate the fact that the condition of speciality plays a role when dealing with partial words with more than one hole. In [19], the very challenging problem of counting the number Ph,k (n) (re (n)) of primitive (respectively, nonprimitive) partial words spectively, Ph,k with h holes of length n over a k-size alphabet was considered. There, formulas for h = 1 and h = 2 in terms of the well known formula for h = 0 were (n). given. Denote by Th,k (n) the sum of Ph,k (n) and Ph,k We first recall the counting for primitive full words. Since there are exactly kn words of length n over a k-size alphabet and every nonempty word w has a unique primitive root v for which w = v n/d for some divisor d of n, the following relation holds: P0,k (d) kn = d|n

Using the Möbius inversion formula, we obtain the following well-known expression for P0,k (n) [92, 105]: P0,k (n) = µ(d)kn/d d|n

where the Möbius function, denoted by µ, is defined as ⎧ if n = 1 ⎨1 µ(n) = (−1)i if n is a product of i distinct primes ⎩ 0 if n is divisible by the square of a prime The cases where h = 1 and h = 2 are stated in the next two theorems. (n) = nP0,k (n) holds. Theorem 24. [19] The equality P1,k

Theorem 25. [19] 1. For an odd positive integer n: P2,k (n)

n P (n) = 2 0,k

2. For an even positive integer n: n n P (n) − (k − 1)T1,k ( ) (n) = P2,k 2 0,k 2


33

Open problem 16 Count the number Ph,k (n) of nonprimitive partial words with h holes of length n over a k-size alphabet for h > 2.

Another problem to investigate is the following. Open problem 17 Study the language of primitive partial words as is done for full primitive words in [105]. We end this section with the following remark. In [18], the authors obtained consequences of the generalizations of Fine and Wilf’s periodicity result to partial words. In particular, they generalized the following combinatorial property: “For any word u over {a, b}, ua or ub is primitive.” This property proves in some sense that there exist very many primitive words. 2.6.2 Borderedness Unbordered partial words were also defined in [9]: A nonempty partial word u is unbordered if no nonempty partial words x1 , x2 , v, w exist such that u = x1 v = wx2 and x1 ↑ x2 . If such nonempty words exist and x is such that x1 ⊂ x and x2 ⊂ x, then we call u bordered and x a border of u. A border x of u is called minimal if |x| > |y| implies that y is not a border of u. Note that there are two types of borders: x is an overlapping border if |x| > |v|, and a nonoverlapping border otherwise. The partial word u = a

ab is bordered with the nonoverlapping border ab and overlapping border aab, the first one being minimal, while the partial word ab c is unbordered. We call a bordered partial word u simply bordered if a minimal border x exists satisfying |u| ≥ 2|x|. Proposition 2. [21] Let u be a nonempty bordered partial word. Let x be a minimal border of u, and set u = x1 v = wx2 where x1 ⊂ x and x2 ⊂ x. Then the following hold: 1. The partial word x is unbordered. 2. If x1 is unbordered, then u = x1 u x2 ⊂ xu x for some u . Note that Proposition 2 implies that if u is a full bordered word, then x1 = x is unbordered. In this case, u = xu x where x is the minimal border of u. Hence a bordered full word is always simply bordered. Corollary 5. [21] Every bordered full word of length n has a unique minimal border x. Moreover, x is unbordered and |x| ≤ n2 . In [20], the problem of enumerating all unbordered partial words with h holes of length n over a k-letter alphabet was considered, a problem that yields some open questions for further investigation. We will denote by Uh,k (n) the number of such words. Let us start with the problem of enumerating all unbordered full words of length n over a k-letter alphabet which gives a conceptually simple and elegant recursive formula: U0,k (0) = 1, U0,k (1) = k, and for n > 0,

34


U0,k (2n) = kU0,k (2n − 1) − U0,k (n) U0,k (2n + 1) = kU0,k (2n) These equalities can be seen from the fact that if a word has odd length 2n + 1 then it is unbordered if and only if it is unbordered after removing the middle letter. If a word has even length 2n then it is unbordered if and only if it is obtained from an unbordered word of length 2n − 1 by adding a letter next to the middle position unless doing so creates a word that is a perfect square. Using these formulas and Proposition 2, we can easily obtain a formula for counting bordered full words. Let Bk (j, n) be the number of full words of length n over a k-letter alphabet that have a minimal border of length j: Bk (j, n) = U0,k (j)kn−2j If we let Bk (n) be the number of full words of length n over a k-letter alphabet with a border of any length, then we have that 2 n

Bk (n) =

Bk (j, n)

j=1

When we allow words to have holes, counting bordered partial words is made extremely more difficult by the failure of Corollary 5 since there is now the possibility of a minimal border that is overlapping as in a bb. We will first concern ourselves with the simply bordered partial words. Note that because borderedness in partial words is defined via containment, it no longer makes sense to talk about the minimal border of a partial word, there could be many possible borders of a certain length. To see inside the structure of the partial words we are trying to count we first define a function. Let fh,k (i, j, n) be the number of partial words of length n with h > 0 holes over a k-letter alphabet that have a hole in position i and a minimal border of length j. When i = 0: n−1 n−h if j = 1 h−1 k fh,k (0, j, n) = 0 if j > 1 It is clear that fh,k (i, j, n) has some symmetry, namely that, fh,k (i, j, n) = fh,k (n − 1 − i, j, n). Throughout this section we will rely on this to consider only i ≤ n2 . We have some general formulas for the evaluation of fh,k (i, j, n). Proposition 3. [20] If 0 < i < j − 1 and j
1 Note that in the case where 0 < i < j − 1 and j < tion 3 reduces to the very simple equality

n 2,

the formula in Proposi-

f1,k (i, j, n) = f1,k (i, j, 2j)kn−2j Similarly, in the case where j − 1 < i, the formula in Proposition 4 reduces to f1,k (i, j, n) = U0,k (j)kn−2j−1 By the above discussion we can restrict our attention to the cases when i > 0, n = 2m and j = m. These are partial words with a border that takes up exactly half the length of the word. We wish to find a complete formula for f1,k (i, m, 2m) where i = m − 1 − i . We proceed by induction on i . When i = 0, we have the following. Lemma 2. [20] For all m ≥ 2, f1,k (m − 1, m, 2m) = U0,k (m). Continuing with the first interesting case i = 1, we have the following lemma. Lemma 3. [20] For all m ≥ 3, f1,k (m − 2, m, 2m) = U0,k (m) − k(k − 1). This kind of analysis quickly becomes much more complicated though. The evaluation breaks up into cases depending on how the periodicity of the words interacts with the length of the border in modular arithmetic. Lemma 4. [20] For all m ≥ 4, the following holds: U0,k (m) − k 2 (k − 1) − k(k − 1) if m ≡ 1 mod 2 f1,k (m − 3, m, 2m) = U0,k (m) − k(k − 1)2 − k(k − 1) if m ≡ 0 mod 2 Lemma 5. [20] For all m ≥ 5, the following holds: f1,k (m − 4, m, 2m) = U0,k (m) − k(k − 1) − g1 (m) − g2 (m) where

36


g1 (m) =

k(k − 1)2 if m ≡ 0 mod 2 0 if m ≡ 1 mod 2

and ⎧ 2 ⎨ k (k − 1)2 if m ≡ 0 mod 3 if m ≡ 1 mod 3 g2 (m) = U0,k (4) ⎩ 2 k (k − 1)2 if m ≡ 2 mod 3 To give an idea of how the values for f1,k (i, m, 2m) behave unpredictably, here is a table of values that has been put together through a brute force count: i f1,2 (i, 2, 4) f1,2 (i, 3, 6) f1,2 (i, 4, 8) f1,2 (i, 5, 10) f1,2 (i, 6, 12) f1,2 (i, 7, 14) f1,2 (i, 8, 16)

0 0 0 0 0 0 0 0

1 2 2 2 6 10 22 42

2 3 4 5 6 7 4 4 6 12 26 52

6 10 16 32 60

12 18 20 34 38 40 66 70 72 74

Open problem 18 Compute the values f1,k (m − i, m, 2m) for m > i. Let Sh,k (n) be the number of simply bordered partial words of length n with h holes over a k-letter alphabet. Clearly if h > n, then Sh,k (n) = 0. Note that when h = 0, Sh,k (n) = Bk (n). Theorem 26. [20] If 0 < h ≤ n, then a formula for Sh,k (n) is given by: Sh,k (2m + 1) = Sh−1,k (2m) + kSh,k (2m) m−1 m 2

Sh,k (2m) =

fh,k (i, j, 2m)

i=0 j=1 h

We can check that S1,k (n) =

n n−1 2

f1,k (i, j, n)

i=0 j=1

Let Nh,k (n) be the number of partial words with h holes, of length n, over a k-letter alphabet that are not simply bordered. Obviously we can find the value of this function by subtracting the value of Sh,k (n) from the total number of partial words with those parameters, but it would be of interest to find a direct formula for Nh,k (n). If h = 0, then


37

N0,k (n) = U0,k (n) since a bordered full word that is not simply bordered is an unbordered full word. It is easy to see that N1,k (0) = 0, N1,k (1) = 1, N1,k (2) = 0, and for h > 1 that Nh,k (1) = 0 and Nh,k (2) = 0. Now, for h > 0, the following formula holds for odd n = 2m + 1: Nh,k (2m + 1) = kNh,k (2m) + Nh−1,k (2m) Open problem 19 What is Nh,k (2m)? If we simplify the problem down to the h = 1 case, then we can again use the values f1,k (i, j, n) to give a formula for N1,k (n): N1,k (2m) = kN1,k (2m − 1) + 2U0,k (2m − 1) −

m

f1,k (i, m, 2m)

i=1

but it rests on the evaluation of the f1,k (i, j, 2j)’s as well. Other interesting questions include the following. Open problem 20 Count the number Oh,k (n) of overlapping bordered partial words of length n with h holes over a k-letter alphabet for h > 0. Open problem 21 Count the number Uh,k (n) of unbordered partial words of length n with h holes over a k-letter alphabet for h > 0. Another open question is suggested by the fact that every partial word of length 5 that has more than two holes is simply bordered. The partial word a

bb shows that this bound on the number of holes for length 5 is tight. For length 6, every partial word with more than 2 holes is simply bordered as well. Open problem 22 What is the maximum number of holes M (n) a partial word of length n can have and still fail to be simply bordered? Some values for small n follow. n M (n) 5 2 6 2 7 3 8 4 9 5 10 5 11 6 12 7 13 8 14 8 15 9

38


We end this section by discussing another open problem related to borderedness in the context of partial words. In 1979, Ehrenfeucht and Silberger initiated a line of research to explore the relationship between the minimal period of a word w of length n, p(w), and the maximum length of its unbordered factors, µ(w) [64]. Clearly, µ(w) ≤ p(w). They conjectured that if n ≥ 2µ(w), then µ(w) = p(w). In [3], a counterexample was given and it was conjectured that 3µ(w) is the precise bound. In 1982, it was established that if n ≥ 4µ(w) − 6, then µ(w) = p(w) [61]. In 2003, the bound was improved to 3µ(w) − 2 in [76] where it is believed that the precise bound can be achieved with methods similar to those presented in that paper. Open problem 23 Investigate the relationship between the minimal weak period of a partial word and the maximum length of its unbordered factors.

2.7 Equations on Partial Words As was seen in Section 2.2, some of the most basic properties of words, like the commutativity and the conjugacy properties, can be expressed as solutions of the word equations xy = yx and xz = zy respectively. It is also well known that the equation xm = y n z p has only periodic solutions in a free semigroup, that is, if xm = y n z p holds with integers m, n, p ≥ 2, then there exists a word w such that x, y, z are powers of w. This result, which received a lot of attention, was first proved by Lyndon and Schützenberger for free groups [96]. Their proof implied the case for free semigroups since every free semigroup can be embedded in a free group. Direct proofs for free semigroups appear in [46, 77, 92]. In this section, we study equations on partial words. When we speak about them, we replace the notion of equality with compatibility. But compatibility is not transitive! We already solved the commutativity equation xy ↑ yx as well as the conjugacy equation xz ↑ zy in Section 2.2. As an application of the commutativity equation, we mention the linear time algorithm for testing primitivity on partial words that was discussed in Section 2.6 [11], and as an application of the conjugacy equation, we mention the efficient algorithm for computing a critical factorization when one exists that was discussed in Section 2.4 [22, 35]. Here, we solve three equations: xm ↑ y n , x2 ↑ y m z, and xm ↑ y n z p . First, let us consider the equation xm ↑ y n , also called the “good pairs” equation. If x and y are full words, then xm = y n for some positive integers m, n if and only if there exists a word z such that x = z k and y = z l for some integers k, l. When dealing with partial words x and y, if there exists a partial word z such that x ⊂ z k and y ⊂ z l for some integers k, l, then xm ↑ y n for some positive integers m, n. For the converse, we need a couple of lemmas.


39

Lemma 6. [13] Let x, y be partial words and let m, n be positive integers such that xm ↑ y n with gcd(m, n) = 1. Call |x|/n = |y|/m = p. If there exists an integer i such that 0 ≤ i < p and xi,p is not 1-periodic, then D(yi,p ) is empty. Lemma 7. [13] Let x be a partial word, let m, p be positive integers, and let i be an integer such that 0 ≤ i < p. Then the relation xm i,p = xi,p x(i−|x|) mod p,p . . . x(i−(m−1)|x|) mod p,p holds. The “good pairs” theorem is stated as follows. Theorem 27. [13] Let x, y be partial words and let m, n be positive integers such that xm ↑ y n with gcd(m, n) = 1. Assume that (x, y) is a good pair, that is, 1. For all i ∈ H(x) the word y n i,|x| is 1-periodic, 2. For all i ∈ H(y) the word xm i,|y| is 1-periodic. Then there exists a partial word z such that x ⊂ z k and y ⊂ z l for some integers k, l. The assumption of (x, y) being a good pair is necessary in the “good pairs” theorem. Indeed, x2 = (a b)2 ↑ (acbadb)1 = y 1 but y(1)y(4) = cd is not 1-periodic, and there exists no partial word z as desired. Corollary 6. [13] Let x and y be primitive partial words such that (x, y) is a good pair. If xm ↑ y n for some positive integers m and n, then x ↑ y. Note that if both x and y are full words, then (x, y) is a good pair. The corollary hence implies that if x, y are primitive full words satisfying xm = y n for some positive integers m and n, then x = y. Second, we consider the “good triples” equation x2 ↑ y m z. Here, it is assumed that m is a positive integer and z is a prefix of y. Nontrivial solutions exist! A solution is trivial if x, y, z are contained in powers of a common word. The equation x2 ↑ y m z has nontrivial solutions. For instance, (a

a)2 ↑ (aab)2 aa where x = a

a, y = aab, and z = aa. The “good triples” theorem follows. Theorem 28. [13] Let x, y, z be partial words such that z is a proper prefix of y. Then x2 ↑ y m z for some positive integer m if and only if there exist partial words u, v, u0 , v0 , . . . , um−1 , vm−1 , zx

40


such that u = ε, v = ε, y = uv, x = (u0 v0 ) . . . (un−1 vn−1 )un = vn (un+1 vn+1 ) . . . (um−1 vm−1 )zx

(2.1) (2.2)

where 0 ≤ n < m, u ↑ ui and v ↑ vi for all 0 ≤ i < m, z ↑ zx , and where one of the following holds: 1. m = 2n, |u| < |v|, and there exist partial words u , un such that zx = u un , z = uun , u ↑ u and un ↑ un . and zx such that 2. m = 2n + 1, |u| > |v|, and there exist partial words v2n un = v2n zx , u = v2n zx , v2n ↑ v2n and zx ↑ zx . A triple of partial words (x, y, z) which satisfy these properties we will refer to as a good triple. Two corollaries can be deduced. Corollary 7. [13] Let x, y, z be partial words such that z is a prefix of y. Assume that x, y are primitive and that x2 ↑ y m z for some integer m ≥ 2. If x has at most one hole and y is full, then x ↑ y. Corollary 8. [13] Let x, y, z be words such that z is a prefix of y. If x, y are primitive and x2 = y m z for some integer m ≥ 2, then x = y. Note that the corollaries do not hold when m = 1. Indeed, the words x = aba, y = abaab and z = a provide a counterexample. Also, the first corollary does not hold when x is full and y has one hole as is seen by setting x = abaabb, y = ab and z = ε. Third, let us consider the equation xm y n ↑ z p . The case of full words is well known. Theorem 29. [96] Let x, y, z be full words and let m, n, p be integers such that m ≥ 2, n ≥ 2 and p ≥ 2. Then the equation xm y n = z p has only trivial solutions, that is, x, y, and z are each a power of a common element. When we deal with partial words, the equation xm y n ↑ z p certainly has a solution when x, y, and z are contained in powers of a common word (we call such solutions the trivial solutions). However, there may be nontrivial solutions as is seen with the compatibility relation (a b)2 (b a)2 ↑ (abba)3 We will classify solutions as Type 1 (or trivial) solutions when there exists a partial word w such that x, y, z are contained in powers of w, and as Type 2 solutions when the partial words x, y, z satisfy x ↑ z and y ↑ z. Note that if z is full, then Type 2 solutions are trivial solutions. The case p ≥ 4 is stated in the following theorem.


41

Theorem 30. [13] Let x, y, z be primitive partial words such that (x, z) and (y, z) are good pairs. Let m, n, p be integers such that m ≥ 2, n ≥ 2 and p ≥ 4. Then the equation xm y n ↑ z p has only solutions of Type 1 or Type 2 unless one of the following holds: 1. x2 ↑ z k zp for some integer k ≥ 2 and nonempty prefix zp of z, 2. z 2 ↑ xl xp for some integer l ≥ 2 and nonempty prefix xp of x. Open problem 24 Solve the equation xm y n ↑ z p on partial words for integers m ≥ 2, n ≥ 2 and p ∈ {2, 3}.

2.8 Unavoidable Sets of Partial Words A set of (full) words X over a finite alphabet A is unavoidable if no two-sided infinite word over A avoids X, that is, X is unavoidable if every two-sided infinite word over A has a factor in X. For instance, the set X = {a, bbb} is unavoidable (if a two-sided infinite word w does not have a as a factor, then w consists only of b’s). This concept was explicitly introduced in 1983 in connection with an attempt to characterize the rational languages among the context-free ones [63]. It is clear from the definition that from each unavoidable set we can extract a finite unavoidable subset, so the study can be reduced to finite unavoidable sets. There is a vast literature on unavoidable sets of words and we refer the reader to [44, 93, 109, 110] for more information. Unavoidable sets of partial words were introduced recently in [15], where the problem of classifying such sets of small cardinality was initiated, in particular, those with two elements. The authors showed that this problem reduces to the one of classifying unavoidable sets of the form {a m1 a . . . a mk a, b n1 b . . . b nl b} where m1 , . . . , mk , n1 , . . . , nl are nonnegative integers and a, b are distinct letters. They gave an elegant characterization of the special case of this problem when k = 1 and l = 1. They proposed a conjecture characterizing the case where k = 1 and l = 2 and proved one direction of the conjecture. They then gave partial results towards the other direction and in particular proved that the conjecture is easy to verify in a large number of cases. Finally, they proved that verifying this conjecture is sufficient for solving the problem for larger values of k and l. In [27], the authors built on the previous work by examining, in particular, unavoidable sets of size three. In [15], the question was raised as to whether there is an efficient algorithm to determine if a finite set of partial words is unavoidable. In [26], it was shown that this problem is NP-hard by using techniques similar to those used in a recent paper on the complexity of computing the capacity of codes that avoid forbidden difference patterns [37]. This is in contrast with the well known feasibility results for unavoidability of a set of full words [93].

42


The contents of Section 2.8 is as follows: In Section 2.8.1, we review basics on unavoidable sets of partial words. In Section 2.8.2, we discuss classifying such sets of size two. And in Section2.8.3, we discuss testing unavoidability of sets of partial words. 2.8.1 Unavoidable Sets We first define some basic terminology. A two-sided infinite word over A is a total function w : Z → A. A finite word u is a factor of w if there exists some i ∈ Z such that u = w(i)w(i + 1) . . . w(i + |u| − 1). A period of w is a positive integer p such that w(i) = w(i + p) for all i ∈ Z. If w has period p for some p, then we call w periodic. If v is a finite word, then v Z denotes the two-sided infinite word w with period |v| satisfying w(0) . . . w(|v| − 1) = ˆ denotes the set of all full words v. If X is a set of partial words, then X ˆ = compatible with a member of X. For instance, if X = {a

a, b b}, then X {aaaa, aaba, abaa, abba, bab, bbb}. The concept of an unavoidable set of full words is defined as follows. Definition 8. Let X ⊂ A∗ . 1. A two-sided infinite word w avoids X if no factor of w is a member of X. 2. The set X is unavoidable if no two-sided infinite word over A avoids X, that is, X is unavoidable if every two-sided infinite word over A has a factor in X. If A = {a, b}, then the following sets are unavoidable: X1 = {ε} (ε is a factor of every two-sided infinite word); X2 = {a, bbb}; X3 = {aa, ab, ba, bb} (this is the set of all words of length 2); and for any n ∈ N, An is unavoidable. If X ⊂ A∗ is finite, then the following three statements are equivalent: (1) X is unavoidable; (2) There are only finitely many words in A∗ with no member of X as a factor; and (3) No periodic two-sided infinite word avoids X. An unavoidable set of partial words is defined as follows. Definition 9. Let X ⊂ A∗ . ˆ 1. A two-sided infinite word w avoids X if no factor of w is a member of X. 2. The set X is unavoidable if no two-sided infinite word over A avoids X, that is, X is unavoidable if every two-sided infinite word over A has a ˆ factor in X. If A = {a, b}, then the following sets are unavoidable: X1 = {a , b}; X2 = { n } for any nonnegative integer n as well as any set containing X2 as a subset (let us call such sets the trivial unavoidable sets); and X3 = {a, bbb} since of course Definition 9 is equivalent to Definition 8 if every member of X is full. We will explore some less trivial examples soon.


43

ˆ a two-sided infinite word w has a factor in X ˆ if By the definition of X, and only if that factor is compatible with a member of X. Thus the two-sided ˆ ⊂ A∗ , infinite words which avoid X ⊂ A∗ are exactly those which avoid X ˆ ⊂ A∗ is unavoidable. Thus with and X ⊂ A∗ is unavoidable if and only if X regards to unavoidability, a set of partial words serves as a representation of a set of full words. The set {a

a, b b} represents {aaaa, aaba, abaa, abba, bab, bbb} We will shortly prove that this set is unavoidable. The smaller X is, the more information is gained by identifying X as unavoidable. Thus it is natural to begin investigating the unavoidable sets of partial words of small cardinality. Of course, every two-sided infinite word avoids the empty set and thus, there are no unavoidable sets of size 0. Unless the alphabet is unary, the only unavoidable sets of size 1 are trivial. If the alphabet is unary, then every nonempty set is unavoidable and in that case there is only one two-sided infinite word. Thus the unary alphabet is not interesting so we will not consider it further. Classifying the unavoidable sets of size 2 is the focus of the next section. 2.8.2 Classifying Unavoidable Sets of Size Two If X is a two-element unavoidable set, then every two-sided infinite unary word has a factor compatible with a member of X. In particular, X cannot have fewer elements than the alphabet. Thus if X has size 2, then the alphabet is unary or binary. We hence assume that the alphabet is binary say with distinct letters a and b since we said above that the unary alphabet is not interesting. So one element of X is compatible with a factor of aZ and the other element is compatible with a factor of bZ , since this is the only way to guarantee that both aZ and bZ will not avoid X. Thus we can restrict our attention to sets of the form Xm1 ,...,mk |n1 ,...,nl = {a m1 a . . . a mk a, b n1 b . . . b nl b}

(2.3)

for some nonnegative integers m1 , . . . , mk and n1 , . . . , nl . For which integers m1 , . . . , mk , n1 , . . . , nl is Xm1 ,...,mk |n1 ,...,nl unavoidable? A simplification is stated in the next lemma. Lemma 8. [15] If p is a nonnegative integer, then set X = Xm1 ,...,mk |n1 ,...,nl and Y = Xp(m1 +1)−1,...,p(mk +1)−1|p(n1 +1)−1,...,p(nl +1)−1 Then X is unavoidable if and only if Y is unavoidable.

44


The easiest place to start is with small values of k and l. Of course, the set {a, b n1 b . . . b nl b} is always unavoidable for if w is a two-sided infinite word which does not have a as a factor, then w = bZ . This handles the case where k = 0 (and symmetrically l = 0). An elegant characterization for the case where k = l = 1 is stated in the following theorem. Theorem 31. [15] The set Xm|n = {a m a, b n b} is avoidable if and only if m + 1 and n + 1 have the same greatest power of 2 dividing them. The next natural step is to look at k = 1 and l = 2, that is, sets of the form Xm|n1 ,n2 = {a m a, b n1 b n2 b} On the one hand, we have identified a large number of avoidable sets of the form {a m a, b n b}. For Xm|n1 ,n2 to be avoidable it is sufficient that {a m a, b n1 b}, {a m a, b n2 b} or {a m a, b n1 +n2 +1 b} be avoidable. On the other hand, the structure of words avoiding {a m a, b n1 b n2 b} is not nearly as nice as those avoiding {a m a, b n b}. Thus a simple characterization seems unlikely, unless perhaps there are no unavoidable sets of this form at all. But there are! The set {a 7 a, b b 3 b} is unavoidable. Seeing that it is provides a nice example of the techniques that can be used. Referring to the figure below, . . . -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 ... b b ... a ... b ... a ... a ... b ... a

... ... ... ... ... ... ... ...

suppose instead that there exists a two-sided infinite word w which avoids it. We know from Theorem 31 that {a 7 a, b b} is unavoidable, thus w must have a factor compatible with b b. Say without loss of generality that w(0) = w(2) = b. This implies that w(6) = a, which in turn implies that w(−2) = b. Then we have that w(−2) = w(0) = b, forcing w(4) = a. This propagation continues: w(−4) = w(−2) = b and so w(2) = a, which makes w(−6) = b giving w(0) = a, a contradiction. The perpetuating patterns phenomenon of the previous example is a special case of a more general result. Theorem 32. [15] If m = n2 − n1 − 1 or m = 2n1 + n2 + 2, and the highest power of 2 dividing n1 + 1 is less than the highest power of 2 dividing m + 1, then Xm|n1 ,n2 is unavoidable.


45

Here are other unavoidability results for k = 1 and l = 2. Proposition 5. [15] The set Xm|n1 ,n2 is unavoidable if Conditions 1 or Conditions 2 or Conditions 3 hold: 1. {a m a, b n1 b} is unavoidable, m = 2n1 + n2 + 2 or m = n2 − n1 − 1, and n1 + 1 divides n2 + 1. 2. n1 < n2 , 2m = n1 + n2 and m − n1 divides m + 1. 3. m = 6, n1 = 1 and n2 = 3. Extensive experimentation suggests that these results (and their symmetric equivalents) give a complete characterization of when Xm|n1 ,n2 is unavoidable. Conjecture 1 [15] The set Xm|n1 ,n2 is unavoidable if and only if one of the following conditions (or symmetric equivalents) holds: 1. {a m a, b n1 b} is unavoidable, m = 2n1 + n2 + 2 or m = n2 − n1 − 1, and n1 + 1 divides n2 + 1. 2. m = n2 − n1 − 1 or m = 2n1 + n2 + 2, and the highest power of 2 dividing n1 + 1 is less than the highest power of 2 dividing m + 1. 3. n1 < n2 , 2m = n1 + n2 and m − n1 divides m + 1. 4. m = 6, n1 = 1 and n2 = 3. Open problem 25 Is Conjecture 1 true or false? If true, then Conjecture 1 implies that the unavoidable sets of size two have been completely classified as stated in the following proposition. Proposition 6. [15] If Conjecture 1 is true, then Xm1 ,...,mk |n1 ,...,nl is avoidable for k = 1 and l ≥ 3, and for k > 1 and l ≥ 2. In order to prove the conjecture, only one direction remains. We must show that if none of the aforementioned conditions hold, then Xm|n1 ,n2 is avoidable. There are some partial results towards this goal. In particular there is an easy way of verifying the conjecture for even values of m. Proposition 7. [15] Assume m is even and 2m ≤ min(n1 , n2 ). Then Xm|n1 ,n2 is avoidable. Thus for any fixed even m we only need to verify the conjecture for finitely many values of n1 and n2 , which is generally easy. For 1. m = 0: X0|n1 ,n2 is always avoidable, and indeed this is the case. 2. m = 2: X2|n1 ,n2 is avoidable except for n1 = 1, n2 = 3 or n1 = 3, n2 = 1. It is easy to find avoiding two-sided infinite words for other values of n1 and n2 less than 5 when m = 2. This is all that is necessary to confirm the conjecture for m = 2.

46


In this way the conjecture has been verified for all even m up to very large values via computer. The odd values of m seem to be much more difficult. The following proposition shows that the conjecture is true for m = 1. Proposition 8. [15] Conjecture 1 is true for m = 1, that is, X1|n1 ,n2 is unavoidable if and only if n1 and n2 are even numbers with |n1 − n2 | = 2. Other results on the avoidability of Xm|n1 ,n2 include the following. Proposition 9. [15] 1. Let s ∈ N with s < m − 2. Then for n > 2(m + 1)2 + m − 1, Xm|m+s,n = {a m a, b m+s b n b} is avoidable. Intuitively this means that if m and n1 are relatively close in value, then the set of integers n2 which make Xm|n1 ,n2 unavoidable is finite. 2. If max(n1 , n2 ) < m < n1 + n2 + 2, then Xm|n1 ,n2 is avoidable. 3. The set X = {a m a, bbb} is avoidable. Classifying the unavoidable sets of partial words of size greater than or equal to two remains an open question. Open problem 26 Classify the unavoidable sets of partial words of size l ≥ 2 over a k-letter alphabet where k ≤ l. 2.8.3 Testing Unavoidability Efficient algorithms to determine if a finite set of full words is unavoidable are well known [45, 93]. For example, we can check whether there is a loop in the finite automaton of Aho and Corasick [1] recognizing A∗ \ A∗ XA∗ . Another approach is the following. We say that a set of words Y is obtained from a finite set of words X by an elementary derivation if 1. Type 1 elementary derivation: There exist words x, y ∈ X such that x is 1 a proper prefix of y, and Y = X \ {y} (this will be denoted by X → Y ). 2. Type 2 elementary derivation: There exists a word x = ya ∈ X with a ∈ A such that, for each letter b ∈ A there is a suffix z of y such that zb ∈ X, 2 and Y = (X \ {x}) ∪ {y} (this will be denoted by X → Y ). A derivation is a sequence of elementary derivations. We say that Y is derived from X if Y is obtained from X by a derivation. If Y is derived from X, then X is unavoidable if and only if Y is unavoidable. Example 2. The following sequence of elementary derivations shows that X = {aaaa, aaba, abaa, abba, bab, bbb} derives {ε}:


47

2

X → {aaaa, aaba, aba, abba, bab, bbb} 2 → {aaaa, aaba, aba, abb, bab, bbb} 2 → {aaaa, aaba, ab, bab, bbb} 2 → {aaa, aaba, ab, bab, bbb} 2 → {aa, aaba, ab, bab, bbb} 1 → {aa, ab, bab, bbb} 2 → {a, ab, bab, bbb} 1 → {a, bab, bbb} 2 → {a, ba, bbb} 2 → {a, ba, bb} 2 → {a, b, bb} 2 → {a, b} 2 → {ε, b} 1 → {ε} The notion of a derivation gives an algorithm to check whether a set is unavoidable: A finite set X is unavoidable if and only if there is a derivation from X to the set {ε}. The above derivation shows that {aaaa, aaba, abaa, abba, bab, bbb} is unavoidable. These algorithms to determine if a finite set of full words is unavoidable, like the one just described, can be used to decide if a finite set of partial ˆ However words X is unavoidable by determining the unavoidability of X. this incurs a dramatic loss in efficiency, as each partial word u in X can ˆ The above derivation shows contribute as many as A H(u) elements to X. that {a

a, b b} is unavoidable as is confirmed by Theorem 31 since m + 1 = 2 + 1 = 3 = 20 3 and n + 1 = 1 + 1 = 2 = 21 . In [15], the question was raised as to whether there is an efficient algorithm to determine if a finite set of partial words is unavoidable. In [26], it was proved that testing the unavoidability of a finite set of partial words is much harder to handle than the similar problem for full words. Indeed, the following theorem holds (note that the case k = 1 is trivial). Theorem 33. [26] The problem of deciding whether a finite set of partial words over a k-letter alphabet where k ≥ 2 is unavoidable is NP-hard. The proof proceeds by reduction from the 3SAT problem that is known to be NP-complete (see [70]). In the 3SAT problem, we are given n binary variables x1 , . . . , xn and m clauses that each contain three literals (a literal can be a variable or its negation), and we search a truth assignment for the variables such that each clause has at least one true literal. In [26], the following related questions on avoidability of sets of partial words were raised.

48


Open problem 27 Is the decision problem of the avoidability of a set of partial words in NP? A similar (stronger) question is the following one. Open problem 28 For any set of partial words X, does there always exist a two-sided infinite periodic word that avoids X, whose period is polynomial in the size of X?

2.9 Freeness of Partial Words In [99], Manea and Mercaş introduce freeness of partial words. There, they extend in a natural way the concepts of square- and overlap-freeness of words to partial words. In [31, 30], some more basic freeness properties of partial words are investigated generalizing the well-known freeness properties of full words. A one-sided infinite word over the alphabet A is a function from N to A. The Thue-Morse word is an example of a one-sided infinite word defined by iterating a morphism. Let φ : {a, b}∗ → {a, b}∗ be the morphism defined by φ(a) = ab and φ(b) = ba. We define t0 = a and ti = φi (a), for all i ≥ 1. ¯ is the word obtained Note that ti+1 = φ(ti ) and that ti+1 = ti ti , where x from x by replacing each occurrence of a with b and each occurrence of b with a. Thus, the limit (the infinite word) t = limi→∞ ti exists. The Thue-Morse word is defined as t, a fixed point for the morphism φ. Computations show that t1 = ab, t2 = abba, t3 = abbabaab, t4 = abbabaabbaababba, and t5 = abbabaabbaababbabaababbaabbabaab

(2.4)

and so on. A one-sided infinite word w is k-free if there is no word x such that xk is a factor of w (a word that is 2-free is also called square-free and a word that is 3-free is called cube-free). It is called overlap-free if it does not contain any factor of the form cycyc with c ∈ A. Any overlap-free word is clearly k-free for all k ≥ 3. Theorem 34. [120, 121] The Thue-Morse infinite word t is overlap-free and hence k-free for all k ≥ 3. A one-sided infinite partial word w over the alphabet A is a partial function from N to A. We call w k-free if for any nonempty factor x1 . . . xk of w, no partial word x exists such that xi ⊂ x for all 1 ≤ i ≤ k. And it is said to be overlap-free if for any factor c1 y1 c2 y2 c3 of w no letter c ∈ A and partial word y over A exist such that ci ⊂ c for all 1 ≤ i ≤ 3 and yj ⊂ y for all 1 ≤ j ≤ 2. In [99], the authors propose an efficient algorithm to test whether or not a partial word of length n is k-free. Both the time and space complexities of the algorithm are O( nk ). In case of full words, the time complexity can be


49

reduced to O(n log n) using suffix arrays [98]. In [99], the authors also give an efficient algorithm to construct in O(n) time a cube-free (and hence k-free for all k ≥ 3) partial word with n holes, and modify the algorithm in the case of a four-letter alphabet to produce such a partial word of minimal length 3n − 2 (which is the minimal length among all the possible cube-free words with n holes regardless of the alphabet over which these words are constructed). Theorem 35. [99] For k ≥ 3, there exist infinitely many k-free infinite partial words over a two-letter alphabet containing an arbitrary number of holes. Note that it is enough to show the result for k = 3. The idea of the proof is to show that there exist infinitely many cube-free infinite partial words containing exactly one hole over a two-letter alphabet. In order to do this, observe that if the underlined b in Equality 2.4 is replaced by , then the resulting partial word is still cube-free. Since there is an infinite number of occurrences of t5 in t, any replacement of the underlined b in such occurrences leads to an infinite partial word with one hole that is cube-free. The result follows since there is an infinite number of nonoverlapping occurrences of t5 in t. A surprising result holds for an alphabet of size four. Theorem 36. [99] There exists an infinite cube-free word over a four-letter alphabet in which we can randomly replace letters by holes and obtain in this way an infinite partial word that is cube-free as long as each pair of two consecutive holes are separated by at least two letters of the alphabet. Moreover, such a word does not exist over a three-letter alphabet. We discuss the concept of square-freeness of partial words in Section 2.9.1 and of overlap-freeness of partial words in Section 2.9.2. 2.9.1 Square-Freeness Let us now consider the k = 2 case. A well known result from Thue states that over a three-letter alphabet there exist infinitely many infinite words that are square-free [120, 121]. To generalize Thue’s result, we wish to find a square-free partial word with infinitely many holes, and an infinite full word that remains square-free even after replacing an arbitrary selection of letters with holes. Unfortunately, every partial word containing at least one hole and having length at least two contains a square (either a or a cannot be avoided, where a denotes a letter from the alphabet). Furthermore, it is obvious that if we replace 2n consecutive letters in a full word with holes, then the corresponding factor of the resulting partial word will be a square. Motivated by these observations, we call a word non-trivial square-free if it contains no factors of the form wk , k ≥ 2, except when |w| ∈ {1, 2} and k = 2. Notice that the cube aaa is considered to be a non-trivial square. For the sake of readability, we shall use the terms non-trivial square and square

50


interchangeably. The study of non-trivial squares is not new. In [106], several iterating morphisms are given for infinite words avoiding non-trivial squares. In particular, the authors give an infinite binary word avoiding both cubes xxx and squares yy with |y| ≥ 4 and an infinite binary word avoiding all squares except 02 , 12 , and (01)2 using a construction that is somewhat simpler than the original one from Fraenkel and Simpson [68]. Remark 1. When we introduce holes into arbitrary positions of a word, we impose the restriction that every two holes must have at least two non-hole symbols between them. With this restriction, the study of square-free partial words becomes much more subtle and interesting. Theorem 37. [31] There exists an infinite word over an eight-letter alphabet that remains square-free after replacing an arbitrary selection of its letters with holes, and none exists over a smaller alphabet. A suggested problem for investigation is the following. Let g(n) be the length of a longest binary full word containing at most n distinct squares. How does the sequence {g(n)} behave? A complete answer appears in [68]. Open problem 29 Compute the maximum number of distinct squares in a partial word with h holes of length n over a k-letter alphabet. 2.9.2 Overlap-Freeness A well known result of Thue states that over a binary alphabet there exist infinitely many overlap-free words [120, 121]. In [99], the question was raised as to whether there exist overlap-free infinite partial words, and to construct them over a binary alphabet if such exist. The following result settles this question. Theorem 38. [31] There exist overlap-free infinite partial words with one hole over a two-letter alphabet, and none exists with more than one hole. The following result relates to a three-letter alphabet. Theorem 39. [31] There exist infinitely many overlap-free infinite partial words with an arbitrary number of holes over a three-letter alphabet. For the following result, we adhere to the restriction described in Remark 1 when replacing an arbitrary selection of letters in a word with holes. Theorem 40. [31] There exists an infinite overlap-free word over a six-letter alphabet that remains overlap-free after an arbitrary selection of its letters are changed to holes, and none exists over a four-letter alphabet.


51

The case of a five-letter alphabet remains open. Open problem 30 Does there exist an infinite word over a five-letter alphabet that remains overlap-free after an arbitrary insertion of holes? Other problems are suggested in [31]. Open problem 31 Extend the concept of square-free (respectively, overlapfree or cube-free) morphism to partial words. From [31, 99], some of the properties of this kind of morphisms already start to be obvious. A further analysis might give additional properties that such morphisms should fulfill. Following the approach of Dejean [56], another interesting problem to analyze is the following. Open problem 32 Identify the exact value of k (related to k-freeness) for a given alphabet size. This value would represent the repetitiveness threshold in an n-letter alphabet. If for full words this value is known for alphabets up to size 11 and it is conjectured that for bigger size alphabets the value is n+1 n , for partial words this value has not yet been investigated.

2.10 Other Open Problems The theory of codes has been widely developed in connection with combinatorics on words [5]. In [7, 32], a new line of research was initiated by introducing pcodes in connection with combinatorics on partial words, and a theoretical framework for pcodes was developed by revisiting the theory of codes of words, as exposited in [5], starting from pcodes of partial words. Pcodes are defined in terms of the compatibility relation as follows. Definition 10. [7] Let X be a nonempty set of partial words over an alphabet A. Then X is called a pcode over A if for all positive integers m, n and partial words u1 , . . . , um , v1 , . . . , vn ∈ X, the condition u1 u2 . . . um ↑ v1 v2 . . . vn implies m = n and ui = vi for i = 1, . . . , m. An area of current interest for the study of pcodes is data communication where some information may be missing, lost, or unknown. While a code of words X does not allow two distinct decipherings of some word in X + , a pcode of partial words Y does not allow two distinct compatible decipherings in Y + . Various ways have been described for defining and analyzing pcodes. In particular, many pcodes can be obtained as antichains with respect to certain partial orderings. Adapting a graph technique related to dominoes [6, 73, 79], the pcode property was shown to be decidable for finite sets of partial words.

52


For example, the set X = {a , a b} is a pcode over {a, b}, but the set Y = {u1 , u2 , u3 , u4 } where u1 = a b, u2 = aa bba, u3 = b, and u4 = ba is not a pcode over {a, b} since u1 u3 u3 u4 u3 ↑ u2 u3 u1 is a nontrivial compatibility relation over Y . It is well known that the two-element set of words {u, v} is a code if and only if uv = vu. However, this is not true in general for partial words. For instance, the set {u, v} where u = a b and v = abbaab satisfies uv ↑ vu, but {u, v} is not a pcode since u2 ↑ v. Open problem 33 Find a necessary and sufficient condition for a twoelement set of partial words to be a pcode. Other suggested problems are the following. Open problem 34 Investigate the concept of tiling periodicity introduced recently by Karhumäki, Lifshits and Rytter’s [81]. There, the authors suggest a number of questions for further work on this new concept. Punctured languages are sets whose elements are partial words. In [91], Lischke investigated to which extent restoration of punctured languages is possible if the number of holes or the proportion of holes per word, respectively, is bounded, and studied their relationships for different boundings. The considered restoration classes coincide with similarity classes according to some kind of similarity for languages. Thus all results he can also formulate in the language of similarity. He shows some hierarchies of similarity classes for each class of the Chomsky hierarchy, and proves the existence of linear languages which are not δ-similar to any regular language for any δ < 12 . Open problem 35 For 12 ≤ δ, do there exist linear languages which are not δ-similar to any regular language? If they exist, then they must be non-slender.

References 1. A.V. Aho and M.J. Corasick. Efficient string machines, an aid to bibliographic research. Comm. ACM, 18:333–340, 1975. 2. J.P. Allouche and J. Shallit. Automatic Sequences: Theory, Applications, Generalizations. Cambridge University Press, 2003. 3. R. Assous and M. Pouzet. Une caractérisation des mots périodiques. Discrete Math., 25:1–5, 1979. 4. J. Berstel and L. Boasson. Partial words and a theorem of fine and wilf. Theoret. Comput. Sci., 218:135–141, 1999. 5. J. Berstel and D. Perrin. Theory of Codes. Academic Press, Orlando, FL, 1985. 6. F. Blanchet-Sadri. On unique, multiset, and set decipherability of three-word codes. IEEE Trans. Inform. Theory, 47:1745–1757, 2001. 7. F. Blanchet-Sadri. Codes, orderings, and partial words. Theoret. Comput. Sci., 329:177–202, 2004.


53

8. F. Blanchet-Sadri. Periodicity on partial words. Comput. Math. Appl., 47:71–82, 2004. 9. F. Blanchet-Sadri. Primitive partial words. Discrete Appl. Math., 148:195–213, 2005. 10. F. Blanchet-Sadri. Algorithmic Combinatorics on Partial Words. Chapman & Hall/CRC Press, 2007. 11. F. Blanchet-Sadri and A.R. Anavekar. Testing primitivity on partial words. Discrete Appl. Math., 155:279–287, 2007. (www.uncg.edu/mat/primitive). 12. F. Blanchet-Sadri, D. Bal, and G. Sisodia. Graph connectivity, partial words and a theorem of Fine amd Wilf. Information and Computation, to appear(www.uncg.edu/mat/research/finewilf3). 13. F. Blanchet-Sadri, D.D. Blair, and R.V. Lewis. Equations on partial words. In R. Královic and P. Urzyczyn, editors, MFCS 2006. 31st International Symposium on Mathematical Foundations of Computer Science, LNCS, vol. 4162, Springer, pp. 167-178. 14. F. Blanchet-Sadri, L. Bromberg, and K. Zipple. Tilings and quasiperiods of words. Preprint (www.uncg.edu/cmp/research/tilingperiodicity). 15. F. Blanchet-Sadri, N.C. Brownstein, and J. Palumbo. Two element unavoidable sets of partial words. In T. Harju, J. Karhumäki, and A. Lepistö, editors, DLT 2007. 11th International Conference on Developments in Language Theory, LNCS, Vol. 4588, Springer, pp. 96-107. 16. F. Blanchet-Sadri and A. Chriscoe. Local periods and binary partial words: an algorithm. Theoret. Comput. Sci., 314:189–216, 2004. (www.uncg.edu/mat/ AlgBin). 17. F. Blanchet-Sadri, E. Clader, and O. Simpson. Border correlations of partial words. Preprint (www.uncg.edu/cmp/research/bordercorrelation). 18. F. Blanchet-Sadri, K. Corcoran, and J. Nyberg. Fine and wilf’s periodicity result on partial words and consequences. In LATA 2007, 1st International Conference on Language and Automata Theory and Applications, GRLMC Report 35/07, Tarragona. 19. F. Blanchet-Sadri and M. Cucuringu. Counting primitive partial words. Preprint. 20. F. Blanchet-Sadri, M. Cucuringu, and J. Dodge. Counting unbordered partial words. Preprint. 21. F. Blanchet-Sadri, C.D. Davis, J. Dodge, R. Mercaş, and M. Moorefield. Unbordered partial words. Preprint (www.uncg.edu/mat/border). 22. F. Blanchet-Sadri and S. Duncan. Partial words and the critical factorization theorem. J. Combin. Theory Ser. A, 109:221–245, 2005. (www.uncg.edu/mat/cft). 23. F. Blanchet-Sadri, J. Fowler, J. Gafni, and K. Wilson. Combinatorics on partial word correlations. Preprint (www.uncg.edu/cmp/research/correlations2). 24. F. Blanchet-Sadri, J. Gafni, and K. Wilson. Correlations of partial words. In W. Thomas and P. Weil, editors, STACS 2007, volume 4393, pages 97–108, Berlin, 2007. (www.uncg.edu/mat/research/correlations). 25. F. Blanchet-Sadri and R.A. Hegstrom. Partial words and a theorem of fine and wilf revisited. Theoret. Comput. Sci., 270:401–419, 2002. 26. F. Blanchet-Sadri, R. Jungers, and J. Palumbo. Testing avoidability of sets of partial words is hard. Preprint. 27. F. Blanchet-Sadri, A. Kalcic, and T. Weyand. Unavoidable sets of partial words of size three. Preprint (www.uncg.edu/cmp/research/unavoidablesets2).

54


28. F. Blanchet-Sadri and D.K. Luhmann. Conjugacy on partial words. Theoret. Comput. Sci., 289:297–312, 2002. 29. F. Blanchet-Sadri, T. Mandel, and G. Sisodia. Connectivity in graphs associated with partial words. Preprint (www.uncg.edu/cmp/research/finewilf4). 30. F. Blanchet-Sadri, R. Mercaş, and G. Scott. Counting distinct squares in partial words. Preprint (www.uncg.edu/cmp/research/freeness). 31. F. Blanchet-Sadri, R. Mercaş, and G. Scott. A generalization of thue freeness for partial words. Preprint (www.uncg.edu/cmp/research/freeness). 32. F. Blanchet-Sadri and M. Moorefield. Pcodes of partial words. Preprint(www.uncg.edu/mat/pcode). 33. F. Blanchet-Sadri, T. Oey, and T. Rankin. Partial words and generalized fine and wilf’s theorem for an arbitrary number of weak periods. Preprint (www.uncg.edu/mat/research/finewilf2). 34. F. Blanchet-Sadri, B. Shirey, and G. Gramajo. Periods, partial words, and a result of guibas and odlyzko. Preprint (www.uncg.edu/mat/bintwo). 35. F. Blanchet-Sadri and N.D. Wetzler. Partial words and the critical factorization theorem revisited. Theoret. Comput. Sci., to appear. (www.uncg.edu/mat/research/cft2). 36. F. Blanchet-Sadri and J. Zhang. On the critical factorization theorem. Preprint. 37. V.D. Blondel, R. Jungers, and V. Protasov. On the complexity of computing the capacity of codes that avoid forbidden difference patterns. IEEE Trans. Inform. Theory, 52:5122–5127, 2006. 38. R.S. Boyer and J.S Moore. A fast string searching algorithm. Comm. ACM, 20:762–772, 1977. 39. D. Breslauer, T. Jiang, and Z. Jiang. Rotations of periodic strings and short superstrings. J. of Algorithms, 24:340–353, 1997. 40. J. Buhler, U. Keich, and Y. Sun. Designing seeds for similarity search in genomic dna. J. Comput. System Sci., 70:342–363, 2005. 41. P. Bylanski and D.G.W. Ingram. Digital transmission systems. IEE, 1980. 42. M.G. Castelli, F. Mignosi, and A. Restivo. Fine and wilf’s theorem for three periods and a generalization of sturmian words. Theoret. Comput. Sci., 218:83– 94, 1999. 43. Y. Césari and M. Vincent. Une caractérisation des mots périodiques. C.R. Acad. Sci. Paris, 268:1175–1177, 1978. 44. C. Choffrut and K. Culik II. On extendibility of unavoidable sets. Discrete Appl. Math., 9:125–137, 1984. 45. C. Choffrut and J. Karhumäki. Combinatorics of words. In G. Rozenberg and A. Salomaa, editors, Handbook of Formal Languages, volume 1, pages 329–438. Springer, Berlin, 1997. 46. D.D. Chu and H.S. Town. Another proof on a theorem of lyndon and schützenberger in a free monoid. Soochow J. Math., 4:143–146, 1978. 47. S. Constantinescu and L. Ilie. Generalised fine and wilf’s theorem for arbitrary number of periods. Theoret. Comput. Sci., 339:49–60, 2005. 48. M. Crochemore, C. Hancart, and T. Lecroq. Algorithms on Strings. Cambridge University Press, 2007. 49. M. Crochemore, F. Mignosi, A. Restivo, and S. Salemi. Text compression using antidictionaries. LNCS, 1644:261–270, 1999. 50. M. Crochemore and D. Perrin. Two-way string matching. J. of the ACM, 38:651–675, 1991.


55

51. M. Crochemore and W. Rytter. Text Algorithms. Oxford University Press, 1994. 52. M. Crochemore and W. Rytter. Jewels of Stringology. World Scientific, NJ, 2003. 53. A. de Luca. On the combinatorics of finite words. Theoret. Comput. Sci., 218:13–39, 1999. 54. A. de Luca and S. Varricchio. Regularity and Finiteness Conditions, volume 1, chapter 11, pages 747–810. Springer, Berlin, 1997. 55. A. de Luca and S. Varricchio. Finiteness and Regularity in Semigroups and Formal Languages. Springer, Berlin, 1999. 56. F. Dejean. Sur un théorème de thue. J. Combin. Theory Ser. A, 13:90–99, 1972. 57. P. Dömösi. Some results and problems on primitive words. In 11th International Conference on Automata and Formal Languages, 2005. 58. P. Dömösi, S. Horváth, and M. Ito. Primitive Words and Context-Free Languages. 59. J.P. Duval. Périodes locales et propagation de périodes dans un mot. Theoret. Comput. Sci., 204: 87-98, 1998 60. J.P. Duval. Périodes et répétitions des mots du monoïde libre. Theoret. Comput. Sci., 9:17–26, 1979. 61. J.P. Duval. Relationship between the period of a finite word and the length of its unbordered segments. Discrete Math., 40:31–44, 1982. 62. J.P. Duval, R. Kolpakov, G. Kucherov, T. Lecroq, and A. Lefebvre. Linear-time computation of local periods. Theoret. Comput. Sci., 326:229–240, 2004. 63. A. Ehrenfeucht, D. Haussler, and G. Rozenberg. On regularity of context-free languages. Theoret. Comput. Sci., 27:311–322, 1983. 64. A. Ehrenfeucht and D.M. Silberger. Periodicity and unbordered segments of words. Discrete Math., 26:101–109, 1979. 65. P. Erdös. Note on sequences of integers no one of which is divisible by another. J. London Math. Soc., 10:126–128, 1935. 66. M. Farach-Colton, G.M. Landau, S.C. Sahinalp, and D. Tsur. Optimal spaced seeds for approximate string matching. In L. Caires, G.F. Italiano, L. Monteiro, C. Palanidessi, and M. Yung, editors, ICALP 2005, LNCS, vol. 3580, Springer, pp. 1251-1262, 2005. 67. N.J. Fine and H.S. Wilf. Uniqueness theorems for periodic functions. In Proc. Amer. Math. Soc., volume 16, pages 109–114, 1965. 68. A.S. Fraenkel and R.J. Simpson. How many squares must a binary sequence contain? Electron. J. Combin., 2, 1995. 69. Z. Galil and J. Seiferas. Time-space optimal string matching. J. Comput. System Sci., 26:280–294, 1983. 70. M.R. Garey and D.S. Johnson. Computers and Intractability -A Guide to the Theory of NP-Completeness. Freeman, 1979. 71. L.J. Guibas and A.M. Odlyzko. Periods in strings. J. Combin. Theory Ser. A, 30:19–42, 1981. 72. D. Gusfield. Algorithms on Strings, Trees, and Sequences. Cambridge University Press, Cambridge, 1997. 73. F. Guzmán. Decipherability of codes. J. Pure Appl. Algebra, 141:13–35, 1999. 74. V. Halava, T. Harju, and L. Ilie. Periods and binary words. J. Combin. Theory Ser. A, 89:298–303, 2000.

56


75. T. Harju. Combinatorics on Words, chapter 19, pages 381–392. Springer, Berlin, 2006. 76. T. Harju and D. Nowotka. Periodicity and unbordered segments of words. Bull. Eur. Assoc. Theor. Comput. Sci. EATCS, 80:162–167, 2003. 77. T. Harju and D. Nowotka. The equation xi = y j z k in a free semigroup. Semigroup Forum, 68:488–490, 2004. 78. T. Head, G. Paun, and D. Pixton. Language Theory and Molecular Genetics, volume 2, chapter 7, pages 295–360. Springer, Berlin, 1997. 79. T. Head and A. Weber. Deciding multiset decipherability. IEEE Trans. Inform. Theory, 41:291–297, 1995. 80. J. Justin. On a paper by castelli, mignosi, restivo. Theoret. Inform. Appl., 34:373–377, 2000. 81. J. Karhumäki, Y. Lifshits, and W. Rytter. Tiling periodicity. In CPM 2007, 18th Annual Symposium on Combinatorial Pattern Matching, 2007. 82. L. Kari, G. Rozenberg, and A. Salomaa. L Systems, volume 1, chapter 7, pages 253–328. Springer, Berlin, 1997. 83. U. Keich, M. Li, B. Ma, and J. Tromp. On spaced seeds for similarity search. Discrete Appl. Math., 138:253–263, 2004. 84. D.E. Knuth, J.H. Morris, and V.R. Pratt. Fast pattern matching in strings. SIAM J. on Comput., 6:323–350, 1977. 85. R. Kolpakov and G. Kucherov. Finding approximate repetitions under hamming distance. Lecture Notes in Computer Science, 2161:170–181, 2001. 86. R. Kolpakov and G. Kucherov. Finding approximate repetitions under hamming distance. Theoret. Comput. Sci., 33:135–156, 2003. 87. G. Landau and J. Schmidt. An algorithm for approximate tandem repeats. Lecture Notes in Computer Science, 684:120–133, 1993. 88. G.M. Landau, J.P. Schmidt, and D. Sokol. An algorithm for approximate tandem repeats. J. Comput. Biology, 8:1–18, 2001. 89. P. Leupold. Languages of partial words - how to obtain them and what properties they have. Grammars, 7:179–192, 2004. 90. P. Leupold. Partial words for dna coding. LNCS, 3384:224–234, 2005. 91. G. Lischke. Restorations of punctured languages and similarity of languages. Math. Logic Quart., 52:20–28, 2006. 92. M. Lothaire. Combinatorics on Words. Cambridge University Press, Cambridge, 1997. 93. M. Lothaire. Algebraic Combinatorics on Words. Cambridge University Press, Cambridge, 2002. 94. M. Lothaire. Applied Combinatorics on Words. Cambridge University Press, Cambridge, 2005. 95. R.C. Lyndon and P.E. Schupp. Combinatorial Group Theory. Springer, Berlin, 2001. 96. R.C. Lyndon and M.P. Schützenberger. The equation am = bn cp in a free group. Michigan Math. J., 9; 289–298, 1962. 97. B. Ma, J. Tromp, and M. Li. Patternhunter: faster and more sensitive homology search. Bioinformatics, 18:440–445, 2002. 98. U. Manber and G. Myers. Suffix arrays: a new method for on-line string searches. SIAM J. on Comput., 22:935–948, 1993. 99. F. Manea and R. Mercas. Freeness of partial words. Preprint.


57

100. D. Margaritis and S. Skiena. Reconstructing strings from substrings in rounds. In FOCS 1995, 36th Annual Symposium on Foundations of Computer Science, pages 613–620, 1995. 101. E.M. McCreight. A space-economical suffix tree construction algorithm. J. of the ACM, 23:262–272, 1976. 102. F. Mignosi, A. Restivo, and S. Salemi. A periodicity theorem on words and applications. LNCS, 969:337–348, 1995. 103. F. Nicolas and E. Rivals. Hardness of optimal spaced seed design. In Apostolico, A., Crochemore, M. and Park, K., editors, CPM 2005, 16th Annual Symposium on Combinatorial Pattern Matching, LNCS, vol. 3537, Srpinger, pages 144–155, 2005. 104. L. Noé and G. Kucherov. Improved hit criteria for dna local alignment. BMC Bioinformatics, 5, 2004. 105. H. Petersen. On the language of primitive words. Theoret. Comput. Sci., 161:141–156, 1996. 106. N. Rampersad, J. Shallit, and M. w Wang. Avoiding large squares in infinite binary words. Theoret. Comput. Sci., 339:19–34, 2005. 107. G. Richomme. Sudo-lyndon. Bull. Eur. Assoc. Theor. Comput. Sci. EATCS, 92:143–149, 2007. 108. E. Rivals and S. Rahmann. Combinatorics of periods in strings. J. Combin. Theory Ser. A, 104:95–113, 2003. 109. L. Rosaz. Unavoidable languages, cuts and innocent sets of words. RAIRO Theoret. Inform. Appl., 29:339–382, 1995. 110. L. Rosaz. Inventories of unavoidable languages and the word-extension conjecture. Theoret. Comput. Sci., 201:151–170, 1998. 111. J.P. Schmidt. All highest scoring paths in weighted grid graphs and their application to finding all approximate repeats in strings. SIAM J. Comput., 27:972–992, 1998. 112. J. Setubal and J.Meidanis. Introduction to Computational Molecular Biology. PWS Publishing Company, Boston, MA, 1997. 113. A.M. Shur and Y.V. Gamzova. Periods’ interaction property for partial words. In T. Harju and J. Karhümaki, editors, Words 2003, volume 27, pages 75–82, 2003. 114. A.M. Shur and Y.V. Gamzova. Partial words and the periods’ interaction property. Izv. RAN, 68:199–222, 2004. (see Shur, A.M., Gamzova, Y.V.: Partial words and the interaction property of periods. Izv. Math. 68 (2004) 405–428, for the English translation). 115. A.M. Shur and Y.V. Konovalova. On the periods of partial words. LNCS, vol. 2136, Springer, pp. 657–665, 2001. 116. H.J. Shyr. Free Monoids and Languages. Hon Min Book Company, Taichung, Taiwan, 1991. 117. H.J. Shyr and G. Thierrin. Disjunctive languages and codes. LNCS, vol.56, Springer, pp. 171–176, 1977. 118. W.F. Smyth. Computing Patterns in Strings. Pearson Addison-Wesley, 2003. 119. J.A. Storer. Data Compression: Methods and Theory. Computer Science Press, Rockville, MD, 1988. 120. A. Thue. Uber unendliche zeichenreihen. Norske Vid. Selsk. Skr. I, Mat. Nat. Kl. Christiana, 7:1–22, 1906. Reprinted in Nagell, T., Selberg, A., Selberg, S., Thalberg, K. (eds.): Selected Mathematical Papers of Axel Thue. Oslo, Norway, Universitetsforlaget (1977) 139–158.

58


121. A. Thue. Uber die gegenseitige lage gleicher teile gewisser zeichenreihen. Norske Vid. Selsk. Skr. I, Mat. Nat. Kl. Christiana, 12:1–67, 1912. Reprinted in Nagell, T., Selberg, A., Selberg, S., Thalberg, K. (eds.): Selected Mathematical Papers of Axel Thue. Oslo, Norway, Universitetsforlaget (1977) 139–158. 122. R. Tijdeman and L. Zamboni. Fine and wilf words for any periods. Indag. Math., 14:135–147, 2003. 123. J. Ziv and A. Lempel. A universal algorithm for sequential data compression. IEEE Trans. Inform. Theory, 23:337–343, 1977.

3 Alignments and Approximate String Matching Maxime Crochemore1 and Thierry Lecroq2 1

2

Department of Computer Science, King’s College London London WC2R 2LS, UK [email protected] and Université Paris Est, France LITIS, Université de Rouen 76821 Mont-Saint-Aignan Cedex, France [email protected]

The alignments constitute one of the processes used to compare strings. They allow to visualize the resemblance between strings. This chapter deals with several methods that perform the comparison of two strings in this sense. The extension to comparison methods of more than two strings is delicate, leads to algorithms whose execution time is at least exponential, and is not treated here. The alignments are based on notions of distance or of similarity between strings. The computations are usually performed by dynamic programming. A typical example is the computation of the longest subsequence common to two strings since it shows the algorithmic techniques to implement in order to obtain an efficient computation. In particular, the reduction of the memory space obtained by one of the algorithms constitute a strategy that can often be applied in the solutions to close problems. Section 3.1.1 describes the basic techniques for the computation of the edit (or alignment) distance and the production of the associated alignments. The chosen methodology allows to highlight a global resemblance between two strings using assumptions that simplify the computation. The search for local similarities between two strings is examined in Section 3.1.2. The possibility of reduction of the memory space required by the computations is presented in Section 3.1.3 concerning the computation of longest common subsequences. We are then interested Section 3.2 in the approximate search for fixed strings. More generally, approximate pattern matching consists in locating all the occurrences of factors inside a text y, of length n, that are similar to a string x, of length m. It consists in producing the positions of the factors of y that are at distance at most k from x, for a given natural integer k. We assume

M. Crochemore and T. Lecroq: Alignments and Approximate String Matching, Studies in Computational Intelligence (SCI) 113, 59–93 (2008) c Springer-Verlag Berlin Heidelberg 2008 www.springerlink.com

60

Maxime Crochemore and Thierry Lecroq

in the rest that k < m ≤ n. We consider the edit distance for measuring the approximation. The edit distance between two strings u and v, that are not necessarily of same length, is the minimal cost of the elementary edit operations between these two strings. The method at the basis for approximate pattern matching is a natural extension of the alignment method by dynamic programming of Section 3.1. It can be improved by using a restricted notion of distance obtained by considering the minimal number of edit operations rather than the sum of their costs. With this distance, the problem is known under the name of approximate pattern matching with k differences. Section 3.2 presents several solutions. The Hamming distance between two strings u and v of same length is the number of positions in which the two strings possess different letters. With this distance, the problem is known under the name of approximate pattern matching with k mismatches. It is treated in Section 3.3. We examine then, in Section 3.4, the case of the search for short patterns. This gives excellent practical results and is very flexible as long as the conditions of its utilization are fulfilled. The Shift-Or algorithm of Section 3.4 is a method that is both very fast in practice and very easy to implement. The method is flexible enough to be adapted to a wide range of similar approximate matching problems.

3.1 Alignments An alignment of two strings x and y of length m and n respectively consists in aligning their symbols on vertical lines. Formally an alignment of two strings x, y ∈ V is a word w on the alphabet (V ∪ {λ}) × (V ∪ {λ}) \ ({(λ, λ)} (λ is the empty word) whose projection on the first component is x and whose projection of the second component is y. Thus an alignment w = (x0 , y 0 )(x1 , y 1 ) · · · (xp−1 , y p−1 ) of length p is such that x = x0 x1 · · · xp−1 and y = y 0 y 1 · · · y p−1 with xi ∈ V ∪ {λ} and y i ∈ V ∪ {λ} for 0 ≤ i ≤ p − 1. The alignment is represented as follows x0 x1 · · · xp−1 y 0 y 1 · · · y p−1 with the symbol − instead of the symbol λ. An example is presented in Fig. 3.1. ACG−−A ATG C T A Fig. 3.1. Alignment of ACGA and ATGCTA.

3 Alignments and Approximate String Matching

61

3.1.1 Global alignment A global alignment of two strings x and y can be obtained by computing the distance between x and y. The notion of distance between two strings is widely used to compare files. The diff command of UNIX operating system implements an algorithm based on this notion, in which lines of the files are treated as symbols. The output of a comparison made by diff gives the minimum number of operations (substitute a symbol, insert a symbol, or delete a symbol) to transform one file into the other. Let us define the edit distance between two strings x and y as follows: it is the minimum number of elementary edit operations that enable to transform x into y. The elementary edit operations are: • the substitution of a character of x at a given position by a character of y, • the deletion of a character of x at a given position, • the insertion of a character of y in x at a given position. A cost is associated with each elementary edit operation. For a, b ∈ V : •

Sub(a, b) denotes the cost of the substitution of the character a by the character b, • Del(a) denotes the cost of the deletion of the character a, • Ins(a) denotes the cost of the insertion of the character a. This means that the costs of the edit operations are independent of the positions where the operations occur. We can now define the edit distance of two strings x and y by edit(x, y) = min{cost of γ | γ ∈ Γx,y } where Γx,y is the set of all the sequences of edit operations that transform x into y, and the cost of an element γ ∈ Γx,y is the sum of the costs of its elementary edit operations. In order to compute edit(x, y) for two strings x and y of length m and n respectively, we make use of a two-dimensional table T of m + 1 rows and n + 1 columns such that T [i, j] = edit(x[0 . . i], y[0 . . j]) for i = 0, . . . , m − 1 and j = 0, . . . , n − 1. It follows edit(x, y) = T [m − 1, n − 1]. The values of the table T can be computed by the following recurrence formula: T [−1, −1] = 0 , T [i, −1] = T [i − 1, −1] + Del(x[i]) , T [−1, j] = T [−1, j − 1] + Ins(y[j]) , ⎧ ⎪ ⎨T [i − 1, j − 1] + Sub(x[i], y[j]) , T [i, j] = min T [i − 1, j] + Del(x[i]) , ⎪ ⎩ T [i, j − 1] + Ins(y[j]) ,

62


for i = 0, 1, . . . , m − 1 and j = 0, 1, . . . , n − 1. The value at position [i, j] in the table T only depends on the values at the three neighbor positions [i − 1, j − 1], [i − 1, j] and [i, j − 1]. The direct application of the above recurrence formula gives an exponential time algorithm to compute T [m − 1, n − 1]. However the whole table T can be computed in quadratic time, technique known as “dynamic programming”. This is a general technique that is used to solve the different kinds of alignments. The computation of the table T proceeds in two steps. First it initializes the first column and first row of T , this is done by a call to a generic function Margin which is an argument of the algorithm and that depends on the kind of alignment that is considered. Second it computes the remaining values of T , that is done by a call to a generic function Formula which is an argument of the algorithm and that depends on the kind of alignment that is considered. Generic-DP(x, m, y, n, Margin, Formula) 1 Margin(T, x, m, y, n) 2 for j ← 0 to n − 1 do 3 for i ← 0 to m − 1 do 4 T [i, j] ← Formula(T, x, i, y, j) 5 return T

Fig. 3.2. Computation of the table T by dynamic programming.

Computing a global alignment of x and y can be done by a call to Generic-DP with the following arguments (x, m, y, n, Global-margin, Global-formula) (see Fig. 3.2, 3.3 and 3.4). The computation of all the values of the table T can thus be done in quadratic space and time: O(m × n). Global-margin(T, x, m, y, n) 1 T [−1, −1] ← 0 2 for i ← 0 to m − 1 do 3 T [i, −1] ← T [i − 1, −1] + Del(x[i]) 4 for j ← 0 to n − 1 do 5 T [−1, j] ← T [−1, j − 1] + Ins(y[j]) Fig. 3.3. Margin initialization for the computation of a global alignment.

An optimal alignment (with minimal cost) can then be produced by a call to the function One-alignment(T, x, m − 1, y, n − 1) (see Fig. 3.5). It consists in tracing back the computation of the values of the table T from


63

Global-formula(T, x, i, y, j) 1 return min{T [i − 1, j − 1] + Sub(x[i], y[j]), T [i − 1, j] + Del(x[i]), T [i, j − 1] + Ins(y[j])} Fig. 3.4. Computation of T [i, j] for a global alignment.

position [m − 1, n − 1] to position [−1, −1]. At each cell [i, j] the algorithm determines among the three values T [i − 1, j − 1] + Sub(x[i], y[j]), T [i − 1, j] + Del(x[i]) and T [i, j − 1] + Ins(y[j]) which has been used to produce the value of T [i, j]. If T [i − 1, j − 1] + Sub(x[i], y[j]) has been used it adds (x[i], y[j]) to the optimal alignment and proceeds recursively with the cell at [i − 1, j − 1]. If T [i − 1, j] + Del(x[i]) has been used it adds (x[i], −) to the optimal alignment and proceeds recursively with cell at [i − 1, j]. If T [i, j − 1] + Ins(y[j]) has been used it adds (−, y[j]) to the optimal alignment and proceeds recursively with cell at [i, j − 1]. Recovering all the optimal alignments can be done by a similar technique. An example of global alignment is given in Fig. 3.6. One-alignment(T, x, i, y, j) 1 if i = −1 and j = −1 then 2 return (λ, λ) 3 else if i = −1 then 4 return One-alignment(T, x, −1, y, j − 1) · (λ, y[j]) 5 else if j = −1 then 6 return One-alignment(T, x, i − 1, y, −1) · (x[i], λ) 7 else if T [i, j] = T [i − 1, j − 1] + Sub(x[i], y[j]) then 8 return One-alignment(T, x, i − 1, y, j − 1) · (x[i], y[j]) 9 else if T [i, j] = T [i − 1, j] + Del(x[i]) then 10 return One-alignment(T, x, i − 1, y, j) · (x[i], λ) 11 else return One-alignment(T, x, i, y, j − 1) · (λ, y[j]) Fig. 3.5. Recovering an optimal alignment.

3.1.2 Local alignment A local alignment of two strings x and y consists in finding the segment of x that is closer to a segment of y. The notion of distance used to compute global alignments cannot be used in that case since the segments of x closer to segments of y would only be the empty segment or individual characters. This is why a notion of similarity is used based on a scoring scheme for edit operations.

64

Maxime Crochemore and Thierry Lecroq T i -1 0 1 2 3

-1 y[j] x[i] 0 A 1 C 2 G 3 A 4 j

0 A 1 0 1 2 3

1 T 2 1 1 2 3

2 G 3 2 2 1 2

3 C 4 3 2 2 2

4 T 5 4 3 3 3

5 A 6 5 4 4 3

Fig. 3.6. Global alignment of ACGA and ATGCTA. The values of the above table have been obtained with the following unitary costs: Sub(a, b) = 1 if a = b and Sub(a, a) = 0, Del(a) = Ins(a) = 1 for a, b ∈ V .

A score (instead of a cost) is associated with each elementary edit operation. For a, b ∈ V : • SubS (a, b) denotes the score of substituting the character b for the character a, • DelS (a) denotes the score of deleting the character a, • InsS (a) denotes the score of inserting the character a. This means that the scores of the edit operations are independent of the positions where the operations occur. For two characters a and b, a positive value of SubS (a, b) means that the two characters are close to each other, and a negative value of SubS (a, b) means that the two characters are far apart. We can now define the edit score of two strings x and y by sco(x, y) = max{score of γ | γ ∈ Γx,y } where Γx,y is the set of all the sequences of edit operations that transform x into y and the score of an element σ ∈ Γx,y is the sum of the scores of its elementary edit operations. An optimal local alignment between the strings x and y is a pair of strings (u, v) for which u is a factor of x, v is a factor of y and sco(u, v) is maximal. For performing its computation, we consider a table T defined, for i = −1, 0, . . . , m − 1 and j = −1, 0, . . . , n − 1, by: T [i, j] is the maximal similarity between a suffix of x[0 . . i] and a suffix of y[0 . . j]. Or also T [i, j] = max{sco(x[ . . i], y[k . . j]) | 0 ≤ ≤ i and 0 ≤ k ≤ j} ∪ {0} is the score of the local alignment in [i, j]. The values of the table T can be computed by the following recurrence formula:


65

T [−1, −1] = 0 , T [i, −1] = 0 , T [−1, j] = 0 ,

⎧ T [i − 1, j − 1] + SubS (x[i], y[j]) , ⎪ ⎪ ⎪ ⎨T [i − 1, j] + Del (x[i]) , S T [i, j] = max ⎪ T [i, j − 1] + Ins S (y[j]) , ⎪ ⎪ ⎩ 0,

for i = 0, 1, . . . , m − 1 and j = 0, 1, . . . , n − 1. Computing the values of T for a local alignment of x and y can be done by a call to Generic-DP with the following arguments (x, m, y, n, Local-margin, Local-formula) in O(mn) time and space complexity (see Fig. 3.2, 3.7 and 3.8). Recovering a local alignment can be done in a way similar to what is done in the case of a global alignment (see Fig. 3.5) but the trace back procedure must start at a position of a maximal value in T rather than at position [m − 1, n − 1]. An example of local alignment is given in Fig. 3.9. Local-margin(T, x, m, y, n) 1 T [−1, −1] ← 0 2 for i ← 0 to m − 1 do 3 T [i, −1] ← 0 4 for j ← 0 to n − 1 do 5 T [−1, j] ← 0 Fig. 3.7. Margin initialization for computing a local alignment.

Local-formula(T, x, i, y, j) 1 return max{T [i − 1, j − 1] + SubS (x[i], y[j]), T [i − 1, j] + DelS (x[i]), T [i, j − 1] + InsS (y[j]), 0}

Fig. 3.8. Recurrence formula for computing a local alignment.

3.1.3 Longest Common Subsequence of Two Strings A subsequence of a string x is obtained by deleting zero or more characters from x. More formally w[0 . . i − 1] is a subsequence of x[0 . . m − 1] if there

66


(a)

T i −1 0 1 2 3 4 5 6 7 8

(b)

AWACQ-GK AW-CQPGK

−1 y[j] x[i] 0 E 0 A 0 W 0 A 0 C 0 Q 0 G 0 K 0 L 0 j

0 E 0 1 0 0 0 0 0 0 0 0

1 R 0 0 0 0 0 0 0 0 0 0

2 D 0 0 0 0 0 0 0 0 0 0

3 A 0 0 1 0 1 0 0 0 0 0

4 W 0 0 0 2 1 0 0 0 0 0

5 C 0 0 0 1 0 2 1 0 0 0

6 Q 0 0 0 0 0 1 3 2 1 0

7 P 0 0 0 0 0 0 2 1 0 0

8 G 0 0 0 0 0 0 1 3 2 1

9 K 0 0 0 0 0 0 0 2 4 3

10 W 0 0 0 1 0 0 0 1 3 2

11 Y 0 0 0 0 0 0 0 0 2 1

Fig. 3.9. Computation of an optimal local alignment of x = EAWACQGKL and y = ERDAWCQPGKWY with scores: SubS (a, a) = 1, SubS (a, b) = −3 and DelS (a) = InsS (a) = −1 for a, b ∈ V , a = b. (a) Values of table T . (b) The corresponding alignment.

exists an increasing sequence of integers (kj | j = 0, . . . , i − 1) such that for 0 ≤ j ≤ i − 1, w[j] = x[kj ]. We say that a string is an lcs(x, y) if it is a longest common subsequence of the two strings x and y. Note that two strings can have several longest common subsequences. Their common length is denoted by llcs(x, y). A brute-force method to compute an lcs(x, y) would consist in computing all the subsequences of x, checking if they are subsequences of y, and keeping the longest ones. The string x of length m has potentially 2m subsequences, and so this method could take O(2m ) time, which is impractical even for fairly small values of m. However llcs(x, y) can be computed with a two-dimensional table T by the following recurrence formula: T [−1, −1] = 0 , T [i, −1] = 0 , T [−1, j] = 0 , T [i − 1, j − 1] + 1 T [i, j] = max{T [i − 1, j], T [i, j − 1]}

if x[i] = y[j], otherwise,

for i = 0, 1, . . . , m−1 and j = 0, 1, . . . , n−1. Then T [i, j] = llcs(x[0 . . i], y[0 . . j]) and llcs(x, y) = T [m − 1, n − 1]. Computing T [m − 1, n − 1] can be done by a call to Generic-DP with the following arguments (x, m, y, n, Local-margin, Lcs-formula) in O(mn) time and space complexity (see Fig. 3.2, 3.7 and 3.10).


67

Formula-lcs(T, x, i, y, j) 1 if x[i] = y[j] then 2 return T [i − 1, j − 1] + 1 3 else return max{T [i − 1, j], T [i, j − 1]} Fig. 3.10. Recurrence formula for computing an lcs.

It is possible afterward to trace back a path from position [m − 1, n − 1] to exhibit an lcs(x, y) in a similar way as for producing a global alignment (see Fig. 3.5). An example is presented in Fig. 3.11. T i −1 0 1 2 3 4

−1 y[j] x[i] 0 A 0 G 0 C 0 G 0 A 0 j

0 C 0 0 0 1 1 1

1 A 0 1 1 1 1 2

2 G 0 1 2 2 2 2

3 A 0 1 2 2 2 3

4 T 0 1 2 2 2 3

5 A 0 1 2 2 2 3

6 G 0 1 2 2 3 3

7 A 0 1 2 2 3 4

8 G 0 1 2 2 3 4

Fig. 3.11. The value T [4, 8] = 4 is llcs(x, y) for x = AGCGA and y = CAGATAGAG. String AGGA is an lcs of x and y.

3.1.4 Reducing the Space: Hirschberg Algorithm If only the length of an lcs(x, y) is required, it is easy to see that only one row (or one column) of the table T needs to be stored during the computation. The space complexity becomes O(min(m, n)) as can be checked on the algorithm of Fig. 3.12. The Hirschberg algorithm computes an lcs(x, y) in linear space and not only the value llcs(x, y). The computation uses the algorithm of Fig. 3.12. Let us define T ∗ [i, n] = T ∗ [m, j] = 0, for 0 ≤ i ≤ m and 0 ≤ j ≤ n T ∗ [m − i, n − j] = llcs((x[i . . m − 1])R , (y[j . . n − 1])R ) for 0 ≤ i ≤ m − 1 and 0 ≤ j ≤ n − 1 and

M (i) = max {T [i, j] + T ∗ [m − i, n − j]} 0≤j C[i] then 7 C[i] ← last 8 else if last < C[i] then 9 last ← C[i] 10 else if x[i] = y[j] then 11 C[i] ← C[i] + 1 12 last ← last + 1 13 return C

Fig. 3.12. O(m)-space algorithm to compute llcs(x, y). Hirschberg(x, m, y, n) 1 if m = 0 then 2 return λ 3 else if m = 1 then 4 if x[0] ∈ y then 5 return x[0] 6 else return λ 7 else j ← n/2 8 C ← LLCS(x, m, y[0 . . j − 1], j) 9 C ∗ ← LLCS(xR , m, y[j . . n − 1]R , n − j) 10 k ←m−1 11 M ← C[m − 1] + C ∗ [m − 1] 12 for j ← −1 to m − 2 do 13 if C[j] + C ∗ [j] > M then 14 M ← C[j] + C ∗ [j] 15 k←j 16 return Hirschberg(x[0 . . k − 1], k, y[0 . . j − 1], j)· Hirschberg(x[k . . m − 1], m − k, y[j . . n − 1], n − j) Fig. 3.13. O(min(m, n))-space computation of lcs(x, y).

M (i) = T [m − 1, n − 1],

for 0 ≤ i < m .

In the algorithm shown in Fig. 3.13 the integer j is chosen as n/2. After T [i, j−1] and T ∗ [m−i, n−j] (0 ≤ i < m) are computed, the algorithm finds an integer k such that T [i, k]+T ∗ [m−i, n−k] = T [m−1, n−1]. Then, recursively, it computes an lcs(x[0 . . k−1], y[0 . . j −1]) and an lcs(x[k . . m−1], y[j . . n−1]), and concatenates them to get an lcs(x, y).


69

The running time of the Hirschberg algorithm is still O(mn) but the amount of space required for the computation becomes O(min(m, n)) instead of being quadratic when computed by dynamic programming.

3.2 Approximate String Matching with Differences Approximate string matching is the problem of finding all approximate occurrences of a pattern x of length m in a text y of length n. Approximate occurrences of x are segments of y that are close to x according to a specific distance: the distance between segments and x must be not greater than a given integer k. With the edit distance (or Levenshtein distance), the problem is known as approximate string matching with k differences. The standard solutions to solve this problem consist in using the dynamic programming technique introduced in Section 3.1. We describe three variations around this technique. Dynamic programming We first examine a problem a bit more general for which the cost of the edit operations is not necessarily one unit. Aligning x with a factor of y amounts to align x with a prefix of y considering that the insertion of any number of letters of y at the beginning of x is not penalizing. With the table T of Section 3.1.1 we check that, to solve the problem, it is sufficient then to initialize to zero the values of the first line of the table. The positions of the occurrences are then associated with all the values of the last line of the table that are less than k. To perform the search for approximate factors, we utilize the table R defined by R[i, j] = min{edit(x[0 . . i], y[ . . j]) | = 0, 1, . . . , j + 1}, for i = −1, 0, . . . , m − 1 and j = −1, 0, . . . , n − 1, where edit is the edit distance of Section 3.1. The computation of the values of the table R utilizes the recurrence relations that follow. For i = 0, 1, . . . , m − 1 and j = 0, 1, . . . , n − 1, we have: R[−1, −1] = 0, R[i, −1] = R[i − 1, −1] + Del(x[i]), R[−1, j] = 0, ⎧ ⎪ ⎨R[i − 1, j − 1] + Sub(x[i], y[j]), R[i, j] = min R[i − 1, j] + Del(x[i]), ⎪ ⎩ R[i, j − 1] + Ins(y[j]).

(3.1)

70


K-diff-DP(x, m, y, n, k) 1 R[−1, −1] ← 0 2 for i ← 0 to m − 1 do 3 R[i, −1] ← i + Del(x[i]) 4 for j ← 0 to n − 1 do 5 R[−1, j] ← 0 6 for i ← 0 to m − ⎧1 do ⎪ ⎨R[i − 1, j − 1] + Sub(x[i], y[j]) 7 R[i, j] ← min R[i − 1, j] + Del(x[i]) ⎪ ⎩ R[i, j − 1] + Ins(y[j]) 8 if R[m − 1, j] ≤ k then 9 Output(j)

Fig. 3.14. Approximate string matching with k differences by dynamic programming.

(a)

R i −1 0 1 2 3 4

−1 0 y[j] C x[i] 0 0 G 1 1 A 2 2 T 3 3 A 4 4 A 5 5

j

1 A 0 1 1 2 3 4

2 G 0 0 1 2 3 4

3 A 0 1 0 1 2 3

4 T 0 1 1 0 1 2

5 A 0 1 1 1 0 1

6 A 0 1 1 2 1 0

7 G 0 0 1 2 2 1

8 A 0 1 0 1 2 2

9 G 0 0 1 1 2 3

10 A 0 1 0 1 1 2

11 A 0 1 1 1 1 1

GATAA CAGAT-AAGAGAA

GATAA CAGATAAGAGAA

GATAA CAGATA-AGAGAA

-GATAA CAGATAAGAGAA

GATAA CAG-ATAAGAGAA

GATAACAGATAAGAGAA

(b)

GATAA CAGATAAGAGAA Fig. 3.15. Search for x = GATAA in y = CAGATAAGAGAA with one difference, considering unit costs for the edit operations. (a) Values of table R. (b) The seven alignments of x with factors of y ending at positions 5, 6, 7 and 11 on y. We note that the fourth and sixth alignments give no extra information comparing to the second.


71

The search algorithm K-diff-DP whose code is given in Fig. 3.14 and that translates the recurrence of the previous proposition performs the approximate search. An example is given in Fig. 3.15. We note that the space used by the algorithm K-diff-DP can be reduced to a single column by reproducing the technique of Section 3.1.3. Besides, this technique is implemented by the algorithm K-diff-cut-off (see Fig. 3.16). As a conclusion we get the following result. The operation K-diff-DP(x, m, y, n, k) that finds the factors u of y for which edit(u, x) ≤ k (edit edit distance with any costs) executes in time O(m× n) and can be realized in space O(m). Diagonal monotony In the rest of the section, we consider that the costs of the edit operations are unitary. This is a simple case for which we can describe more efficient computation strategies that those described above. The restriction allows to state a property of monotony on the diagonals that is at the basis of the presented variations. Since we assume that Sub(a, b) = Del(a) = Ins(b) = 1 for a, b ∈ V , a = b, the recurrence relation 3.1 simplifies and becomes R[−1, −1] = 0, R[i, −1] = i + 1, R[−1, j] = 0, ⎧ R[i − 1, j − 1] if x[i] = y[j], ⎪ ⎪ ⎪ ⎨R[i − 1, j − 1] + 1 if x[i] = y[j], R[i, j] = min ⎪ R[i − 1, j] + 1, ⎪ ⎪ ⎩ R[i, j − 1] + 1.

(3.2)

for i = 0, 1, . . . , m − 1 and j = 0, 1, . . . , n − 1. A diagonal d of the table R consists of the positions [i, j] for which j − i = d (−m ≤ d ≤ n). The property of diagonal monotony expresses that the sequence of values on each diagonal of the table R increases with i and that the difference between two consecutive values is at most one (see Fig. 3.15). Before formally stating the property, we give intermediate results. The first result means that two adjacent values on a column of the table R differ from at most one. The second result is symmetrical to the first one for the lines of R. For each position j on the string y, we have −1 ≤ R[i, j] − R[i − 1, j] ≤ 1 for i = 0, 1, . . . , m − 1. For each position i on the string x, we have

72


−1 ≤ R[i, j] − R[i, j − 1] ≤ 1 for j = 0, 1, . . . , n − 1. We now can state the result concerning the property of monotony on the diagonals announced above: For i = 0, 1, . . . , m − 1 and j = 0, 1, . . . , n − 1, we have: R[i − 1, j − 1] ≤ R[i, j] ≤ R[i − 1, j − 1] + 1. Partial computation The property of monotony on the diagonals is exploited in the following way to avoid to compute some values in the table R that are greater than k, the maximal number of allowed differences. The values are still computed column by column, in the increasing order of the positions on y and for each column in the increasing order of the positions on x, as done by the algorithm Kdiff-DP. When a value equal to k + 1 is found in a column, it is useless to compute the next values in the same diagonal since those latter are all strictly greater than k. For pruning the computation, we keep, in each column, the largest position at which is found an admissible value. If qj is this position, for a given column j, only the values of lines −1 to qj + 1 are computed in the next column (of index j + 1). The algorithm K-diff-cut-off, given in Fig. 3.16, realizes this method. K-diff-cut-off(x, m, y, n, k) 1 for i ← −1 to k − 1 do 2 C1 [i] ← i + 1 3 p←k 4 for j ← 0 to n − 1 do 5 C2 [−1] ← 0 6 for i ← 0 to p do 7 if x[i] = y[j] then 8 C2 [i] ← C1 [i − 1] 9 else C2 [i] ← min{C1 [i − 1], C2 [i − 1], C1 [i]} + 1 10 C1 ← C2 11 while C1 [p] > k do 12 p←p−1 13 if p = m − 1 then 14 Output(j) 15 p ← min{p + 1, m − 1} Fig. 3.16. Approximate string matching with k differences by partial computation.

The column −1 is initialized until line k − 1 that corresponds to the value k. For the next columns of index j = 0, 1, . . . , n − 1, the values are computed until line


pj = min

73

1 + max{i | 0 ≤ i ≤ m − 1 and R[i, j − 1] ≤ k}, m − 1.

The table R is implemented with the help of two tables C2 and C1 that allow to memorize respectively the values of the column during the computation and the values of the previous column. The process is similar to the one that is used in the algorithm LLCS of Section 3.1.4. At each iteration of the loop Lines 7–10, we have: C1 [i − 1] = R[i − 1, j − 1], C2 [i − 1] = R[i − 1, j], C1 [i] = R[i, j − 1]. We compute then the value C2 [i] that is also R[i, j]. We find thus at this line an implementation of Relation 3.2. An example of computation is given in Fig. 3.17. R i −1 0 1 2 3 4

−1 0 y[j] C x[i] 0 0 G 1 1 2 A T A A j

1 A 0 1 1

2 G 0 0 1 2

3 A 0 1 0 1

4 T 0 1 1 0 1

5 A 0 1 1 1 0 1

6 A 0 1 1 2 1 0

7 G 0 0 1 2 2 1

8 A 0 1 0 1 2 2

9 G 0 0 1 1 2

10 A 0 1 0 1 1

11 A 0 1 1 1 1 1

Fig. 3.17. Pruning of the computation of the dynamic programing table for the search for x = GATAA in y = CAGATAAGAGAA with one difference (see Figure 3.15). We notice that seventeen values of table R (those that are not shown) are not useful for the computation of occurrences of approximate factors of x in y.

We note that the memory space used by the algorithm K-diff-cut-off is O(m). Indeed, only two columns are memorized. This is possible since the computation of the values for one column only needs those of the previous column. Diagonal computation The variant of search with differences that we consider now consists in computing the values of the table R according to the diagonals and by taking into account the property of monotony. The interesting positions on the diagonals are those where changes of values happen. These changes are incrementation because of the chosen distance. For a number q of differences and a diagonal d, we denote by L[q, d] the index i of the line on which R[i, j] = q for the last time on the diagonal

74

Maxime Crochemore and Thierry Lecroq R i −1 0 1 2 3 4

j x[i] G A T A A

−1 0 y[j] C

1 A

2 G

3 A

4 T 0

5 A

6 A

7 G

8 A

9 G

10 11 A A

1 1 2 2 3

Fig. 3.18. Values of table R on diagonal 5 for the approximate search for x = GATAA in y = CAGATAAGAGAA. The last occurrences of each value on the diagonal are in gray. The lines where they occur are stored in table L by the algorithm of diagonal computation. We thus have L[0, 5] = −1, L[1, 5] = 1, L[2, 5] = 3, L[3, 5] = 4.

j − i = d. The idea of the definition of L[q, d] is shown in Fig. 3.18. Formally, for q = 0, 1, . . . , k and d = −m, −m + 1, . . . , n − m, we have L[q, d] = i if and only if i is the maximal index, −1 ≤ i < m, for which there exists an index j, −1 ≤ j < n, with R[i, j] ≤ q and j − i = d. In other words, for fixed q, the values L[q, d] mark the lowest borderline of the values less than q in the table R (gray values in Fig. 3.19). The definition of L[q, d] implies that q is the smallest number of differences between x[0 . . L[q, d]] and a factor of the text ending at position d + L[q, d] on y. It moreover implies that the letters x[L[q, d] + 1] and y[d + L[q, d] + 1] are different when they are defined. The values L[q, d] are computed by iteration on d, for q going from 0 to k + 1. The principle of the computation relies on Recurrence 3.2 and the above statements. A simulation of the computation on the table R is presented in Fig. 3.19. For the approximate pattern matching with k differences problem, only the values L[q, d] for which q ≤ k are necessary. If L[q, d] = m − 1, it means that there is an occurrence of the string x at the diagonal d with at most q differences. The occurrence ending at position d + m − 1, this is only valid if d + m ≤ n. We get another approximate occurrences at the end of y when L[q, d] = i and d+i = n−1; in this case the number of differences is q+m−1−i. The algorithm K-diff-diag, given in Fig. 3.21 performs the approximate search for x in y by computing the values L[q, d]. It uses the function lcp where lcp(u, v) gives the length of the longest common prefix of two strings u and v. Let us note that the first possible occurrence of an approximate factor of x in y can end at position m − 1 − k on y, this corresponds to diagonal −k. The last possible occurrence starts at position n − m + k on y, this corresponds


(a)

(b)

R i −1 0 1 2 3 4

−1 0 y[j] C x[i] 0 0 G A T A A

1 A 0

R i −1 0 1 2 3 4

−1 0 y[j] C x[i] 0 0 G 1 1 A T A A

1 A 0 1 1

j

j

2 G 0 0

3 A 0

4 T 0

5 A 0

6 A 0

7 G 0 0

0

8 A

9 G

10 11 A A

8 A

9 G

10 11 A A

0 1

1

75

0 0 0 0

2 G 0 0 1

3 A 0 1 0 1

4 T 0 1 1 0 1

5 A 0 1 1 1 0 1

6 A 0 1 1 1 0

7 G 0 0 1

1 1

1

Fig. 3.19. Simulation of the diagonal computation for the search for x = GATAA in y = CAGATAAGAGAA with one difference (see Figure 3.15). (a) Values computed during the first step (Lines 8–13 for q = 0 of Algorithm L-diff-diag); they detect the occurrence of x at right position 6 on y (since R[4, 6] = 0). (b) Values computed during the second step (Lines 8–13 for q = 1); they indicate the approximate factors of x with one difference at right positions 5, 7 and 11 on y (since R[4, 5] = R[4, 7] = R[4, 11] = 1).

to diagonal n − m + k. Thus only diagonals going from −k to n − m + k are considered during the computation (the initialization is also done on the diagonals −k − 1 and n − m + k + 1 to simplify the writing of the algorithm). Fig. 3.20 shows the table L obtained on the example of Fig. 3.15. d −2 −1 0 1 q = −1 −2 −2 −2 q = 0 −1 −1 −1 −1 0 1 4 q=1

2 −2 4 4

3 −2 −1 4

4 −2 −1 1

5 −2 −1 1

6 −2 −1 2

7 8 9 −2 −2 −2 1 −1 4

Fig. 3.20. Values of table L of the diagonal computation when x = GATAA, y = CAGATAAGAGAA and k = 1. Lines q = 0 and q = 1 correspond to a state of the computation simulated on table R in Figure 3.19. Values 4 = |GATAA| − 1 on line q = 1 indicate the presence of occurrences of x with at most one difference ending at positions 1 + 4, 2 + 4, 3 + 4 and 7 + 4 on y.

The algorithm K-diff-diag computes the table L. For every string x of length m, every string y of length n and every integer k such that k < m ≤ n, the operation K-diff-diag(x, m, y, n, k) computes the approximate occurrences of x in y with at most k differences.

76


K-diff-diag(x, m, y, n, k) 1 for d ← −1 to n − m + k + 1 do 2 L[−1, d] ← −2 3 for q ← 0 to k − 1 do 4 L[q, −q − 1] ← q − 1 5 L[q, −q − 2] ← q − 1 6 for q ← 0 to k do 7 for d ← −q ⎧ to n − m + k − q do ⎪ ⎨L[q − 1, d − 1] 8 ← max L[q − 1, d] + 1 ⎪ ⎩ L[q − 1, d + 1] + 1 9 ← min{, m − 1} 10 L[q, d] ← + |lcp(x[ + 1 . . m − 1], y[d + + 1 . . n − 1])| 11 if L[q, d] = m − 1 or d + L[q, d] = n − 1 then 12 Output(d + m − 1) Fig. 3.21. Approximate string matching with k differences by diagonals.

In the way that the algorithm K-diff-diag is described, the memory space for the computation is principally used by the table L. We note that it is sufficient to memorize a single line to correctly perform the computation, this gives an implementation in space O(n). It is however possible to reduce the space to O(m) obtaining a space comparable to algorithm K-diff-cut-off. If the computation of lcp(u, v) is realized in time O(|lcp(u, v)|), the algorithm K-diff-diag executes in time O(m × n). But it is possible to prepare the strings x and y in such a way that any lcp(u, v) query is answered in constant time. For this, we utilize the suffix tree, of the string z = x$y where $ ∈ alph(y). The string w = lcp(x[ + 1 . . m − 1], y[d + + 1 . . n − 1]) is nothing else but the string lcp(x[ + 1 . . m − 1]$y, y[d + + 1 . . n − 1]) since $ ∈ alph(y). Let f and g be the external nodes of the suffix tree associated with suffixes of x[ + 1 . . m − 1]$y and y[d + + 1 . . n − 1] of the string z. Their common prefix of maximal length is then the label of the path leading from the initial state to the lowest node that is a common ancestor to f and g. This reduces the computation of w to the computation of this node. The problem of the common ancestor that we are interested in here is the one for which the tree is static. A linear preprocessing of the tree allows to get a response in constant time to the queries (see notes). The consequence of this result is that on a fixed alphabet, after preparation of the strings x and y in linear time, it is possible to execute the algorithm K-diff-diag in time O(k × n).


77

3.3 Approximate String Matching with Mismatches In this section, we are interested in the search for all the occurrences of a string x of length m in a string y of length n with at most k mismatches (k ∈ N , k < m ≤ n). The Hamming distance between two strings u and v of same length is the number of mismatches between u and v and is defined by: Ham(u, v) = card{i | u[i] = v[i], i = 0, 1, . . . , |u| − 1}. The problem can then be expressed as the search for all the positions j = 0, 1, . . . , n − m on y that satisfy the inequality Ham(x, y[j . . j + m − 1]) ≤ k. 3.3.1 Search automaton A natural solution to this problem consists in using an automaton that recognizes the language V ∗ {w | Ham(x, w) ≤ k}. To do this, we can consider the non-deterministic automaton defined as follows: •

each state is a pair (, i) where is the level of the state and i is its depth, with 0 ≤ ≤ k, −1 ≤ i ≤ m − 1 and ≤ i + 1; • the initial state is (0, −1); • the terminal states are of the form (, m − 1) with 0 ≤ ≤ k; • the transitions are, for 0 ≤ ≤ k, 0 ≤ i < m − 1 and a ∈ V , either of the form ((0, −1), a, (0, −1)), or of the form ((, i), x[i + 1], (, i + 1)), or of the form ((, i), a, ( + 1, i + 1)) if a = x[i + 1] and 0 ≤ ≤ k − 1.

The automaton possesses k + 1 levels, each level allowing to recognize the prefixes of x with mismatches. The transitions of the form ((, i), a, (, i + 1)) correspond to the equality of letters while those of the form ((, i), a, ( + 1, i + 1)) correspond to the inequality of letters. The loop on the initial state allows to find all the occurrences of the searched factors. During the analysis of the text with the automaton, if a terminal state (, m−1) is reached, this indicates the presence of an occurrence of x with exactly mismatches. It is clear that the automaton possesses (k + 1) × (m + 1 − k2 ) states and that it can be build in time O(k × m). An example is shown in Fig. 3.22. Unfortunately, the total number of states obtained by determinizing the automaton is Θ(min{mk+1 , (k + 1)!(k + 2)m−k+1 }). We can check that a direct simulation of the automaton produces a search algorithm whose execution time is O(m × n) using the dynamic programming as in the previous section. Actually by using a method adapted to the problem we get, in the rest, an algorithm that performs the search in time O(k×n). This produces a solution of same complexity as the one of algorithm K-diff-diag that nevertheless solves a more general problem. But the solution that follows is based on a simple management of lists without using a search algorithm for common ancestor.

78

Maxime Crochemore and Thierry Lecroq a, b, c, d

0,-1

a

0,0

b, c, d

1,0

b

0,1

c

0,2

d

0,3

a, c, d

a, b, d

a, b, c

b

c

d

1,1

a, c, d 2,1

1,2

1,3

a, b, d

a, b, c

c

d

2,2

2,3

Fig. 3.22. The (non-deterministic) automaton of approximate pattern matching with two mismatches for the string abcd on the alphabet V = {a, b, c, d}.

3.3.2 Specific implementation We show how to reduce the execution time of the simulation of the previous automaton. To obtain the desired time, we utilize during the search a queue F of positions that stores detected mismatches. Its update is done by letter comparisons, but also by merging with queues associated with string x. The sequences that they represent are defined as follows. For a shift q of x, 1 ≤ q ≤ m − 1, G[q] is the increasing sequence, of maximal length 2k + 1, of the positions on x of the leftmost mismatches between x[q . . m − 1] and x[0 . . m − q − 1]. The sequences are determined during a preprocessing phase that is described at the end of the section. The searching phase consists in performing attempts at all the positions j = 0, 1, . . . , n − m on y. During the attempt at position j, we scan the factor y[j . . j + m − 1] of the text and the generic situation is the following (see Fig. 3.23): the prefix y[j . . g] of the window has already been scanned during f

j

g

y

Fig. 3.23. Variables of Algorithm K-mismatches. During the attempt at position j, variables f and g spot a previous attempt . The mismatches between y[f . . g] and x[0 . . g − f ] are stored in the queue F .

a previous attempt at position f , f < j, and no comparison already happens on the suffix y[g + 1 . . n − 1] of the text. During the comparison of the already scanned part of the text, y[j . . g], around k tests can be necessary. Fig. 3.24 shows a computation example.


(a)

y

a b a b c b b a b a b a a c b a b a b a b b b a b

x

a b a c b a b a

x

a b a c b a b a

(b)

x y

(c)

79

a b a c b a b a

a b a b c b b a b a b a a c b a b a b a b b b a b x

a b a c b a b a

Fig. 3.24. Search with mismatches of the string x = abacbaba in the text y = ababcbbababaacbabababbbab. (a) Occurrence of the string with exactly three mismatches at position 0 on y. The queue F of mismatches contains positions 3, 4 and 5 on x. (b) Shift of length 1. There are seven mismatches between x[0 . . 6] and x[1 . . 7], this corresponds to the fact that G[1] contains the sequence 1, 2, 3, 4, 5, 6, 7 (see Figure 3.26). (c) Attempt at position 1: the factor y[1 . . 7] has already been considered but the letter y[8] = b has never been compared yet. The mismatches at positions 0, 1, 5 and 6 on x can be deduced from the merge of the queues F and G[1]. Three letter comparisons are necessary at positions 2, 3 and 4 in order to find the mismatch at position 2 since these three positions are simultaneously in F and G[1]. An extra comparison provides the mismatch at position 7.

The positions of the mismatches detected during the attempt at position f are stored in a queue F . Their computation is done by scanning the positions in increasing order. For the search with k mismatches, we only keep in F at most k + 1 mismatches (the leftmost ones). Considering a possible (k + 1)-th mismatch amounts to compute the longest prefix of x that possesses exactly k mismatches with the aligned factor of y. The code of the search algorithm with mismatches, K-mismatches, is given in Fig. 3.25. The processing at position j proceeds in two steps. It first starts by comparing the factors x[0 . . g − j] and y[j . . g] using the queues F and G[j −f ]. The comparison amounts to perform a merge of these two queues (Line 8); this merge is described further. The second step is only applied when the obtained sequence contains less than k positions. It resumes the scanning of the window by simple letter comparisons (Lines 11–18). This is during this step that an occurrence of an approximate factor can be detected. An example of table G and of successive values of the queue F of the mismatches is presented in Fig. 3.26. In the algorithm K-mismatches, the positions stored in the queues F or J are positions on x. They indicate mismatches between x and the factor aligned at position f on y. Thus, if p occurs in the queue, we have x[p] = y[f + p]. When the variable f is updated, the origin of the factor of y is replaced by j, and we should thus perform a translation, that is to say to decrease the

80


K-mismatches(x, m, G, y, n, k) 1 F ← Empty-Queue() 2 (f, g) ← (−1, −1) 3 for j ← 0 to n − m do 4 if Length(F ) > 0 and Head(F ) = j − f − 1 then 5 Dequeue(F ) 6 if j ≤ g then 7 J ← Mis-merge(f, j, g, F, G[j − f ]) 8 else J ← Empty-Queue() 9 if Length(J) ≤ k then 10 F ←J 11 f ←j 12 do 13 g ←g+1 14 if x[g − j] = y[g] then 15 Enqueue(F, g − j) 16 while Length(F ) ≤ k and g < j + m − 1 17 if Length(F ) ≤ k then 18 Output(j)

Fig. 3.25. Approximate string matching with k mismatches.

positions by the quantity j − f . This is realized in the algorithm Mis-merge during the addition of a position in the output queue. If the merge realized by the algorithm Mis-merge executes in linear time, the execution time of the algorithm K-mismatches is O(k × n) in space O(k × m). 3.3.3 Merge The aim of the operation Mis-merge(f, j, g, F, G[j − f ]) (Line 8 of the algorithm K-mismatches) is to produce the sequence of positions of the mismatches between the strings x[0 . . g −j] and y[j . . g], relying on the knowledge of the mismatches stored in the queues F and G[j −f ]. This algorithm is given in Fig. 3.28. The positions p in F mark the mismatches between x[0 . . g−f ] and y[f . . g], but only those that satisfy the inequality f + p ≥ j (by definition of F we already have f + p ≤ g) are useful to the computation The objective of the test in Line 5 of the algorithm K-mismatches is precisely to delete from F the useless values. The positions q of G[j − f ] denote the mismatches between x[j − f . . m − 1] and x[0 . . m − j + f − 1]. Those that are useful must satisfy the inequality f +q ≤ g (we already have f +q ≥ j). The test in Line 19 of the algorithm Mis-merge takes into account this constraint. Fig. 3.27 illustrates the merge (see also Fig. 3.24). Let us consider a position p on x such that j ≤ f + p ≤ g. If p occurs in F , this means that y[f + p] = x[p]. If p is in G[j − f ], this means that


i 0 1 2 3 4 5 6 7

x[i] a b a c b a b a

G[i] 1, 2, 3, 4, 5, 6, 7 3, 4, 5 3, 6, 7 4, 5, 6, 7 6, 7 (a)

j 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

y[j] a b a b c b b a b a b a a c b a b a

81

F 3, 4, 5 0, 1, 2, 5 2, 3 0, 1, 2, 3 0, 2, 3 0, 3, 4, 5 0, 1, 2, 3 3, 4, 6, 7 0, 1, 2, 3 3, 4, 5, 6 0, 1 1, 2, 3, 4 1, 2, 3 3, 4, 5, 7 0, 1, 2, 3 3, 4, 5, 7 0, 1, 2, 3 3, 5, 6, 7 (b)

Fig. 3.26. Queues used for the approximate search with three mismatches of x = abacbaba in y = ababcbbababaacbabababbbab. (a) Values of table G for string abacbaba. The queue G[3] for instance contains 3, 6 and 7, positions on x of the mismatches between its suffix cbaba and its prefix abacb. (b) Successive values of queue F of the mismatches computed by Algorithm K-mismatches. The values at positions 0, 2, 4, 10 and 12 on y possess less than three elements, which reveals the presence of occurrences of x with at most three mismatches at these positions. At position 0, for instance, the factor ababcbba of y possesses exactly three mismatches with x: they are at positions 3, 4 and 5 on x.

x[p] = x[p − j + f ]. Four situations can arise for a position p whether it occurs or not in F and G[j − f ]. (see Fig. 3.24 and 3.27): 1. The position p is neither in F nor in G[j − f ]. We have y[f + p] = x[p] and x[p] = x[p − j + f ], thus y[f + p] = x[p − j + f ]. 2. The position p is in F but not in G[j − f ]. We have y[f + p] = x[p] and x[p] = x[p − j + f ], thus y[f + p] = x[p − j + f ]. 3. The position p is in G[j − f ] but not in F . We have y[f + p] = x[p] and x[p] = x[p − j + f ], thus y[f + p] = x[p − j + f ]. 4. The position p is in F and in G[j − f ]. We have y[f + p] = x[p] and x[p] = x[p−j +f ], this does not allow to conclude on the equality between y[f + p] and x[p − j + f ]. Among the enumerated cases, only the last three can lead to a mismatch between the letters y[f + p] and x[p − j + f ]. Only the last case requires an

82

Maxime Crochemore and Thierry Lecroq y

(a)

(b)

x

a b a c b a b a

x

a b a c b a b a x

y (c)

a b a b c b b a b a b a a a b a b a b a b b b a b

a b a c b a b a

a b a b c b b a b a b a a c b a b a b a b b b a b x

a b a c b a b a

Fig. 3.27. Merge during the search with three mismatches of x = abacbaba in y = ababcbbababaacbabababbbab. (a) Occurrence of x at position 4 on y with three mismatches at positions 0, 2 and 3 on x; F = 0, 2, 3. (b) There are three mismatches between x[2 . . 7] and x[0 . . 5]; G[2] = 3, 4, 5. (c) The sequences conserved for the merge are 2, 3 and 3, 4, 5, and this latter produces the sequence 2, 3, 4, 5 of positions of the four first mismatches between x and y[6 . . 13]. A single letter comparison is necessary at position 3, between x[1] and y[7], since the other positions only occur in one of the two sequences.

extra comparison of letters. They are processed in this respective order at Lines 7–8, 10–11 and 12–15 of the algorithm of merge. The algorithm Mis-merge (see Fig. 3.28) executes in linear time. 3.3.4 Correctness proof The correctness proof of the algorithm K-mismatches relies on the proof of the function Mis-merge. One of the main arguments of the proof is a property of the Hamming distance that is stated below. Let u, v and w be three strings of same length. Let us set d = Ham(u, v), d = Ham(v, w), and assume d ≤ d. We then have: d − d ≤ Ham(u, w) ≤ d + d . When the operation Mis-merge(f, j, g, F, G[j − f ]) is executed in the algorithm K-mismatches, the next conditions are satisfied: 1. 2. 3. 4. 5.

f < j ≤ g ≤ f + m − 1; F = p | x[p] = y[f + p] and j ≤ f + p ≤ g); x[g − f ] = y[g]; Length(F ) ≤ k + 1; G = p | x[p] = x[p − j + f ] and j ≤ f + p ≤ g for an integer g such that j ≤ g ≤ f + m − 1.

Moreover, if g < f + m − 1, Length(G) = 2k + 1 by definition of G. By taking these conditions as assumption we get the following result. Let J = Mis-merge(f, j, g, F, G[j − f ]). If Length(J) ≤ k,


83

Mis-merge(f, j, g, F, G) 1 J ← Empty-Queue() 2 while Length(J) ≤ k and Length(F ) > 0 and Length(G) > 0 do 3 p ← Head(F ) 4 q ← Head(G) 5 if p < q then 6 Dequeue(F ) 7 Enqueue(J, p − j + f ) 8 else if q < p then 9 Dequeue(G) 10 Enqueue(J, q − j + f ) 11 else Dequeue(F ) 12 Dequeue(G) 13 if x[p − j + f ] = y[f + p] then 14 Enqueue(J, p − j + f ) 15 while Length(J) ≤ k and Length(F ) > 0 do 16 Dequeued(F, p) 17 Enqueue(J, p − j + f ) 18 while Length(J) ≤ k and Length(G) > 0 and Head(G) ≤ g − f do 19 Dequeued(G, q) 20 Enqueue(J, q − j + f ) 21 return J

Fig. 3.28. Algorithm for merging queues.

J = p | x[p] = y[j + p] and j ≤ j + p ≤ g, and, in the contrary case, Ham(y[j . . g], x[0 . . g − j]) > k. The result that follows is on the correctness of algorithm K-mismatches. It assumes that the sequences G[q] are computed in accordance with their definition. If x, y ∈ V ∗ , m = |x|, n = |y|, k ∈ N and k < m ≤ n, the algorithm K-mismatches detects all the positions j = 0, 1, . . . , n − m on y for which Ham(x, y[j . . j + m − 1]) ≤ k. 3.3.5 Preprocessing The aim of the preprocessing phase is to compute the values of the table G that is required by the algorithm K-mismatches. Let us recall that for a shift q of x, 1 ≤ q ≤ m − 1, G[q] is the increasing sequence of positions on x of the leftmost mismatches between x[q . . m − 1] and x[0 . . m − q − 1], and that this sequence is limited to 2k + 1 elements.

84


The algorithm Pre-K-mismatches is given in Fig. 3.29. The computation of the sequences G[q] is realized in an elementary way by the function whose code follows. Pre-K-mismatches(x, m, k) 1 for q ← 1 to m − 1 do 2 G[q] ← Empty-Queue() 3 i←q 4 while Length(G[q]) < 2k + 1 and i < m do 5 if x[i] = x[i − q] then 6 Enqueue(G[q], i) 7 i←i+1 8 return G

Fig. 3.29. Preprocessing for the approximate string matching with mismatches.

The execution time of the algorithm is O(m2 ), but it is possible to prepare the table in time O(k × m × log m).

3.4 Shift-Or Algorithm We are interested in this Section in the case of the search for short patterns. We first present an algorithm to solve the exact string matching problem, but that extends readily to the approximate string matching problems.

j y

.. .

x[0]

i=0

1

x[0 . . 1]

i=1

0

x[0 . . 2]

i=2

1

.. .

.. .

.. .

x

i=m−1

0 R0j

Fig. 3.30. Meaning of vector R0j . Each matching prefix of x is associated with value 1 in Rj0 .


85

Let R0 be a bit array of size m. Vector R0j is the value of the entire array R0 after text character y[j] has been processed (see Fig. 3.30). It contains information about all matches of prefixes of x that end at position j in the text. It is defined, for 0 ≤ i ≤ m − 1, by 0 if x[0 . . i] = y[j − i . . j], 0 Rj [i] = 1 otherwise. Therefore, R0j [m − 1] = 0 is equivalent to saying that an (exact) occurrence of the pattern x ends at position j in y. The vector R0j can be computed after R0j−1 by the following recurrence relation: 0 if R0j−1 [i − 1] = 0 and x[i] = y[j], 0 Rj [i] = 1 otherwise, and R0j [0]

0 if x[0] = y[j], = 1 otherwise.

The transition from R0j−1 to R0j can be computed very fast as follows. For each a ∈ V , let Sa be a bit array of size m defined, for 0 ≤ i ≤ m − 1, by Sa [i] = 0

iff

x[i] = a.

The array Sa denotes the positions of the character a in the pattern x. All arrays Sa are preprocessed before the search starts. And the computation of R0j reduces to two operations, SHIFT and OR: R0j = SHIFT(R0j−1 ) OR Sy[j] . An example is given in Fig. 3.31. Approximate String Matching with k Mismatches The Shift-Or algorithm easily adapts to support approximate string matching with k mismatches. To simplify the description, we shall present the case where at most one substitution is allowed. We use arrays R0 and S as before, and an additional bit array R1 of size m. Vector R1j−1 indicates all matches with at most one substitution up to the text character y[j − 1]. The recurrence on which the computation is based splits into two cases. • There is an exact match on the first i characters of x up to y[j − 1] (i.e., R0j−1 [i − 1] = 0). Then, substituting x[i] to y[j] creates a match with one substitution (see Fig. 3.32). Thus, R1j [i] = R0j−1 [i − 1] .

86


G A T A A

C 1 1 1 1 1

SA SC SG ST 1 1 0 1 0 1 1 1 1 1 1 0 0 1 1 1 0 1 1 1 AGATAAGA 10111101 11011110 11101111 11110111 11111011

G 0 1 1 1 1

A 1 0 1 1 1

A 1 1 1 1 1

Fig. 3.31. String x = GATAA occurs at position 2 in y = CAGATAAGAGAA.

•

There is a match with one substitution on the first i characters of x up to y[j − 1] and x[i] = y[j]. Then, there is a match with one substitution of the first i + 1 characters of x up to y[j] (see Fig. 3.33). Thus, R1j−1 [i − 1] if x[i] = y[j], 1 Rj [i] = 1 otherwise.

This implies that R1j can be updated from R1j−1 by the relation: R1j = SHIFT R1j−1 OR Sy[j] AND SHIFT R0j−1 .

j -1 j y x i -1 i Fig. 3.32. If R0j−1 [i − 1] = 0 then R1j [i] = 0.

An example is presented in Fig. 3.34. Approximate String Matching with k Differences We show in this section how to adapt the Shift-Or algorithm to the case of only one insertion, and then dually to the case of only one deletion. The method is based on the following elements. One insertion is allowed: here, vector R1j−1 indicates all matches with at most one insertion up to text character y[j − 1]. R1j−1 [i − 1] = 0 if the first i


87

j -1 j y x

∼ i -1 i Fig. 3.33. R1j [i] = R1j−1 [i − 1] if x[i] = y[j].

G A T A A

C 0 1 1 1 1

A 0 0 1 1 1

G 0 1 1 1 1

A 0 0 1 1 1

T 0 1 0 1 1

A 0 0 1 0 1

A 0 0 1 1 0

G 0 1 1 1 1

A 0 0 1 1 1

G 0 1 0 1 1

A 0 0 1 0 1

A 0 0 0 1 0

Fig. 3.34. String x = GATAA occurs at positions 2 and 7 in y = CAGATAAGAGAA with no more than one mismatch.

characters of x (x[0 . . i − 1]) match i symbols of the last i + 1 text characters up to y[j −1]. Array R0 is maintained as before, and we show how to maintain array R1 . Two cases arise. •

There is an exact match on the first i + 1 characters of x (x[0 . . i]) up to y[j − 1]. Then inserting y[j] creates a match with one insertion up to y[j] (see Fig. 3.35). Thus, R1j [i] = R0j−1 [i] .

• There is a match with one insertion on the i first characters of x up to y[j − 1]. Then if x[i] = y[j] there is a match with one insertion on the first i + 1 characters of x up to y[j] (see Fig. 3.36). Thus, R1j−1 [i − 1] if x[i] = y[j], 1 Rj [i] = 1 otherwise. This shows that R1j can be updated from R1j−1 with the formula OR Sy[j] AND R0j−1 . R1j = SHIFT R1j−1 An example is given in Fig. 3.37. One deletion is allowed: we assume here that R1j−1 indicates all possible matches with at most one deletion up to y[j − 1]. As in the previous solution, two cases arise. • There is an exact match on the first i characters of x (x[0 . . i − 1]) up to y[j] (i.e., R0j [i − 1] = 0). Then, deleting x[i] creates a match with one deletion (see Fig. 3.38). Thus,

88

Maxime Crochemore and Thierry Lecroq j -1 j

y

+ x i Fig. 3.35. If R0j−1 [i] = 0 then R1j [i] = 0. j -1 j

y

+ x i -1 i Fig. 3.36. R1j [i] = R1j−1 [i − 1] if x[i] = y[j].

G A T A A

C 1 1 1 1 1

A 1 1 1 1 1

G 1 1 1 1 1

A 0 1 1 1 1

T 1 0 1 1 1

A 1 1 0 1 1

A 1 1 1 0 1

G 1 1 1 1 0

A 0 1 1 1 1

G 1 0 1 1 1

A 0 1 1 1 1

A 1 0 1 1 1

Fig. 3.37. GATAAG is an occurrence of x = GATAA with exactly one insertion in y = CAGATAAGAGAA. j -1 j y x

− i -1 i Fig. 3.38. If R0j [i] = 0 then R1j [i] = 0.

R1j [i] = R0j [i − 1] . • There is a match with one deletion on the first i characters of x up to y[j − 1] and x[i] = y[j]. Then, there is a match with one deletion on the first i + 1 characters of x up to y[j] (see Fig. 3.39). Thus, R1j−1 [i − 1] if x[i] = y[j], 1 Rj [i] = 1 otherwise. The discussion provides the following formula used to update R1j from R1j−1 :


OR Sy[j] AND R1j = SHIFT R1j−1

89

SHIFT R0j .

j -1 j y x

− i -1 i Fig. 3.39. R1j [i] = R1j−1 [i − 1] if x[i] = y[j].

An example is presented in Fig. 3.40.

G A T A A

C 0 1 1 1 1

A 0 0 1 1 1

G 0 0 1 1 1

A 0 0 0 1 1

T 0 1 0 0 1

A 0 0 1 0 0

A 0 0 1 1 0

G 0 0 1 1 1

A 0 0 0 1 1

G 0 0 1 1 1

A 0 0 0 1 1

A 0 0 1 0 1

Fig. 3.40. GATA and ATAA are two occurrences with one deletion of x = GATAA in y = CAGATAAGAGAA

Wu–Manber Algorithm We present in this section a general solution for the approximate string matching problem with at most k differences of the types: insertion, deletion, and substitution. It is an extension of the problems presented above. The following algorithm maintains k + 1 bit arrays R0 , R1 , . . . , Rk that are described now. The vector R0 is maintained similarly as in the exact matching case. The other vectors are computed with the formula (1 ≤ ≤ k) OR Sy[j] Rj = SHIFT Rj−1 AND SHIFT R−1 j−1 AND SHIFT Rj−1 AND R−1 j−1 which can be rewritten into OR Sy[j] Rj = SHIFT Rj−1 AND R−1 AND SHIFT R−1 j j−1 AND R−1 j−1 .

90


G A T A A

C 0 1 1 1 1

A 0 0 1 1 1

G 0 0 1 1 1

A 0 0 0 1 1

T 0 0 0 0 1

A 0 0 0 0 0

A 0 0 1 0 0

G 0 0 1 1 0

A 0 0 0 1 1

G 0 0 0 1 1

A 0 0 0 0 1

A 0 0 0 0 0

Fig. 3.41. Here x = GATAA and y = CAGATAAGAGAA and k = 1. The output 5, 6, 7, and 11 corresponds to the segments GATA, GATAA, GATAAG, and GAGAA which approximate the pattern GATAA with no more than one difference. WM(x, m, y, n, k) 1 for each character a ∈ V do 2 Sa ← 1m 3 for i ← 0 to m − 1 do 4 Sx[i] [i] ← 0 5 R0 ← 1m 6 for ← 1 to k do 7 R ← SHIFT(R−1 ) 8 for j ← 0 to n − 1 do 9 T ← R0 10 R0 ← SHIFT(R0 ) OR Sy[j] 11 for ← 1 to k do 12 T ← R 13 R ← (SHIFT(R ) OR Sy[j] ) AND (SHIFT((T AND R−1 )) AND T 14 T ← T 15 if Rk [m − 1] = 0 then 16 Output(j)

Fig. 3.42. Wu–Manber approximate string matching algorithm.

An example is given in Fig. 3.41. The method, called the Wu–Manber algorithm, is implemented in Fig. 3.42. It assumes that the length of the pattern is no more than the size of the memory word of the machine, which is often the case in applications. The preprocessing phase of the algorithm takes O(σm + km) memory space, and runs in time O(σm + k). The time complexity of its searching phase is O(kn).

3.5 Bibliographic notes The techniques described in this chapter are overused in molecular biology for comparing sequences of chains of nucleic acids (DNA or RNA) or of amino acids (proteins). The books of Deonier, Tavaré and Waterman [9], Setubal


91

and Meidanis [28], Gusfield [10] and Böckenhauer and Bongartz [4] constitute excellent introductions to problems of the domain. The book of Sankoff and Kruskal [26] contains numerous applications of alignments. The book of Crochemore, Hancart and Lecroq [7] presents in detail, together with their correctness proofs, the algorithms for computing alignments and solving the approximate string matching problems. The notion of longest common subsequence to two strings is used for file comparison. The command diff of the UNIX system implements an algorithm based on this notion by considering that the lines of the files are letters of the alphabet. Among the algorithms at the basis of this command are those of Hunt and Szymanski [14] and of Myers [20]. A general presentation of the algorithms for searching for common subsequences can be found in an article by Apostolico [1]. Wong and Chandra [32] shown that the algorithm LCS-simple is optimal in a model where we limit the access to letters to equality tests. Without this condition, Hirschberg [13] gave a (lower) bound Ω(n × log n). On a bounded alphabet, Masek and Paterson [18] gave an algorithm running in time O(n2 / log n). The extension of this result to the general computation of alignments is an open problem (see Apostolico and Giancarlo [2]). Using the Lempel-Ziv factorization of the two strings, Crochemore, Landau and Ziv-Ukelson designed an algorithm for computing alignments running in time O(hn2 / log n) where h ≤ 1 is the entropy of the text. The initial algorithm of global alignment, from Needleman and Wunsch [24], runs in cubic time. The algorithm of Wagner and Fischer [31], as well as the algorithm of local alignment of Smith and Waterman [29], run in quadratic time (see [10], page 234). The method of dynamic programming was introduced by Bellman (1957; see [6]). Sankoff [25] discusses the introduction of the dynamic programming in the processing of molecular sequences. The algorithm LCS is from Hirschberg [12]. A generalization of this method has been proposed by Myers and Miller [21]. Charras and Lecroq created the site [5], accessible on the Web, where animations of alignment algorithms are available. The book of Navarro and Raffinot [23] is an excellent introduction to exact and approximate string matching. The algorithm K-diff-cut-off is from to Ukkonen [30]. The algorithm Kdiff-diag together with its implementation with the help of the computation of common ancestors was described by Landau and Vishkin [16]. Harel and Tarjan [11] presented the first algorithm running in constant time that solves the problem of the common ancestor to two nodes of a tree. An improved version is from Schieber and Vishkin [27]. Landau and Vishkin [15] conceived the algorithm K-mismatches. The size of the automaton of Section 3.3 was established by Melichar [19]. The approximate pattern matching for short strings in the way of the algorithm K-diff-short-pattern is from Wu and Manber [33] and also from Baeza-Yates and Gonnet [3].

92


A synthesis and experimental results on the approximate pattern matching is presented by Navarro [22].

References 1. A. Apostolico. String Editing and Longest Common Subsequences. G. Rozenberg and A. Salomaa, editors. Handbook of Formal Languages, 1997, 361–398. Springer-Verlag. 2. A. Apostolico and R. Giancarlo. Sequence alignment in molecular biology. J. Comput. Bio., 5(2):173–196, 1998. 3. R. A. Baeza-Yates and G. H. Gonnet. A new approach to text searching. Comm. ACM, 35(10):74–82, 1992. 4. H.-J. Böckenhauer and D. Bongartz. Algorithmic Aspects of Bioinformatics. Springer-Verlag, 2007. 5. C. Charras and T. Lecroq. Sequence Comparison. http://monge.univ-mlv.fr/ lecroq/seqcomp/ 6. T. H. Cormen, C. E. Leiserson and R. L. Rivest. Introduction to Algorithms. The MIT Press, 1990. 7. M. Crochemore, C. Hancart and T. Lecroq. Algorithms on strings. Cambridge University Press, 2007. 8. M. Crochemore, G. M. Landau and M. Ziv-Ukelson. A sub-quadratic sequence alignment algorithm for unrestricted cost matrices. SIAM J. Comput., 32(6):1654–1673, 2003. 9. R. C. Deonier, S. Tavaré and M. S. Waterman. Computational Genome Analysis: An Introduction (Statistics for Biology & Health). Springer-Verlag, 2005. 10. D. Gusfield. Algorithms on Strings, Trees and Sequences: Computer Science and Computational Biology. Cambridge University Press, 1997. 11. D. Harel and R. E. Tarjan. Fast algorithms for finding nearest common ancestors. SIAM J. Comput., 13(2):338–355, 1984. 12. D. S. Hirschberg. A linear space algorithm for computing maximal common subsequences. Comm. ACM, 18(6):341–343, 1975. 13. D. S. Hirschberg. An Information-Theoretic Lower Bound for the Longest Common Subsequence Problem. Inform. Process. Lett., 7(1):40–41, 1978. 14. J. W. Hunt and T. G. Szymanski. A fast algorithm for computing longest common subsequences. Comm. ACM, 20(5):350–353, 1977. 15. G. M. Landau and U. Vishkin. Efficient string matching with k mismatches. Theoret. Comput. Sci., 43(2–3):239–249, 1986. 16. G. M. Landau and U. Vishkin. Fast string matching with k differences. J. Comput. System Sci., 37(1):63–78, 1988. 17. V. I. Levenshtein. Binary codes capable of correcting deletions, insertions and reversals. Sov. Phys. Dokl., 6:707–710, 1966. 18. W. J. Masek and M. S. Paterson. A faster algorithm for computing string edit distances. J. Comput. Syst. Sci., 20(1):18–13, 1980. 19. B. Melichar. Approximate String Matching by Finite Automata. Computer Analysis of Images and Patterns, V. Hlavác and R. Sára, editors, Lecture Notes in Computer Science, volume 970, 342–349, Springer-Verlag, Berlin, 1995. 20. E. W. Myers. An O(N D) difference algorithm and its variations. Algorithmica, 1:251–266, 1986.


93

21. E. W. Myers and W. Miller. Optimal alignment in linear space. CABIOS, 4(1):11–17, 1988. 22. G. Navarro. A guided tour to approximate string matching. ACM Comp. Surv., 33(1):31–88, 2001. 23. G. Navarro and M. Raffinot. Flexible Pattern Matching in Strings – Practical online search algorithms for texts and biological sequences. Cambridge University Press, 2002. 24. S. B. Needleman and C. D. Wunsch. A general method applicable to the search for similarities in the amino acid sequence of two proteins. J. Mol. Biol., 48:443–453, 1970. 25. D. Sankoff. The early introduction of dynamic programming into computational biology. Bioinformatics, 16:41–47, 2000. 26. D. Sankoff and J. Kruskal, editors. Time Warps, String Edits, and Macromolecules: The Theory and Practice of Sequence Comparison. Cambridge University Press, second edition, 1999. 27. B. Schieber and U. Vishkin. On finding lowest common ancestors: simplification and parallelization. SIAM J. Comput., 17(6):1253–1262, 1988. 28. J. C. Setubal and J. Meidanis. Introduction to Computational Molecular Biology. PWS Publishing Company, 1997. 29. T. F. Smith and M. S. Waterman. Identification of common molecular sequences. J. Mol. Biol., 147:195–197, 1981. 30. E. Ukkonen. Algorithms for approximate string matching. Inform. and Control, 64(1–3):100–118, 1985. 31. R. A. Wagner and M. Fischer. The string-to-string correction problem. J. ACM, 21(1):168–173, 1974. 32. C. K. Wong and A. K. Chandra. Bounds for the string editing problem. J. ACM, 23(1):13–16, 1976. 33. S. Wu and U. Manber. Fast text searching allowing errors. Comm. ACM, 35(10):83–91, 1992.

4 An Introductory Course on Communication Complexity Carsten Damm Institut für Informatik, Universität Göttingen Lotzestr. 16-18, D-37073 Göttingen [email protected] Summary. This text was written for a short course on communication complexity given at the 5th International PhD School in Formal Languages and Applications at Tarragona, Spain (June 5/6, 2006). The course was planned for three lectures.The text is intended to beginners and it may also serve as a guide for further reading. Some easy exercises are spread out through the text with solutions given at the end. The present version is a revision of the one used in class.

4.1 Motivation, Definition, and Background 4.1.1 Introduction Whenever several persons, computers, or components of a system jointly complete a certain task, that none of them can do alone, there is need for communication. Sometimes, part of the actions of the participants consists of obvious information exchange between parties (Alice knows the family name, and Bob the first name of a person. By communicating they can find out the person’s phone number), sometimes communication takes place invisibly and implicitely inside a system (bits exchanged in the processing unit of a personal computer). “How much of communication” is absolutely necessary to complete the task? If I knew this minimum “amount of communication”, what does it tell me about the complexity of the task? Communication complexity deals with questions of this type in a variety of settings. Properly defined, communication problems focus attention to the combinatorial core of an otherwise very complicated setup. Because of this concentration to the essential it allows to derive interesting computational bounds with incontrovertible rigour and is therefore at the heart of several lower bound proofs of computational complexity. Everyone interested in complexity of computations should have some ideas from communication complexity in the toolbox. This text is organized as follows. In the current chapter we introduce some preliminaries, give an example application of communication complexity and C. Damm: An Introductory Course on Communication Complexity, Studies in Computational Intelligence (SCI) 113, 95–123 (2008) c Springer-Verlag Berlin Heidelberg 2008 www.springerlink.com

96

Carsten Damm

mention some useful references. The next chapter is devoted to lower bound techniques for communication complexity. Chapter 4.3 and 4.4 deal with some problems of interest for formal language theory. In Chapter 4.5 we survey some other communication setups that are studied in the literature. Throughout the text there are exercises. Usually the exercises refer to the topic introduced immediately before. It makes therefore sense to discuss them right where they appear. I conclude the introduction with a possible schedule for a short course on communication complexity based on the present material. LECTURE 1 1.2 Notation and definitions, 1.3 Some “benchmark functions” for communication complexity, 1.4 An application, 1.5 Some history and some references, LECTURE 2 2.1 The range bound and the tiling method, 2.2 The fooling set method, 2.3 The rank method, 2.4 Comparison of lower bound methods LECTURE 3 3. Communication complexity and Chomsky hierarchy, 4. Communication complexity applications for finite automata and Turing machines, 5.1 Different modes of communication, 5.2 Different partitions, 5.3 Different games This plan can be completed by discussing a short research paper dealing with surprising communication protocols. A very nice example is given in [4] (others include, e.g., the protocols in [12] or in [19]). Actually [4] was chosen for the PAPER session to this little course at the PhD school in Tarragona. This selfcontained, two-page paper is available online (see references) and its exposition can hardly be improved. So I decided to not cover it directly in this text. Instead a set of slides from our PAPER session can be sent on email-request. 4.1.2 Notations and Definitions First we need some general mathematical notions and notations. Throughout this text we use log to denote the base 2 logarithm. If A is a finite set, |A| denotes the number of elements in it. Exercise 1. Suppose you are about to make a catalogue of the books on your bookshelf. You decide to label the books by binary strings. If you have N books on the shelf, how many bits (binary digits) do you need for the labels? How many decimal digits do you need if you use decimal labels instead? A pair (A1 , A2 ) of subsets of A is a partition of A if (1) A1 ∪ A2 = A and (2) A1 ∩ A2 = ∅. More general, a family {A1 , . . . , AT } of subsets of A such T that (1) i=1 Ai = A and (2) Ai ∩ Aj = ∅ if i = j is also called a partition of A. This is obviously equivalent to saying, that each element of A belongs to exactly one of the subsets Ai . n For vectors v ∈ {0, 1}n the weight of v is defined to be i=1 vi . The weight of v is denoted v.

4 An Introductory Course on Communication Complexity

97

Now let’s start with some simple notions from communication complexity. We want to formally describe the situation, that something is to be jointly computed by several participants, none of which has complete information on the input. The actors in a communication setting are called parties or players. In a two-party setting, the parties are almost always called Alice and Bob. Their task is to compute a function value f (x, y), where x is Alice’s part of the input and y Bob’s. Alice has absolutely no information about y and Bob has no idea, what x is. f is a function of shape f : X × Y → Z, for some non-empty sets X, Y , and Z. For most examples in this text we consider X = Y = {0, 1}n , where n is fixed and Z = {0, 1}. This means, that input bits are partitioned into two equally-sized sets of bits. One set is given to Alice, the other to Bob. So, Alice has an n-bit-vector x, Bob has an n-bit-vector y, and they want to know the bit z = f (x, y). The function as such is known to both players, but they only want to determine this particular value. The point is, that the players are not charged for computation time or memory usage, but for the number of bits they need to exchange until they both know z. The following trivial strategy enables Alice and Bob to compute z: (1) When the game starts Alice sends her input x to Bob. (2) Bob, now knowing the complete input pair (x, y) can compute z and sends this bit to Alice. This takes n + 1 bits of communication and it works for every function f . But, e.g., for the parity function PARITY : {0, 1}n × {0, 1}n → {0, 1} this would be a very bad strategy. This function is defined by 1 , if x + y is odd, PARITY(x, y) = 0 , else. PARITY(x, y) is simply the parity of all input bits. A much better strategy for PARITY is: (1) Alice sends the parity of her input bits. (2) Bob adds the parity of his input bits and sends the result back to Alice. This takes only 2 bits of communication. Exercise 2. For a natural number m ≥ 1 consider the function MODk: {0, 1}n × {0, 1}n → {0, 1, . . . , m − 1} defined by MODk (x, y) = x + y

(mod k).

Try to find a good communication strategy for this function. How many bits of communication are sufficient when following this strategy? The players’ aim is to communicate in such a clever way, that z can be determined without wasting too many bits of communication. For this they agree in advance on a set of rules that govern the communication and the interpretation of sent messages. This set of rules is called the protocol. Here is a formal definition following [21]:

98

Carsten Damm

Definition 1. A communication protocol P over X × Y and with range Z is a binary tree where each internal node v is either labeled by a function av : X → {0, 1} or a function bv : Y → Z and each leaf is labeled by some z ∈ Z. [Inner nodes labeled by some av “belong” to Alice, others “belong” to Bob.] Further, the two edges leaving a node are labeled 0 and 1, respectively. Execution of the protocol on input (x, y) consists of walking down the tree from the root to one of its leafs. If an internal node v is reached, the next node is the one that is reached by following the edge labeled av (x) or bv (y) depending on which function v is labeled with. The output of the protocol is the label of the leaf that is finally reached. In this definition the nodes in the tree represent the knowledge about the input pair that is common to both players. The functions av , bv determine the next bit to be send by a player depending on her or his input part. It may happen that, in executing the protocol we walk from node v to node v that both “belong to Alice”. This corresponds to the fact that in this case Alice sends two consecutive bits. In the sequel we will describe protocols more conveniently by combining maximum sequences of consecutive bits sent by one and the same player to binary strings, called messages. Here is the corresponding terminology (giving also another definition of protocols equivalent to the one above): 1. The players take turns in communication. We consider all sent messages to be binary strings of non-zero length. 2. Each sending of a message is considered a “round” in executing the protocol. The message sent in round i is denoted mi . 3. Alice sends the first message. Message mi is sent by Bob, if i is even and by Alice otherwise. 4. The protocol determines in each step i the message to be sent. It depends on the sequence m1 , . . . , mi−1 of previous messages (which is empty in case i = 1) and the part of the input known to the player whose turn it is. The protocol also specifies when it is finished and the output of the protocol. Remark 1. The sequence m1 , . . . , mi−1 does not contain more information than the sequence of edge labels from the root-to-leaf-path from Definition 1. This is due to the fact, that the labeled tree is fixed in advance and known to both players. Given such a path each player can infer from the node labels who sent which bit. Remark 2. In general it is required that after execution of the protocol both players know the output. If the output cannot be inferred from the messages sent so far and the known input part, the last message will be the output value. This is the case in most of our examples. Example 1. The trivial protocol for functions f : X × Y → Z is given by:


99

TRIVIAL(f ) : m1 (x) := x output := m2 (m1 ; y) := f (m1 , y) The “clever parity protocol” described above is given by PARITYn : m1 (x) := x (mod 2) output := m2 (m1 ; y) := m1 + y (mod 2) Let P be a certain protocol. The sequence (m1 , m2 , . . . , mr ) of messages sent while executing P on input pair (x, y) is called transcript of P on (x, y) and denoted sp (x, y). Let |m ir| denote the length of mi , i.e., the number of bits in it and let |sP (x, y)| = i=1 |mi | denote the total length of the transcript. Definition 2. The communication complexity of the protocol P is the number of bits exchanged by the protocol in the worst case: CC(P ) :=

max

(x,y)∈X×Y

|sP (x, y)|.

Exercise 3. Specify the rules of the strategy from Exercise 2 as a protocol MODk and determine its complexity. Let fP (x, y) denote the output generated by following P on input pair (x, y). fP : X × Y → Z is the function computed by P . Definition 3. The communication complexity of a function f is the communication complexity of the best protocol for f : CC(f ) := min CC(P ) . fP =f

Proposition 1. For any Boolean function {0, 1}n × {0, 1}n → {0, 1}n holds CC(f ) ≤ n + 1 Proof. CC(TRIVIAL(f )) = n + 1. Exercise 4. Clearly for every f : X × Y → Z holds CC(f ) ≥ 0. Give a characterization for the set of functions, for which CC(f ) = 0. 4.1.3 Some “Benchmark Functions” for Communication Complexity When studying papers on communication complexity one very often meets special functions whose communication complexity is investigated. A simple function of shape {0, 1}n × {0, 1}n → {0, 1} that is important in communication complexity is the following:

100

Carsten Damm

equality

1 , if x = y EQn (x, y) = 0 , otherwise.

Proposition 2. CC(EQn ) = n + 1. Proof. We show that protocol TRIVIAL(EQn ) is optimal. First observe, that for every input (x, y) both players need to send at least one bit — otherwise the output would not depend on the silent players input part contradicting the definition of the function. Suppose Alice (who starts the communication) sends less than n bits in total. Then there are two input parts x1 , x2 ∈ {0, 1}n , on which Alice sends the same messages. Then for all possible input parts y of Bob (whose messages depend only on y and received messages) the sequence of messages on (x1 , y) and (x2 , y) are the same. This means, by traveling down the protocol tree (x1 , y) and (x2 , y) reach the same leaf and in particular, the computed value will be the same. Now, suppose y = x1 . Then EQn (x1 , y) = 1 and EQn (x2 , y) = 0, hence the protocol cannot compute the function EQn . Other functions often met are: greater-than

1 , if x > y GTn (x, y) = 0 , otherwise.

Here, x, y are considered as binary representations of numbers in {0, . . . , 2n − 1} — likewise in the next example. inner-product n xi · yi (mod 2). IPn (x, y) = i=1

disjointness 1 , if there is no index i such that xi = yi = 1 DISJn (x, y) = 0 , otherwise. Think of x, y as descriptions of sets A, B ⊆ {1, . . . , n} — xi (yi , respectively) is 1 iff i is contained in A (B, respectively). Then DISJn (x, y) indicates, whether A and B are disjoint sets. Sometimes, when the length n of the input is understood or is unimportant, or we speak about all functions of the sequence, we omit the index n and write simply EQ, GT etc. These functions perhaps don’t look too practical, and knowing about their communication complexities does not seem to be of great value, but it is! Consider the problem of computing some function value F (a), where a is an N -bit input. To have something specific in mind, let a be n groups of n bits (interpreted as n numbers in the range 0 to 2n − 1) and let F (a) = MAX (a)


101

be the index of the maximum of the numbers (the smallest such index in case of a draw). Then every computational device that computes MAX can also be used to compute the function GE. Exercise 5. “Reduce” the computation of GT to the computation of MAX : Given x and y, construct a, such that the value of GT(x, y) can be inferred from MAX (a). In other words, the computation of GT is a by-product of computing MAX . Thus knowing CC(GT) gives us a truly unbeatable lower bound met by every computational device for MAX , that is “somehow charged” for communication (we’ll see an example in the next section). The point is, that MAX is more complicated and more special than GT. Therefore when deriving such lower bound for MAX directly there is good chance that essentials are hidden behind a bunch of peculiarities. On the other hand the bound would be a less general statement — it would apply only to exactly this function. Further “benchmark functions” are symmetric functions: Definition 4. A function is called symmetric, if its value does not change when input positions are permuted. Exercise 6. Prove that the following is an equivalent definition for symmetric boolean functions: Definition 5. A function defined on binary inputs is called symmetric, if its value does only depend on the number of input bits that are equal to 1. Example 2. MODk is a symmetric function, but IP and the other examples above are not. Exercise 7. Prove that if f : {0, 1}n × {0, 1}n → {0, 1} is symmetric, then CC(f ) ≤ log(n + 1) + 1. 4.1.4 An Application This section serves to illustrate the idea from the last section, that real-life computations can be “charged for communication” in non-obvious ways. It shows also that, as mentioned in the introduction, communication can take place implicitly within a system. The exposition is taken from the beautiful survey [24]. Consider the following problem in VLSI-design. A “microchip” is to be layed-out. Its purpose is to compute EQn (x, y). Two groups of n bits each x = (x1 , x2 , . . . , xn ), y = (y1 , y2 , . . . , yn ), arrive simultaneously at 2n places (the “input processors”) arranged in exactly the given bit order along one edge of the layout. There is also an “output processor” — some place at the border that produces the required output bit 1. There may be more processors as well as connecting wires between them on the chip.

102

Carsten Damm

Considering the processors and wires as nodes and edges of a graph, a VLSI-design can be considered a special embedding of that graph into the plane with some geometric restrictions. It is embedded into a grid of squares each of which is either empty or contains exactly one of the following as shown in Figure 4.1: a processor, a straight line connecting two opposite sides of the square (in two orientations), a sharp bend connecting to adjacent sides of the square (in 4 orientations) or a crossing of two lines, that each connect two opposite sides of the square.

empty

port

straight line

sharp bend

crossing lines

Fig. 4.1. Squares in an VLSI-design and an example layout

The chip works in distinct time steps. At each step, each processor reads the bits sent to it, computes some bit for some of its connecting wires and sends it to the other end of this wire. It is essential to note, that we assume “fast wires”, i.e., the time that a signal needs to travel from one to the other end of a wire is considered to be a non-zero constant independent from its length. The smallest possible layout contains only the input processors lined up in a row, one of them serving also as output processor. Let p1 , . . . p2n be the nodes, each containing a processor and the have wires to read in the input (see figure 4.2). We describe the functioning of this layout. In the first time step all nodes except pn+1 “sleep”. In this step pn+1 reads y1 , sends it to the left and sends an “alarm bit” to the right. The alarm bit wakes up pn+2 , who sends y2 to the left and alarms its right neighbor, and so on. Each processor, that receives some bit from the right, will transmit it to the left in the next step.

x1

x2

x3

x4

x5

x6

x7

x8

y1

y2

y3

y4

y5

y6

y7

y8

p1

p2

p3

p4

p5

p6

p7

p8

p9

p10

p11

p12

p13

p14

p15

p16

Fig. 4.2. Linear VLSI layout


103

When p1 eventually receives y1 it compares it with x1 and sends bit 1 to the right if they are equal and bit 0 if they are not. p2 receives this bit simultaneously with y2 . It sends bit 1 to the right, if it received 1 from the left and x2 and y2 are the same, otherwise it sends bit 0. Finally, pn receives one bit from the left (the resulting bit from all previous comparisons) and the bit yn from the right. If the bit from the left is 1 and xn and yn are equal, then pn gives the overall output bit 1, else 0. This design is small, but computation takes n time steps, since each wire can only carry one bit at a time. With a more generous design, the chip would finish work earlier. The idea is to provide extra wires that pass x1 and y1 , x2 and y2 , . . . , xn and yn at the same time to special comparator nodes and to collect the outcomes of comparison in a binary tree design to give the final output value (see Figure 4.3).

Fig. 4.3. Tree-like VLSI layout

In this design O(log n) time steps are sufficient to complete the task. By using bends in the tree part of the layout it is possible to use a little less of the chip’s area. However, this would not reduce the total length of the wires. Since every unit of the wire occupies some place on the chip (a square), it makes sense to consider the total length of wires as occupied chip area. Adopting this point of view our layout uses more than an amount of n2 of area (the right half of the design). Combining ideas from the small-but-slow linear and the fast-but-big tree-like design, it is possible to make a design that still works in time O(log n) but uses area only O(n2 / log n). However, still, the product of area and time is at least n2 . And this is true for any VLSI-design! This is a special case of results of [31], which build on communication complexity ideas: Proposition 3. If a VLSI-chip computes EQn , then for the number T of time steps needed and the total number A of wire units satisfies

104

Carsten Damm

AT ≥ n2 . Proof. Divide the design into two pieces by a vertical line, so that on the left hand side we have inputs x1 , . . . , xn , on the right hand side we have y1 , . . . , yn . We concentrate on the communication that crosses the line. Since by Proposition 2 we have CC(EQn ) = n + 1 at least n bits must cross the line until the result is known. Why not n + 1? Since only the output node (sitting on one of the sides) needs the final result, it is not necessary to “inform” the other. Because there are only T time steps, in one of the steps at least n/T bits cross the line. But as the wires carry only one bit at a time, this means at least n/T wires cross this line. How many wires do cross the line, if its position were one place to the right? In this case x1 and y1 were on the same side, but still communication is needed to compute EQn−1 (x2 , . . . , xn ), (y2 , . . . , yn )). By the same argument we conclude, that at least (n − 1)/T wires cross this line. Now we shift one position further to the right, and as before we can conclude that at least (n−2)/T wires cross the line, and so on. Similarly we can also shift the line by 1, 2, . . . places to the left and the line will cross at least (n−1)/T, (n−2)/T, . . . wires. In summing up, our lines crossed at least n−1 n−2 1 n2 n +2 + ... + 2 = T T T T T wires, which proves the statement. 4.1.5 Some History and Some References Communication complexity arguments where first applied in the late 1970s [1, 32]. The result from the last section is from [31]. As the technique became known widely, more and more results in a variety of areas where based on it or techniques were formulated in the language of communication complexity. In particular, also the model was extended to more sophisticated situations: more players, non-determinism, randomization, different charging, other input partitions etc. The primary motivation for communication complexity study comes from applications in the field of distributed and parallel computing (including VLSIcomputing). However, communication complexity is a neat field of study on its own. There are even analogies to NP-completeness theory and structural complexity. There is one difference however to “classical” complexity theory: in communication complexity we can prove the separations, that we only conjecture in the classical setting. Unfortunately, there seems no way to carry things over. . . This line of research was begun by [29] and continued by, e.g., [3, 13, 18, 23, 8] (a random selection). One of the latest developments is quantum communication complexity [9]. The achievements of communication complexity theory feed back into other fields of application and theory.


105

Nowadays, communication complexity is an established technique that is characterized by strict pinpointing combinatorial situations that appear in computations of all kind, beautiful mathematics, appealing puzzles, and surprising results. This little tutorial can only give a glimpse on some of the basic ideas in this nice toolbox. There are some very good surveys on communication complexity around, that cover more or different material: [28, 24, 14]. There are also books devoted to the subject of communication complexity: [21, 15].

4.2 Some Lower Bound Methods and Results 4.2.1 The Range Bound and the Tiling Method Definition 6. Let f : X × Y → Z be a function. The range of f is the set of all values, that can be taken by the function: Range(f ) := {z ∈ Z|∃(x, y) ∈ X × Y : f (x, y) = z}. Proposition 4. CC(f ) ≥ log |Range(f )|. Proof. If f (x1 , y1 ) = f (x2 , y2 ), then the players must use different transcripts on (x1 , y1 ) and (x2 , y2 ) — otherwise the protocol would make an error. Hence, we must have at least |Range(f )| many different transcripts, which are binary strings of length at least CC(f ). By Exercise 1 CC(f ) must be at least log |Range(f )| to give these many strings. Exercise 8. Apply Proposition 4 to the function MODk . For functions f : X × Y → {0, 1} with boolean output Proposition 4 is not very helpful, so we try to refine the argument. Before going into details it is useful to have the following picture in mind: We regard f as a matrix of order |X| × |Y | with rows indexed by inputs x (Alice’s part) and columns indexed by inputs y (Bob’s part). Consequently, the entry in row x and column y is f (x, y). We refer to this matrix as the communication matrix of f and denote it by Mf — however, as said, it is nothing else than f itself, written down in a special manner.

y

x

f(x,y)

Fig. 4.4. The communication matrix of f

106

Carsten Damm

Exercise 9. Write down the communication matrix of PARITY : {0, 1}3 × {0, 1}3 → {0, 1}. (After writing the first few bits in a row and in a column you will quickly see the structure and can stop writing . . . ) Definition 7. A subset R ⊆ X×Y is called a rectangle in X×Y , if R = A×B for some subsets A ⊆ X and B ⊆ Y . Please note, that it is not required, that A and B are consecutive in any sense! This requirement would even be meaningless, without specifying the order of entries in the communication matrix.

B B

A A

Fig. 4.5. Two examples for rectangles

Lemma 1 (Combinatorial Characterization of Rectangles). Let R ⊆ X × Y . The following are equivalent: 1. R is a rectangle. 2. For any two points (x1 , y1 ), (x2 , y2 ) ∈ R holds (x1 , y2 ) ∈ R. 3. R = RX × RY , where RX = {x|∃y : (x, y) ∈ R} and RY = {y|∃x : (x, y) ∈ R}. Proof. 3. ⇒ 1. is obvious. 1. ⇒ 2. Let R = A × B. Since (x1 , y1 ), (x2 , y2 ) ∈ R we know on the one hand side, that x1 ∈ A and on the other hand side, that y2 ∈ B. Hence, (x1 , y2 ) ∈ A × B = R. 2. ⇒ 3. We show (1) R ⊆ RX × RY and (2) RX × RY ⊆ R. (1) Let (x, y) ∈ R. Clearly, x ∈ RX (since there is an y, such that (x, y) ∈ R). Similarly we have y ∈ RY . Together we obtain (x, y) ∈ RX × RY . (2) Let x1 ∈ RX and y2 ∈ RY . By construction of RX , RY this means, there are y1 ∈ Y and x2 ∈ X, such that (x1 , y1 ) ∈ R and (x2 , y2 ) ∈ R. By the assumption we can conclude (x1 , y2 ) ∈ R. Let P be a protocol computing f and let α = (m1 , . . . , mr ) be a transcript between Alice and Bob following this protocol. (Reminder: r depends on the input pair.) We denote the set of input pairs on which the protocols transcript is α as follows: Rα = {(x, y)|sP (x, y) = α}.


107

Fact 1. Rα ⊆ X × Y is a rectangle. Proof. We use induction and seemingly show a little bit more: The set Rαi of input pairs whose transcript only starts with αi = (m1 , . . . , mi ) is a rectangle. For i = 0 (no message sent) Rα0 = X × Y , which is a rectangle. For i = 1 (Alice sent her first message) Rα1 = A × Y for some A ⊆ X, since m1 depends only on Alice’s input x (A is simply the set of those x, for which Alice would send m1 ). Suppose Rαi is a rectangle A×B for some i ≥ 1. Without loss of generality we assume, that the next message is to be sent by Alice. By definition of the protocol this message depends only on x and the previous messages. Let A be the subset of A, on which Alice, given previous messages m1 , . . . , mi , sends mi+1 . Then Rαi+1 = A × B which again is a rectangle. Let’s think about the induction argument for a second. It says that after every round the input pairs, that are still “in the game” form a set of rectangular shape. That’s interesting, isn’t it? Fact 2. f (x, y) is the same for every input pair (x, y) ∈ Rα . Proof. Follows directly from the definition of the output. Fact 3. The family of sets RP = {Rα |α is a transcript of P on some input pair} is a partition of X × Y . This partition is called the protocol partition of P . Proof. Input pair (x, y) ∈ X × Y belongs to RsP (x,y) but to no other of the sets Rα . Definition 8. Let f : X ×Y → Z. We consider f like a coloring of the entries of X × Y . A rectangle R ⊆ X × Y is called f -monochromatic if f is constant on R. If f is understood, we sometimes simply speak of monochromatic rectangles. Especially, for z ∈ Z an f -monochromatic rectangle R ⊆ X × Y is called a z-rectangle if f (x, y) = z for all (x, y) ∈ R. We consider the partition number of f , which is the smallest number of f -monochromatic rectangles in a partition of X × Y : Cov D (f ) := min{T |∃ partition of X × Y into T monochromatic rectangles}. Remark 3. The superscript D in Cov D (f ) reminds to “disjoint” — we want a partition, not just a covering by monochromatic rectangles. Example 3. Recall how communication matrices of PARITY-functions look like (see Exercise 9). What is the partition number of such functions? Let’s first look at our PARITY3 -example from the exercise. Since order is not an issue for the rectangles we have in mind, we order the rows and columns such that first we have all n-vectors with even parity, and then we

108

Carsten Damm

have all n-vectors with odd parity. Then we fill in the matrix entries. As the example shows, this gives 4 rectangles: 0 0 0 0 1 1 1 1

0 0 0 0 1 1 1 1

0 0 0 0 1 1 1 1

0 0 0 0 1 1 1 1

1 1 1 1 0 0 0 0

1 1 1 1 0 0 0 0

1 1 1 1 0 0 0 0

1 1 1 1 . 0 0 0 0

It is obvious, that this holds in general: Entries of type even-even are 0-entries, as well as odd-odd-type entries. The 1-entries are covered by an even-odd and an odd-even rectangle. Hence the partition number of a PARITY-function is at most 4. Can it be smaller? If it was smaller, then we could combine, say, the 0-rectangles into one (the argument for 1-rectangles is similar). So lets take an even-even entry (x1 , y1 ) and an odd-odd entry (x2 , y2 ) and assume they are in the same monochromatic rectangle. But then, by the Characterization Lemma 1 also (x1 , y2 ) belongs to this rectangle. But this entry is of even-odd type, which is a contradiction. Hence Cov D (f ) = 4 for all n. Proposition 5. For every function f : X × Y → Z holds CC(f ) ≥ log Cov D (f ). Proof. By the above mentioned facts the protocol partition of P is a partition of X × Y into monochromatic rectangles. If P is an optimal protocol, on every input pair at most CC(f ) bits are exchanged. Therefore the number of possible transcripts (and therefore the number of rectangles in the particular partition induced by P ) is at most 2CC(f ) . Hence we obtain 2CC(f ) ≥ Cov D (f ). For later referencing we call this lower bound argument the tiling method. Since the partition number of PARITY-functions is 4, we can conclude, that at least 2 bits have to be exchanged to jointly determine PARITYn (x, y), hence the protocol PARITYn is optimal. Well, no big surprise . . . Example 4. Also no big surprise, but hopefully instructive: We reprove Proposition 2 CC(EQn ) = n + 1. But now we use partition numbers. An EQn -monochromatic partition clearly needs to have 2n 1-rectangles to cover all 1-entries (by an argument as in Example 3 we see, that no two of the 1’s can live in the same monochromatic rectangle.). On the other hand, we need at least one 0-rectangle to cover the 0-entries of the communication matrix. This means C(f ) > 2n , from which we conclude CC(EQn ) > n by the help of Proposition 5. But since CC(EQn ) is an integer number, it is at least n + 1.


109

Exercise 10. As the examples might suggest, it is tempting to believe, that each partition of the communication matrix of f into monochromatic rectangles there is indeed already a protocol partition. However, this is not the case. Can you find an example function of shape {1, 2, 3} × {1, 2, 3} → {0, 1} and partition into monochromatic rectangles, that is not a protocol partition? Remark 4. It is still open, whether for all f : {0, 1}n × {0, 1}n → {0, 1} holds CC(f ) = O(Cov D (f )). This question was posed in [25]. 4.2.2 The Fooling Set Method It is sometimes hard to prove lower bounds on monochromatic partition numbers. The following method is often easier to apply. It was first used in [32] and in a more elaborated form in [22]. Let f : X × Y → Z. Definition 9. A set of input pairs {(x1 , y1 ), . . . , (x , y )} is called a fooling set for f , if there exists some z ∈ Z, such that 1. for all i, f (xi , yi ) = z, 2. for all i = j, either f (xi , yj ) = z or f (xj , yi ) = z. For fixed z the set is called z-fooling set. Proposition 6. If f has a fooling set of size , then CC(f ) ≥ log . Proof. By Proposition 5 it is sufficient to prove Cov D (f ) ≤ . Suppose the opposite is true, i.e., suppose there is a partition of X × Y into less than monochromatic rectangles. Then there exist two pairs (xi , yi ), (xj , yj ) in the fooling set, that are in the same rectangle A × B. By definition z is the “color” that f gives to this rectangle. By the Characterization Lemma 1 we know that also (xi , yj ) and (xj , yi ) belong to this rectangle. But by definition of the fooling set, one of those pairs has a different color that z, which is a contradiction. Hence, there must be at least rectangles in any f -monochromatic partition of X × Y . Remark 5. The proof considers only rectangles colored z and therefore in fact we get a lower bound on the number of z-rectangles. By lower-bounding the number of rectangles of any color z ∈ Z and summing these bounds up, we can improve the bound: Let Z = {z1 , . . . , zt } and for i = 1, . . . , t let si be the size of some zi -fooling set for f , then CC(f ) ≤ log(s1 + . . . + st ). We make use of this argument in the following example. Example 5. We apply the bound to the disjointness function. Recall the interpretation: n-bit vectors x are considered as “encoding” subsets A from {1, . . . , n}: xi = 1 iff i ∈ A.

110

Carsten Damm

We claim, that the following is a 1-fooling set for DISJn : {(A, Ac ) | A ⊆ {1, . . . , n}}, where Ac denotes the complement of A in {1, 2, . . . , n}. Indeed, DISJn (A, Ac ) = 1 for all A and on the other hand for A = B either A ∩ B = ∅ or Ac ∩ B c = ∅. Hence we conclude CC(DISJn ) ≥ n. But this concerns only 1-rectangles. Since the function is non-constant, there must be at least one 0-rectangle. Hence, CC(DISJn ) ≥ log(2n + 1) = n + 1. Exercise 11. Use the fooling-set bound to show that CC(GTn ) = n + 1. Remark 6. The communication argument from Section 4.1.4 on the area-timeproduct of VLSI-circuits is easily seen to extend to functions like GE, GT and so on. 4.2.3 The Rank Method Recall that Mf denotes the communication matrix of f , i.e., the matrix with rows and columns indexed with inputs x ∈ X and y ∈ Y and entries f (x, y). The following bound is due to [26] Proposition 7. Let f : X × Y → {0, 1} and let rank(M ) denote the rank of a matrix over the rationals. Then CC(f ) ≥ log rank(Mf ). Proof. Consider a partition of X × Y into Cov D (f ) rectangles that are f monochromatic. Let R1 , . . . , Rt ⊆ X × Y be the 1-rectangles in this partition. By the tiling bound it is sufficient to prove rank(Mf ) ≥ t. We associate the following matrices of order |X| × |Y | to them: 1 , if (x, y) ∈ Ri . Mi (x, y) = 0 , else. Observe, that all non-zero rows of Mi are the same, t hence rank(Mi ) = 1 for all i. Since the Ri do not intersect, we have Mf = i=1 Mi . By the properties of the rank we conclude rank(Mf ) ≤

t

rank(Mi ) = t.

i=1

There is good chance to successfully apply the rank lower bound in cases, where the communication matrix features some algebraic property.


111

Example 6. Now we can prove a lower bound on the inner product function IP: CC(IPn ) ≥ n. To this end, let’s have a look on the communication matrices of IP1 , IP2 , IP3 , . . .. We use the natural binary ordering on rows and columns (e.g., in case n = 2 the order is 00, 01, 10, 11). Then the resulting matrices are: ⎛

MIP1

⎛ 0 ⎜ 0 00 = , MIP2 = ⎜ ⎝ 01 0 0

0 1 0 1

0 0 1 1

⎞ 0 1⎟ ⎟ , MIP3 1⎠ 0

0 ⎜0 ⎜ ⎜0 ⎜ ⎜0 =⎜ ⎜0 ⎜ ⎜0 ⎜ ⎝0 0

0 1 0 1 0 1 0 1

0 0 1 1 0 0 1 1

0 1 1 0 0 1 1 0

0 0 0 0 1 1 1 1

0 1 0 1 1 0 1 0

0 0 1 1 1 1 0 0

⎞ 0 1⎟ ⎟ 1⎟ ⎟ 0⎟ ⎟,... 1⎟ ⎟ 0⎟ ⎟ 0⎠ 1

Observe that except the first one, each matrix contains copies of its predecessor matrix in the left-upper, left-lower, and in the right-upper corner and a copy of the complement (0–1 exchanged) in the right-lower corner. These matrices are known as Sylvester-matrices and, their rank, e.g., over the rationals is one less 2 than full-rank. To see this, consider entry (x, z) of (MIPn ) . By definition this entry equals z∈{0,1}n IPn (x, z)IPn (z, y). This is the number of z ∈ {0, 1}n for which IPn (x, z) = IPn (z, y) = 1. This is an inhomogeneous linear system of equations modulo 2 in n indeterminates (z1 , . . . , zn ). In case x = 0 = y the number of solutions is 2n−1 if x = y and 2n−2 else. If one of x, y is identically zero the number of solutions is zero. We can conclude that rank(MIPn ) = 2n −1 and Proposition 7 gives CC(IPn ) ≥ n. Exercise 12. Apply Proposition 7 to the function GTn . 4.2.4 Comparison of Lower Bound Methods The fooling-set method and the rank-method rely on the tiling bound, therefore it is the potentially strongest longer bound method for communication complexity. However, direct application of the tiling-bound is difficult. It is interesting, to compare the relative power of lower bound methods on communication complexity. This study has been started in [2] and was continued in [13] and [10]. We report here some of the results from such study. For ease of exposition we adopt the following notations from [10], that all refer to a function of shape f : {0, 1}n × {0, 1}n → {0, 1}: • r(f ) = log rank(Mf ), • fs(f ) = log , where is the maximum size of a 1-fooling set for f , • t(f ) = log Cov D (f ).

112

Carsten Damm

Remark 7. Since in any case rank(Mf ) ≤ 2n , the rank lower bound can give no larger bound than n. On the other hand, the fooling set lower bound accounting for also 0-rectangles can give the optimal bound n + 1 (see Example 5). The reason for this somehow annoying difference is simple: An all-0-row is a 0-rectangle and would contribute to the fooling-set bound. However, its rank is 0 — so it would not contribute to the rank lower bound. Hence, it is only fair in a comparison, to restrict consideration to 1-fooling sets only. The following general inequalities are known: • t(f ) ≤ CC(f ) ≤ (t(f ) + 1)2 • r(f ) ≤ t(f ), fs(f ) ≤ t(f ) (see proofs above) • If n is sufficiently large, there are example functions f , such that CC(f ) = n but fs(f ) = O(log n) — the fooling-set bound may be very weak. Concerning the comparison of fs and r the following more detailed facts were shown 1. For almost all f : {0, 1}n × {0, 1}n → {0, 1} holds r(f ) = n and simultaneously fs(f ) = O(log n). Explicit constructions of such functions are known. 2. For all f : {0, 1}n × {0, 1}n → {0, 1} holds fs(f ) ≤ 2r(f ) and explicit constructions for f are known, for which fs(f ) = n but r(f ) < 0.8n. Let us comment on the first of these results: What does “for almost all” mean? It says, that if a function f : {0, 1}n ×{0, 1}n → {0, 1} is taken at random from the uniform distribution on the set of all such functions, then the probability that is has the mentioned properties tends to 1 as n grows to infinity. Exercise 13. For each k ∈ {0, 1, . . . , 2n} let EXACTk : {0, 1}n × {0, 1}n → {0, 1} be defined by 1 , if x + y = k EXACTk (x, y) = 0 , else. Show that CC(EXACTk ) ≥ log(k + 1). For the proof you can use any of the mentioned lower bound methods. Exercise 14. We know that symmetric functions f : {0, 1}n × {0, 1}n → {0, 1} have communication complexity bounded by O(log n). Give an example of such a function with CC(f ) ≥ log n. Remark 8. Taking into account that there must exist at least one 0-rectangle one can improve the result of Exercise 13 to CC(EXACTk ) ≥ log(k + 2) = log((k + 1) + 1).


113

Exercise 15. If the range of the function under consideration is greater than {0, 1}, the simple idea from the last remark can be extended. E.g., it can be shown that CC(MODk ) ≥ log(2k − 1) = log(k + (k − 1)), by presenting a 0-fooling set for and by taking into account, that there must be at least one 1-rectangle, at least one 2-rectangle, . . . , at least one (k − 1)-rectangle. Try this! But for this function it is easy to give better lower bounds on the number of those z-rectangles. This way one can prove CC(MODk ) ≥ 2 log k, which is your exercise.

4.3 Communication Complexity and Chomsky Hierarchy It is natural to ask, how communication complexity relates to other complexity measures or to restricting features of computational devices. We saw already one example concerning complexity measures in the field of VLSIdesign. Next we concentrate on the Chomsky hierarchy. We take some ideas from the exposition in [15]. Recall the Chomsky hierarchy: REG ⊂ CFL ⊂ CS ⊂ RE ⊂ ALL, where the notations in this order denote the classes of regular, context-free, context-sensitive, recursively enumerable, and all languages. (As usual, by a language we mean a subset of Σ ∗ for some fixed finite alphabet Σ.) To bring these two concepts in touch we first need to translate functions (studied in communication complexity) into languages (studied in formal language theory) and vice versa. We confine to Boolean functions and to languages over the alphabet {0, 1}. The key idea is to bring sequences of functions in correspondence to languages. Here is how: Definition 10. For every natural number N let fN : {0, 1}N → {0, 1} be a Boolean function. We denote by f the sequence f1 , f2 , . . .. The language defined by f is the set Lf = {w ∈ {0, 1}∗ |f|w| (w) = 1} (|w|, as usual, denotes the length of the string w). If L ⊆ {0, 1}n , then the Boolean function sequence fL defined by L is the sequence fL,N : {0, 1}N :→ {0, 1}, N = 1, 2, . . . with ∀w ∈ {0, 1}N : fL,N (w) = 1 ⇔ w ∈ L. We want to apply communication complexity ideas to member functions from sequences fL for arbitrary L. In the Boolean function examples studied so far, the input was always partitioned in equal sized parts and distributed to Alice and Bob. This is not possible for functions fL,N , if N is odd. But for these

114

Carsten Damm

cases we simply consider the input space as product of X = {0, 1}N/2 and Y = {0, 1}N/2 +1 — so we give the first N/2 bits to Alice (this input part is denoted x) and the remaining bits to Bob (this input part is denoted y). This said, we can now speak about the communication complexity of the language L by considering communication complexities of the member functions in the corresponding Boolean function sequence: CC(fL,N ) . For any g : N → N let CC(g(N )) denote the set of languages L ⊆ {0, 1}∗ , such that ∀N : CC(fL,N ) ≤ g(N ). We start with the top of the Chomsky hierarchy. The result of the following exercise seems disappointing at first glance, but at least it is instructive. We’ll comment on it afterwards. Exercise 16. First recall the solution to Exercise 4. Then use a diagonalization argument to prove the following statement: Proposition 8. There is a language L ⊆ {0, 1}∗ which is not recursively enumerable but has zero communication complexity: L ∈ CC(0) \ RE. Remark 9. The result is not really surprising. Each Turing machine is a finite object, processing every input uniformly by the same algorithm. Proposition 8 shows, that it is inadequate to compare a uniform computational mechanism (acceptors for languages in the Chomsky hierarchy) to non-uniform ones, like infinite sequences of communication protocols, that provide for each input length an own algorithm. There is a standard way to translate uniform devices to corresponding nonuniform ones (see [20]), that we introduce shortly: A computational device is called non-uniform if there is some infinite sequence α1 , α2 , . . . called advice that is used like an oracle mechanism: Instead of x the device processes the combination x#α|x| . If C is a certain complexity class, defined by a resource bounded uniform computational device, then C/g(N ) denotes the class of languages accepted by the same resources but with advice of length at most g(N ) for inputs of length N . To define this formally we introduce the following notation: C/g(N ) = {L : α|L ∈ C, α = (αN )N ∈N : ∀N |αN | ≤ g(N )} where L : α = {w|w#α|w| ∈ L}}. The proposition shows, that in order to accept the language L , a Turing machine acceptor needed the information which constant function is computed at inputs of length 1, length 2, . . . . This is only one bit of advice, or — in the terminology of [20] — CC(0) ⊆ RE/1. In fact, already a finite automaton equipped with this advice could accept L . It is straight-forward to extend the advice mechanism to finite automata (see [6]). Hence, the following is true: CC(0) ⊆ REG/1.


115

Proposition 9. For each regular language L ⊆ {0, 1}∗ there exists a constant k, such that L ∈ CC(k). Proof. See Chapter 4.4. So, for every regular language a constant number of communicated bits is sufficient to decide membership in that language. We express this fact as REG ⊆ CC(O(1)). On the other hand in [15] it is shown, that no single constant number of communicated bits is sufficient to decide about any regular language: Proposition 10. For every natural number k, there is a regular language Lk outside CC(k). Proof. Given k consider Lk = {w ∈ {0, 1}∗ : w = 2k+1 } ∈ REG. By Exercise 13 any deterministic communication protocol needs at least k + 1 bits of communication to decide w ∈ Lk . Remark 10. Let const (poly, respectively) denote the class of advice sequences α1 , α2 , . . . ∈ {0, 1}∗ such that ∀n : |αn | ≤ k (∀n : |αn | ≤ nk , respectively) for some constant k. Using notation and ideas from Remark 9, the proof of Proposition 9 and [6] one can show CC(O(1)) ⊃ REG/const. However, providing more than constant length advice does not help (see [6]): REG/const = REG/poly. How about context-free languages? Already in this language class there are languages requiring the highest possible — namely linear — communication complexity. Proposition 11. There is a context-free language L ⊆ {0, 1}∗ , such that L ∈ CC(o(N )). Proof (sketch). It is easy to show, that the set of palindromes, which is a context free language, has maximum communication complexity. Remark 11. Palindromes are hard to recognize in our basic two party communication model since the partition of input bits among Alice and Bob is worstcase. For other partitions (in the case n = 4, e.g., ({x1 , x2 , y1 , y2 }, {x3 , x4 , y3 , y4 })) communication complexity 2 would be sufficient. This is related to “best partition communication complexity”, discussed in Section 4.5.2. In [15] an example of a context-free set is presented that has communication complexity Ω(n) regardless how the (balanced!) partition of input bits among Alice and Bob looks like.

116

Carsten Damm

4.4 Communication Complexity Applications for Finite Automata and Turing Machines The following can be found in [21]. Lemma 2. Let f : {0, 1}n × {0, 1}n → {0, 1} and let there be a deterministic finite automaton with accepted language L, such that fL,2n = f (i.e., the “length 2n slice of L” is exactly the set of inputs xy with f (x, y) = 1). Then for the number s of its states holds CC(f ) ≤ log s + 1. Proof. Consider a deterministic finite automaton accepting L. Let {q0 , . . . , qs−1 } be its states and let k = log s+1. Then the following is a deterministic communication protocol, that accepts the input pair (x, y) if and only if the concatenation xy belongs to L. Alice simulates the automaton on the input x and passes the state qr in which the automaton ends in a binary encoding (no more than k bits) to Bob. Bob then starts simulating the automaton on y but with qr as starting state. He sends to Alice bit 1, if this computation ends accepting and 0 otherwise. Hence log s + 1 bits of communication are sufficient. Proof (Proposition 9). If L is regular, then it is recognized by some deterministic finite automaton with s states. From the lemma we can conclude, that L ⊆ CC(log s + 1). Here is another application: Exercise 17. Prove by means of communication complexity that the languages L = {xy||x| = |y| and EQ(x, y) = 1} and

L = {xy||x| = |y| and EQ(x, y) = 0}

are not regular. The third application concerns the following well-known simulation: For every s-state non-deterministic automaton there is a 2s state deterministic finite automaton that accepts the same language. We can show, that this construction is essentially optimal: Proof. We know from Exercise 17 that a deterministic finite automaton that accepts the language Ln = {xy||x| = |y| = n and EQn (x, y) = 0} needs at least 2n states. On the other hand, it is easy to design a nondeterministic finite automaton with O(n) states for Ln : It simply guesses two input positions i, 1 ≤ i ≤ n and j, n + 1 ≤ j ≤ 2n and accepts, if and only if the input differs in these positions.


117

Communication complexity finds also application for Turing machines. The idea of the below lower bound is quite similar to the situation with deterministic finite automata (and it reminds also to the VLSI-example from Section 4.1.4). We concentrate on the following example language: PALINDROME = {wwR |w ∈ {0, 1}∗ }, where xR denotes the reversed string x. Exercise 18. Prove that PALINDROME can be recognized by a Turing machine (1) in linear time using linear space or (2) in quadratic time using logarithmic space. Proposition 12. If a Turing machine accepts PALINDROME in time T (N ) using space S(N ) then T (N ) · S(N ) = Ω(N 2 ). Proof. Consider a Turing machine M that decides this language. Alice and Bob compute EQ(x, y) for (x, y) ∈ {0, 1}n × {0, 1}n by simulating the work of M on x0n yR ∈ {0, 1}N where N = 3n. Clearly M accepts if and only if EQ(x, y) = 1. Whenever the read-only input head is located in the “xregion” it is Alice who simulates and if it is in the “y-region” it is Bob who simulates. When the head enters the “0-region” the current player can continue to simulate until the head enters the region corresponding to the other player. It takes at least time n to move from the x- to the y-region, thus this happens at most T (N )/n times. Further, when the head crosses the responsibility border (i.e., when it walks out of the 0-region), the current player passes the necessary information to the other: the state (O(1) bits) and the contents of the work tape(s) (O(S(N )) bits). In total they exchange O(S(N ) · T (N )/n) bits while finding EQ(x, y). Because of Proposition 2 we can conclude T (N ) · S(N ) = Ω(n2 ). The idea works obviously also for other functions with linear communication complexity. Remark 12. In principle the same idea should also work for pushdown automata (divide the input in two parts and let Alice simulate on the left and Bob on the right). However, passing information between players is more difficult, because of the large amount of information that can be stored in the stack. In [16] a more sophisticated communication model was defined, that is applicable for lower bound proofs in this situation. The details are beyond the scope of this tutorial.

118

Carsten Damm

4.5 A Survey on Communication Problems and Applications 4.5.1 Different Modes of Communication The model of communication that was introduced in Section 4.1.2 is the basic deterministic model. However, the concept of non-determinism known from computational complexity theory turns out to be fruitful also in communication complexity. We survey some of these and related notions and results. A non-deterministic communication protocol is a protocol, that may allow players to choose one of several messages to send. We define the computed value to be 1 if at least one transcript gives output 1. The length of the protocol transcript for the worst-case input pair is the complexity of the nondeterministic protocol. The complexity of the best nondeterministic protocol for a function f : {0, 1}n × {0, 1}n → {0, 1} is called the nondeterministic communication complexity of f and is denoted CC N (f ) . Still, it holds that the set Rα of all input pairs with transcript α is a monochromatic rectangle. It is not hard to see, that every nondeterministic protocol for f defines a (not necessarily disjoint) covering of {0, 1}n × {0, 1}n by monochromatic rectangles. Let Cov 1 (f ) denote the minimum number of 1-rectangles required to cover Mf . Exercise 19. Prove that CC N (f ) = log Cov 1 (f ). Let Cov 0 (f ) denote the minimum number of 0-rectangles needed to cover Mf . The following is proved in [13]: Proposition 13. CC(f ) = O(Cov 1 (f ) · Cov 0 (f )). Cov 1 (f ) and Cov 0 (f ) are called cover numbers. Let ms(f ) denote the maximum size of an f -monochromatic 1-rectangle. The following is obvious: Lemma 3. CC(f ) ≥ Cov 1 (f ) + Cov 0 (f ). CC N (f ) ≥ #f −1 (1)/ms(f ). Exercise 20. Use properties of Sylvester-matrices (see Example 6) to prove n #IP−1 n (1) > 3 . It can be shown, that 1-rectangles of IPn have size at most 2n : ms(IPn ) ≤ 2n . Using the lemma and the exercise this gives: Proposition 14. CC N (IPn ) = Ω(n).


119

The rectangle size argument was extended to randomized communication protocols in [17]. In randomized communication protocols players make use of random bits and the input is to be accepted with a certain probability. There is a “private coins model” (each player uses a separate source of randomness) and a “public coin model” (players share the random bits). In the latter model the random bits are not taken into account for communication. Example 7. There is a private coin random communication protocol that computes EQn within O(log n) bits of communication and with error at most 1/n. Proof (sketch). Let p, n2 < p < 2n2 be a prime and consider the input vectors as coefficients of degree n − 1 polynomials. So every player has a polynomial. Alice picks some r, 0 ≤ r < p at random and evaluates her polynomial at r. She sends r as well as the value modulo p to Bob. Bob evaluates his polynomial at the point r. If his result equals hers modulo p, then he accepts. Otherwise he rejects. No more than O(log n) bits are communicated and the probability of error is small: If the input parts are the same, the polynomials are the same and no error occurs. If the inputs are different, the polynomials differ and there is good chance that this will be detected by random evaluation. Another type of “communication mode” concerns rounds. The protocols mentioned in previous sections are all 1-round protocols: Alice sends some bits and Bob can compute the function value. The question is, whether it is sufficient to restrict attention to protocols bounded to a certain number of rounds or how much round restrictions affect length of protocols. This study was started in [29]. We mention some results: •

The protocol used in the proof of Proposition 13 uses binary search in a number of rounds that grows as log Cov 0 (f ). No deterministic protocol with a constant number of rounds is known, that gives the same upper bound. • For each fixed k functions are known that need exponentially more communicated bits in k − 1-round games compared to k-round games (see [11] for the deterministic version and [27] for the randomized). • There is a connection between round-restricted protocols and depth/sizetradeoffs for Boolean circuits [27]. 4.5.2 Different Partitions

Recall the VLSI-example from Section 4.1.4. Processors compute EQn but bits to be compared are at a large distance from each other. This is, what makes the computation costly. Without this restriction, the lower bound AT ≥ n2 would no longer be applicable. Exercise 21. Describe a layout for a VLSI-circuit with arbitrary arrangement of inputs that achieves an area-time product o(n2 ).

120

Carsten Damm

In view of this it seems meaningful to consider communication complexity of functions on base of arbitrary input partitions. This model was introduced in [29] and is heavily used for VLSI lower bound proofs: Definition 11. Let (S, T ) be a partition of the set of inputs of a boolean function with #S = #T . An (S, T )-communication protocol is a communication protocol that enables Alice and Bob (given the bits with index i ∈ S and with index i ∈ T , respectively) to jointly compute the value of f on the given input. Its complexity is defined as usual. The communication complexity of f with respect to (S, T ) (denoted CCS,T (f )) is the complexity of the best protocol with respect to S, T . The best partition communication complexity of f is the minimum of CCS,T (f ) over all input set partitions (S, T ): CC best (f ) := min CCS,T (f ) . (S,T )

The “benchmark function” for best partition communication complexity is: shifted equality

1 , if x(i) = y SEQn (x, y, i) = 0 , otherwise,

where x(i) denotes the cyclic shift of x by i places. Proposition 15. CC best (SEQn ) = Ω(m), where m = 2n + log n is the size of the input. Proposition 16. Every VLSI-chip that computes f satisfies AT 2 ≥ (CC best (f ))2 . For proofs see [21]. 4.5.3 Different Games Depending on the situation to be modeled different versions of “communica tion games” are in use. The basic model is the two party communication game as introduced. To give an impression we mention only two of them: • Multiparty communication: The input consists of k parts x1 , . . . , xk . An obvious and hopefully useful generalization of the two-party case is the following version (1): The share of player i consists of xi . Less obvious but also useful is version (2): The share of player i consists of all xj , 1 ≤ i ≤ k except xi . Version (2) is dubbed “number-on-the-forehead-model” (NOF) while version (1) is the “number-in-the-hand-model”. The NOF version was introduced by [5]. There are surprising connections between this model and Boolean circuit complexity. And there are also surprising protocols (see, e.g., [12, 7]).


•

121

Communication complexity of relations, which was introduced in [19]: A relation is a subset R ⊆ X ×Y ×Z. Alice is given x ∈ X, Bob is given y ∈ Y and their task is to find some z ∈ Z, such that (x, y, z) ∈ R. For an example consider the so-called universal relation: Un ⊆ {0, 1}n ×{0, 1}n ×{1, . . . , n} is defined to be the set of all triples (x, y, i) such that x = y and xi = yi . The corresponding communication game is the following: Alice is given x and Bob is given y. They know in advance, that x = y. Their task is to find some bit i, in which their inputs differ.

The communication complexity of such protocols is defined in the usual way — it is the length of the best protocol on the worst case input. Let us elaborate somewhat more on communication complexity of relations. Let f : {0, 1}n → {0, 1} be a non-constant Boolean function. Suppose Alice is given some x ∈ f −1 (0) and Bob is given some y ∈ f −1 (1). Then they know, that their input parts differ in at least one bit and they can communicate to find one such position. This is the communication game on the following relation associated to f : Rf := {(x, y, i)|x ∈ f −1 (0), y ∈ f −1 (1), xi = yi }. Obviously, every protocol for Un gives a protocol for Rf — this is where the name “universal relation” comes from. The trivial protocol for Un gives an upper bound of n + log n communicated bits. In [30] several protocols are given that achieve an upper bound of n + 2 (while the lower bound — proved in the same paper — is n + 1). We denote by CC(Rf ) the communication complexity of the relation Rf . Further let d(f ) denote the minimum depth of a Boolean circuit for f that uses only ∧-, ∨-, and ¬-gates. The following interesting connection was proved in [19]: Lemma 4. d(f ) = CC(Rf ) . This lemma was used in [4] for a new proof of the following fact, that is known from 1950s: Proposition 17. For every symmetric Boolean function f holds d(f ) = O(log n). Remark 13. The proof relies on a the construction of an O(log n) protocol for Rf . Please note, that this is not related to the result of Exercise 7.

References 1. H. Abelson. Lower bounds on information transfer in distributed computations. In Proceedings of the 19th IEEE FOCS. IEEE Computer Society, Ann Arbor, pp. 151-158, 1978.

122

Carsten Damm

2. A. Aho, J. Ullman, and M. Yannakakis. On notions of information transfer in VLSI-circuits. In Proceedings of the 15th Annual ACM Symposium on Theory of Computing, pages 133–139, 1983. 3. L. Babai, P. Frankl, and J. Simon. Complexity classes in communication complexity theory. In Proceedings of the 27th Annual Symposium on Foundations of Computer Science, FOCS 86. IEEE Computer Society, Toronto, pp. 337-347, 1986. 4. G.S. Brodal and T. Husfeldt. A communication complexity proof that symmetric functions have logarithmic depth. Brics report series. http://www.brics. dk/RS/96/1/BRICS-RS-96-1.ps.gz. 5. A.K. Chandra, M.L. Furst, and R.J. Lipton. Multi-party protocols. In STOC., Boston, ACM, pages 94–99, 1983. 6. C. Damm and M. Holzer. Automata that take advice. In Mathematical Foundations of Computer Science, LNCS, vol 969, pp. 149-158, 1995. 7. C. Damm, S. Jukna, and J. Sgall. Some bounds on multiparty communication complexity of pointer jumping. Computational Complexity, 7(2):109–127, 1998. 8. C. Damm, M. Krause, C. Meinel, and S. Waack. On relations between counting communication complexity classes. Journal of Computer and System Sciences, 69:259–280, 2004. 9. R. de Wolf. Quantum communication and complexity. Theoretical Computer Science, 287:337–353, 2002. 10. M. Dietzfelbinger, J. Hromkovič, and G. Schnitger. A comparison of two lowerbound methods for communication complexity. Theoretical Computer Science, 168(1):39–51, 1996. 11. P. Duriš, Z. Galil, and G. Schnitger. Lower bounds on communication complexity. IC, 73:1–22, 1987. 12. V. Grolmusz. The bns lower bound for multi-party protocols is nearly optimal. Information and Computation, 112:51–54, 1994. 13. B. Halstenberg and R. Reischuk. Different modes of communication. SIAM Journal on Computing, 22, 1993. 14. J. Hromkovič. Randomized communicating protocols (a survey). In Proceedings of the International Symposium on Stochastic Algorithms: Foundations and Applications. LNCS, vol. 2264, Springer, pp. 1-32. 15. J. Hromkovič. Communication Complexity and Parallel Computing. Springer, Berlin, 1997. 16. J. Hromkovič and G. Schnitger. Pushdown automata and multicounter machines, a comparison of computation modes. In Automata, Languages and Programming: 30th International Colloquium, pages 66–80, 2003. 17. M. Karchmer, E. Kushilevitz, and N. Nisan. Fractional covers and communication complexity. SIAM Journal on Discrete Mathematics, 8:76–92, 1995. 18. M. Karchmer, I. Newman, M. Saks, and A. Wigderson. Non-deterministic communication complexity with few witnesses. Journal of Computer and System Sciences, 49:247–257, 1994. 19. M. Karchmer and A. Wigderson. Monotone circuits for connectivity require super-logarithmic depth. In STOC., pages 539–550, 1988. 20. R.M. Karp and R.J. Lipton. Turing machines that take advice. L’Enseignement Mathématique, 28:191–209, 1982. 21. E. Kushilevitz and N. Nisan. Communication Complexity. Cambridge University Press, 1996.


123

22. R.J. Lipton and R. Sedgewick. Lower bounds for VLSI. In 13th Annual ACM Symposium on the Theory of Computing, pages 300–307, 1981. 23. S.V. Lokam. Spectral methods for matrix rigidity with applications to size-depth tradeoffs and communication complexity. In FOCS, pages 6–15, 1995. 24. L. Lovász. Communication complexity: A survey. In B. Korte, L. Lovász, H.J. Prömel, and A. Schrijver, editors, Paths, Flows, and VLSI Layout. Algorithms and Combinatorics 9, pages 235–265. 1990. 25. L. Lovász and M. Saks. Lattices, Möbius functions and communication complexity. Journal of Computer and System Sciences, 47:322–349, 1993. 26. K. Mehlhorn and E.M. Schmidt. Las vegas is better than determinsm in VLSI and distributed computing. In 14th Annual ACM Symposium on Theory of Computing, pages 330–337, 1982. 27. N. Nisan and A. Wigderson. Rounds in communication complexity revisited. SIAM Journal on Computing, 22, 1993. 28. A. Orlitsky and A. Gamal. Communication complexity. In Complexity in Information Theory, pages 16–61. 1988. 29. C. Papadimitriou and M. Sipser. Communication complexity. JCSS, 28:260–269, 1984. 30. G. Tardos and U. Zwick. The communication complexity of the universal relation. In Proceedings of the 12th IEEE Conference on Computational Complexity, pages 247–259, 1997. 31. C. Thompson. Area-time complexity for VLSI. In Proceedings of the 11th ACM STOC. ACM, Atlanta, pp. 81-88. 32. A.C.C. Yao. Some complexity questions related to distributive computing. In Proceedings of the 11th Annual ACM Symposium on the Theory of Computing, Atalanta, pp. 209-213, 1979.

5 Formal Languages and Concurrent Behaviours Jetty Kleijn1 and Maciej Koutny2 1

2

LIACS, Leiden University P.O.Box 9512, NL-2300 RA Leiden, The Netherlands [email protected] School of Computing Science, Newcastle University Newcastle upon Tyne, NE1 7RU, United Kingdom [email protected]

Summary. This is a tutorial based on a course delivered as part of the International PhD School in Formal Languages and Applications located at the Rovira i Virgili University in Tarragona, Spain. It is focused on an application of formal language theory to represent behaviours of concurrent systems necessitating a generalisation of language theory to traces, which originates with the work of Mazurkiewicz in 1977. The tutorial uses Petri nets as an underlying system model which allows one to clearly distinguish between causality and independence between executions of actions, a major feature of concurrent behaviour.

Keywords: formal languages, traces, concurrency, causality, independence, partial orders, Petri nets, inhibitor arcs, processes.

5.1 Introduction The dynamic behaviours of concurrent systems are not always adequately captured with the help of purely functional input-output descriptions. Often one is more interested in the modelling of ongoing evolutions of such systems at the interface with the environment, e.g., when communicating or reacting to external stimuli. One way of representing such evolutions is to use sequences of executed actions, which in a natural way leads to a formal language semantics of dynamic systems. A successful example of this approach are finite state machines and their languages which have numerous applications in almost every branch of Computer Science. Another example are Turing machines and their languages which delimit the effectiveness of computational behaviour. Both classes of machines are of a sequential nature which makes languages a suitable semantical domain. However, plain words and languages are only of limited usefulness when it comes to faithful representation of concurrent behaviours. For example, a sequential description of behaviour cannot be used to describe the result of action refinement for which the information about J. Kleijn and M. Koutny: Formal Languages and Concurrent Behaviours, Studies in Computational Intelligence (SCI) 113, 125–182 (2008) c Springer-Verlag Berlin Heidelberg 2008 www.springerlink.com

126

Jetty Kleijn and Maciej Koutny

concurrency or independence, as opposed to causality, is of crucial importance. The problem of insufficient expressibility of sequential descriptions was recognised in the 1970s when concurrent systems became a focus of intensive research activity carried out by various groups, and when it was realised that additional information should be added to the sequential descriptions of system behaviours. An example coming from the process algebra world is Milner’s observational equivalence [39] — replacing language equivalence as a means of comparing different systems — which allows one to identify the exact points when choices between alternative actions were made during system executions. Another such example are traces — introduced by Mazurkiewicz [35] — providing explicit information on the causal dependencies between executed actions. It is this latter approach which is the subject of this tutorial. A key concept behind traces is that a given concurrent run can be observed in different ways depending on the viewpoint of the observer and on the particular way of recording this behaviour. Trace theory then provides a tool which allows one to identify in a precise way different observations of a concurrent behaviour. In its most basic form, traces equate sequences of executed actions on basis of given independencies between such actions. The original idea of Mazurkiewicz was to use the well-developed tools of formal language theory for the analysis of concurrent systems, understood as Petri nets [44, 45, 46]. Petri nets are an operational model which directly generalises state machines (labelled transition systems) through the notions of a state and state change. Both models allow a graphical representation. What makes Petri nets radically different from state machines is their ability to represent states as distributed entities, and state changes as affecting only local parts of these distributed states in a way prescribed by the underlying graphical structure. Whereas the executions or runs of a sequential system consist of actions executed one-by-one in a totally ordered fashion, the actions of a run of a concurrent system are not necessarily executed one after the other. Having distributed states and local effects makes concurrency aspects explicit since actions may be independent in the sense that they involve disjoint parts of distributed states. Hence words and languages (sets of words) which are appropriate models for the behaviours of sequential systems are no longer sufficient since actions executed in a concurrent system are only partially ordered. The concern of trace theory is how to add information to observations in order to convey the essence of causality between executed actions (i.e., the necessary ordering in the sense that cause must precede effect). Trace theory has as its starting point an alphabet of action names enriched with information about which (occurrences of) actions are independent (or non-interfering). A key assumption is that the order of observation of two independent actions is accidental. Hence two sequential observations (words) which differ only w.r.t. the order of occurrence of independent actions are in fact observations of the same concurrent run and consequently may be identified. A trace is then simply the resulting equivalence class of all

5 Formal Languages and Concurrent Behaviours

127

sequential observations of the same underlying concurrent run. In this way, formal language theory is lifted to quotient structures consisting of equivalent observations, and a number of standard language theoretic tools can therefore be adapted and applied in the analysis of concurrent systems. Moreover, by explicitly recording the dependencies between executed actions a unique (causal) partial order can be associated to each trace. In other words, traces can be seen as partial orders in the same way as words can be seen as total orders. The tutorial is organised in the following way. After a preliminary section on sets, graphs and languages, we introduce traces and recall their main properties, including the underlying dependence graphs. We then consider Elementary Net systems [47] which are generally regarded as the most fundamental class of Petri nets, and were indeed the model which inspired the introduction of traces. We investigate both sequential and non-sequential ways of executing them. The trace-based behaviour is obtained by taking sequential executions and combining them with the structural information about the dependencies between executed actions obtained from the graph structure of a net. That this approach is sound follows from the fact that the partial orders defined by traces coincide with the partial order semantics of nets represented by the non-sequential observations captured by operationally derived processes. This treatment is then repeated for two significant, and practically relevant, extensions of Elementary Net systems (note that ‘Petri net’ is actually a generic name rather than a single model). The first extension consists in adding inhibitor arcs to the net, and the other in extending the notion of a global state. In each case we demonstrate the necessary generalisations of the concept of action independence, leading to comtraces and local traces, respectively. The first is based on the enhanced structure of the net whereas the other is history dependent. The tutorial is based on existing work and contains no proofs. The main text presents key definitions and results as numbered items, whereas supporting observations (and suggested exercises for the reader) are marked with the symbol. For further background, proofs and references the reader is provided with bibliographical remarks at the end of each technical section. 5.1.1 A Running Example Throughout this tutorial, we will discuss various aspects of simple yet practically relevant concurrent systems consisting of producers, buffers and consumers. We start with a model consisting of three (sequential) components: a producer Prod, buffer of capacity one Buff and consumer Cons. Each component is characterised by its language which is the set of finite sequences of atomic actions it can execute. Prod can execute three actions: m representing making of an item, a representing adding of a newly produced item to the buffer, and r representing the retirement of the producer. Cons can execute two actions:

128

Jetty Kleijn and Maciej Koutny init

init

init

r m

a

FSMProd

g

a

FSMBuff

g

u

FSMCons

Fig. 5.1. Finite state machines for the running example.

g representing getting an item from the buffer, and u representing using the newly acquired item. Buff can execute the a and g actions. The sequences of allowed action executions are given by the regular languages generated by the three finite state machines shown in Figure 5.1 (all states in these machines are considered final). The three components together form a concurrent system in which they operate independently except for the requirement that the actions shared by two processes are executed if both of the processes (can and) do so. For example, ama is a valid action sequence for the producer, but it is not a valid behaviour of the combined system, because the buffer does not allow two a actions without an intermediate g. On the other hand, the sequence amgau can be executed by the whole system, and so it is a valid history. Note that amgru is a history leading to a deadlock, i.e., a global state in which none of the five actions can be executed.

5.2 Preliminaries df

A relational tuple is a tuple reltuple = (X1 , . . . , Xm , Q1 , . . . , Qn ) where the Xi ’s are disjoint sets forming the domain, and the Qi ’s are relations involving the elements of the domain and perhaps some other elements.1 For example, directed graphs and finite state machines can be regarded as relational tuples. In fact, in all cases considered later on, a relational tuple can be viewed as a graph of some sort and we will use the usual graphical conventions to represent its nodes (i.e., the elements of its domain), various relationships between these nodes, and some particular characteristics of these nodes (e.g., the initial state of a finite state machine, or a labelling of the elements). A particular issue which links together various kinds of relational tuples is the idea that what really matters is the structures they represent rather than the identities of the elements of their domains. A technical device which can be used to capture such a view is as follows: two relational tuples, reltuple and reltuple , are isomorphic if there is a bijection ψ from the domain of reltuple to the domain of reltuple such that if we replace throughout reltuple 1 In this tutorial, m ≤ 2 and n ≤ 4. Note that suitable Qi ’s can represent functions on the domain as well as subsets and individual elements of the domain.


129

each element x in its domain by ψ(x) then the result is reltuple .2 It is then standard to consider isomorphic relational tuples as undistinguishable. 5.2.1 Set Theoretic Notations N denotes the set of natural numbers including zero. The powerset of a set X is denoted by P(X), and the cardinality of a finite set X by |X|. Sets X1 , . . . Xn form a partition of a set X if they are non-empty disjoint subsets of X such that X = X1 ∪ . . . ∪ Xn . A labelling for a set X is a function from X to a set of labels. The df labelling can be applied to finite sequences of elements of X, (x1 . . . xn ) = df (x1 ) . . . (xn ), and to finite sequences of subsets of X, (X1 . . . Xn ) = (X1 ) . . . (Xn ). The composition R ◦ Q of two relations R ⊆ X × Y and Q ⊆ Y × Z comprises all pairs (x, z) in X ×Z for which there is y in Y such that (x, y) ∈ R and (y, z) ∈ Q. Definition 1 : relations Let R be a binary relation on a set X. • • • • • • •

R−1 = {(y, x) | (x, y) ∈ R} denotes the inverse of R. df R0 = id X = {(x, x) | x ∈ X} is the identity relation on X. df Rn = Rn−1 ◦ R is the n-th power of R, for n ≥ 1. + df R = R1 ∪ R2 ∪ . . . is the transitive closure of R. df R∗ = R0 ∪ R+ is the transitive and reflexive closure of R. R is symmetric / reflexive / irreflexive / transitive if, respectively, R = R−1 / id X ⊆ R / id X ∩ R = ∅ / R ◦ R ⊆ R. R is acyclic if R+ is irreflexive. df

The restriction of a function f : X → Y to a subset Z of X is denoted by f |Z , and of a relation R ⊆ X × Y to a subset Z of X × Y by R|Z . df The domain of R is dom R = {x | (x, y) ∈ R} and its codomain is given by df codom R = {y | (x, y) ∈ R}. We will often use the infix notation x R y to denote (x, y) ∈ R. Definition 2 : equivalence relations A binary relation R on a set X is an equivalence relation if it is reflexive, symmetric and transitive. An equivalence class of R is any maximal subset of equivalent elements.

2 This definition is not strictly formal, but it should convey sufficient meaning to make the presentation clear.

130


In other words, R is an equivalence relation iff R = (R ∪ R−1 )∗ holds . If R is an equivalence relation on X, then X/R denotes the set of all equivalence classes of R. Given an equivalence relation R on X and a function f defined for ntuples of elements of X, it is often useful to lift f to n-tuples of equivalence df classes of R by setting f (Rx1 , . . . , Rxn ) = f (x1 , . . . , xn ) where each Rxi is the equivalence class of R containing xi . We say that f is well-defined on X/R if the value returned does not depend on the choice of the representing element xi from Rxi . Definition 3 : partial orders A binary relation R on a set X is a partial order if it is irreflexive and transitive. In other words, R is a partial order iff R = R+ \ id X holds

.

Definition 4 : partially ordered sets A labelled partially ordered set (or poset) po = (X, ≺, ) is a relational tuple consisting of a finitea set X, a partial order ≺ on X, and a labelling of X. The poset is total (or linear) if, in addition, all distinct elements of X are ordered. df

a

For simplicity we restrict ourselves to finite posets.

More precisely, po is total if x ≺ y or y ≺ x for all x = y in X. Two elements x = y of X are unordered if neither x ≺ y nor y ≺ x; we denote this by x y. Moreover, we write x y if x ≺ y or x = y. A total poset tpo is a linearisation of a poset po if they have the same domain and labelling, and the partial order relation of the former extends (includes) the partial order relation of the former. The set of all linearisations of po is denoted by lin(po). The intersection T PO of a non-empty set of total posets tpo = (X, ≺tpo , ) with the same domain X and labelling is (X, ≺, ) where ≺ is the relation comprising all pairs (x, y) of elements of X such that x ≺tpo y for each tpo inT PO. The set of all linearisations of a poset po is non-empty and po = lin(po) which means that any poset can be identified with its set of linearisations . 5.2.2 Directed Acyclic Graphs As usual, we define a labelled directed graph (or simply graph) G as a relational tuple (V, A, ) consisting of a set of nodes V , a set of arcs A ⊆ V × V ,


131

and a labelling of V . (For simplicity we restrict ourselves to finite graphs, i.e., with a finite set of nodes). Definition 5 : dags A graph G = (V, A, ) is acyclic (transitive) if A is an acyclic (transitive) relation. A dag is a (directed) acyclic graph. Figure 5.2 shows an example of a dag G with six nodes. Any poset is a dag and, conversely, any dag defines a (unique) poset after adding all arcs implied by transitivity: for a dag G = (V, A, ) the relation A+ is a partial order on df V , and we refer to the graph G+ = (V, A+ , ) as the transitive closure of G. Deleting all transitive arcs from G yields its Hasse diagram, i.e., the graph df hasse(G) = (V, A \ (A ◦ A+ ), ). Figure 5.2 shows examples of both these notions. The Hasse diagram of a poset po is the minimal (w.r.t. the number of arcs) dag G such that G+ = po. Moreover, a dag is a poset iff it is its own transitive closure, and the transitive closures of two dags coincide iff their Hasse diagrams coincide .

G

G+

hasse(G)

Fig. 5.2. A dag, its transitive closure with added arcs indicated by dotted lines, and its Hasse diagram. Note that node labels are omitted as they are irrelevant.

5.2.3 Words and Languages Alphabets, words, and languages are the main notions for recording the sequential view of a system’s behaviour. Definition 6 : alphabets An alphabet Σ is a finite non-empty set of symbols. A word (over Σ) is any finite sequence a1 · · · an of symbols ai (from Σ), and a language (over Σ) is any set of words (over Σ). In the case that n = 0 in the above definition, one is dealing with the empty sequence or empty word, denoted by λ. The set of all words over Σ including λ, is denoted by Σ ∗ .

132

Jetty Kleijn and Maciej Koutny λ

cantotalposet(agmau)

a

ag

a

g

m

a

u

1

g1

m1

a2

u1

a

agm

agma

agmau

Fig. 5.3. Hasse diagram of the prefix ordering for the word agmau, and Hasse diagram of its canonical total poset showing the identities of its nodes (bottom) and labels (top).

Definition 7 : words Let u = a1 . . . an and v = b1 . . . bm be two words over an alphabet Σ, (m, n ≥ 0). • • • • •

df

uv = a1 . . . an b1 . . . bm is the concatenation of u and v. df length(u) = n is the length of u. alphabet(u) comprises all symbols occurring within u. #a (u) is the number of occurrences of a symbol a within u. occ(u) is the set of symbol occurrences of u comprising all indexed symbols ai with a ∈ alphabet(u) and 1 ≤ i ≤ #a (u).

For the running example in this tutorial, we will use the alphabet Σ = {a, g, m, r, u}. For the string u = agmau over Σ, we have: length(u) = 5, alphabet(u) = {a, g, m, u}, #a (u) = 2 and occ(u) = {a1 , a2 , g1 , m1 , u1 }. Fact 8 : The set of all words over an alphabet Σ with concatenation and the empty word forms a monoid. That is, concatenation is associativea and λ is its unit.b a b

(uv)w = u(vw) for all u, v, w in Σ ∗ . λu = uλ = u for each u in Σ ∗ .

A word u is a prefix of a word v if v = uw for some word w. We denote this by u v. Moreover, if u v and u = v then we write u v. The prefix relation on words is a partial order . For example, ag agma. For a given word, is a total order on its prefixes which can be interpreted as saying that every word has a unique history (see Figure 5.3). Moreover, every word, being a sequence of symbols, corresponds directly to a total poset. This is an important relationship now made precise. To start with, the elements of a total poset tpo = (X, ≺, ) can be listed as a (unique) sequence x1 . . . xn such that xi ≺ xj iff i < j. The word generated df by tpo is then defined as word (tpo) = (x1 . . . xn ). Total orders are isomorphic iff the words they generate are the same . Now, given a word u it is clearly possible to see it as corresponding to any total poset tpo such that word (tpo) = u. Since all such total posets are


133

isomorphic it does not really matter which one is chosen and for our purposes it is convenient to single out one such poset. It is called the canonical total df poset of u and is defined as cantotalposet(u) = (occ(u), ≺, ) where ai ≺ bj if the i-th occurrence of a precedes the j-th occurrence of b within u, and df (ai ) = a, for all symbol occurrences ai and bj in occ(u). Distinct words have distinct canonical total posets, and the word generated by the canonical total poset of a word is that word itself . 5.2.4 Bibliographical Remarks We have recalled the necessary notions and results used later in this tutorial. Most of them are standard. We only mention that the fact that any poset can be identified with its set of linearisations is usually known as ‘Szpilrajn’s Theorem’ and has been first given in [49] in a fully general setting.

5.3 Traces Words represent a sequential view of the actions executed by a system. As such, no further information is provided on the intrinsic dependencies among the actions and the resulting necessary ordering of their occurrences. The introduction of traces starts from the definition of a concurrency alphabet, which simply states which symbols are considered as representing independent actions (not interfering with each other) and thus should be treated as concurrent. Definition 9 : concurrency alphabets df

A concurrency alphabet is a pair CA = (Σ, Ind ) where Σ is an alphabet and Ind is an irreflexive and symmetric binary relation over Σ called an independence relation. When two symbols are not independent (i.e., they do not appear as a pair in the given independence relation), they are said to be dependent. Note that this dependence relation is reflexive. Let (Σ, Ind ) be a concurrency alphabet and let u, v ∈ Σ ∗ . We write u ∼Ind v if there are words w and z and independent symbols (a, b) ∈ Ind such that u = wabz and v = wbaz. Thus u ∼Ind v means that they are the same word except for a change of order of two adjacent occurrences of independent symbols. Trace equivalence (with respect to (Σ, Ind )) is the equivalence relation ≡Ind over Σ ∗ obtained as the reflexive and transitive closure of ∼Ind . Thus, two words are trace equivalent if one can be obtained from the other by changing (repeatedly) the order of adjacent occurrences of independent symbols. This means that u ≡Ind v iff #a (u) = #a (v) for all a ∈ Σ and

134


the order of occurrences ai and bj is the same within u and v for all pairs a, b of dependent symbols and for all 1 ≤ i ≤ #a (u) and 1 ≤ j ≤ #b (u). Continuing our running example with Σ = {a, g, m, r, u}, we assume CA = (Σ, Ind) is the concurrency alphabet with independence relation Ind given by: Ind = {(r, g), (g, r), (r, u), (u, r), (m, g), (g, m), (m, u), (u, m), (a, u), (u, a)}. Then agmr ∼Ind amgr ∼Ind amrg and so agmr ≡Ind amrg. Definition 10 : traces A trace over a concurrency alphabet (Σ, Ind ) is any equivalence class of the trace equivalence relation ≡Ind , and a trace language is any set of traces over (Σ, Ind ). The trace containing a given word u is denoted by [u]Ind , and the set of all traces over (Σ, Ind ) by Σ ∗ /≡Ind . Whenever the independence relation Ind is understood, we may drop the subscript Ind . Note that the empty trace [λ] is {λ} rather than the empty set. For the running example, [amgr] = {amgr, agmr, amrg}. If two words are trace equivalent, then both their lengths and alphabets are the same . Hence, the alphabet of a trace and its length, defined in the df df natural way as alphabet(α) = alphabet(u) and length(α) = length(u), where u is any word belonging to the trace α, are both well-defined notions. Also, trace concatenation (or sequential composition, in operational terms) defined df as [u]◦[v] = [uv] is a well-defined operation. This follows from the observation that uv and u v are trace equivalent whenever [u] = [u ] and [v] = [v ]. Fact 11 : The set Σ ∗ /≡Ind of all traces over a concurrency alphabet (Σ, Ind ) is a monoid. That is, concatenation of traces is an associativea operation with the empty trace [λ] as its unit.b a b

(α ◦ β) ◦ γ = α ◦ (β ◦ γ) for all α, β, γ in Σ ∗ /≡ . α ◦ [λ] = [λ] ◦ α = α for each α in Σ ∗ /≡ .

The last result can be pushed a little bit further. Fact 12 : The trace monoid Σ ∗ /≡Ind is partially commutative in the sense that, for any pair of traces α and β, alphabet(α)×alphabet(β) ⊆ Ind implies α ◦ β = β ◦ α and, moreover, the converse holds whenever the alphabets of α and β are disjoint. We lift the prefix relationship on words to the level of traces, by stating that a trace α is a prefix of a trace β if β = α ◦ γ for some trace γ. We denote this by α β. Moreover, if α β and α = β then we write α β. For


135

[λ] m1

[a] [am] [amr]

r1

[ag] [amg]

a1

g1 Fig. 5.4. Hasse diagram of the prefix ordering for the trace [amrg] and its dependence graph (labelling is obvious and therefore omitted). [amrg]

example, [a] [amr]. It then follows that u v and v ≡ w implies [u] [w] where u, v, and w are words . Consequently, u v implies [u] [v], but [u] [v] does not necessarily imply that u v holds . Unlike words which have a unique history that is captured through the prefix relation (see Figure 5.3), the prefix relation for traces can be interpreted as associating with a trace several histories which are sequential observations (represented by the words in the trace) of possibly concurrent behaviour. One could say that the concurrency between occurrences has been ‘flattened’ to choosing an order of occurrence. In Figure 5.4 the trace [amrg] is depicted with the ordering of its prefixes. That trace has three histories in correspondence with the three directed paths defining its elements amgr, agmr, and amrg. To extract information on the dependencies between the symbol occurrences in a trace, dependence graphs are used. In these graphs, the relationship between dependence and order is made explicit. Definition 13 : dependence graphs A dependence graph over a concurrency alphabet (Σ, Ind ) is a dag in which two nodes are connected iff they are labelled with dependent symbols. In other words, a dag G = (V, A, ) is a dependence graph over (Σ, Ind ) if : V → Σ is a labelling such that for all distinct nodes x, y ∈ V , there is an / Ind . arc (x, y) ∈ A ∪ A−1 if and only if ((x), (y)) ∈ Every dag defines a language, consisting of all words that can be read from its labels without violating the order implied by its arcs (e.g., by using a topological sorting procedure). Formally, the language of a dag G comprises all words associated with the total extensions of its transitive closure, i.e., df language(G) = word (lin(G+ )). Their languages provide a fairly precise characterisation of dependence graphs, as two dependence graphs over the same concurrency alphabet are isomorphic iff their languages are the same . Conversely, assuming a given concurrency alphabet (Σ, Ind ), with every word u ∈ Σ ∗ a dependence graph can be associated. The canonical dependence df graph of u is the dag candepgraph Ind (u) given by (occ(u), ≺, ) where (ai ) = a,

136


G = candepgraph Ind (agmau) g1

g1

u1

a1

a2

a1

a1

g1

u1

m

g1

u1 a2

a2 1

a1

u1 a2

m1

m1

m1

G

G+

hasse(G+ )

a1

g1

m1

a2

u1

= cantotalposet(agmau)

a1

g1

m1

u1

a2

= cantotalposet(agmua)

a1

g1

u1

m1

a2

= cantotalposet(aguma)

a1

m1

g1

a2

u1

= cantotalposet(amgau)

a1

m1

g1

u1

a2

= cantotalposet(amgua)

Fig. 5.5. The canonical dependence graph (twice) of agmau, its transitive closure, the Hasse diagram of its transitive closure, and Hasse diagrams of its total extensions. (Labelling of the nodes is obvious and therefore omitted.)

for all a ∈ alphabet(u) and ai in occ(u), and ai ≺ bj if the i-th occurrence of a (strictly) precedes the j-th occurrence of b within u and (a, b) ∈ / Ind . Note that the canonical dependence graph of the empty word is candepgraph Ind (λ) = (∅, ∅, ∅), and that candepgraph ∅ (u) = cantotalposet(u). Figure 5.5 shows an example of a canonical dependence graph, and its total extensions. >From the definition of u ∼Ind v, it follows that the canonical dependence graphs of two words are equal whenever they are trace equivalent . Hence the canonical dependence graph of a trace α can be defined as df candepgraph(α) = candepgraph(u) where u is any word in α. What is more, the language defined by the canonical dependence graph of a trace consists exactly of the words comprising that trace . It therefore follows that distinct traces have distinct canonical dependence graphs . Hence there is a one-to-one correspondence between dependence graphs and traces. On basis of the canonical dependence graph of a trace, we define the canondf ical poset of a trace α as canposet(α) = candepgraph(α)+ . >From the above observations, it follows that the canonical poset of a trace properly captures the behaviours represented by the words in that trace.


137

Fact 14 : Let α be a trace. • lin(canposet(α)) = cantotalposet(α).a • word (lin(canposet(α))) = α. a

Note that in cantotalposet(α) the trace α is treated as a set of words.

All information on the dependencies between the occurrences in a trace is represented in its uniquely associated poset. 5.3.1 Bibliographical Remarks Main independent sources of trace theory are [7] (in the context of combinatorial problems) and [35, 29] (in the context of concurrency theory). An extensive account of trace theory is provided by [11] which, in particular, contains a chapter on dependence graphs [20]. For a bibliography on traces see [15].

5.4 Elementary Net Systems In this section we first briefly discuss Petri nets as a system model, or rather as a framework for the modelling of concurrent systems. Then we introduce in more detail Elementary Net systems, the most basic Petri net model. In this model the key primitive notions underlying the operation of concurrent systems are explicitly represented and as such it has been the inspiration for the development of trace theory. In later sections, we will discuss more expressive net classes and how they lead to generalizations of traces. The description of a Petri net comes in two parts, giving its static and dynamic aspects. The (static) structure of a Petri net is a graph specifying the local states (called places) of the system being modelled and its possible actions (called transitions). Global (system) states consist of combinations of the local states and it is the role of transitions to change those states in accordance with the given (dynamic) rules. Each transition has a neighbourhood of places with which it is linked and there are specific rules when transitions can occur (concurrently) and the effect of such occurrence. Both notions are fully determined by the transition’s neighbourhood, i.e., every transition occurrence depends on neighbouring local states and also its effect when it occurs is completely local. A net system is fully specified when also an initial state is supplied from which possible behavioural scenarios are initiated. By varying the kind and nature of the relationships between places and transitions, as well as the precise notions of global state, and the enabling and occurrence rules, one obtains different classes of Petri nets. First we introduce the basic structure underlying every Petri net. The definition below captures what presumably is the most fundamental class of nets.

138


N

p1 r p2 m

r p5

p4 a

EN

p1

g

p3

p2 u

p6

p5

p4

m

g

a p3

u p6

Fig. 5.6. A net without and with configuration (an EN-system) for the running example with a producer and a consumer subnet connected by a (buffer) place p4.

Definition 15 : nets A net N is a relational tuple (P, T, F ) with P and T disjoint finite sets of nodes, called respectively places and transitions, and F ⊆ (T ×P )∪(P ×T ) the flow relation. In diagrams, places are drawn as circles, and transitions as rectangles. The flow relation is represented by directed arcs between them. Hence nets are drawn as bipartite graphs. Figure 5.6 shows the net N = (P, T, F), where P = {p1, p2, p3, p4, p5, p6} is the set of places, T = {a, g, m, r, u} is the set of transitions, and the flow relation F comprises the following twelve arcs: (r, p1) (a, p4)

(p2, r) (p4, g)

(p3, m) (p5, g)

(m, p2) (g, p6)

(p2, a) (p6, u)

(a, p3) (u, p5) .

Let (P, T, F ) be a net. The inputs and outputs of a node x ∈ P ∪ T are the sets • x and x• , respectively comprising all y such that yF x and xF y, and the neighbourhood • x• of x is the union of its inputs and outputs. The dot-notations readily extend to sets of nodes, e.g., • X comprises all inputs of the nodes in X. It is assumed here that each net is T-restricted which means that every transition has at least one input (cause) and at least one output (effect). For the net N in Figure 5.6, • g = {p4, p5} and p3• = {m}. 5.4.1 Configurations and Transition Occurrence df

In this and the next section, the states of a net N = (P, T, F ) are given by subsets of places representing the conditions that hold at a given global situation. Definition 16 : configurations A configuration of a net is a subset of its places.


139

In diagrams, a configuration C is represented by drawing in each place p in C a token (a small black dot). A possible configuration for the net in Figure 5.6 is C = {p2, p5}, as illustrated on the right of Figure 5.6. Transitions represent actions which may occur at a given configuration and then lead to a new configuration. Definition 17 : transition occurrences A transition t can occur (or is enabled) at a configuration C if • t ⊆ C and t• ∩ C = ∅. Its occurrence then leads to a new configuration (C \ • t) ∪ t• . Thus a (potential) occurrence of a transition depends only on its neighbours. If t can occur at C then we write C[t, and if its occurrence leads to C we write C[tC . Note that through such an occurrence, all inputs of t cease to hold, and all outputs start to hold. Hence the change caused by the occurrence of a transition is always the same and does not depend on the current global state. For the configuration C shown in Figure 5.6, the enabled transitions are r and a. Moreover, we have C[a{p3, p4, p5} and C[r{p1, p5}. Figure 5.7 provides further intuition about the enabling and occurrence rules for net transitions. We now lift the execution of transitions to a concurrent context by allowing the simultaneous occurrence of transitions provided that they do not interfere with one another, i.e., their neighbourhoods are mutually disjoint. Definition 18 : steps A step of a net is a subset of its transitions. A step can occur (or is enabled) at a configuration C if the neighbourhoods of its transitions do not overlap, and each transition is enabled. The effect of its occurrence is the cumulative effect of the occurrences of the transitions it comprises. In other words, a step U is enabled at C if • t• ∩ • t• = ∅ for all distinct transitions t and t in U , and C[t for each transition t in U . We denote this by C[U . The occurrence of an enabled step leads to a new configuration C given by (C \ • U ) ∪ U • , and we denote this by C[U C . Note that C[U C iff the step U is empty . For the configuration C shown in Figure 5.6, we have C[{a}C where C = {p3, p4, p5}; moreover, we further have: C [{m, g}{p2, p6}

C [{m}{p2, p4, p5}

C [{g}{p3, p6} .

We are now ready to introduce sequences of transitions and step occurrences. Definition 19 : step sequences A step sequence of a net is a finite sequence of non-empty steps occurring one after another from a given configuration.

140


[t t is not enabled

t is not enabled

t is enabled

t has occurred

Fig. 5.7. Local change-of-state produced by the occurrence of a transition.

In other words, a step sequence from a configuration C to a configuration C is a possibly empty sequence σ = U1 . . . Un of non-empty steps Ui such that C[U1 C1 , . . . , Cn−1 [Un C , for some configurations C1 , . . . , Cn−1 . We also write C[σC or C[σ, and say that C is a configuration reachable from C. The set of all configurations reachable from C will be denoted by [C. Note that we always have C ∈ [C. If n = 0, thus σ = λ the empty (step) sequence, then C = C . The converse implication however does not hold . For the configuration C shown in Figure 5.6, C[{a}{m, g}{u, r}{p1, p5}, and, as we will see later on, the set [C comprises twelve reachable configurations. To improve the readability of the notations when discussing examples, we will often drop the curly brackets when writing a singleton step, e.g., we can write a{m, g}ur instead of {a}{m, g}{u}{r}. A special kind of step sequences are those that consist of singleton steps only. Such sequences (of transitions) are referred to as firing sequences. For example, amgur is a firing sequence from C to {p1, p5}. Reachability of configurations does not depend on whether one uses step sequences or firing sequences. If, however, the structure of a net is enriched with inhibitor arcs as we will do it in the next section, then reachability may be affected by the restriction to firing sequences. 5.4.2 Concurrency and Causality The definition of concurrent behaviour on basis of non-interference, as introduced above, allows one to investigate some intricate relationships in the way transitions can occur. As a first observation we have that transitions which can be executed simultaneously (at some configuration) do not have to occur together. They can still occur one after another. Moreover, whenever transitions can occur in any order, they must be concurrently enabled and non-interfering. Fact 20 : Let C, C be configurations and U, U be steps of a net. • C[U ∪ U C and U ∩ U = ∅ implies C[U U C . • C[U U C and C[U implies U ∩ U = ∅ and C[U ∪ U C . This fact is often referred to as a ‘diamond property’. The reason is that if we have, say, C[{a, b}C , it then follows that we also have C[{a}C [{b}C


141

and C[{b}C [{a}C where C and C are distinct configurations . In drawing this yields a diamond shape. Note that the two statements together show that for the dynamics of nets defined sofar, diamonds imply concurrency and vice versa. For the configurations C = {p3, p4, p5} and C = {p2, p6} of the net shown in Figure 5.6, we have C [{m, g}C as well as C [mgC and C [gmC , and the resulting ‘diamond’ can be seen with a little bit of effort at the centre of the upper state graph in Figure 5.8. The first part of Fact 20 implies that every step of simultaneously occurring transitions can be split into any partition of subsets occurring in sequence, with the same effect as the original step. As a consequence, every step sequence eventually gives rise to a valid (but not necessarily unique) firing sequence. And so the configurations reachable from a given one are the same for step sequences and firing sequences. Fundamental relationships between transitions can be classified in a way which reflects their causal dependence (occurrence of one enables the other), competition for shared resources (both can occur, but they cannot occur together), or concurrency (they can occur together). Definition 21 : fundamental situations - behavioural Let t and t be distinct transitions, and C be a configuration of a net. • t causally depends on t at C if ¬C[t and C[t t. • t and t are in conflict at C if C[t, C[t and ¬C[{t, t }. • t and t are concurrent at C if C[{t, t }. For the configuration C shown in Figure 5.6, we have that g causally depends on a, and the latter is in conflict with r. Moreover, m and g are concurrent at the configuration C = {p3, p4, p5}. It is interesting to note the difference between conflict and concurrency in terms of firing sequences: in case of conflict at a configuration, both are enabled to occur, but the occurrence of one disables the other, whereas in case of concurrency, the two transitions can occur in either order. Fact 22 : Let t and t be transitions, and C be a configuration of a net. • If t causally depends on t at C then ¬C[tt and C[t t. • If t and t are in conflict at C then ¬C[tt and ¬C[t t. • If t and t are concurrent at C then C[tt and C[t t. These fundamental relationships between transitions are defined dynamically by referring to a global state. However, if two transitions are in one of these three relationships at some configuration, then none of the other relationships will ever hold for them (at whatever configuration) . In fact, the (potential) relationships between transitions are determined by the graph structure.

142


Definition 23 : fundamental situations - structural Let t and t be two distinct transitions of a net N . • • • •

t t t t

and and and and

t t t t

are structurally causally related if • t ∩ t• = ∅ or t• ∩ • t = ∅. are in structural backward conflict if • t ∩ • t = ∅. are in structural forward conflict if t• ∩ t• = ∅. are structurally independent if • t• ∩ • t• = ∅.

For the net shown in Figure 5.6, a and g are structurally causally related, a and r are in structural forward conflict, and r and u are structurally independent. 5.4.3 EN-Systems and Their State Spaces Having defined nets with states and dynamics, it is now time to study them as systems which start their operation from an initial state. Definition 24 : EN-systems An elementary net system (or EN-system) consists of an underlying net and an initial configuration. Its state space consists of all configurations reachable from the initial configuration. In other words, an elementary net system EN is a relational tuple (P, T, F, Cinit ) such that the first three components form its underlying net and Cinit ⊆ P is the initial configuration. Figure 5.6 shows on the right an EN-system EN = (P, T, F, Cinit ), where Cinit = C = {p2, p5}, modelling our running example. Its state space consists of twelve configurations: [Cinit = {{pi, pj} | i = 1, 2, 3 ∧ j = 5, 6} ∪ {{pi, p4, pj} | i = 1, 2, 3 ∧ j = 5, 6} .

The state graph of EN is a relational tuple stategr (EN ) = ([Cinit , LA, Cinit ) df with node set [Cinit , set of labelled arcs LA = {(C, U, C ) | C ∈ [Cinit ∧ C[U C }, and initial node Cinit . Restricting the arcs of the state graph to those labelled by singletons steps yields the sequential state graph of EN , denoted by seqstategr (EN ). Figure 5.8 gives examples of each kind of state graph for the EN-system ENsimple in Figure 5.9. Since every configuration reachable from the initial configuration by a step sequence is also reachable by a firing sequence, all nodes in seqstategr (EN ) are reachable from the initial node. Interestingly, also stategr (EN ) can be recovered from the sequential state graph seqstategr (EN ) by saturating the latter with non-singleton step labelled edges using the diamond property (Fact 20) . To illustrate the above idea, let us consider the state graphs in Figure 5.8, and two nodes, C = {p3, p4, p6} and C = {p2, p4, p5}. Looking at the sequential state graph, we can deduce that C[{m}{u}C and C[{u}. Hence, by the df


143

init : {p2, p5} {m} {a}

{p3, p5}

{m, u} {p3, p4, p5} {m}

{u}

{g} {a, u} {m, g}

{p2, p4, p5}

{p3, p6}

{g} {u}

{u}

{m}

{u}

{p2, p6} {m, u} {a}

{p2, p4, p6}

stategr (ENsimple )

{m} {p3, p4, p6}

init m a u

u

m

g

g

m

u

u a

seqstategr (ENsimple )

m

Fig. 5.8. The state graph of ENsimple from Figure 5.9 and its sequential state graph.

144


second part of Fact 20, we have C[{m, u}C and so the concurrent step {m, u} from C to C in the state graph has been deduced from purely sequential information. For a behavioural comparison of EN-systems, isomorphism is too discriminating, because then there would be essentially only one structure defining the behaviour under consideration. Therefore, in EN-system theory it is the state graph which provides the main reference point for any behaviour related analysis. However, all information on (the relevant, active, part of) the net underlying the EN-system can still be recovered from its state graph; the places belonging to reachable configurations, transitions which actually occur and thus appear in the steps labelling the arcs, and their neighbourhood relations, are all explicitly represented in the state graph. Using the state graph itself would thus lead to a similar identification of net structure and behaviour. To abstract from the concrete information on places and transitions, state graph isomorphism is used as an equivalence notion for the comparison of concurrent behaviours. Already the structure of its state graph provides a complete and faithful representation of the behaviour of an EN-system. In particular, causality, conflict, and concurrency among (possibly renamed) transitions can be determined from it. Note that two EN-systems have isomorphic state graphs iff also their sequential state graphs are isomorphic . After isomorphism of EN-systems, state graph isomorphism is the second strongest notion of equivalence employed in the behavioural analysis of EN-systems. With this equivalence it is possible to transform EN-systems in order to realise a desired property or feature (a normal form) without affecting their dynamic properties in an essential way, i.e., the state graph remains the same up to isomorphism and the resulting system is considered behaviourally equivalent. An important application of this idea is the following. The enabling relation for transitions checks explicitly for the emptiness of their output places. This may be regarded as somewhat unsatisfactory. It would be more efficient and intuitively more appealing if it would be sufficient to check only whether all input conditions are fulfilled. Definition 25 : contact-freeness An EN-system is contact-free if for every reachable configuration C and every transition t, it is the case that • t ⊆ C implies t• ∩ C = ∅. In other words, a contact-free system is one where the test for transition enabledness can simply be • t ⊆ C without changing anything. The EN-system shown in Figure 5.6 is not contact-free . Not all EN-systems are contactfree, but the simple transformation described next turns any EN-system into a behaviourally equivalent contact-free version. Two places, p and q, are complements of one another if • p = q • , p• = • q and exactly one of them belongs to the initial configuration Cinit . The of EN is obtained by adding, for each place p without a complementation EN


p2 m

p5 g

a p3

p4

p2 u

p6

m

p4

p3

ENsimple

p5 g

a

145

4 p

u p6

EN simple

Fig. 5.9. A simplified version of the EN-system from Figure 5.6 and its complementation.

complement, a fresh complement place p; moreover, if the initial configuration does not contain p then p is added there as well. The result is clearly an ENsystem and the two systems have isomorphic state spaces. In fact, only the reachable configurations have to be renamed in the case that new complement places have been added; the arc labels between corresponding states however are the same . is contact-free and its state space is isomorphic to that Fact 26 : EN of EN . The construction is illustrated by the non-contact-free EN-system ENsimple in Figure 5.9 and its contact-free complementation EN simple . The state spaces of the two EN-systems are respectively: Conf ∪ {{pi, pj} | i = 2, 3 ∧ j = 5, 6} Conf ∪ {{pi, p 4, pj} | i = 2, 3 ∧ j = 5, 6}

(left) (right)

where Conf = {{pi, p4, pj} | i = 2, 3 ∧ j = 5, 6}. It is can be seen that a suitable isomorphism for their state graphs maps each {pi, pj} to {pi, p 4, pj}, and is the identity for the configurations in Conf. Fact 26 assumes that one adds complements for all non-complemented places. But it is also possible to add complementation selectively and, in general, we have that any EN-system with an arbitrary, added set of new complement places has a state space which is isomorphic to that of the original EN-system . For the EN-system EN modelling the running example we can add a complement of the buffer place which results in the equivalent EN-system shown in Figure 5.10. In this case already the selective complementation yields a contact-free EN-system . Since it is always possible to ensure contact-freeness without changing the behaviour represented in the state-graph, we now make a simplifying assumption. In the rest of this tutorial all EN-systems are contact-free.

146

Jetty Kleijn and Maciej Koutny p1

ENcfree

r p2 m

p4

p3

p5 g

a

u

p7

p6

Fig. 5.10. A contact-free version of the EN-system from Figure 5.6 where the place p4 has been complemented, i.e., p7 = p 4.

5.4.4 Behaviour of EN-Systems

Let EN = (P, T, F, Cinit ) be a fixed EN-system for the rest of this section. In addition to the state graph, we can also associate firing sequences and step sequences as behavioural notions to EN-systems. The set of all firing sequences firseq(EN ) of EN consists of those sequences u ∈ T ∗ such that Cinit [u and, similarly, the set of all step sequences stepseq(EN ) of EN comprises all step sequences of EN from Cinit . Each firing sequence corresponds to a finite labelled path through the sequential state graph from the initial node. Since the set of reachable configurations of an EN-system is finite, the sequential state graph is a finite state machine. Hence the set of firing sequences of an EN-system is a prefix-closed regular language. However, it consists of purely sequential observations of the EN-system’s behaviour without any reference to the possible independence of transitions. Yet such causality information is often of high importance for system analysis and design. Let us first demonstrate how the theory of traces can be applied to extract partial orders from firing sequences as representations of the necessary causal ordering of transition occurrences within these sequences. Definition 27 : concurrency alphabets of EN-systems df

The concurrency alphabet of EN is CAEN = (T, Ind EN ) where the structural independence relation Ind EN comprises all pairs of distinct transitions with disjoint neighbourhoods. Defined in this way, Ind EN = {(t, t ) | t, t ∈ T ∧ • t• ∩ • t• = ∅} is a symmetric and irreflexive relation and so it is indeed an independence relation. For the EN-system ENcfree in Figure 5.10, IndENcfree = Ind where Ind was defined at the beginning of Section 5.3. An important observation is now that in a firing sequence adjacent occurrences of independent transitions could have occurred


147

also in the other order (see the diamond property, Fact 20). Hence, for every firing sequence of EN , all its trace equivalent words from T ∗ are also firing sequences of EN . Fact 28 : firseq(EN ) =

u∈firseq(EN )

[u].

Taking, for example, ENcfree in Figure 5.10, we have agm ∈ firseq(ENcfree ) and [agm] = {agm, amg}. Clearly, amg is also a firing sequence of ENcfree . The step sequences of an EN-system obviously provide important insights into concurrency aspects of its behaviour. They are nevertheless still sequential rather than concurrent in nature in the sense that the sequential ordering of the steps obscures the true causal dependencies between the occurrences of transitions. Petri net models can however easily support a formal approach where this information is readily available by unfolding behaviours into structures allowing an explicit representation of causality and concurrency. 5.4.5 Non-Sequential Observations Rather than describing the behaviour of the system in terms of sequential observations, like firing sequences and step sequences, we now present a semantics based on a class of acyclic Petri nets, called occurrence nets. What one essentially tries to achieve here is to record the changes of configurations due to transitions being executed along some legal behaviour of the EN-system, and in doing so record which places were emptied (served as inputs) and which filled (as outputs). The resulting occurrence nets may be viewed as partial net unfoldings, with each transition representing an occurrence of a transition in the original net (thus occurrence nets are acyclic), and each place corresponding to the occurrence of a token on a place of the original net. Conflicts between transitions are resolved and thus the places in an occurrence net do not branch. Definition 29 : occurrence nets df

An occurrence net is a relational tuple ON = (B, E, R, ) such that (B, E, R) is an underlying net,a is a labelling for B ∪ E, R is an acyclic flow relation, and |• b| ≤ 1 and |b• | ≤ 1, for every b ∈ B. a

The dot-notations, configurations, firing rule, etc, for ON are as those defined for the underlying net.

The places of an occurrence net are usually called conditions (‘Bedingungen’ in German) and its transitions events (‘Ereignisse’ in German). The default initial configuration of ON consists of all conditions without incoming arcs, ON comprises all conditions b ∈ B such that • b = ∅, and the default i.e., Cinit

148


e1 a

b1 p7 p4

b5

e2 m

p2

b2

p7

b3

p6

b7 u e5

p5

g e4 p2

b11 p3

a

b4 p7 b8

b9 p6

g e6

b12 p2

b13

p3

m

p7

p3

e3 a p4

b6

p5 b10

p3

a

p7

p7

p4

p4

p5

p6 g

p5 u

p6 g

Fig. 5.11. An occurrence net ON with nodes labelled by places and transitions of the EN-system ENcfree in Figure 5.10 (top), and the same occurrence net with identities of the nodes omitted (bottom). ON final configuration Cfin consists of all conditions without outgoing arcs. The ON = default initial configuration of the occurrence net in Figure 5.11 is Cinit ON = {b4 , b9 , b13 }. {b1 , b5 , b10 } and the default final configuration is Cfin The sets of firing and step sequences of ON are defined w.r.t. the default initial configuration. However, since an occurrence net is meant to represent a record of a concurrent run of an EN-system, what really counts is not the identities of its events, but their labels which are linked to the occurrences of transitions in the EN-system. The language of ON is the set language(ON ) of all sequences (u) such that u is a firing sequence from the default initial configuration of ON to the default final configuration. By abstracting from the conditions we associate with the occurrence net ON = (B, E, R, ) a directed acyclic graph with E as its set of nodes. This df dag dag(ON ) = (E, R ◦ R|E×E , |E ) represents the direct causal relationships between the events. Its transitive closure dag(ON )+ , see Figure 5.12, then gives all, direct and indirect, causal dependencies. For example, e4 directly causes e5 , but there is only an indirect causal link from e4 to e6 . ON with its default initial configuration is basically a contact-free ENsystem . Interestingly, all the sets occurring in any step sequence σ from the initial configuration to another configuration C, are mutually disjoint . Moreover, C is the default final configuration iff the steps in σ use all the events of the occurrence net . A slice of ON is a maximal (w.r.t. set inclusion) subset S of events from ON which are causally unrelated, i.e., (S ×S)∩R+ = ∅. The set of all slices of ON is denoted by slices(ON ). Clearly, both default configurations are slices ON = slices(ON ), i.e., slices are exactly those configuand, in general, [Cinit rations which are reachable from the initial configuration . Moreover, the

5 Formal Languages and Concurrent Behaviours e2 e1

m

a

g

a

dag(ON)

m

e3

e4

g

u

e5

e6

a

a

dag(ON)+

149

g g

u

Fig. 5.12. Direct causality among the events in the occurrence net in Figure 5.11, and full causality (node identities omitted).

final configuration of ON is always reachable from any configuration reachable from the initial one . Essentially, this means that ON is deadlock-free until its final configuration has been reached. The processes of an EN-system are occurrence nets reflecting its structure and possible behaviour through their labelling and initial configuration. Definition 30 : processes of EN-systems A process of EN is an occurrence net ON = (B, E, R, ) such that: • labels conditions with places and events with transitions. • is injective on the default initial configuration of ON , as well as on the sets of input and output conditions of each event. ON ) = Cinit and, for every e ∈ E, (• e) = • (e) and (e• ) = (e)• . • (Cinit The occurrence net ON in Figure 5.11 is a process of the EN-system in Figure 5.10. Processes can be used to investigate the behaviours of EN-systems. Due to the second and third conditions in Definition 30, we can relate the firing sequences, step sequences and configurations of EN to their labelled versions in ON [σC then Cinit [(σ)(C) ON . More precisely, if we take a step sequence Cinit holds. This can be proved by an inductive argument from which it also follows that labelling of ON is injective on all its slices and hence also on the sets occurring in any step sequence of ON . If σ is a step sequence from the default initial configuration of ON , then (σ) is referred to as a labelled step sequence of ON . Similar to the language of ON , the step language of ON is defined as the set steplanguage(ON ) of all sequences (σ) such that σ is a step sequence from the default initial configuration of ON to the default final configuration. In general, it follows that all firing and step sequences of EN-systems can be derived from their processes.

150


Fact 31 : Let ON be the set of all processes of EN . • firseq(EN ) = ON ∈ON language(ON ). • stepseq(EN ) = ON ∈ON steplanguage(ON ). Definition 30 does not provide any clues as to how to derive a process of an EN-system. This is rectified in the next definition which shows how to construct a process corresponding to a given step sequence. Definition 32 : processes construction The occurrence net ON σ generated by a step sequence σ = U1 . . . Un of EN is the last element in the sequence N0 , . . . , Nn where each Nk is an occurrence net (Bk , Ek , Rk , k ) constructed thus. df df Step 0: B0 = {p1 | p ∈ Cinit } and E0 = R0 = ∅. Step k: Given Nk−1 we extend the sets of nodes and arcs as follows: Bk = Bk−1 ∪ {p1+p | p ∈ Uk• } df Ek = Ek−1 ∪ {t1+t | t ∈ Uk } df Rk = Rk−1 ∪ {(pp , t1+t ) | t ∈ Uk ∧ p ∈ • t} ∪ {(t1+t , p1+p ) | t ∈ Uk ∧ p ∈ t• } . df

In the above, the label of each node xi is set to be x, and #x denotes the number of nodes of Nk−1 labelled by x. The construction is illustrated in Figure 5.13 for the ENcfree in Figure 5.10 and its step sequence σ = a{m, g}{a, u}g. The resulting occurrence net is isomorphic to the occurrence net ON in Figure 5.11 which is a process of ENcfree . Fact 33 : Each occurrence net constructed as in Definition 32 is a process of EN and, for each process of EN , there is a run of the construction from Definition 32 generating an isomorphic occurrence net. Thus the operationally defined processes and the axiomatically defined processes of an EN-system are essentially the same. Finally, we return to the trace semantics of EN-systems in relation to processes. First note that each trace gives rise to only one process, since interchanging adjacent occurrences of independent transitions has no effect on the construction of a process. So, ON u = ON w whenever u and w are trace equivalent firing sequences . Hence ON [u] the process associated to a trace is a well-defined notion. Conversely, the language of a process is identical to its defining trace . Thus we have a one-to-one correspondence between traces and the processes of an EN-system. Moreover, even though the dag

5 Formal Languages and Concurrent Behaviours p21

p31

m1

p22

a1 1

p7

p32 a2

p4

1

2

p42

p7 1

g p61

p3

m1

u1

p2

1

p52

p62

p3 2

a p7

p73 2

g p51

p2

151

a p4

p7

p4

g1 p5

p7 g2

p6

u1

p5

p6

Fig. 5.13. The occurrence net ONa{m,g}{a,u}g generated for the EN-system in Figure 5.10: node-oriented view (top), and label-oriented view (bottom).

defined by a process is not necessarily isomorphic to the dependence graph of its trace , they always define the same partial order on their transition occurrences. Fact 34 : Let u be a firing sequence of EN . • [u] = language(ON u ). • canposet([u]) = dag(ON u )+ . To conclude, the trace semantics and the process semantics of EN-systems lead to one partial order semantics by providing for each EN-system the same (isomorphic) partial orders modelling the causalities in its concurrent executions. This provides a strong argument in favour of the view that both these approaches capture the essence of causality in the behaviours of EN-systems. 5.4.6 Bibliographical Remarks Over the past 40 or so years different classes of Petri nets have been introduced by varying the kind of underlying net, notion of local state, or transition relation. An early systematic treatment of basic notions in net theory and ENsystems can be found in [48]. Other extensions of the EN-systems approach adopt notions like priorities, real-time behaviour, or object-orientation. (In fact, we consider two such extensions later in this tutorial.) The general question of the intrinsic or common properties of nets is discussed in [8]. The problem of associating non-sequential semantics with Petri nets is dealt with, in particular, in [35, 40, 36, 37, 41, 42, 19, 47]. There is a systematic way of dealing with process semantics of various classes of Petri nets proposed in [31] which makes it possible to separately discuss behaviour, processes, causality,

152


p2 m

p4

p3

p5 g

a p7

ENI

ENI

r

p2 u

p6

p1 r

m

p4

p3

p5 g

a p7

u p6

Fig. 5.14. Two ENI-systems modelling two variations of the running example.

and their mutually consistency. General Petri net related resources can be found in the web pages at [26].

5.5 Adding Inhibitor Arcs This section extends the treatment of concurrency considered so far in ENsystems in order to accommodate the practically relevant case of nets with inhibitor arcs. In particular, we will demonstrate how the original definition of traces may be extended to describe in an adequate way also the additional features of the resulting new kind of concurrent behaviours. To see why inhibitor arcs can be a convenient modelling device, let us imagine that a designer would like to modify the running example so that the producer cannot retire if the customer is waiting for an item. Such a modification is easily achieved by taking the EN-system of Figure 5.10 and adding to it an inhibitor arc linking the place p5 and transition r. This yields the net system ENI shown on the left of Figure 5.14. (Inhibitor arcs are drawn with small open circles as arrowheads.) Adding this arc means that r cannot be enabled if p5 contains a token, and so the producer indeed cannot retire if the consumer is waiting for an item. Elementary net systems with inhibitor arcs, or simply ENI-systems, thus extend EN-systems. The usefulness of inhibitor arcs stems from their ability to detect a lack rather than the presence of specific resources, i.e., tokens in specific places. That such an addition to the EN-system syntax is a true extension of their modelling power follows from the observation that there is no EN-system with exactly the same set of firing sequences as ENI. This can be shown by considering two firing sequences of ENI, amgru and amgu. If there was an EN-system generating the same firing sequences as ENI, then, due to the second statement in Fact 20, it would also have to generate the firing sequence amgur. But such a firing sequence is not generated by ENI as executing the last transition would contradict the defining characteristic of the inhibitor arc between r and p5. We will return to this example after introducing ENI-systems more formally.


153

Definition 35 : ENI-systems An elementary net system with inhibitor arcs (or ENI-system) is a reladf tional tuple ENI = (P, T, F, Cinit , Inh) such that the first four components form an (underlying) EN-system and Inh ⊆ P ×T is a set of inhibitor arcs.

As inhibitor arcs are introduced on top of the model of EN-systems, wherever it is possible notions and notations concerning the structure and configurations of an ENI-system are inherited from its underlying EN-system. Thus, for example, the initial configuration of the ENI-systems in Figure 5.14 is the initial configuration of the EN-system in Figure 5.10. The only new notation is ◦ t denoting the set of all the places p where the presence of a token inhibits the enabling of a transition t, i.e., (p, t) ∈ Inh. For example, we have ◦ r = {p5} and ◦ a = ◦ m = ◦ g = ◦ u = ∅ in the case of ENI. The dynamic aspects of an ENI-system are also derived from the underlying EN-system, with proper attention being paid to the inhibiting features of the new kind of arcs. In fact, all one needs to re-define is the enabling condition for steps, by stating that a step of transitions U of an ENI-system is enabled at a configuration C if it is enabled at C in the underlying EN-system and, in addition, no place in ◦ U belongs to C, where ◦ U consists of all places connected by inhibitor arcs to transitions in U . It is important here to stress that the change of state effected by an executed step is exactly the same as in the underlying EN-system; in other words, inhibitor arcs have only impact on the enabling of steps. In the case of ENI, a is a singleton step enabled in the initial configuration, but the other singleton step r enabled in the initial configuration of the underlying EN-system is not since p5 ∈ ◦ r ∩ Cinit . Note that it would be quite natural and harmless as it has no effect on the dynamics of an ENI-system to additionally assume that for each of its transitions t, the sets • t, t• and ◦ t are mutually disjoint . As far as executing step sequences of ENI are concerned, we have Cinit [a{p3, p4, p5} and Cinit [a{m, g}{p2, p6, p7}. Having introduced the step sequence semantics of ENI-systems, we have df another look at ENI. This ENI-system generates the step sequence σexmpl = df a{m, g}{u, r} Splitting the last step into ur leads to the sequence σinvalid = a{m, g}ur which is not a valid behaviour of ENI (yet the splitting of {u, r} into ru leads to a valid step sequence). Thus, also the first part of Fact 20 does not, in general, hold for ENI-systems. In fact, not only the diamond property no longer holds for ENI-systems, but even the property that every step sequence of an EN-system can be linearised to yield some valid firing sequence is not true for ENI-systems. Consider, for example, the ENI-system ENI on the right of Figure 5.14 which has been obtained from ENI by adding an inhibitor arc between p1 and u ensuring that the consumer can only use an item if there is still a chance that the producer may produce another item in the future. It is easy to see that ENI generates the step sequence amg{u, r}, but any attempt to linearise its only non-singleton step, ur, results in an illegal behaviour.

154


Note that ENI does not even have a firing sequence leading to a configuration including both p1 and p5. Hence the reachability of configurations is affected by the restriction to firing sequences, and so ordinary words are insufficient to capture all potential behaviours of ENI-systems. Consequently, the generalisation of trace theory we will present next will be based on step sequences rather than on words. 5.5.1 Comtraces We will now show how the notions of independence and causality developed for EN-systems can be lifted to the level of ENI-systems. The concurrency model used for EN-systems is not directly applicable to nets with inhibitor arcs; in particular, the current notion of transition independence needs to be replaced by a device which can be used to disallow some linearisations of executed steps. We therefore start by modifying the notion of concurrency alphabet. Definition 36 : combined concurrency alphabets df

A combined concurrency alphabet is a triple CCA = (Σ, sim, ser ) where Σ is an alphabet and ser ⊆ sim are binary relations over Σ called respectively simultaneity and serialisability. It is assumed that sim is irreflexive and symmetric. The two relations in a combined concurrency alphabet serve two distinct purposes, one of which is to define valid steps and the other is to define valid ways of splitting such steps. More precisely, if (a, b) ∈ sim then a and b may occur together in a step, while (a, b) ∈ ser means that a and b may occur in a step {a, b} and, in addition, such a step can be split into the sequence {a}{b}. Although it may not be immediately clear, combined concurrency alphabets subsume concurrency alphabets used earlier on. More precisely, the independence relation Ind of concurrency alphabets used in the definition of traces represents the situation that simultaneity and serialisability coincide with concurrency, i.e., sim = ser = Ind (note that this implies ser = ser −1 and that the two relations define diamonds). Intuitively, this means that simultaneity of symbols implies that they are totally independent and executing one has no impact on the subsequent executability of the other. In the examples, the combined concurrency alphabet CCA corresponds to the running ENI-system example, i.e., ENI in Figure 5.14. Thus CCA = (Σ, sim, ser), where Σ is as before, sim = {(r, u), (u, r), (g, m), (m, g), (u, m), (m, u), (u, a), (a, u)} and ser = sim \ {(u, r)}. As we will later see, these relations can be derived from the structure of ENI. Note that the problem with the step sequence σinvalid above is addressed by excluding (u, r) from ser. Next, the notion of a step is extended so that it does not necessarily depend on a net, but may also be defined relative to a combined concurrency


155

alphabet CCA = (Σ, sim, ser ) by stating that any non-empty set U ⊆ Σ is a step (over CCA) if (a, b) ∈ sim for all distinct a and b in U . (If sim and ser are not relevant we silently assume that sim = (Σ × Σ) \ id Σ and ser = sim; in that case every finite sequence of non-empty subsets of Σ is a step sequence over Σ.) We then lift in the obvious way to step sequences the following notions previously defined for words: concatenation, the set of symbol occurrences, and what it means for the i-th occurrence of a to precede the j-th occurrence of b within a step sequence. Note that neither the i-th occurrence of a precedes the j-th occurrence of b, nor the j-th occurrence of b precedes the i-th occurrence of a, if the two occurrences belong to the same step. Taking as an example the combined concurrency alphabet CCA defined above, we have that a{m, g}am{u, r} is a step sequence where m1 precedes a2 and the latter symbol occurrence precedes m2 . We can now introduce comtraces generalising traces and based on a combined concurrency alphabet. When defining the trace equivalence relation, we were able to swap any pair of neighbouring occurrences of independent symbols, e.g., amu ≡ aum. A similar effect can be achieved using the extended concurrency alphabet, but with an additional intermediate phase where the symbols being swapped are put together into a single step which is then linearised. Thus the elementary transformation needed to define comtraces is step splitting and combining rather than symbol swapping. To this end, we introduce which is a relation comprising all pairs (σ, ρ) of step sequences such that σ = τ U χ and ρ = τ U U χ where τ, χ are possibly empty step sequences, and U , U form a partition of U such that U × U ⊆ ser . Then we define the comtrace equivalence to be the reflexive symmetric transitive closure of . Coming back to the trace equivalence amu ≡ aum and assuming the combined concurrency alphabet of the running example, we have amu aum which follows from amu −1 a{m, u} aum. Another example of comtrace equivalence is a{m, g}{u, r} a{m, g}ru, but we also have that a{m, g}{u, r} a{m, g}ur since (u, r) ∈ ser. In fact, a{m, g}{u, r} and a{m, g}ur are not comtrace equivalent step sequences. Definition 37 : comtraces A comtrace over a combined concurrency alphabet is any equivalence class of its comtrace equivalence relation. A comtrace containing a given step sequence σ will be denoted σ. Note that λ = {λ} is the empty comtrace. The comtrace comprising the step sequence σexmpl is made up of the following six step sequences: a{m, g}{u, r}, a{m, g}ru, amgru, agmru, amg{u, r} and agm{u, r}. Comtraces enjoy a number of the key properties satisfied by traces, and so certain notions introduced for the latter can be re-defined for comtraces.

156


To start with, σρ and σ ρ are comtrace equivalent whenever σ = σ and df ρ = ρ . Hence comtrace concatenation σ $ ρ = σρ is a welldefined operation, and we can recover the monoidal structure of traces. Fact 38 : The set of all comtraces over a combined concurrency alphabet with comtrace concatenation and the empty comtrace forms a monoid. Finally, similarly as it was done for traces, comtraces can be equipped with a prefix relation which reflects their possible histories . 5.5.2 Stratified Posets and Comdags Traces have posets as their underlying dependency structures. For comtraces however, we will need to provide another notion of causal dependence. We start by providing a characterization of step sequences as a rather specific kind of posets, similar to the way that total posets correspond to words. Definition 39 : stratified posets A poset is stratified if being an unordered pair of elements is a transitive relation, and all elements labelled with the same label are linearly ordered. In other words, a poset spo = (X, ≺, ) is stratified if its elements can be partitioned into non-empty sets X1 , . . . , Xk such that is injective on each of them and the precedence relation is equal to the union of sets Xi × Xj , for all i < j. This further implies that the unorderedness relation spo is equal to the union of sets Xi × Xi \ id Xi , for all i, and that spo ∪ id X is an equivalence relation . Since the partitioning of X into these Xi ’s is unique, one can df associate with spo the step sequence steps(spo) = (X1 ) . . . (Xn ), and thus it is possible to view a stratified poset as a step sequence. The converse move is also possible, and the definition resembles that of the canonical total poset of a word. The canonical stratified poset canstratposet(σ) of a step sequence σ is defined as (occ(σ), ≺, ) where ai ≺ bj if the i-th occurrence of a precedes the jdf th occurrence of b within σ, and (ai ) = a, for all symbol occurrences ai and bj in occ(u). Figure 5.15 shows the canonical stratified poset of the step sequence σexmpl . One can immediately note that steps(canstratposet(σexmpl )) = σexmpl and since this is a general property holding for any σ, step sequences can be identified with the corresponding stratified order . We will also need structures generalising dependence graphs. Recall that these dags result from total posets (words) by deleting some of the precedence relationships between elements. Similarly, the new structures can be interpreted as stratified posets from which certain relationships have been deleted, while taking into account that the simultaneity within the steps also defines a weak (mutual) dependency between elements that can be deleted

5 Formal Languages and Concurrent Behaviours m

r

m

m1

r1

m1

a

r

157 r1

a

1

a1

a

g

u

1

1

g

u g g1 u u1 Fig. 5.15. Hasse diagram of the canonical stratified poset for σexmpl , and a comdag.

cdag

(in either direction). The main idea here is that standard causal precedence captures the ‘happened before’ relationship and a new weak causality relation stands for ‘happened before or simultaneously’. Definition 40 : comdags A labelled directed acyclic combined graph (or simply comdag) is a redf lational tuple comdag = (X, ≺, 0 ⇐⇒ δ ((a, 0, λ), (s, i, g1 g2 · · · gk )) = (a, 0, λ) δ(s, g1 , λ) = (s , λ) ⇐⇒ δ ((s, 0, g1 g2 · · · gk ), #) = (s , 0, g2 g3 · · · gk λ) and δ ((s, 0, g1 g2 · · · gk ), ($, 0, gk+1 gk+2 · · · g2k )) = (s , 0, g2 g3 · · · gk gk+1 )

s

ai ai+1 · · · an

an

· · · ai+1 ai

g1 .. . gm ⊥

s g1 .. .

$

$ $ g mk +1 .. . gm . ··· ..

#

gk

δ(s, g1 , λ) = (s+ , λ) s+ g2 .. . gm ⊥

ai ai+1 · · · an

an

· · · ai+1 ai

s+ g2 .. .

$

$ $ g mk +2 .. . gm . ··· ..

#

gk+1

Fig. 6.30. A λ-transition of the pushdown automaton M (left) and the corresponding transition of the OCA M (right). Counters are not depicted.

6 Cellular Automata – A Computational Point of View

221

The OCA M takes at most (k + 1) · n time steps. It has to simulate n nonλ-transitions of M. This takes n time steps. During each of these transitions some p symbols are pushed onto the stack which cause k − p packing steps. In addition, there are at most p additional λ-transitions that erase the p symbols. So, every non-λ-transition causes at most k further steps. It follows that M obeys the time complexity (k + 1) · n. & ' Altogether we obtain the hierarchy depicted in Figure 6.31, where the only known proper inclusions are at the top and the lower end. L (CS)

RE

L (CA)

DCSL

L (OCA)

CFL

Llt (CA) Llt (OCA)

Lrt (CA) = LltR (OCA)

Lrt+log (OCA)

DCFL

METALIN Lrt (OCA)

LIN

REG

Fig. 6.31. Basic hierarchy of language families. A solid arrow indicates a proper inclusion, a dashed arrow an inclusion, and a double arrow an equality. Linear, metalinear, and deterministic context-free languages are denoted by LIN, METALIN, and DCFL. Regular, context-free, deterministic context-sensitive, and recursively enumerable languages are denoted by REG, CFL, DCSL, and RE.

6.5.5 Summary of Closure Properties and Decidability Problems Finally, this subsection is devoted to summarize closure properties of and decidability results for the language families in question. Closure properties The closure properties of L (CS) and L (CA) are those of the recursively enumerable and deterministic context-sensitive languages. In [8, 22] strong closure properties are derived for the family of OCA languages. It is shown that

222

Martin Kutrib

L (OCA) is an AFL, that is, an abstract family of languages (cf., e.g., [58]) which is in addition closed under reversal. The closure under reversal is of crucial importance. It is an open problem for Lrt (CA) and, equivalently, for Llt (OCA). Moreover, it is linked with the open closure property under concatenation for the same family. If the answer to the open reversal closure of Lrt (CA) is negative, we have to deal with two different language families. Since the properness of the inclusion Lrt (CA) ⊆ Llt (CA) is also open, the problem gains in importance. A negative answer of the former problem would imply a proper inclusion. A language L ∈ Lrt (CA) whose reversal does not belong to Lrt (CA) may serve as witness since Llt (CA) is closed under reversal. In fact, the following stronger relation is shown in [23]. Theorem 12. The family Lrt (CA) is closed under reversal if and only if Lrt (CA) and Llt (CA) are identical. The question whether or not the family Lrt (OCA) is closed under concatenation was open for a long time. It has been solved negatively in [64]. The question whether or not one of the families Lrt (CA) = LltR (OCA) or Llt (CA) is closed under concatenation is another famous open problem in this field. Nevertheless, it is shown in [23] that the closure of Lrt (CA) under reversal implies its closure under concatenation. Since in this case we obtain Lrt (CA) = Llt (CA), the family of linear-time CA languages were also closed under concatenation. The concatenation closure for unary real-time CA languages has been solved in the affirmative [23]. Table 6.1 summarizes some closure properties of the language families in question. Decidability problems It is well known that all nontrivial decidability problems for Turing machines are undecidable [55]. Moreover, many of them are not even semidecidable, for example, neither finiteness nor infiniteness. Now we turn to summarize undecidable properties of cellular automata. Most of the early results are shown in [59] by reductions of Post Correspondence Problems. In terms of trellis automata the undecidability of emptiness, equivalence, and universality is derived in [28]. Here we present improved results that show the nonsemidecidability of the properties. Almost all results in this section are proved in [43] by reductions of Turing machine problems. To this end, valid computations of Turing machines are considered. Roughly speaking, these are histories of accepting Turing machine computations which can be encoded in small grammars [20]. The generated languages are accepted by real-time OCAs. Theorem 13. For any language family that effectively contains Lrt (OCA) emptiness, universality, finiteness, infiniteness, equivalence, inclusion, context-freeness, and regularity are not semidecidable.


∪,∩ complementation, − reversal concatenation λ-free iteration concatenation REG REG concatenation marked concatenation marked λ-free iteration hom−1 deterministic gsm−1 gsm−1 inj. length-pres. hom λ-free hom λ-free gsm λ-free substitution hom

223

Lrt (OCA) Lrt (CA) Llt (CA) L (OCA) L (CA) L (CS) + + + + + + + + + + + − + ? + + + + − ? ? + + + − ? ? + + + + + ? + + + + ? ? + + + + + + + + + + + + + + + + + + + + + + + + + + + − ? ? + + + + + + + + + − ? ? + + + − ? ? + + + − ? ? + + + − − − − − +

Table 6.1. Summary of closure properties. Concatenation REG denotes the concatenation with regular languages at the right, REG concatenation at the left, hom denotes homomorphisms, gsm generalized sequential machine mappings, and inj. length-pres. abbreviates injective length-preserving. A + indicates closure, a − non-closure, and a question mark an open problem.

Next the question arises whether some structural properties of cellular language acceptors are (semi)decidable. For example, whether or not a realtime two-way language is a real-time one-way language. The questions turned out to be not even semidecidable. Theorem 14. For any language family L that effectively contains Lrt (CA) it is not semidecidable whether L ∈ L is a real-time OCA language. In general, a family L of languages possesses a pumping lemma in the narrow sense if for each L ∈ L there exists a constant n ≥ 1 computable from L such that each z ∈ L with |z| > n admits a factorization z = uvw, where |v| ≥ 1 and u v i w ∈ L, for infinitely many i ≥ 0. The prefix u and the suffix w depend on u, w and i. Theorem 15. Any language family whose word problem is semidecidable and that effectively contains Lrt (OCA) does not possess a pumping lemma (in the narrow sense). Theorem 16. There is no minimization algorithm converting some CA or OCA (with arbitrary time complexity) to an equivalent automaton of the same type with a minimal number of states. Nevertheless, there are nontrivial decidable properties of cellular spaces. It is known that injectivity of the global transition function is equivalent to

224

Martin Kutrib

the reversibility of the automaton. It is shown in [2] that global reversibility is decidable for one-dimensional CSs, whereas the problem is undecidable for higher dimensions [36].

References 1. K. Albert, Čulik II. A simple universal cellular automaton and its one-way and totalistic version. Complex Systems, 1:1–16, 1987. 2. S. Amoroso and Y.N Patt. Decision procedures for surjectivity and injectivity of parallel maps for tessellation structures. J. Comput. System Sci., 6:448–464, 1972. 3. R.M. Balzer. An 8-state minimal time solution to the firing squad synchronization problem. Inform. Control, 10:22–42, 1967. 4. E.R. Berlekamp, J.H. Conway, and R.K. Guy. Winning Ways for your Mathematical Plays. volume 2, chapter 25, Academic Press, 1982. 5. Th. Buchholz and M. Kutrib. On the power of one-way bounded cellular time computers. In In Developments in Language Theory (DLT 1997), pp. 365-375, 1997. 6. Th. Buchholz and M. Kutrib. Some relations between massively parallel arrays. Parallel Comput., 23:1643–1662, 1997. 7. Th. Buchholz and M. Kutrib. On time computability of functions in one-way cellular automata. Acta Inform., 35:329–352, 1998. 8. J.H. Chang, O.H. Ibarra, and A. Vergis. On the power of one-way communication. J. ACM, 35:697–726, 1998. 9. C. Choffrut and K. Čulik II. On real-time cellular automata and trellis automata. Acta Inform., 21:393–407, 1984. 10. E.F. Codd. Cellular Automata. Academic Press, New York, 1968. 11. S.N. Cole. Real-time computation by n-dimensional iterative arrays of finitestate machines. In IEEE Symposium on Switching and Automata Theory (SWAT 1966). 12. S.N. Cole. Real-time computation by n-dimensional iterative arrays of finitestate machines. IEEE Trans. Comput., pages 349–365, 1969. 13. J.C. Dubacq and V. Terrier. Signals for cellular automata in dimension 2 or higher. In Theoretical Informatics (LATIN 2002), LNCS, vol. 2286, pp. 147-163. 14. C.R. Dyer. One-way bounded cellular automata. Inform. Control, 44:261–281, 1980. 15. P.C. Fischer. Generation of primes by a one-dimensional real-time iterative array. J. ACM, 12:388–394, 1965. 16. M. Gardner. Mathematical games. Sci. Amer, 224:112–117, 1971. 17. S. Ginsburg. The Mathematical Theory of Context-Free Languages. McGraw Hill, New York, 1996. 18. E. Goto. A minimal time solution of the firing squad problem. Technical report. 19. A. Grasselli. Synchronization of cellular arrays: The firing squad problem in two dimensions. Inform. Control, 28:113–124, 1975. 20. J. Hartmanis. Context-free languages and Turing machine computations. In Proceedings of the Symposia in Applied Mathematics, 19, pages 42–51, 1967. 21. G.T. Herman, W.H. Liu, S. Rowland, and A. Walker. Synchronization of growing cellular arrays. Inform. Control, 25:103–122, 1974.


225

22. O.H. Ibarra and T. Jiang. On one-way cellular arrays. SIAM J. Comput., 16:1135–1154, 1987. 23. O.H. Ibarra and T. Jiang. Relating the power of cellular arrays to their closure properties. Theoret. Comput. Sci., 57:225–238, 1998. 24. O.H. Ibarra, S.M. Kim, and S. Moran. Sequential machine characterizations of trellis and cellular automata and applications. SIAM J. Comput., 14:426–447, 1985. 25. O.H. Ibarra and M.A. Palis. Some results concerning linear iterative (systolic) arrays. J. Parallel Distributed Comput., 2:182–218, 1985. 26. O.H. Ibarra, M.A. Palis, and S.M. Kim. Fast parallel language recognition by cellular automata. Theoret. Comput. Sci., 41(2): 231-246, 1985. 27. K. Čulik II and S. Dube. An efficient solution of the firing mob problem. Theoret. Comput. Sci., 91:57–69, 1991. 28. K. Čulik II, J. Gruska, and A. Salomaa. Systolic trellis automata: Stability, decidability and complexity. Inform. Control, 71:218–230, 1986. 29. K. Čulik II and S. Yu. Iterative tree automata. Theoret. Comput. Sci., 32:227–247, 1984. 30. A.R. Smith III. Simple computation–universal cellular spaces. J. ACM, 18:339–353, 1971. 31. A.R. Smith III. Real-time language recognition by one-dimensional cellular automata. J. Comput. System Sci., 6:233–253, 1972. 32. K. Imai and K. Morita. Firing squad synchronization problem in reversible cellular automata. Theoret. Comput. Sci., 165:475–482, 1996. 33. K. Imai and K. Morita. A computation-universal two-dimensional 8-state triangular reversible cellular automaton. Theoret. Comput. Sci., 231:181–191, 2000. 34. C. Iwamoto, T. Hatsuyama, K. Morita, and K. Imai. Constructible functions in cellular automata and their applications to hierarchy results. Theoret. Comput. Sci., 270:797–809, 2002. 35. T. Jiang. The synchronization of nonuniform networks of finite automata. Inform. Comput., 97:234–261, 1992. 36. J. Kari. Reversibility and surjectivity problems of cellular automata. J. Comput. System Sci., 48:149–182, 1994. 37. A. Klein and M. Kutrib. Fast one-way cellular automata. Theoret. Comput. Sci., 1(3):233–250, 2003. 38. K. Kobayashi. The firing squad synchronization problem for a class of polyautomata networks. J. Comput. System Sci., 17:300–318, 1978. 39. K. Kobayashi. On the minimal firing time of the firing squad synchronization problem for polyautomata networks. Theoret. Comput. Sci., 7:149–167, 1978. 40. M. Kutrib. Automata arrays and context-free languages, pages 139–148. Kluwer Academic Publishers, 2001. 41. M. Kutrib and R. Vollmar. The firing squad synchronization problem in defective cellular automata. IEICE Trans. Inf. Syst., pages 895–900, 1995. 42. M. Kutrib and Th. Worsch. Investigation of different input modes for cellular automata. In Ch. Jesshope, V. Jossofov and W. Wihelmi, editors, Parallel Processing by Cellular Automata and Arrays (Parcella 1994), Akademie Verlag, Berlin, pp. 141-150, 1994. 43. A. Malcher. Descriptional complexity of cellular automata and decidability questions. J. Autom. Lang. Comb., 7:549–560, 2002. 44. M. Margenstern. Frontier between decidability and undecidability: a survey. Theoret. Comput. Sci., 231:217–251, 2000.

226

Martin Kutrib

45. B. Martin. Efficient unidimensional universal cellular automaton. In Mathematical Foundations of Computer Science (MFCS 1992), volume 629 of Lecture Notes in Computer Science, pages 374–382, Berlin, 1992. 46. B. Martin. A universal cellular automaton in quasi-linear time and its S–m–n form. Theoret. Comput. Sci., 123:199–237, 1994. 47. J. Mazoyer. A minimal time solution to the firing squad synchronization problem with only one bit of information exchanged. Technical report. 48. J. Mazoyer. A six-state minimal time solution to the firing squad synchronization problem. Theoret. Comput. Sci., 50:183–238, 1987. 49. J. Mazoyer and V. Terrier. Signals in one-dimensional cellular automata. Theoret. Comput. Sci., 217:53–80, 1999. 50. E.F. Moore. The firing squad synchronization problem, pages 213–214. AddisonWesley, 1964. 51. F.R. Moore and G.C. Langdon. A generalized firing squad problem. Inform. Control, 12:17–33, 1968. 52. K. Morita. Computation-universality of one-dimensional one-way reversible cellular automata. Inform. Process. Lett., 42:325–329, 1992. 53. K. Morita. Reversible simulation of one-dimensional irreversible cellular automata. Theoret. Comput. Sci., 148:157–163, 1995. 54. K. Morita and S. Ueno. Computation-universal models of two-dimensional 16state reversible cellular automata. Trans. IEICE, 75:141, 1992. 55. H.G. Rice. Classes of recursively enumerable sets and their decision problems. Trans. Amer. Math. Soc., 89:25–59, 1953. 56. Y. Rogozhin. Small universal turing machines. Theoret. Comput. Sci., 168:215–240, 1996. 57. P. Rosenstiehl, J.R. Fiksel, and A. Holliger. Intelligent graphs: Networks of finite automata capable of solving graph problems. Academic Press, New York. 58. A. Salomaa. Formal Languages. Academic Press, New York, 1973. 59. S.R. Seidel. Language recognition and the synchronization of cellular automata. Technical report. 60. I. Shinahr. Two- and three-dimensional firing-squad synchronization problems. Inform. Control, 24:163–180, 1974. 61. A.R. Smith. Cellular automata and formal languages. In IEEE Symposium on Switching and Automata Theory (SWAT 1970). IEEE Press, pp. 216-224, 1970. 62. R. Sommerhalder and S.C. van Westrhenen. Parallel language recognition in constant time by cellular automata. Acta Inform., 19:397–407, 1983. 63. H. Szwerinski. Time optimal solution of the firing squad synchronization problem for n-dimensional rectangles with the general at an arbitrary position. Theoret. Comput. Sci., 19:305–320, 1982. 64. V. Terrier. On real time one-way cellular array. Theoret. Comput. Sci., 141:331–335, 1995. 65. V. Terrier. Language not recognizable in real time by one-way cellular automata. Theoret. Comput. Sci., 156:281–287, 1996. √ 66. V. Terrier. Construction of a signal of ratio n + n. Unpublished manuscript, 2002. 67. H. Umeo. A simple design of time-efficient firing squad synchronization algorithms with fault-tolerance. IEICE Trans. Inf. Syst., pages 733–739, 2004. 68. H. Umeo and N. Kamikawa. A design of real-time non-regular sequence generation algorithms and their implementations on cellular automata with 1-bit inter-cell communications. Fund. Inform., 52:257–275, 2002.


227

69. H. Umeo and N. Kamikawa. Real-time generation of primes by a 1-bitcommunication cellular automaton. Fund. Inform., 58:421–435, 2003. 70. H. Umeo, M. Maeda, M. Hisaoka, and M. Teraoka. A state-efficient mapping scheme for designing two-dimensional firing squad synchronization algorithms. Fund. Inform., 74:603–623, 2006. 71. H. Umeo, K. Morita, and K. Sugata. Deterministic one-way simulation of twoway real-time cellular automata and its related problems. Inform. Process. Lett., 14:158–161, 1982. 72. J. von Neumann. Theory of Self-Reproducing Automata. University of Illinois Press. Edited and completed by Arthur W. Burks. 73. A. Waksman. An optimum solution to the firing squad synchronization problem. Inform. Control, 9:66–78, 1996.

7 Probabilistic Parsing Mark-Jan Nederhof1 and Giorgio Satta2 1

2

School of Computer Science, University of St Andrews North Haugh, St Andrews, Fife, KY16 9SX, Scotland http://www.cs.st-andrews.ac.uk/mjn Department of Information Engineering, University of Padua via Gradenigo, 6/A, I-35131 Padova, Italy [email protected]

7.1 Introduction A paper in a previous volume [51] explained parsing, which is the process of determining the parses of an input string according to a formal grammar. Also discussed was tabular parsing, which solves the task of parsing in polynomial time by a form of dynamic programming. In passing, we also mentioned that parsing of input strings can be easily generalised to parsing of finite automata. In applications involving natural language, the set of parses for a given input sentence is typically very large. This is because formal grammars often fail to capture subtle properties of structure, meaning and use of language, and consequently allow many parses that humans would not find plausible. In natural language systems, parsing is commonly one stage of processing amongst several others. The effectiveness of the stages that follow parsing generally relies on having obtained a small set of preferred parses, ideally only one, from amongst the full set of parses. This is called (syntactic) disambiguation. There are roughly two ways to achieve this. First, some kind of filter may be applied to the full set of parses, to reject all but a few. This filter may look at the meanings of words and phrases, for example, and may be based on linguistic knowledge that is very different in character from the grammar that was used for parsing. A second approach is to augment the parsing process so that weights are attached to parses and subparses. The higher the weight of a parse or subparse, the more confident we are that it is correct. This is called weighted parsing. If the weights are chosen to define a probability distribution over parses or strings, this may also be called probabilistic parsing. Disambiguation is achieved by computing the parse with the highest weight or, where appropriate, highest probability. The simplest form of probabilistic parsing relies on an assignment of probabilities to individual rules from a context-free grammar. These probabilities M.-J. Nederhof and G. Satta: Probabilistic Parsing, Studies in Computational Intelligence (SCI) 113, 229–258 (2008) c Springer-Verlag Berlin Heidelberg 2008 www.springerlink.com

230

Mark-Jan Nederhof and Giorgio Satta

are then multiplied upon combination of rules to form parses. Models that are close to this basic idea, such as [18, 21], have been highly influential from the 1990s onward. The success of probabilistic parsing is due to its flexibility and scalability, in contrast to approaches to disambiguation that rely on much deep knowledge of language. For general discussions about statistical natural language processing see [12, 43, 9]. In Section 7.2 we discuss both weighted and probabilistic context-free grammars. We investigate intersection of weighted context-free grammars and finite automata in Section 7.3. By normalisation, discussed in Section 7.4, it can be shown that for the sake of disambiguation we may restrict our attention to probabilistic context-free grammars. Parsing is treated in Section 7.5, and how the probabilities of grammar rules can be obtained empirically is explained in Section 7.6. Section 7.7 discusses the computation of prefix probabilities, and probabilistic push-down automata are the subject of Section 7.8. By considering semirings, a number of computations involving context-free grammars and push-down automata can be unified, as demonstrated in Section 7.9. We end with additional bibliographic remarks in Section 7.10.

7.2 Weighted and Probabilistic Context-Free Grammars A weighted context-free grammar (WCFG) G is a 5-tuple (Σ, N, S, R, µ), where (Σ, N, S, R) is a context-free grammar and µ is a mapping from rules in R to positive real numbers. We refer to these numbers as weights, and they should be thought of as a measure of the desirability of using the corresponding rules. In general, a rule with a high weight is preferred over one with a low weight. Let d = π1 · · · πm ∈ R∗ be a string of rules (or alternatively, of labels that uniquely identify rules), and let α and β be strings of grammar symbols. d

The expression α ⇒ β means that β can be obtained from α by a left-most derivation in m steps, and the i-th step replaces the left-most nonterminal Ai by γi according to rule πi = (Ai → γi ). All derivations in this paper are d

assumed to be left-most. If S ⇒ w, we define the yield of d as y(d) = w. #m d d We now define µ(α ⇒ β) to be i=1 µ(πi ) if α ⇒ β holds and to be d

0 otherwise. In words, if the expression α ⇒ β denotes a valid left-most derivation, we compute the product of the weights of the used rules, and otherwise we take 0. This notation allows us to define the weight of a string w as: d µ(S ⇒ w). (7.1) µ(w) = d

In words, to obtain the weight of a string we sum the weights of all left-most d derivations of that string. For choices of d such that S ⇒ w does not denote a

7 Probabilistic Parsing

231

d

valid left-most derivation, nothing is contributed to the sum. If S ⇒ w holds, d we also write µ(d) in place of µ(S ⇒ w). Example 1. In the grammar below, the rules are labelled by names πi and the weights are the numbers between brackets. π1 : S → A A (3) π2 : S → a a (1) (2) π3 : A → a This grammar is ambiguous, as there are two left-most derivations of aa, π π π namely S ⇒1 AA ⇒3 aA ⇒3 aa with weight µ(π1 ) · µ(π3 ) · µ(π3 ) = 3 · 2 · 2 = π2 12 and S ⇒ aa with weight µ(π2 ) = 1. The weight of aa is therefore µ(aa) = π π π π µ(S 1⇒3 3 aa) + µ(S ⇒2 aa) = 12 + 1 = 13. d We say a WCFG is convergent if d,w µ(S ⇒ w) is a finite number. A WCFG can be called a probabilistic context-free grammar (PCFG) if µ maps all rules to numbers no greater than 1 [28, 29, 10, 73]. Where we are dealing with PCFGs, we will often replace the name µ of the weight assignment by p. We say a WCFG is proper if for every nonterminal A: µ(π) = 1. (7.2) π=(A→α)

In other words, for each nonterminal A in a parse tree or in a sentential form, µ gives us a probability distribution over the rules that we can apply. A WCFG is said to be consistent if: d µ(S ⇒ w) = 1. (7.3) d,w

This means that µ is a probability distribution over derivations of terminal strings. An equivalent statement is that µ is a probability distribution over terminal strings, as: d µ(S ⇒ w) = µ(w). (7.4) w

d,w

Clearly, consistency implies convergence. Properness and consistency are two closely related concepts but, as we will see below, neither implies the other. An important auxiliary concept for much of the theory that is to follow is the partition function Z, which maps each nonterminal A to: d µ(A ⇒ w). (7.5) Z(A) = d,w

Note that a WCFG is consistent if and only if Z(S) = 1.

232


By decomposing derivations into smaller derivations, and by making use of the fact that multiplication distributes over addition, we can rewrite: µ(π) · Z(α), (7.6) Z(A) = π=(A→α)

where we define: Z() = 1, Z(aβ) = Z(β),

(7.7) (7.8)

Z(Bβ) = Z(B) · Z(β), for β = .

(7.9)

The partition function may be approximated by only considering derivations up to a certain depth. We define for all A and k ≥ 0: d Zk (A) = p(A ⇒ w), (7.10) d,w:depth(d)≤k

where the depth of a left-most derivation is the largest number of rules visited on a path from the root to a leaf in the familiar representation as parse tree. d More precisely, depth() = 0 and if π = (A → X1 · · · Xm ) and Xi ⇒i wi (1 ≤ i ≤ m), then depth(πd1 · · · dm ) = 1 + maxi depth(di ). By again decomposing derivations, we obtain a recursive characterisation: p(π) · Zk (α), (7.11) Zk+1 (A) = π=(A→α)

and Z0 (A) = 0 for all A, where we define: Zk () = 1, Zk (aβ) = Zk (β), Zk (Bβ) = Zk (B) · Zk (β), for β = .

(7.12) (7.13) (7.14)

lim Zk (A) = Z(A).

(7.15)

Naturally, for all A: k→∞

If we interpret (7.6) together with (7.7) through (7.9) as a system of polynomial equations over variables Z(A), for the set of nonterminals A ∈ N , then there may be several solutions. The intended solution, as given by (7.5), is the smallest non-negative solution. This follows from the fact that the operation implied by (7.11) that computes values Zk+1 (A) from values Zk (B) is monotone, and the least fixed-point of this operation corresponds to (7.5), following (7.15). The values Z(A) may be approximated by computing Zk (A) for k = 1, . . . until the values stabilise. Another option is to use Newton’s method [22]. In special cases, the solution can be found analytically.


233

Example 2. Consider the following proper WCFG: S → S S (q) S → a (1 − q) for a certain choice of q between 0 and 1. Using (7.6) through (7.9), we obtain: Z(S) = q · Z(S)2 + (1 − q).

(7.16)

We can solve this equation, distinguishing between two cases. If q ≤ 12 , then Z(S) = 1 and if q > 12 , then Z(S) = 1−q q . We make use of the fact that we need the smallest non-negative solution. It follows that the WCFG is consistent only if q ≤ 12 . The intuition for the case q > 12 is that probability mass is lost in ‘infinite derivations’. A WCFG can also be consistent without being proper. An example is: q ) S † → S ( 1−q S → S S (q) S → a (1 − q)

for

1 2

< q < 1.

7.3 Weighted Intersection It was shown by [5] that context-free languages are closed under intersection with regular languages. The proof relies on the construction of a new CFG out of an input CFG and an input finite automaton. Here we extend that construction by letting the input grammar be a weighted CFG. For an even more general construction, where also the finite automaton is weighted, we refer to [49]. To avoid a number of technical complications, we assume here that the finite automaton has no epsilon transitions, and has only one final state. Thus, a finite automaton (FA) M is a 5-tuple (Σ, Q, q0 , qf , ∆), where Σ and Q are two finite sets of input symbols and states, respectively, q0 is the initial state, qf is the final state, and ∆ is a finite set of transitions, each of the form a s → t, where s, t ∈ Q and a ∈ Σ. For a FA M as above and a PCFG G = (Σ, N, S, R, µ) with the same set Σ, we construct a new PCFG G∩ = (Σ, N∩ , S∩ , R∩ , µ∩ ), where N∩ = Q × (Σ ∪ N ) × Q, S∩ = (q0 , S, qf ), and R∩ is the set of rules that is obtained as follows. •

For each A → X1 · · · Xm in R and each sequence s0 , . . . , sm ∈ Q, with m ≥ 0, let (s0 , A, sm ) → (s0 , X1 , s1 ) · · · (sm−1 , Xm , sm ) be in R∩ ; if m = 0, the new rule is of the form (s0 , A, s0 ) → . Function µ∩ assigns the same weight to the new rule as µ assigned to the original rule.

234

•

Mark-Jan Nederhof and Giorgio Satta a

For each s → t in ∆, let (s, a, t) → a be in R∩ . Function µ∩ assigns weight 1 to this rule.

Observe that a rule of G∩ is constructed either out of a rule of G or out of a transition of M. On the basis of this correspondence between rules and transitions of G∩ , G and M, it can be stated that each derivation d∩ in G∩ deriving a string w corresponds to a unique derivation d in G deriving the same string and a unique computation c in M recognising the same string. Conversely, if there is a derivation d in G deriving string w, and some computation c in M recognising the same string, then the pair of d and c corresponds to a unique derivation d∩ in G∩ deriving the same string w. Furthermore, the weights of d and d∩ are equal, by the definition of µ∩ . Parsing of a string w = a1 · · · an can be seen as a special case of the construction, where there is a linear FA, with states q0 = s0 , s1 , . . . , sn = qf , a and transitions of the form si−1 →i si (1 ≤ i ≤ n). The intersection grammar constructed as explained above can be seen as a succinct representation of all parses of w. As weights are copied unchanged from G to G∩ , we can find the parse of w with the highest weight on the basis of G∩ . We will return to this issue in Section 7.5. We say a nonterminal in a CFG is generating if at least one terminal string can be derived from that nonterminal. We say a nonterminal is reachable if a string containing that nonterminal can be derived from the start symbol. A nonterminal is called useless if it is non-generating or non-reachable or both. A grammar G∩ as obtained above generally contains a large number of useless nonterminals, to the extent that the construction as given may not be practical. Introduction of non-generating nonterminals can be avoided by constructing rules in a bottom-up phase. That is, a rule is introduced only if all the members in the right-hand side have been found to be generating. This ensures that the left-hand side nonterminal is also generating. In a following top-down phase, the non-reachable nonterminals can be eliminated, by a standard technique that is linear in the size of the grammar [65]. Below, we will assume one more improvement. The motivation is that the number of rules of the form (s0 , A, sm ) → (s0 , X1 , s1 ) · · · (sm−1 , Xm , sm ) is exponential in m. Our improvement effectively postpones enumeration of all relevant combinations of s1 , . . . , sm−1 until (s0 , A, sm ) is found to be reachable in the top-down phase. During the bottom-up phase, given in Figure 7.1, such rules are constructed incrementally by items of the form (s0 , A → α • β, si ), where A → αβ is a rule and i = |α|. Existence of such an item in table I means that there are s1 , . . . , si−1 such that (s0 , X1 , s1 ), . . . , (si−1 , Xi , si ) are all generating nonterminals, with α = X1 · · · Xi . We also have a separate table N to store such generating nonterminals. The bottom-up phase is similar to a bottom-up variant of the parsing algorithm by [26], and the complexity is very similar. The time complexity in


235

first_phase: N = ∅ {table of generating nonterminals for N∩ } I = ∅ {table of items, partially representing rules for R∩ } A = ∅ {agenda, items yet to be processed} a for all (s → t) ∈ ∆ do add _symbol (s, a, t) for all s ∈ Q do for all (A → ) ∈ R do A = A ∪ {(s, A → •, s)} while A = ∅ do choose (s, A → α • β, t) ∈ A A = A − {(s, A → α • β, t)} add _item(s, A → α • β, t) add _symbol (s, X, t): if (s, X, t) ∈ /N N = N ∪ {(s, X, t)} for all (r, A → α • Xβ, s) ∈ I do A = A ∪ {(r, A → αX • β, t)} for all (A → Xβ) ∈ R do A = A ∪ {(s, A → X • β, t)} add _item(r, A → α • β, s): if (r, A → α • β, s) ∈ /I I = I ∪ {(r, A → α • β, s)} if β = add _symbol (r, A, s) else let Xγ = β for all (s, X, t) ∈ N do A = A ∪ {(r, A → αX • γ, t)} Fig. 7.1. The bottom-up phase of the intersection algorithm. Input are PCFG G and FA M. The tables N and I will be used in the subsequent top-down phase.

our case is cubic in the number of states of M and linear in the size of G. The space complexity is quadratic in the number of states of M. Let us now turn to the construction of G∩ out of N and I in the topdown phase, given in Figure 7.2. From the start symbol (q0 , S, qf ), we descend and construct rules for reachable nonterminals that were also found to be generating in the bottom-up phase. Nonterminals are individually added to N∩ in such a way that rules cannot be constructed more than once. Some remarks about the implementation are in order. First, the agenda A is here represented as a set to avoid the presence of duplicate elements. The maximum number of elements the agenda may contain at any given time is thereby quadratic in the number of states of M. If we alternatively represent

236


second_phase: make_rules(q0 , S, qf ) make_rules(r, A, s): {if second argument is nonterminal} if (r, A, s) ∈ / N∩ N∩ = N∩ ∪ {(r, A, s)} for all π = (A → X1 · · · Xm ) ∈ R do s0 = r sm = s for all s1 , . . . , sm−1 ∈ Q such that (s0 , X1 , s1 ), . . . , (sm−1 , Xm , sm ) ∈ N do ρ = (r, A, s) → (s0 , X1 , s1 ) · · · (sm−1 , Xm , sm ) R∩ = R∩ ∪ {ρ} µ∩ (ρ) = µ(π) for all i such that 1 ≤ i ≤ m do make_rules(si−1 , Xi , si ) make_rules(r, a, s): {if second argument is terminal} if (r, a, s) ∈ / N∩ N∩ = N∩ ∪ {(r, a, s)} ρ = (r, a, s) → a R∩ = R∩ ∪ {ρ} µ∩ (ρ) = 1 Fig. 7.2. The top-down phase of the intersection algorithm. On the basis of table N , the nonterminals and rules of G∩ are constructed, together with the weight function µ∩ on rules.

the agenda as a queue or stack, allowing elements to be present more than once, the space complexity may become cubic. Second, one may use I in the top-down phase to guide the search for relevant rules from G and states from M. This process is further simplified by having the bottom-up phase record a list of the reasons why a certain element is in N or I. For example, if (r, A → αX • β, t) was obtained from (r, A → α • Xβ, s) and (s, X, t), then the mentioned list for (r, A → αX • β, t) contains amongst others the pair consisting of (r, A → α • Xβ, s) and (s, X, t). Such a pair is recorded in the list by add _symbol if (s, X, t) is added to N after (r, A → α • Xβ, s) is added to I, and it is recorded by add _item otherwise. The additional bookkeeping however is at the cost of having larger tables at the end of the bottom-up phase. This increase is from square to cubic in the number of states of M, as a pair consisting of (r, A → α • Xβ, s) and (s, X, t) contains three states. See also [3, Exercise 4.2.21]. With or without the above optimisations, the space complexity is O(|Q|r+1 ), where r is the length of the longest right-hand side. This can be reduced to O(|Q|3 ), either by transforming the original grammar to binary form (that is, with r = 2) before the intersection, or by refining the intersection algorithm to return a grammar in binary form.

7 Probabilistic Parsing M:

G: S→aSa S→bSb S→b

b

( 12 ) ( 18 ) ( 38 )

a

a

q0

→ → → → → → → →

qf

q1 b

G∩ : (q0 , S, qf ) (q1 , S, q1 ) (qf , S, q0 ) (qf , S, q0 ) (qf , S, q0 ) (q0 , S, q0 ) (q0 , S, q0 ) (q0 , S, q0 )

237

(q0 , a, q1 ) (q1 , S, q1 ) (q1 , a, qf ) (q1 , a, qf ) (qf , S, q0 ) (q0 , a, q1 ) (qf , b, q0 ) (q0 , S, q0 ) (q0 , b, q0 ) (qf , b, q0 ) (q0 , S, qf ) (qf , b, q0 ) (qf , b, q0 ) (q0 , b, q0 ) (q0 , S, q0 ) (q0 , b, q0 ) (q0 , b, q0 ) (q0 , S, qf ) (qf , b, q0 ) (q0 , b, q0 )

( 12 ) ( 12 ) ( 18 ) ( 18 ) ( 38 ) ( 18 ) ( 18 ) ( 38 )

(q0 , b, q0 ) (q0 , a, q1 ) (q1 , a, qf ) (qf , b, q0 )

→ → → →

b a a b

(1) (1) (1) (1)

Fig. 7.3. Example of intersection of PCFG G and FA M, resulting in G∩ , which is presented here without useless nonterminals.

Example 3. Figure 7.3 shows the end result G∩ of applying the algorithm in Figures 7.1 and 7.2 on an example PCFG G and FA M.

7.4 Normalisation An obvious question is whether general convergent WCFGs have any advantages over proper and consistent PCFGs. In this section we will show that if we are only interested in the ratios between the weights of derivations, rather than in absolute values, the answer is negative. This allows us to restrict our attention to proper and consistent PCFGs for the purpose of disambiguation. The argument hinges on normalisation of WCFGs [69, 1, 49, 52], which can be defined as the construction of a proper and consistent PCFG (Σ, N, S, R, p) out of a convergent WCFG (Σ, N, S, R, µ). The function p is given by: p(π) =

µ(π) · Z(α) , Z(A)

(7.17)

for each rule π = (A → α). In words, the probability of a rule is normalised to the portion it represents of the total weight mass of derivations from the left-hand side nonterminal A. That the ratios between weights of derivations are not affected by normalisation follows from a result in [49]: d

d

p(S ⇒ w) =

µ(S ⇒ w) , Z(S)

(7.18)

238

Mark-Jan Nederhof and Giorgio Satta q

Z(S) 1

1 0.5

0

q 0

0.5

0

1

q 0

0.5

1

Fig. 7.4. The values of Z(S) and q as functions of q, for Example 4.

for each derivation d and string w. In other words, the weights of all derivations change by the same factor. Note that this factor is 1 if the original grammar is already consistent. This implies that consistent WCFGs and proper and consistent PCFGs describe the same class of probability distributions over derivations. Example 4. Let us return to the WCFG from Example 2, with the values of µ between brackets: S → S S (q) S → a (1 − q) The result of normalisation is the proper and consistent PCFG below, with the values of p between brackets: S → S S (q ) S → a (1 − q ) For q ≤ 12 , we have q = q. For q > q =

q·Z(SS ) Z(S)

=

q·Z(S )2 Z(S)

1 2

however, we have:

= q · Z(S ) = q ·

1−q q

= 1 − q.

(7.19)

The values of Z(S) and q as functions of q are represented in Figure 7.4.

7.5 Parsing As explained in Section 7.3, context-free parsing is strongly related to computing the intersection of a context-free grammar and a finite automaton. If the input grammar is probabilistic, the probabilities of the rules are simply copied to the intersection grammar. The remaining task is to find the most probable derivation in the intersection grammar. Note that the problem of finding the most probable derivation in a PCFG does not rely on that PCFG being the intersection of another PCFG and a FA. Let us therefore consider an arbitrary PCFG G = (Σ, N, S, R, p), and


239

d

our task is to find d and w such that p(S ⇒ w) is maximal. Let pmax denote this maximal value. We further define pmax (X) to be the maximal value of d

p(X ⇒ w), for any d and w, where X can be a terminal or nonterminal. Naturally, pmax = pmax (S) and pmax (a) = 1 for each terminal a. Much of the following discussion will focus on computing pmax rather d than on computing a choice of d and w such that pmax = p(S ⇒ w). The justification is that most algorithms to compute pmax can be easily extended to a computation of relevant d and w using additional data structures that record how intermediate results were obtained. These data structures however make the discussion less transparent, and are therefore largely ignored. Consider the graph that consists of the nonterminals as vertices, with an edge from A to B iff there is a rule of the form A → αBβ. If G is non-recursive, then this graph is acyclic. Consequently, the nonterminals can be arranged in a topological sort A1 , . . . , A|N | . This allows us to compute for j = |N |, . . . , 1 in this order: pmax (Aj ) =

max

π=(Aj →X1 ···Xm )

p(π) · pmax (X1 ) · . . . · pmax (Xm ).

(7.20)

The topological sort ensures that any value for a nonterminal in the right-hand side has been computed at an earlier step. A topological sort can be found in linear time in the size of the graph [19]. See [44] for an application strongly related to ours. In many cases however, there is a topological sort that follows naturally from the way that G is constructed. For example, assume that G is the intersection of a PCFG G in Chomsky normal form and a linear FA with states s0 , . . . , sn as in Section 7.3. We impose an arbitrary linear ordering ≺N on the set of nonterminals from G . As topological sort we can now take the linear ordering ≺ defined by: (si , A, sj ) ≺ (si , A , sj ) iff j > j ∨ (j = j ∧ i < i ) ∨ (j = j ∧ i = i ∧ A ≺N A ).

(7.21)

By this ordering, the computation of the values in (7.20) can be seen as a probabilistic extension of CYK parsing [31]. This amounts to a generalised form of Viterbi’s algorithm [71], which was designed for probabilistic models with a finite-state structure. If G is recursive, then a different algorithm is needed. We may use the fact that the probability of a derivation is always smaller than (or equal to) that of any of its subderivations. The reason is that the probability of a derivation is the product of the probabilities of a list of rules, and these are positive numbers not exceeding 1. We also rely on monotonicity of multiplication, i.e. for any positive numbers c1 , c2 , c3 , if c1 < c2 then c1 · c3 < c2 · c3 . The algorithm in Figure 7.5 is a special case of an algorithm by Knuth [35], which generalises Dijkstra’s algorithm to compute the shortest path in a weighted graph [19]. In each iteration, the value of pmax (A) is established for a

240


E =Σ repeat F = {A | A ∈ / E ∧ ∃A → X1 · · · Xm [X1 , . . . , Xm ∈ E]} if F = ∅ report failure and halt for all A ∈ F do p(π) · pmax (X1 ) · . . . · pmax (Xm ) q(A) = max π=(A→X1 ···Xm ): X1 ,...,Xm ∈E

choose A ∈ F such that q(A) is maximal pmax (A) = q(A) E = E ∪ {A} until S ∈ E output pmax (S) Fig. 7.5. Knuth’s generalisation of Dijkstra’s algorithm, applied to finding the most probable derivation in a PCFG.

nonterminal A. The set E contains all grammar symbols X for which pmax (X) has already been established; this is initially Σ, as we set pmax (a) = 1 for each a ∈ Σ. The set F contains the nonterminals not yet in E that are candidates to be added next. Each nonterminal A in F is such that a derivation from A exists consisting of a rule A → X1 · · · Xm , and derivations from X1 , . . . , Xm matching the values of pmax (X1 ), . . . , pmax (Xm ) found earlier. The nonterminal A for which such a derivation has the highest probability is then added to E. Knuth’s algorithm can be combined with construction of the intersection grammar, along the lines of [46], which also allows for variants expressing particular parsing strategies. See also [34]. A problem related to finding the most probable parse is to find the k most probable parses. This was investigated by [32, 54, 27]. Much of the discussed theory of probabilistic parsing carries over to more powerful formalisms, such as probabilistic tree adjoining grammars [58, 63]. We want to emphasise that finding the most probable string is much harder than finding the most probable derivation. In fact, the decision version of the former problem is NP-complete if there is a specified bound on the string length [64], and it remains so even if the PCFG is replaced by a probabilistic finite automaton [11]; see also [70]. If the bound on the string length is dropped, then this problem becomes undecidable, as shown in [55, 8].

7.6 Parameter Estimation Whereas rules of grammars are often written by linguists, or derived from structures defined by linguists, it is very difficult to correctly estimate the probabilities that should be attached to these rules on the basis of linguistic


241

intuitions. Instead, one often relies on two techniques called supervised and unsupervised estimation. 7.6.1 Supervised Estimation Supervised estimation relies on explicit access to a sample of data in which one can observe the events whose probabilities are to be estimated. In the case of PCFGs, this sample is a bag D of derivations of terminal strings, often called a tree bank. We assume a fixed order d1 , . . . , dm of the derivations in tree bank D. The bag is assumed to be representative for the language at hand, and the probability of a rule is estimated by the ratio of its frequency in the tree bank and the total frequency of rules with the same left-hand side. This is a form of relative frequency estimation. Formally, define C(π, d) to be the number of occurrences of rule π in derivation d. Similarly, C(A, d) is the number of times nonterminal A is expanded in derivation d, or equivalently, the sum of all C(π, d) such that π has left-hand side A. Summing these numbers for all derivations in the tree bank, we obtain: C(π, dh ), (7.22) C(π, D) = 1≤h≤m

C(A, D) =

C(A, dh ).

(7.23)

1≤h≤m

Our estimation for the probability of a rule π = (A → α) now is: pD (π) =

C(π, D) . C(A, D)

(7.24)

One justification for this estimation is that it maximises the likelihood of the tree bank [16]. This likelihood for given p is defined by: $ p(dh ). (7.25) p(D) = 1≤h≤m

The PCFG that results by taking estimation pD as above is guaranteed to be consistent [13, 60, 16]. Note that supervised estimation assigns probability 0 to rules that do not occur in the tree bank, which means that probabilistic parsing algorithms ignore such rules. A tree bank may contain zero occurrences of rules because it is too small to contain all phenomena in a language, and some rules that do not occur in one tree bank may in fact be valid and would occur if the tree bank were larger. To circumvent this problem one may apply a form of smoothing, which means shifting some probability mass from observed events to those that did not occur. Rules that do not occur in the tree bank thereby obtain a small but non-zero probability. For a study of smoothing techniques used for natural language processing, see [14].

242


Example 5. Consider the following CFG: π1 π2 π3 π4 π5

:S :S :S :S :S

→ → → → →

SS aSb ab ba c

Assume a tree bank consisting of only two derivations, π1 π3 π5 and π2 π3 , with yields abc and aabb, respectively. Without smoothing, the estimation is p(π1 ) = 15 , p(π2 ) = 15 , p(π3 ) = 25 , p(π4 ) = 05 , p(π5 ) = 15 . 7.6.2 Unsupervised Estimation We define an (unannotated) corpus as a bag W of strings in a language. As in the case of tree banks, the bag is assumed to be representative for the language at hand. We assume a fixed order w1 , . . . , wm of the strings in corpus W. Estimation of a probability assignment p to rules of a CFG on the basis of a corpus is called unsupervised as there is no direct access to frequencies of rules. A string from the corpus may possess several derivations, each representing different bags of rule occurrences. A common unsupervised estimation for PCFGs is a form of ExpectationMaximisation (EM) algorithm [20]. It computes a probability assignment p by successive refinements p0 , p1 , . . ., until the values stabilise. The initial assignment p0 may be arbitrarily chosen, and subsequent estimates pt+1 are computed on the basis of pt , in a way to be explained below. In each step, the likelihood pt (W) of the corpus increases. This likelihood for given p is defined by: $ p(wh ). (7.26) p(W) = 1≤h≤m

The algorithm converges to a local optimum (or a saddlepoint) with respect to the likelihood of the corpus, but no algorithm is known to compute the global optimum, that is, the assignment p such that p(W) is maximal. Computation of pt+1 on the basis of pt corresponds to a simple idea. With unsupervised estimation, we do not have access to a single derivation for each string in the corpus, and therefore cannot determine frequencies of rules by simple counts. Instead, we consider all derivations for each string, and the counts we would obtain for individual derivations are combined by taking a weighted average. The weighting of this average is determined by the current (d) , where y(d) = w, which is assignment pt , which offers us probabilities pptt(w) the conditional probability of derivation d given string w. More precisely, an estimated count Ct (π) of a rule π in a corpus, given assignment pt , can be defined by:


Ct (π) =

1≤h≤m d:y(d)=wh

243

pt (d) · C(π, d). pt (wh )

(7.27)

pt (d) · C(A, d). pt (wh )

(7.28)

Similarly: Ct (A) =

1≤h≤m d:y(d)=wh

Using these values we compute the next estimation pt+1 (π) for each rule π = (A → α) as: pt+1 (π) =

Ct (π) . Ct (A)

(7.29)

Note the similarity of this to (7.24). Example 6. Consider the CFG from Example 5, with a corpus consisting of strings w1 = abc, w2 = acb and w3 = abab. The first two strings can only be derived by d1 = π1 π3 π5 and d2 = π2 π5 , respectively. However, w3 is ambiguous as it can be derived by d3 = π1 π3 π3 and d4 = π2 π4 . For a given pt , we have: pt (d3 ) pt (w3 ) pt (d4 ) Ct (π2 ) = 1 + pt (w3 ) pt (d3 ) Ct (π3 ) = 1 + 2 · pt (w3 ) pt (d4 ) Ct (π4 ) = pt (w3 ) Ct (π5 ) = 2 pt (d4 ) pt (d3 ) +2· Ct (S) = 5 + 3 · pt (w3 ) pt (w3 ) Ct (π1 ) = 1 +

The assignment that pt converges to depends on the initial choice of p0 . We investigate two such initial choices: pt (π1 ) pt (π2 ) pt (π3 ) pt (π4 ) pt (π5 ) t = 0 0.200 0.200 0.200 0.200 0.200 t = 1 0.163 0.256 0.186 0.116 0.279 t = 2 0.162 0.257 0.184 0.117 0.279 .. .. .. .. .. .. . . . . . . t = ∞ 0.160 0.260 0.180 0.120 0.280 and:

244


pt (π1 ) pt (π2 ) pt (π3 ) pt (π4 ) pt (π5 ) t = 0 0.100 0.100 0.600 0.100 0.100 t = 1 0.229 0.156 0.330 0.028 0.257 t = 2 0.236 0.146 0.344 0.019 0.255 .. .. .. .. .. .. . . . . . . t = ∞ 0.250 0.125 0.375 0.000 0.250 In the first case, the likelihood of the corpus is 2.14 · 10−5 and in the second case 2.57 · 10−5 . As strings may allow a large number of derivations, a direct implementation of (7.27) and (7.28) is often not feasible. To obtain a more practical algorithm, we first rewrite Ct (π) as below. Treatment of Ct (A) is similar. Ct (π) =

1≤h≤m

1 pt (wh )

pt (d) · C(π, d).

(7.30)

d:y(d)=wh

The value pt (wh ) is just Z(St,h ), where St,h is the start symbol of the intersection of the PCFG with probability assignment pt and the linear FA accepting the singleton language {wh }. How this value can be computed has already been explained in Section 7.2. Let us therefore concentrate on the second part of the above expression, fixing an assignment p, rule π = (A → α) and string w = a1 · · · an . We rewrite: d p(d) · C(π, d) = p(S ⇒1 a1 · · · ai Aβ) · p(π) · (7.31) i,j

d:y(d)=w

d1 ,d2 ,d3 ,β

d

p(α ⇒2 ai+1 · · · aj ) · d

=

p(β ⇒3 aj+1 · · · an ) outer (A, i, j) · p(π) · inner (α, i, j),

(7.32)

i,j

where we define: outer (A, i, j) =

d

d

p(S ⇒1 a1 · · · ai Aβ) · p(β ⇒3 aj+1 · · · an ), (7.33)

d1 ,d3 ,β

inner (α, i, j) =

d

p(α ⇒2 ai+1 · · · aj ).

(7.34)

d2

The intuition is that the occurrences of π in the different d such that y(d) = w are grouped according to the substring ai+1 · · · aj that they cover. For each choice of i and j we look at the sum of probabilities of matching derivations, dividing them into the subderivations that are ‘inside’ and ‘outside’ the relevant occurrence of π. The values of inner (α, i, j) can be computed similarly to the computation of the partition function Z, which was explained in Section 7.2. For the remaining values, we have:


245

outer (A, i, j) = δ(A = S ∧ i = 0 ∧ j = n) + outer (B, i , j ) · p(π) · inner (γ, i , i) · inner (η, j, j ),(7.35) π=(B→γAη),i ,j

with δ defined to return 1 if its argument is true and 0 otherwise. Here we divide the derivations ‘outside’ a nonterminal occurrence into parts outside parent nonterminal occurrences, and the parts below siblings on the left and on the right. A special case is if the nonterminal occurrence can be the root of the parse tree, which corresponds to a value of 1, which is the product of zero rule probabilities. If we fill in the values for inner , we obtain a system of linear equations with outer (A, i, j) as variables, which can be solved in polynomial time. The system is of course without cyclic dependencies if the grammar is without cycles. The algorithm we have described is called the inside-outside algorithm [4, 39, 31, 57]. It generalises the forward-backward algorithm for probabilistic models with a finite-state structure [6]. Generalised PCFGs, with righthand sides representing regular languages, were considered in [37]. The insideoutside algorithm is guaranteed to result in consistent PCFGs [60, 16, 52].

7.7 Prefix Probabilities Let p be the probability assignment of a PCFG. The prefix probability of a string w is defined to be Pref (w) = v p(wv). Prefix probabilities have important applications in speech recognition. For example, assume a prefix of the input is w, and the next symbol suggested by the speech recogniser is a. The probability that a is the next symbol according to the PCFG is given by: Pref (wa) . Pref (w)

(7.36)

For given w, there may be infinitely many v such that p(wv) > 0. As we will show, the difficulty of summing infinitely many values can be overcome by isolating a finite number of auxiliary values whose computation can be carried out ‘off-line’, that is, independent of any particular w. On the basis of these values, Pref (w) can be computed in cubic time for any given w. We first extend left-most derivations to ‘dotted’ derivations written as d S ⇒ w • α. The dot indicates a position in the sentential form separating the known prefix w and a string α of grammar symbols together generating an unknown suffix v. No symbol to the right of the dot may be rewritten. The rationale is that this would lead to probability mass being included more than once in the theory that is to follow.

246


Formally, a dotted derivation can be either A ⇒ • A, which represents the dπ empty derivation, or it can be of the form A ⇒ wv • αβ, where π = (B → vα) d

π

with v = , to represent the (left-most) derivation A ⇒ wBβ ⇒ wvαβ. In the remainder of this section, we will assume that the PCFG is proper and consistent. This allows us to rewrite: d d d p(S ⇒ w • α) · p(α ⇒ v) = p(S ⇒ w • α). (7.37) Pref (w) = d ,v

d,α

d,α

Note that derivations leading from any α in the above need not be considered individually, as the sum of their probabilities is 1 for proper and consistent PCFGs. Example 7. We investigate the prefix probability of bb, for the following PCFG: π1 : S → A a (0.2) π2 : S → b (0.8) π3 : A → S a (0.4) π4 : A → S b (0.6) d

The set of derivations d such that S ⇒ bb • α, some α, can be described by the regular expression (π1 (π3 ∪ π4 ))∗ π1 π4 π2 . By summing the probabilities of these derivations, we get: m p(π1 ) · (p(π3 ) + p(π4 )) · p(π1 ) · p(π4 ) · p(π2 ) Pref (bb) = m≥0

=

p(π1 )m · p(π1 ) · p(π4 ) · p(π2 ).

m≥0

As

m≥0

p(π1 )m =

1 1−p(π1 )

= 1.25, we obtain:

Pref (bb) = 1.25 · 0.2 · 0.6 · 0.8 = 0.12. The remainder of this section derives a practical solution for computing the value in (7.37), due to [30]. This requires that the underlying CFG is in Chomsky normal form, or more precisely that every rule has the form A → BC or A → a. We will ignore rules S → here. We first distinguish two kinds of subderivation. For the first kind, the yield falls entirely within the known prefix w = a1 · · · an . For the second, the yield includes the boundary between known prefix w and unknown suffix v. We do not have to investigate the third kind of subderivation, whose yield falls entirely within the unknown suffix, because the factors involved are always 1, as explained before. For subderivations within the known prefix we have values of the form: d

d

p(A ⇒ ai+1 · · · aj ),

(7.38)


247

with i and j between 0 and n. These values can be computed using techniques already discussed, in Section 7.2 for Z, and in Section 7.6.2 for inner . Here, the computation can be done in cubic time in the length of the prefix, since there are no cyclic dependencies. Let us now look at derivations at the boundary between w and v. If the relevant part of w is empty, we have: d p(A ⇒ • A) = 1. (7.39) d

If the relevant part of w is only one symbol an , we have: d d p(A ⇒ an • α) = p(A ⇒ Bα) · p(π). d,α

(7.40)

π=(B→an ) d,α

Here B plays the role of the last nonterminal in a path in a parse tree from A down to an , taking the left-most child at each step. It is easy to see that: d d p(A ⇒ Bα) = δ(A = B) + p(π) · p(C ⇒ Bα). (7.41) d,α

d,α

π=(A→CD)

d If we replace expressions of the form d,α p(A ⇒ Bα) by variables chain(A, B), then (7.41) represents a system of linear equations, for fixed B and different A. This system can be solved with a time complexity that is cubic in the number of nonterminals. Note that this is independent of the known prefix w, and is therefore an off-line computation. If the derivation covers a larger portion of the prefix (i + 1 < n) we have: d p(A ⇒ ai+1 · · · an • α) = d,α

d

p(A ⇒ Dα) · p(π) ·

π=(D→BC) d,α

k:i i. These values can be computed in quadratic time if the other values in (7.42) have already been obtained. The resulting value of prefix _inside(S, 0) is the required prefix probability of w. In many applications, the prefix probabilities need to be computed for increasingly long strings. For example in real-time speech recognition, the input grows as the acoustic signal is processed and sequences of sounds are recognised as words. The algorithm above has the disadvantage that the values in (7.42) are specific to the length n of the prefix w. If w grows on the right by one more symbol, the computation of the values has to be done anew. The algorithm by [66], which is based on Earley’s algorithm, suffers less from this problem. Most of the values it computes can be reused as the prefix grows. A very similar algorithm was described by [56]. It differs from [66] in that it does not explicitly isolate any off-line computations. Prefix probabilities for tree adjoining and linear indexed grammars were investigated by [48, 47].

7.8 Probabilistic Push-Down Automata A paper in a previous volume [51] argued that a parsing strategy can be formalised as a mapping from CFGs to push-down automata that preserves the described languages. In this section we investigate the extension of this notion to probabilistic parsing strategies [53], which are to preserve probability distributions over strings. As in [51], our type of push-down automaton does not possess states. Hence, a push-down automaton (PDA) M is a 5-tuple (Σ, Γ, Xinit , Xfinal , ∆), where Σ is a finite set of input symbols, Γ is a finite set of stack symbols, Xinit ∈ Γ is the initial stack symbol, Xfinal ∈ Γ is the final stack symbol, and ∆ is the set of transitions. Each transition can have one of the following three x forms: X → XY (a push transition), YX → Z (a pop transition), or X → Y (a swap transition); here X, Y, Z ∈ Γ , x ∈ Σ ∪ {}. Note that in our notation, stacks grow from left to right, i.e., the top-most stack symbol will be found at the right end. Without loss of generality, we assume that any PDA is such that for a given stack symbol X = Xfinal , there are either one or more push transitions X → XY , or one or more pop transitions YX → Z, or one or more swap x transitions X → Y , but no combinations of different kinds of transition. If a PDA does not satisfy this normal form, it can easily be brought in this form by introducing for each stack symbol X = Xfinal three new stack symbols Xpush , Xpop and Xswap and new swap transitions X → Xpush , X → Xpop and X → Xswap . In each existing transition that operates on top-of-stack X, we then replace X by one from Xpush , Xpop or Xswap , depending on the type of that transition. We also assume that Xfinal does not occur in the left-hand side of any transition, again without loss of generality.


249

As usual, the process of recognition of a string w starts with a configuration consisting of the singleton stack Xinit . If a list of transitions leads to singleton stack Xfinal when the entire input w has been scanned, then we say that w is recognised. Such a list of transitions is called a computation. The language accepted by the PDA is the set of all strings that can be recognised. We assume below that a PDA is always reduced, which means that each stack symbol can be used in some computation that recognises a string. For more precise definitions, we refer to [51]. A weighted push-down automaton (WPDA) M is a 6-tuple (Σ, Γ, Xinit , Xfinal , ∆, µ), where (Σ, Γ, Xinit , Xfinal , ∆) is a PDA, and µ is a mapping from transitions in ∆ to positive real numbers. Thereby, a WPDA assigns weights to computations and strings, in the same way as WCFGs assign weights to derivations and strings. A probabilistic push-down automaton (PPDA) is a WPDA with the restriction that the values assigned to transitions are no greater than 1 [61]. Consistency is defined as for WCFGs. We say a WPDA is proper if: •

Στ =(X → XY ) p(τ ) = 1 for each X ∈ Γ such that there is at least one

•

transition of the form X → XY ; x Στ =(X → Y ) p(τ ) = 1 for each X ∈ Γ such that there is at least one transi-

•

tion of the form X → Y ; and Στ =(Y X → Z) p(τ ) = 1 for each X, Y ∈ Γ such that there is at least one

x

transition of the form Y X → Z. For each stack that may arise in the recognition of a string, exactly one of the above three clauses applies, depending on the symbol on top (provided this is not Xfinal ). The conditions ensure that the sum of probabilities of next possible transitions is always 1. An obvious question is whether parsing strategies, mapping CFGs to PDAs, preserve the capacity to describe probability distributions on strings. In many cases, PDAs are able to describe a wider range of probability distributions than the CFGs they were derived from by a parsing strategy. Consider for example the parsing strategy of top-down parsing. The stack symbols of the constructed PDA are of the form [A → α • β], where A → αβ is a rule in the CFG. The transitions are given by: a

• [A → α • aβ] → [A → αa • β] for each rule A → αaβ; • [A → α • Bβ] → [A → α • Bβ] [B → • γ] for each pair of rules A → αBβ and π = B → γ; and • [A → α • Bβ] [B → γ •] → [A → αB • β]. We assume without loss of generality that the start symbol has only one defining rule, say S → σ. The initial stack symbol is then [S → • σ] and the final stack symbol is [S → σ •]. The probability distribution described by a proper and consistent PCFG can be carried over to a PPDA implementing the top-down parsing strategy

250


if we let transitions of the first and third kind above have probability 1, and let those of the second kind have the same probability as the rule B → γ from the PCFG. The reverse does not hold in general. In the PPDA we may assign different probabilities to two different transitions of the form:

• [A → α • Bβ] → [A → α • Bβ] [B → • γ]; and • [A → α • Bβ ] → [A → α • Bβ ] [B → • γ]. Such a distinction between the different contexts for an occurrence of nonterminal B cannot normally be encoded into the original CFG. This observation is related to a technique from [17] that allows probability distributions more refined than those that can be expressed in terms of a given CFG. See also [33]. Example 8. Consider the CFG: π1 π2 π3 π4 π5

: S→A :A→aB :A→bB :B→c :B→d

If p is the probability distribution over strings induced by a proper PCFG that extends the CFG above, then we must have: p(bc) p(ac) = . p(ad) p(bd)

(7.43)

Another way of looking at this is that we have a 2-dimensional parameter space, as there are only two free parameters: once we choose p(π2 ) and p(π4 ), then p(π3 ) must be 1−p(π2 ) and p(π5 ) must be 1−p(π4 ). Naturally p(π1 ) = 1. Consider now the corresponding top-down PDA. If we are to turn this into a proper PPDA, all transitions must have probability 1, except the following six: • • • • • •

[S → • A] → [S → • A] [A → • aB], [S → • A] → [S → • A] [A → • bB], [A → a • B] → [A → a • B] [B → • c], [A → a • B] → [A → a • B] [B → • d], [A → b • B] → [A → b • B] [B → • c], and [A → b • B] → [A → b • B] [B → • d].

Now (7.43) no longer restricts the space of available probability distributions. Seen in a different way, we have a 3-dimensional parameter space, as there are three free parameters; the probabilities of the first, third and fifth transitions above determine those of the others.


251

Another parsing strategy that preserves the allowable probability distributions is left-corner parsing [68]. This preservation does not hold for all parsing strategies however, as pointed out for bottom-up parsing by [1]. Another strategy for which it does not hold is LR parsing. In [53], an example is given of a PCFG with a probability distribution that cannot be expressed in terms of the corresponding PDA implementing the LR strategy. However, as shown by [50], this problem disappears if we abandon the requirement that the PPDA be proper.

7.9 Semirings Let us compare the computation of Z (Section 7.2) with the computation of pmax (Section 7.5). An important similarity is that values coming from members in the right-hand side of a rule are multiplied, as can be witnessed in both (7.9) and (7.20). An important difference is that in (7.6) we add the values coming from alternative derivations, whereas in (7.20) these values are combined by maximisation. By allowing other operations in place of those mentioned above, possibly with another domain of weights, we obtain a general class of computations involving context-free grammars. The domain and operations are subject to a number of constraints, which can be expressed as an algebraic structure. Formally, a semiring is a 5-tuple (D, ⊕, ⊗, 0, 1), where D is a set, ⊕ and ⊗ are binary operations on D, and 0, 1 ∈ D, with the following properties for all a, b, c ∈ D: additive identity a ⊕ 0 = 0 ⊕ a = a, additive commutativity a ⊕ b = b ⊕ a, additive associativity (a ⊕ b) ⊕ c = a ⊕ (b ⊕ c), multiplicative identity a ⊗ 1 = 1 ⊗ a = a, annihilation a ⊗ 0 = 0 ⊗ a = 0, multiplicative associativity (a ⊗ b) ⊗ c = a ⊗ (b ⊗ c), distributivity a ⊗ (b ⊕ c) = (a ⊗ b) ⊕ (a ⊗ c) and (a ⊕ b) ⊗ c = (a ⊗ c) ⊕ (b ⊗ c). The semiring (R+ , +, ·, 0, 1) underlies the computation of Z, where R+ stands for the non-negative real numbers, and + and · stand for ordinary addition and multiplication. By replacing + by max, we obtain the semiring (R+ , max, ·, 0, 1), which underlies the computation of pmax . These and several other semirings were discussed in relation to context-free grammars by [25]. Semirings are firmly rooted in the theory of context-free grammars and other formalisms, often in connection with formal power series [36]. For applications of semirings with a focus on finite-state transducers, see [45]. In the remainder of this section, we investigate the semiring (R+ ∪ {∞}, min, +, ∞, 0). The domain includes the symbol ∞, which also acts as the ‘zero’ element. This means that min(a, ∞) = min(∞, a) = a and a + ∞ = ∞ + a = ∞ for all a. We will show that this semiring is useful

252


for error correction of programming languages, which is closely related to the problem of computing the optimal parse on the basis of WCFGs, as is known from [67]. Assume a CFG G, and assume two functions on terminals, called d and i, and a function s on pairs of terminals. These functions define the costs of correcting a string by deleting or inserting a terminal, or by substituting one terminal by another. Costs are non-negative real numbers. We assume that s(a, b) ≤ d(a) + i(b), or in words, it is at least as costly to delete a and insert b as to substitute a by b. Naturally, s(a, a) = 0 for all a, which means that leaving a terminal unaffected can be treated as substituting it by itself. The minimum edit distance between two strings w and v, denoted by dist(w, v), is defined as the minimum sum of costs of a list of deletions, insertions and substitutions needed to turn w into v [72]. We are now asked to solve the following problem. Given a string w, compute the string v in the language generated by G that minimises dist(w, v). Similarly to our presentation in Section 7.5, the algorithm we will show computes this minimal value dist(w, v), but not the relevant v itself nor the used edit operations. These can be computed by a simple extension of the basic mechanism. Let us assume a PDA A instead of a CFG as representation of a contextfree language, which slightly simplifies the discussion. The PDA may implement any parsing strategy. We now construct a WPDA A as follows. For each stack symbol X in A, A has two distinct stack symbols X and Xdel . A symbol of the form Xdel will be on top of the stack immediately after a substitution, or at the beginning of the input. While it is on top of the stack, we allow an uninterrupted sequence of deletions, but no other actions. We thereby effectively force a canonical ordering on the edit operation and stack manipulations, placing deletions as early as possible. If the initial stack symbol of A is X, then that of A is Xdel . Further: Pop and push transitions are copied unchanged from A to A . Also swap transitions of the form X → Y are copied unchanged. The weight of all these transitions is 0. a • For each transition X → Y in A, A has the following transitions: b – X → Ydel with weight s(b, a), for each b, and – X → Y with weight i(a). • For each stack symbol Xdel , A has the following transitions: a – Xdel → Xdel with weight d(a), for each a, and – Xdel → X with weight 0. •

As dictated by the specified semiring, weights in a computation are added. In the presence of ambiguity, we take the minimum weight of all the computations that recognise a string. Example 9. Consider the following grammar and the top-down PDA obtained from it:


[S → • aAa]del → [S → • aAa] [S → • aAa] → [S → a • Aa] [S → a • Aa] → [S → a • Aa] [A → • a] a [A → • a] → [A → a •]del [A → a •]del → [A → a •] [S → a • Aa] [A → a •] → [S → aA • a] b

[S → aA • a] → [S → aAa •]del b

[S → aAa •]del [S → aAa •]del → [S → aAa •] [S → aAa •]del →

253

(0) (i(a)) (0) (0) (0) (0) (s(b, a)) (d(b)) (0)

Fig. 7.6. One possible list of transitions that can be applied in order to recognise string abb with error correction, in Example 9. The weights of these transitions are given between brackets.

S→aAa A→bAb A→a One of several ways to recognise the string abb, while allowing for error correction, is by application of the list of transitions in Figure 7.6. The total weight of the computation is i(a) + s(b, a) + d(b). There are other computations recognising the same string, which may have lower weight, depending on the values of i, d and s. In transforming a PDA to become an error-correcting WPDA, new nondeterminism is introduced. By tabulation however, all computations can be simulated in cubic time in the input length, in a way that allows extraction of the computation with the lowest weight [38]. Depending on the chosen parsing strategy, the result may be similar to Earley’s algorithm [2, 42], to CYK parsing [67], or to tabular LR parsing [41, 40].

7.10 Further References Some interpretations of probabilistic formalisms differ from what we have described above in that they define acceptance by cut-point. This means that a probabilistic grammar or automaton with probability assignment p is paired with a number c between 0 and 1. The language that is thereby defined consists of all strings w such that p(w) > c. For PCFGs this was investigated by [59], and for PPDAs by [28, 24, 62, 23]. Assume a PCFG with probability assignment p. If we let λπ = loge p(π) for each rule π, then the probability of a derivation can be rewritten as: $ $ p(π)C(π,d) = eλπ ·C(π,d) . (7.44) p(d) = π

π

254


This equation stresses that the probability of a derivation is determined only by the frequencies of individual rules occurring in it. We cannot express, say, preference for combinations or patterns of rules. For this, we need to generalise the framework to log-linear models [7, 15]. Such a model allows us to specify a number of arbitrary features c1 , . . . , cm on derivations. The features map derivations to non-negative numbers. Further, there is an equal number of weights λ1 , . . . , λm . The model thereby defines a probability distribution on derivations as: 1 $ λi ·ci (d) 1 p(d) = e = e i λi ·ci (d) , (7.45) z i z

where z is a normalisation constant, that is, z is the sum of e d that are valid left-most derivations.

i

λi ·ci (d)

for all

References 1. S. Abney, D. McAllester, and F. Pereira. Relating probabilistic grammars and automata. In 37th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pages 542–549, Maryland, USA, 1999. 2. A.V. Aho and T.G. Peterson. A minimum distance error-correcting parser for context-free languages. SIAM Journal on Computing, 1:305–312, 1972. 3. A.V. Aho and J.D. Ullman. Parsing, volume 1 of The Theory of Parsing, Translation and Compiling. Prentice-Hall, Englewood Cliffs, N.J., 1972. 4. J.K. Baker. Trainable grammars for speech recognition. In J.J. Wolf and D.H. Klatt, editors, Speech Communication Papers Presented at the 97th Meeting of the Acoustical Society of America, pages 547–550, 1979. 5. Y. Bar-Hillel, M. Perles, and E. Shamir. On formal properties of simple phrase structure grammars. In Y. Bar-Hillel, editor, Language and Information: Selected Essays on their Theory and Application, chapter 9, pages 116–150. Addison-Wesley, Reading, Massachusetts, 1964. 6. L.E. Baum. An inequality and associated maximization technique in statistical estimation of probabilistic functions of a Markov process. Inequalities, 3:1–8, 1972. 7. A. Berger, S. Della Pietra, and V. Della Pietra. A maximum entropy approach to natural language processing. Computational Linguistics, 22:39–71, 1996. 8. V.D. Blondel and C. Canterini. Undecidable problems for probabilistic automata of fixed dimension. Theory of Computing Systems, 36:231–245, 2003. 9. R. Bod, J. Hay, and S. Jannedy, editors. Probabilistic Linguistics. MIT Press, 2003. 10. T.L. Booth and R.A. Thompson. Applying probabilistic measures to abstract languages. IEEE Transactions on Computers, C-22:442–450, 1973. 11. F. Casacuberta and C. de la Higuera. Computational complexity of problems on probabilistic grammars and transducers. In A. Oliveira, editor, Grammatical Inference: Algorithms and Applications, volume 1891 of Lecture Notes in Artificial Intelligence, pages 15–24. Springer-Verlag, 2000. 12. E. Charniak. Statistical Language Learning. MIT Press, 1993.


255

13. R. Chaudhuri, S. Pham, and O.N. Garcia. Solution of an open problem on probabilistic grammars. IEEE Transactions on Computers, C-32:748–750, 1983. 14. S.F. Chen and J. Goodman. An empirical study of smoothing techniques for language modeling. In 34th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pages 310–318, Santa Cruz, California, USA, 1996. 15. Z. Chi. Statistical properties of probabilistic context-free grammars. Computational Linguistics, 25:131–160, 1999. 16. Z. Chi and S. Geman. Estimation of probabilistic context-free grammars. Computational Linguistics, 24:299–305, 1998. 17. M.V. Chitrao and R. Grishman. Statistical parsing of messages. In Speech and Natural Language, Proceedings, pages 263–266, Hidden Valley, Pennsylvania, 1990. 18. M. Collins. Three generative, lexicalised models for statistical parsing. In 35th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pages 16–23, Madrid, Spain, 1997. 19. T.H. Cormen, C.E. Leiserson, and R.L. Rivest. Introduction to Algorithms. The MIT Press, 1990. 20. A.P. Dempster, N.M. Laird, and D.B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. J. Royal Statistical Society Series, B 39:1–38, 1977. 21. J. Eisner and G. Satta. Efficient parsing for bilexical context-free grammars and head automaton grammars. In 37th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pages 457–464, Maryland, USA, 1999. 22. K. Etessami and M. Yannakakis. Recursive Markov chains, stochastic grammars, and monotone systems of nonlinear equations. In 22nd International Symposium on Theoretical Aspects of Computer Science, volume 3404 of Lecture Notes in Computer Science, pages 340–352, Stuttgart, Germany, 2005. Springer-Verlag. 23. R. Freivalds. Probabilistic machines can use less running time. In Proceedings of IFIP Congress 77, pages 839–842, Toronto, 1977. 24. K.S. Fu and T. Huang. Stochastic grammars and languages. International Journal of Computer and Information Sciences, 1:135–170, 1972. 25. J. Goodman. Semiring parsing. Computational Linguistics, 25:573–605, 1999. 26. S.L. Graham, M.A. Harrison, and W.L. Ruzzo. An improved context-free recognizer. ACM Transactions on Programming Languages and Systems, 2:415–462, 1980. 27. L. Huang and D. Chiang. Better k-best parsing. In Proceedings of the Ninth International Workshop on Parsing Technologies, pages 53–64, Vancouver, British Columbia, Canada, 2005. 28. T. Huang and K.S. Fu. On stochastic context-free languages. Information Sciences, 3:201–224, 1971. 29. S.E. Hutchins. Moments of strings and derivation lengths of stochastic contextfree grammars. Information Sciences, 4:179–191, 1972. 30. F. Jelinek and J.D. Lafferty. Computation of the probability of initial substring generation by stochastic context-free grammars. Computational Linguistics, 17:315–323, 1991. 31. F. Jelinek, J.D. Lafferty, and R.L. Mercer. Basic methods of probabilistic context free grammars. In P. Laface and R. De Mori, editors, Speech Recognition and

256

32.

33. 34. 35. 36.

37. 38. 39.

40.

41.

42. 43. 44. 45.

46. 47.

48.

Mark-Jan Nederhof and Giorgio Satta Understanding — Recent Advances, Trends and Applications, pages 345–360. Springer-Verlag, 1992. V.M. Jiménez and A. Marzal. Computation of the n best parse trees for weighted and stochastic context-free grammars. In Advances in Pattern Recognition, volume 1876 of Lecture Notes in Computer Science, pages 183–192, Alicante, Spain, 2000. Springer-Verlag. M. Johnson. PCFG models of linguistic tree representations. Computational Linguistics, 24:613–632, 1998. D. Klein and C.D. Manning. Parsing and hypergraphs. In Proceedings of the Seventh International Workshop on Parsing Technologies, Beijing, China, 2001. D.E. Knuth. A generalization of Dijkstra’s algorithm. Information Processing Letters, 6:1–5, 1977. W. Kuich. Semirings and formal power series: their relevance to formal languages and automata. In G. Rozenberg and A. Salomaa, editors, Handbook of Formal Languages, Vol. 1, chapter 9, pages 609–677. Springer, Berlin, 1997. J. Kupiec. Hidden Markov estimation for unrestricted stochastic context-free grammars. In ICASSP’92, volume I, pages 177–180, 1992. B. Lang. A generative view of ill-formed input processing. In ATR Symposium on Basic Research for Telephone Interpretation, Kyoto, Japan, 1989. K. Lari and S.J. Young. The estimation of stochastic context-free grammars using the Inside-Outside algorithm. Computer Speech and Language, 4:35–56, 1990. A. Lavie. An integrated heuristic scheme for partial parse evaluation. In 32nd Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pages 316–318, Las Cruces, New Mexico, USA, 1994. A. Lavie and M. Tomita. GLR∗ – an efficient noise-skipping parsing algorithm for context free grammars. In Third International Workshop on Parsing Technologies, pages 123–134, Tilburg (The Netherlands) and Durbuy (Belgium), 1993. G. Lyon. Syntax-directed least-errors analysis for context-free languages: A practical approach. Communications of the ACM, 17:3–14, 1974. C.D. Manning and H. Schütze. Foundations of Statistical Natural Language Processing. MIT Press, 1999. A. Martelli and U. Montanari. Optimizing decision trees through heuristically guided search. Communications of the ACM, 21:1025–1039, 1978. M. Mohri. Statistical natural language processing. In M. Lothaire, editor, Applied Combinatorics on Words, chapter 4, pages 210–240. Cambridge University Press, 2005. M.-J. Nederhof. Weighted deductive parsing and Knuth’s algorithm. Computational Linguistics, 29:135–143, 2003. M.-J. Nederhof, A. Sarkar, and G. Satta. Prefix probabilities for linear indexed grammars. In Fourth International Workshop on Tree Adjoining Grammars and Related Frameworks, pages 116–119. Institute for Research in Cognitive Science, University of Pennsylvania, 1998. M.-J. Nederhof, A. Sarkar, and G. Satta. Prefix probabilities from stochastic tree adjoining grammars. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, volume 2, pages 953–959, Montreal, Quebec, Canada, 1998.


257

49. M.-J. Nederhof and G. Satta. Probabilistic parsing as intersection. In 8th International Workshop on Parsing Technologies, pages 137–148, LORIA, Nancy, France, 2003. 50. M.-J. Nederhof and G. Satta. An alternative method of training probabilistic LR parsers. In 42nd Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pages 551–558, Barcelona, Spain, 2004. 51. M.-J. Nederhof and G. Satta. Tabular parsing. In C. Martín-Vide, V. Mitrana, and G. Pˇ aun, editors, Formal Languages and Applications, pages 529–549. Springer, 2004. 52. M.-J. Nederhof and G. Satta. Estimation of consistent probabilistic context-free grammars. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 343–350, New York, USA, 2006. 53. M.-J. Nederhof and G. Satta. Probabilistic parsing strategies. Journal of the ACM, 53:406–436, 2006. 54. L.R. Nielsen, D. Pretolani, and K.A. Andersen. Finding the k shortest hyperpaths using reoptimization. Operations Research Letters, 34:155–164, 2006. 55. A. Paz. Introduction to Probabilistic Automata. Academic Press, New York, 1971. 56. E. Persoon and K.S. Fu. Sequential classification of strings generated by SCFG’s. International Journal of Computer and Information Sciences, 4:205–217, 1975. 57. D. Prescher. Inside-outside estimation meets dynamic EM. In Proceedings of the Seventh International Workshop on Parsing Technologies, Beijing, China, 2001. 58. P. Resnik. Probabilistic tree-adjoining grammar as a framework for statistical natural language processing. In Proceedings of the fifteenth International Conference on Computational Linguistics, volume 2, pages 418–424, Nantes, 1992. 59. A. Salomaa. Probabilistic and weighted grammars. Information and Control, 15:529–544, 1969. 60. J.-A. Sánchez and J.-M. Benedí. Consistency of stochastic context-free grammars from probabilistic estimation based on growth transformations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19:1052–1055, 1997. 61. E.S. Santos. Probabilistic grammars and automata. Information and Control, 21:27–47, 1972. 62. E.S. Santos. Probabilistic pushdown automata. Journal of Cybernetics, 6:173– 187, 1976. 63. Y. Schabes. Stochastic lexicalized tree-adjoining grammars. In Proceedings of the fifteenth International Conference on Computational Linguistics, volume 2, pages 426–432, Nantes, 1992. 64. K. Sima’an. Computational complexity of probabilistic disambiguation. Grammars, 5:125–151, 2002. 65. S. Sippu and E. Soisalon-Soininen. Parsing Theory, Vol. I: Languages and Parsing, volume 15 of EATCS Monographs on Theoretical Computer Science. Springer-Verlag, 1988. 66. A. Stolcke. An efficient probabilistic context-free parsing algorithm that computes prefix probabilities. Computational Linguistics, 21:167–201, 1995. 67. R. Teitelbaum. Context-free error analysis by evaluation of algebraic power series. In Conference Record of the Fifth Annual ACM Symposium on Theory of Computing, pages 196–199, 1973.

258


68. F. Tendeau. Stochastic parse-tree recognition by a pushdown automaton. In Fourth International Workshop on Parsing Technologies, pages 234–249, Prague and Karlovy Vary, Czech Republic, 1995. 69. R.A. Thompson. Determination of probabilistic grammars for functionally specified probability-measure languages. IEEE Transactions on Computers, C23:603–614, 1974. 70. E. Vidal, F. Thollard, C. de la Higuera, F. Casacuberta, and R.C. Carrasco. Probabilistic finite-state machines — part I. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27:1013–1025, 2005. 71. A.J. Viterbi. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory, IT13:260–269, 1967. 72. R.A. Wagner and M.J. Fischer. The string-to-string correction problem. Journal of the ACM, 21:168–173, 1974. 73. C.S. Wetherell. Probabilistic languages: A review and some open questions. Computing Surveys, 12:361–379, 1980.

8 DNA-Based Memories: A Survey Andrew J. Neel and Max H. Garzon Department of Computer Science, The University of Memphis 209 Dunn Hall, Memphis, TN 38152 3240 {aneel,mgarzon}@memphis.edu

Summary. DNA-based computers have been made possible by biotechnology developed in the last two decades. They can make advances on challenges caused by limiting features of conventional silicon computers. General and application specific DNA-based computers both require memory systems for DNA computers capable of either sophisticated processing capabilities or the storage of massive amounts of data and, more importantly, effective methods to extract information meaningful to human brains from massive corpora of data. We survey the challenges and methods to build such memories, as well as some applications where they offer very good potential. The DNA memories discussed here do not require an intelligent, outside “brain” to extract the relevant features from given data. Hybridization affinity naturally selects the relevant features in the input and display them on a DNA chip signature as a 2D graphical and semantic representation, by relatively simple parallel procedures that would take forbidding amounts of time to operate on equivalent massive amount of data in digital form by conventional computers.

Keywords: Noncrosshybridizing oligonculeotides, PCR, PCR Selection, DNAbased memories, information retrieval, semantic retrieval, DNA chips, textual entailment.

8.1 Introduction The original motivation for DNA Computing was the real feasibility, demonstrated by [1], to utilize Deoxyribonucleic Acid (DNA) molecules as data processors capable of solving problems intractable with conventional solutions. The original vision was to ultimately replace conventional computers with biological computers made of DNA. As such, DNA computers would then require input-output protocols, processors, and memory systems to process information for difficult general-purpose computational applications. This vision has substantially evolved in the last decade. Research has expanded from building DNA Computers for general solutions toward the use of DNA A.J. Neel and M.H. Garzon: DNA-Based Memories: A Survey, Studies in Computational Intelligence (SCI) 113, 259–275 (2008) c Springer-Verlag Berlin Heidelberg 2008 www.springerlink.com

260

Andrew J. Neel and Max H. Garzon

as special-purpose computers for specific applications. Here scientists apply lessons learned in biology to solve difficult computational problems by taking advantage of the massively parallel nature of DNA molecules. A survey of the basic biological ideas and biotechnology that made DNA Computing a reality can be found in [15, 21, 22]. Both the original and new visions still require memory systems for DNA computers capable of either sophisticated processing capabilities (such as selfassembly of DNA into useful molecular structures [31, 32], or capable of storing, in principle, large amounts of data (order of terabytes and larger) for information retrieval [2]. In addition to its computational role, Eric Baum [2] suggested that DNA is capable of storing data more compactly than is possible using the best technologies conventional expertise can create. A terabyte of data can, in principle, fit well within a gram of DNA material. The sheer capacity best conventional counterparts of laser media such as CD / DVD which store about 8 Gigs max on 120mm disk, solid state media which stores about the same per cm2 , and magnetic hard disks which stores up to one Terabyte over several 3.5in platters [4]. Given the current state of biotechnology, it seems conceivable that large caches of data can be represented in so-called DNA-based memories in small volumes. Further, by taking advantage of massive parallelism naturally occurring in DNA interactions, it may be possible not only to store terabytes of data compactly, but also mine data from it in just a few hours. The most striking of possibilities is the potential to apply what is known about DNA computing to create so-called memories capable of retrieving information where only data is stored. This idea is best illustrated by a hypothetical memory that would store results of a national survey as data but retrieves information in the form of line graphs/charts, or answers to complex questions which require aggregation of data or even reasoning about the contents of the memory. The realization of this vision poses a number of challenges. The critical one is “Can such a memory be created to store even genetic data which is native to the source media?” A second question can be framed as “Can one realistically design a DNA based memory to store and mine a terabyte of data in only a few hours?” Third, “Can it be done with the same degree of accuracy and reliability standard on conventional computers?” Finally, with the potential quantity of data so vast and the memory sizes so minuscule, “How will the results become specifically useful to humans?” The next section presents two of these challenges (specifically, “How to test DNA memories?” and “How to build DNA memories?”). The following sections continue by summarizing solutions to many of the critical challenges facings DNA memories. The final section concludes by summarizing some of the known applications and some other potential applications of DNA memories.

8 DNA-Based Memories: A Survey

261

8.2 Challenges to DNA Memory Implementations This section introduces several very critical challenges in developing and testing DNA Memories. The first of these challenges is technological in nature and must be met on two fronts: biotechnology and computer technology. Biotechnologies have advanced enough to retrieve and sequence results from DNA memories and can provide a means to create DNA of any desirable species. However, no efficient bio-process or bio-technology standard is readily available to systematically and automatically encode text and data into DNA form. As such, years may be required to bring DNA memories to the commercial world. Computer technologies need to advance in order to enable faster development and lower cost testing for memory protocols. Alternatives to testing in vitro provide a platform capable of testing principle implementations of protocols in silico but are bound to relatively small memories given computational boundaries of conventional hardware. Until these issues are addressed, researchers will be challenged to constantly replace and update hardware for development and testing purposes. The final analysis is that the technologies are available for DNA-based memories but they are not always efficient. The next section shows how Baum’s proposed memory could be implemented with extant biotechnologies and briefly analyzes the efficiency of the technologies that enable implementation. The largest class of challenges to overcome is centered on the very chemistry that makes DNA-based memories so intriguing. First in this class is the need for input protocols that encode data into DNA such that undesirable hybridizations do not occur. For example, consider the disastrous result of an input protocol that encodes two very different inputs into two very chemically similar DNA structures. The resulting memory would have two data elements capable of hybridizing together and to queries. Similarly, if two queries were encoded as Watson-Crick complements of each other, the retrieval protocol certainly fail to retrieve. Again, consider the input protocol that encodes queries as WC complements of DNA structures that represent very different data in the memory. The retrieval protocol could provide none or very confusing results. This challenge is best expressed as the problem of finding input protocols that store data into DNA reliably to enable reliable retrieval and to prevent data loss or confusion of data. Still another challenge in this class is to overcome the effects of kinetic forces that negatively influence motion and chemical reactions within the test tube. The kinetic effect on the motion of DNA may prevent queries from reaching a particular target DNA structure in the memory. By analogy, a single person in a dense crowd of people may see his date across the room but not be able to reach her because he is continually buffeted by many a people moving in different directions. Even if two DNA molecules get close enough to hybridize, it may be that the crowd of DNA prevents a desirable fold or bend in the DNA or causes partial hybridization to more than one molecule (e.g. a 3-way knot). The root cause of this problem is too many people (i.e., too

262


much DNA) in too small a room (or test tube holding the DNA memory). As a result, this challenge is to find a concentration ideal for retrieval protocols to operate. Another class of challenges is found as a result of the capacity potential. Specifically, it may be that so much data is stored that retrieval will eventually fail to return anything useful to humans (typical internet seacrhes comes to mind.) Even with the potential for DNA to act as a computer and create information where data is stored, it is also conceivable such protocols will not scale to the maximum data storage capacity of DNA-based memories. As such, too much data would result in far too little information. The challenge thus remains to invent scalable retrieval protocols and DNA structures that capture exactly the data and information expected by a human querying the system. Even more challenging is to create these structures and protocols to provide very informative and very specific results. This challenge is best expressed as one of retrieving (getting what humans want) by the semantics of the query constructed from human language that expresses what is desired. More succinctly, this challenge is one of semantic retrieval. The remainder of this article demonstrates solutions to two of these challenges. Section 8.3 tackles the critical challenge of overcoming the time and expense of in vitro experimentation, a challenge made even more difficult due to a glass wall that separates the computer scientist, who is interested in DNA memories as solutions to information storage and retrieval problems, and the chemist who has the education and experience to perform experiments in vitro and produce actual results. This solution uses a computer software, called EdnaCo [14], that is capable of reliable simulation of in vitro experiments. Section 8.4 continues with a second critical problem of producing the raw material needed to encode such volumes of data into DNA. The ideal solution to this challenge, known as the encoding solution, is a DNA library that is resistant to self hybridization and whose complement library is also resistant to self-hybridization. The remaining challenges of the capacity potential and kinetic effects are discussed elsewhere [14].

8.3 Virtual Test Tubes In this section we summarize the experimental tool used to test and benchmark the memory protocols. Full details of virtual test tubes can be found in several sources [14, 13]. 8.3.1 Test Tubes in Silico A virtual test tube (VTT) is “any type of (simulated) biomolecular reactions in electronic media that captures fairly closely the environment and kinetics of the molecular interactions, while making minimal assumptions about the global behavior of molecular populations.” (as defined by [14]). EdnaCo [14] is


263

a distributed VTT implementation. The computational framework of EdnaCo is a complex of interacting data structures distributed over several processing nodes which are interconnected, transparently to the user, in order to produce a single test tube in each run. The VTT of EdnaCo consists of an organized network space of a cellular automaton [16] arranged in grid formation. The nodes (cells) represent quanta of 2D or 3D space that may be empty or occupied by nucleotides, molecules, or other reactants. Each cell can also be characterized by associated parameters that render the tube conditions in a realistic way, such as temperature, salinity, covalent bonds, etc. The entire tube is distributed over a cluster of processors in such a way that each local processor holds only a segment of the entire tube. A segment is itself a copy of Edna, so that one can check the contents of the tube and manage deletions (when a strand leaves the local tube segment) and additions (incoming strands, strand additions at the outset of the simulation, and hybridization events). No strand is split between two different nodes, but their Brownian motion may include migration to any other different node. Further details on the performance and implementation of EdnaCo can be found in [14]. Real nucleotides are replaced in EdnaCo with virtual ones implemented as C++ objects. Polymers, called strands when implemented in simulation, are represented as complex structural combinations of these nucleotide objects (e.g., linked lists in a software implementation.) Strands carry context information (meta-information, such as position, velocity, direction, etc.), in addition to their internal structure, which may include even morphological information. Strand interaction in silico occurs similarly to how it would occur if the interaction were to occur in vitro. Two strands encounter each other when they come into close proximity to each other. At this point of the encounter, the tube attempts to hybridize one strand to hybridize to the other, according to some pre-specified criterion for local interactions, for example, some approximation of the Gibbs Energy released by the real molecules. One approximation is the Hamming distance [30] with provides an error count or count of mismatches between two DNA sequences of equal length perfectly lined up to each another. A second approximation is the h-measure developed by [16] that computes the Hamming distance with frame shifts but still assumes the strands are rigid and do not form buldges, in particular. A third approximation is the simplified dynamic programming algorithm of [9]. Each strand is moved about by a motion engine that mimics random-like Brownian motion of the real test tube with actually random motion. The motion engine tracks when any molecule moves beyond the border of a cell. When a border is crossed, the motion model transfers the molecule to the migration engine, which moves it from cell to cell (processor to processor). Motion occurs in discrete steps, called iterations. Each iteration corresponds to about 1 ms of real time in the real test tube, roughly equivalent to the time it takes two molecules to settle a hybridization event.

264


8.3.2 Validation How can a naturally suspecting biologist or chemist give any credence to the outcome of a simulation or any conclusions based on their analyses? To validate the simulation to suspecting chemists, it is necessary to develop controlled experiments that will quantify the degree of reliability and fidelity of the test tube experiments. For validation, Adleman’s experiment was successfully recreated [13] inside a virtual test tube, except at a larger scale that well illustrates the power and scope of the simulations. Random graph configurations [28] were chosen as instances by selecting varying edge densities depending on a fixed probability (0.2, 0.4, and 0.6) of including an edge from the set of possible edges. The size of the graph (number of vertices) varied from 5-9. Each graph was a positive instance, where one witness Hamiltonian path was placed randomly, connecting source to destination. Vertex strands were selected as polymers of 20 bases. Edges were constructed from two vertices by taking the last 10 bases of one vertex and ligating it to first 10 bases of another (but in Watson-Crick complementary form.) The experiment was done under conditions that capture a mildly stringent hybridization criterion where hybridization occurs only if the strands perfectly match in all but two or three places, but were otherwise unconstrained [19], including frame-shifts. The experiment was performed 30 times for 3000 iterations. The results were averaged to provide accurate results. Over 99.4% of nearly 500 total instances of the problem systematically returned the correct answer and solution. The result of this simulation not only validates the results of the real test tube but also illustrates the power of EdnaCo’s VTT’s to provide very realistic results. Quantifying the success rate by performing the same number of experiments would be very costly in the wet lab where each experiment would cost at least several hundreds of dollars. Here, DNA computers in silico provided the solution to Adleman’s initial experiment in about 1200 iterations of the simulation. Moreover, [13] were able to show how to improve the efficiency of Adleman’s technique (which is essentially brute force and blindly attempts to build all possible paths (most of which will fail to be Hamiltonian) by introducing the concept of a fitness function. The fitness functions were later used to improve the efficiency of the simulation and suggest that protocols may exist to achieve similar improvements to Adleman’s experiment in vitro. More information on this experiment or the enhancements that derived from this validation can be found in [13]. In [25], a similar second validation, which uses the PCR protocol to selectively increase the concentration of specific DNA species, is reported. EdnaCo has proven itself equally useful and reliable in implementing PCR [20], DNA Chips [27], and Baum’s associative memory [2, 24].


265

8.4 Noncrosshybridizing Bases and PCR Selection The encoding problem E [15, 16] is defined as the problem of representing, as DNA structures, the data set D in a humanly understood language (such as English), or in the bits of conventional computers (although English will continue to be the running example.) An ideal representation would be unambiguous and fully available for retrieval. To be considered truly unambiguous, two requirements must be met. First, D must be encoded unambiguously to represent English words WEnglish (i.e. every WEnglish in DEnglish must have exactly one counterpart DNA codeword WDN A in DDN A and, likewise, every WDN A in DDN A must have exactly one counterpart WEnglish in DEnglish ). Second, DDN A must remain unambiguous over time and persist through any protocols (e.g. PCR). Encoding solutions are fully available when all strands in D can be queried and retrieved. Of specific concern are encoding solutions that cross-hybridize and thus would create ambiguity in the encoding (because queries may match similar DNA) or may hide information (two memory strands may hybridize and prevent retrieval.) Ideally, every WDN A in DDN A will not hybridize to itself or any other WDN A in DDN A and complementary WDN A in the complementary set of DDN A will not hybridize to itself or any other complementary WDN A in the complementary set of DDN A . An ideal solution to the encoding problem will produce an ideal data representation efficiently in a manner that is both scalable and reversible. First, encoding solutions are expected to represent data with all the characteristics of DDN A expected of an ideal solution (as described in Section 8.2). The encoding process itself is expected to be reversible (i.e. a data set DEnglish is encoded as DDN A such that DEnglish can be reconstructed exactly from the reverse encoding DDN A to DEnglish ). The complexity of the encoding algorithm must be low and ideally linear to the size of D. For example, DEnglish or DBytes with n elements should require at most n steps to produce DDN A . The solution E must be scalable to encode very large amounts of data and could ideally encode DDN A from any size D). The solution that is generally considered to be ideal is substitution of words and phrases with a codeword that captures the significance of that word or phrase. In DNA memories, a codeword is a ssDNA (single-stranded DNA) that unambiguously represents bytes or a language construct (e.g. word, phrase or concept). For example, Adleman hashed graph vertices and edges unambiguously into DNA codewords and demonstrated that his model was feasible in vitro by producing a solution to the Hamiltonian Path Problem. This demonstration essentially proved that his representation was fully available over the full length of the experiment and unambiguous in its representation. This encoding solution requires a single pass over the text to substitute codewords with language constructs that completes in linear time. For example, n words require n substitutions to produce to DDN A and m phases

266


require m substitutions to produce DDN A In more general terms, any set of abstract concepts of size n can be represented in DNA after n substitutions of DNA for concept. By similar substitution of the language constructs with DNA words, the process can be reversed. The encoding solution is scalable to the size of the number of the codewords available. This solution is common and generally considered adequate. The critical shortcoming of this approach is the lack of available DNA to encode the words of the English language (currently estimated to be around a million by http://www.languagemonitor.com/) or word concepts expressed in WordNet as approximately 207,000 different meanings used worldwide and expressed by different words (http://wordnet.princeton.edu/man/wnstats.7WN). Finding DNA sets of sufficient quantity and quality for even subsets of these databases is the so-called word design problem. Its solution has motivated a decade long search for an optimal set of codewords [5, 20, 15, 18, 3, 13, 10, 16]. As pointed out in [20], relatively small DNA strands of 20-mers could easily represent 1 terabyte of data if just one byte corresponded to one 20-mer. If representing abiotic data in the form of words and phrases, or more conceptual ideas in the form of word meanings or word relationships, the potential exists to realize Baum’s [2] first estimates of exceeding the capacity of the human brain. Because of the importance of the encoding problem for the entire field, there has been many efforts to find good codeword designs since early days. Surveys can be found in [15, 11, 12, 16] The ideal approach is to use DNA-based computing to solve the problem because the results is largely independent of Gibbs energy models and likely to yield optimal performance in vitro. The resulting PCR Selection (PCRS) [3, 10] protocol was designed to take advantage of PCR’s capabilities to select, amplify DNA, and obtain noninteracting (referred to below as noncrosshybridizing, for short nxh) codewords. The protocol uses PCR to selectively amplify DNA in a test tube and then selectively separate the amplified DNA from the rest of the test tube. Step one of PCRS is initialized with a seed set of dsDNA D1 that is placed into a test tube at a low temperature. Each D1 is initialized as a pair of DNA with universal, unchanging primers attached such that P1 attached at the 5 end and primer P2 attached at 3 end. This protocol begins by heating the test tube to a temperature warm enough to melt less all DNA, then quickly cooling it to allow more complementary DNA to re-hybridized. The protocol continues by amplifying the melted ssDNA (the more nxh DNA in the test tube) by PCR. Amplification [23] of the nxh DNA only occurs because the primers, inserted to initiate PCR extension, will not hybridize to the dsDNA (the more stable and more complementary DNA in the test tube.) The capability of PCRS to identify nxh subsets was evaluated experimentally in [6, 7]. The validation began with an initial DNA set of DNA seeded with the full set of random 20-mers. The primers, P1 and P2 , were nxh 20mers and were excluded from the seed set. Figure 1.1 in cite14 shows the templates (red) in the top row (centered above each gel) with each primer (black)


267

attached at the 5 and 3 ends. The first template was fully complementary while the last was nxh. Two other templates were selected as intermediate steps between fully complementary and nxh. The template sequences were designed using an in silico software tool [10] that selects nxh DNA from an initially random pool. In a first round of experiments, PCR was performed on each of the four templates representative of the spectrum of conditions between fully crosshybridizing and nxh extremes. The test tube was heated to 52o C, 58o C, 64o C, 70o C, and 74o C (centigrade) and PCR extension was allowed to run for 1 round that lasted one hour. Each template was incubated in a PCR buffer of 50 mM KCI, 10 mM Tris-HCI, 0.1% Triton X-100, 2.5mM MgCl2 , 0.4 nM 4 dNTP, and 4 U Taq DNA polymerase in total 10ul volume. The results of the 20 experiments (5 temperatures for each of the four species) were then placed into a denature gel at 400V. Denaturing was allowed to run for one hour before being captured by autoradiography. In the resulting gels, amplification occurred at 52o C, 58o C, and 64o C for all species. At temperatures at or above 70o C, very little amplification occurred for the three nearly crosshybridizing templates. However, amplification for the nxh DNA templates occurred at all temperatures with maximum yield at 52o C. This result proves that PCRS can selectively amplify nxh DNA from a seed set and eventually extract a maximal subset. This experiment was repeated a second time to determine the ideal conditions for nxh amplification. In this experiment, only the maximally similar and maximally dissimilar templates were used. The range of temperatures included 37o C, 40o C, 43o C, 46o C, 48o C, 50o C, 56o C, 62o C, 68o C, and 72o C. It was determined that no amplification occurs at 43o C when the templates are complementary. However, plenty of amplification occurs when the temperature was 43o C and below. This range of temperatures allows PCRS to operate efficiently. The results of PCRS were evaluated and the nxh quality of each set was confirmed to be very high in [7]. PCRS was again performed to produce a set of template species. The nxh quality was then evaluated by spectrophotometric quantification, a method that measures optical density by spectrography. The critical property being exploited is that UV light absorption at 260nm is less for ssDNA than for dsDNA. By melting the DNA results fully, a spectrophotometer can measure the amount of light absorbed by the ssDNA. Thus, a rough census of nxh to crosshybridizing DNA can be taken over time. As the test tube cools, hybridization will occur naturally only if the DNA can form energetically stable bonds. The measurement of this concentration of dsDNA over time curve is called the CoT curve. (for Concentration-Time.) A steep decline in the CoT indicates the DNA is very crosshybridzing while a flatter CoT curve indicates the DNA is more nxh. PCRS is ideal for creating nxh sets. Because PCRS is massively parallel in nature, it is maximally efficient in vitro. As a result, the time of completion for a single round of PCRS is in the order of minutes to hours. The product

268


set is maximally nxh according to the natural process of DNA hybridization and not an approximation of hybridization applied in other approaches. The characterization of several product sets has been given in [7]. The remaining challenge thus becomes to discover a scalable solution that can extract the codeword sequences from these nxh product sets. Even if one is willing to bear the cost, in terms of money, of sequencing a sample of the code set in vitro, it remains forbiddingly infeasible in terms of time to sequence the whole product set of 20-mers, estimated to be 10,000 - 50,000 species [6]. In order to get useful information on the composition of such nxh set, PCRS has been implemented in silico with EdnaCo. In simulation, a model is required to of decide whether two strands hybridize. An approximation of Gibbs energy that has proved to provide very good results up to 60-mers can be used [8]. Scalability is still a challenge due to the sheer size of input sets and the number of rounds needed to refine the codeword set. The advantage is that the codeword set will be fully sequenced and of comparable quality. We will discuss this solution in full details below. In [20], PCR selection (PCRS) was implemented in silico on EdnaCo to identify quality codeword sets. PCRS protocol was implemented exactly as it would be in vitro with one exception: all copies of a codeword were removed from the virtual test tube at the moment any copy of the codeword formed a duplex according to Gibbs energy. This variation significantly reduced the overhead and helped the apparent kinetic problems that emerged in vitro required to converge to an ideal nxh codeword set. The codeword product sets were evaluated by both size and quality. Quality was evaluated by three methods to assess the application of each set in vitro. The first method performed an O(n2 ) search of R = {Σr} to determine if any r would hybridize to any member of R and if any r (the complement of r) would hybridize to any member of R (the complement set of R). Because Gibbs energy evaluation is the primary criterion for hybridization in silico and is an excellent predictor of hybridization in vitro, this first method is an excellent measure of the nxh quality of R. The second method estimates the Gibbs energy of each pair of codewords. The third method characterizes each r in R as blocks of 2, 3, and 4-mers. PCRS was performed with various input sets of uniform length n where n = 5, 6, 7, 8, 10, 13, 16, and 20 [20]. No matter how efficiently represented in simulation, a seed set of length 20 contains at most 420 strands. The resources needed to perform PCRS with that much input are too high a computational and memory requirement even for supercomputers of today. This challenge motivated a pre-processing filtering that reduces the complexity and search space of the input DNA. The inputs to PCRS, called S, were filtered to minimize wasting simulation cycles. Each S was filtered to remove a priori those obvious candidate DNA species likely to be removed by PCR Selection. In every full n-mer DNA set, a sizable number of them is immediately available to be discarded as they are palindromes or contain hairpins (i.e., reverse WC complements of self or other strands in the set), contain k-mer runs (k > 3) of


269

G/Cs. The above filters were implemented on a full seed input of all n-mers. Where it was feasible, the input set was scanned to remove any hybridizing DNA that would crosshybridize even with very stringent hybridization thresholds according to the h-distance [17] which is less accurate but executes faster than the Gibbs approximation used by EdnaCo. Because scanning for crosshybridization was only possible over several months for 12-mers and 13-mers, a different pre-processing step was used for n > 13. The serialized filter was replaced by a distributed filter to do the work to PCRS running on EdnaCo (a parallel application). By removing these strands from the input sets, were able to reduce the complexity of these problems by orders of magnitude [20]. As shown in Figure 8.1, the net gain is a filtered set (middle bars) that is at least one order of magnitude smaller in size than the full set (left bars). The size of n-mer set seems to grow exponentially with a power law 40.72n+1.66 . The results and capabilities of PCRS to scale and produce large sets of nxh were first discussed in [20]. Below is a recap of the evidence presented in that paper that proves PCRS as a scalable solution capable of produce fully sequenced nxh codewords efficiently in a matter of hours for small examples and months for very large codeword sets. The next section addresses the issue of scalability and is followed by two evaluations of the nxh quality of the codeword product sets.

Log of Size (Base 4)

Size of Filtered Seed Sets for PCRS++ 20.00 18.00 16.00 14.00 12.00 10.00 8.00 6.00 4.00 2.00 0.00 4−

5−

6−

7−

8−

9−

10−

11−

12−

13−

14−

15−

16−

17−

18−

19−

20− mers

n-mers Pre-Filter

Filtered Set

pow(4,0.72*n+1.66)

Fig. 8.1. Pre-filtering of the space of n-mers reduces the full set by at least an order of magnitude. The filtering process removes from the full input set (left bars) the DNA species that contain hairpins or short runs of the same base and those species that are palindromes or are too much like other strands already passed by the filter. The size of the resulting filtered set (middle bars) can be estimated by the power law 40.72n+1.66 (right bars). [20]

Figure 8.2 compares the size of filtered n-mer input set (dark blue) is compared with the PCRS product set (red) and an ideal PCRS product set (black). Due to the sheer complexity of the task, the experiments for 20-mers were not completed after 10 months into the simulation. The growth in size

270


for n-mer sets in range from 5-10 was calculated to be 4.19sqrt((n)+0.3) . By extrapolation, the sizes of 20-mer nxh sets are estimated to be around 416 or 4.7 billion. The set of 16-mers began with 413.23474901 or about 93 million DNA polymers. PCRS produced a set of about 48 , which was reduced to about 47 or 16KDNA polymers by additional exhaustive filtering.

Log of Size (Base 4)

Size of Code Sets 20 18 16 14 12 10 8 6 4 2 0 4−

5−

6−

7−

8−

9−

10−

11−

12−

13−

14−

15−

16−

17−

18−

19−

20− mers

n-mers Filtered Input to PCRS++

PCRS++ Product

Ideal product after post-filtering

pow(4,1.9*sqrt(n)+0.3)

Fig. 8.2. Size of the PCR Selection product obtained from the filtered input sets (left bars). The size of the resulting set (middle bars) can be estimated by the sub-power law 41.9pn+0.6 (light bars on the right). [20]

[20] obtained very large sets of very good nxh quality that are guaranteed to perform well under a variety of reaction conditions in wet test tubes. The method is constructive, not in the sense that it selects the actual wet strands as the PCR Selection protocol does in vitro [10], but in the complementary and more advantageous sense that the sequence and composition of the actual codewords is known, thereby bypassing a costly or impossible sequencing procedure. The method is also nearly optimal, in the sense that the size of the codeset obtained is in the order of magnitude of the theoretical maximum size of a set for the given hybridization stringency under which the simulation was conducted. The net result is that there is in hand a number of codeword sets of n-mers for values of n = 4-10, 13-,16- and 20mers. More importantly, we have the protocol that is capable of producing far larger codeword sets. The analysis for some of these sets has been presented, for example, G/C-content analysis and a bias analysis of the most frequent k-mer-blocks for k = 2, 3, 4. These results suggest further analyses that could be conducted with the products of the protocol in vitro to obtain further characterization of its products (so far only conducted in vitro for 20-mers [7]). These PCRS experiments (including the analysis) have lasted over 10 months and are expected to bear full fruit up to 20-mers (once resources become available). PCRS can easily be scaled to sets of longer polymers with feasible (but long) run times up to about 60-mers when the Gibbs energy model used begins to fail. At that point, optimality


271

will be out of range. Given the very encouraging results of PCRS [20], tensor products [15], and shuffle codes [29], it may be necessary to apply tensor product and shuffle codes as filters to PCRS as the size of the n-mers increase much beyond 20-mers. [20] argued that the size of these codeword sets may be in the order of magnitude of maximal sets and provided a good practical estimate of the capacity of a DNA memory [20] based on Baum’s construction [2]. With these codes, we have the support required to implement virtually any kind of application of biomolecular computing, and in particular, the primary ingredient for DNA memories [25, 18, 25, 33].

8.5 Some Applications of DNA Memories In this Section, we summarize a few applications of DNA memories capable of retrieving both biotic and abiotic data under development or consideration. Given a nxh basis B of N oligos of length n, arbitrary data sets can be represented by a so-called signature. [18, 5] define the representation as a signature as follows. Without loss of generality, it will be assumed that strings x to be encoded are written in a four letter alphabet {a,c,g,t}. The signature of a string x with respect to B is a vector V of size N obtained by shredding. x to fragments of size n or less and allowing the fragments to hybridize under saturation conditions to the oligos in B. The vector V can be visualized as a 2D matrix by arranging the basis strands in a 2D DNA-chip, with a fixed common number of basis strand per spot. The vector V may appear not to be well-defined, since it is clear that its calculation depends on the concentration of basis as well a strands x used in an experiment to compute it. To avoid these difficulties, this situation is idealized to the case where only the same number of copies (say, a fixed large number of copies) of each basis strand is present in the tube, that the target input x is poured in saturation concentrations over the chip, and that the tube is small enough that all possible hybridization occur within reasonable time. These idealizations are supported by a number experiments performed on EdnaCo that essentially normalize the representation and make it unique to within small variation in intensity for each pixel, but with basically the same pattern over the entire chip. Precise details can be found in [18] An obvious and very intriguing application of the type of DNA memories described above is in storage and retrieval of genomic data of biological organisms. One cannot help but wonder what type of signatures given organisms may produce and how they compare to one another. These issues have been explored in several cases, of which we mention three, two with genomic data and one with abiotic/textual data. Precise details can be found in [20, 26, 27, 25] In the first application, a number of plasmids of lengths varying between 3Kbp (kilobase pairs) and 4.5Kbps were shredded to fragments of size 40 or

272


less and thrown into a tube containing a basis B of three different nxh quality consisting of 36-mers. The each member of the basis B was permanently fixed to a spot on a thin piece of glass. Their signature were approximated by simulation on EdnaCo by a simple process described above, i.e. measuring the degree in which the shredded target DNA interacted with the DNA on the chip. The variability of the signatures was telling of the nature and origin of the plasmids. More nxh bases produced signatures with less variability and noise. As the noise decreased, the size of the basis could be reduced so that the same result could be achieved with far fewer nxh DNA and far smaller DNA chips. On a high-quality nxh basis, the plasmid signatures appeared were just as different as the organisms that originated them. This application shows how genetic data could be captured by merely measuring hybridization affinity to an nxh set of DNA. In a second application, the DNA chip can reveal emergent information from a set of shredded DNA that the shredded DNA itself became the DNA memory. The shredded DNA is placed into a test tube. PCR selection protocol is applied to identify an nxh subset, and the resulting species become the DNA memory placed onto a DNA chip. When the shredded DNA is allowed to interact with the DNA chip, the resulting signature is far sharper and crisper. By performing this process on many such species, it is possible to create a single, re-usable DNA chip capable of identifying the origin species of any DNA sample. Precise details can be found in [20]. Furthermore, this same protocol can be extended to very different data sets. For example, the shredded DNA may be replaced with English words from a Shakespearian play encoded in DNA structures. PCR selection protocol could then produce a meaning subset of words (arguably the most meaningful subset of words) which could then be placed on a DNA chip. The application of this chip could assist in natural language processing problems of question answering and information summarization. Precise details of this protocol can be found [26, 27] (for genetic data) and [25] (for abiotic data). In the third application, DNA memories are used to reveal the meaning of human language in applications such as recognizing textual entailment (RTE). Here, the capacity potential of DNA to retrieve fine-grained information from vast amounts of data could be truly leveraged. Data representing the full complement of word-phrase meanings is stored into a DNA memory. This potential has been documented in the case of textual entailment challenge (RTE), A typical instance of RTE consists of two paragraphs (one called the text and the other called the hypothesis). The problem is to decide whether the hypothesis is entailed by the text. This is not a problem about logical inference, but rather a problem that requires bringing into play background knowledge about the world, usually left implicit in the text. Using DNA memory representations of semantic networks such as WordNet, entailment can be solved automatically by disambiguating the words and phrases of each paragraph to meanings in the DNA memory and then examining the overlap of the corresponding semantic concepts contained in both text and hypothesis. This application shows how DNA memories might be able to capture semantical knowledge in superior


273

ways to that standard syntactic and lexical representations. More details can be found in [25, 27]. In conclusion, we mention an important advantage on this type of DNA memory. Information retrieval normally requires the identification of the relevant data, retrieval of the data from the DNA-based memory, and further processing of the desired information. The critical issue is how to “identify the correct data to retrieve”. The simplest and most frequent conventional approach is to “let the human decide” which data to retrieve. For example, conventional computers ask the user to search for information by opening a series of files on a file system, reading each file, and deciding from the content which file is correct. The underlying system may provide tools to help the human search by iterating through a list of files searching for keywords. This “key-word” search approach will miss files and content that express similar concepts with words different than the keywords. Ideally, data retrieval tools could not only retrieve lexically (by matching key words); but also retrieved by matching meanings of words captured in the query to meanings of words captured in the data in the memory. By contrast, the DNA memories discussed here do not require an intelligent, outside “brain” to extract the relevant features from the given data. Hybridization affinity naturally selects the relevant features in the input and display them on the DNA chip signature as a 2D graphical and semantic representation.

References 1. L.M. Adleman. Molecular computation of solutions to combinatorial problems. Science, 266:1021, 1994. 2. E. Baum. Building an associative memory vastly larger than the brain. Science, 268:583–585, 1995. 3. H. Bi, J. Chen, R. Deaton, M. Garzon, H. Rubin, and D. Wood. A PCR based protocol for in vitro selection of non-crosshybridizing oligonucleotides. Journal of Natural Computing, 2(4):461–477, 2003. 4. BES Computers Blog. Hitachi 1tb hard drive scores popular mechanics editors choice award. http://blog.bescomputers.net/2007/01/20/hitachi-1tb-harddrive-scores-popular-mechanics-editors-choice-award-ces/, 2007. 5. K. Bobba, A. Neel, V. Phan, and M. Garzon. Reasoning and talking DNA: Can DNA understand English? In Proceedings of the 12th International Meeting on DNA Computing. Springer. 6. J. Chen, R. Deaton, M. Garzon, J.W. Kim, D.H. Wood, H. Bi, D. Carpenter, J.S. Lee, and Y.Z. Wang. Sequence complexity of large libraries of DNA oligonucleotides. In N. Pierce and A. Carbone, editors, DNA Computing: Preliminary Proceedings of the 11th International Workshop on DNA-Based Computers. 7. J. Chen, R. Deaton, M. Garzon, J.W. Kim, D.H. Wood, H. Bi, D. Carpenter, and Y.Z. Wang. Characterization of noncrosshybridizing DNA oligonucleotides manufactured in vitro. Journal of Natural Computing, 1567. 8. R. Deaton, J. Chen, H. Bi, M. Garzon, H. Rubin, and D.H. Wood. A PCRbased protocol for in vitro selection of non-crosshybridzing oligonucleotides. In M. Hagiya and A. Ohuchi, editors, DNA Computing VIII.

274


9. R. Deaton, R. Murphy, M. Garzon, D. Franceschetti, and Jr.S. Stevens. Good encodings for DNA-based solutions to combinatorial problems. In Proceeedings of the Second Annual DIMACS Workshop on DNA Based Computers, 1996. 10. R.J. Deaton, J. Chen, H. Bi, and J.A. Rose. A software tool for generating noncrosshybridizing libraries of DNA oligonucleotides. In M. Hagiya and A. Ohuchi, editors, Proceedings of the 8th International Meeting of on DNA-Based Computers. 11. R.M. Dirks, M. Lin, E. Winfree, and A.N. Pierce. Paradigms for nucleic acid design. Nucleic Acids Res., 32:1392–1403, 2004. 12. U. Feldkamp and H. Ruahue W. Banzhaf. Software tools for sequence design. Genetic Programming and Evolvable Machines, 4:153–171, 2003. 13. M. Garzon, D. Blain, K. Bobba, A. Neel, and M. West. Self-assembly of DNAlike structures in silico. Journal of Genetic Programming and Evolvable Machines, 4:185–200, 2003. 14. M. Garzon, D. Blain, and A. Neel. Virtual test tubes. Journal of Natural Computing, 3(4):461–477, 2004. 15. M. Garzon and R. Deaton. Codeword design and information encoding in DNA ensembles. Journal of Natural Computing, 3(33):253–292, 2004. 16. M. Garzon, R. Deaton, P. Neathery, D.R. Franceschetti, and R.C. Murphy. A new metric for DNA computing. In Proceedings of the 2nd Genetic Programming Conference. Morgan Kaufman. 17. M. Garzon, R. Deaton, P. Neathery, R.C. Murphy, D.R. Franceschetti, and Jr.E. Stevens. On the encoding problem for DNA computing. In Preliminary Proceedings of the Third DIMACS Workshop on DNA-based Computing, pages 230–237, University of Pennsylvania, 1997. 18. M. Garzon, A. Neel, and K. Bobba. Efficiency and reliability of semantic retrieval in DNA-based memories. In Proceedings of DNA Based Computing 2003. Spinger. 19. M. Garzon and C. Oehmen. Biomolecular computation on virtual test tubes. In N. Jonoska and N. Seeman, editors, Proceedings of the 7th International Workshop on DNA-based Computers, 2001. 20. M. Garzon, V. Phan, S. Roy, and A. Neel. In search of optimal codes for DNA computing. In Proceedings of DNA Computing, 12th International Meeting on DNA Computing. Springer. 21. M.H. Garzon and R.J. Deaton. Biomolecular computing and programming: A definition. Kunstliche Intelligenz, 1. 22. M.H. Garzon and R.J. Deaton. Biomolecular computing and programming. IEEE Transactions on Evolutionary Computation, 3(3):36–50, 1999. 23. K.B. Mullis. The unusual origin of the polymerase chain reaction. Scientific American, 262(4):56–61, 64–65, 2001. 24. A. Neel and M. Garzon. Efficiency and reliability of genomic information storage and retrieval in DNA-based memories with compaction. In IEEE Congress for Evolutionary Computation 2003, pages 2733–2739, 2003. 25. A. Neel and M. Garzon. Semantic retrieval in DNA memories with gibbs energy models. Biotechnology Progress, 22(1):86–90, 2006. 26. A. Neel, M. Garzon, and P. Penumatsa. Improving the quality of semantic retrieval in DNA-based memories with learning. In Knowledge-Based Intelligent Information & Engineering Systems. KES-2004. Springer.


275

27. A. Neel, M. Garzon, and P. Penumatsa. Soundness and quality of semantic retrieval in DNA based memories with abiotic data. In G. Greenwood and G.Fogel, editors, Proceedings of the IEEE Conference on Evolutionary Computation, CEC 2004. 28. E.M. Palmer. Graphical Evolution. Wiley, New York, 1985. 29. V. Phan and M. Garzon. Information encoding using DNA. In Proceedings of the 10th International Conference on DNA Computing. Springer. 30. J. Roman. The Theory of Error-Correcting Codes. Springer, Berlin, 1995. 31. N.C. Seeman. DNA engineering and its application to nanotechnology. Trends in Biotechnology, 17:437–443, 1999. 32. E. Winfree, F. Liu, L.A. Wenzler, and N.C. Seeman. Design and self-assembly of two dimensional DNA crystals. Nature, 394:539–544, 1998. 33. B. Yurke and A.P. Mills. Using DNA to power nanostructures. Genetic Programming and Evolvable Machines, 4:111–112, 2003.

New developments in formal languages and applications

Recent Advances in Formal Languages and Applications

Formal Languages

New Developments and Applications in Sensing Technology

Formal Languages and Compilation

New Trends in Formal Languages - Control, Cooperation, and Combinatorics

New Developments and New Applications in Animal Cell Technology

Developments and Applications in Solubility

Formal Models, Languages And Applications (Machine Perception and Artifical Intelligence)

Formal groups and applications

New Developments in Distributed Applications and Interoperable Systems

Multiple Sensorial Media Advances and Applications: New Developments in Mulsemedia

Visual Languages and Applications

Models of Computation and Formal Languages

Larch: Languages and Tools for Formal Specification

Formal languages and their relation to automata

Automata, Formal Languages and Algebraic Systems

An introduction to formal languages and automata

Formal languages and their relation to automata

Modern Formal Methods and Applications

Formal groups and applications MAtg

Automata, Formal Languages and Algebraic Systems

Elementary Computability, Formal Languages, and Automata

Formal Languages and Compilation (2nd edition)

Automata, Formal Languages and Algebraic Systems

Formal Languages and Their Relation to Automata

An Introduction to Formal Languages and Automata

New developments in parsing technology

New Developments in Differential Equations

New Developments in Biomedical Engineering

New Developments in Advanced Welding

New developments in formal languages and applications

Recent Advances in Formal Languages and Applications

Formal Languages

New Developments and Applications in Sensing Technology

Formal Languages and Compilation

New Trends in Formal Languages - Control, Cooperation, and Combinatorics

New Developments and New Applications in Animal Cell Technology

Developments and Applications in Solubility

Formal Models, Languages And Applications (Machine Perception and Artifical Intelligence)

Formal groups and applications

New Developments in Distributed Applications and Interoperable Systems

Multiple Sensorial Media Advances and Applications: New Developments in Mulsemedia

Visual Languages and Applications

Models of Computation and Formal Languages

Larch: Languages and Tools for Formal Specification

Formal languages and their relation to automata

Automata, Formal Languages and Algebraic Systems

An introduction to formal languages and automata

Formal languages and their relation to automata

Modern Formal Methods and Applications

Formal groups and applications MAtg

Automata, Formal Languages and Algebraic Systems

Elementary Computability, Formal Languages, and Automata

Formal Languages and Compilation (2nd edition)

Automata, Formal Languages and Algebraic Systems

Formal Languages and Their Relation to Automata

An Introduction to Formal Languages and Automata

New developments in parsing technology

New Developments in Differential Equations

New Developments in Biomedical Engineering

New Developments in Advanced Welding

Recommend Documents