Grammatical Inference: Algorithms and Applications: 5th International Colloquium, ICGI 2000, Lisbon, Portugal, September 11-13, 2000 Proceedings

Lecture Notes in Artificial Intelligence Subseries of Lecture Notes in Computer Science Edited by J. G. Carbonell and J...

Author: Arlindo L. Oliveira

71 downloads 619 Views 3MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Lecture Notes in Artificial Intelligence Subseries of Lecture Notes in Computer Science Edited by J. G. Carbonell and J. Siekmann

Lecture Notes in Computer Science Edited by G. Goos, J. Hartmanis and J. van Leeuwen

1891

¿ Berlin Heidelberg New York Barcelona Hong Kong London Milan Paris Singapore Tokyo

Arlindo L. Oliveira (Ed.)

Grammatical Inference: Algorithms and Applications 5th International Colloquium, ICGI 2000 Lisbon, Portugal, September 11-13, 2000 Proceedings

½¿

Series Editors Jaime G. Carbonell,Carnegie Mellon University, Pittsburgh, PA, USA Jörg Siekmann, University of Saarland, Saarbrücken, Germany

Volume Editor Arlindo L. Oliveira INESC / IST R. Alves Redol 9 1000 Lisbon, Portugal E-mail: [email protected] Cataloging-in-Publication Data applied for Die Deutsche Bibliothek - CIP-Einheitsaufnahme Grammatical inference: algorithms and applications : 5th international colloquium ; proceedings / ICGI 2000, Lisbon, Portugal, September 11 13, 200. Arlando L. Oliveira (ed.). - Berlin ; Heidelberg ; New York ; Barcelona ; HongKong ; London ; Milan ; Paris ; Singapore ; Tokyo : Springer, 2000 (Lecture notes in computer science ; Vol. 1891 : Lecture notes in artificial intelligence) ISBN 3-540-41011-2

CR Subject Classification (1998): I.2, F.4.2-3, I.5.1, I.5.4, J.5, F.2 ISBN 3-540-41011-2 Springer-Verlag Berlin Heidelberg New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag. Violations are liable for prosecution under the German Copyright Law. Springer-Verlag Berlin Heidelberg New York a member of BertelsmannSpringer Science+Business Media GmbH c Springer-Verlag Berlin Heidelberg 2000 Printed in Germany Typesetting: Camera-ready by author, data conversion by PTP-Berlin, Stefan Sossna Printed on acid-free paper SPIN 10722523 06/3142 543210

Preface

The Fifth International Colloquium on Grammatical Inference (ICGI-2000) was held in Lisbon on September 11–13th, 2000. ICGI-2000 was the fifth in a series of successful biennial international conferences in the area of grammatical inference. Previous conferences were held in Essex, U.K.; Alicante, Spain; Montpellier, France; and Ames, Iowa, USA. This series of meetings seeks to provide a forum for the presentation and discussion of original research on all aspects of grammatical inference. Grammatical inference, the process of inferring grammar from given data, is a field that is not only challenging from a purely scientific standpoint but also finds many applications in real world problems. Despite the fact that grammatical inference addresses problems in a relatively narrow area, it uses techniques from many domains, and intersects a number of different disciplines. Researchers in grammatical inference come from fields as diverse as machine learning, theoretical computer science, computational linguistics, pattern recognition and artificial neural networks. From a practical standpoint, applications in areas such as natural language acquisition, computational biology, structural pattern recognition, information retrieval, text processing and adaptive intelligent agents have either been demonstrated or proposed in the literature. ICGI-2000 was held jointly with CoNLL-2000, the Computational Natural Language Learning Workshop and LLL-2000, the Second Learning Language in Logic Workshop. The technical program included the presentation of 24 accepted papers (out of 35 submitted) as well as joint sessions with CoNLL and LLL. A tutorial program organized by Gabriel Pereira Lopes took place after the meetings and included tutorials by Raymond Mooney, Gregory Grefenstette, Walter Daelemans, Ant´ onio Ribeiro, Joaquim Ferreira da Silva, Gael Dias, Nuno Marques, Vitor Rossio, Jo˜ ao Balsa and Alexandre Agostini. The joint realization of these events represents a unique opportunity for researchers in these related fields to interact and exchange ideas. I would like to thank Claire Nédellec, Claire Cardie, Walter Daelemans, Colin de la Higuera and Vasant Honavar for their help in several aspects of the organization; the members of the technical program committee and the reviewers for their careful evaluation of the submissions; the members of the local organizing committee, Ana Teresa Freitas and Ana Fred, for their help in setting up the event; and Ana de Jesus for her invaluable secretarial support.

September 2000

Arlindo Oliveira Technical Program Chair

Technical Program Committee Pieter Adriaans Michael Brent Walter Daelemans Pierre Dupont Dominique Estival Ana Fred Jerry Feldman Lee Giles Colin de la Higuera Vasant Honavar Laurent Miclet G. Nagaraja Jacques Nicolas Arlindo Oliveira Jose Oncina Carratala Rajesh Parekh Lenny Pitt Yasubumi Sakakibara Arun Sharma Giora Slutzki Esko Ukkonen Stefan Wermter Enrique Vidal Thomas Zeugmann

Syllogic/University of Amsterdam, The Netherlands Johns Hopkins University, USA Tilburg University, The Netherlands University de St. Etienne, France Syrinx Speech Systems, Australia Lisbon Technical University, Portugal ICSI, Berkeley, USA NEC Research Institute, USA EURISE, University de St. Etienne, France Iowa State University, USA ENSSAT, France Indian Institute of Technology, India IRISA, France INESC/IST, Portugal Universidade de Alicante, Spain Allstate Research and Planning Center, USA University of Illinois at Urbana-Champaign, USA Tokyo Denki University, Japan University of New South Wales, Australia Iowa State University, USA University of Helsinki, Finland University of Sunderland, UK University Politecnica de Valencia, Spain Kyushu University, Japan

Organizing Committee Conference Chair: Tutorials: Local Arrangements: Social Program: Secretariat:

Additional Reviewers Daniel Gildea Mitch Harris Satoshi Kobayashi Eric Martin Franck Thollard Takashi Yokomori

Arlindo Oliveira, INESC/IST Gabriel Pereira Lopes, Universidade Nova de Lisboa Ana Fred, Lisbon Technical University Ana Teresa Freitas, INESC/IST Ana de Jesus, INESC

Table of Contents

Inference of Finite-State Transducers by Using Regular Grammars and Morphisms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Francisco Casacuberta

1

Computational Complexity of Problems on Probabilistic Grammars and Transducers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Francisco Casacuberta and Colin de la Higuera Efficient Ambiguity Detection in C-NFA, a Step Towards the Inference of Non Deterministic Automata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Fran¸cois Coste and Daniel Fredouille Learning Regular Languages Using Non Deterministic Finite Automata . . . 39 Fran¸cois Denis, Aurélien Lemay, and Alain Terlutte Smoothing Probabilistic Automata: An Error-Correcting Approach . . . . . . . 51 Pierre Dupont and Juan-Carlos Amengual Inferring Subclasses of Contextual Languages . . . . . . . . . . . . . . . . . . . . . . . . . . 65 J.D. Emerald, K.G. Subramanian, and D.G. Thomas Permutations and Control Sets for Learning Non-regular Language Families 75 Henning Fernau and José M. Sempere On the Complexity of Consistent Identification of Some Classes of Structure Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 Christophe Costa Florêncio Computation of Substring Probabilities in Stochastic Grammars . . . . . . . . . 103 Ana L. N. Fred A Comparative Study of Two Algorithms for Automata Identification . . . . 115 P. Garc´ıa, A. Cano, and J. Ruiz The Induction of Temporal Grammatical Rules from Multivariate Time Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 Gabriela Guimar˜ aes Identification in the Limit with Probability One of Stochastic Deterministic Finite Automata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 Colin de la Higuera and Franck Thollard Iterated Transductions and Efficient Learning from Positive Data: A Unifying View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 Satoshi Kobayashi

VIII

Table of Contents

An Inverse Limit of Context-Free Grammars - A New Approach to Identifiability in the Limit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 Pavel, Martinek Synthesizing Context Free Grammars from Sample Strings Based on Inductive CYK Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 Katsuhiko Nakamura and Takashi Ishiwata Combination of Estimation Algorithms and Grammatical Inference Techniques to Learn Stochastic Context-Free Grammars . . . . . . . . . . . . . . . . 196 Francisco Nevado, Joan-Andreu S´ anchez, and José-Miguel Bened´ı On the Relationship between Models for Learning in Helpful Environments 207 Rajesh Parekh and Vasant Honavar Probabilistic k-Testable Tree Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 Juan Ram´ on Rico-Juan, Jorge Calera-Rubio, and Rafael C. Carrasco Learning Context-Free Grammars from Partially Structured Examples . . . . 229 Yasubumi Sakakibara and Hidenori Muramatsu Identification of Tree Translation Rules from Examples . . . . . . . . . . . . . . . . . 241 Hiroshi Sakamoto, Hiroki Arimura, and Setsuo Arikawa Counting Extensional Differences in BC-Learning . . . . . . . . . . . . . . . . . . . . . . 256 Frank Stephan and Sebastiaan A. Terwijn Constructive Learning of Context-Free Languages with a Subpansive Tree . 270 Noriko Sugimoto, Takashi Toyoshima, Shinichi Shimozono, and Kouichi Hirata A Polynomial Time Learning Algorithm of Simple Deterministic Languages via Membership Queries and a Representative Sample . . . . . . . . 284 Yasuhiro Tajima and Etsuji Tomita Improve the Learning of Subsequential Transducers by Using Alignments and Dictionaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298 Juan Miguel Vilar Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313

Inference of Finite-State Transducers by Using Regular Grammars and Morphisms Francisco Casacuberta? Departamento de Sistemas Inform´ aticos y Computaci´ on, Instituto Tecnol´ ogico de Inform´ atica, Universidad Politécnica de Valencia, 46071 Valencia, Spain. [email protected]

Abstract. A technique to infer finite-state transducers is proposed in this work. This technique is based on the formal relations between finitestate transducers and regular grammars. The technique consists of: 1) building a corpus of training strings from the corpus of training pairs; 2) inferring a regular grammar and 3) transforming the grammar into a finite-state transducer. The proposed method was assessed through a series of experiments within the framework of the EuTrans project.

1

Introduction

Formal transducers [8] give rise to an important framework in syntactic pattern recognition [20]. Many tasks in automatic speech recognition can be viewed as simple translations from acoustic sequences to sub-lexical or lexical sequences (acoustic-phonetic decoding) or from acoustic or lexical sequences to sequences of commands to a data-base management system or to a robot (semantic decoding). Other similar application is the recognition of continuous handwritten characters. Other more complex applications of formal transducers are language translations (e.g. English to Spanish) [21,1] from text to text, from speech to text or speech [1] or from continuous handwritten characters to text, etc. Regular transductions [2] constitute an important class within the formal translation field. Regular transduction involves regular or finite-state machines to deal with the input and output languages that are defined in a formal translation. Even though these translations are much more limited than other more powerful ones, the computational costs of the algorithms that are needed to deal with them are much lower. One of the most important interest in finite-state machines for translation comes from the fact that these machines can be learned automatically from examples [20]. However, there are few techniques that infer finite-state transducers [16,14,19,12]. Nevertheless, there is an important number of techniques to infer regular grammars from finite sets of learning strings that have been used successfully in automatic speech recognition [20]. Some of these techniques ?

This work has been partially funded by the European Union under grant IT-LTROS-30268.

A.L. Oliveira (Ed.): ICGI 2000, LNAI 1891, pp. 1–14, 2000. c Springer-Verlag Berlin Heidelberg 2000

2

F. Casacuberta

are based on results from formal language theory. In particular, complex regular grammars can be built by inferring simple grammars that recognize local languages [9]. A finite-state transducer or a regular syntax-directed translation scheme, T , is a tuple < N, Σ, ∆, R, S >, where N is a finite set of non-terminal symbols or states, S is the initial state, Σ is a finite set of input terminal symbols, ∆ is a finite set of output terminal symbols (Σ ∩ ∆ = ∅) and R is a set of rules of the form A → aB, ωB or A → a, ω for A, B ∈ N , a ∈ Σ, ω ∈ ∆? . A pair (x, y) ∈ Σ ? × ∆? is a translation pair if there is a translation form t(x, y) in T : t(x, y) : (S, S) → (x1 A1 , y1 A1 ) → (x1 x2 A2 , y1 y2 A2 ) → ... → (x, y) A regular translation is the set of all translation pairs for some finite-state transducer. A stochastic finite-state transducer, TP , is a tuple < N, Σ, ∆, R, S, P >, where N , Σ, ∆, R, S are defined as above and P : R → a(j) and for j 00 : j ≤ j 00 ≤ j + l, a(j 00 ) ≤ a(j) zi =  otherwise xi The procedure of alignment consists of an assignment of each word from y to the corresponding word from x given by the alignment a if the output order is not violated. In the other case, the output word is assigned to the first input word that does not violate the output order. The set of extended symbols is a subset of Σ ∪ (Σ + D), where D ⊂ ∆? is the finite set of output substrings that appear in the training data.

Inference of Finite-State Transducers

7

The application of the procedure to the example gives in section 2.1: he+I+have hecho+made la+a reserva+reservation de+for una+a habitaci´ on doble+double+room con+with+a tel´ efono+telephone y+and+a televisi´ on+tv a nombre+for de Rosario+Rosario Cabedo+Cabedo .+. The extended symbols obtained by using this procedure are called extended symbols of type I. Obviously, other assignments are possible. For example, after the application of the above procedure, consecutive isolated input words (without any output symbol) can be joined to the first extended symbol with an assigned output symbol: Let x, y and a be an input string, an output string and an alignment function, respectively, and let z be the labelled string obtained from the above procedure. A new labelled string can be defined by the substitution of all the substrings zk zk+1 ... zk+l by zk − zk+1 − ... − zk+l , if for 1 ≤ k, k + l ≤ |z| (“+” and “-” are symbols not in Σ or ∆): – zk0 ∈ Σ for all k 0 : k ≤ k 0 < k + l – zk−1 , zk+l 6∈Σ A procedure to implement this second approach is straightforward from the previous one. The application of this algorithm to the above example allows us to obtain: he+I+have hecho+made la+a reserva+reservation de+for una+a habitaci´ on-doble+double+room con+with+a tel´ efono+telephone y+and+a televisi´ on+tv a-nombre+for de-Rosario+Rosario Cabedo+Cabedo .+. The extended symbols obtained by the last method are called extended symbols of type II. In the example, the differences between both types of symbols are in “habitaci´ on doble+double+room” that becames “habitaci´ ondoble+double+room”, “a nombre+for” that becames “a-nombre+for”, and “de Rosario+Rosario” that becames “de-Rosario+Rosario”. In practice, many of these extended symbols define reasonable correspondences between pairs of input and output segments (substrings). Step 2: Inferring the regular grammar In the second step of the proposed procedure, a (stochastic) regular grammar is built from the strings which are built from first step. The so-called n-grams are particular cases of stochastic regular grammars that can be inferred from training samples with well known techniques [18]. These models represent the stochastic languages of strings of the forms x, whose probability is given by Pr(x) =

|x| Y i=1

Pr(xi |xi−n+1 , ..., xi−1 )

8

F. Casacuberta

where xk = $ if k ≤ 0 and $ is a symbol that is not in the alphabet. These probabilities can be estimated by counting the substrings of length n in a training set. In practice, substrings of length n that have not appeared in the training set can appear in the analysis of a new input string. To deal with this problem, smoothed n-grams are used and they are a type of combination of k-gram ∀k ≤ n [10]. Obviously, other possible grammatical inference techniques can be used. However, in this work, n-grams are used due to the availability of a good public tool-kit to build smoothed (back-off) models [18]. On the other hand, the efficiency of smoothed n-grams has been proven successfully in some areas such as language modeling [18,10]. Step 3: Building the finite-state transducer The process of transforming a grammar of extended symbols into a finite-state transducer is based on the application of the two morphisms: if a ∈ Σ and b1 , b2 , ...bk ∈ ∆, hΣ (a + b1 + b2 + ... + bk ) = a h∆ (a + b1 + b2 + ... + bk ) = b1 b2 ...bk The procedure consists in transforming a rule or transition of the inferred regular grammar A → a + b1 + b2 + ... + bk B where a ∈ Σ and b1 , b2 , ..., bk ∈ ∆ into a transition of the finite-state transducer A → aB, b1 b2 ...bk B This procedure is illustrated in Fig. 3. The probabilities associated to the transitions in the finite-state transducer are the same as those of the original stochastic regular grammar. "habitacion+room" 1

"una+a"

2

3

"habitacion" 4

"a"

6

"individual+single+room"

"nombre+for"

7

"de"

8

5

Grammar of extended symbols

"habitacion" / "room" 1

"una" / "a"

2

3

"habitacion" / "" 4

"a" / ""

"individual" / "single room"

6

"nombre" / "for"

7

"de" / ""

8

5

Finite-State Transducer

Fig. 3. An example of an inferred grammar of extended symbols, and the corresponding finite-state transducer obtained from the morphisms hΣ and h∆ .


9

When extended symbols of type II are used, rules of the form: A → a1 − a2 − ... − al + b1 + b2 + ... + bk B

a1 , ..., al ∈ Σ and b1 , ..., bk ∈ ∆

can appear in the inferred grammar. In this case, a grammar rule is transformed into a set of transitions of the finite-state transducer: A → a1 B1 , b1 b2 ...bk B1 B1 → a2 B2 , B2 ... Bl−2 → al−1 Bl−1 , Bl−1 Bl−1 → al B, B The probability associated to the first transition in the above transformation is the same as that of the original rule in the stochastic regular grammar. The probabilities of the rest of transitions are set to 1.0.

3

Experimental Results

Two tasks of different levels of difficulty were selected to assess the inference method proposed in the framework of the EuTrans project [11]: a SpanishEnglish task (EuTrans-I) and an Italian-English task (EuTrans-II). In all the experiments reported in this paper, the approximate stochastic translations of the input test strings were computed, and the word-error rate (WER) for the translations was used as an error criterium. The WER was computed as the minimum number of substitution, insertion and deletion operations that had to be performed in order to convert the string which was hypothesized by the translation system into a given reference word string [11]. 3.1

Results Using the Corpus EuTrans-I

A Spanish-English corpus was generated semi-automatically for the EuTrans-I task which is a subtask of the “Traveller Task” [21]. The domain of the corpus is a human-to-human communication situation at a reception desk of a hotel. A summary of the corpus used in the experiments is given in Table 1. Extended symbols of type I: These first experiments corresponded to the use of extended symbols of type I and smoothed (back-off) n-grams as stochastic regular grammars. The results are presented in Table 2. The smoothed n-grams were built using the CMU Statistical Language Modeling Toolkit [18] and were represented by a stochastic grammar [13]. The number of transitions did not correspond to the number of free parameters of the model, since the weights were computed from a combination of the probability transitions and back-off weights [13]. The number of free parameters was approximately three times the number of states.

10

F. Casacuberta Table 1. The EuTrans-I task [1]. Spanish English Train: Sentences 10,000 Words 97,131 99,292 Vocabulary 686 513 Test: Sentences 2,996 Words 35,023 35,590 Bigram Perplexity 8.6 5.2

Extended symbols of type I from the a-priori segmented training pairs: We were able to segment the parallel training corpus due to the existence of some punctuation marks and some special words ([11]) (a-priori segmentation). The idea was to apply the statistical alignments only in each pair of segments and not in the entire sentences. The segments were shorter than the whole sentences; therefore, the alignment probability distributions were better estimated than for whole sentences. Extended symbols were built from these alignments. The strings of extended symbols corresponding to the segments of the same original string pair were concatenated. The best result achieved was a WER of 10.2 % using smoothed five-grams. In this case, the results were slightly worse than the ones in Table 2. The main reason is that the corpus was generated semi-automatically and the statistical alignments in the whole sentence could capture the relation between words in a training pair quite well. Extended symbols of type II: New experiments were performed in order to test extended symbols of type II. The WER for n = 3 was 23.2%, a result which was worse than that achieved using extended symbols of type I. One possible cause for this result could be the size of the achieved finite-state transducers that were twice the size of the finite-state transducers which were obtained using extended symbols of type I. Consequently, the assigned probabilistic distributions were poorly estimated. Summary of the results with the corpus EuTrans-I: The best result achieved using the proposed technique in the EuTrans-I was a WER of 9.7%. This result was achieved by using single extended symbols built from alignments Table 2. Results with the standard corpus EuTrans-I. The regular models were smoothed n-grams for different values of n. The number of states and transitions of the transducer are also reported. n-grams 2 3 4 5

states transitions WER 2,911 34,106 13.2 13,309 133,791 10.3 33,245 300,843 9.7 66,655 592,721 9.8


11

which were defined from the output to the input strings and from four-grams as stochastic regular grammars. This result was as good as the ones achieved by other finite-state techniques (a WER of 8.3% using Omega [11] another technique to infer some types of finite-state transducers) in similar experimental conditions. However, a statistical template technique allowed us to achieve a WER of 4.4% [11]. A WER of 13.9% was achieved [11] using a statistical alignment model (similar to IBM Model 2 used for the alignments). 3.2

Results with the Corpus EuTrans-II

The EuTrans-II task consists of two corpora acquired in the EuTrans project [11]: an Italian-Spanish corpus and an Italian-English corpus, consisting of transcriptions of spoken dialogues within the framework of hotel reception desk person-to-person communications. A summary of the corpus (only from Italian to English) used in the experiments is given in Table 3. Table 3. The EuTrans-II task. Italian English Train: Sentences 3,038 Words 55,302 64,176 Vocabulary 2,459 1,712 Test: Sentences 300 Words 6,121 7,243 Bigram Perplexity 31 25

The same translation procedure and error criterium used for EuTrans-I were used for EuTrans-II. Extended symbols of type I: The first experiment performed with this corpus was similar to the first experiment reported for EuTrans-I. In this case, the best WER achieved was 43.0 % using smoothed bigrams. This result was worse than for EuTrans-I since the task was also more difficult (the perplexity of the first task was 5.2 and the perplexity of the second task was 25). Extended symbols of type I from the a-priori segmented training pairs: In this experiment, the training corpus was previously segmented as for EuTrans-I. The results are reported in Table 4. The results of this experiment were clearly better than the corresponding experiments with non-segmented training data. These experiments showed a behaviour which was opposite to the one for EuTrans-I. One possible reason is that this corpus was more spontaneous than the first one and, consequently, had a higher degree of variability. Moreover, the size of the training data was less than the corresponding data of EuTrans-I.

12

F. Casacuberta

Table 4. Results with the standard corpus EuTrans-II. The regular models were smoothed n-grams [18] for different values of n. The training set was segmented using some a-priori knowledge. The statistical alignments were constrained to be in each parallel segment. n-grams 2 3 4 5

states transitions WER 7,988 77,453 27.2 31,157 254,143 28.6 66,507 472,518 28.3 110,197 768,024 28.0

Extended symbols of type II: More experiments were carried out. One of them was designed to test extended symbols of type II and bigrams. The main results were a WER of 48.6% for segmented training and and a WER of 77.0% for non-segmented training. In all of these experiments, the results were clearly worse than the results using extended symbols of type I and segmented training data. Summary of the results using the corpus EuTrans-II: The best result achieved with the proposed technique in the EuTrans-II was a WER of 27.2%. This result was achieved by using extended symbols of type I and a-priori segmentation of the training pairs. A smoothed bigram was the best regular model. This result was one of the best among the others reported in [17]. The above statistical template technique achieved a WER of 25.1% and a WER of 61.0 % using the IBM Model 2.

4

Conclusions

The method proposed in this paper to infer stochastic finite-state transducers from stochastic regular grammars allowed us to achieve good results in two real translation problems with different levels of difficulty. The method seemed to work better than others when the training data was scarce. However, when the available training data was sufficient, the technique presented a behaviour which was similar to the other finite-state approaches. The reasons for the results achieved by this method are due to: 1. the method of transforming training pairs into strings of extended symbols: a) it was based on a statistical alignment model b) it preserved the order of the input string and the output string in each training pair. 2. the use of smoothed n-grams that were trained from extended symbols. These models proved that they can deal adequately with the problem of unseen strings in the training set.


13

This method could be improved by using more powerful statistical alignment models (for example, the so called IBM Model 3 and 4). Another way of improving this method could be by adding an accurate output language model to recover the possible output syntactic errors that can be produced in the translation process. Acknowledgements. The author wishes to thank the anonymous reviewers for their criticisms and suggestions.

References 1. J. C. Amengual, J. B. Bened´ı, F. Casacuberta, A. Casta˜ no, A. Castellanos, V.M.Jiménez, D. Llorens, A. Marzal, M. Pastor, F. Prat, E. Vidal and J. M. Vilar: The EuTrans-I speech translation system. To be published in Machine Translation, 2000. 2. J. Berstel: Transductions and context-free languages. B. G. Teubner Stuttgart, 1979. 3. P. Brown, J. Cocke, S. Della Pietra, V. Della Pietra, J. Jelinek, J. Lafferty, R. Mercer and P. Roossin: A statistical approach to machine translation, Computational Linguistics, Vol. 16, N. 2, pp. 79–85, 1990. 4. P.F. Brown, J.C. Lai and R.L. Mercer: Aligning sentences in parallel corpora, 29th Annual Meeting of the ACL, pp. 169–176, 1991. 5. P. Brown, S. Della Pietra, V. Della Pietra and R. Mercer: The mathematics of statistical machine translation: parameter estimation, Computational Linguistics, Vol. 19, N. 2, pp. 263–310, 1993. 6. F. Casacuberta: Maximum mutual information and conditional maximum likelihood estimations of stochastic syntax-directed translation schemes, in: Grammatical inference: learning syntax from sentences, L. Miclet and C. de la Higuera (eds), Lecture Notes in Artificial Intelligence, Vol. 1147, Springer-Verlag, Berlin, pp. 282–291, 1996. 7. F. Casacuberta and C. de la Higuera: Computational complexity of problems on probabilistic grammars and transducers, Proceedings of the 5th International Colloquium on Grammatical Inference. 2000. 8. K.S. Fu: Syntactic pattern recognition and applications, Prentice-Hall, Englewood Cliffs, NJ. 1982. 9. P. Garc´ıa, E. Vidal and F. Casacuberta: Local languages, the succesor method and a step towards a general methodology for the inference of regular grammars, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 9, No. 6, pp. 841– 844, 1987. 10. H. Ney, S. Martin and F. Wessel: Statistical language modeling using leaving-oneout, in Corpus-based methods in language and speech processing, Chap. 6, Kluwer Academic Publishers, 1997 11. Instituto Tecnol´ ogico de Inform´tica, Fondazione Ugo Bordoni, Rheinisch Westf¨ alische Technische Hochschule Aachen Lehrstuhl f¨ ur Informatik VI and Zeres GmbH Bochum: Example-based language translation systems. Second year progress report, EuTransproject, Technical report deliverable D0.1b. Information Technology. Long Term Research Domain. Open scheme. Project Number 32026. 1999. 12. K. Knight and Y. Al-Onaizan: Translation with finite-state devices, Proceedings of the 4th. ANSTA Conference, 1998.

14

F. Casacuberta

13. D. Llorens: Suavizado general de aut´ omatas finitos, Ph.D. Thesis. Universitat Politècnica de València. To be published in 2000. 14. E. M¨ akinen: Inferring finite transducers, University of Tampere, Report A-1999-3, 1999. 15. F. Maryanski and M.G. Thomason: Properties of stochastic syntax-directed translation schemata, International Journal of Computer and Information Science, Vol. 8, N. 2, pp. 89–110, 1979. 16. J. Oncina, P. Garc´ıa and E. Vidal: Learning subsequential transducers for pattern recognition tasks, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 15, pp. 448–454, 1993. 17. Rheinisch Westf¨ alische Technische Hochschule Aachen Lehrstuhl f¨ ur Informatik VI and Instituto Tecnol´ ogico de Inform´ atica: Statistical Modeling Techniques and Results and Search Techniques and Results, EuTransproject, Technical Report Deliverables D3.1a and D3.2a Information Technology. Long Term Research Domain. Open scheme. Project Number 32026, 1999. 18. P.R. Clarkson and R. Rosenfeld: Statistical Language Modeling Using the CMUCambridge Toolkit, Proceedings ESCA Eurospeech, Vol. 5, pp. 2707–2710, 1997. 19. E. Vidal, P. Garc´ıa and E. Segarra: Inductive learning of finite-state transducers for the interpretation of unidimensional objects, Structural Pattern Analysis, R, Mohr, Th.Pavlidis, A. Sanfeliu (eds.), pp. 17–35, World Scientific pub. 1989. 20. E. Vidal, F. Casacuberta and P. Garc´ıa: Grammatical inference and automatic speech recognition, in Speech recognition and coding: new advances and trends, A.Rubio, J. L´ opez (eds.) pp. 174–191, NATO-ASI Vol. F147, Springer-Verlag, 1995. 21. E. Vidal: Finite-state speech-to-speech translation, Proceedings of the International Conference on Acoustic, Speech and Signal Processing. Munich (Germany), Vol. I, pp. 111–114, 1997.

Computational Complexity of Problems on Probabilistic Grammars and Transducers Francisco Casacuberta1? and Colin de la Higuera2 1

2

Departamento de Sistemas Inform´ aticos y Computaci´ on, Universidad Politécnica de Valencia, 46071 Valencia, Spain. [email protected] EURISE, Faculté des Sciences et Techniques, Université de Saint Etienne - Jean Monnet, 42023 Saint Etienne, France. [email protected]

Abstract. Determinism plays an important role in grammatical inference. However, in practice, ambiguous grammars (and non determinism grammars in particular) are more used than determinism grammars. Computing the probability of parsing a given string or its most probable parse with stochastic regular grammars can be performed in linear time. However, the problem of finding the most probable string has yet not given any satisfactory answer. In this paper we prove that the problem is NP-hard and does not allow for a polynomial time approximation scheme. The result extends to stochastic regular syntax-directed translation schemes.

1

Introduction

As the problem of not having negative evidence arises in practice when wishing to learn grammars, different options as how to deal with the issue have been proposed. Restricted classes of deterministic finite-state automata can be identified [1,10] heuristics have been proposed [22] and used for practical problems in speech recognition or pattern recognition [15], and stochastic inference has been proposed as a mean to deal with the problem [2,24,21]. Stochastic grammars and automata have been used for some time in the context of speech recognition [20,16]. Algorithms that (heuristically) learn a context-free grammar have been proposed (for a recent survey see [23], and other algorithms (namely the forward-backward algorithm for hidden Markov models, close to stochastic finite automata or the inside-outside algorithm for stochastic context-free grammars) that compute probabilities for the rules have been realised [20,14]. But in the general framework of grammatical inference it is important to search for algorithms that not only perform well in practice, but that provably converge to the optimal solution, using only a polynomial amount of time. For the case of stochastic finite automata the problem has been dealt with ?

This work has been partially funded by the European Union and the Spanish CICYT, under grants IT-LTR-OS-30268 and TIC97-0745-C02, respectively.


16

F. Casacuberta and C. de la Higuera

by different authors: in [24] stochastic deterministic finite automata are learnt through Bayes minimisation, in [3], through state merging techniques common to classical algorithms for the deterministic finite-state automaton inference problem. Along the same line in [21] acyclic stochastic deterministic finite automata are learnt, proving furthermore that under certain restrictions the inferred automaton is probably approximately correct. Work in the direction of learning this sort of object has been followed these last years, with new algorithms proposed in [27,25]. In a general sense the models that have been inferred are always deterministic. It is not obvious why this should be so as non-deterministic stochastic automata are strictly more powerful than their deterministic counter parts. They can also be of a smaller size and thus be more understandable. One reason may be that in the normal (non-stochastic) paradigm, it can be proved that non deterministic machines can not be identified in polynomial time [17]. In this work we point out that the difference between deterministic and non-deterministic stochastic automata (or regular grammars) is also that some reasonably easy problems in the deterministic case become intractable in the non deterministic case. An appealing feature of stochastic regular grammars is the existence of efficient algorithms for parsing. The probability of generating a given string by a stochastic regular grammar can be computed in linear time with the length of the string. The same holds for the search of the derivation with the highest probability. In spite of the existence of polynomial algorithms for dealing with some problems that involve stochastic regular grammars, there is another important problem which does not have an efficient solution. This is to find the most probable string that can be generated by a stochastic regular grammar. Other useful models which are closely related to stochastic regular grammars are the stochastic regular syntax-directed translation schemes [9,13,26]. Stochastic grammars are adequate models for classification tasks; however, there are many practical situations which do not fit well within the classification framework but can be properly tackled through formal translation [18]. For translation, efficient (linear) algorithms are only known for the computation of the highest probability translation form [4]. In this framework, given an input string, the goal is to find its most probable translation. However, there is no efficient solution for this problem. Under the complexity theory framework [11], we report some results about the difficulty of different computations regarding probabilistic finite state machines.

2

The Most Probable String Problem

The following definition is classical [13]. Definition 1: A stochastic regular grammar (SRG) G is a tuple < N, Σ, R, S, P >, where N is a finite set of non-terminal symbols; Σ is a

Computational Complexity of Problems on Probabilistic Grammars

17

finite set of terminal symbols; R is a set of rules of the form A → aB or A → a for A, B ∈ N and a ∈ Σ (for simplicity, empty rules are not allowed); S is the starting symbol and P : R → Q+ (the set of the positive rational numbers) is a function such that X X P (A → aB) + P (A → a) = 1 ∀A ∈ N a∈Σ,B∈N :(A→aB)∈R

a∈Σ:(A→a)∈R

Stochastic grammars are probabilistic generators of languages; therefore, the concept of probability that a string is generated by a SRG can be defined. Definition 2: Given w ∈ Σ ? (the set of finite-length strings over Σ), the probability that a SRG G generates w is defined as: X prG (w) = prG (d(w)) ∀d(w)

where d(w) is a complete derivation of w in G of the form: S → w1 A1 → w1 w2 A2 → ... → w1 w2 ...w|w|−1 A|w|−1 → w1 w2 ...w|w|−1 w|w| = w and prG (d(w)) = P (S → w1 A1 )P (A1 → w2 A2 )...P (A|w|−1 → w|w| ) Some important problems arise with these definitions. Namely the computation for a given string of its probability (PS) or of its most probable derivation (MPDS), the computation of the most probable derivation (MPD) and the computation of the most probable string (MPS). The PS, MPDS and MPD problems have been widely addressed. The PS and MPDS are classical parsing problems, and can be solved in time O(|w||N |2 ) [5]. The MPD problem can also be dealt with using Dijkstra’s algorithm [6] to compute the shortest path in a weighted path and requires no more than O(|N |2 ) time. The MPS problem, although straightforward, has not been dealt with. Let us define the associated decision problem as follows: Problem Most probable string (MPS). Instance A SRG G, and p ∈ Q+ . Question Is there a string x ∈ Σ ? with |x| ≤ |N |, such that prG (x) ≥ p ? A more restricted problem is the following : Problem Restricted Most Probable String (RMPS). Instance A SRG G, d ∈ N (the set of natural numbers), d ≤ |N |, and p ∈ Q+ . Question Is there a string x ∈ Σ d such that prG (x) ≥ p? RMPS is not just a special case of MPS. We will prove that both MPS and RMPS are NP-hard. As the probability of any string can be computed in

18


polynomial time1 , both MPS and RMPS are in NP. We prove that MPS and RMPS are NP-hard by reduction from the “satisfiability” problem (SAT) [11]. The proof relies on a technical encoding of a set of clauses: Given an instance of SAT: 1) a collection v1 , ..., vn of n boolean variables and 2) a collection c1 , ..., ck of k clauses over the n variables, consider the following SRG G = (N, Σ, R, S, P ): – Σ = {f, t, $, #} – For 1 ≤ j ≤ k, – Aj0 ∈ N – the rule S → $Aj0 is in R with probability 1/k and rules Bn → $ and Ajn → # are in R with an associated probability 1. – for 1 ≤ i ≤ n with an associated probability 1/2: • Aji , Bi ∈ N • the rules Bi−1 → tBi and Bi−1 → f Bi are in R. • if vi appears as a positive literal in cj then the rules Aji−1 → f Aji and Aji−1 → tBi are in R. • if vi appears as a negative literal in cj then the rules Aji−1 → tAji and Aji−1 → f Bi are in R. • if vi does not appear in cj then the rules Aji−1 → tAji and Aji−1 → f Aji are in R. Each of these rules have an associated probability of 1/2. – For RMPS fix d = n + 2. To illustrate this construction, consider an example where one of the clauses is ¯3 ∨ x5 with n = 6. Then the corresponding part of the automaton2 cj = x2 ∨ x for this clause is shown in the Figure 1 This SRG must verify that if a clause is evaluated to true by the assignment of values to the boolean variables, then a complete derivation has to exist and vice-versa. On the other hand, if a clause is evaluated to false, no such derivation associated to the clause has to exist in the SRG nor vice-versa. Theorem 1: RMPS and MPS are NP-complete. Proof of Theorem 1: From the above construction, if the clause is satisfied for some solution (x1 , ..., xn ) the corresponding string $X1 ...Xn $ (Xi is t if xi is true, and Xi is f if xi is f alse) will have probability 1/(k 2n ) for each derivation linked to the clause. Note that the construction is in O(k n). The string length is n + 2 ≤ |N | = k · (2n + 1). Fix3 p to 1/2n . Let m be a solution of SAT; it can be considered as a string in {f, t}n , hence the corresponding $m$ is a string generated by G with k derivations all of probability 1/(k 2n ); so the probability of $m$ by G is 1/2n . On 1 2

3

This can be done in O(|w||N |2 ) A SRG can be interpreted by its associated graph. Notice that some states (Aj6 and B1 ) are useless). The fact that the grammar is not in proper normal form is irrelevant. Encoding of p only requires n bits

Computational Complexity of Problems on Probabilistic Grammars f

f 1/2 Aj1 Aj0 t

f 1/2

Aj2

t 1/2

1/2

$

t

1/k

1/2

S

f

.. . Fig. 1.

B 1 1/2

t 1/2

f

1/2 Aj3 t 1/2

f t 1/2

Aj5

f 1/2 t 1/2

Aj6

# 1

t

1/2

B 2 1/2

f Aj4 1/2

19

1/2

f

B 3 1/2 t 1/2

f

B 4 1/2

t 1/2

f

B 5 1/2 t 1/2

B6 $ 1

Part of the SRG corresponding to clause cj = x2 ∨ x ¯3 ∨ x5 with n = 6.

the other hand, if the instance of SAT does not admit any solution, then as the only strings that have non null probability for the associated grammar are of length n + 2 (= d), and at least one clause is not satisfied (for example if the clause j is not satisfied, the corresponding derivation ends in Ajn ), then no string 5. has probability 1/2n Consequently the corresponding optimization problems (finding the most probable string) are NP-hard. More can be said about the NP-optimization problem: Problem Maximum probability of a string (MaxPS). Instance A SRG G , and p ∈ Q+ . Solution A string x ∈ Σ ? Measure prG (x). By reduction from maximum satisfiability (Max-SAT) [12,19], Theorem 2: MaxPS is APX-hard. Maximum satisfiability is the NP-optimization problem corresponding to SAT. It concerns finding a subset of clauses such that there is a truth assignment satisfying each clause in the subset. The associated measure is just the number of clauses. The problem is APX-complete, i.e. it is complete for the class APX. Being APX-complete implies that you can not do better than a constant approximation (a bound of the constant approximation is proposed by Goemans and Williamson [12]) and that no PTAS (polynomial time approximation scheme) is feasible. Proof of Theorem 2: The proof is straight forward and involves the same construction as for the NP-hardness of MPS: Given an instance I of Max-SAT, and a rational , construct an instance f (I, ) of MaxPS as in the proof of theorem 1. Now given a string x on the input alphabet of the associated SRG f (I, ), the following holds: prf (I,) (x) =

c k 2n

⇒ c = g(I, x, ) clauses of I can be satisfied.

20


Finally we have, for any instance I of Max-SAT, any rational and any string x solution to f (I, ): opt(I) opt(f (I, )) = m(f (I, ), x) m(x, g(I, x, )) where opt denotes the optimal result (maximum number of satisfied clauses or maximum probability) and m is the measure function (number of actual satisfied clauses for a given assignment and probability of a given string). It follows that with playing a dummy part the reduction inequation can be obtained [7]: opt(I) opt(f (I, )) ≤⇒ ≤ m(f (I, ), x) m(x, g(I, x, )) All these constructions are polynomial.

3

5

Stochastic Regular Syntax-Directed Translation Scheme

The last problem deals with the search of an optimal translation of a given input string according to a translation scheme [13]. Definition 3: A stochastic regular syntax-directed translation scheme (SRT) E is a tuple < N, Σ, ∆, R, S, P >, where N and S are defined as in SRGs, Σ is a finite set of input terminal symbols, ∆ is a finite set of output terminal symbols (Σ ∩ ∆ = ∅); R is a set of rules of the form A → aB, ωB or A → a, ω for A, B ∈ N , a ∈ Σ, ω ∈ ∆? and and P : R → Q+ is a function such that X X P (A → aB, ωB) + P (A → a, ω) = 1, ∀A ∈ N ∀a ∈ Σ, ω ∈ ∆? , B ∈ N : A → aB, ωB ∈ R

∀a ∈ Σ, ω ∈ ∆? : A → aB, ωB

For simplicity, empty input rules (A → λB, wB or A → λ, w where λ is the empty string) are not allowed. SRGs and SRTs are closely related and given a SRT E, the probability of a translation pair (x, y) ∈ Σ ? × ∆? , prE (x, y) is defined in a way similar to that for SRGs: Definition 4: The probability of a translation pair (x, y) ∈ Σ ? × ∆? according to the scheme E is defined as: X prE (x, y) = prE (t(x, y)) ∀t(x,y)


21

where t(x, y) is a translation form of (x, y) in E: (S, S) → (x1 A1 , y1 A1 ) → (x1 x2 A2 , y1 y2 A2 ) → ... → (x, y) and the corresponding probability of the translation form is: pr(t(x, y)) = P (S → x1 A1 , y1 A1 )P (A1 → x2 A2 , y2 A2 )....P (A|x|−1 → x|x| , y|x|) The following example is presented to illustrate the above definitions. Example 1. N = {S, A, B}, Σ = {0, 1} , ∆ = {a, b} and the rules of the Table 1. The input string 010 has two possible translations: abbbba and Table 1. Set of rules and probabilities corresponding to SRT of Example 2. Rules (R) Probabilities (P) S → 0A, aA 3/10 S → 0B, abB 7/10 A → 1B, aaB 2/7 A → 1A, aaaA 4/7 A → 0, a 1/7 B → 1A, bbbA 2/5 B → 0, aa 3/5

aaaaa. The first one can be obtained as S → 0B, abB → 01A, abbbbA → 010, abbbba with probability 1/25, and the second one with probability 6/245 as S → 0A, aA → 01A, aaaaA → 010, aaaaa or with probability 9/175 as S → 0A, aA → 01B, aaaB → 010, aaaaa. Therefore, 010 can be translated into abbbbba with probability 1/25, or into aaaaa with probability 6/245 + 9/175 =93/1225. An interesting question is thus the one of computing the most probable translation of some input string. Formally: Problem Most probable translation (MPT). Instance A SRT E , x ∈ Σ ? and p ∈ Q+ . Question Is there an output string y ∈ ∆? , |y| ≤ |N | · lmax (lmax is the maximum length of the output string in a rule) such that prE (x, y) ≥ p? In Example 1, the second translation (aaaaa) has the highest probability, therefore it is the most probable translation of 010. If the translation defined by E from Σ ? to ∆? is not ambiguous (E defines a function from Σ ? to ∆? ), there is an efficient algorithm that computes an answer to the MPT problem in linear time. Basically, this algorithm performs a parsing of the input with the input grammar. The MPT problem can be reduced from RMPS as follows: given a SRG G =< N, Σ, R, S, P >, an integer n and a rational p, construct: a SRT E =< N 0 , Σ 0 , ∆, R0 , S 0 , P 0 > with

22


N0 = N, ∆ = Σ, Σ 0 = {$}, for every rule A → aB ∈ R, a rule A → $B, aB is in R0 with P 0 (A → $B, aB) = P (A → aB) – for every rule A → a ∈ R, a rule A → $, a is in R0 with P 0 (A → $, a) = P (A → a) – an input string $n (n ≤ |N |) – a rational p – – – –

Theorem 3: MPT is NP-complete. Proof of the Theorem 3: From the above reduction, it follows that: 1) the construction is polynomial; and 2) $n has an output string y ∈ ∆? such that prG (y) ≥ p if and only if prE ($n , y) ≥ p. The length of y 5 And the associated optimization problem of computing the most probable translation is NP-hard. Without proof, (it follows from the previous different results and proofs) for the associated NP optimization problem (MaxPT) we give a final result: Theorem 4: MaxPT is APX-hard.

4

Conclusions

In this paper we have presented computational complexity results regarding parsing problems for stochastic regular grammars and stochastic regular syntaxdirected translation schemes. In particular, the problems of searching for the most probable string in a SRG and of searching for the most probable translation of an input string given a SRT are NP-hard problems and the associated optimization problems do not admit polynomial approximation schemes. Future work can be conducted in the following direction: we have proved that both NPoptimization problems are APX-hard. Do they belong to APX? Such a result would require a polynomial time algorithm that always meets a given bound. Acknowledgements. The authors wish to thank the anonymous reviewers for their criticisms and suggestions.

References 1. D. Angluin, Inference of reversible languages. Journal of the ACM, Vol. 29(3), pp. 741–765, 1982. 2. R. Carrasco and J. Oncina, Learning stochastic regular grammars by means of a state merging method, in Grammatical Inference and Applications. Proceedings of ICGI ’94, Lecture Notes in Artificial Intelligence 862, Springer Verlag ed., pp. 139–150, 1994.


23

3. Carrasco, J. Oncina, Learning deterministic regular grammars from stochastic samples in polynomial time. Informatique Théorique et Applications, Vol. 33(1), pp. 1–19, 1999. 4. F.Casacuberta, Maximum mutual information and conditional maximum likelihood estimations of stochastic syntax-directed translation schemes, in: L. Miclet and C. de la Higuera (eds), Grammatical Inference: Learning Syntax from Sentences. Lecture Notes in Artificial Intelligence, Vol 1147, pp. 282–291, Springer-Verlag, 1996. 5. F.Casacuberta, Growth transformations for probabilistic functions of stochastic grammars. International Journal on Pattern Recognition and Artificial Intelligence. Vol. 10, pp. 183–201, 1996. 6. T. Cormen, Ch. Leiserson and R. Rivest, Introduction to algorithms. The MIT Press, 1990. 7. P. Crescenzi and V. Kann, A compendium of NP optimization problems, http://www.nada.kth.se/ viggo/problemlist/compendium.html (1995). 8. K.S. Fu and T.L.Booth, Grammatical inference: introduction and survey. Part I and II, IEEE Transactions on System Man and Cybernetics, Vol. 5, pp. 59–72/409– 23, 1985. 9. K.S. Fu, Syntactic pattern recognition and applications. Prentice-Hall, Englewood Cliffs, NJ. 1982. 10. P. Garc´ıa and E. Vidal, Inference of K-testable languages in the strict sense and applications to syntactic pattern recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. Vol. 12(9). pp. 920–925. 1990. 11. M.R. Garey and D.S. Johnson, Computers and intractability: a guide to the theory of NP-completeness W.H. Freeman, San Francisco, 1979. 12. M.X. Goemans and D. O. Williamson, 878-approximation algorithms for MAXCUT and MAX-2SAT. Proc. Twenty sixth Ann. ACM Symposium on Th. of Comp., 422–431, 1994. 13. R. Gonz´ alez and M. Thomason, Syntactic pattern recognition: an introduction. Addison-Wesley, Reading, MA 1978. 14. K. Lari, and S. Young, Applications of stocashtic context-free grammars. Computer Speech and Language. Vol. 5. 237–257. 1991. 15. S. Lucas, E. Vidal, A. Amiri, S. Hanlon and J-C.Amengual, A comparison of syntactic and statistical techniques for off-line OCR. Proceedings of the International Colloquium on Grammatical Inference ICGI-94 (pp. 168–179). Lecture Notes in Artificial Intelligence 862, Springer-Verlag, 1994. 16. H. Ney, Stochastic grammars and Pattern Recognition, in Speech Recognition and Understanding. edited by P. Laface and R. de Mori, Springer-Verlag, pp. 45–360, 1995. 17. C. de la Higuera, Characteristic sets for grammatical inference Machine Learning, 27 pp. 1–14, 1997 18. J. Oncina, P. Garc´ıa and E. Vidal, Learning subsequential transducers for pattern recognition tasks, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 15, pp. 448–458, 1993. 19. C.H. Papadimitriou and M. Yannakakis, Optimisation approximation and complexity classes. Journal Computing System Science, Vol. 43, pp. 425–440, 1991. 20. L. Rabiner and B.H. Juang, Fundamentals of Speech Recognition. Prentice-Hall, 1993. 21. D. Ron, Y. Singer and N. Tishby, On the Learnability and Usage of Acyclic Probabilistic Finite Automata, Proceedings of COLT 1995 , pp. 31–40, 1995.

24


22. H. Rulot and E. Vidal, Modelling (sub)string-length-based constraints through grammatical inference methods. Devijver and Kittler eds. Sringer-Verlag 1987. 23. Y. Sakakibara, Recent Advances of Grammatical Inference. Theoretical Computer Science Vol. 185, pp. 15–45, 1997. 24. A. Stolcke and S. Omohundro, Inducing Probabilistic Grammars by Bayesian Model Merging, in Grammatical Inference and Applications. Proceedings of ICGI ’94, Lecture Notes in Artificial Intelligence 862, Springer Verlag ed., pp. 106–118, 1994. 25. F. Thollard and P. Dupont and C. de la Higuera, Probabilistic DFA Inference using Kullback-Leibler Divergence and Minimality. ICML2000 (International Colloquium on Machine Learning), Stanford, 2000. 26. E.Vidal, F.Casacuberta and P.Garc´ıa, Syntactic Learning Techniques for Language Modeling and Acoustic-Phonetic Decoding, in: A. Rubio (ed.) New Advances and Trends in Speech Recognition and Coding Chap. 27, NATO-ASI Series SpringerVerlag, pp. 174–191, 1995. 27. M. Young-Lai and F.W. Tompa, Stochastic Grammatical Inference of Text Database Structure, to appear in Machine Learning, 2000.

Efficient Ambiguity Detection in C-NFA A Step Towards the Inference of Non Deterministic Automata Fran¸cois Coste and Daniel Fredouille IRISA/INRIA Rennes Campus Universitaire de Beaulieu, 35042 RENNES Cedex, France Phone : +33 2 99 84 71 00 Fax: +33 2 99 84 71 71 {Francois.Coste|Daniel.Fredouille}@irisa.fr

Abstract This work addresses the problem of the inference of non deterministic automata (NFA) from given positive and negative samples. We propose here to consider this problem as a particular case of the inference of unambiguous finite state classifier. We are then able to present an efficient incompatibility NFA detection framework for state merging inference process. key words : regular inference, non deterministic automata, finite state classifier, sequence discrimination

Introduction This work addresses the problem of the inference of non deterministic automata (NFA) from given positive and negative samples. This problem has been extensively studied for the inference of deterministic automata (DFA), for which state merging algorithms have been proven efficient [OG92,Lan92,CN97,LPP98, OS98]. Whereas DFA are polynomially identifiable from given data [Gol78, dlH97], this result does not hold for NFA [dlH97]. In contrast, it is well known that there exist languages such that their representation by DFA requires an exponential number of states with respect to the NFA representation. Considering the inference of NFA instead of DFA allows therefore to obtain smaller solution which we expect to require less samples to be characterized. Few studies have been made on the inference of NFA. Yokomori [Yok94] has proposed an algorithm that needs an oracle and can infer NFA that determinizes polynomialy in polynomial time. We propose here to consider the inference of compatible NFA as a particular case of the inference of unambiguous finite state classifier presented in section 1. A first algorithm for checking unambiguousness of a C-NFA is given in this section. The second section proposes an incremental version of this algorithm for a state merging inference process, ensuring the compatibility of the corresponding NFA without parsing the sample. We conclude with a first experimentation comparing minimum sized NFA and DFA inference with respect to the size of the training sample. A.L. Oliveira (Ed.): ICGI 2000, LNAI 1891, pp. 25–38, 2000. c Springer-Verlag Berlin Heidelberg 2000

26

1

F. Coste and D. Fredouille

Inference of Unambiguous Finite State Classifier

The purpose of this section is to introduce the inference of finite state classifier by means of state merging algorithms. Using this representation allows unbiased inference [AS95,Alq97]. We propose here to take advantage of the simultaneous representation of a set of languages for the inference of unambiguous automata. 1.1

Definitions and Notations

Definition 1. A C-classes non deterministic finite state automata (C-NFA) is defined by a 6-tuple (Q, Q0 , Σ, Γ, δ, ρ) where: Q is a finite set of states; Q0 ⊆ Q is the set of initial states; Σ is a finite alphabet of input symbols; Γ is a finite alphabet of C output symbols; δ is the next-state function mapping Q × Σ to 2Q (if δ maps Q × Σ to Q, the automaton is said deterministic and is denoted by C-DFA); ρ is the output function mapping Q to 2Γ . The function realized by a C-NFA is the classification of sequences. The classification function γ mapping Σ ∗ × Q to 2Γ is defined by: [ γ(q, w) = ρ(q 0 ) q 0 ∈δ(q,w)

where δ has been extended to sequences following the classical way by: S ∀q ∈ Q, ∀w ∈ Σ ∗ , ∀a ∈ Σ ∪ {}, δ(q, ) = {q}, δ(q, wa) = q0 ∈δ(q,w) δ(q 0 , a) The classification of a sequence w by a C-NFA may then be defined as the set of classifications obtained from the initial states. We also denote by γ this function mapping Σ ∗ to 2Γ : [ γ(w) = γ(q, w) q∈Q0

Given a C-NFA M , a sequence w is said classified if its classification is defined (ie: γ(w) 6= ∅). The set of classified sequences is named the domain of M . The classification over this domain defines a C-tuple of regular languages denoted L(M ): L(M ) = hLc (M )ic∈Γ where ∀c ∈ Γ, Lc (M ) = {w ∈ Σ ∗ |c ∈ γ(w)}. A C-NFA allows to handle simultaneously a set of languages. In this paper, we focus on unambiguous C-NFA: Definition 2. A C-NFA is said unambiguous if each sequence is classified in at most one class. From the definition, it follows that a C-NFA M is unambiguous iff the C-tuple of languages represented by M are mutually disjoint, i.e.: ∀i, j ∈ Γ, Li (M ) ∩ Lj (M ) = ∅. The unambiguousness property is important for the search of compatible automata from positive and negative samples and other applications dealing with

Efficient Ambiguity Detection in C-NFA

27

discrimination of sequences by finite state machines. The choice of a C-NFA representation of a set of languages, instead of the classical automata representation, allows to efficiently characterize the disjunction of the recognized languages. We propose to take advantage of this property in the next sections devoted to the inference of unambiguous C-NFA. 1.2

State Merging Inference

The problem of inferring a C-NFA may be seen as a C-regular inference problem [Cos99]. We assume that a training sample S = hSc ic∈Γ is given such that each Sc is a sample from the target language Lc (M ), i.e. a finite subset of Lc (M ). One classical assumption made in grammatical inference is that the sample is structurally complete with respect to the target machine. Under this assumption, the inference of C-NFA may then be done by means of state merging algorithm, which proceeds by merging states of the Maximal Canonical Automaton, denoted by MCA(S), which is the automaton resulting from the union of the canonical C-NFA for each sequence of S (figure 1 and algorithm 1). When looking for

Figure 1. MCA(S) for S = h{ab}, {aaa, aa}i.

unambiguous C-NFA, the search is pruned as soon as the current automaton is detected ambiguous, since all automata obtained by merging states of an ambiguous automaton are ambiguous. Algorithm 1 Greedy state merging algorithm 1: 2: 3: 4: 5: 6: 7: 8:

Greedy SMA(S) /* Input: training sample S */ /* Output: a C-NFA compatible with S */ A ← Maximal Canonical Automaton(S) while Choose States To Merge(q1 ,q2 ) do A0 ← Merge(A,q1 ,q2 ) if A0 is not ambiguous then A ← A0

28


Detecting ambiguity is simple in the deterministic case. It can be done by checking that no states of different classes have been merged, or even by parsing the automaton with the training set. In the non-deterministic case, parsing may be done by a viterbi-like procedure. For classical automata parsing the negative sample is sufficient to ensure compatibility. For non deterministic C-NFA, compatibility with samples and unambiguousness should not be confused: whenever all the samples are correctly labeled by the automaton, sequences outside the training set may have more than one classification. We propose in the next section a first algorithm to detect the ambiguousness of C-NFA. 1.3

Ambiguity Detection

Only two cases of ambiguity exist. A C-NFA is ambiguous if: – There exists a state such that its output function returns two different classifications. For C-DFA, it is the unique case of ambiguity. – There exist paths labeled by the same sequence w leading to states with defined and different classifications.

or We introduce the notation γ1 6∼γ2 (γ1 incompatible with γ2 ) for two different and defined classifications γ1 and γ2 : γ1 6∼γ2 ⇔ ((γ1 6= γ2 ) ∧ (γ1 6= ∅) ∧ (γ2 6= ∅)). Otherwise, the classifications are said compatible (denoted γ1 ∼ γ2 ). It is easy to detect whether the first case holds. For the second case, we need to introduce the definition of incompatible pair of states. Two states q1 and q2 are incompatible, (denoted q1 6∼q2 ), if there exists a word whose classifications from these states are incompatible: q1 6∼q2 ⇔ ∃w ∈ Σ ∗ , ∃(s1 , s2 ) ∈ δ(q1 , w) × δ(q2 , w), ρ(s1 ) 6∼ρ(s2 ) Otherwise, the states are said compatible (denoted q1 ∼ q2 ). Then, ambiguity detection for a C-NFA reduces to checking if a state is incompatible with itself or if two initial states are incompatible. To mark incompatible states, we propose an algorithm (algorithm 2) inspired by the algorithm of Hopcroft and Ullman designed to mark non equivalent states


29

for automaton minimization [HU80] 1 . Since the automata we consider are not necessarily deterministic, the original algorithm has been changed by inverting the propagation direction of the marking process, which results in O(n2 ) time complexity for tree-like automata. This algorithm may be used to construct the set of incompatible pairs of states E6∼ and to raise an exception if it detects ambiguity. Algorithm 2 Incompatible states and C-NFA ambiguity. 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18:

Incompatible States(A = (Σ, Γ, Q, Q0 , δ, ρ)): /* Search of the set of incompatible states of A */ /* and ambiguity detection of A */ E6∼ ← ∅ /* set of incompatible states */ for all {qi , qj } ∈ Q × Q, ρ(qi ) 6∼ρ(qj ) do if {qi , qj } 6∈ E6∼ then Set Incompatible And Propagate(qi , qj ) return E6∼ Set Incompatible And Propagate(q1 ,q2 ): /* ambiguity detection */ if (q1 = q2 ) ∨ (q1 ∈ Q0 ∧ q2 ∈ Q0 ) then throw exception(“ambiguous C-NFA”) /* Incompability memorization */ E6∼ ← E6∼ ∪ {q1 , q2 } /* Propagation */ for all a ∈ Σ, {p1 , p2 } ∈ δ −1 (q1 , a) × δ −1 (q2 , a) do if {p1 , p2 } 6∈ E6∼ then Set Incompatible And Propagate(p1 , p2 )

In the worst case, the complexity of algorithm 2 is O(|Σ|n4 ) : O(n2 ) calls of the function Set Incompatible And Propagate, whose body needs O(|Σ|n2 ) steps. However, if we denote by ta the maximal number of incoming transitions with the same symbol in a state, one can refine the complexity result. The complexity of Set Incompatible And Propagate body with respect to ta is O(|Σ|t2a ) which leads to a global complexity of O(|Σ|t2a n2 ). Therefore the complexity lies more practically between O(|Σ|n2 ) and O(|Σ|n4 ) according to the value of ta . In an inference process, this algorithm may be used to determine whether each candidate is unambiguous. In the next section, we propose an incremental version of this algorithm to detect ambiguity in an extension of the classical state merging framework.

1

The partition refinement algorithm to minimize automata may not be used here since the state equivalence relation is transitive whereas the compatibility relation is not.

30

2


Considering Unmergeable States During Inference

We propose here to extend the classical state merging algorithm to consider pairs of unmergeable states (denoted for two states q1 and q2 of a C-NFA by q1 6'q2 ). At each step of the inference, instead of always merging the chosen pair of states, the algorithm will be allowed to set this pair of states unmergeable. This may be used to guide the search or to prune an entire part of the search space: either because it has already been explored or either because it is known that no solution may be found in it. 2.1

Detection of Unmergeable States Due to Ambiguity

During the inference of unambiguous automata, some pairs of states may be detected to have no other choice than being set unmergeable to ensure unambiguousness. The first relation that can be used is that two incompatible states are also unmergeable: ∀q1 , q2 ∈ Q × Q, q1 6∼q2 ⇒ q1 6'q2 . We can detect more unmergeable states by considering the counterpart of merging for determinization used in the deterministic framework [OG92], that is considering pairs of states that are reachable by a common word from the initial states. Definition 3. Two states q1 and q2 are said to be in relation k, denoted by q1 k q2 , if they are reachable by a common S word from initial states. More formally, we have q1 k q2 ⇔ ∃w ∈ Σ ∗ , q1 , q2 ∈ q0 ∈Q0 δ(q0 , w). The algorithm computing relation k is very similar to algorithm 2 for incompatible states. The loop in line 5 is replaced by a loop on pairs of initial states, and backward propagation in line 17 is replaced by forward propagation (using δ instead of δ −1 ). This algorithm can also detect ambiguity since it tries to put in relation k two states with incompatible output. Thanks to relation k we can detect new unmergeable states with the following equation which is illustrated in figure 2: q1 6∼q2 ∧ q2 k q3 ⇒ q1 6'q3 Relation k enables also earlier ambiguity detection: to detect ambiguity, we can check that no incompatible states have to be set in relation k (or that no states in relation k have to be set incompatible). This property comes from the fact that if two states are in relation k due to a word w1 and that they are incompatible due to a word w2 , then the word w1 w2 has an ambiguous classification. Notice also that this detection can replace the one given in section 1.3 (algorithm 2 line 11) since all initial states are in relation k, and every state is in relation k with itself.


31

Figure 2. Illustration of the equation q1 6∼q2 ∧ q2 k q3 ⇒ q1 6'q3 : given a relation q2 k q3 involved by a word w1 , and an incompatibility q1 6∼q2 involved by a word w2 , the merging of q1 and q3 is not possible since it entails the acceptation of the word w1 w2 in two different classes. We can also notice that the relation q1 6∼q2 ⇒ q1 6'q2 is due to a particular case of q1 6∼q2 ∧ q2 k q3 ⇒ q1 6'q3 with q2 = q3 thus thanks to the fact that every state is in relation k with itself.

To summarize, before computing a merge we can check in some cases if it will lead to ambiguity, but this checking is not always possible (we do not detect all mergings leading to ambiguity, see figure 3). In this case, ambiguity is detected during the merge thanks to the addition of new relation k and 6∼.

(a)

(b)

Figure 3. part a. States q0 and q1 are unmergeable but not detected with our equations (the automaton resulting from the merge, figure3. part b, is ambiguous; for example, in this automaton the word aa is both classified c1 and c2.

We dispose of various relations between states which are useful not only to detect ambiguity, but also to prevent merging of states that leads to ambiguity. We now propose to maintain these relations after each merge during an inference algorithm.

32


2.2

Incremental Maintenance of Relations

Let E6'(q) (resp. E6∼(q), Ek (q)) denote the set of states unmergeable (resp. incompatible, in relation k) with state q. At the beginning of an inference algorithm, E6'(q), E6∼(q) and Ek (q) have to be initialized. E6∼(q) and Ek (q) can be computed with algorithm 2 and its counterpart for states in relation k, but update of E6'(q) must also be done ; for that reason we use the function Initialize (algorithm 3). Function Merge’ Algorithm 3 Initialization of E6∼, Ek and E6' 1: 2: 3: 4: 5: 6:

Initialize(A=< Σ, Γ, Q, Q0 , δ, γ >) ∀q ∈ Q, E6∼(q) = ∅; Ek (q) = ∅; E6'(q) = ∅ for all {q1 , q2 } ∈ Q0 × Q0 do SetCP1 (q1 ,q2 ) /* maintain E6', add k relation and propagate */ for all {q1 , q2 } ∈ Q × Q, γ(q1 ) 6∼γ(q2 ) do SetIncompatible(q1 ,q2 ) /* maintain E6', add incompatibility and propagate */

(algorithm 4) realizes the merging of two states and update sets Ek , E6∼ and E6'. This update is realized by propagating existing relations incompatible and k on the state created by the merging (functions PropagateIncompatibility and PropagateCP1 , algorithm 5). For example, the ambiguity of the automaton figure 3 part b, may be detected during the merging thanks to addition of new relations: the incompatibility q0 6∼ q2 is transformed into q01 6∼q2 by the merging, then this relation is propagated to q01 6∼q01 by the function PropagateIncompatibility. At this step an exception is thrown since it would imply a k relation and an incompatibility between the same states. Algorithm 4 Merge two states and update E6∼, Ek and E6' 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 1

Merge’(A,q1,q2) /* detection of unmergeable states */ if q1 ∈ E6'(q2 ) then throw exception else A ← Merge(A,q1 ,q2 ) /* substitute q2 by q1 in A and E6', E6∼, Ek */ for all q 0 ∈ Ek (q1 ) do PropagateCP(q 0 ,q1 ) for all q 0 ∈ E6∼(q1 ) do PropagateIncompatibility(q 0 ,q1 ) return A

CP stands for Common Prefix, and correspond to the k relation. We do not detail the functions SetCP and PropagateCP which are the counterpart for relation k of functions SetIncompatible and PropagateIncompatibility shown in algorithm 5.


33

Algorithm 5 Add a new incompatibility in E6∼ and propagates its effects 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17:

SetIncompatible(q1 ,q2 ) if q1 6∈ E6∼(q2 ) then if q1 ∈ Ek (q2 ) then throw exception else /* add q1 to E6∼(q2 ) and q2 to E6∼(q1 ) */ E6∼(q1 ) ← E6∼(q1 ) ∪ {q2 } ; E6∼(q2 ) ← E6∼(q2 ) ∪ {q1 } /* Propagation */ PropagateIncompatibility(q1 ,q2 ) /* Update the blocs in relation 6'*/ for all q ∈ Ek (q1 ) do SetUnmergeable(q2 ,q) for all q ∈ Ek (q2 ) do SetUnmergeable(q1 ,q) PropagateIncompatibility(q1 ,q2 ) for all a ∈ Σ, {p1 , p2 } ∈ δ −1 (q1 , a) × δ −1 (q2 , a)} do SetIncompatible(p1 ,p2 )

Algorithm 6 Add unmergeable states in E6' 1: SetUnmergeable(q1 ,q2 ) 2: if q1 = q2 then 3: throw exception 4: else 5: if q1 6∈ E6'(q2 ) then 6: E6'(q1 ) ← E6'(q1 ) ∪ {q2 }; E6'(q2 ) ← E6'(q2 ) ∪ {q1 }

Every time an incompatibility or a relation k between two states has to be added (functions SetIncompatible and SetCP1 , algorithm 5), two actions are to be taken: (1) we check that the new relation does not mean ambiguity of the C-NFA (algorithm 5, line 3), (2) we compute new unmergeable states using the relation q1 k q2 ∧ q2 6∼q3 ⇒ q1 6'q2 (algorithm 5, line 11-14, and algorithm 6). Thanks to this new algorithm, we are able to infer efficiently non deterministic and non ambiguous C-NFAs. This algorithm can also be directly applied to the inference of classical NFA by inferring a 2-NFA and enabling only merges between states of M CA(hS+ , S− i) created by the positive sample S+ . In this framework, branches created by the negative sample are only used to check the ambiguity of the current C-NFA. To find the corresponding NFA, we suppress in the C-NFA the part of M CA(hS+ , S− i) corresponding to the negative sample. We present in the next section experiments applying this approach for the inference of classical NFAs.

34

3


Experiments

We have implemented our algorithm to carry out first experiments in order to test the validity of our approach. Our idea is to compare the information needed to correctly infer a NFA versus its determinized version. We first present the benchmark designed for this experiment and the state merging algorithm we have used before giving the experimental results. 3.1

Benchmark

We have chosen for this benchmark different non deterministic automata (figure 4) inspired by different papers [Dup96,SY97,DLT00] or specifically designed for the benchmark. We have tried to represent the various processes of determinization. The benchmark contains: a DFA such that no smaller NFA that recognizes the same language is expected to exist (L1) ; a NFA such that its determinization is polynomial (L2) ; NFAs with exponential determinization, representing a finite language (L3) or not (L4, L5) ; a simple NFA common in the DFA literature [Dup96], with a transition added in its transition function (L6). The various properties of these automata are summarized in table 1. A parameter n is set for some of the automata allowing to tune their size, the value chosen for n in the benchmark is indicated in the third column of table 1.

Figure 4. Automata of the benchmark


35

Table 1. Characteristics of the benchmark’s automata number Language with Σ = {a, b} L1

L2 L3 L4 L5 L6

all word such that the absolute value of its number of a minus its number of b modulo n is 0

n in the size size benchmark of NFA of DFA 8

n

n

4 5

n n+1

Σ ∗ aΣ n 2 {(b∗ a)(n−1).x+n.y |x ∈ IN, y ∈ IN+ } 3 see automaton -

n+2 n 3

(n − 1)2 + 2 2n/2+1 − 1 if n is even 3 ∗ 2bn/2c − 1 if n is odd 2n+1 2n − 1 7

{w ∈ Σ|w = uav, |w| < n ∧|v| = bn/2c − 1}

Samples of training and testing sets were generated following a normal distribution for length and a uniform distribution for words of a given length. Training and testing sets are disjoint. 3.2

Algorithm

In these experiments, we consider the inference of a minimum sized non deterministic automaton. We propose to use the “coloring” scheme which has been proven efficient for this search in the deterministic case [BF72,CN97,OS98]. We briefly describe the algorithm we have used. A set C of colored states (the states of the target automaton) is maintained. Search space exploration is performed by a function choosing at each step a state q of Q − C and calling itself recursively, first after each successful merging between q and a state of C, and second, after the promotion of q in C. Adopting a Branch & Bound strategy, the search is pruned if the number of states in C is greater than in the smallest solution found. The same heuristic than in [CN97] has been used both for the deterministic and the non deterministic automaton inference: at each step the state, having the maximum number of colored states unmergeable with it, is chosen to be colored. This algorithm has been used in the upper bound framework [BF72, OS98], which means that it tries to find a solution of size one and increments the size until a solution is found. Within this framework, we guarantee that the solution found is of minimum size and structurally complete with the samples. 3.3

Results

The result of algorithm’s runs are given in figure 5 and table 2. We can verify that for all the experiments except one, identification of the NFA requires

36


a smaller sample than identification of its deterministic version. The only exception is L1 which has been constructed so as to be hard to identify in the non deterministic approach. For all other languages non deterministic approach seems clearly better suited to this task as sparse training data are available. We may interpret this result by applying Occam’s razor principle: smaller automaton compatible with positive and negative samples are more likely to identify the target language. This results may also be explained by the amount of data needed to ensure structural completness with respect to the target automaton. Table 2. convergence observed language number of samples needed to reach “stable” 100% of recognition deterministic case non deterministic case L1 166 278 L2 372 23 L3 > 500 79 L4 65 22 L5 100 32 L6 190 27

Conclusion We have proposed an algorithm to detect whether a C-NFA is ambiguous. This algorithm may be used incrementally in a state merging inference process, taking into account not only the possible state merging but also the impossible ones. We have applied this approach for the exact search of minimal NFA with a saturation strategy. Experimental results are promising and tend to show that less data may be needed to identify the non deterministic automata representation of a language than its deterministic representation. However the main problem for the inference of non deterministic automata remains the lack of canonical form. Denis & al. [DLT00] have presented very recently a first response to this problem by constructing a subclass of NFA for which a canonical form can be defined. Their results could be integrated in the state merging framework in order to reduce the search space and to obtain identification results. Acknowledgments. The authors wish to thank Jacques Nicolas for helpful discussions about this work and Tallur Basavanneppa for valuable comments on the manuscript.


37

Figure 5. Graphs giving recognition level on testing set and size of automaton found (ordinate) compared to the number of samples in training set (abscissa). Inference of DFAs and NFAs is given on same graphs for each language.

38


References [Alq97] [AS95]

[BF72] [CN97] [Cos99] [dlH97] [DLT00] [Dup96] [Gol78] [HU80] [Lan92] [LPP98] [OG92] [OS98] [SY97] [Yok94]

Alquézar (R.). – Symbolic and connectionist learning techniques for grammatical inference. – Thèse de PhD, Universitat Politecnica de Catalunya, mars 1997. Alquézar (R.) et Sanfeliu (A.). – Incremental grammatical inference from positive and negative data using unbiased finite state automata. In : Shape, Structure and Pattern Recognition, Proc. Int. Workshop on Structural and Syntactic Pattern Recognition, SSPR’94, Nahariya (Israel), pp. 291–300. – 1995. Biermann (A. W.) et Feldmann (J. A.). – On the synthesis of finite-state machines from samples of their behaviour. IEEE Transactions on Computeurs C 21, 1972, pp. 592 – 597. Coste (F.) et Nicolas (J.). – Regular inference as a graph coloring problem. In : Workshop on Grammar Inference, Automata Induction, and Language Acquisition (ICML’ 97). – Nashville, TN., USA, juillet 1997. Coste (F.). – State merging inference of finite state classifiers. – Rapport technique n˚ INRIA/RR-3695, IRISA, septembre 1999. de la Higuera (C.). – Characteristic sets for polynomial grammatical inference. Machine Learning, vol. 27, 1997, pp. 125–138. Denis (F.), Lemay (A.) et Terlutte (A.). – Apprentissage de langages réguliers a l’aide d’automates non détérministes. In : Conf´ ` erence d’apprentissage CAp’00. – 2000. Dupont (P.). – Utilisation et apprentissage de mod` eles de langages pour la reconnaissance de la parole continue. – Thèse de PhD, Ecole Nationale Supérieure des Télécommunications, 1996. Gold (E. M.). – Complexity of automaton identification from given data. Information and Control, vol. 37, 1978, pp. 302 – 320. Hopcroft (J.) et Ullman (J.). – Introduction to Automata Theory, Languages, and Computation. – N. Reading, MA, Addison-Wesley, 1980. Lang (K. J.). – Random dfa’s can be approximately learned from sparse uniform examples. 5th ACM workshop on Computation Learning Theorie, 1992, pp. 45 – 52. Lang (K. J.), Pearlmutter (B. A.) et Price (R. A.). – Results of the abbadingo one DFA learning competition and a new evidence-driven state merging algorithm. Lecture Notes in Computer Science, vol. 1433, 1998, pp. 1–12. Oncina (J.) et Garcia (P.). – Inferring regular languages in polynomial update time. Pattern Recognition and Image Analysis, 1992, pp. 49 – 61. Oliveira (A. L.) et Silva (J. P. M.). – Efficient search techniques for the inference of minimum size finite automata. In : South American Symposium on String Processing and Information Retrieval. – 1998. Salomaa (K.) et Yu (S.). – Nfa to dfa transformation for finite languages. In : First international workshop on implementing automata, WIA’96, p. 188. – 1997. Yokomori (T.). – Learning non-deterministic finite automata from queries and counterexamples. Machine Intelligence, vol. 13, 1994, pp. 169–189.

Learning Regular Languages Using Non Deterministic Finite Automata ? Fran¸cois Denis, Aurélien Lemay, and Alain Terlutte Bˆ at. M3, GRAPPA-LIFL Université de Lille I 59655 Villeneuve d’Ascq Cedex France {denis, lemay, terlutte}@lifl.fr

Abstract. We define here the Residual Finite State Automata class (RFSA). This class, included in the Non deterministic Finite Automata class, strictly contains the Deterministic Finite Automata class and shares with it a fundamental property : the existence of a canonical minimal form for any regular language. We also define a notion of characteristic sample SL for a given regular language L and a learning algorithm (DeLeTe). We show that DeLeTe can produce the canonical RFSA of a regular language L from any sample S which contains SL . We think that working on non deterministic automata will allow, in a great amount of cases, to reduce the size of the characteristic sample. This is already true for some languages for which the sample needed by DeLete is far smaller than the one needed by classical algorithms. Key words : regular inference, non deterministic automata.

1

Introduction

Regular language learning is still one of the major thema of the grammatical inference field. This class of language, the simplest in the Chomsky hierarchy, is neither efficiently learnable in the Gold Model [Pit89], nor in the Valiant one [KV94]. Nevertheless, this class is polynomially learnable from given data [Gol78] : one can build for each regular language L a sample SL in polynomial time relatively to the size of the smallest Deterministic Finite Automata (DFA) recognizing L and characterizing L in the way that L can be found back from any sample S containing SL . We could think that this theoritical result would have few consequences since nothing assures that a natural sample contains SL . But some learning algorithms in this last model, like RPNI made by Oncina and Garcia [OG92], are already interesting from an experimental point of view : correctly adapted and with good heuristics, they can turn into powerful algorithms [LPP98]. Can we go further ? The RPNI algorithm calculates a deterministic automaton compatible with the sample in polynomial time. But it is a well known fact that regular languages have representations much cheaper in term of ?

This work was partially supported by “Motricité et Cognition : Contrat par objectif région Nord/Pas-de-Calais”


40

F. Denis, A. Lemay, and A. Terlutte

size : the minimal DFA recognizing the language Σ ∗ 0Σ n has O(2n ) states while this language is described here by a regular expression with O(log n) symbols. Looking after a non deterministic finite automaton (NFA) could seem to be a promising way but it has been proved that NFA are not polynomially identifiable from given data [Hig97]. That is nevertheless the way we will explore here. We define a sub-class of NFA, the Residual Finite States Automata (RFSA) that has some good properties of DFA (existence of a canonical representation) and NFA (often concise minimal representation), but also some of their drawbacks. We will show how to associate with a DFA A a characteristic sample SA of polynomial cardinal and we will define an algorithm (DeLeTe) that builds the canonical RFSA from any sample containing SA . The cardinal of SA is, in any cases, of the same order than the cardinal of the sample needed by RPNI, and it may be exponentially smaller in best cases. For example, RPNI needs O(2n ) examples to learn the language Σ ∗ 0Σ n although DeLeTe can learn it from sample of O(n2 ) words. However, this new algorithm is not a learning algorithm from given data, we will show that it is probably not a serious problem in a PAC context (Probably Approximatively Correct [Val84]) where samples are produced using a probability distribution, where the allowed running time depends on the length of the drawn examples and where the returned hypothesis can be an approximation of the target. After some preliminaries on languages, automata and the learning model from given data presented in Sect. 2, we introduce the Residual Finite States Automata in Sect. 3. Then we show how to associate a representative set with any regular language in Sect. 4.1, we present the DeLeTe algorithm in Sect. 4.2 and we demonstrate the main result of this article in Sect. 4.4. Then, we comment this result and argue that it seems promising.

2 2.1

Preliminary Languages and Automata

Let Σ be a finite alphabet and Σ ∗ be the set of words built on Σ. We note ε the empty word and |u| the length of a word u of Σ ∗ . We assume that words on Σ ∗ are ordered the following way : u < v iff [|u| < |v| or (|u| = |v| and u is before v in the lexicographical order)]. A language is a subset of Σ ∗ . If L is a language, we note pref (L) = {u ∈ Σ ∗ |∃v ∈ Σ ∗ such that uv ∈ L}. A Non deterministic Finite Automaton (NFA) is a quintuplet A = hΣ, Q, Q0 , F, δi where Q is a finite set of states, Q0 ⊆ Q is the set of initial states, F ⊆ Q is the set of terminal states and δ is a (partial) transition function defined from a subset of Q × Σ to 2Q . We also note δ the extended transition function defined on (a subset of) 2Q × Σ ∗ . A language L is regular if there exists a NFA A = hΣ, Q, Q0 , F, δi such that L = {u ∈ Σ ∗ |δ(Q0 , u) ∩ F 6= ∅}. We note REG the set of regular languages. Let A = hΣ, Q, Q0 , F, δi be a NFA and q a state of A, we note Lq the language Lq = {u ∈ Σ ∗ |δ({q}, u) ∩ F 6= ∅}. An automaton is said Deterministic (DFA) if Q0 contains only one element, and if, for each state q and each letter x, δ(q, x) contains at most one element. A finite automaton A is trimmed if every state is accessible and if, from each state, we can access to a terminal

Learning Regular Languages Using Non Deterministic Finite Automata

41

state. Any non empty regular language is accepted by a unique (with respect to an isomorphism) minimal deterministic trimmed automaton. If L is a regular language, and if u is a word of Σ ∗ , we note u−1 L the residual language of L by u defined by u−1 L = {v ∈ Σ ∗ | uv ∈ L}. According to the Myhill-Nerode theorem, the set of distincts residual languages of a regular language is finite. Furthermore, if A = hΣ, Q, {q0 }, F, δi is the minimal DFA recognizing L, u−1 L → δ(q0 , u) is a bijection from the set of residual states of L to Q. Let A = hΣ, Q, Q0 , F, δi be a NFA and L the language recognized by A. We define As = hΣ, Q, Qs0 , F, δ s i where Qs0 = {q ∈ Q|Lq ⊆ L} and δ s (q, x) = {q 0 ∈ Q|Lq0 ⊆ x−1 Lq } for any state q and any letter x. As is said to be the saturated of A and we say that an automaton A is saturated if it is isomorph with As . One can show that an automaton and its saturated recognize the same language [DLT00b]. 2.2

Learning Languages

Our framework is regular language learning from examples. If L is a language defined on the alphabet Σ, an example of L is a pair (u, e) where u ∈ Σ ∗ and e = 1 if u ∈ L (positive example), and e = 0 otherwise (negative example). A sample S of L is a finite set of examples of L. We note S + = {u|(u, 1) ∈ S} and S − = {u|(u, 0) ∈ S}. The size of a sample S (noted ||S||) is the sum of the length of all the words in it.Gold showed that the class of regular languages is polynomially learnable from given data [Gol78]. Goldman and Mathias introduced a learning model with teacher [GM96] that De la Higuera extended to languages and showed equivalent to the learning model from given data [Hig97]. To show that the REG class, represented by DFA, is polynomially learnable from given data is equivalent to show that there exists two algorithms T and L such that for any regular language L with a minimal DFA A : – T takes A as input and gives in polynomal time a sample SL of L (of polynomial size with respect to the size of A) – for each sample S of L, L takes S as input and gives in polynomial time a DFA compatible with S, equivalent with A if S contains SL . The RPNI algorithm ([OG92]) is a learning algorithm for the regular language class in this model. RPNI builds from a sample S the most specific DFA recognizing S + (the prefix tree), then examines if it is possible to merge two states, beginning with the root, while keeping the automaton deterministic and consistant with the sample. The purpose of the characteristic sample is here to avoid that two states are merged when they do not have to be. So, that is not a surprise that RPNI can identify an automaton from any sample containing its characteristic sample : but the main interesting point of this algorithm is that it keeps working in a proper way in “deteriorated” mode, that is when the sample does not contain the characteristic sample. It is here necessary to define heuristics precising whether two states can be merged or not, even when the conditions to do it are not fulfilled.

42

3


RFSA

Definition 1. A Residual Finite State Automaton (RFSA) is a finite automaton A = hΣ, Q, Q0 , F, δi such that, ∀q ∈ Q, ∃u ∈ Σ ∗ such that Lq = u−1 L. Deterministic automata are obviously RFSA, but some non deterministic finite automata are not (cf. figure 1). 0

- q0

/

0

q q 1

Fig. 1. This automaton is not a RFSA as Lq1 = {ε} is not a residual of 0+ .

One of the major interests of the RFSA class is that we can define a notion of canonical RFSA associated with a language. Definition 2. Let L be a regular language. A residual language u−1 L is said to be prime if it is not union of other residual languages of L, that is if [ {v −1 L | v −1 L ( u−1 L} ( u−1 L. We say that a residual language is composed if it is not prime. Example 1. Let L = a∗ + b∗ . This language possesses 3 non empty residuals : L, a∗ and b∗ . The first one is composed, and the other two are prime. Definition 3. Let L be a regular language on Σ and let A = hΣ, Q, Q0 , F, δi be the automaton defined by : – – – –

Q is the set of prime residual non-empty languages of L Q0 is the set of prime residual languages included in L. F is the set of prime residual languages of L containing ε• δ is defined by δ(u−1 L, x) = {v −1 L ∈ Q|v −1 L ⊆ ux−1 L} We say that A is the canonical RFSA of L.

One can show that the canonical RFSA of a regular language L is an RFSA, that it is saturated, that it recognizes L, that it is minimal in the number of states and that any other minimal saturated RFSA recognizing L is isomorph with it [DLT00b]. As a result, the RFSA class shares at least two important properties with DFA : states are defined by residual languages and there exists a canonical element. On the other hand, canonical RFSA of a regular language can be much smaller than the minimal DFA that recognizes this language : that is the case of the language Σ ∗ 0Σ n , often showed in the litterature to illustrate the fact that there can be an exponential gap between the size of the minimal DFA and the size of a minimal NFA recognizing a language.

Learning Regular Languages Using Non Deterministic Finite Automata a

- q1

43

b

/

- q2

/

Fig. 2. canonical RFSA accepting the language a∗ + b∗ . 0, 1

- ε

0

0

/ k

0

s

0,1

0 0,1

/ k

0,1 0

s

0,1 01

s

011

0,1 Fig. 3. The canonical RFSA of Σ ∗ 0Σ 2 : a minimal NFA has 4 states too, although its minimal DFA has 8 states. The 4 prime residual languages are L = Σ ∗ 0Σ 2 , 0−1 L = L ∪ Σ 2 , 01−1 L = L ∪ Σ, 011−1 L = L ∪ {ε}. The other 4 residuals are composed.

4 4.1

Learning Regular Languages Using RFSA A Characteristic Sample

Let A = (Σ, Q, q0 , F, δ) be a minimal trimmed DFA. For every state q of A, we define uq as being the smallest word of Σ ∗ such that δ(q0 , uq ) = q. As a consequence, we have uq0 = ε. We assume that Q = {q0 , . . . , qn } is ordered using uq . In other words, qi < qj iff uqi < uqj . We note P C(L) = {uq |q ∈ Q} and U (L) = P C(L)∪{ux|u ∈ P C(L), x ∈ Σ}. Definition 4. We say that a sample S is characteristic for the minimal DFA if – for every state – for every state – if Lq0 \ Lq00 – if Lq00 \ Lq0

q of Q, there exists a word of the form uq v in S + q and q 00 of Q and every letter x, if q 0 = δ(q, x) then 6= ∅ then there exists w such that uq xw ∈ S + and uq00 w ∈ S − 6= ∅ then there exists w such that uq xw ∈ S − and uq00 w ∈ S +

Let S be a sample, u and v be 2 words of pref (S + ). We note : – u ' v if no word w exists such that uw ∈ S + and vw ∈ S − or the opposite – u ≺ v if no word w exists such that uw ∈ S + and vw ∈ S − As, in a learning context, we do not know residual languages but only the learning sample, we will use these relations to estimate relations between residual languages. We show here that it is reasonable to do that if S is characteristic for the minimal DFA, and if u and v are in U (L).

44


Lemma 1. If u, v ∈ U (L) and if S is a sample of L then u ' v ⇐ u−1 L = v −1 L, u ≺ v ⇐ u−1 L ⊆ v −1 L If S is characteristic for the minimal DFA A, then the converses are true. Proof. The if-parts are straightforward. Let us assume that S is characteristic for the DFA A and that u−1 L \ v −1 L 6= ∅. Then u and v are distincts and one of those two words at least is different from ε. Let us assume that u = u1 x. We verify that a state q1 exists such that uq1 = u1 . There exists a word w such that uw = u1 xw ∈ S + and vw ∈ S − , thus u 6'v and u 6≺v. The other three cases (u−1 L \ v −1 L 6= ∅ and v = v1 x, v −1 L \ u−1 L 6= ∅ and v = v1 x, v −1 L \ u−1 L 6= ∅ and u = u1 x) are treated in a same way. Let us introduce a new notation : – if q is a prime state, let vq be the smallest word of Lq such that, for every state q 0 , Lq0 ( Lq ⇒ v 6∈Lq0 . – if q is a composed state, there exists a highest index k such that S = {L |L – L q q q i i ( Lq and i ≤ k} and S – {Lqi |Lqi ( Lq and i < k} ( Lq . We then define vq as being the smallest word of Lq \ {Lqi |Lqi ⊂ Lq , i < k}. Definition 5. We say that a sample S is characteristic for the canonical RFSA of a regular language L if it is characteristic for the minimal DFA of L and if – vq0 – for – –

∈ S + and for every state q such that Lq ⊆ L\{vq0 }, we have uq vq0 ∈ S − , every state q and for every letter x, if q 0 = δ(q, x) then uq xvq0 ∈ S + and for every state q 00 such that Lq00 ⊆ Lq0 \ {vq0 }, uq00 vq0 ∈ S −

Let S be a sample, u0 , . . . , un ∈ pref (S). We note u0 = ⊕{u1 , . . . , un } if ui ≺ u0 for every i = 1 . . . n and if for every word v, u0 v ∈ S + implies that there exists at least one index i > 0 such that ui v 6∈S − . Lemma 2. If u0 , . . . , un ∈ U (L) and if S is a sample of L then −1 u−1 0 L = ∪{ui L|i = 1 . . . n} ⇒ u0 = ⊕{u1 , . . . , un }

Furthermore, if we assume that S is characteristic for the canonical RFSA and that U = {u1 , . . . , un } verify the following property : ∀v ∈ P C(L), v ≺ u0 and ∃u ∈ U v ≤ u ⇒ v ∈ U Then the reciprocal is true. Proof. The necessary part of this statement is straightforward. Let us assume that u0 = ⊕{u1 , . . . , un }. As S is characteristic for the DFA, we have a u−1 i L⊆ −1 u−1 0 L for each i = 1 . . . n because of the previous lemma. If we had ∪{ui L|i = −1 −1 1 . . . n} ( u−1 0 L we would also have ui L ⊆ u0 L \ {vδ(q0 ,u0 ) } for every index i and so ui v0 ∈ S − , which is wrong because of our hypothesis


45

Remarks – ≺, ' and ⊕ operators have the intended properties as long as we require the good conditions for the working sample and as long as we only use words of U (L). This means that, if we have a characteristic sample, we can use those operators to evaluate relations between residual languages ; that is actually what we will do in the learning algorithm presented here, – of course, there exists characteristic samples for the minimal DFA of polynomial size with respect to the size of this DFA ; we can also observe that the RPNI algorithm can find back this minimal DFA from such a sample, – there also exists characteristic samples for RFSA whose cardinal is polynomial with respect to the size of the minimal DFA recognizing the language L ; however, it happens that the smallest characteristic sample for the canonical RFSA recognizing a language L contains words of length exponential with respect to the size of the minimal DFA of this language. Let us note p1 , . . . , pn the first n prime numbers and let us define for every index i the language Li = {ε} ∪ {ak |pi does not divide k}. Let us introduce Sn + 1 extra letters {x0 , . . . , xn } and let us consider the language L = x0 a∗ ∪ {xi Li |i = 1 . . . n}. −1 The residual x−1 0 L is not the reunion of the residuals xi L but the first word p1 ...pn that can show that is a and its length is exponential with respect to the size of the minimal DFA recognizing L. 4.2

The DeLeTe Algorithm

We present here a grammatical inference algorithm that builds a NFA from a sample of a target language L. We show that if this sample is characteristic, the resulting automaton is the canonical RFSA of L. This algorithm is divided in 4 main phases. Suppose that the input sample is characteristic : – the marking phase marks some prefixes of the input sample : each marked word is in P C(L) (lemma 3) and each state of the canonical RFSA of L will correspond to one of those words. – in the saturation phase, we build the prefix tree automaton corresponding to the positive part of the input sample. We establish the ≺ relation between the marked prefixes and their successors. From our hypothesis and lemmas 1 and 2, these relations are correct and the resulting automaton contains the canonical RFSA of L, – in the cleaning phase, we delete un-marked states. Again, we will show that having a good sample allows us to do that without changing the language recognized by the automaton, – in the reduction phase, we delete marked states that are recognized as composed by the algorithm. In these conditions, the resulting automaton is the canonical RFSA of L. If the sample is not characteristic, we can at least show that the resulting automaton is consistant with the sample.

46


Input : a sample S of a language L (we suppose S + is not empty). We note QS = {u0 , . . . , un } prefixes of S + ordered using the usual order. Marking phase Initialisation : let Q˙ = {ε}(= {u0 }) For i from 1 to n Do If ∃j < i and x ∈ Σ such that ui = uj x and uj ∈ Q˙ Then ˙ < i, uk ≺ ui } Let Ei = {uk ∈ Q|k ˙ If ui 6= ⊕Ei Then Add qi to Q. End For We note QM the set Q˙ obtained at the end of this phase. Saturation phase ˙ by setting ˙ Q˙ 0 , F˙ , δi Initialisation : We create an automaton A˙ = hΣ, Q, + ˙ ˙ ˙ ˙ Q = QS , Q0 = {u0 }, F = S and δ(u, x) = {ux ∈ QS } for every word u ∈ QS ¨ = QM ∪ {ux ∈ pref (S + ) | x ∈ Σ, u ∈ QM } and every x ∈ Σ. We note Q For i from 0 to n Do For j from 0 to n Do ¨ ui ≺ uj Then If i 6= j, ui ∈ QM , uj ∈ Q, If uj = ux Then ˙ x) if it does not imply losing consistance with S − Add ui to δ(u, If uj = ε Then Add ui to Q˙ 0 if it does not imply losing consistance with S − End For End For We call As = hΣ, QS , Q0S , FS , δS i the automaton A˙ obtained after this phase. Cleaning phase ˙ = AS ˙ Q˙ 0 , F˙ , δi Initialisation : We set A˙ = hΣ, Q, ˙ For every states u ∈ Q \ QM , Do Suppress u of Q˙ if it does not imply losing the consistance with S + End For We note AC = hΣ, QN , Q0N , FN , δN i the automaton A˙ obtained after this phase. Reduction phase ˙ = AC ˙ Q˙ 0 , F˙ , δi Initialisation : We take A˙ = hΣ, Q, ˙ For every states ui ∈ Q Do If ∀v such that ui v ∈ S + , ∃uk ∈ Q˙ such that uk ≺ ui , uk 6= ui and uk v 6∈S − Then Suppress ui of Q˙ if it does not imply losing the consistance with S + End For We note AR = hΣ, QR , Q0R , FR , δR i the automaton A˙ obtained after this phase. Output : The automaton AR .


4.3

47

Example

Let L = Σ ∗ 0Σ. The minimal DFA that recognizes L has 4 states corresponding to the residual languages L, 0−1 L = L∪Σ, 00−1 L = L∪Σ ∪ε and 01−1 L = L∪ε. All of them are prime except 00−1 L = L ∪ 0−1 L ∪ 01−1 L. 1

-

0

/

0

07

Y 1

0

j

1

/

/

1

Fig. 4. minimal DFA recognizing L

We have P C(L) = {ε, 0, 00, 01} and U (L) = {ε, 0, 1, 00, 01, 000, 001, 010, 011}. Let us study the behaviour of Delete on the following sample : S + = {00, 01, 000, 100, 0000, 0100, 01000, 01100} S − = {ε, 0, 1, 10, 010, 011, 0110} We first mark the words. We can observe for instance that 1 is not marked, as we have 1 ' ε. At the end of the marking phase, we have QM = {ε, 0, 00, 01} ¨ = {ε, 0, 1, 00, 01, 000, 010, 011}. and Q ¨ we’ve got the following non trivial relations : Between words of QM and Q, for every u, ε ≺ u, 000 ' 00, 010 ' 0, 011 ' ε, 1 ' ε, 0 ≺ 00, 01 ≺ 00, 0 ≺ 000, 01 ≺ 000. With those relations, we can build the automaton of figure 5 at the end of the saturation phase. At the cleaning phase, we suppress every un-marked state. As the state 00 is composed of all the others, we can suppress it during the reduction phase and we obtain the automaton of figure 6 at the end of the reduction phase. This automaton is the canonical RFSA of L. 4.4

Results

We suppose in this whole section that S is a characteristic sample of the RFSA. Lemma 3. For every word u ∈ pref (S + ), we have – if u ∈ QM then u ∈ P C(L) – if u ∈ P C(L), then there exists u1 , . . . , ul ∈ QM such that ui ≤ u for every index i and such that u = ⊕{u1 , . . . , ul }.

48


Fig. 5. Output Automaton at the end of the saturation phase 0, 1

-

0

/ i k

0 0, 1

/

q

i 0, 1

0, 1

q

0

Fig. 6. Output automaton of DeLeTe

One can prove both properties at the same time using a recurrence on the length of u (see full proof in [DLT00a]). This lemma does imply two main results : first, as every marked word u is in P C(L), the ≺ and ⊕ relations between those words do correspond to real inclusions and union relations between their representative languages, that is, if u and v are in QM , u ≺ v ⇒ u−1 L ⊆ v −1 L ; second, every prime word of P C(L) is in QM . We can then show that the automaton AS obtained after the saturation phase has an essential property : the language LS recognized by AS is exactly equal to the target language L. We can prove that using the following lemma. Lemma 4. ∀u ∈ QM , u−1 LS = u−1 L. Sketch of proof : (see full proof in [DLT00a]) The proof is mainly based on the fact that, due to the precedent lemma, if we have two words u and v in QM such that u ≺ v, then we have u−1 L ⊆ v −1 L. We first have to show that consistance tests in the algorithm are useless when the sample has the conditions required here, and that we can neglect them in those proofs.


49

We also have to show that our transition function is correct, that is if u ∈ δS (v, x), then we have u−1 L ⊆ (vx)−1 L. That can be proved using the fact that it is true in the prefix tree, and all changes in the transition function keep this property. In particular, this implies that ∀u ∈ QM , u−1 LS ⊆ u−1 L. Furthermore, we can show that the automaton AS contains all the states and transitions of the canonical RFSA of L. So, we can conclude that, at least for words u ∈ QM such that u−1 L is a prime residual of L, we have u−1 L ⊆ u−1 LS . That is not too hard a step to also show it for every other states of QM . t u Theorem 1. If we give a characteristic sample for the canonical RFSA of a language L as input to the DeLeTe algorithm, the automaton given in output is the canonical RFSA of L. Sketch of proof : (see full proof in [DLT00a]) We already know that the automaton AS contains the states and the transitions of the canonical RFSA of L, and it is not hard to prove that the cleaning phase and the reduction phase suppress from this automaton all the states that are not prime. Furthermore, we can show that all the remaining transitions have the properties required in the canonical RFSA and that the remaining initial states are the initial states of the canonical RFSA. t u Despite of this theorem, DeLeTe is not a learning algorithm for regular language from given data as it can happen that the smallest characteristic sample for the canonical RFSA of the target language contains a word whose length is exponential with respect to the size of the minimal DFA recognizing it.

5

Remarks and Conclusion

Using DFA to represent regular languages is too strong a constraint since many regular languages possess shorter representations. That is the case of languages like Σ ∗ 0Σ n for which the minimal DFA has a number of states exponential in n. So, it is impossible for an algorithm like RPNI to have good results with those languages from samples of polynomial size in n. Nevertheless, it is quite easy to show that DeLeTe can find back those languages from a sample of cardinal in O(n2 ) and of size in O(n3 ). For instance, one can verify that the sample {01i 01j , 101i , 01k , 1i |0 ≤ i, j ≤ n + 1, 0 ≤ k ≤ 2n} correctly labeled, of size n2 +O(n) behaves like a strongly representative sample in the sense that DeLeTe calculates exactly the target language from all the examples it contains. The reason of this good behaviour is that those languages have few prime residuals and that they are defined by short words. But DeLeTe is not built to succeed on those “academic” languages. The ideas we propose here shares the same philosophy as the one that motivated the conception of RPNI or that explains its performances. In ideal conditions, it is an algorithm of exact learning ; in less ideal conditions (but not designed to make it fail), it is still a good algorithm for approximative learning. This

50


can surely be explained by the fact that, if an incorrect merge has been made, it is because there was no example to forbid it, and that it could probably have been made anyway without important consequences. This remark is also true when examples are distributed using a probability distribution and when asked performances are at the same time approximative and relative to the used distribution, that is in the PAC framework. This analysis is still basic and we think a deeper analysis is still to be done. The previous remark can be used in our case too. We said in the introduction that the representative sample of a DFA can contain words of exponential length, but in a PAC learning context, it is probably not a serious problem. Indeed, those long words are used, in an exact learning context, to be sure not to replace a residual by the reunion of the residuals it contains when we should not. But if no example forbid that reunion, it is probably not serious to suppose we could do it. All of this need to be formalized and precised : we’d like to find an algorithm that, when examples are distributed using a given probability, gives us a RFSA near to the target relatively to this distribution - that is still up to do. We think the ideas exposed here are original and promising. The present paper is a first step. It shows that it is relevant to study grammatical inference of regular languages using RFSA representation. This work will be carry on.

References [DLT00a] [DLT00b] [GM96] [Gol78] [Hig97] [KV94] [LPP98] [OG92] [Pit89]

[Val84]

F. Denis, A. Lemay, and A. Terlutte. Learning regular languages using non deterministic finite automata. Technical Report 7, 2000. F. Denis, A. Lemay, and A. Terlutt. Les automates finis ` a états résiduels (afer). Technical report, ftp://ftp.grappa.univliller3. fr/pub/reports/after.ps.gz, 2000S.A. Goldman and H.D. Mathias. Teaching a smarter learner. Journal of Computer and System Sciences, 52(2): 255-267, 1996. E.M. Gold. Complexity of automaton identification from given data. Inform. Control, 37:302-320, 1978. Colin De La Higuera. Characteristic sets for polynomial grammatical inference. Machine Learning, 27: 125-137, 1997. M. Kearns and L. Valiant. Cryptographic limitations on learning boolean formulae and finite automata. Journal of the ACM, 41(1):67-95, 1994. K. J. Lang, B. A. Pearlmutter, and R. A. Price. Results of the abbadingo one DFA learning competition and a new evidence-driven state merging algorithm. Lecture Notes in Science, 1433: 1-12, 1998. J. Oncina and P. Garcia. Inferring regular languages in polynomial update time. In Pattern Recognition and Image Analysis, pages 49-61, 1992. L. Pitt. Inductive Inference, DFAs, and Computational Complexity. In Proceedings of AII-89 Workshop on Analogical and Inductive Inference; Lecture Notes in Artificial Intelligence 397, pages 18-44, Heidelberg, October 1989. Springer-Verlag. L.G.Valiant. A theory of the learnable. Commun. ACM, 27(11):1134-1142, November 1984.

Smoothing Probabilistic Automata: An Error-Correcting Approach Pierre Dupont1 and Juan-Carlos Amengual2 1

EURISE, Université Jean Monnet 23, rue P. Michelon 42023 Saint-Etienne Cedex – France [email protected] 2 Universidad Jaume I de Castell´ on Campus de Riu Sec 12071 Castell´ on – Spain [email protected]

Abstract. In this paper we address the issue of smoothing the probability distribution defined by a probabilistic automaton. As inferring a probabilistic automaton is a statistical estimation problem, the usual data sparseness problem arises. We propose here the use of an error correcting technique for smoothing automata. This technique is based on a symbol dependent error model which guarantees that any possible string can be predicted with a non-zero probability. We detail how to define a consistent distribution after extending the original probabilistic automaton with error transitions. We show how to estimate the error model’s free parameters from independent data. Experiments on the ATIS travel information task show a 48 % test set perplexity reduction on new data with respect to a simply smoothed version of the original automaton.

1

Introduction

The goal of learning a probabilistic deterministic finite automaton (PDFA) is to induce a DFA structure from data and estimate its constituent transition probabilities. As the structure itself constrains the probability distribution over the set of possible strings, the inference procedure can be considered to be a single problem of statistical estimation. Several learning algorithms for probabilistic automata have been proposed [16,3,15], but the smoothing issue has not been addressed. In particular, when probabilistic automata are used for modeling real data, as in the case of natural language interfaces, the usual problem of data sparseness arises. In other words only a few strings are actually observed in the training sample and many strings that could be observed receive a zero probability of being generated even after the generalization introduced by the inference algorithm. Smoothing the probability distribution fundamentally requires us to discount a certain probability mass from the seen events and to distribute it over unseen events which would otherwise have a zero probability. Considering that a string with zero probability is a string for which there is no path between the initial A.L. Oliveira (Ed.): ICGI 2000, LNAI 1891, pp. 51–64, 2000. c Springer-Verlag Berlin Heidelberg 2000

52

P. Dupont and J.-C. Amengual

state and an accepting state in a probabilistic automaton1 , error-correcting techniques [2] can be used towards this end. Error-correcting techniques extend automata to allow acceptance of, in principle, any string. Using error correction allows us to compute the probability of accepting the string with minimal error. Several criteria can be used to guide this process. For instance, we can look for the minimal number of editing operations necessary to accept a string. Alternatively, we can search for the accepting path of maximal probability in a probabilistic error-correcting parser. In the latter case, the error model parameters need to be estimated and possibly smoothed as well. Definitions and notations are given in section 2.1. The ALERGIA algorithm, which will be used for PDFA inference, is briefly presented in section 2.2. The criterion for evaluating the quality of a PDFA, that is the perplexity computed on an independent test sample, is detailed in section 2.3. We present in section 3 our baseline smoothing technique using linear interpolation with a unigram model. The formal definition of the proposed error-correcting model and the method for estimating its free parameters are fully described in section 4. Experiments on the ATIS task, a spoken language interface to a travel information database, were performed in order to assess the proposed smoothing techniques. The task is presented in section 5. Finally we show how error-correcting techniques improve the baseline perplexity. These experiments are detailed in section 6.

2

Preliminaries

In this section we detail the formal definition of a probabilistic DFA (PDFA). Next, we review briefly the ALERGIA algorithm which will be used in our experiments to infer PDFA. Finally we present the measure for estimating the quality of PDFA inference and smoothing. 2.1

Definitions

A PDFA is a 5-tuple (Q, Σ, δ, q0 , γ) in which Q is a finite set of states, Σ is a finite alphabet, δ is a transition function, i.e. a mapping from QxΣ to Q, q0 is the initial state, γ is the next symbol probability function, i.e. a mapping from QxΣ ∪{#} to [0, 1]. A special symbol #, not belonging to the alphabet Σ, denotes the end of string symbol. Hence γ(q, #) represents the probability of ending the generation process in state q and q is an accepting state if γ(q, #) > 0. The probability function must satisfy the following constraints: P

γ(q, a) = 0 , if δ(q, a) = ∅, ∀a ∈ Σ γ(q, a) = 1 , ∀q ∈ Q a∈Σ∪{#}

The probability PA (x) of generating a string x = x1 . . . xn from a PDFA A = (Q, Σ, δ, q0 , γ) is defined as 1

We assume here that no existing transition in an automaton has a zero probability.

Smoothing Probabilistic Automata

53

 Qn i n  i=1 γ(q , xi ) γ(q , #)       if δ(q i , xi ) 6= ∅ with q i+1 = δ(q i , xi ), PA (x) = for 1 ≤ i < n and q 1 = q0        0, otherwise The language L(A) generated by a PDFA A is made of all strings with non-zero probability: L(A) = {x | PA (x) > 0} Our definition of probabilistic automaton is equivalent to P a stochastic deterministic regular grammar used as a string generator. Thus, x∈Σ ∗ PA (x) = 1. Note that some work onPthe learning of discrete distributions uses distributions defined on Σ n (that is x∈Σ n P (x) = 1, for any n ≥ 1), instead of Σ ∗ . Let I+ denote a positive sample, i.e. a set of strings belonging to a probabilistic language we are trying to model. Let P T A(I+ ) denote the prefix tree acceptor built from a positive sample I+ . The prefix tree acceptor is an automaton that only accepts the strings in the sample and in which common prefixes are merged together resulting in a tree shaped automaton. Let P P T A(I+ ) denote the probabilistic prefix tree acceptor. It is the probabilistic extension of the P T A(I+ ) in which each transition has a probability proportional to the number of times it is used while generating, or equivalently parsing, the positive sample. 2.2

PDFA Inference

Several inference algorithms for probabilistic automata have been proposed [16, 3,15] but only Carrasco and Oncina’s ALERGIA algorithm, a stochastic extension of the RPNI algorithm [14], is free from the restriction to the learning of acyclic automata. This algorithm has been applied to information extraction from text [7] or structured documents [17], speech language modeling [5] and probabilistic dialog modeling [10]. The ALERGIA algorithm performs an ordered search in a lattice of automata Lat(P P T A(I+ )). This lattice is the set of automata that can be derived from P P T A (I+ ) by merging some states. The specific merging order, that is the order in which pair of states are considered for merging, is explained in detail and fully motivated in [4]. At each step of this algorithm, two states are declared compatible for merging, if the probability of any of their suffixes are similar within a certain threshold α. This parameter α indirectly controls the level of generalization of the inferred PDFA. 2.3

Evaluation Criterion

Evaluation of non-probabilistic inference methods is usually based on correct classification rates of new positive and negative data [12]. In the case of PDFA inference, the model quality can no longer be measured by classification error rate, as the fundamental problem has become the estimation of a probability distribution over the set of possible strings.

54


The quality of a PDFA A = (Q, Σ, δ, q0 , γ) can be measured by the per symbol log-likelihood of strings x belonging to a test sample according to the distribution defined by the solution PA (x) computed on a test sample S:   |S| |x| X X 1 j LL = − log P (xi |q i ) kSk j=1 i=1 where P (xji |q i ) denotes the probability of generating xji , the i-th symbol of the jth string in S, given that the generation process was in state q i . This average loglikelihood is also related to the Kullback-Leibler divergence between an unknown target distribution and the proposed solution by considering the test sample as the empirical estimate of the unknown distribution (see e.g. [5]). The test sample perplexity P P is most commonly used for evaluating language models of speech applications. It is given by P P = 2LL . The minimal perplexity P P = 1 is reached2 when the next symbol xji is always predicted with probability 1 from the current state q i (i.e. P (xji |q i ) = 1) while P P = |Σ| corresponds to random guessing from an alphabet of size |Σ|.

3

Interpolation with a Unigram Model

In this section we present the basic smoothing technique which will serve as our reference model for smoothing probabilistic automata. A unigram model is a probabilistic model in which the probability of any symbol a from Σ is independent from its context. It can be simply estimated by computing the frequency C(a) of a in a training sample containing N tokens. The probability P (a) is given by P (a) =

C(a) N

and the probability P1 (x) of a string x = x1 . . . x|x| is given by P1 (x) =

|x| Y

P (xi )

i=1

In general not all symbols are observed in the training sample and the unigram distribution is smoothed according to a discounting parameter d [13]: ( C(a)−d , if C(a) > 0 N Pb(a) = (1) D otherwise N0 where D is the total discounted probability mass X d D= N {a | C(a)>0}

and N0 is the number of unseen symbols in the training sample 2

Such a perfectly informed model cannot be constructed in general.


X

N0 =

55

1.

{a | C(a)=0}

A smoothed unigram model is guaranteed to assign a non-zero probability to any string which will be denoted Pb1 (x). It is equivalent to the universal automaton built from the alphabet Σ with transitions probabilities defined according to equation (1). If PA (x) denotes the (possibly null) probability assigned to a string x by a PDFA A, a smoothed distribution is obtained by linear interpolation with the smoothed unigram model: Pb(x) = β · PA (x) + (1 − β) · Pb1 (x) , with 0 ≤ β < 1. This smoothing technique is very rudimentary but, because it is so simple, it best reflects the quality of the PDFA itself. This smoothed probabilistic distribution serves as our reference model. In the sequel we study whether errorcorrecting techniques can improve over this reference model, that is whether a probabilistic model with smaller perplexity on independent data can be obtained.

4

Error-Correcting Model

Given A, a PDFA, and its language L(A), error transitions can be added in order to make it possible to accept any possible string from Σ ∗ with a non-zero probability. This error model is fully described in section 4.1. The problem of estimating the error-correcting model free parameters, which are the probabilities of error transitions, is detailed in section 4.2. Once the error model has been estimated from data, there may still be some string which cannot be generated with a non-zero probability. This is due to the fact that some error transitions may not have been seen during the estimation of the error model. Smoothing of the error model is then required as explained in section 4.3. The adaptation of the original PDFA distribution in order to include the error transition probabilities to build a consistent model is described in section 4.4. Once the error model has been defined and its parameters estimated, the probability Pe(x) of generating any string x from the original PDFA extended with error transitions can be computed. An efficient algorithm to compute a path of maximal probability in any probabilistic automaton A (i.e. an automaton possibly including cycles and possibly non-deterministic) was recently proposed [1]. This algorithm is briefly presented in section 4.5. We use this algorithm here to reestimate iteratively the error model as described in section 4.6. 4.1

Model Definition

Our error model is based on the addition of error transitions to an existing PDFA resulting in an extended automaton. These error transitions accounts for the possibility of inserting any symbol at any state, of substituting any existing transition labelled by a symbol a by any other symbol from the alphabet or of deleting the transition (or equivalently substituting a by the empty string λ). Figure 1 illustrates the addition of error transitions to a PDFA. Initially there are only two transitions from state q labeled by a and b, respectively. The original

56


automaton is extended with insertion transitions, substitution transitions and deletion transitions. Note that for this example the alphabet is assumed to have b two symbols, Σ = {a, b}. a q

b

b a

a

q

b

λ

a

λ (a) Original automaton

(b) Extended automaton

Fig. 1. Addition of error transitions to a PDFA

The parameters of the general error model are the following: – P (λ → a | q) which denotes the probability of inserting symbol a while being in state q – P (a → b | q, q 0 ) which denotes the probability of substituting a by b while going from q to q 0 . In particular P (a → a | q, q 0 ) denotes the probability of substituting a by a, that is of taking the original transition labeled by a from state q. – P (a → λ | q, q 0 ) which denotes the probability of deleting a while going from q to q 0 . Estimating an error model consists of estimating the error transitions probabilities. In order to minimize the number of free parameters, these probabilities can be made dependent on the symbol but independent of the transitions (or the state) they apply to. The parameters of the symbol dependent error model now become: – P (λ → a) which denotes the probability of inserting symbol a in any state. – P (a → b) which denotes the probability of substituting a by b while taking any transition labeled by a. – P (a → λ) which denotes the probability of deleting a while taking any transition labeled by a. Alternatively, the error model can be made state dependent instead of symbol dependent. In our case, we adopt a symbol dependent error model as the alphabet is usually known before the automaton inference process. State independence also allows us to merge several error models as described in section 6.4. 4.2

Estimation of an Error Model

Once a PDFA is given or inferred from a training sample, the parameters of the error model can be estimated on an independent sample. For any string x from this independent sample, the probability of generating the string can be computed. This requires that a consistent probability distribution can be defined for


57

the extended automaton as detailed in section 4.4. Note also that after the extension of the original automaton with error transitions, the new automaton is no longer deterministic. Following a Viterbi criterion, the probability of generating x can be approximated by the probability of the most likely path to generate x. An efficient algorithm to compute this path is described in section 4.5. The set of editing operations used while generating the independent sample from the extended automaton can be stored and the associated counts can be computed: – C(λ, a) denotes the number of insertions of the symbol a. – C(a, b) denotes the number of substitutions of the symbol a by b. In particular, C(a, a) denotes the number of times the symbol a was asserted, that is, not substituted while parsing the independent sample. – C(a, λ) denotes the number of deletions of the symbol a. – C(#) denotes the number of (end of) strings. As the proposed error model is state independent several estimates of its parameters for various underlying automata can be computed. Combining these estimates simply amounts to summing the respective error counts. This property will be used in our experiments as explained in section 6.4. 4.3

Smoothing of the Error Counts

Some counts associated with error transitions may be null after estimating the error model. This is the case when some error transitions are never used along any of the most likely paths computed while parsing the independent sample. This problem can be solved by adding real positive values to the error counts. We use four additional parameters εins , εsub ,εdel and εnoerr to smooth the error counts : – – – – 4.4

b a) = C(λ, a) + εins C(λ, b b) = C(a, b) + εsub if a 6= b C(a, b C(a, a) = C(a, a) + εnoerr b λ) = C(a, λ) + εdel C(a, Definition of the Extended PDFA Distribution

In the original PDFA P (a | q) = γ(q, a) denotes the probability of generating the symbol a from state q whenever such a transition exists. This transition probability can be estimated from the training sample from which the PDFA was built. The maximum likelihood estimate for γ(q, a) is given by γ(q, a) =

Cq (a) Cq

where Cq (a) denotes the number of times the symbol a was generated from state q and Cq denotes the number of times the state q was observed while parsing the training sample. We can assume that the counts Cq (a) and Cq are strictly

58


positive, as any transition (or state) which would not satisfy this constraint would be initially removed from the original PDFA. The probability distribution of the extended automaton can be defined as follows. The total insertion count Cins is defined as X b a) C(λ, Cins = a∈Σ

and its complementary count Cins is defined as Cins =

X

X

b b) C(a,

+ C(#)

a∈Σ b∈Σ∪{λ}

ins denote the probability of inserting any symbol. The Let Pins = CinsC+C ins probability of the transitions from any state q in the extended automaton are computed as follows:

– the probability of inserting a while being in state q: b a) C(λ, . P (λ → a|q) = P (λ → a) = Pins · Cins

(2)

– the probability of substituting a by b, for any symbol b in the alphabet Σ (including the case where a = b), from state q : " # b b) C(a, . P (a → b|q) = P (a → b) · γ(q, a) = (1 − Pins ) · P · γ(q, a) b b∈Σ∪{λ} C(a, b) (3) – the probability of deleting a from state q : # " b λ) C(a, . · γ(q, a) P (a → λ|q) = P (a → λ) · γ(q, a) = (1 − Pins ) · P b b) C(a, b∈Σ∪{λ}

– the probability of generating the end of string symbol # from state q : . P (# | q) = (1 − Pins ) · γ(q, #) 4.5

(4) (5)

Computation of the Most Likely Path

The general problem of finite-state parsing with no error correction can be formulated as a search for the most likely path or equivalently the minimum cost3 path through a trellis diagram associated to the PDFA A and the string x to be parsed. This trellis is a directed acyclic multistage graph, where each node qkj corresponds to a state qj in a stage k. The stage k is associated with a symj bol xk of the string to be parsed and every edge of the trellis tk = (qki , qk+1 ) 3

Sums of negative log probabilities rather than products of probabilities are used.


59

stands for a transition between the state qi in stage k and the state qj in stage k + 1 (Fig. 2 (a)). Thanks to the acyclic nature of this graph, dynamic programming can be used to solve the search problem, leading to the well-known Viterbi algorithm [6].

(a) K

(b) K+1

K

(c) K+1

K

(d) K+1

K

K+1

Fig. 2. Trellis with: a) Substitution and proper PDFA transitions b) Insertion transitions c) Deletion transitions in an acyclic PDFA d) Deletion transitions in a cyclic PDFA. Every edge is labeled with a symbol of Σ.

The trellis diagram can be extended in a straightforward fashion to parse errors produced by substitution and insertion actions. Efficient error correcting parsing can be implemented because such an extended trellis diagram still has the shape of a directed acyclic multistage graph (Fig. 2 (a),(b)). However, the extension of the trellis diagram to parse errors produced by deletion of one or more (consecutive) symbol(s) in the original string results in a graph form that includes edges between nodes belonging to the same stage k (Fig. 2 (c)). In particular, when the automaton A has cycles dynamic programming can no longer be used as the problem becomes one of finding a minimum cost path through a general directed cyclic graph (Fig. 2 (d)). As noted in [8], we can still take advantage of the fact that most edges, for this kind of graph, still have a left-to-right structure and consider each column as a separate stage like in the Viterbi algorithm. An efficient algorithm for computing the most likely acceptance path that includes error operations in general automata was proposed in [1]. This algorithm can be considered as an extension of the Viterbi algorithm. The main difference lies in the fact that an order has to be defined when parsing deletion transitions (see Fig. 2 (c), (d)) for adequately performing the required computations during local (state) minimizations. In particular, it is based on the definition of a pseudo-topological state ordering, that is an extension to cyclic graphs of the usual topological ordering. This pseudo-topological ordering is computed and efficiently stored in a hash table during a preprocessing stage which detects the backward edges, i.e. those edges which produce cycles in A. This leads to a fixed order for the traversal of the list of nodes (states of the PDFA) at any stage of the parsing process in order to update the cumulated costs whenever required.

60


Full details of the computation of this state ordering, the resulting parsing algorithm and practical evaluations are presented in [1]. We use here this algorithm to compute the most likely path of generating a string from the extended PDFA. 4.6

Greedy Reestimation of the Smoothed Distribution

Computing the most likely path using the technique described in section 4.5 is equivalent to computing the path of minimum cumulated cost. For example the cost Dq (a → b) of substituting a by b from state q is given by Dq (a → b) = − log P (a → b|q). Thus the maximization of a product of probabilities becomes a minimization of additive costs. The initial error model can not be derived from the probabilistic error model described in section 4.4, as the error counts are initially unknown and the extended (smoothed) PDFA distribution can not be computed. However a set of editing costs can be defined a priori, for instance according to the Levenshtein distance [11]: Dq (λ → a) = 1, Dq (a → b) = 1 if a 6= b, Dq (a → a) = 0 and Dq (a → λ) = 1. Once the initial editing costs are defined, the counts of insertions, substitutions and deletions that minimize the Levenshtein distance criterion on an independent sample can be computed as described in section 4.1. Note that, in this particular case, only the structure of the PDFA is required. A new error model can then be derived from these error counts, and this estimation can be iterated with a true probabilistic error model. This reestimation process is performed until a maximum number of iterations is reached (typically 10) or until the relative change of perplexities computed on two consecutive iterations falls below a certain threshold (typically 1%). During this iterative procedure, the original PDFA distribution can also be reestimated by adding to the original counts, Cq and Cq (a), their values computed on the independent sample and by modifying accordingly the estimate of γ(q, a). This will be referred to as reestimation of non-error transitions.

5

The ATIS Task

The Air Travel Information System (ATIS) corpus [9] was developed under a DARPA speech and natural language program that focussed on developing language interfaces to information retrieval systems. The corpus consists of speakers of American English making information requests such as, “Uh, I’d like to go from, uh, Pittsburgh to Boston next Tuesday, no wait, Wednesday”. Each user was given several goal scenarios to work with, in which he or she had to try to make travel arrangements between multiple cities in North America. A database containing information from the Official Airline Guide was at the heart of the system. Users could ask questions about a wide variety of items in the database, ranging from flight information to aircraft equipment descriptions and even meals served on particular flights. They could speak naturally to the machine, as there was no fixed interaction language or required sequence of events. Spoken language phenomena such as truncated words, hesitations, false starts, and verbal error recovery are common in the corpus. It is commonplace


61

to find multiple turn interactions (and thus multiple utterances from a user) between the user and machine for solving each scenario.

6 6.1

Experiments Data Sets

We use the ATIS-2 sub-corpus in the experiments reported here. This portion of the corpus was developed under Wizard-of-Oz conditions in which a human being secretly replaced the speech recognition component of an otherwise fully automated dialogue system. The ATIS-2 collection is officially defined as containing a training set and two evaluation sets. The training set, which we used for inferring PDFAs, contains 13,044 utterances (130,773 tokens). The vocabulary contains 1,294 words. We used the first evaluation set (Feb92, 974 utterances, 10636 tokens) as a validation set to estimate the baseline perplexity and an error model. The second evaluation set (Nov92, 1001 utterances, 11703 tokens) was used as our independent test set. In the context of these experiments, alphabet symbols represent words from the ATIS vocabulary and strings represent utterances. 6.2

Baseline Perplexity

A PDFA is inferred from the training set using the ALERGIA algorithm. The resulting PDFA consists of 414 states and 12,303 transitions. It accepts 55 % (532 strings) of the validation set, illustrating the need for smoothing the PDFA distribution. In particular the validation set perplexity is infinite without smoothing. Figure 3(a) shows the perplexity obtained after smoothing by interpolating with a unigram model as explained in section 3. The optimal perplexity (70) is obtained for β equal to 0.5. 6.3

Validation Set Perplexity with Error Model

The initial error model parameters are estimated from training and validation sets by counting the observed editing operation frequencies so as to minimize the Levenshtein distance (see section 4.6). As some error transitions are not observed during this process, the initial error table is then smoothed (see section 4.3). The additional smoothing parameters (εins , εsub , εdel and εnoerr ) are adjusted in order to minimize the perplexity on the last 10 % of the validation set while estimating the error model only on the first 90 % of the validation set. Their optimal values are εins = 0.1, εsub = 0.1, εdel = 0.1 and εnoerr = 0.0. Figure 3(b) shows the perplexity obtained on the validation set during reestimation of the error model. The initial perplexity (41) is achieved after the initial estimation of error parameters, based on the counts of the editing operations which minimize Levenshtein distance. In the first case (type I model), only error transitions are reestimated resulting in a 10% relative perplexity improvement (from 41 to 37). In the second case (type II model), error and non-error transitions probabilities are reestimated. The perplexity obtained after 10 iterations is 28.

62


(a)

6.4

Fig. 3. Perplexity results

(b)

Estimating the Error Model by Cross-Validation

In the experiments described in section 6.3, the error model was constructed and reestimated on the validation set. The training set, which represents about 13 times more data, was not used for estimating the error model as the original automaton is guaranteed to accept all training strings without errors. However a better estimate of the error model can be obtained using cross-validation. This procedure can be summarized as follows: – Concatenate training and validation set in a single data set. – Construct N (typically 10) different partitions of this data set. – For each partition, infer a PDFA on the first part (typically 90 % of the data set) and estimate an error model on the second part (typically the remaining 10 %) following the greedy procedure described in section 4.6. – Merge all error models by summing up the error counts obtained on each partition. Merging of several error models is simple in our case, as these models are symbol dependent but do not depend on the structure of the underlying automaton. Once the error model is estimated by cross-validation, a final reestimation on the validation set can be performed using the original automaton constructed on the training set only. 6.5

Independent Test Set Perplexity

Table 1 summarizes the final results computed on an independent test set. The reference model is a PDFA interpolated with a unigram using the optimal interpolation parameter estimated on the validation set (β = 0.5). Type I error model refers to the model obtained after reestimating only the error transition probabilities on the validation set. Type II error model refers to the model obtained after reestimating both error and non error transitions probabilities. In both cases the error model probabilities may be simply computed on the validation set or can be estimated by cross-validation (CV) following the procedure described in section 6.4.

Smoothing Probabilistic Automata Table 1. Test set perplexity Model Unigram smoothing (β = 0.5) Type I error model Type II error model CV + Type I error model CV + Type II error model

63

Perplexity 71 40 41 37 37

The reestimation of non-error transitions does not improve the perplexity of the extended PDFA on an independent test set. The significant perplexity decrease on the validation set, as seen in figure 3(b), is thus a result of overfitting to the validation data. On the other hand, cross-validation allows for up to 10 % relative perplexity reduction. Finally these results show a 48 % relative perplexity reduction as compared to the perplexity obtained by interpolating with a unigram model.

7

Conclusions and Future Work

We have examined the issues of smoothing probabilistic automata by adding error transitions to an original probabilistic automaton structure. The probability distribution of the extended automaton is such that any possible string can be predicted with non-zero probability. We explained how to define a consistent error model and how to estimate its free parameters from independent data. Practical experiments on the ATIS travel information task show a 48 % test set perplexity reduction on new data with respect to a simply smoothed version of the original automaton.These experiments illustrate the risk of overfitting when both the error model and the initial non error transitions are reestimated. On the other hand, cross-validation allows us to estimate a more reliable error model which results in significant perplexity reduction on new data. The error model proposed here is symbol dependent but state independent. In particular, the probability of inserting a given symbol a does not depend on where this symbol is inserted. In order to refine the error model without significantly increasing the number of free parameters, the relative weight of error versus non-error transitions could also be estimated for each state. We presented here the error correcting approach as a method for extending a probabilistic deterministic automaton. Most existing inference algorithms produce deterministic machines which after extension with error transitions become non-deterministic. The techniques presented here handle this non-determinism. Thus, smoothing of automata which are non-deterministic from the start is also something we can pursue. Clustering alphabet symbols before PDFA inference was shown to reduce perplexity on new data [5]. Combination of this technique with error correcting will also be investigated in the future.

References 1. J.-C. Amengual and E. Vidal. Efficient error-correcting viterbi parsing. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-20(10), October 1998.

64


2. J.-C. Amengual, E. Vidal, and J.-M. Benedí. Simplifying language through errorcorrecting techniques. In International Conference on Spoken Language Processing, pages 841–844, 1996. 3. R. Carrasco and J. Oncina. Learning stochastic regular grammars by means of a state merging method. In Grammatical Inference and Applications, ICGI’94, number 862 in Lecture Notes in Artificial Intelligence, pages 139–150, Alicante, Spain, 1994. Springer Verlag. 4. R. Carrasco and J. Oncina. Learning deterministic regular gramars from stochastic samples in polynomial time. Theoretical Informatics and Applications, 33(1):1–19, 1999. 5. P. Dupont and L. Chase. Using symbol clustering to improve probabilistic automaton inference. In Grammatical Inference, ICGI’98, number 1433 in Lecture Notes in Artificial Intelligence, pages 232–243, Ames, Iowa, 1998. Springer Verlag. 6. G.D. Forney. The Viterbi algorithm. IEEE Proceedings, 3:268–278, 1973. 7. D. Freitag. Using grammatical inference to improve precision in information extraction. In Workshop on Automata Induction, Grammatical Inference, and Language Acquisition, Fourteenth International Conference on Machine Learning, Nashville, Tennessee, 1997. 8. G.W. Hart and A. Bouloutas. Correcting dependent errors in sequences generated by finite-state processes. IEEE Trans. on Information Theory, 39(4):1249–1260, July 1993. 9. L. Hirschman. Multi-site data collection for a spoken language corpus. In Proceedings of DARPA Speech and Natural Language Workshop, pages 7–14, Arden House, NY, 1992. 10. K. Kita, Y. Fukui, M. Nagata, and T. Morimoto. Automatic acquisition of probabilistic dialogue models. In Proceedings of ISSD96, workshop of the International Conference on Spoken Language Processing, pages 196–199, Philadelphia, October 1996. 11. J.B. Kruskal. An overview of sequence comparison. In D. Sankoff and J.B. Kruskal, editors, Time Warps, String Edits, and Macromolecules: the Theory and Practice of Sequence Comparison, pages 1–44. Addison-Wesley, Reading, Massachusetts, 1983. 12. K.J. Lang, B.A. Pearlmutter, and R.A. Price. Results of the abbadingo one DFA learning competition and a new evidence-driven state merging algorithm. In Grammatical Inference, number 1433 in Lecture Notes in Artificial Intelligence, pages 1–12, Ames, Iowa, 1998. Springer-Verlag. 13. H. Ney, U. Essen, and R. Kneser. On structuring probabilistic dependences in stochastic language modelling. Computer Speech and Language, 8:1–38, 1994. 14. J. Oncina and P. Garc´ıa. Inferring regular languages in polynomial update time. In N. Pérez de la Blanca, A. Sanfeliu, and E.Vidal, editors, Pattern Recognition and Image Analysis, volume 1 of Series in Machine Perception and Artificial Intelligence, pages 49–61. World Scientific, Singapore, 1992. 15. D. Ron, Y. Singer, and N. Tishby. On the learnability and usage of acyclic probabilistic automata. In Proceedings of the Eighth Annual Conference on Computational Learning Theory, pages 31–40, Santa Cruz, CA, 1995. ACM Press. 16. H. Rulot and E. Vidal. An efficient algorithm for the inference of circuit-free automata. In G. Ferratè, T. Pavlidis, A. Sanfeliu, and H. Bunke, editors, Advances in Structural and Syntactic Pattern Recognition, pages 173–184. NATO ASI, Springer-Verlag, 1988. 17. M. Young-Lai and F. Tompa. Stochastic grammatical inference of text database structure. To appear in Machine Learning, 2000.

Inferring Subclasses of Contextual Languages J.D. Emerald, K.G. Subramanian, and D.G. Thomas Department of Mathematics Madras Christian College Madras - 600 059, INDIA

Abstract. In this paper, we show that the languages generated by internal contextual grammars are not inferable from positive data only. We define two subclasses of internal contextual languages, namely, k-uniform and strictly internal contextual languages which are incomparable classes and provide an algorithm to learn these classes. The algorithm can be used when the rules are applied in a parallel mode. Keywords : internal contextual grammars, strict, k-uniform, identification in the limit from positive data.

1

Introduction

Contextual grammars of Marcus [5] originated in an attempt to translate the central notion of context from the analytical models into the framework of generative grammars. Basically, a contextual grammar gives rise to a language as follows: starting with a given finite set of strings, called axioms, pairs of strings, called contexts, associated to sets of words, called selectors, are added iteratively to the strings already obtained. Among different variations of contextual grammars, we are concerned here with internal contextual grammars [2,3]. It is known that internal contextual grammars with maximal use of selectors are the most appropriate to model the generative capacity of natural languages because they are able to describe all the usual restrictions appearing in such languages. In this paper, we show that the class of internal contextual languages is not inferable from positive data only. Thus, it is natural to look for subclasses of these languages which can be identified in the limit from positive examples only. Motivated by this, two such subclasses are introduced here and inference procedures are provided for learning these classes in this framework. In addition, another variation of internal contextual grammar, which requires a parallel mode of derivation, is also introduced and inference procedure for this class is indicated.


66

2

J.D. Emerald, K.G. Subramanian, and D.G. Thomas

Perliminaries and Definitions

For a language L ⊆ Σ ∗ , the length set of L, is length (L) = {|x|/x ∈ L}. We recall that for x, y ∈ Σ ∗ , y is said to be a subword of x, if x = x1 yx2 , for x1 , x2 ∈ Σ ∗ . The set of all subwords of a string x is denoted by sub(x). If w = xy then x−1 w = y and wy −1 = x. Definition 2.1 [3] An internal contextual grammar (ICG) is a triple G = (Σ, A, P ), where Σ is an alphabet, A is a finite set of strings over Σ, and P is a finite set of pairs (z, u$v) where z, u, v are strings over Σ and $ is a reserved symbol not in Σ. The elements of A are called axioms, those in P are called productions. For a production π = (z, u$v), z is called the selector of π and u$v (or the pair (u,v)) is the context of π. For x, y ∈ Σ ∗ , we define the derivation relation x ⇒ y (with respect to G) if and only if there is a production π ∈ P, π = (z, u$v), such that x = x1 zx2 and y = x1 uzvx2 , for some x1 , x2 ∈ Σ ∗ . (The context (u,v) is adjoined to a substring z of x provided this substring is the selector of a production whose context is u$v). ⇒∗ denotes the reflexive and transitive closure of the relation ⇒. The language generated by G is L(G) = {x ∈ Σ ∗ /w ⇒∗ x for some w ∈ A}. We use ICL to denote the family of all languages generated by internal contextual grammars. Clearly, the productions (z, u$v) with u = v = λ have no effect. Example 2.1 L1 = {an bn am bm /n, m ≥ 1} is an ICL generated by an ICG G1 = ({a, b}, {abab}, {(ab, a$b)}).

3

A Negative Result for Identifiability of ICL

We first consider the learnability of the class of internal contextual languages in the limit from positive data and show using the result of Angluin [1] that C, the class of ICL is not learnable from positive data only. Theorem 3.1 [1] If an indexed family of nonempty recursive languages is inferable from positive data, then there exists, on any input i, i ≥ 1, a finite set of strings Ti such that Ti ⊆ Li and for all j ≥ 1, if Ti ⊆ Lj , then Lj is not a proper subset of Li . Theorem 3.2 The class of internal contextual languages C, is not inferable from positive data only.

Inferring Subclasses of Contextual Languages

67

Proof : We derive a contradiction with Theorem 3.1. Consider the language L = {cbc} ∪ {can ban c/n ≥ 1}. L is in C, since it can be generated from the axiom cbc with production (b, a$a). Let T be any nonempty finite subset of L, and let T 0 = T − {cbc}. In fact, let T 0 = {can1 ban1 c, ..., canp banp c} where ni ≥ 1. Consider an internal contextual grammar H with axiom cbc and with production {(b, an1 $an1 ), ..., (b, anp $anp )}. We have T 0 ⊆ L(H) ⊆ L contradicting Theorem 3.1. Remark : It is observed in [3] that internal contextual grammars are equivalent to pure grammars [6] which generate a class of languages included in the context-sensitive family and hence ICL ⊂ CS. Koshiba et al.[4] have shown that pure contextfree languages are not inferable from positive data only. As a consequence, pure languages are also not inferable from positive data only. In view of this, Theorem 3.2 is of significance, as ICL is a subclass of the family of pure languages.

4

Subclasses of Internal Contextual Grammars

We now define a strictly internal contextual grammar and a k-uniform internal contextual grammar. Definition 4.1 A strictly internal contextual grammar (SICG) is a 6-tuple G = (Σ, S, C, F, w, P ) where i) Σ is the alphabet. ii) S, C and F are sets of strings over Σ (i.e., S, C, F ⊆ Σ ∗ ) called Selector, Context and Factor sets respectively, such that for any u, v ∈ S ∪ C ∪ F , f irst(u) 6= f irst(v) and u is not a subword of v and vice versa, where f irst(w) denotes the first letter of the string w. iii) w ∈ (S ∪ F )∗ is the axiom such that w = w1 w2 ...wn where wi ∈ S or wi ∈ F , 1 ≤ i ≤ n and wi 6= wj , 1 ≤ i, j ≤ n. iv) P is a finite set of pairs (z, u$v) called productions where z ∈ S and u, v ∈ C ∪ {λ}, u 6= v such that there is at most one production for each z ∈ S. The language generated by a strictly internal contextual grammar (SICG) is called a strictly internal contextual language (SICL). Example 4.1 The grammar G2 = ({a, b, c}, {abca}, {(abca, bb$cc)} is a SICG generating the language L2 = {(bb)m abca(cc)m /m ≥ 0}.

68


Definition 4.2 An internal contextual grammar is called a k-uniform internal contextual grammar, (k − U ICG), k ≥ 1, if the elements in axiom A are of length mk, for some m ≥ 1, and if in each production (z, u$v), i) |z| = |u| = |v| = k ii) for any w0 ∈ A, |x| = mk for some m ≥ 1, where x = z −1 w0 or x = w0 z −1 for some z with (z, u$v) in P . iii) for each selector, there is at most one production iv) given any axiom w = w1 w2 ...wm , |wi | = k, 1 ≤ i ≤ m, in A, there is no rule of the form (x, u$v) with either u = wi or v = wi for any i(1 ≤ i ≤ m). The language generated by a k-UICG is called a k-uniform internal contextual language (k-UICL). Example 4.2 The grammar G3 = ({a, b, c}, {ab}, {(ab, ba$bc)}) is a 2-UICG generating the language L3 = {(ba)m ab(bc)m /m ≥ 0}. Proposition 4.1 The class of mono-PCF languages is a subclass of the class of internal contextual languages. Proof : Inclusion can be seen as follows: Every mono-PCF language L is a finite union of languages of the form {xun bv n y/n ≥ 1}, where x, u, v, y are strings and b is a symbol [8]. We can construct an internal contextual grammar to generate the language {xun bv n y / n ≥ 1} as follows: G = (Σ, {xubvy}, {(b, u$v)}). Thus L is an internal contextual language, being a finite union of such languages. Proper inclusion follows from the fact that the internal contextual language L = {an bn am bm /n, m ≥ 1} generated by an internal contextual grammar G = ({a, b}, {abab}, {(ab, a$b)} is not a mono-PCF language. Proposition 4.2 The class of k-uniform internal contextual languages is incomparable with the class of strictly internal contextual languages.


69

Proof : i) The language L3 given in example 4.2 is a 2-uniform internal contextual language. But it can be seen that it cannot be generated by any strictly internal contextual grammar. ii) The strictly internal contextual language L2 given in example 4.1 cannot be generated by any k-uniform internal contextual grammar, for the requirement of |z| = |u| = |v| = k cannot be met with for any rule (z, u$v). iii) The language L4 = {(ba)m ab(cb)m /m ≥ 0} generated by the grammar G4 = ({a, b, c}, {ab}, {(ab, ba$cb)} is both a strictly internal contextual language and a 2-uniform internal contextual language. It is commonly asserted that natural languages are not context-free. This assertion is based on the existence of some restrictions in natural as well as artificial languages outrunning the context-free barrier. Therefore, different ways of adjoining contexts in order to capture these features have been defined. One such approach is considering an internal contextual grammar working in a parallel derivation mode i.e., contexts are adjoined in parallel [7]. We now define a variation in the two classes considered, by requiring a parallel mode of derivation i.e., contexts are adjoined simultaneously to all the selectors in a word. Definition 4.3 Parallel derivation for a k-uniform internal contextual grammar or a strictly internal contextual grammar, can be defined in the following manner: x ⇒p y if and only if x = x1 z1 x2 z2 ...xm zm xm+1 and y = x1 u1 z1 v1 x2 ...xm um zm vm xm+1 where (zi , ui $vi ) ∈ P . Example 4.3 The language L5 = {ba(eb)m ab(f a)m db(gb)m cb(hc)m ra/m ≥ 0} is generated by a strictly internal contextual grammar G5 = ({a, b, c, d, e, f, g, h, r}, {baabdbc bra}, (ab, eb$f a), (cb, gb$hc)), with parallel mode of derivation. In fact the language generated is context-sensitive.

5

Identification of Subclasses of Internal Contextual Languages

Since the class of internal contextual languages is not inferable from positive data only, we show that certain subclasses of internal contextual languages namely the class of strictly internal contextual languages and the class of k-uniform internal contextual languages are inferable from positive data using an identification algorithm to learn these languages. For simplicity, we consider the algorithm which learns these grammars with a single axiom and non-empty contexts. The

70


idea behind the algorithm can be informally described as follows: The sample words are first factored using their factorization and then compared to obtain the context rules using the procedure CONTEXT i.e., in the case of k-uniform ICL, the sample words are factored into factors of length k while in the case of strictly ICL, the words are factored according to the subprocedure UPDATE [9], where the first letters of the factors are all different. While comparing the factorized words x and y, the maximum common prefix and suffix factors of the words are removed and new words x1 and y1 are obtained from x and y respectively. x1 and y1 are then compared to find the first occurrence of the maximum prefix factor of word x1 in y1 (assuming |x1 | < |y1 |). This maximum prefix is conjectured to be the selector of the rule and the factors to the left and the same number of factors to the right of this maximum prefix in y are conjectured as the contexts of the rules. The same procedure is repeated with the remaining factors of the words, and the algorithm terminated when the correct conjecture is given as output. Notations |w| denotes the number of letters in the string w; #(w) denotes the number of factors in w, in a given factorization; Assuming #(x) < #(y) we use the following notations: common-pref (x,y) denotes the maximum common prefix factors of x and y. common-suf (x,y) denotes the maximum common suffix factors of x and y. max-pref (x,y) denotes the first occurrence in y of the maximum prefix factors of x, in considering factor subwords of y from left to right. remove-pref (u, δ) = u0 if u = δu0 . remove-suf (v, δ) = v 0 if v = v 0 δ. When max-pref(x, y) is well defined, we can describe x = max-pref(x, y)u where u ∈ Σ ∗ and u = (u1 )(u2 )...(uk ), k ≥ 1; and y = v max-pref(x, y)v 0 where v, v 0 ∈ Σ ∗ and 0 )n, m ≥ 1 v = (v1 )(v2 )...(vn ); v 0 = (v10 )(v20 )...(vm rem(x) = (u1 )(u2 )...(uk ) rem-left(y, max-pref(x, y)) = (v1 )(v2 )...(vn ) 0 ) rem-right(y, max-pref(x, y)) = (v10 )(v20 )...(vm first(w) is the first letter of w. factorize(w,T) is a function which factorizes the word w over the set T of factors obtained from calling the subprocedure UPDATE given in [9]. Algorithm A Input: A positive presentation of a strictly internal contextual language and its alphabet S. Output: a sequence of SICG for SICL. Procedure:


let C = φ {/* C is the set of contexts */} let P = {(a1 , λ$λ), (a2 , λ$λ), ..., (am , λ$λ)} where Σ = {a1 , a2 , ..., am } read the first positive example w1 ; let T1 = {w1 }; let w10 = factorize (w1 , T1 ); let A = {w1 }; let Γ = {w1 }; output G1 = (Σ, P, A) let i = 2 repeat (forever) begin [i-th stage] let Gi−1 = (Σ, P, A) be the (i − 1)th conjectured SICG let a be the unique element in A read the next positive example wi let Ti = UPDATE (Ti−1 , wi ); for n = 1 to i do let wn0 = factorize(wn , Ti ); 0 let a = factorize(a, Ti ); if wi0 ∈ L(Gi−1 ) then output Gi (= Gi−1 ) as the i-th conjecture else begin if |wi | ≤ |a| then begin call CONTEXT (P, wi0 , a0 ); A = {wi } end else for all w ∈ Γi−1 do the following begin if |wi | < |w| then call CONTEXT (P, wi0 , w0 ) else call CONTEXT (P, w0 , wi0 ) end end end let Γi = Γi−1 ∪ {wi } output Gi = (Σ, P, A) as the i-th conjecture; i=i+1 CONTEXT (P, u, v) begin let P = {(a1 , ta1 $t0a1 ), (a2 , ta2 $t0a2 ), ...(am , tam $t0am )} let f = #(u); let g = #(v); if common-pref (u,v) = φ and common-suf (u,v) = φ then call PRE-RULE (u,v) else begin if ω = common-pref (u,v)

71

72


then

begin let u = remove-pref (u,ω); v = remove-pref (v,ω); end if ω 0 = common-suf (u,v) then begin let u = remove-suf(u,ω 0 ); let v = remove-suf (v,ω 0 ); end call PRE-RULE (u,v) end PRE-RULE (u,v) let max-pref (u,v) = α; if α 6= φ then begin let rem(u) = α0 ; let h = #(α); let rem-left (v,a) = x; let j = #(x); let β = (v1 )(v2 )...(v2j+h ) where β = xαx0 ; #(x) = #(x0 ); let β 0 = v2j+h+1 ...vg let rem-left (β, α) = δ; let rem-right (β, α) = δ 0 if α 6∈C then begin case (γ = γ 0 αγ 00 ) for some γ in C replace (γ, tγ $t0γ ) by (α, γ 0 $γ 00 ) let C = C − {γ} ∪ {α} case (α = α1 γα2 ) for some γ in C call Pre-rule (P, γ, α) case (γ 6= γ 0 αγ 00 ) ∧ (α 6= α1 γα2 ) for all γ in C let P = P ∪ {(α, δ$δ 0 )} let C = C ∪ {α} end else begin if (#(δ) < #(tα )) ∧ (#(δ 0 ) < #(t0α )) (except when δ = δ 0 = λ) then replace (α, tα $t0α ) by (α, δ$δ 0 ) in P end if {((α0 6= λ) ∧ (β 0 6= λ)) then if common-pref (α0 , β 0 ) = φ then call Pre-rule (P, α0 , β 0 ) (assuming |α0 | < |β 0 |) else if ω = common-pref (α0 , β 0 ) then begin let α0 = remove-pref(α0 , ω);


end

end

73

let β 0 = remove-pref(β 0 , ω); call Pre-rule (α0 , β 0 ) (assuming |α0 | < |β 0 |) end

Remark : The above algorithm can also be used to identify a k-uniform internal contextual grammar. A modification required in the algorithm is that k is also given along with the positive presentation as input to the algorithm and instead of the function factorize(w), we use the function split (w,k) where split(w,k) = (w1 )(w2 )...(wn ), |wi | = k, 1 ≤ i ≤ n. A strictly and k-uniform contextual grammars with derivations in parallel mode can also be inferred using the above algorithm. Correctness of the Algorithm and Characteristic Sample The correctness of the algorithm can be noticed in view of the fact that the specific features of the subclasses considered allow the positive examples to have a unique factorization. Indeed, the factors allow us to infer the rules correctly. Also, it can be seen that the algorithm runs in time polynomial in the sum of the lengths of the examples provided. The correctness of the algorithm A, can be seen by considering a characteristic sample for a target strictly internal contextual language. Let L be a strictly internal contextual language. A finite set S is called a characteristic sample of L if and only if L is the smallest SICL containing S. We illustrate the method of forming a characteristic sample, with an example. Consider a SICG, G = {Σ, S, C, F, w, P } where Σ = {a, b, c, d, e, f, g, h}, S = {ab, dc}, C = {ebd, f c, gb, hcd}, F = {ba, cb}, w = baabcbdc and P = {(ab, ebd$ f c), (dc, gb$hcd)} generating a language L = {ba(ebd)m ab(f c)m cb(gb)n dc(hcd)n /m, n ≥ 0}. We construct the characteristic sample S by taking a finite number of strings derived from the axiom till each of the rules of the grammar finds its application at most twice in the derivation of these strings. In the grammar considered above, S = {baabcbdc, baebdabf ccbdc, baabcbgbdchcd, baebdabf ccbgbdchcd, baebdebdabf cf ccbdc, baabcbgbgbdchcdhcd, baebdebdabf cf ccbgbgbdchcdhcd, baebdabf ccbgbgbdchcdhcd, baebdebdabf cf ccbgbdchcd}. When the input set of the algorithm contains all the elements of S, the algorithm A converges to a correct SICG for the target language L. Hence, it is clear from

74


the manner in which the characteristic sample S is formed that, the class of SICL is identifiable in the limit from positive data. Similarly, the UICL can also be identified in the limit from positive data by constructing a characteristic sample in a similar manner. References [1] D. Angluin, Inductive inference of formal languages from positive data, Information and Control 45 (1980), 117-135. [2] A. Ehrenfeucht, Gh. Paun and G. Rozenberg, Contextual grammars and natural languages, In Handbook of Formal Languages, Springer-Verlag, Vol. 2 (1997), 237-293. [3] A. Ehrenfeucht, Gh. Paun and G. Rozenberg, On representing recursively enumerable languages by internal contextual languages, Theoretical Computer Science 205 (1998), 61-83. [4] T. Koshiba, E. M¨ akinen and Y.Takada, Inferring pure context-free languages from positive data, Technical Report A-1997-14, University of Tampere, Finland (To appear in Acta Cybernetica). [5] S. Marcus, Contextual grammars, Rev. Roum. Math. Pures Appl., 14 (10) (1969), 1525-1534. [6] H.A. Maurer, A. Salomaa and D. Wood, Pure Grammars, Information and Control 44 (1980), 47-72. [7] V. Mitrana, Parallelism in contextual grammars, Fundamenta Informaticae 33 (1998), 281-294. [8] N. Tanida and T. Yokomori, Inductive inference of monogenic pure contextfree languages, Lecture Notes in Artificial Intelligence 872, Springer-Verlag (1994), 560-573. [9] T. Yokomori, On polynomial-time learnability in the limit of strictly deterministic automata, Machine Learning 19 (1995), 153-179.

Permutations and Control Sets for Learning Non-regular Language Families Henning Fernau1 and José M. Sempere2 1

2

Wihelm-Schickard-Institut f¨ ur Informatik, Universit¨ at T¨ ubingen, D-72076 T¨ ubingen, Germany, Email: [email protected] Departamento de Sistemas Inform´ aticos y Computaci´ on, Universidad Politécnica de Valencia, Valencia 46071, Spain, Email: [email protected]

Abstract. We discuss two versatile methods which can be used to transfer learnability results from one language class to another. We apply these methodologies to three learning paradigms: (1) Learning in the limit, (2) Morphic generator grammar inference, and (3) Query learning. Keywords:Formal languages, universal grammars, control sets, learning from positive data.

1

Introduction

In this paper, we will present two methods for transferring learnability results from one language class to another by simple preprocessing. We mainly concentrate on the paradigm “learning in the limit from positive data”. It is not hard to see that similar techniques can be used to develop efficient learning algorithms in other paradigms as well. In the case of query learning, this has been done (within the framework of matrix grammars, see below) in [9]. We will detail such extensions at the end. Here, we will focus on the following version of the learning model “identification in the limit” proposed by Gold [18]: – An inference machine (a “learner”) IM is given the task to infer a language from a certain fixed language class F for which a description formalism (in our case, a grammar formalism) is also fixed. – To the inference machine IM, a language L ∈ F is presented by giving all the elements of L to IM one by one (maybe, with repetitions), i.e., L = { wi | i ≥ 0 }, and wi is given to IM at time step i. – After having received wi , IM responds with a hypothesis grammar Gi . Of course, we can see Gi as the result of computing a recursive (i + 1)-ary function fi : Gi = fi (w0 , . . . , wi ).


(1)

76

H. Fernau and J.M. Sempere

The inference machine IM is called a learner for F if 1. the process described above always converges in the discrete space of Fgrammars, i.e., for all presentations {wi | i ≥ 0} ∈ F, the corresponding grammar sequence G0 , G1 , G2 , . . . converges to a limit grammar G, which means that there is an i0 such that for all i ≥ i0 we find G = Gi0 = Gi ; 2. the limit grammar G is independent of the presentation of the language L and, moreover, L ⊆ L(G). Note that there are a lot of language families known to be identifiable in the limit from positive data. According to Gregor [19], the most prominent examples of identifiable regular language families are: – k-testable languages [15,17] (see below), – k-reversible languages [4] and – terminal distinguishable regular languages [30,31]. Generalizations of these language classes are discussed in [1,10,11,21,32]. Further identifiable language families, especially also non-regular ones, can be found as references in the quoted papers. All these language classes can be learned efficiently, i.e., the time complexity for the computation of the hypothesis function(s) fi in Eq. (1) is only polynomial in the size of its input, which is the total length of the input sample words up to step i.

2

Formal Language Definitions

Notations: Σ k is theSset of all words of length k over the alphabet Σ, Σ

Grammatical Inference: Algorithms and Applications: 5th International Colloquium, ICGI 2000, Lisbon, Portugal, September 11-13, 2000 Proceedings

Grammatical Inference and Applications: Second International Colloquium, ICGI-94, Alicante, Spain, September 21-23, 1994. Proceedings

Grammatical Inference: Algorithms and Applications: 8th International Colloquium, ICGI 2006, Tokyo, Japan, September 20-22, 2006, Proceedings