Computer Programming & Formal Systems

CO MPUTER P R O G R A M M ING AND

FORMAL SYSTEMS

Edited by

P. BRAFFORT Centre de Traitement de l’lnformation Scientijique, Euratom (Ispra)

and

D. HIRSCHBERG IBM Belgium and Universitd Libre de Bruxelles

1963

NORTH-HOLLAND PUBLISHING COMPANY AMSTERDAM

No part of this book may be reproduced in any form by print, microfilm or any other means without written permission from the publisher

PREFACE This book is a product of two seminars held in the IBM World Trade European Education Center of Blaricum (Holland) in 1961, the first of which was dedicated to a general survey of non-numerical applications of computers whereas the second was more specifically concerned with some aspects of the theory of formal systems. Professor E. W. Beth, who took part in both seminars, was kind enough to sponsor the publication of the proceedings in the “Studies of Logic and Foundations of Mathematics”. Rather than publishing everything, a choice was made on the basis of relevance to the subject matter described in the title of this volume. Most authors preferred moreover to present a revised version of their contributions. Symbol manipulation plays an important role both in the theory of formal systems and in computer programming and one would therefore expect some important relationships to exist between these domains. It may therefore seem surprising that specialists in the two fields have only recently become interested in one another’s techniques. This situation is probably due to an original differencein motivation and to a phaseshift in time. Electronic computers made their appearance about fifteen years ago, and have up to now been used essentially to solve problems in numerical mathematics and to process commercial data, two activities with a rather well-established methodology. Moreover, although a computer is indeed a general purpose symbol manipulator, advanced linguistic techniques for prescribing or describing its behaviour are unnecessary as long as the structure of the problems does not differ too much from the arithmetical structure built into the hardware of the machine. Research in formal logics, on the other hand, received its original impetus from the foundation problems of mathematics, which originated at the end of the XIXth century. During the last three decades the emphasis has shifted from the study of particular formal systems, capable of formalizing mathematical disciplines, to the investigation of the fundamental properties of formal systems in general, such as the existence or nonexistence of decision procedures for certain questions. In these investigations practical feasibility has not been taken into account and the results are therefore not directly related to actual computing problems, as was pointed out by Professor Hao Wang.

v1

PREFACE

The current interest in formal logics, manifested by certain computer programmers, is due to a desire to broaden the scope of computer usage beyond the numerical area. It has become customary to call “nonnumerical” a class of computer applications, typical members of which are : language translation, information retrieval, game playing, pattern recognition and theorem proving. In addition to exhibiting a structure which cannot easily be reduced, by an appropriate coding, to the arithmetical structure, most of these applications are not amenable to standard decision procedures and some are extremely complex. Hence the need for new efficient algorithms and for “heuristics”, i.e. shortcuts. Some logicians are similarly interested in going beyond the “asymptotic” concept of decidability, which corresponds to a machine with unlimited storage capacity and computing time. On the basis of experiments in theorem proving on actual computers they want to assess the relative efficiencies of various proof procedures and to investigate the practical impact of classical results on decidability. The concept of a computing mechanism occurred to the logicians many years before the advent of the electronic stored program machine, but the existence of real computers and of corresponding programming techniques has had a considerable influence on present day research in automata theory. These are the various aspects of the relationships between computer programming and the theory of formal systems discussed in the following pages. We have attempted to group as much as possible articles according to similarities of interest; it is however clear that often more than one aspect has been considered by the authors so that our classification remains quite arbitrary. THEEDITORS

MECHANICAL MATHEMATICS AND INFERENTIAL ANALYSIS H A 0 WANG Cambridge, Mass.

1. GENERAL SPECULATIONS

If we compare calculating with proving, four differences strike the eye: (1) Calculations deal with numbers; proofs, with propositions. (2) Rules of calculation are generally more exact than rules of proof. (3) Procedures of calculation are usually terminating (decidable, recursive) or can be made so by fairly well-developed methods of approximation. Procedures of proof, however, are often nonterminating (undecidableor nonrecursive, though recursively enumerable), indeed incomplete in the case of number theory or set theory, and we do not have a clear conception of approximate methods in theorem-proving. (4) We possess efficient calculating procedures, while with proofs it frequently happens that even in a decidable theory, the decision method is not practically feasible. Although shortcuts are the exception in calculations, they seem to be the rule with proofs in so far as intuition, insight, experience, and other vague and not easily imitable principles are applied. Since the proof procedures are so complex or lengthy, we simply cannot manage unless we somehow discover peculiar connections in each particular case. Undoubtedly, it is such differences that have discouraged responsible scientists from embarking on the enterprise of mechanizing significant portions of the activity of mathematical research. The writer, however, feels that the nature and the dimension of the difficulties have been misrepresented through uncontrolled speculation and exaggerated because of a lack of appreciation of the combined capabilities of mathematical logic and calculating machines. Of the four differences, the first is taken care of either by quoting Godel representations of expressions or by recalling the familiar fact that alphabetic information can be handled on numerical (digital) machines. The second difference has largely been removed by the achievements of mathematical logic in formalization during the past eighty years or so. Item (3) is not a difference that is essential to the task of proving

2

H A 0 WANG

theorems by machine. The immediate concern is not so much theoretical possibility as practical feasibility. Quite often a particular question in an undecidable domain is settled more easily than one in a decidable region, even mechanically. We do not and cannot set out to settle all questions of a given domain, decidable or not, when, as is usually the case, the domain includes infinitely many particular questions. In addition, it is not widely realized how large the decidable subdomains of an undecidable domain (e.g., the predicate calculus) are. Moreover, even in an undecidable area, the question of finding a proof for a proposition known to be a theorem, or formalizing a sketch into a detailed proof, is decidable theoretically. The state of affairs arising from the Godel incompleteness is even less relevant to the sort of work envisaged here. The purpose here is at most to prove mathematical theorems of the usual kind, e.g., as exemplified by treatises on number theory, yet not a single “garden-variety” theorem of number theory has been found unprovable in the current axiom system of number theory. The concept of approximate proofs, though undeniably of a kind other than approximations in numerical calculations, is not incapable of more exact formulation in terms of, say, sketches of and gradual improvements toward a correct proof. The difference (4) is perhaps the most fundamental. It is, however, easy to exaggerate the degree of complexity which is necessary, partly because abstract estimates are hardly realistic, partly because so far little attention has been paid to the question of choosing more efficient alternative procedures. The problem of introducing intuition and experience into machines is a bit slippery. Suffice it to say for the moment, however, that we have not realized that much of our basic strategies in searching for proofs is mechanizable, because we had little reason to be articulate on such matters until large, fast machines became available. We are in fact faced with a challenge to devise methods of buying originality with plodding, now that we are in possession of servants which are such persistent plodders. In the more advanced areas of mathematics, we are not likely to succeed in making the machine imitate the man entirely. Instead of being discouraged by this, however, one should view it as a forceful reason for experimenting with mechanical mathematics. The human inability to command precisely any great mass of details sets an intrinsic limitation on the kind of thing that is done in mathematics and the manner in which it is done. The superiority of machines in this respect indicates that machines, while following the broad outline of

MECHANICAL MATHEMATICS AND INFERENTIAL ANALYSIS

3

paths drawn up by people, might yield surprising new results by making many new turns which man is not accustomed to taking. The attempt to mechanize, as much as possible, mathematical thinking opens up a large area of research. In my opinion, the theoretical core of this area is a new branch of applied logic which may be called inferential analysis, characterized by an emphasis on the explicitlessand practical feasibility of methods of inference. This discipline enjoys a measure of autonomy not shared by numerical analysis which, for example, does not deal with logical operations on the choice and arrangement of numerical methods. It is believed that the development of mechanical mathematics will influence pedagogical and research methods in mathematics, as well as affect certain epistemological questions of a specifically mathematical coloring. The governing general principle is: what can be formulated exactly can be put on a machine, subject to the practical limitations on the manageable dimension and complexity. Repetitions are a good indication of the suitability of a mechanical treatment. Thus, for example, much of the activity of teaching mathematics is tedious and requires patience. If no interaction between pupil and teacher were necessary, televisions, or sometimes just gramophones, would be sufficient to replace teachers. As it is, these ready-made conveniences are only used as partial substitutes but, even so, teaching has already begun to enjoy to a certain extent the advantages of mass production. However, interesting problems of mechanical mathematics arise only when we come to tasks which call for an active agent to give answers, advices, and criticisms, the simplest being the correction of exercises and examination papers. Psychologically, the pupil has many reasons for preferring a patient machine teacher when the problem is, as in the majority of situations, a matter of drill rather than inspiration. The result may be that human teachers will employ mechanical devices as teaching assistants. In a similar fashion, since in mathematical research there is also a great deal of mechanizable tedious work, mechanical devices may be used to aid individual mathematicians. In this connection, ,in view of the fact that specific mathematical discoveries are made essentially once and for all, there are less of exact repetitions, but more of the problem of using mechanical devices flexibly by, for example, designing and combining programs on general purpose computers. In order to use machines either to aid research or to aid teaching, the results, methods, and spirit of formalization in mathematical logic are to play an essential role.

4

HA0 WANG

The advance of mechanical mathematics may also affect some of our concepts in the philosophy of mathematics. We get, not only in theory, but even in practice, an objective criterion of mathematical rigor in mechanical terms. The range of feasible mathematical methods will be extended so that the theoretically feasible and the practically feasible begin to converge, and we have a more realistic guidance to the improvement of feasibility. As we understand more fully the range of mechanical mathematics, we get a clearer view of the relation between complexity and conceptual difficulty in mathematics, since we would probably wish to say that mechanizable pieces, even when highly complex, are conceptually easy. When alternative proofs are fully mechanizable, we obtain also a quantitative measure of the simplicity of mathematical proofs, to supplement our vaguer but richer intuitive concept of simplicity. With the increasing power to formalize and mechanize, we are freed from tedious details and can more effectively survey the content and conceptual core of a mathematical proof.

2. THECENTRAL ROLEOF LOGIC In theory all mathematical arguments can be formalized in elementary logic (quantification theory, predicate calculus). If we add equality and the quantifiers “for all x” and “for some y” to the propositional connectives “and”, “if“, “or”, “not”, etc., we obtain the predicate calculus, in which, as logicians will know, every usual mathematical discipline can be so formulated that each theorem T in the latter becomes one in the former when the relevant mathematical axioms A are added as premises. That is to say, if T is the theorem in the mathematical discipline, then “if A , then T” is a theorem of logic. From this fact it is clear that in order to prove mathematical theorems by machines a major step is to deal with theorems of the predicate calculus. One may question the advantage of thus handling mathematics, on the ground that the peculiar mathematical content of each individual branch is lost when the disciplines are thus uniformly incorporated into the predicate calculus by formalization and abstraction. Now it is indeed true that we must add special methods for each special mathematical discipline. An adequate treatment of the predicate calculus is, however, of dominant importance, and for each discipline the basic additional special methods required are fairly uniform. For number theory, the essential new feature is largely concentrated in mathematical induction

MECHANICAL MATHEMATICS AND INFERENTIAL ANALYSIS

5

as a method of proof and of definition; for set theory, in the axiom of comprehension, i.e., the axiom specifying all the conditions which define sets. So there is the problem of choosing the formula to make induction on, or of choosing the condition for defining a set. While it seems doubtful that there is any uniform efficient mechanical method for making such selections, there are often quite feasible partial methods. For example, for making such selections in number theory the obvious uninspired method of trying the desired conclusion of one or another of its clauses as the induction formula should suffice in many cases. It would seem that, once a feasible way of doing logic is given, fairly simple additional methods could carry us quite some way into special mathematical disciplines. Since most of us learned Euclid and number theory without worrying about the predicate calculus, it might seem that the natural course is to bypass logic and go directly to mathematics. But in fact such an approach is ill-advised, so long as the aim is to prove more and harder theorems rather than merely to re-enact the history of mathematical thinking. What is natural for people need not be natural for the machine. If logic is not treated in an explicit and systematic way, constant subsequent additions of ad hoc devices keep slowing our progress toward interesting theorems, while multiplying the sources of possible confusion. In general, a vast machinery specifically designed to obtain a few easy theorems is wasteful ; results obtained from whatever approaches should be measured against the generality and economy of the machinery used. Foundations, furthermore, should be scaled to large future superstructures. It is our conviction that to treat logic only by the way would score very poorly by both criteria. 3.

SOME POSSIBLE

DIRECTIONS FOR FURTHER EXPLORATION

Results so far are too rudimentary to provide us with any decisive conclusions as to the dimension of the long-range effects of the pursuit of mechanical mathematics. Nevertheless, I shall venture a few comments drawn from my own restricted experience. (a) I have examined the theoretically undecidable domain of the predicate calculus and managed to make an IBM 704 prove all theorems (over 350) of Principia mathematica in this domain in less than 9 minutes; this suggests that we usually do not use the full power of strong mathematical methods and should not be prevented from trying to handle an area on account of

6

H A 0 WANG

pessimistic abstract estimates of the more difficult cases in the region. (b) Care in the theoretical design of the procedures is essential and a certain amount of sophistication in mathematical logic is indispensable, because most existing methods are not immediately applicable on machines. (c) In particular, one often has to reformulate available methods or even invent fundamentally new ones ; sometimes theoretically insignificant improvements could increase the speed or reduce the necessary storage by several orders of magnitude, for example, a device to try out certain “preferred” substitutions first. (d) Long-rangeplanning and efforts to make results cumulative are necessary; ad hoc measures and desire for quick sensation should be avoided because otherwise the limit of diminishing return will be reached too soon; the correct course would increase reward per unit of work more and more quickly with greater and greater efforts. (e) While more can be done with larger machines, the design and choice of methods is, at least at the present stage, more crucial because we are far from having made full use of an IBM 704 or 7090 yet. (f) Distrust luck and do not, for example, use obscure methods with the hope that something wonderful might happen since we do not know what will happen ; the chances of undesirable consequences are much bigger. At the present stage, mechanical mathematics seems to be one of the areas in information processing which promise the highest reward for each unit of labor. Only accidental circumstances such as the lack of alliance of potential contributors in administration, programming, and logic have so far sabotaged more rapid developments. The laziest solution of this practical difficulty is for one to attack problems in isolation and hope that the pieces will miraculously fit together in due course. This is not the most satisfactory solution but perhaps the most feasible, given all the facts of competition, sales exaggeration, desire for liberty and independence. There are at least three groups of preliminary work necessary for genuine advances in the long run :a good common (idealized programming) language for crystallization, communication, and accumulation of results; a decent library of subroutines for simple algebraic and logical manipulations of symbols, as well as for simple basic proof and decision procedures; and a fairly sophisticated logical analysis of a number of specific mathematical proofs with a view to bringing out the details which have to be taken care of in order to thoroughly formalize and mechanize them. It is of course not excluded that one would often run into blind

MECHANICAL MATHEMATICS AND INFERENTIAL ANALYSIS

7

alleys. But I am confident major wastes can be avoided through careful planning and alert flexibility. With these provisos in mind, I now proceed to list a few possible directions which, in my opinion, are worthy of at least some preliminary exploration. That proof procedures for elementary logic can be mechanized is familiar. In practice, however, were we slavishly to follow these procedures without further refinements, we should encounter a prohibitively expansive element. It is desirable to study underlying properties of such expansions in order to increase efficiency. In this way we are led to a closer study of reduction procedures and of decision procedures for special domains, as well as of proof procedures of more complex sorts. Such deeper considerations of elementary logic also provide us with a systematic approach to axiomatic theories viewed as applied predicate calculus. The insights thus obtained can complement our direct treatment of specific mathematical disciplines. For the sake of a more concrete goal to guide the choice of theoretical questions, we may set ourselves the aim of programming machines to formalize and “discover” proofs in quantifier-free number theory and axiomatic set theory. These areas are chosen both because they are so central and because it seems desirable to isolate the two basically difficult mathematical concepts: functions and quantifiers. It is possible that the quantifier-free theory of positive integers, including arbitrary simple recursive definitions, can be handled mechanically with relative ease, and yield fairly interesting results. It is clear from works in the literature that this restricted domain of number theory is rather rich in content. It goes beyond logic in an essential way because of the availability of (quantifierfree) mathematical induction. On the other hand, in axiomatic set theory, the explicit use of functions can be postponed for quite a long time. Moreover, here certain general concepts often prove difficult; perhaps machines will more quickly excel in areas where people’s intuitions are not strong. A case in point would be Quine’s axiomatic system “New Foundations,” which was obtained by relaxing certain syntactical restrictions of the theory of types. While the ulterior aim is to use machines to aid mathematical research with the assistance of logic, machines can also be used to aid our theoretical research in logic at the present stage. Computers can be put to good use in the quantity production of concrete examples, which we constantly need as a means of clarifying our concepts and so expediting general theoretical results.

8

HA0 WANG

Already in the limited experience in the mechanizing of logical procedures, the machine outputs have from time to time brought out features of the procedures which one had not thought out clearly in advance. Such experiences have sufficed to persuade us that we would do well to experiment with computing machines even if it were only for purposes of theoretical logic. Some other possible directions are: (1) Experiment with redoing school and college mathematics by machines ;instruct a machine to compete with the average student by using its patience to compensate its lack of intuition; partial decision procedures in algebra, trigonometry, analytic geometry, the calculus ;prove theorems in elementary geometry and algebra with extensive use of methods dealing with the underlying logic. (2) Try to combine numerical and inferential methods so that principles can be introduced for the machine to choose particular methods to apply according to the nature of the given problems; this aims at delegating to the machine as much as possible of the work which now requires a maths matical analyst. (R.W. Hamming is much interested in work along this direction.) (3) In fields like algebraic topology where often definitions are long but proofs are short, it is not unlikely that mechanized logical methods will prove to be of practical use in helping to sort out logical consequences of new concepts. (4)Fairly simple mathematical researches involving combinatorial considerations such as questions of completeness, independence, deducibility in the various systems of the propositional calculus can presumably be helped radically by a few suitably devised machine programs. ( 5 ) Use this type of work as data to guide us in the design of more realistic idealized programming languages. With regard to the formulation of programming languages, it seems desirable not to pursue the task in isolation and then look for applications afterwards. One must not let the initial investment in a programming language control the choice of problems to be programmed, but frequent revisions of a fixed language require a prohibitive amount of energy and work which can easily prevent one from meeting new demands with an open mind. A good compromise between rigidity and complete lack of organization would seem to be the isolation of necessary devices such as the designing of MACROinstructions at every stage, as is called for by specific but typical occasions. In this way, a body of quite well-organized data would gradually emerge as more programs are written. Attention to the accumulation of good MACROinstructions also brings

MECHANICAL MATHEMATICS AND INFERENTIAL ANALYSIS

9

into the somewhat routine task of programming a theoretically more interesting element of formulating exactly concepts which are intuitively familiar. 4. CASESTUDIES AND STOCK-OF-TRADE SYSTEMS

To analyze in detail specific mathematical proofs is clearly a useful preliminary step toward the mechanization of types of arguments. One might attempt to work out a few examples of such case studies drawn from number theory, geometry, and axiomatic set theory. In number theory, one might compare quantifier and free variable proofs of the infinitude of primes and of the fundamental theorem of arithmetic, putting emphasis on recursive functions and mathematical induction. In geometry and axiomatic set theory, one might consider mildly difficult theorems which are proved with quantifiers but without functions. In each case, two types of problems can be conveniently separated : deriving very elementary properties such as the commutativity of addition from the basic axioms on the one hand, and organizing such elementary properties to obtain a basis for further derivations on the other. For the human being, the first type of problem is rather artificial and contraintuitive. For example, the very early theorems in elementary geometry are more abstruse than the simple exercises about parallels, triangles, etc. A good organization of elementary properties plus an exact formulation of familiar methods of trying to find a proof would presumably yield in each discipline something similar to the principles obtained from what is often called the “heuristic approach”. It is here proposed that such organizations be called stock-of-trade systems. However, despite terminological disputes, everybody probably agrees as to roughly what sort of thing is to be done, and the more relevant question is how good a result one gets. It is with regard to this last point that a patient study of special cases with ample use also of the stocks in trade of mathematical logic appears indispensible. For instance, even a formalization of Euclid’s proof of the infinitude of primes contains a few surprises, and there are quite a number of theoretically interesting questions connected with the problem of proving the irrationality of f2 with no appeal to quantifiers. We consider here only an example in axiomatic set theory derived from a paper of Hintikka [13]. If a theorem is proved in a system, even one with only a finite set

10

HA0 WANG

of axioms, it would seem that one major problem is to select the axioms needed for the proof. It stands to reason to expect that it would be easier for the machine to begin with the selected axioms. In so doing, we may lose some alternative proof which uses other axioms but that is something which we do not have to worry about yet. Moreover, it appears easier to select and prove intermediate lemmas and break up the whole proof into parts. In both cases, if we do not have the selection to begin with, it is not easy to decide whether it is advantageous to take all to begin with, or to add routines to select. In the long run, one would expect to use the latter alternative. But when the methods of selecting subproblems and branching out are as cumbersome as some existing crude attempts appear to be, it is not necessarily more efficient to use the selection alternative. The example to be considered is of special interest because it lies in an area which has not been developed nearly as much as old subjects such as number theory. Consequently, we can draw very little from a cumulative intuition, and our advantages over machines are not great. Moreover, this area has been pursued with a considerable emphasis on formal arguments. Let Hxy and 3zGxyz be short for:

#x A (1) u # v3z(z

(2) y # a = (3) y # b (4) y # c = (5)y#d3

z

# Y A FZY A Fyz).

(Fyu = Hay). (Fyb r l H b y ) . (Fyc = ( y = a v y (Fyd-y=c).

=

b)).

The assertion is that the conjunction of (1)-(5) is contradictory. More exactly, this says that the following formula is a theorem of the predicate calculus. (I) 13 u 3 v 3 u 3 b 3 c 3 d v y v z 3 w 3 x {u#v A b#a A [v # b A [Y#C A [y # d

3 3 =)

((Fya A G w w ) V ( 1 F y a A 1 G a y z ) ) l ((Fyb A 1 G b y z ) V ClFvb A Gbyx))] (Fyc=(y=a V y=b))l (Fyd-y = ~ 1 1 ) .

If the system does not include =, then we have to treat a = b as

MECHANICAL MATHEMATICS AND INFERENTIAL ANALYSIS

11

an abbreviation for Vx(Fxu E Fxb),

and add the axiom: u =b

3

Vy(Fuy E Fby).

This incidentally illustrates the fact that for mechanical mathematics it is in practice desirable to include = to begin with. In that case, the formula is in one of the familiar decidable cases since it contains only two consecutive 3 ’ s (for validity). In terms of satisfiability, the part without the i n i t i a l 1 has no model and can be decided by the 3VzX satisfiability case (see [23]). On the whole, it seems easier to make machines do some of the formalizing work which logicians sometimes have to do. This may be viewed as an application of the principle “Charity begins at home.” Some malicious soul might use this as evidence for his favorite view that logic is trivial, and he will be wrong for too many reasons which it is tiresome to elaborate. In general, what is needed for mechanization is not just axiomatic systems with emphasis on economy and elegance but rather “stock-oftrade systems” and formalizations which are exact and yet remain as close to good common expositions as possible.

5. SOMETHEORETICAL DIFFICULTIES In order to mechanize proof procedures of the predicate calculus, it seems natural to use Herbrand’s Theorem. This has been suggested and carried out to varying degrees of completion by different people (see [21], [22], [lo], [16], [6]). The crucial part contains the generation from a given formula of a sequence of propositional or Boolean conditions, and the testing of them. It is clear, both by theoretical estimates and from results obtained so far, that (i) doing the expansion and the testing both by a brute force approach is not feasible; (ii) even greatly speeding up the testing part is not adequate to dealing with fairly interesting cases because often we have to generate a large number of Boolean conditions. Hence, a central theoretical problem is to find ways of selecting only useful terms from each sequence of Boolean conditions. This problem has been explored in a preliminary manner in [22] and [23]. One element is to develop decision procedures for subdomains of the predicate calculus.

12

H A 0 WANG

Another element is to use miniscope forms instead of prenex forms. A third element is to develop semidecision procedures whose range of application we do not know exactly in advance ([23], p. 30). The decision procedures appear not to include the formulas which are of the most interest to us. More specifically, the decision procedures mostly deal with formulas in the prenex form, and when we derive a theorem from a few axioms, even though the theorem and the axioms are separately of simple forms, putting the implication (of the theorem by the axioms) in a prenex form quickly gets us outside the decidable subdomains. This suggests that we should try to extend the decision procedures to truth functions of formulas in the prenex form. Property C in Herbrand’s dissertation [l 11 (see below) seems relevant to this question. The semidecision procedure of [23] is not developed far enough in that the conditions under which we are to terminate the procedure are not specified explicitly. For example, if we encounter a periodic situation (a torus), we can naturally stop; but since the initial columns occupy a special place, we permit also cases while the initial columns and the others have two periods which can be fitted together. Closer examination is necessary in order to lay down broader stopping conditions. The miniscope form defined in [22] is different from the one developed in Herbrand’s dissertation because it permits the breaking up of a quantifier into several. While this permits more extensive reductions, it makes an elegant general treatment difficult. Hence, it seems desirable to return to Herbrand’s treatment which is again connected intimately with his Property C and Property B. Both for these reasons and for the additional reason that Herbrand’s dissertation contains a wealth of relevant material which has been largely overlooked hitherto in the literature, we shall reproduce here in part lecture notes given at Oxford in the Michaelmas term of 1960 on Herbrand’s dissertation, especially on the Properties B and C. His Property A also appears interesting and has been revived in Ackermann’s book ([l], p. 93), but will not be discussed here because we do not understand fully its implications for mechanization. 6. HERBRAND’S DISSERTATION 6.1. Herbrand’s System H. The primitive logical constants are V, (+v) (or Vv), (-v) (or 3v),with 2 , A , = defined in the usual manner. To avoid confusion, we shall use p,q,r,...,Fx,Gxy ,... as atomic

1,

MECHANICAL MATHEMATICS AND INFERENTIAL ANALYSIS

13

formulas and X,Yx, ... as arbitrary formulas which may or may not be atomic. By a “tautology” we shall mean a formula that is always true according to the customary interpretations of truth-functional (Boolean) connectives, abstracting from an analysis of parts which contain quantifiers. The system H contains six rules ([ll], pp. 31-32). RT. Rule of tautology. Every quantifier-free tautology is a theorem. (For example, p 1 p , although not VxFx 1VxFx, falls under this.) RI.

Rules of inversion. Given a theorem X of H , we get another theorem of H if we replace within X a part which is of one of the following forms by its dual:

17 -I(*V)YV (&v)(Yv v 2 )

Y (TV)lYV (fv)Yv v

2 not containing v .

RU. Rule of universal generalization. Xxx +

z

+yXyy (“+”

for infer).

RE. Rule of existential generalization. Xxx + -yXxy. RC. Rule of contraction. X V X + X .

RD. Rule of detachment (cut, modus ponens). X , X 2 Y + Y. The difference between RU and RE can be brought out by: x=x +y(y=y) x = x --f - y ( x = y ) x = x 7+ + y ( x = y ) . --f

The first important result is a direct proof of the following ([ll], p. 36). 6.2. THEOREM 1. Every tautology is a theorem of H; in other words, if we substitute quantifier expressions for parts in RT, we again get theorems of H . 6.2.1. X V

... V

X-+ X

6.3. Positive and Negative Parts. It is familiar that every formula can be brought into a prenex normal form with a matrix in the conjunctive (or disjunctive) normal form: ( f v 1 ) . ..(& vn)Xv1..

.vn

such that X is in, say, a conjunctive normal form.

HA0 WANG

14

+

If we wish to determine whether a quantifier turns into or -, or whether an atomic proposition (an occurrence of it) gets negated or not in such a normal form, we do not have to carry out the transformation but may “calculate” directly by using the notion of positive and negative parts. 6.3.1. Signs of occurrences of propositional parts ([ 111, p. 21 and p. 35). (a) The sign of the occurrence of X in X is (b) The sign changes from Y to 1 Y ; i.e., if an occurrence of 1 Y in X has one sign, the same occurrence of Y in X has the other sign. (c) The sign does not change from X V Y to X or Y. (d) The sign does not change from (& v)Xv to Xv.

+.

6.3.2. A positive occurrence of form; a negative occurrence of

+ v or -v remains + v or -v in a prenex + v or -v becomes -v or + v.

6.3.3. When a formula X without quantifiers is transformed into a conjunctive normal form Y, the sign of each occurrence of each atomic formula is preserved; in particular, a negative or a positive occurrence of, e.g., a letter p turns respectively into one or more occurrences of p preceded b y 1 or not. 6.3.4. If p has only positive occurrences in X@), then

k H@

4)

(X(P)

X(q));

if p has only negative occurrences in X@), then

k H@

4)

(X(q)

X@)).

Proof by induction on the number of logical constants in X . ([ 111, p. 36.) 6.3.5. General and restricted variables. A quantified variable is general if it is and begins a positive part, or - and begins a negative part; in the other two cases it is called a restricted variable. ([ 1I], p. 35.)

+

6.3.6. Miniscope form. A formula is in the miniscope form if the quantifiers cannot be driven inwards any further by the rules of inversion RI. Each formula has a unique miniscope form.

6.4. Champs (sections, cycles). In a given formula, we may replace each general variable v by an indexing function (a Skolem function, an Herbrand function) with the restricted variables governing Y in the formula as arguments. Since ordinary functions (the descriptive functions) add no

15

MECHANICAL MATHEMATICS A N D INFERENTIAL ANALYSIS

basic complications, we shall exclude them. Functions for ungoverned general variables are (indexing) constants, they are always contained in CZas defined below. For an arbitrary set S of indexing functions, we can define the associated champs as follows. The champ of order 1, CI, contains only one fixed object al, which may be taken as the number 1. Given the champs CI,.. .,c k , c k + l consists of all and only the indexing constants and the values of the indexing functions in Swith arguments drawn from CI, ...,ck. We shall assume that the values of different functional expressions are always different. Thus, given a set of indexing functions, we can always determine the number nK of the members of the union of CI, .. ,c k .

.

7. PROPERTY B AND PROPERTY C Given an arbitrary formula, we replace each general variable by a function of all the restricted variables which govern it. Then we can define the champs and a function nk which gives the number of numbers occurring in C1,...,ck.

7.1. The reduced form of order k of a formula X. Let N = nk. (1) An atomic formula is its own reduced form. (2) If the reduced forms of Y and 2 are Y* and Z*, those o f l Y , Y V 2 a r e l Y * , Y* v Z*. (3) If + x or -x is a general variable and the reduced form of Yx is Y* x, that of xYx or -xYx is Y* x(y1,. ..,ym),where y l , ...,y , are the restricted variables which govern x. (4) If -x is restricted, the reduced form of -xYx is:

+

Y*l

If

v ... A

Y*N.

... A

Y*N.

+ x is restricted, that of + xYx is: Y*l A

7.2. A proposition X has Property C of order k if its reduced form of order k is a tautology. It has Property B of order k if its miniscope form has Property C of order k. A very important theorem in Herbrand’s dissertation ([ 1l],pp. 101-105) is :

THEOREM 2. Ifaproposition has Property C (or B) of order k, then every proposition derivable from it by the rules of inversion RI has Property C (or B) of order k.

16

H A 0 WANG

The first rule for negation exchanging Z a n d l l Z obviouslymakes no difference. The rules of exchanging (kv ) l Z v and l ( r v ) Z v , (*v) (Yv v Z ) and (& v) Yv V Z do not affect the reduced forms when (kv) is a general variable, since a general variable remains general by the transformation. When (& v) is a restricted variable, the rules of negation do not affect Property C because of the definition of A in terms of V , from which we have : ( 1 Y l A ... A l Y N ) r l ( Y l V ... V Y N ) ( T Y l V ... V l Y N ) = l ( Y 1 A ... A Y N ) . The complex cases are with the rules of disjunction when (&v) is a restricted variable. Case (a). The exchange of ( - v ) Yv v 2 and ( - v ) (Yv v Z ) . Suppose U and V are the same except that an occurrence of ( - v ) Yv V Z in U is replaced by (- v) (Yv V Z ) in V. Let U* and V* be the reduced forms of U and V of order k. We wish to show that U* is a tautology if and only if V* is. The difference between lJ* and V* is that, for every general variable y in Z , we use y(x1, ...,x I ) to get U*, y(x1, ...,xn,v) to get V*. The number n k in the two cases are, say, NI and N,, NI < Nz. (al). If V* is a tautology, then U* is one. If V* is a tautology, then, in particular, if y(x1, ...,x n , l ) ,...,y(x1,...,x,,Nz) are all identified, the result is again a tautology, since we are thereby merely restricting the choice of truth values of certain atomic propositions. Hence, if we delete repetitions, we can derive U* from V*. Hence, U* is also a tautology. (a2). If some assignment falsifies V*, then there is also some assignment which falsifies U*. We choose for each falsifying assignment of V* a falsifying assignment of a suitably chosen U** which can be shown to be implied by U*. Since X I , . .., X n are the (only) free variables in (- v) Yv V Z and (-v)(Yv V Z ) , the two parts may multiply their occurrences in U* and V* according to the values given to X I , . . ., x n from { 1 , . ., N I } and { 1 , . ..,N z } respectively. Now we define U** from U* by choosing one number i from { 1 , . ,N z } for each fixed set of values of X I , . .., X I from { 1,. ..,Nz), and replacingy(x1,...,xn) in U* by y(x1,. ..,xn,il.It is essential that U ** is modified to be over { 1,. .,N z } rather than { 1 , . ..,N i } . First, we make the choice of i for U** for a fixed falsifying assignment

.

..

.

MECHANICAL MATHEMATICS AND INFERENTIAL ANALYSIS

T of V* and a fixed choice of (XI,. sponding parts in U** and V * are: Y*l V ( Y * l v Z*1)

17

..,xn) from { 1 ,...,N2}. The corre-

... V Y*N2 V Z*i v ... v ( P N 2 v Z*N2).

(1) (2)

If in the assignment T, (2) gets the value true, then either Y * j is true for some j , and we can take any number, say 1 as i, or else, Z*j is true for some j , and we choose the smallest such j to be i. If (2) gets the value false, then i can be any number, say 1 . It is clear that ( 1 ) and (2) get the same truth value by the choice, and we can similarly choose an i for each set of values X I , . .., x n from { 1 , . ..,Nz}.In this way, it follows that the assignment T falsifying V* also falsifies U** which is like V* except for containing parts like ( 1 ) in place of parts like (2). Now we have to show that U* implies U**. Just the replacement of y(x1,. ..,x,) by y(x1,...,xn,i) makes no difference since we merely give the former a new name. But the fact that U* is relative to { 1 ,...,N1}, but U** is relative to { 1 ,...,Nz}, N1< N2, means there are more clauses in U** than in U*. If we look at (4) of 7.1, we see that a positive occurrence of ( - x ) Wx is replaced by a disjunction, a negative occurrenceof (+x ) Wx is replaced by a conjunction. Hence, using 6.3.4, we can get U* 3 U** from : (W1 V ... V WNi) 3 (W1 V ... V WN2) (W1 A ... A WN2) 2 (W1 A ... A WNi). Therefore, a falsifying assignment of V* yields one for U*, i.e., V* is a tautology if U* is. By (al) and (a2), we have proved Theorem 2 for Case (a). Case (b). The exchange of (+v)Yv v 2 and (+ v)(Yv V Z). The proof is similar to Case (a) except that in choosing i for U**, we use : (Y*l A ... A Y*N2) V Z*i (I*> ( Y * l V Z l ) A ... A (Y*N2 V Z*N2). (2*> If (2*) gets true in the assignment T, then take any number, say 1 , as i . If (2*) gets false, then ( Y * j v Zj) must be false for somej, take the smallest such j as i. This completes the proof of Theorem 2. It is clear that, from this theorem, we can use a prenex form or the miniscope form and retain the same order k of Property C(or Property B).

18

HA0 WANG

8. HERBRAND’S THEOREM The fundamental theorem proved by Herbrand ([ 111, pp. 112-1 13) is somewhat different from what is commonly known as Herbrand’s Theorem : HERBRAND’S FUNDAMENTAL THEDREM HX;one can effectively find (i) If X has Property B of order k , then the proof of X from the number k . (ii) Given a proof in H of a proposition X, we can effectively jind a number k such that X has Property B of order k . By Theorem 2, if X has Property B of order k, a prenex form of X has Property B of order k and can be proved in H with the help of 6.2.1, in a manner which is familiar nowadays. Hence, by RI, we get a proof of X in H . And (i) is proved. To prove (ii), we make induction on the number of steps in the proof. An axiom must be a case of RT and has Property B or C of order 1. By Theorem 2, RI does not affect the order of Property B or C . RU does not change the reduced form. If Xxx has Property C of order k with nr = N1, then (-y)Xxy has Property C of order k or k 1. We may assume Xxx in the prenex form (by Theorem 2), and then the reduced form of (-y)Xxy of order k 1 must be a disjunction X*jl V . V X*jNz which includes all the disjuncts of the reduced form of Xxx of order k , since N1< Nz.For example, (-z)(Gzx V 1 G v x ) is of order 1 since its reduced form G12 V 7 G 1 2 is a tautology; (-y)(-z)(Gzx V 1 G v y ) is of order 2 since its reduced form (G12 V 1 G l l ) V (G22 V l G 1 1 ) V (G12 V 7G12) v (G22 V 7G12) is a tautology and y does take the fixed value 2 of x . Hence, RE preserves Property B or C with the possible increase of the order by 1 . RC does not change the order of Property B because, given a falsifying assignment of the reduced form of X,we get one for that of X V X by the same assignment of truth values to the atomic formulas. RD preserves the order of Property B. First, if X and 2 have Property B of order k, then X A 2 has the same order. Thus, take the reduced form XI V V X j of X and Y1 v V Yt or Y. Then the reduced form of X A 2 clearly contains all disjuncts X, Yb, 1 I; a I; j , 1 Ib 5; t , and is therefore a tautology. Hence, if X and X =Y Y are of order k, so is also X A (X 2 Y).

+

+

..

...

...

MECHANICAL. MATHEMATICS AND INFERENTIAL ANALYSIS

19

Let X be ( f y l f ...f y,) My1 ...y ,xi...xn, and Y be ( f z i f ...f z g ) Nz1 ...zqxl...xn. To avoid conflicts of variables, we replace yi ,...,y, in X by u1, ...,up to get XI, and consider X A (Xi 3 Y). To get this into a prenex form, we can pair off ut with yr, one of which must be positive while the other is negative. We put them pairwise at the beginning, with the negative preceding the positive in eachcase. Let the negative one of the pair yt and ut be vt and the other be wi, then we have: (-v1+w1-

...- v p + w p & z 1 f . . . kzg) [ M y l . .. y p ~ 1 ... ~ A n MU^. . UpX1..

.

.Xn

3

N z ~...ZqX1.. .xn)].

Now we form a champ of order k for this formula, and identify suitable elements in it to get one for Y. Thus all elements wt+l(vl,...,vt), with arbitrary v1,. ..,v4, are identified with vt+l; all elements zt(vi,.. .,vp, zfl,. ..,z4,) with a fixed zt(u1,. ..,up,ztl,. ..,zt,). In this way, we get a champ for Y of order p since the only indexing functions are those for the negative quantifiers among z1,. ..,zq plus the indexing constants for the free variables xi,. ..,xn. By hypothesis, X A (X = Y) has Property B of order p and has, therefore, a disjunction: which is a tautology. After the identifications, we get a new tautology in which certain disjuncts are repeated and can be deleted. Suppose the formula is the result after the deletions, then we can derive by the propositional calculus: Y1 v ... v YN. But then this tautology is essentially the reduced form of order k of the formula Y. This completes the exposition of Herbrand’s fundamental theorem. Recently Peter Andrews and Burton Dreben discovered in Herbrand’s thesis (see their forthcoming paper “Some difticulties in Herbrand’s Theorem”) a subtle mistake which is preserved in the above exposition in the sketched proof of Case (b) on p. 17. We have to replace the two occurrences of “order k” in the statement of Theorem 2, on p. 15 above, by “some finite order.” The fundamental Theorem remains correct except that the detailed calculations of the order of Property B become more complex. Dreben is planning a thorough discussion of these questions. (October, 1962).

20

H A 0 WANG

REFERENCES ACKERMANN, W., Solvable Cases of the Decision Problem. Amsterdam, 1954. GOODSTEIN, R. L., Recursive Number Theory. Amsterdam, 1957. BTER, R., Rekursive Funktionen. 2nd edition, Budapest, 1957. QUINE,W. V., Mathematical Logic. Revised Edition, Cambridge, Mass.,1950. SURANYI,J., Reduktionstheorie des Entscheidungsproblems.Budapest, 1959. DAVIS,M and H. PUTNAM,“A Computing Procedure for Quantification Theory.” Journal ACM, Vol. 7 (1960), pp. 201-215. [7] DUNHAM, B., R. FRIDSHAL,G. L. SWARD,“A Nonheuristic Program for Proving Elementary Logical Theorems.” Proceedings Z.C.Z.F., Paris, 1959 (pub. 1960), p. 284. [8] -, J. H. NORTH,Exploratory Mathematics by Machine (to be published in a symposium volume at Perdue University). [9] GELERNTER, H., “Realization of a Geometry Theorem Proving Machine.” Proceedings Z.C.Z.F., Pans, 1959 (pub. 1960). [lo] GILMORE, P. C., “A Proof Method for Quantification Theory: Its Justification and Realization.” ZBM Journal, Vol. 4 (1960), pp. 28-35. [Ill HERBRAND, J., Recherches sur la ThPorie de la Dkmonstration. Warsaw, 1930. [I21 -, “Sur le Problkme Fondamental de la Logique Mathkmatique.” CR, Warsaw, No. 24 (1931). [13] HINTIKKA,K. J. J., “Vicious Circle Principle and the Paradoxes.” Journal of Symbolic Logic, Vol. 22 (1957), pp. 245-248. [14] MINSKY,M., “Steps Toward Artificial Intelligence.” Proceedings Z.R.E., Vol. 49 (1961), pp. 8-30. A., J. C. SHAW,H. A. SIMON,“Empirical Explorations of the Logical [15] NEWELL, Theory Machine: A Case Study in Heuristics,” Proceedings W.J.C.C., (19571, pp. 218-230. D., H. PRAWITZ,N. VOGHERA,“A Mechanical Proof Procedure and [16] PRAWITZ, its Realization in an Electronic Computer.” Journal ACM, Vol. 7 (1960), pp. 102-128. [17] -, “An Improved Proof Procedure.” Theoria, Vol. 26 (1960), pp. 102-139. [18] ROBINSON, A., “On the Mechanization of the Theory of Equations.” Bulletins of the Research Council of Israel, Vol. 9F, No. 2 (Nov. 1960), pp. 47-70. [19] SHEPHERDSON, J., The Principle of Induction in Free Variable Systems of Number Theory. (Lecture at the Polish Academy, Spring, 1961, to be published). [20] SKOLEM, Th., “Begriindung der elementaren Arithmetik.” Kristiania, 1923,38 pp. [21] WANG,H., “A Variant to Turing’s Theory of Computing Machines.” Journal ACM, Vol. 4 (1957), pp. 63-92. [22] -, “Toward Mechanical Mathematics,” ZBM Journal, Vol. 4 (1960), pp. 2-22. [23] -, “Proving Theorems by Pattern Recognition.” Part I, Communications ACM, Vol. 3 (1960), pp. 220-234; Part 11, Bell System Technical Journal, Vol. 40 (1961), pp. 1-41. [l] [2] [3] [4] [5] [6]

OBSERVATIONS CONCERNING COMPUTATION, DEDUCTION, AND HEURISTICS E. W. BETH University of Amsterdam

1. Introduction. It is not a very easy matter to decide to what extent the results of recent investigations in the domains of logic and foundations are relevant with a view to the problems which a programmer has to face. I feel, however, that the following observations can be offered.1) In constructing codes and machine languages one is concerned, generally speaking, with the problem of finding suitable notations for certain given concepts, and this is a subject to which logicians have given much attention. In point of fact, Leibniz's attempts at formalizing logic and mathematics were strongly motivated by his interest in arithmetical and logical machines [4].Only much later, mainly through Frege's influence, was research in this direction connected with investigations into the foundations of logic and mathematics. In this kind of work, two main directions must be distinguished. (a) We may ask whether, in principle, agivensystemof notation makes allowance for the expression of certain specific concepts, and to what extent. (b) We may ask which system of notation is the most convenient one for a certain specific purpose. The discussion of problem (a) leads to the development of a theory of definition the importance of which would seem to be theoretical rather than practical. To this point I shall return later on; we shall see that nevertheless the theory of definition has a certain practical importance because of its connections with the theory of deduction. Problem (b), on the other hand, leads to discussions of a practical and experimental nature. In point of fact, logicians have collected a tremendous amount of experience which, even though their purpose and standards of evaluation may differ from those of a computer specialist, would seem of considerable interest with a view to programming. In my opinion, this is a domain where the co-operation of logicians and coml) The above remarks have to some extent been inspired by the results of certain investigations carried out under Contract No. 010-60- 12 DOH between Euratom and the University of Amsterdam.

22

E. W.BETH

puter specialists will prove particularly fruitful. Nevertheless, I will not dwell upon this subject [3]. 2. Computation as Deduction. Logic is primarily concerned with the study of deduction. Numerical computation is represented in the more familiar formal systems as deduction on the basis of a suitable axiom system for arithmetic. The formulas to be deduced are of the type: f*(m*) = n*, where f * denotes a certain arithmetical function f whose values are to be computed and where m* and n* are numerals denoting the natural numbers m and n. We wish such a formula to be deducible if and only if n is the value of the function f for the argument value m. The functions for which this wish can be fulfilled are found to be the same for a great variety of formal systems. They are denoted as general recursive functions. It seems clear that the study of formal systems, of axiom systems for arithmetic, of general recursive functions, and of related function classes, will be highly relevant to any systematic treatment of the problems arising in computer programming and will even provide its theoretical basis. Such an approach will, however, look somewhat less obvious if we take account of the kind of discussion that is more or less characteristic of the field of study just mentioned. By way of illustration, I should wish to discuss the following Let T be a formal system of arithmetic, formalized within THEOREM. the framework of elementary logic, in which the conditions: X+Y = 2, x - y = z, xv = z, can be expressed and which provides a basisfor the numerical computation of the functions which they define; then the system T also provides a basis for the definition and the numerical computation of the function f characterized by the conditions: f(1)

f(m

1,

+ 1) = ( m + 1) - f ( m ) . =

[The conditions imposed upon the system T are already satisfied by

OBSERVATIONS CONCERNING COMPUTATION, DEDUCTION, AND HEURISTICS 23

rather weak axiomatizations of arithmetic, such as the system Q discussed in the Tarski-Mostowski-Robinson monograph [7] or the system A described in my own book on Formal Methods [la,2]. Note that we clearly have f ( m ) = m ! ] Proof (in outline). The condition : (1)

f(m) = n

can be restated as follows: (11) There is a finite sequence of natural numbers bl,62,. ..,bm such that: (1) bl = 1, (2) for every i such that 1 5 i < m, we have br+l = (i+ l).bt, and (3) bm = n. If we now replace the numbers 61, b2.. .,bm by:

al = 3 . 5b1, a2 = 32.5'2 , . . . , a m = 3m.5bm, then it will be possible to restate the above condition (11) as follows: (111) There is ajinite set M = {ai,a2...,arn}ofnaturalnumberssuchthat: (1) if, for any u in M , we have u = 3 . 9 , then k = 1, (2) ...,and (3) ... . Finally, we may still replace the finite set M by one single natural number : A = 2'"1+ 2"2 ... 2'"m,

+ +

and, accordingly, we shall obtain the following restatement of the above condition (111) : (IV) There is a natural number A such that: (1) i f , for any u, v, w, k , we haveA = 2 " * ( 2 . v 1) w, w< 2U,u = 3.5", thenk = 1,(2) ...,and(3).... Now this last version (IV) of our condition can be expressed in T by the formula [lb]:

+ +

+ 1) + w & w < 2U & u = 3.5"] -+

(EA)[(u)(v)(w)(k){[A =2 U - ( 2 . ~

and this formula (EA)[. .. A, m, n, (m)(n){f*(m)= n

f----r

+k

=

l } & ... & ...I

...] clearly defines the function f: @ A ) [ ...A,m,n,. ..I}.

Moreover, whenever for a specific m and n we have f ( m ) = n we can, on the basis of the formal system T, compute the relevant value of A and show that it has all the above properties. Therefore, the relevant formula: (EA)[. ..A,m*,n*,

or :

f*(m*) = n*

...]

24

E. W. BETH

can be deduced in T, and this amounts to saying that the numerical value off(m) can be computed in T. This completes our proof. Now take m = 6. Then we have f(m) = 6 ! = 720. In accordance with the above discussion, this means that a certain natural number A has certain properties. Therefore, computing 6 ! on the basis of the formal system T (to which the above definition has been added) amounts to first computing A and then proving that A has all those properties. By a rough but conservative estimate, we have : A = 103.10~~~. I wonder how many micro-seconds one of the more powerful computers would need to provide us with a precise value of A. At any rate, one should remember that we have taken a fairly small argument value. The above discussion has certainly not been impartial, and I shall take it up once more in section 3. Nevertheless, it will already have served a purpose if it has shown that the above-mentioned disciplines, and in particular the theory of recursive functions, are not primarily concerned with the construction of practical methods of computation, but rather with the problem of characterizing those functions which are, or are not, effectively computable in principle. Therefore these disciplines deal with a kind of problem with which, in general, the programmer will hardly be concerned. With respect to the functions which he has to handle the question of their computability in principle will hardly ever arise. In most cases he will be exclusively concerned with the problem of programming their computation in the most efficient manner. 3. The Deduction Problem Re-Examined. Let us first state the problem somewhat more precisely. We consider the formal system T’ obtained from T by adding the formula : (1)

(m)(n){f*(m)

=

n

-

.

(EA) [. .A,m,n,. ..I}

as an axiom. [This step is meant to eliminate all possible complications arising from the fact that T may not be strong enough to establish the functional character offas defined by formula (l).] We know that in the formal system T’ the formula: (2)

f * ( 6 ) = 720

can be proved, and we wish to examine the various proofs for which T’

OBSERVATIONS CONCERNING COMPUTATION, DEDUCTION, AND HEURISTICS 25

makes allowance. In particular, we anticipate two ways of proving formula (2). 1". One starts by computing the numerical value of A . As we have seen, this preliminary part of the proof will prematurely exhaust the capacity of any physically conceivable machine. 2". The other starts by characterizing A as: 23.5 + 232.52+ 233.56+ p4.sZ4+235.5120+ 236.5720. In this notation, A is readily found to have all necessary properties. Now suppose that we wish to construct an automaton that is capable of proving theorems on the basis of the formal system T'. It will be clear that we should prevent the automaton from trying to produce a proof of the first kind and that, conversely, it should rather be stimulated to produce proofs of the second kind. Since from a purely logical point of view the two kinds of proofs are equally conclusive, it follows that, in addition to instructions corresponding to the relevant methods of deduction, the automaton should be given instructions concerning the selection to be made among these methods. Such instructions may be taken to represent a form of theorem-proving heuristics. Before discussing the possibility of such a form of heuristics, I wish to make still another remark. In computing a valuef*(m*) of an arithmetical function f we do more than simply deducing a certain conclusionf*(m*) = n*: we start, so to speak, with an incomplete formula f*(m*) = ..., we somehow find a suitable numeral n*, and only then do we deduce the conclusionf*(m*) = n*. It will be clear that our automaton should be given instructions for finding a suitable numeral n * ; such instructions may be taken to represent a form of theorem-$nding heuristics. In the particular case under discussion, which is concerned with the probability of formulasf*(m*) = n* on the basis of the formal system T', it is not difficult to see that a suitable theorem-finding and theoremproving heuristics can indeed be established. I shall not give an explicit statement of the relevant heuristic rules, but it will be clear that these rules are based on the following facts. 1". The function f defined by the formula (1) is the well-known factorial, f ( m ) = n!, which is characterized by the following conditions :

I!

=

1,

(n

+ l)! = (n + l ) * n !

2". In order to find f(m), we compute in successionf(l), f(2), f(m - 11, f h ) .

...,

26

E. W. BETH

3”. In proving f*(m*) = n*, we use the formulasf*( 1) = 1, f * ( 2 ) = 2, ...,f * ( m - 1) = ..., as lemmas. 4”. The proofs of these formulas can be relatively brief because we have a very concise and appropriate notation for the numbers A involved. It will be clear that this heuristic is not based on psychological considerations but rather on certain metamathematical insights concerning the formal system T’. Therefore it is reasonable to ask whether our metamathematical insight could be suitably deepened and expanded. It follows from Godel’s Theorem and from related results that the possibilities in this direction are relatively restricted. In this connection it may be observed that, as shown by my own analysis of the Berry Paradox [2], pp. 512ff., the introduction of a concise notation acts, so to speak, as a double-edged sword. It is true that such a notation tends to shorten deductions, but conversely it also tends to shorten theorems. 4. Other Systems of Heuristics. In this connection I wish to state the opinion that often exaggerated claims are made concerning the strength of other systems of heuristics. In this connection I may offer the following example [8].

“A simple programme produced a new proof of the well-known theorem of Euclid, which states that the two angles at the base of an isosceles triangle are equal to each other. This is the proof.. . .

A

B

C.

AF3 = AC (given) AC = AB (given) LBAC = LCAB AABC = AACB LABC = LACB, Q.E.D.

This example seems to support the thesis of Bernal that we are perhaps on the threshold of a new direction in science. The proof is original, elegant, and some one hundred generations of schoolboys and their mentors failed to think of it.” The proof produced by the programme is certainly elegant, but it is, of course, by no means original or new. In point of fact, it was already given by Pappus [ 5 ] (about 300 A.D.) and reproduced in many editions of

OBSERVATIONS CONCERNING COMPUTATION, DEDUCTION, AND HEURISTICS

27

Euclid’s Elements. In recent years, it was again clearly suggested by D. Hilbert [a]. 5. Non-Numerical Problems. Several of the above remarks apply as well to non-numerical data processing, inasmuch as this can, in many cases at least, be reduced to numerical computation by means of a suitable arithmetization. A new situation arises, however, if among the information kept in stock there are data referring not to particular facts but to rules of a general nature and if we wish to retrieve, not simply these general rules themselves, but rather the results of their application in specific cases. It will be helpful to discuss a concrete example. Suppose that we wish to construct an automaton which answers concrete questions concerning social insurance. As an example of such a question we may take the following one : A woman of 61 has been working for 42 years and during 37 years she has paid a compulsory contribution to the General Retirement Fund; owing to an accident she is no longer able to work. Is she entitled to a pension from the General Retirement Fund, or to any other allowance?

The complete answer to this question may be of the following kind: If she is not married, has no children, and has no other sources of income, then at the age of 65 she will receive from the General Retirement the pension which she would have received if she had continued to work; and, meanwhile, she will receive 80% of that pension as an allowance from the Special Pension Fund. If she is married,. . . .

It would clearly not be practical to accept questions and provide answers in this elaborate form. It is possible (and customary) to simplify the procedure by compelling those seeking information to fill a Question form and thus to provide all information that may be relevant; in the case of our example, one would ask whether the woman concerned was married, had children, had any other sources of income, and so on. And, conversely, in reply to a question one might give, on a Reply form, a full statement of the worker’s rights and obligations. An automaton as required might operate on a purely inductive basis. The machine would have access to a complete stock of information regarding all allowances hitherto granted or refused by Labour Boards. It should be able to retrieve and to compare all decisions taken in cases similar to the one in question; if all these decisions agree, this would

28

E. W.BETH

settle the matter. This solution of our problem will be considered no further.

6. Deductive Treatment. I am rather interested in an automaton that would proceed by deduction. Such a machine should be able to carry out deductions on the basis of the general rules laid down in the law and of the special data concerning the case in question. It is reasonable to ask to what extent its operation would be similar to that of a machine programmed for computing the values of an arithmetical function. Because of the fact that the answer to a question is to be given in the form of a full statement of rights and obligations we have created for the deductions to be carried out by our automaton a situation which agrees with numerical computation in this respect that the form of the conclusion is to a considerable extent fixed in advance. In point of fact, the situation becomes very simple if the general rules which are to be applied refer exclusively to the worker immediately concerned. For in this case these rules will be expressed by formulas with one prenex general quantifier and so our deduction admits of a decision procedure. For instance, in order to complete, in the Reply Form, the entry : “Entitled to pension from General Retirement Fund?’ we shall try to deduce, on the basis of general rules such as:

(x>I(Fem(x) & A G 5 ( X ) }

+

Wx)I

and of special data such as: the conclusion :

Fem(a)7A61(a)7Ma(a)7* *

*

Pe(a).

If we are, or if the machine is, successful, this means that the woman is entitled to a pension; if the deduction is impossible, she is not. The situation will become much more involved if the answer to be produced by the automaton depends not only on data about the woman concerned but also on data about, say, her husband and her employer. In that case the general rules which are to be applied will contain several quantifiers instead of only one. In order to make this point completely clear I have first to explain in what manner quantifiers do create complications in deduction problems. We consider the problem of deducing in accordance with the rules of classical logic a certain conclusion Z from certain premisses, one of which

OBSERVATIONS CONCERNING COMPUTATION, DEDUCTION, AND HEURISTICS 29

has the form (x)(Ey)( z ) U(x,y, z ) whereas the other ones are not specified and form a set K'. One approach to such a problem consists in constructing a semantic tableau, as follows. True

False

It will be clear that, if nothing intervenes, formula (2) alone will contribute to the tableau, besides formulas (4)-(19, and infinitely many other ones; in general, we do not know in advance which formulas will be useful and which will not. Now this is exactly what makes mathematical theorem proving such a difficult (and exciting) matter. Suppose, for instance, that 2 is a theorem of arithmetic and that formula (2) is an axiom or a theorem previously established. Then a will stand for an arbitrary natural number. By formula (2), there is a natural number, b, which stands to a in the relationship expressed by formula (5). Moreover, there is another natural number, c, which stands to b in the same relationship, and so on. Because there are, as we know, infinitely many natural numbers, the possibility of an infinite regress cannot be excluded. Nevertheless, there are reasons to believe that in connection with problems such as the one in our above example an infinite regress can hardly arise. For the text of the law will enable us to draw up once for all a n exhaustive list of all persons possibly involved in any given case. In connection with our problem of social insurance this list may include:

30

E. W.BETH

the woman concerned, her husband and children, her employer, her physician, the Labour Board and their medical adviser, the Minister of Labour, and perhaps a few others. The children of the woman’s physician and the wife of the Minister of Labour will certainly not be involved, and hence we have no reason to anticipate the kind of infinite regress that complicated matters in the case of our above semantic tableau. Suppose that the persons possibly involved in accordance with the above-mentioned list are only 30 in number; then we shall be obliged at most 30 times to introduce an element a, b, c, . . as a result of the appearance of certain quantifiers. If, for a given class C of deduction problems, the corresponding number of elements can be shown to be - k, then k will be denoted as a critical number for the class C ; if no I such number k can be found, then the critical number for the class C is said to be infinite. The possibility of an automatic treatment of a class C of deduction problems depends, in the first place, upon its critical number; if this critical number is relatively large, then it is rather unlikely that an automatic treatment will prove possible. If the critical number is small, then we have still to take into account the number and complexity of the general rules which have to be applied and of the special data which have to be considered.

.

7. Deduction from Legal Fictions. To conclude I wish to make a few remarks on legal fictions and on counter-factual conditionals because I feel that the above discussion may throw some light on the specific status of these statements and, perhaps, on the status of legal statements in general. Among the general rules to be applied there may be one or more of the following kind : If a worker satisfies the condition A , then he shall be treated as i f he satisfied the condition B, for instance: if a worker has been divorced, then he shall be treated as if he had never been married. A rule of this kind can be expressed by a formula : (X)[AW

+

{(Y)[Bdv)-+ w Y ) l

+

Pe(x))l.

Two possible interpretations will be considered. (a) A lawyer might argue as follows: “So far, two workers have been found to satisfy the condition B and each of the two has been granted a

OBSERVATIONS CONCERNING COMPUTATION, DEDUCTION, AND HEURISTICS 3 1

pension. So we have ( y ) [ B ( y )-+ Pe(y)].My client, a, satisfies the condition A and so we have A(a). It follows from the general rule that Pe(a). Therefore, my client is entitled to a pension.” (b) It needs hardly saying that this is not the way in which the rule is meant to be applied. A worker a will benefit from the rule, if and only if, on the basis of the general rules laid down in the law and of the special data concerning a, the conclusions A (a) and can be deduced. True

(Y) “ A -+ P e w 1

I

False

The above semantic tableau shows clearly how a’s lawyer should proceed. In the first place he must establish formula (5), that is, he must show that a qualifies for being treated as if he satisfied the condition B. Secondly, he must show that an arbitrary worker b, who satisfies the condition B, will receive a pension. In such a situation, however, one would normally use a counter-factual conditional, and argue as follows: “NOWsuppose that a satisfied the condition B ... .” But, although this way of speaking is familiar, it is clearly by no means unavoidable. In this connection we should consider once more the possibility of an automaton operating on an inductive basis. In the first place, I should wish to emphasize that such an apparatus would not necessarily produce the unsatisfactory conclusion mentioned under (a). Secondly, however, the fact that such a conclusion might be propounded points to the necessity that its stock of information should contain all special data on which the decisions have actually been based. For otherwise the automaton might present cases as similar in which the

32

E. W. BETH

same decision has actually been based on different special data and in this way it might produce incorrect anticipations in regard of future decisions based on special data not accounted for in the operations of the machine. REFERENCES [la] Beth, E. W., Formal Methods, An Introduction to Symbolic Logic and to the Study of Effective Operations in Arithmetic and Logic, Dordrecht 1962. [I b] For more details, cf. Beth, E. W., Formal Methods. [2] Beth, E. W., i%e Foundations of Mathematics (Studies in Logic), Amsterdam 1959. [3] On these matters, cf. Braffort, P., D. Hirschberg and J. Mommens, La programmation des problhes non numdriques (mimeographed). [4] Couturat, L., (ed.), Opuscules et fragments inidits de Leibniz, Paris 1903, Index nominum et rerum S.V. “Machina”, p. 650. [5] Heath, Th. L., The Thirteen Books of Euclid’s Elements, vol. I, 2nd ed. (republished), New York 1956, p. 254. [6] Hilbert, D., Grundlagen der Geometrie, 8. Aufl. Stuttgart 1956, S. 15. [7]Tarski, A., A. Mostowski and R. M. Robinson, Undecidable Theories (Studies in Logic), Amsterdam 1953. [8] The Phantasy Generator. Artorga, Communication 33 (September, 1961), p. 2, referring to work done by M. Minsky.

A BASIS FOR A MATHEMATICAL THEORY OF COMPUTATION 1)

JOHN McCARTHY Computation is sure to become one of the most important of the sciences. This is because it is the science of how machines can be made to carry out intellectual processes. We know that any intellectual process that can be carried out mechanically can be performed by a general purpose digital computer. Moreover, the limitations on what we have been able to make computers do so far clearly come far more from our weakness as programmers than from the intrinsic limitations of the machines. We hope that these limitations can be greatly reduced by developing a mathematical science of computation. There are three established directions of mathematical research relevant to a science of computation. The first and oldest of these is numerical analysis. Unfortunately, its subject matter is too narrow to be of much help in forming a general theory, and it has only recently begun to be affected by the existence of automatic computation. The second relevant direction of research is the theory of computability as a branch of recursive function theory. The results of the basic work in this theory, including the existence of universal machines and the existence of unsolvable problems, have established a framework in which any theory of computation must fit. Unfortunately, the general trend of research in this field has been to establish more and better unsolvability theorems, and there has been very little attention paid to positive results and none to establishing the properties of the kinds of algorithms that are actually used. Perhaps for this reason the formalisms for describing algorithms are too cumbersome to be used to describe actual algorithms. The third direction of mathematical research is the theory of finite automata. Results which use the finiteness of the number of states tend not to be very useful in dealing with present computers which have so l) This paper is a corrected version of the paper of the same title given at the Western Joint Computer Conference, May 1961. A tenth section discussing the relations between mathematical logic and computation has been added.

34

JOHN MCCARTHY

many states that it is impossible for them to go through a substantial fraction of them in a reasonable time. The present paper is an attempt to create a basis for a mathematical theory of computation. Before mentioning what is in the paper, we shall discuss briefly what practical results can be hoped for from a suitable mathematical theory. This paper contains direct contributions towards only a few of the goals to be mentioned, but we list additional goals in order to encourage a gold rush. 1. To develop a universal programming language. We believe that this goal has been written off prematurely by a number of people. Our opinion of the present situation is that ALGOLis on the right track but mainly lacks the ability to describe different kinds of data, that COBOLis a step up a blind alley on account of its orientation towards English which is not well suited to the formal description of procedures, and that UNCOLis an exercise in group wishful thinking. The formalism for describing computations in this paper is not presented as a candidate for a universal programming language because it lacks a number of features, mainly syntactic, which are necessary for convenient use. 2. To define a theory of the equivalence ofcomputationprocesses. With such a theory we can define equivalence preserving transformations. Such transformations can be used to take an algorithm from a form in which it is easily seen to give the right answers to an equivalent form guaranteed to give the same answers but which has other advantages such as speed, economy of storage, or the incorporation of auxiliary processes. 3. To represent algorithms by symbolic expressions in such a way that significant changes in the behavior represented by the algorithms are represented by simple changes in the symbolic expressions. Programs that are supposed to learn from experience change their behavior by changing the contents of the registers that represent the modifiable aspects of their behavior. From a certain point of view, having a convenient representation of one’s behavior available for modification is what is meant by consciousness. 4. To represent computers as well as computations in a formalism that permits a treatment of the relation between a computation and the computer that carries out the computation. 5. To give a quantitative theory of computation. There might be a quantitative measure of the size of a computation analogous to Shannon’s measure of information. The present paper contains no information about this.

A BASIS FOR A MATHEMATICAL THEORY OF COMPUTATION

35

The present paper is divided into two sections. The first contains several descriptive formalisms with a few examples of their use, and the second contains what little theory we have that enables us to prove the equivalence of computations expressed in these formalisms. The formalisms treated are the following: 1. A way of describing the functions that are computable in terms of given base functions, using conditional expressions and recursive function definitions. This formalism differs from those of recursive function theory in that it is not based on the integers, strings of symbols, or any other fixed domain. 2. Computable functionals, i.e. functions with functions as arguments. 3. Non-computable functions. By adjoining quantifiers to the computable function formalism, we obtain a wider class of functions which are not a priori computable. However, such functions can often be shown to be equivalent to computable functions. In fact, the mathematics of computation may have as, one of its major aspects, rules which permit us to transform functions from a non-computable form into acomputable form. 4. Ambiguous functions. Functions whose values are incompletely specified may be useful in proving facts about functions where certain details are irrelevant to the statement being proved. 5. A way of defining new data spaces in terms of given base spaces and of defining functions on the new spaces in terms of functions on the base spaces. Lack of such a formalism is one of the main weaknesses of ALGOL,but the business data processing languages such as FLOWMATIC and COBOLhave made a start in this direction, even though this start is hampered by concessions to what the authors presume are the prejudices of business men. The second part of the paper contains a few mathematical results about the properties of the formalisms introduced in the first part. Specifically, we describe the following : 1. The formal properties of conditional expressions. 2. A method called recursion induction for proving the equivalence of recursively defined functions. 3. Some relations between the formalisms introduced in this paper and other formalisms current in recursive function theory and in programming. We hope that the reader will not be angry about the contrast between the great expectations of a mathematical theory of computation and the meager results presented in this paper.

36

JOHN MCCARTHY

FORMALISMS FOR DESCRIBING COMPUTABLE FUNCTIONS AND RELATED ENTITIES In this part we describe a number of new formalisms for expressing computable functions and related entities. The most important section is 1, the subject matter of which is fairly well understood. The other sections give formalisms which we hope will be useful in constructing computable functions and in proving theorems about them. 1. Functions Computable in Terms of Given Base Functions. Suppose we are given a base collection 9 of functions (including predicates) having certain domains and ranges. In the case of the non-negative integers, we may have the successor function and the predicate of equality, and in the case of the S-expressions discussed in reference 7, we have the five basic operations. Our object is to define a class of functions C { 9 } which we shall call the class of functions computable in terms of 9. Before developing C { 9 } formally, we wish to give an example, and in order to give the example, we first need the concept of conditional expression. In our notation a conditional expression has the form (pi -+ e1,pz -+ ez, . . . , pn -+ en)

which corresponds to the ALGOL60 reference language (12) expression

i f p l then el else i f p z then ez

... else ifpn then en.

Here p l , . . . , p n are propositional expressions taking the values T or F standing for truth and falsity respectively. The value of (PI -+ el, p z -+ e2,... , p n -+ en) is the value of the e corresponding to the first p that has value T. Thus (4< 3

-+

7 , 2 > 3 +- 8 , 2 < 3

-+

9,4< 5

-+

7)

= 9.

Some examples of the conditional expressions for well known functions are 1x1 = ( x < 0 -+ -x,x 2 0 -f x) 611 = (i=j -+ 1, i # j + 0) and the triangular function whose graph is given in figure 1 is represented

A BASIS FOR A MATHEMATICAL. THEORY OF COMPUTATION

by the conditional expression tri(x) = (XI - 1 + 0 , x s 0 + x + 1, x < 1 + 1-x, x

37

> 1 + 0).

Fig. 1

Now we are ready to use conditional expressions to define functions recursively. For example, we have

n! = (n = 0 + 1, n # 0 + n*(n- I)!) Let us evaluate 2! according to this definition. We have 2! = (2 = 0 + 1,2 # 0 + 2.(2- I)!) = 2.1! = 2.(1 = 0 + 1, 1 # 0 + l ’ ( 1 - I)!) =2*1*0! = 2 . 1 * ( 0 = 0 + 1 , 0 #O+O.(O-l)!) = 2.1-1 = 2.

The reader who has followed these simple examples is ready for the construction of C{S} which is a straightforward generalization of the above together with a tying up of a few loose ends. Some notation. Let 9 be a collection (finite in the examples we shall give) of functions whose domains and ranges are certain sets. C { 9 } will be a class of functions involving the same sets which we shall call computable in terms of S. Suppose f is a function of n variables, and suppose that if we write y = f(x1,. ..,x,), each xi takes values in the set Ut and y takes its value in the set V. It is customary to describe this situation by writing

...x Un + V (XI,. ..,x,) is called

f : U l X UZX

...

The set U1 x X Un of n-tuples and the set V is called the range off.

the domain of f,

38

JOHN MCCARTHY

Forms and functions. In order to make properly the definitions that follow, we will distinguish between functions and expressions involving free variables. Following Church [l] the latter are called forms. Single letters such a s f , g, h, etc. or sequences of letters such as sin are used to denote functions. Expressions such as f(x,y), f(g(x),y), x2+y are called forms. In particular we may refer to the function f defined by f(x,y) = X ~ + J J . Our definitions will be written as though all forms involving functions were written f(, ...,) although we will use expressions like x JJ with infixes like in examples. Composition. Now we shall describe the ways in which new functions are defined from old. The first way may be called (generalized) composition and involves the use of forms. We shall use the letters x,y, ... (sometimes with subscripts) for variables and will suppose that there is a notation for constants that does not make expressions ambiguous. (Thus, the decimal notation is allowed for constants when we are dealing with integers.) The class of forms is defined recursively as follows: (i) A variable x with an associated space U is a form, and with this form we also associate U.A constant in a space U is a form and we also associate U with this form. (ii) If el,. ..,en are forms associated with the spaces U1,...,Un respectively, then f(e1,...,en) is a form associated with the space V. Thus the form f(g(x,y),x) may be built from the forms g(x,y) and x and the function f. If all the variables occurring in a form e are among X I , . ..,Xn, we can define a function h by writing h(x1,...,xn) = e. We shall assume that the reader knows how to compute the values of a function defined in this way. If fi,...,fm are all the functions occurring in e we shall say that the function h is defined by composition from fi,...,fm. The class of functions definable from given functions using only composition is narrower than the class of function computable in terms of these functions. Partial functions. In the theory of computation it is necessary to deal with partial functions which are not defined for all n-tuples in their domains. Thus we have the partial function minus, defined by minus (x,y) = x - y , which is defined on those pairs (x,y) of positive integers for which x is greater than y. A function which is defined for all n-tuples in its domain is called a totalfunction. We admit the limiting case of a partial function which is not defined for any n-tuples.

+

+

A BASIS FOR A MATHEMATICAL THEORY OF COMPUTATION

39

The n-tuples for which a function described by composition is defined is determined in an obvious way from the sets of n-tuples for which the functions entering the composition are defined. If all the functions occurring in a composition are total functions, the new function is also a total function, but the other processes for defining functions are not so kind to totality. When the word “function” is used from here on, we shall mean partial function. Having to introduce partial functions is a nuisance, but an unavoidable one. The rules for defining computable functions sometimes give computation processes that never terminate, and when the computation process fails to terminate, the result is undefined. It is well known that there is no effective general way of deciding whether a process will terminate. Predicates and propositional forms. The space 17 of truth values whose only elements are T (for truth) and F (for falsity) has a special role in our theory. A function whose range islTis called apredicate. Examples of predicates on the integers are prime defined by prime@) =

T if x is prime F otherwise

Iess(x,y) =

Tifx n + 0,T .+f ( m ) +t=m+l 2 f(i)).

t=m

Functions with functions as arguments are called functionuls. Another example is the functional feast(p) which gives the least integer n such that p(n) for a predicate p . We have least(p) = least 2(p,O)

where

leust2(p,n) = (p(n) -+ n,T

-+

leust2(p,n+ 1)).

In order to use functionals it is convenient to have a notation for naming functions. We use Church's [l] lambda notation. Suppose we have a function f defined by an equation f(x1,...,xn) = e where e is some expression in X I , . ..,xn. The name of this function is A( ( X I ,...,xn),e). For example, the name of the function f defined byf(x,y) = x 2 + y is 4 (XYY), x2 Y). Thus we have %(X,Y)YX2+ Y)(3,4) = 13, but 4cv,x),x2 Y)(3,4) = 19.

+

+

The variables occurring in a il definition are dummy or bound variables and can be replaced by others without changing the function provided the replacement is done consistently. For example, the expressions 4(XYY)Y

and

+Y), + 4,

x2

4(u,v),u2

all represent the same function. In the notation

n

i2

t -1

integer n for which n2

is represented by sum(l,n,il((i),i2))and the least

> 50 is represented by leust (A( (n),n2 > 50)).

46

JOHN MCCARTHY

When the functions with which we are dealing are defined recursively, a difficulty arises. For example, considerfactorial defined by fuctoriul(n) = (n = 0 + 1,T + n-fuctoriuZ(n-1)). The expression A((n),(n = 0 + 1,T + n.fuctoriul(n-1)))

cannot serve as a name for this function because it is not clear that the occurrence of “factorial” in the expression refers to the function defined by the expression as a whole. Therefore, for recursive functions we adopt an additional convention. Namely, lubel(f,l((xl,...,x,),e)) stands for the function f defined by the equation f(x1,...,x n )

=e

where any occurrences of the function letter f within e stand for the function being defined. The letter f is a dummy variable. The factorial function then has the name

-

lubeZ(fuctoriul,A( (n),(n = 0 + 1,T + n fuctoriuZ(n- l ) ) ) ) ,

and since factorial and n are dummy variables the expression lubel(g,A((r),(r= 0 -+ 1,T + r - g ( r - 1 ) ) ) )

represents the same function. If we start with a base domain for our variables, it is possible to consider a hierarchy of functionals. At level 1 we have functions whose arguments are in the base domain. At level 2 we have functionals taking functions of level 1 as arguments. At level 3 are functionals taking functionals of level 2 as arguments, etc. Actually functionals of several variables can be of mixed type. However, this hierarchy does not exhaust the possibilities, and if we allow functions which can take themselves as arguments we can eliminate the use of Zubel in naming recursive functions. Suppose that we have a function f defined by

f(x)

=

4m

where S(x,f, is some expression in x and the function variable5 This function can be named ,ubelCfAI((x),b(x,f))).

A BASIS FOR A MATHEMATICAL THEORY OF COMPUTATION

47

However, suppose we define a function g by or We then have

g(xYv) = g=

A((&?4X,d))

(xyv)9&x,

(X)Y&Yd

1)1.

f(x) =

since g(x,g) satisfies the equation g(x,g) = a(xY4(x),g(x,g))).

Now we can write f as

f = mx), w , 9 v ( y Y m h ? J ( u 9 d ) ) )(x, W l d Y

m),v(uYv))))>).

This eliminates label at what seems to be an excessive cost. Namely, the expression gets quite complicated and we must admit functionals capable of taking themselves as arguments. These escape our orderly hierarchy of functionals. 4. Non-Computable Functions and Functionals. It might be supposed that in a mathematical theory of computation one need only consider computable functions. However, mathematical physics is carried out in terms of real valued functions which are not computable but only approximable by computable functions. We shall consider several successive extensions of the class C { 9 } . First we adjoin the universal quantifier y to the operations used to define new functions. Suppose e is a form in a variable x and other variables associated with the space 17 of truth values. Then V( ( 4 9 4

is a new form in the remaining variables also associated with 17.V( (x), e) has the value T for given values of the remaining variables if for all values of x, e has the value T. V((x),e) has the value F if for at least one value of x, e has the value F. In the remaining case, i.e. for some values of xye has the value T and for all others e is undefined,V( (x), e) is undefined. If we allow the use of the universal quantifier to form new propositional forms for use in conditional forms, we get a class of functions H a { 9 } which may well be called the class of functions hyper-arithmetic over 9 since in the case where 9= {successor, equality} on the integers, H a { 9 } consists of Kleene’s hyper-arithmetic functions.

48

JOHN MCCARTHY

Our next step is to allow the description operator 1. i((x),p(x))stands for the unique x such that p ( x ) is true. Unless there is such an x and it is unique, L((x),p(x))is undefined. In the case of the integersL( ( ( x ) , p ( x ) ) can be defined in terms of the universal quantifier using conditional expressions, but this does not seem to be the case in domains which are not effectively enumerable, and one may not wish to do so in domains where enumeration is unnatural. The next step is to allow quantification over functions. This gets us to Kleene’s [ 5 ] analytic hierarchy and presumably allows the functions used in analysis. Two facts are worth noting. First V(cf>,tp(f>) refers to all functions on the domain and not just the computable ones. If we restrict quantification to computable functions, we get different results. Secondly, if we allow functions which can take themselves as arguments, it is difficult to assign a meaning to the quantification. In fact, we are apparently confronted with the paradoxes of naive set theory. 5. Ambiguous Functions. Ambiguous functions are not really functions.

For each prescription of values to the arguments the ambiguous function has a collection of possible values. An example of an ambiguous function is less(n) defined for all positive integer values of n. Every non-negative integer less than n is a possible value of less@). First we define a basic ambiguity operator amb(x,y) whose possible values are x and y when both are defined: otherwise, whichever is defined. Now we can define less(n) by less (n) = amb (n- 1, less (n- 1)). less(n) has the property that if we define then

ult(n) = (n = 0 + 0,T + ulr(less(n)))

v((n),ult(n)= 0) = T.

There are a number of important kinds of mathematical arguments whose convenient formalization may involve ambiguous functions. In order to give an example, we need two definitions. Iff and g are two ambiguous functions, we shall say thatfis a descendant of g if for each x every possible value of f ( x ) is also a possible value of Secondly, we shall say that a property of ambiguous functions is

A BASIS FOR A MATHEMATICAL THEORY OF COMPUTATION

49

hereditary if whenever it is possessed by a function g it is also possessed by all descendants of g . The property that iteration of an integer valued function eventually gives 0 is hereditary, and the function less has this property. So, therefore, do all its descendants. Therefore any integerfunction g satisfying g(0) = 0 and n > 0 = g(n) < n has the property that g*(n) = (n = 0 +- 0,T +- g*(g(n))) is identically 0 since g is a descendant of less. Thus any function, however complicated, which always reduces a number will if iterated sufficiently always give 0. This example is one of our reasons for hoping that ambiguous functions will turn out to be useful. With just the operation amb defined above adjoined to those used to generate C{F}, we can extend F to the class C * { Y } which may be called the computably ambiguous functions. A wider class of ambiguous functions is formed using the operator Am(x,n(x)) whose values are all x’s satisfying n(x).

6. Recursive Definitions of Sets. In the previous sections on recursive definition of functions the domains and ranges of the basic functions were prescribed and the defined functions had the same domains and ranges. In this section we shall consider the definition of new sets and the basic functions on them. First we shall consider some operations whereby new sets can be defined. 1. The Cartesian product A x B of two sets A and B is the set of all ordered pairs ( a - b ) with a E A and b E B. If A and B are finite sets and n(A) and n(B) denote the numbers of members of A and B respectively then n(A x B ) = n(A)*n(B). Associated with the pair of sets (A,B) are two canonical mappings: ZA,B: A x B + A defined by nA,B((a*b))= a @ A , B : A X B +- B defined by @A,B((a*b)) = b.

The word “canonical” refers to the fact that Z A , B and @ A , B are defined by the sets A and B and do not depend on knowing anything about the members of A and B. The next canonical function y is a function of two variables YA,B:A,B+A x B defined by y ~ , ~ ( a ,= b )( a - b ) . For some purposes functions of two variables, x from A and y from B, can be identified with functions of one variable defined on A x B.

50

JOHN MCCARTHY

2. The direct union A o B of the sets A and B is the union of two non-intersecting sets one of which is in 1- 1 correspondence with A and the other with B. If A and B are finite, then n(A o B ) = n(A)+ n(B) even if A and B intersect. The elements of A o B may be written as elements of A or B subscripted with the set from which they come, i.e. U A Or bB. The canonical mappings associated with the direct union A 8 B are ~ A , B :+ A A 0 B defined by iA,B(a) = a ~ , j A , B : B + A 0 B defined by jA,B(b) = be, PA,B:Ao B + 17defined by P A , B ( X ) = T if and only i f x comes fromd, qA,B: A O B +. 17 defined by qA,B(X) = T if and only if x comes from B. There are two canonical partial functions rA,BandsA,B. ~ A , B :0 A B-+ A is defined only for elements coming from A and satisfies rA,B(iA,B(U))= a. Similarly, SA,B: A O B + B satisfies sA,B(jA,B(b)) = b. 3. The power set A B is the set of all mappingsf: B + A. The canonical mapping OIA,B:AB x B + A is defined by aA,B(f,b)=f(b). Canonical mappings. We will not regard the sets A x ( B x C ) and ( A x B) x C as the same, but there is a canonical 1-1 mapping between them, gA,B,C:(A X B) X c -+ A X ( B X c)

to express the fact that these sets are canonically isomorphic. Other canonical isomorphisms are 1. fA,B:A X B + B X A defined by t(u) = ~B,A(eA,B(u),~A,B(~)) 2. d ~ : A X ( B O C ) - + A X B O A x C 3. u ~ : ( A O B ) O C +- A O ( B Q C ) 1. &:AC x BC + ( A X B)c 5. d3:ABX Ac +- AB@C 6. s~:(AB)C+- ABxC

We shall denote the null set (containing no elements) by 0 and the set

A BASIS FOR A MATHEMATICAL THEORY OF COMPUTATION

51

consisting of the integers from 1 to n by n. We have AOO2:A

AxOllO A x l l l A A x 2 21 A 8 A (n terms, associate to left by convention) AOII 1 (by convention) Al- A An I IAX .x A (n terms, associate to left by convention)

..

Suppose we write the recursive equation

S={A} e A x S . We can interpret this as defining the set of sequences of elements of A as follows : 1. Interpret A as denoting the null sequence. Then the null 'sequence (strictly an image of it) is an element of S. 2. Since a pair consisting of an element of A and an element of S is an element of S, a pair (a,A)is an element of S. So, then, are al-(a2.A))

and

al~(a~-(u~~A))) etc.

Thus S consists of all sequences of elements of A including the null sequence. Suppose we substitute { A > 8 A x S for S in the right side of S = { A } 8AxS.Weget

s = {A} 8 A x ({A}

8A

x S).

If we again substitute for S and expand by the distributive law expressed in equation (2) above we get s={A}oAx{A}8AxAx{A>o

...

which, if we now denote the set {A} by 1, becomes S=l 8AOA2QASQ

...

which is another way of writing the set of sequences. We shall denote the set of sequences of elements of A by seq(A). We can also derive this relation by writing S = 1 8 A x S and solving formally for S, getting S = 1/(1-A) which we expand in geometric series to get S = 1 8 A 8 A2 8 ... just as before.

52

JOHN MCCARTHY

Another useful recursive construction is S

=A

O S x S.

Its elements have the forms a or (al-a2)or ((uI*u~)*u~) or (a1-(a2-a3)) etc. Thus we have the set of S-expressions on the alphabet A which we may denote by sexp(A). This set is the subject matter of Reference 7, and the following paragraph refers to this paper. When sets are formed by this kind of recursive definition, the canonical mappings associated with the direct sum and Cartesian product operations have significance. Consider, for example, sexp(A). We can define the basic operations of Lisp, i.e. atom, eq, car, cdr and cons by the equations atom(x) = P A , S ~ S ( X ) eq(x,y) = ( ~ A , S X S ( X = ) iA,sxs(y)) assuming that equality is defined on the space A .

Definition of the set of integers. Let 0 denote the null set as before. We can define the set of integers I by

Its elements are then

I = (0) 0 (0) x I. O,(O.O), (O.(O.O)),

etc.

which we shall denote by 0,1,2,3 etc. The successor and predecessor functions are then definable in terms of the canonical operations of the defining equation. We have

PROPERTIES OF COMPUTABLE FUNCTIONS The first part of this paper was solely concerned with presenting descriptive formalisms. In this part we shall establish a few of the proper-

A BASIS FOR A MATHEMATICAL THEORY OF COMPUTATION

TTT TTF TTu

T T T

a a a

a a a

a b

a

U

a

T F T T F F TFu

F F F

b b b

b b b

a b

b b b

TuT TuF Tuu

U U

U

U U

a

U

b

U U

U

U

U

U

U

FTT F T F FTu

T F

a b

a a

a

a

b

b

U

U

a

U

U

F F T F F F F F u

T F

a

a b

b

U

U

b b b

U

U

FuT FuF Fuu

T F

a b

U

a b

a b

U

U

U U

U

U

U

uTT uTF uTu

b

U

a

a

U

U

a

U U

U U

a

a b

a

U

U U

uFT u F F uFu

U

U

U

U

a

U

b b b

a

U U

U

U U

uuT uuF

U U

U U

U U

a b

U U

u u u

I(

U

U

U

U

53

54

JOHN MCCARTHY

ties of the entities we previously introduced. The most important section is section 8 which deals with recursion induction. 7. Formal Properties of Conditional Forms. The theory of conditional expressions corresponds to analysis by cases in mathematics and is only a mild generalization of propositional calculus. We start by considering expressions called generalized Boolean forms (gbf) formed as follows: 1. Variables are divided into propositional variables p , q, r, etc. and general variables x , y, z, etc. 2. We shall write (p -+ x,y) for (p --f x,T -+ y). (p -+ x,y) is called an elementary conditional form (ecf) of which p , x , and y are called the premiss, conclusion and the alternative, respectively. 3. A variable is a gbf, and if it is a propositional variable it is called a propositional form (pf). 4. If n is a pf and 01 and B are gbfs, then (n+or$) is a gbf. If, in addition, 01 and /?are pfs, so is (n-+ 01,p). The value of a gbf 01 for given values (T, F or undefined)of the propositional variables will be T or F in case 01 is a pf or a general variable otherwise. This value is determined for a gbf (n -+ 01,m according to the table value ((n-+ or,B)) value (n) T F undefined

value (a) value (B) undefined

We shall say that two gbfs are strongly equivalent if they have the same value for all values of the propositional variables in them including the case of undefined propositional variables. They are weakly equivalent if they have the same values for all values of the propositional variables when these are restricted to F and T. The equivalence of gbfs can be tested by a method of truth tables identical to that of propositional calculus. The table for ((p -+ q,r) -+ a,b and (p -+ ( q -+ a,b)(r -+ a$)) is given on the foregoing page. According to the table, ((p -+ q,r) -+ a,b)and (p -+ (q -+ a,b),(r + a$)) are strongly equivalent. For weak equivalence the u case can be left out of the table. Consider the table

A BASIS FOR A MATHEMATICAL THEORY OF COMPUTATION

t

P4

3

T T F T

a b a b

T F T T

t

t

t

W

rn

1. 4

4

4

C

a

U

d

b

a

C

C

C

d

d

C

b b d d

55

t

rn

W

a b C

d

which proves that (p -+ (q -+ a,b),(q -+ c,d)) and ( q + (p -+ a,c),(p -+ b,d)) are weakly equivalent. They are also strongly equivalent. We shall write f a and -w for the relations of strong and weak equivalence. There are two rules whereby an equivalence can be used to generate other equivalences. 1. If dc = /Iand al= is the result of substituting any gbf for any variable in dc = p, then dc1= PI. This is called the rule of substitution. 2. If a = p and a is subexpression of y and 6 is the result of replacing an occurrence of a in y by an occurrence of /?, then y = 6. This is called the rule of replacement. These rules are applicable to either strong or weak equivalence and in fact to much more general situations. Weak equivalence corresponds more closely to equivalence of truth functions in propositional calculus than does strong equivalence. Consider the equations 1) ( p -+ a,a) -w

2) (T 3) (F

a

a,b)- g u -+ a,b) -8 b 4) (p -+ T , F ) r g p 5 ) (p (p a,b),c) (p a,c) 6 ) ( p a,(p b ~ )? s)( p W) 7) ((p q,r) -S(p (4 a,b),(r a,b)) 8) ((p (4 + a,b)7(q -+ c,d)) (4 (p j -+

--f

-+

-+

-+

-+ b 7 d ) )

All are strong equivalence except the fmt, and all can be proved by truth tables. These eight equations can be used as axioms to transform any gbf into any weakly equivalent one using substitution and replacement.

56

JOHN MCCARTHY

In fact, they can be used to transform any gbf into a canonical form. This canonical form is the following. Let PI,. ..,pn be the variables of the gbf a taken in an arbitrary order. Then a can be transformed into the form where each at has the form at

(p1 + ao,a1) = (p2 + Ut0,Uil)

and in general for each k = 1, ...,n-1 at l...fk

and each at,,

...,at

= (Pk+l

-+

atl...tkO,

afl...tk1)

is a truth value or a general variable.

For example, the canonical form of ((P

-+

494

-+

4)

with the variables taken in the order r, q,p is (r

+.

(4 +. ( p + a,a),(p

+

b,a)),(q + (P + a,b),(p

-+

b,b))).

In this canonical form, the 212 cases of the truth or falsity of p l , . ..,pn are explicitly exhibited. An expression may be transformed into canonical form as follows: 1) Axiom 7 is used repeatedly until in every subexpression the n in (n --f a$) consists of a single propositional variable. 2) The variable p~ is moved to the front by repeated application of axiom 8. There are three cases: (q + (PI + a,b),(pl -+ c,d)) to which axiom 8 is directly applicable; (q -+ a,(pl -+ c,d)) where axiom 8 becomes applicable after axiom 1 is used to make it (q -+ (pl -+ a,a),(pl -+ c,d)); the case (q + ( p l + a,b),c) which is handled in a manner similar to that of case 2. Once the main expression has the form ( p l -+ a$) we move any pl’s which occur in a and p to the front and eliminate them using axioms 5 and 6 . We then bringpz to the front ofa and j3 using axiom 1 if necessary to guarantee at least one occurrence of p2 in each of LY and p. The process is continued until the canonical form is achieved. There is also a canonical form for strong equivalence. Any gbf a is strongly equivalent to one of the form ( p l + or$), where a and j3 do not contain p l and are themselves in canonical form. However, the variable p l may not be chosen arbitrarily but must be an inevitable propositional variable of the original gbf and can be chosen to be any

A BASIS FOR A MATHEMATICAL THEORY OF COMPUTATION

57

inevitable variable. An inevitable variable of a gbf (n -+ a$) is defined to be either the first propositional variable or else an inevitable variable of both a and /?.Thus p and q are the inevitable variables of (P

+.

(r

+

(4

-+

a,b),(q +. c,d)),(q

+.

e,f)).

A gbf a may be put in strong canonical form as follows: 1) Use axiom 7 to get all premises as propositional variables. 2) Choose any inevitable variable, say P I , and put a in the form (pl -+ a,/?)by using axiom 8. 3) The next step is to eliminate occurrences of p l in a and B. This can be done by the general rule that in any ecf occurrences of the premiss in the conclusion can be replaced by T and occurrences in the alternative by F. However, if we wish to use substitution and replacement on formulas we need the additional axioms ( 4 +. a,b),c)-8 (P

(9) and

(p

+.

(10)

(p

+ a,(q +

-+

( 4 +. (P

+.

b,c)) -8 ( p + a,(q + ( P

w ) , O , +. b,b)),c)

-+

W,(P+. c,c))).

Suppose there is an occurrence of p l in the conclusion; we want to replace it by T. To do this, we use axioms 9 and 10 to move in a p l until the objectionable pl occurs as the inner p l of one of the forms or In either case, the objectionable p l can be removed by axiom 5 or 6, and the pl’s that were moved in can be moved out again. Thus we have ( P I + a,/?)with p l missing from a and /?. 4) Inevitable variables are then brought to the front of a andpand so forth. Two gbfs are equivalent (weakly or strongly) if and only if they have the same (weak or strong) canonical form. One way this is easy to prove; if two gbfs have the same canonical form they can be transformed into each other via the canonical form. Suppose two gbfs have different weak canonical forms when the variables are taken in the same order. Then values can be chosen for the p’s giving different values for the form proving non-equivalence. In the strong case, suppose that two gbfs do not have the same inevitable propositional variables. Let p be inevitable

58

JOHN MCCARTHY

in a but not in b. Then if the other variables are assigned suitable values b will be defined with p undefined. However, a will be undefined since p is inevitable in a which proves non-equivalence. Therefore, strongly equivalent gbfs have the same inevitable variables, so let one of them be put in front of both gbfs. The process is then repeated in the conclusion and alternative etc. The general conditional form

(pi -+ el,. ..,pn

-+

en)

can be regarded as having the form (pi

-+

el,(p2

-+

ez, ...,(pn

.+

en,U),...))

where u is a special undefined variable and their properties can be derived from those of gbf's. The relation of functions to conditional forms is given by the distributive law

..

f ( ~ i , .,xi-i,(pi

.

-+ el,. .,pn

.

-+ en),xi+i,.. , x k ) = (pi -+ f(xi,. ..,xi-i, ei,xr+i,...,x k ) , ...,Pn -+ f(xi,. ..,xi-i, en,xt+i,...,x k ) ) . -+

The rule of replacement can be extended in the case of conditional expressions. Suppose a is an occurrence of a subexpression of an expression p. We define a certain propositional expression n called the premiss of a in /3 as follows: 1) The premiss of 01 in 01 is T 2) The premiss of 01 in f ( x 1 , . ..,xr, ...,xn) where 01 is part of x{ is the premiss of a in xg. 3) If 01 occurs in eg and the premiss of a in er is n, then the premiss of a in (pi -+ el ,...,pi -+ et ,...,pn -+ en) is ( N p i A ... A pt-1) A pr A n. 4) If a occurs in pt and the premiss of a in pg is n, then the premiss pi-i A Z. of 01 in (pi -+ el,. ..,pi -+ et,. ..,pn -+ en) is p i A ... The extension of the rule of replacement is that an occurrence of (n -+ a') where n is the 01 in may be replaced by a' if (n -+ a) premiss of a in p. Thus in a subcase one needs only prove equivalence under the premiss of the subcase. N

N

N

8. RecursionInduction. Suppose a function f is defined recursively by (1)

.

f(x1,. .,xn) = g(x1,. . x n f } e ,

59

A BASIS FOR A MATHEMATICAL THEORY OF COMPUTATION

where d is an expression that in general contains $ Suppose that d is the set of n-tuples ( X I , . ..,xn) for which f is defined. Now let g and h be two other functions with the same domain as f and which are defined for all n-tuples in d . Suppose further that g and h satisfy the equation which defined$ We assert that g(X1,. .., x n )

= h(x1,.

.., ~ n )

for all ( X I , . ..,xn) in d .This is so, simply because equation (1) uniquely determines the value that any function satisfying it has for arguments in d which in turn follows from the fact that (1) can be used to compute f ( x 1 , . ..,xn) for (XI,. ..,x,) in d . We shall call this method of proving two functions equivalent by the name of recursion induction. We shall develop some of the properties of the elementary functions of integers in order to illustrate proof by recursion induction. We recall the definitions m + n = (n = 0 + m,T + m‘+n-) mn = (n = 0 + O,T + m+mn-) Th. 1. m + O Proof m 0

+

=m = (0 = 0

- m.

+ m,T + m’

+ 0-)

Only the definition of addition and the properties of conditional expressions were used in this proof.

+

+

Th. 2. (m n)’ = m‘ n Proof Definef(m,n) = (n = 0 + m‘,T + f(m,n-)). It is easily seen what f(m,n) converges for all m and n and hence is completely defined by the above equation and is computable from it. Now (m+n)’

= (n = 0

m’+n

= (n = 0

= (n = 0

+ m,T + m’+n-)’

+

+ m‘,T -+ (m’ n-)‘), while -+ m’,T + (m’)‘+n-).

It is easily seen that the functions g and h defined by the equations g(m,n) = (m n)’ and h(m,n) = m’ n both satisfy the equation $ For example, it is clear that g(m‘,n) = (m’ n-)’ and h(m’,n-) = (m’)’ n-. Therefore, by the principle of recursion induction h and g are equivalent functions on the domain of where f is defined, but this is the set of all pairs of integers. The fact that the above defined f(m,n) converges for all m and n is a

+

+

+

+

60

JOHN MCCARTHY

case of the more general fact that all functions defined by equations of the form f ( n , x,...,z) = (n = 0 -+ g(x ,...,z), T -+ h(n,x,...,z, f(n-,r(x,. ..,z),. ..,t ( x , . ..,z)), f(n-,u(x,. ..,z), ...,w(x,. ..,z)),etc.)) converge. We are not yet able to discuss formal proofs of convergence. In presenting further proofs we shall be more terse. Th.3. ( m + n ) + p = ( m + p ) + n . Proof Let f(m,n,p) = (p = 0 + m verges for all m, n, p. We have (m+n)+p

= (p = 0 -+ = (p = 0 -+

(m+p)+n = (p = 0 -+ = (p = 0 -+

+ n,T -+ f(m’,n,p-)).

Again f con-

m+n,T (m+n)’+p-) m n,T -+ (m’ n) +p-) using Th. 2. m,T + m’+p-)+n m+n,T -+ (m‘+p-)+n).

+

-+

+

Each of these forms satisfies the equation for f(m,n,p). Setting m = 0 in Theorem 3 gives (O+n)+p

= (O+A+n

so that if we had O+m = m we would have commutativity of addition. In fact, we cannot prove 0 m = m without making some assumptions that take into account that we are dealing with the integers. For suppose our space consisted of the vertices of the binary tree in figure 2, where

+

Fig. 2

m’is the vertex just above and to the left, and m- is the vertex just below, and 0 is the bottom of the tree. m n can be defined as above and of course satisfies Theorems 1, 2, and 3 but does not satisfy O+m = m. For example, in the diagram 0 a = b although a 0 = a.

+

+

+

A BASIS FOR A MATHEMATICAL THEORY OF COMPUTATION

61

We shall make the following assumptions : 1. m‘ # 0 2. (m’)- = m 3. ( m # 0) 2 ((m-)‘ = m) which embody all of Peano’s axioms except the induction axiom.

+

Th. 4. 0 n = n. Proof Let f ( n ) = (n = 0 -+ O,T -+ f(n-)‘) 0 n = (n = 0 -+ O,T -+ 0’ n-) = (n =O -+ O,T -+ (0 n-)’) n = (n = 0 + n,T -+ n) axiom 3 = (n = 0 -+ O,T -+ (n-)’)

+ +

+

Th.5. m + n = n + m . Proof By 3 and 4 as remarked above. Th.6. ( m + n ) + p = m + ( n + p ) Proof ( m n) + p = ( m + p ) n =(p m) n =( p n) m =m (n + p )

+ + + + + +

+

Th. 3. Th. 5. Th. 3. Th. 5. twice.

Th. 7. ma0 = 0. Proof m ~ O = ( O = O - + O , T + m + n ~ O - )

=o

Th. 8. 0 . n = 0. Proof Let f ( n ) = (n = 0 + O,T + f ( n - ) ) 0 . n = (n = 0 + O,T -+ O+O*n) = (n = 0 0 = (n = 0 -+ O,T + 0) Th. 9. mn’ Proof mn‘

=m

+ mn.

= (n’ = 0 =m+mn

+

-+

O,T

+

-+

O,T + 0-n)

+

m m (n‘)-) axioms 1 and 2.

-+

Th. 10. m(n p ) = mn mp. Proof Let f(m,n,p) = (p = 0 -+ mn,T -+ f(m,n‘,p-)) m(n+p) = m(p = 0 -+ n,T -+ n’+p-)) = (p = 0 -+ mn,T -+ m(n’ + p - ) ) mn+ mp= mn+ (p = 0 -+ O,T -+ m + mp-) =(p=O-+mn+O,T-+mn+(m+mp-)) =(p=O-+mn,T-+(mn+m)+mp-) = (p = 0 -+ mn,T -+ mn’ mp-)

+

62

JOHN MCCARTHY

Now we shall give some examples of the application of recursion induction to proving theorems about functions of symbolic expressions. The rest of these proofs depend on an acquaintance with the Lisp formalism. We start with the basic identities. cur [cons [ x ; y ] ]= x cdr [cons[x;y]] = y atom[x] 2 [cons[cur[x];cdr[XI = x atom [cons[ x ; y ] ]= F nulZ[x]= eq[x;NIL] Let us define the concatenation x*y of two lists x and y by the formula x*y

=

[nuN[x]-+ y ; T

-+

cons[cur[x];cdr[x]*y]]

Our first objective is to show that concatenation is associative. Th. 11. [ x * ~ ] *= z x*[~*z]. Proof We shall show that [x*y]*zand x*[y*z]satisfy the functional equation

f [ x ; y ; z ]= [null[x]-+y*z; T -+ cons[cur[x]; f[cdr[x];y;z]]] First we establish an auxiliary result: cons[u;u]*v= [nuN[cons[u;u]] -+ v ; T -+ cons[cur[cons[u;u]]; cdr [cons[a;u]]*v]] = cons[u;u*v] Now we write

and

[x*y]*z= [null[x]+ y ; T -+ cons[cur[x];cdr[x]*y]]*z = [null[x]-+ y*z; T -+ cons[cur[ X I ; cdr [x]*y]*z] = [null[x]-+ y*z; T -+ cons [car[ X I ; [cdr[x]*y]*z] ] x*[y*z]= [null[x]+ y*z; T + cons[cur[x];cdr[x]*[y*z]]].

From these results it is obvious that both [x*y]*zand x*ly*z] satisfy the functional equation. Th. 12. NIL*x x*NIL Proof NIL*x

=x =X.

= [null[NIL]-+ x ; T -+ cons[cur[NIL];cdr[NIL]*x]] =x x*NIL = [nuN[x]+ N I L ; T -+ cons[cur[x];cdr[x]*NIL]]. Let f [ x ]= [null[x]+ N I L ; T -+ cons[cur[x];f[cdr[x]]]].

A BASIS FOR A MATHEMATICAL THEORY OF COMPUTATION

63

x*NIL satisfies this equation. We can also write for any list x x = [nuZl[x] + X ; T + X] = [nulZ[x] + NIL; T

+ cons[cur[x];c d r [ x ] ] ]

which also satisfies the equation. Next we consider the function reverse[x] defined by reverse[x] = [nuN[x] + NIL; T + reverse[cdr[x]]*cons[cur [x];NIL].

It is not difficult to prove by recursion induction that and

reverse [x*y]

= reverse [y]*reverse[x]

reverse [reverse[ x ]]

= x.

Many other elementary results in the elementary theory of numbers and in the elementary theory of symbolic expressions are provable in the same straightforward way as the above. In number theory one gets as far as the theorem that if a prime p divides ub, then it divides either u or b. However, to formulate the unique factorization theorem requires a notation for dealing with sets of integers. Wilson's theorem, a moderately deep result, can be expressed in this formalism but apparently cannot be proved by recursion induction. One of the most immediate problems in extending this theory is to develop better techniques for proving that a recursively defined function converges. We hope to find some based on ambiguous functions. However, Godel's theorem disallows any hope that a complete set of such rules can be formed. The relevance to a theory of computation of this excursion into number theory is that the theory illustrates in a simple form mathematical problems involved in developing rules for proving the equivalence of algorithms. Recursion induction, which was discovered by considering number theoretic problems, turns out to be applicable without change to functions of symbolic expressions. 9. Relations to Other Formalisms. Our characterization of C(P} as the set of functions computable in terms of the base functions in .F cannot be independently verified in general since there is no other concept with which it can be compared. However, it is not hard to show that all partial recursive functions in the sense of Church and Kleene are in C{succ,eq}. In order to prove this we shall use the definition of partial recursive functions given by Davis [3]. I f we modify definition 1.1 of page

64

JOHN MCCARTHY

41 of Davis [3] to omit reference to oracles we have the following: A function is partial recursive if it can be obtained by a finite number of applications of composition and minimalization beginning with the functions on the following list: 1) x’

2) Urn(xl,...,xn) = xt, 1 5 i 5 n 3) X + Y 4) X-Y = (x-y > O -+ X-y, T -+ 0) 5) XY

All the above functions are in C{succ,eq}. Any C{F} is closed under composition so all that remains is to show that C{succ,eq} is closed under the minimalization operation. This operation is defined as follows: The operation of minimalization associates with each total function f(y,x1,. ..,xn) the function h(x1,. ,xn) whose value for given X I , . ,xn is the least y for which f(y,x1,...,x,) = 0, and which is undefined if no such y exists. We have to show that iff is in C{succ,eq} so is h. But h may be defined by

..

where

hz(y,xl,. .., x n )

Wl,.

=

, x n ) = hZ(O,Xl,.

..

..,Xr)

XI,..., x n ) = 0 -+ y,T

+ ~ z ( Y ’ , x I ., ..,~n)).

The converse statement that all functions in C{succ,eq} are partial recursive is presumably also true but not quite so easy to prove. It is our opinion that the recursive function formalism based on conditional expressions presented in this paper is better than the formalisms which have heretofore been used in recursive function theory both for practical and theoretical purposes. First of all, particular functions in which one may be interested are more easily written down and the resulting expressions are briefer and more understandable. This has been observed in the cases we have looked at, and there seems to be a fundamental reason why this is so. This is that both the original Church-Kleene formalism and the formalism using the minimalization operation use integer calculations to control the flow of the calculations. That this can be done is noteworthy, but controlling the flow in this way is less natural than using conditional expressions which control the flow directly. A similar objection applies to basing the theory of computation on Turing machines. Turing machines are not conceptually different from the

A BASIS FOR A MATHEMATICAL THEORY OF COMPUTATION

65

automatic computers in general use, but they are very poor in their control structure. Any programmer who has also had to write down Turing machines to compute functions will observe that one has to invent a few artifices and that constructing Turing machines is like programming. Of course, most of the theory of computability deals with questions which are not concerned with the particular ways computations are represented. It is sufficient that computable functions be represented somehow by symbolic expressions, e.g. numbers, and that functions computable in terms of given functions be somehow represented by expressions computable in terms of the expressions representing the original functions. However, a practical theory of computation must be applicable to particular algorithms. The same objection applies to basing a theory of computation on Markov’s [9] normal algorithms as applies to basing it on properties of the integers; namely flow of control is described awkwardly. The first attempt to give a formalism for describing computations that allows computations with entities from arbitrary spaces was made by A. P. Ershov [4]. However, his formalism uses computations with the symbolic expressions representing program steps, and this seems to be an unnecessary complication. We now discuss the relation between our formalism and computer programming languages. The formalism has been used as the basis for the Lisp programming system for computing with symbolic expressions and has turned out to be quite practical for this kind of calculation. A particular advantage has been that it is easy to write recursive functions that transform programs, and this makes compilers and other program generators easy to write. The relation between recursive functions and the description of flow control by flow charts is described in Reference 7. An ALGOLprogram can be described by a recursive function provided we lump all the variables into a single state vector having all the variables as components. If the number of components is large and most of the operations performed involve only a few of them, it is necessary to have separate names for the components. This means that a programming language should include both recursive function definitions and ALGOL-like statements. However, a theory of computation certainly must have techniques for proving algorithms equivalent, and so far it has seemed easier to develop proof techniques like recursion induction for recursive functions than for ALGOL-like programs.

66

JOHN MCCARTHY

10. On the Relations between Computation and Mathematical Logic In what follows computation and mathematical logic will each be taken in a wide sense. The subject of computation is essentially that of artificial intelligence since the development of computation is in the direction of making machines carry out ever more complex and sophisticated processes, i.e. to behave as intelligently as possible. Mathematical logic is concerned with formal languages, with the representation of information of various mathematical and non-mathematical kinds in formal systems, with relations of logical dependence, and with the process of deduction. In discussions of relations between logic and computation there has been a tendency to make confused statements, e.g. to say that aspect A of logic is identical with aspect B of computation, when actually there is a relation but not an identity. We shall try to be precise. There is no single relationship between logic and computation which dominates the others. Here is a list of some of the more important relationships. 1. Morphological parallels The formal command languages in which procedures are described, e.g. ALGOL;the formal languages of mathematical logic, e.g. first order predicate calculus; and natural languages to some extent: all may be described morphologically (i.e., one can describe what a grammatical sentence is) using similar syntactical terms. In my opinion, the importance of this relationship has been exaggerated, because as soon as one goes into what the sentences mean the parallelism disappears.

2. Equivalent classes of problems Certain classes of problems about computations are equivalent to certain classes of problems about formal systems. For example, let El be the class of Turing machines with initial tapes, E2 be the class of formulas of the first order predicate calculus, E3 be the class of general recursive functions, E4 be the class of formulas in a universal Post canonical system, Eg be a class of each element which is a Lisp S-function f together with a suitable set of arguments ul, ...,uk, Es be a program for a stored program digital computer. About El we ask: Will the machine ever stop? About E2 we ask: Is the formula valid? About Es we ask: Isf(0) defined?

A BASIS FOR A MATHEMATICAL THEORY OF COMPUTATION

67

About Eq we ask: Is the formula a theorem? About E5 we ask : Is f [ u l ;... ;u k ] defined? About E6 we ask: Will the program ever stop? For any pair (Et,Ej) we can define a computable map that takes any one of the problems about elements of Et into a corresponding problem about an element of E4 and which is such that the problems have the same answer. Thus, for any Turing machine and initial tape we can find a corresponding formula of the first order predicate calculus such that the Turing machine will eventually stop if and only if the formula is valid. In the case of E6 if we want strict equivalence the computer must be provided with an infinite memory of some kind. Practically, any 15 present computer has so many states, e.g. 236.2 , that we cannot reason from finiteness that a computation will terminate or repeat before the solar system comes to an end and one is forced to consider problems concerning actual computers by methods appropriate to machines with an infinite number of states. These results owe much of their importance to the fact that each of the problem classes is unsolvable in the sense that for each class there is no machine which will solve all the problems in the class. This result can most easily be proved for certain classes (traditionally Turing machines), and then the equivalence permits its extension to other classes. These results have been generalized in various ways. There is the work of Post, Myhill, and others, on creative sets and the work of Kleene on hierarchies of unsolvability. Some of this work is of potential interest for computation even though the generation of new unsolvable classes of problems does not in itself seem to be of great interest for computation. 3. Proof procedures and proof checking procedures The next relation stems from the fact that computers can be used to carry out the algorithms that are being devised to generate proofs of sentences in various formal systems. These formal systems may have any subject matter of interest in mathematics, in science, or concerning the relation of an intelligent computer program to its environment. The formal system on which the most work has been done is the first order predicate calculus which is particularly important for several reasons. First, many subjects of interest can be axiomatized within

68

JOHN MCCARTHY

this calculus. Second, it is complete, i.e. every valid formula has a proof. Third, although it seems unlikely that the general methods for the first order predicate calculus will be able to produce proofs of significant results in the part of arithmetic axiomatizable in this calculus (or in any other important domain of mathematics), the development of these general methods will provide a measure of what must be left to subjectmatter-dependent heuristics. It should be understood by the reader that the first order predicate calculus is undecidable; hence there is no possibility of a program that will decide whether a formula is valid. All that can be done is to construct programs that will decide some cases and will eventually prove any valid formula but which will run on indefinitely in the case of certain invalid formulas. Proof-checking by computer may be as important as proof generation. It is part of the definition of formal system that proofs be machine checkable. In my forthcoming paper [9], I explore the possibilities and applications of machine checked proofs. Because a machine can be asked to do much more work in checking a proof than can a human, proofs can be made much easier to write in such systems. In particular, proofs can contain a request for the machine to explore a tree of possibilities for a conventional proof. The potential applications for computerchecked proofs are very large. For example, instead of trying out computer programs on test cases until they are debugged, one should prove that they have the desired properties. Incidentally, it is desirable in this work to use a mildly more general concept of formal system. Namely, a formal system consists of a computable predicate check [statement;proofl of the symbolic expressions statement and proof. We say that proof is a proof of statement provided

check [statement;prooA has the value T. The usefulness of computer checked proofs depends both on the development of types of formal systems in which proofs are easy to write and on the formalization of interesting subject domains. It should be remembered that the formal systems so far developed by logicians have heretofore quite properly had as their objective that it should be convenient to prove metatheorems about the systems rather than that it be convenient to prove theorems in the systems.

A BASIS FOR A MATHEMATICAL THEORY OF COMPUTATION

69

4. Use of formal systems by computer programs

When one instructs a computer to perform a task one uses a sequence of imperative sentences. On the other hand, when one instructs a human being to perform a task one uses mainly declarative sentences describing the situation in which he is to act. A single imperative sentence is then frequently sufficient. The ability to instruct a person in this way depends on his possession of common-sense which we shall define as the fact that we can count on his having available any sufficiently immediate consequence of what we tell him and what we can presume he already knows. In my paper [lo] I proposed a computer program called the Advice Taker that would have these capabilities and discussed its advantages. The main problem in realizing the Advice Taker has been devising suitable formal languages covering the subject matter about which we want the program to think. This experience and others has led me to the conclusion that mathematical linguists are making a serious mistake in their almost exclusive concentration on the syntax and, even more specially, the grammar of natural languages. It is even more important to develop a mathematical understanding and a formalization of the kinds of information conveyed in natural language. 5. Mathematical theory of computation In the earlier sections of this paper I have tried to lay a basis for a theory of how computations are built up from elementary operations and also of how data spaces are built up. The formalism differs from those heretofore used in the theory of computability in its emphasis on cases of proving statements within the system rather than metatheorems about it. This seems to be a very fruitful field for further work by logicians. It is reasonable to hope that the relationship between computation and mathematical logic will be as fruitful in the next century as that between analysis and physics in the last. The development of this relationship demands a concern for both applications and for mathematical elegance.

70

JOHN MCCARTHY

REFERENCES [l] CHURCH, A., The Calculi of Lambda-Conversion, A n n a l s of Mathematics Studies, no. 6, Princeton, 1941, Princeton University Press. [2] -, Introduction to Mathematical Logic, Princeton, 1952, Princeton University Press. [3] DAVIS, M., computability and Unsolvability, New York, 1958, McGraw-Hill. [4] ERSHOV,A. P., On Operator Algorithms (Russian), Doklady Akademii Nauk, vol. 122, no. 6, pp. 967-970. [5] KLEENE, S. C., Recursive Predicates and Quantifiers, Transactions of the American Mathematical Society, vol. 53, 1953, p. 41. [6] MCCARTHY,J., letter to the editor, Communications of the Association for Computing Machinery, vol. 2, August, 1959, p. 2. [7] -, Recursive Functions of Symbolic Expressionsand Their Computation by Machine, Part I, Communications of the ACM, vol. 3, April, 1960, pp. 184-195. [8] -, The LISP Programmer’s Manual, M.I.T. Computation Center, 1960. [9] -, Computer Programs for Checking Mathematical Proofs, to be published in the Proceedings of the American Mathematical Society’s Symposium on Recursive Function Theory, held in New York, April, 1961. [lo] -, Programs wirh Common Sense, Proceedings of the TeddingtonConference on the Mechanization of Thought Processes, H. M. Stationery Office, 1960. [l 11 MARKOV, A. A., Theory of Algorithms (Russian), Moscow, 1954, USSR Academy of Sciences, Steklov Mathematical Institute. [12] NAUR,P., et al., Report on the Algorithmichnguage ALGOL 60, Communications of the ACM, vol. 3, May 1960. [13] TURINO,A. M., On Computable Numbers with an Application to the Entscheidungs Problem, Proceedings of the London Mathematical Society, ser. 2, vol. 43,1937, p. 230; correction, ibid, vol. 43, 1937, p. 544. [14] YANOV,Y. I., The Logical Schemes of Algorithms, from Problems of Cybernetics I, translated from the Russian by Nadler, Griffiths, Kiss, and Mu&, New York, 1960, Pergamon PresslLtd.. pp. 82-140.

AN ABSTRACT COMPUTER WITH A LISP-LIKE MACHINE LANGUAGE WITHOUT A LABEL OPERATOR P. C. GILMORE Yorktown Heights, N . Y.

1. INTRODUCTION

Given a universe U, the class of all functions with arguments and values from U is of no interest in a theory of computable functions. But if a finite class F of functions are taken to be primitive, then the class of all functions %(F) that are computable, assuming that the primitive functions are computable, is of interest. The problem of defining the class %‘(F), for any finite class F, has been solved in several ways. One of the more recent ways, and certainly the way most interesting to programmers, is by means of the language of Lisp, which can be regarded not only as a language suitable for defining any member of %(F) for certain classes F, but also as a programming language.1) In our estimation, however, Lisp suffers from two defects, one minor and one major, although these defects are more important when Lisp is regarded as a language for computable functions, than when it is regarded as a programming language. One of the operators that must be assumed to be primitive in Lisp, independently of any class F, is a conditional operator. The first defect, the minor one, from which Lisp suffers is in the conditional operator that has been chosen for it. For unfortunately the conditional operator of Lisp requires that there occur in U elements representing the truth values, and that the class F include functions (predicates) which take these truth values as their values. Therefore strictly speaking the conditional operator of Lisp limits the classes F, for which the language is capable of defining W(F), to those classes F including certain predicates. Fortunately this defect in the conditional operator is easily avoided by defining it in such a way that a conditional expression, that is a formula representing the l) A description of the Lisp language can be found in the article “Recursive functions of symbolic expressions and their computation by machine, part Ic, by John McCarthy, appearing in the Communications of the Association for Computing Machinery”, vol. 3 (1960), no. 4, pp. 184-195. Familiarity with the first three sections of this paper will be presumed.

72

P. C. GILMORE

application of arguments to the conditional operator, becomes a commonly used method of defining functions. In section 3 the new conditional operator is defined and it is shown there how the Lisp conditional operator can be defined in terms of it whenever there is an element representing the truth value truth in the universe U. In addition to the conditional operator, the quote and label operators must be assumed to be primitive in Lisp. Neither of these operators belong naturally to a theory of computable functions, and it is the fact that they are required in Lisp that is the second and major defect of that language.1) The need for the quote and label operators arises in Lisp, we believe, from a failure to recognize early enough in the design of Lisp the necessity to distinguish three different uses of symbols. These uses are: (1) as an object in U which may be an argument for a function or operator ; (2) as a name for an object in U ; (3) as a name for a function or operator, either primitive or defined. The quote operator of Lisp is used to create names for members of U that are symbols, in particular for the symbol T used to represent the truth value truth; the value of the quote operator for a symbol is a name for the symbol. Its main use occurs in Lisp when names of defined functions are created. The label operator is used to create names for functions defined recursively. To avoid the use of the quote and label operators it is not necessary to alter drastically the very simple syntax of Lisp, but it is necessary to redefine the semantics. The formal semantics of Lisp has been defined by defining a function evaluate in the language of Lisp itself. That is, in order to know how to determine the value of any function defined in Lisp for any of its arguments, it is only necessary to understand from the definition of the evaluate function how its values are determined for its arguments. For the language we will describe in this paper we could also define an evaluate function in the language and thus describe precisely the semantics of the language. However, this paper has another purpose secondary to a presenThe label operator can actually be dispensed with in the Lisp language as is shown in the paper A basis for a mathematical theory of computation, preliminary report by John McCarthy, “Proceedings of the Western Joint Computer Conference”, Los Angeles, California, May 9-1 1, 1961. It is necessary to assume that functions can take themselves as arguments, and one must accept very complicated definitions :for even simple recursive functions. 1)

ABSTRACT COMPUTER WITH A LISP-LIKE MACHINE LANGUAGE

73

tation of a Lisp-like language without a quote or label operator and with a modified conditional operator. It is our belief that important purposes can be served by defining the semantics of a programming language by defining an abstract computer for which the programming language is the machine language. These purposes include both a better understanding of the language, as well as of the difficulties that are likely to be faced in implementing it. Further, the definition of an abstract computer permits an experimentation with novel concepts of machine organization which may prove useful in machine design and use. In this short paper we do not intend to define what we mean by an abstract computer in general or by its machine language, but will rather define a particular abstract computer and the machine language for it. The machine language will be a language similar to Lisp, only it will have a modified conditional operator and no quote or label operators. It will also be a universal language in the sense that for any finite class F, any function ing(F) can be defined in the language, and therefore the abstract computer will be a universal machine. More precisely, if it is assumed that the primitive operations of the computer include ones corresponding to the primitive functions in F, then any function in %(F) can be defined in the machine language of the computer and the definition is a “program” for the computer which will cause it to compute the value of the function for any specified arguments. Providing a combined definition of an abstract computer and its machine language is easier in many respects than defining a language and an evaluate function for the language within the language itself. For some of the distinctions that must be drawn between uses of symbols in the language can be made by the “hardware” of the computer rather than by operators or other overt devices of the language. The result is that the language remains closer to the kinds of languages employed by humans in real situations. Thus, for example, label and quote can be dispensed with in the language described in this paper because the distinctions between usages of symbols which forced their introduction into Lisp are being made in the “hardware” and through the organization of the abstract computer for which the language is the machine language. Some of the features of the abstract computer described in this paper may, therefore, be of interest to designers and users of computers.

74

P. C. GILMORE

2.

PRELIMINARY

DECISIONS

The abstract computer must be capable of distinguishing between a finite number of letters of an alphabet A. It is necessary to have at least two letters in A, although it is not necessary to have more than two. Since, however, statements in the machine language of the computer must be expressed in letters from the alphabet A, as large an alphabet as possible is desirable. By a string is meant any finite sequence of letters from the alphabet. Some representation of members of U as strings must be decided upon. Once this representation has been chosen the functions in F can be regarded as functions whose arguments and values are the strings that represent the members of U. We will, therefore, regard U as a set of strings in the alphabet A. Some representation as strings must also be chosen for the functions in F, for the conditional operator, and for the lambda abstraction operator. We will assume for the sake of convenience that the strings “cond” and “lambda” are strings in the alphabet A and that they represent respectively the conditional and lambda operators. A class of strings must be chosen as the class of constants for the language. The constants are used as names for strings in U, as names for strings representing functions in F, and one is used as a name for the string “cond”. The computer is assumed capable of determining the string which a constant denotes. If D is a function that is undefined for an argument that is not a constant and takes as its value for an argument that is a constant the string that the constant denotes, then the computer is assumed capable of determining, for any string in the alphabet A, whether or not D is defined for that string, and if so, of determining the value of D for that string. Since there is no reason for not permitting a string to denote itself, the string “cond” will be assumed to be the constant denoting itself. A class of strings must also be chosen as the class of variables for the language. The variables are used as unspecified, but specifiable, names for the strings that the constants denote, as well as names for representations of defined functions in the class %?(F). Note that the variables are not grouped into two classes, one for members of U and the other for strings representing functions and the conditional operator. What is to be understood by a sequence in the remainder of the paper is defined inductively: (1) any finite sequence of constants, variables, or

ABSTRACT COMPUTER WITH A LISP-LIKE MACHINE LANGUAGE

75

instances of the string “lambda”, is a sequence; and (2) any finite sequence of constants, variables, instances of the string “lambda”, or sequences, is a sequence. Some means must be found for representing any sequence as a string. For example, if “x”, “I”, “(“, “)”, and “,” are the only letters of an alphabet, and “x” either alone or followed by any number of instances of “1” is a variable, and there are no constants, then parenthesis and commas can be used in a frequently used manner to permit a representation of any sequence. Thus “(((xl ,x),x11,(x,x)),xl)” is a string representing the sequence of strings “((xl,x),xl l,(x,x))”, and “xl”, the first string of which represents the sequence of strings “(xl,~)”,“xll”, and “(x,~)”, each one of which represents a sequence of variables. If S is a function that is defined for any string which represents a sequence and has as its value the finite sequence of strings representing the immediate components of the sequence, then the computer is assumed capable of determining, for any string in the alphabet A, whether or not S is defined for that string, and if so, of determining the value of S for that string. Thus, for the simple example given above, a computer would be capable of recognizing that the value of S for the string “((xl,x),xl, (xl1,x))” is the sequence of strings “(xl,x)”,“xl”, and “ ( ~ 1 1 , ~ ) ” . In general the string representing a sequence OL,PI,. ..,Pn is used to express in the machine language the application of arguments denoted by PI,. ..,Pn to an operator or function denoted by m. Thus again for the simple example given above, “(x,xl, xl 1, xl 1 1, xl 1 1 1)” represents the application of arguments denoted by the variables “xl”, “x1l”, “x1 1 l”, and “ x l l l l ” to a function or operator denoted by “x”. Such a string generally results therefore in the computer evaluating OL for the arguments PI,. ..,pa. The one exception is when the string is a lambda expression; that is, when OL actually is the string “lambda”. The computer is assumed capable of executing what is called the initial load operation. This operation will be described precisely later but it can be understood to be an operation that places a specified string into a specified memory location. It is mentioned here only because the strings used to represent the application of arguments to the initial load operator are not strings representing a sequence of three strings. Rather a special letter, which we will take to be “:”, is assumed to be in A. Then any string of letters of A containing but one instance of “:”, this not being the first or last letter in the string, is regarded as an instruction to the computer to place the string occurring after the instance of “:” into the

76

P. C. GILMORE

memory location corresponding to the string occurring before the instance of “:”. Such a string is called an initial load statement. Assuming that all the above decisions have been made, the class of statements for the machine language can finally be defined: any string representing a sequence (perhaps of one component) is a statement, as well as any string which is an initial load statement. The syntax of the language could be further elaborated so that knowledge of the syntax will assist one in recognizing those statements of the language which will lead to useful computations of the computer. For example, one might not recognize as “meaningful” those statements representing sequences the first member of which does not denote an operator or function. But assistance in recognizing meaningful statements can also be got from a knowledge of the semantics of the language, or in this case of the abstract computer.

3. THEOPERATIONS OF THE ABSTRACT COMPUTER The conditional operator used here differs from the conditional operator of Lisp in having exactly four arguments, rather than any even number of arguments, and in making a comparison between two computed strings, rather than between a fixed string “T” and a computed string. The operator would be conventionally defined : cond(x,y,z, w) =

z, if x = y

w,otherwise.

The conditional operator of Lisp can therefore be obtained by “nesting” this conditional operator if a string “T” is assumed to be in U and represents the truth value truth. For example, the conditional expression (PI + el,pz + ez) of Lisp can be defined to be cond(pl,T,el,cond (pz,T, ez,e)), where e is any undefined expression. A statement of the machine language expressing the application of arguments to the conditional operator is a control statement of the abstract computer that does not result in computations in the same sense that other statements of the machine language do. However, the “evaluation” of such a control statement is carried out in much the same way that the evaluation of any other statement is carried out. There is no operation of the computer corresponding to the lambda operator, although an operation of lambda removal is needed by the computer. However, the operation of lambda removal, unlike the other

ABSTRACT COMPUTER WITH A LISP-LIKE MACHINE LANGUAGE

77

operations of the komputer, has no string representing it. Rather any statement representing a sequence consisting of a lambda expression followed by other statements is interpreted as a call for the operation of lambda removal from the lambda expression, using as arguments the statements following the lambda expression in the sequence. Thus, for example, the string “( (lambda,x,x I), 2)”, when the lambda removal operation has been performed, results in the string “2 1”. One complication of the operation of lambda removal we choose to ignore in the remainder of the paper. The complication is the possible clash of bound variables. There are simple ways of avoiding the complication. One can, for example, assume that an initial part of the operation of lambda removal is an operation of renaming all bound variables so that each bound variable is bound by a single lambda, and is different from any variable occurring in the arguments. Or one can assume that it is the responsibility of the user of the language to avoid clashes. In any case, we will assume that this complication has been taken care of in some manner so that we need no longer be concerned about the clash of bound variables in the operation of lambda removal. The first version of the abstract computer discussed below has no other “universal” primitive operations than the conditional, lambda removal, and the initial load operation. The other primitive operations of the computer correspond to functions in the class F under consideration. For each of these functions, it is assumed that the computer is capable of evaluating the function for any of its arguments. A second enlarged version of the abstract computer that is discussed briefly in $5 has two additional “universal” primitive operations, one called the sequence operation and the other called the dynamic load operation. These operations can be best understood after the first version of the computer has been described. In the flowchart given in the next section and describing the operation of the computer, those parts that are relevant only to the additional universal operations have been outlined in dotted lines.

+

+

4. THEABSTRACT COMPUTER

In this section sufficient details of the computer will be given to permit an understanding of the flowchart given below, and the operation of the computer will be illustrated with two examples of computations. In the alphabet A there is assumed to be one more special letter in addition to the special letter “:”. This new special letter must be such

CHECK

PART.

FOR

I

INTO d

PUT VALUE

I

W

CHECK

ABSTRACT COMPUTER WITH A LISP-LIKE MACHINE LANGUAGE

81

that it does not occur in any statement of the language other than possibly an initial load statement. We will take this special letter to be "=". There are denumerably many storage locations in the computer each location being capable of storing a string of any length. Strings of nonspecial letters, or such strings to which "=" has been attached at the end, are used to name the locations. Thus if u is a string of non-special letters, there is a memory location corresponding to u itself, and one to u =. In the discussion below the assertion that a string /? occurs in the memory location corresponding to the string a will be abbreviated to a :p, since the effect of this initial load statement is to put the string /? into the memory location corresponding to the string a. Similarly, a : abbreviates the assertion that the memory location corresponding to the string a is empty. The purpose of having a memory location corresponding to u = as well as u for a string u is to permit the computer to record a string which denotes the same string as u denotes, should the string denoted by u not yet be recorded in the memory location corresponding to u. Thus, should the state of the memory be described by u : and u = : y , the computer will go to the location corresponding to y in order to fill the location corresponding to u. Should u : and u= : be the case, then the computer must analyse the string u itself in order to determine what the contents of u should be. There is the possibility that D is defined for u, that is that u is a name for another string, so that D(u) can be put into the location corresponding to u. There is also the possibility that u is a lambda expression, in which case the computer will put u into the location corresponding to u since it regards a lambda expression as denoting itself. Finally there is the possibility that S is defined for u and that S(u) is a sequence expressing the application of arguments to an operation. Then the value of the operation for the arguments can be put into the location corresponding to u. Clearly some care has to be exercised in evaluating an operation for certain arguments in order to be certain that the arguments are available in their intended form. Most simply in the case when the arguments must be calculated one must be assured that the arguments have actually been calculated before the operation is applied to them. Or in the case of the conditional operation the first two arguments must be evaluated before the conditional operation can be applied. For this reason the computer has a push-down list on which may be stored any number of strings in

82

P. C . GILMORE

either a checked or unchecked state. By a push-down list is meant a list to which new members are added to the top and members are taken off from the top, so that the last string put onto such a list is the first string to be taken off. Roughly speaking, an unchecked string from the push-down list is never evaluated; only when one is assured that the next time a statement is taken from the push-down list to be evaluated, it will be ready for evaluation, is the statement put onto the push-down list in the checked state. It is important to notice that the generality of the machine language of the computer is such as to permit defined functions to have functions as arguments or as values or both. Extra complication in the computer is necessary to handle such defined functions. Initially all memory locations of the computer are empty. One “programs” the computer by putting statements onto the push-down list in the unchecked state. Included in the statements will be initial load statements which will result in some of the memory locations being filled before the main part of the computation begins. The computer will only stop when the push-down list is empty. The first example illustrates how a recursive definition of a function can be given to the computer in its machine language and how the computer evaluates such a function for a given argument. For this example we will assume that U includes the integers, and that F includes multiplication and subtraction, which have been represented in the machine language by “ x ” and “-” respectively. We also assume that “!” and “x” are variables of the language, and that “ x ” and “-” are constants denoting themselves. “!” has been assumed to be a variable of the language because it will be used to denote a statement representing the factorial function; as mentioned earlier, the variables of the language do not fall into two classes, one for members of U and one for functions. Finally we assume that the numerals are constants denoting themselves. To abbreviate the description of the computation slightly we will say that one string is in a second string, or that a second string is empty, when we mean to say that one string is in the memory location corresponding to a second location, or that the memory location is empty. The computer is “programmed” by putting the following two statements onto its push-down list in the order listed: !: (lambda,x,cond(x,1,1,xx (x-l)!)) 2! A combination of the brackets and comma method and standard

ABSTRACT COMPUTER WITH A LISP-LIKE MACHINE LANGUAGE

83

mathematical notation for representing sequences has been used. Thus S((lambda,x,cond(x, l , l , x x (x-1) !))) is the sequence of strings “lambda”, “x”, and “cond(x, l , l , x x (x-l)!)”; S(cond(x, l , l , x x (x-l)!)) is the sequence of strings “cond”, “x”, “l”, “l”, and ‘‘x x (x-l)!”; S(x x (x-I)!) is the sequence of strings “x”, “x”, and “(x- l)!”; S( (x- l)!) is the sequence of strings “!”, and “(x-1)”; and S((x-1)) is the sequence of strings “-”, “x” and “1”. The effect of the first statement on the list is to cause the computer to put the string (lambda,x,cond(x, l,l,x x (x-l)!)) into !. The statement is then dropped from the push-down list and only 2! remains. When 2! is read from the list, 2! : and 2!= : is the case so that the computer eventually reaches the point B in the flowchart. S is defined for 2! and has the sequence of strings “!” and “2” as its value. Since ! contains a lambda expression, the lambda is removed from the expression using the argument 2 and the resulting string “(cond(2,1,1,2 x (2- l)!)” is put into 2!= and onto the top of the list, and the statement “2!” is checked on the list. When (cond(2,1,1,2 x (2- l)!) is read from the list neither the location corresponding to it nor the location corresponding to it with a “=” attached has a string in it, so that the point B in the flowchart is reached again. S is defined for this string and has the sequence “cond”, “2”, “l”, “l”, and “2 x (2- l)!” as its value. The location corresponding to cond is at present empty so that cond is put onto the list. But D is defined for cond and has cond as its value so that cond is put into cond and is struck from the push-down list.1) When (cond(2,1,1,2 x (2-1)!) is read again from the list the operation has been determined so that the statements “2” and “1” are put onto the list, and it is checked. Again D is defined for 2 and 1 and has these same numerals as its values so that 2 is put into 2 and 1 is put into 1 and 2 and 1 are removed from the list. Now when (cond(2,1,1,2 x (2-l)!) is read again from the list it has been checked and is ready for evaluation. Since the contents of 2 is different from the contents of 1, 2 x (2-l)! is put into (cond(2,1,1,2 x (2- l)!) = and onto the push-down list. Skipping now some of the details of the operation of the computer, it 1) The operation of the computer would be simplified here and later with the evaluation of the numerals if the location correspondingto cond, 1 and 2 were initially loaded with the strings cond, 1 and 2 respectively.

84

P. C. GILMORE

can be seen that the effect of reading 2 x (2- l)! from the list will eventually be to have it checked and have (2-l)! put above it on the list. The effect of reading (2- l)! from the list will be to put the string (cond((2- l), 1,1,(2-1) x ((2-1)-l)! into (2-l)!= and onto the list, and to have (2- l)! checked on the list. x ((2-1)-l)!) The reading of the statement cond((2-1),1,1,(2-1) from the list eventually results in 1 being put into (2-1) and into cond((2-1),1,1,(2-1) x ((2-1)-l!)= and onto the list, and result s i n the statement being checked on the list. 1 is quickly removed from the list. When (cond((2-1),1,1,(2-1) x ((2-1)-l)!) isreadagainfromthe list, 1 is put into its location and it is struck from the list. The statement (2-l)! appears next on the list in a checked state, and hence, since(cond((2-1),1,1,(2-1) x ((2-l)-l)!)appearsin(2-1)! = 1 is put into (2- l)! and the statement is struck from the list. The statement 2 x (2-l)! is now on the top of the list and is in the checked state so that it can be evaluated and the result 2 put into its location, and the statement can be struck from the list. The statement (cond(2,1,1,2 x (2-l)!) is now on the top of the list in the checked state, and hence, since 2 x (2- l)! appears in (cond(2,1,1, 2 x (2- l)!) =, 2 is put into the location corresponding to the statement and the statement is struck from the list. Lastly the statement 2! is read from the list in the checked state and since (cond(2,1,1,2 x (2- l)!) appears in 2! =, 2 is put into 2! and the statement is struck from the list. Since the list is now empty, the computer stops. The last example we will follow in considerably less detail than the previous since the example is only given to illustrate how the computer handles defined functions that have functions as their arguments and values. For this example we presume that the list of variables has been extended to include , y , f ,and “g”. The constant string denotes itself and represents addition. The push-down list initially contains the following statements in the listed order: f : (lambda,u,u + 3) g : (lambda,y, (lambda,x,yCv(x)))) scf)(4> In this example standard mathematical notation is being used again so that S(u+ 3) is the sequence of strings “u”, and “3”; S(y(y(x))) C6u11

66

1,

46

“+”

>9

“+”,

ABSTRACT COMPUTER WITH A LISP-LIKE MACHINE LANGUAGE

85

is the sequence “y” and “y(x)”; S ( y ( x ) ) is the sequence “y” and “2’; S(g(A(4)) is the sequence “g(f)” and “4”; and S(g(f)) is the sequence and 66g7,

6bf99.

The first two initial load statements result in a loading of the computer memory locations corresponding to f and g. The third statement eventually results in g(f) being put above it on the list. Thestatement g(f> on the list results in alambda removal operation since g contains a lambda expression. The result is that g(f) is checked and (lambda,x,f(f(x))) is put into g ( n = and onto the list. But when (lambda,x,f(f(x))) is read from the list it is recognized as a lambda expression, is put into its own location and is struck from the list; eventually therefore it is put into g ( n and g(f) is struck from the list. The statement g(n(4) is again read from the list and results in a lambda removal operation so that it is checked andf(f(4)) is put into gcf)(4)= and onto the list. The statement f(f(4))results in a lambda removal operation so that the statement is checked and f(4) 3 is put into f(f(4)) = and onto the list. The statementf(4) 3 is eventually checked afterf(4) and 3 have been put onto the list. The statement 3 is quickly removed because D is defined for it. The statement f(4) results in a lambda removal operation so that it is checked and 4 3 is put into f(4) = and onto the list. Eventually therefore 7 is put intof(4), 10 is put intof(4) 3 and then into fcf(4)) and g(f)(4). The computation then ends.

+

+

+

+

The Extended Abstract Computer. If the operations of sequence and dynamic load are assumed available for the abstract computer, definitions can be given in its machine language of operators which correspond to ones used at present in the programming of actual computers. In this section we will illustrate this point with two examples and leave any further discussion for another time. The sequence operator, like the initial load operator, does not require that its arguments be first evaluated since the statement expressing the application of arguments to the sequence operator must state what the arguments are. Thus if (T is recognized as expressing the application of are put onto arguments PI,...,Pn to the sequence operator, then PI,. ..,/In the push-down list in that order and (T is dropped from the list. The dynamic load operator, unlike the initial load operator, does

86

P. C . GILMORE

require that both of its arguments be evaluated. The effect of the dynamic load operator is to place the value of its second argument into the location corresponding to the value of its first argument. Flowcharts are a commonly used device in programming and can be regarded as operators accepting as arguments the operations expressed in the boxes occurring within the flowchart. Thus, for example, the simple loop:

Fig. 4

can be regarded as an operator with four arguments, w, x , y, and z, where the decision box means that the upper line is followed if x is 1 and the lower line otherwise. Denoting this operator by “L”, it can be recursively defined as follows. L : (lambda, w, x,y, z, seq(w,cond ( x , 1,seq(y,L(w, x,y, z)), z))). Two nested loops are expressed by nesting the operator L as for example in the following chart

j=i+ I

5

rig. 3

which is expressed by seq(s = 0, i = 1, j

=

I,L(L(s = s + a i j , j < m yj = j + 1, j = l), i rank(d) in P.4. Thus, in a backwards application of P.3 or P.4 we never replace a term c with a term d of higher rank. We say that a sequent r 2 is directly demonstrable if it is demonstrable by means of the restricted postulates P. 1 -P.4 only. We note that we can always decide if a given sequent is directly demonstrable or not.

+

+

+

A SIMPLIFIED PROOF METHOD FOR ELEMENTARY LOGIC

91

We also restrict the rules P.16 and P.17 by the requirement that the term c shall occur in the conclusion below the line or - if there are no terms in the conclusion - c shall be the alphabetically first individual parameter. We shall also change the formulation of the rule. When we apply the rule we shall not have to choose the term c immediately. Instead we replace x by a dummy oc and make a note in the margin that oc stands for one of the terms in the conclusion. Thus the rules are given the following form:

r,F,, uXFX, A =$ z r,uXFX, A + z 1:

P.16

aIc1,.

-.,C n

where CI, ...,C n are the terms occurring in the conclusion; if there are no such terms, oc is the first individual parameter.

P.17 where a,...,C n are the terms occurring in the conclusion ; if there are no such terms, a is the first individual parameter. The note aIc1,. ..,C n we may call a substitution list for the dummy a and the terms c1,. ..,c n we call the values of oc. 1) I shall now describe the routine of the proof method. Suppose we wish to demonstrate that r =$ Z. We start from below with Z and going upwards we construct a tree of sequents by applications of the rules. We divide the construction of the tree into stages. Within each stage we apply only the rules for truth-functions and quantification, i.e. rules P5-Pl8. At the end of each stage we test the top sequents of each branch of the stage. If the test gives a positive result we stop the construction of the tree, otherwise we construct a new stage. Within each stage we prefer applications of rule P.15 and P.18 to applications of the truth-functional rules, i.e. rules P.5-P.14, and we prefer applications of truth-functional rules to applications of P.16 or P.17.When we apply P.16 and P.17 and when more than one term occurs in the conclusion, we shall always introduce a new dummy which is not yet introduced in the tree and we shall always give the substitution list for the dummy. Moreover, when we apply P.16 and P.17 we shall prefer to split a formula G rather than a formula which has been split more times than G in previous application of P.16 or P.17 in the branch

r+

1)

Note that dummies (other than a) may occur in the list

el,

. . ., c,,.

92

STIG KANGER

in question. When we apply P. 15 or P. 18, the parameter introduced shall be new and of course different from the values of the dummies in the conclusion. At the end of each stage we stop for a moment and check whether we can choose values for the dummies from their substitution lists in such a way that all top sequents will be directly demonstrable when we replace the dummies by their values. If there is such a choice, our demonstration has succeeded. If there is no such choice, we keep the dummies and construct a new stage of the tree by continuing the construction of each branch which is not completed in the sense that its top sequent is directly demonstrable for every value of the dummies. It remains to fix the extension of the stages. We let a stage be completed when each branch of the stage has a top sequent d A such that (1) every formula in d is either an atomic formula or a formula beginning with a universal quantifier, and every formula i n n is either an atomic formula or a formula beginning with an existential quantifier, and (2) every nonatomic formula in d and A has been split equally many times by previous applications of P.16 or P.17 in the branch. This concludes the description of the proof method. To illustrate the ExUy((x = method, I shall give a demonstration of the sequent f ( x ) ) = cfcf(y)) = y)). The formula in this sequent we may abbreviate as ExF.The demonstration shall be read from below.

+

+

choice of values: alj

A SIMPLIFIED PROOF METHOD FOR ELEMENTARY LOGIC

93

We shall compare this demonstration with the demonstrations of the sequents U y E x ( ( x = f ( x ) ) 1 ( f ( f ( y ) )= y ) ) and (Ux(x = f ( x ) ) 1 U y ( f ( f ( y ) )= y ) ) . The formulas here are logically equivalent with the formula in the example, but the demonstrations will be simpler. We need only two stages, since th econstant j which we substitute for y will now be available as a dummy value already at the first application of P.17 or P.16. The simplest demonstration is that of the last sequent:

+

+

f(i)),Ux(x = f(x)) + (i=i) (i= f ( i ) ) . W x = f(x)) + (f(i)= i) stration ( j = f ( j ) ) , W x = f ( 4 )+ ( f ( f ( i = ) )j )

(i=

PA P.4

choice of values :

Thus, to obtain simple demonstrations it usually pays not to extend the scopes of the quantifiers more than necessary.

94

STIG KANGER

REFERENCES [l] BETH,E. W., Formalmethods. Dordrecht 1962. H. and ROCHESTER, N., Intelligent behavior in problem-solving [2] GELERNTER, machines. ZBM Journal of Research and Development, vol. 2 (1958), pp. 336-345. P. C., A proof method for quantification theory: its justification and [3] GILMORE, realization. ZBM Journal of Research and Development, vol. 4 (1960), pp. 28-35. S., Provability in logic. Stockholm 1957. [4] KANGER, [5] -, Handbok i logik. Stockholm 1959 (Mimeographed). [6] NEWELL, A. and SIMON,H., The logic theory machine. IRE Transactions on Information Theory, vol. IT-2, no. 3 (1956), pp. 61-79. D., PRAWITZ, H. and VOGERA, N., A mechanical proof procedure and [7] PRAWITZ, its realization in an electronic computer. The Journal of the Association for ComputingMuchinery, vol. 7 (1960), pp. 102-128. D., An improved proof procedure. Theoria,vol. 26(1960), pp. 102-139. [8] PRAWITZ, [9] WANG, H., Toward mechanical mathematics. ZBM Journal of Research and Development, vol. 4 (1960), pp. 2-22. [lo] -, Proving theorems by pattern recognition. Part I, Communications of the Association for Computing Machinery, vol. 3 (1960), pp. 220-234; Part I1 Bell System Technical Journal, vol. 40 (1961) pp. 1 4 1 .

A BASIS FOR THE MECHANIZATION OF THE THEORY OF EQUATIONS A. ROBINSON Hebrew University of Jerusalem and Princeton University

1. Introduction. The work which will be described here represents an attempt to provide a foundation for a limited but definite advance in the realm of the mechanization of mathematical theories. Further details, including proofs of some of the statements made below, will be found in ref. 2. We shall fix our attention on the elementary theory of equations and on certain generalizations of that theory. It is one of our basic ideas that a preliminary step in the development of a theory, as it takes place in the mind of a scientist, is a deliberate limitation of his conceptual language to an extent which is just adequate for the purpose in hand. In other words, while the language should be sufficient for the analysis and solution of the problem under consideration, it should not be too rich, i.e. it should not include too many features which are redundant relative to the treatment of that problem. In the case of the elementary theory of equations it appears that a suitable framework is provided by a free variable theory in the first order functional calculus. It is the fundamental problem of the elementary theory of equations (which was the theory of equations until the advent of Abel and Galois) to find an expression for the solution of a given equation in terms of the field operations and in terms of the extraction of the nth roots, n = 2,3,. .. .For example, the fact that a well-known formula yields the solution of the general quadratic equation can be expressed as follows, x = (1/2a)(-b

+ Vb2-4ac)

3

a

=0

v

ax2

+ bx + c = 0.

In this formula, 3 and V are the connectives of implication and of disjunction, as usual, and the atomic formulae are x =(l/k)(-b

+ Vb2-4ac)

a=O U X ~ +bx+ c

= 0.

Thus, the atomic formulae are equations tl

= t2

where the terms tl and

96

A. ROBINSON

12 are obtained from the variables x , a, b, c, and from certain constants, in the present case 0, 1, 2, by the repeated use of the functors which are given by the field operations, addition, multiplication, subtraction and division (where 0-1 is defined in an arbitrary but definite way) -and by the operation of square root extraction. The resulting sentence is to be interpreted as if all variables had been subjected to universal quantification. This example will help to explain the reason why we choose our formal language as we do.

2. A Free Variable Calculus of the First Order. In general, we shall deal with a formal language which is based on a pre-assigned set of variables, x , y , ..., of individual constants, e.g. 0, 1, ... and of functors of an arbitrary number of variables. Additional atomic symbols are, the sign of equality, =, the propositional constants T and F, the connectives A , V , 3 (A for conjunction), and the brackets [ and 1. Terms are obtained by applying functors repeatedly to individual constants or variables, atomic formulae are either equations of the form t1 = t2, where tl and t2 are terms, or propositional constants (Tor F). Molecular formulae are obtained by the (repeated) application of the connectives A and V to atomic formulae and, finally, sentences are expressions of the form A 3 B where A and Bare molecular formulae. Sentences can be interpreted in the usual way in a given mathematical structure, where the free variables are regarded as if they were quantified universally. A calculus of deduction is introduced into the language by means of the following definitions. There are three classes of axioms. The first class consists of the propositional axioms, which are all sentences of one of the forms A = A , F 3 A , A = T, A A B = A , A = A A B , A V B = B V A , A A B 3 B A A, A A [B A Cl [ A A B1 A C, A V [B V Cl [ A V Bl V C, A A [B V Cl [ A A Bl V [ A V Cl, [ A A B1 V [ A A Cl A A [ B V C1,A V [ B A Cl 2 [ A V Bl A [ A V Cl, A V [B A Cl. [ A V Bl A [ A V Cl The second class of axioms consists of the axioms of equality, T -

-

t = t, s = t t = S, r $(t,t2,. ..,tn),...,s = t

A s = t 1r = t, s = t $(s,t2 ,...,tn) $(tl,. ..,tn-l,s) = $(tl,. ..,tn-1, t),

=S

where r, s, t, and tl, ...,tn are arbitrary terms and $ is an arbitrary functor.

MECHANIZATION OF THE THEORY OF EQUATIONS

97

The third class of axioms consists of a specified set of sentences, K*. Unlike the previous axioms, K* depends on the mathematical theory under consideration. In the particular cases in which we are interested here, K* will include a set of axioms for the notion of a commutative field. Such a set may be formulated by means of the individual constants 0 and 1 (which will have their usual significance), and of the functors o(x,y), $(x,y), ,u(x), and e(x), which stand for x y , x - y, - x and x-l respectively, with the amendment that e(0) = 0. A sentence is a theorem of the calculus if it is an axiom or if it is deducible from the axioms by the (repeated) use of the following rules of deduction. If A = B and B C are theorems then A =I C is a theorem; if A 2 B and A 1C are theorems then A 1B A C is a theorem; if A C and B 1 C are theorems then A V B = C is a theorem. It is shown in ref. 2 that the calculus under consideration is semantically complete. That is to say if, for given K*, a sentence A B is not a theorem of the calculus, then there exists a structure which satisfies all the axioms as well as T =I A and B = F. On the other hand, it is possible to specify ajinite K* such that the corresponding calculus i s undecidable. We mention in passing that the propositional formulae admitted in our calculus are not sufficient to represent all propositional functions. However, all propositional functions can be represented by conjunctions of the formulae admitted here.

+

3. A Framework for the Elementary Theory of Equations. Let {py(x,yl,..., yj,)} be a set of polynomials with rational (or, which is of equal generality, with integer) coefficients. For every p r let there be given a functor $&I,. ,yj,) and let X , be the sentence

..

T

= PY($ICvl,

*

-

*

,Y$,Yl,

***

dj,) = 0.

Thus, X , states that $r(yl,...,yjJ provides a solution for the polynomial equation p,(x,yl,. ..,yjJ = 0 with x as “unknown”. For the classical theory of equations the polynomials pv will be the polynomials x-yly, v = 2,3, ..., and the functors $&I) will provide solutions of the “pure equations” x - y f = 0, i.e. vth roots. The set K* is defined as a set of axioms for the notion of a field of characteristic 0, together with the set of sentences { Xp}for a given set of polynomials {p,}, as explained. Let p(x,yl,.. ,yj), j 2 0, be a polynomial of x,yl,. . ,yj with rational

.

.

98

A. ROBINSON

coefficients. The fundamental problem of the theory of equations of one unknown consists in finding a term t which is formulated by means of the functors and individual constants of K* and by means of the variables y l , . ..,yj such that the sentence x

=

t

= p(x,y1,...,Yj)

=0

is a theorem. It is well known that no such t need exist. It is also possible that the term t provides a solution only in general (compare the case of the quadratic equation mentioned in the introduction). Thus, to find a solution to p,(x,yl,...,yj) = 0 in general means to find a term t as above and a non-vanishing polynomial q(y1,. .., y j ) with rational coefficients such that x = r = q(y1,. ..,yj) = 0 v p ( x , . ~ l-. , ,yj) = 0 is a theorem. Moreover, even if we are interested primarily in equations with a single unknown we may wish to solve certain systems of equations with several unknowns at some intermediate stage. This problem can be formalized in a similar way. Finally, we may be interested in finding all roots polynomials. In cases where there are no exceptions this amounts to finding terms t l , . ..,tk such that

and

x

= rg

= p ( x , y l , . ..,yj) = 0,

p(x,y1, ...)y j )

=

x

= tl

i = 1, ...,k

v ... v

x = tk

are all theorems of the given calculus.

4. Strategy and Heuristics. The problems detailed in the preceding section require in the first place the discovery of a suitable term (or terms) t which satisfies the given equation. For any polynomial p ( x , y l , ..., y j ) and any term t , the question whether or not p ( t , y l , . ..,yj) = 0 - or, more generally, the question whether or not t1 = tz for given terms tl and tz - is subject to a uniform decision procedure which depends on a standard result (due to D. Hilbert) in the theory of polynomial ideals. However, in practice, the discovery of a suitable term t and the proof that it satisfies the given polynomial equation will be carried out by a working mathematician at one and the same time. Imitating this procedure in a formal program, we have suggested a number of specified moves which are designed to “unwrap” the x in the equation p ( x , y , ...,yj) = 0 i.e. to

MECHANIZATION OF THE THEORY OF EQUATIONS

99

transform it step-by-step into an equation x = t, t a term involving y l , ...,yj , which entails p(x,yl, ...,y j ) = 0. However, the question how long to persist in the application of each move constitutes the one element of the program which is by necessity tentative and in which gradual improvements may be suggested by experience. All other so-called heuristic principles apparently cannot escape the fate of being transformed into definite rules or sets of rules as soon as one considers their formalization. We digress for a moment in order to offer a comment on the use of (simulated) diagrams as a heuristic aid in proving theorems of elementary geometry by machine (ref. 1). It is stated in ref. 2 that the usefulness of this idea is associated with the fact that a (classical) geometer is greatly stimulated by diagrams. However, while this statement is empirically correct we may add, on second thoughts, that after all these diagrams are used chiefly in order to examine a general situation by means of special cases. Provided the data of the diagrams are chosen to lie in a convenient range it is provably correct that the diagram will most probably display the same features as the “general case”. No such positive statement can be made about the use of examples in other branches of mathematics such as Number theory. To sum up, we suggest that the elementary theory of equations expressed within the framework of an appropriately designed deductive calculus provides a good subject for mechanization. It is easy to generalize the theory e.g. by the introduction of inequalities. However, while I share the long range optimism of other workers in this field, I believe that unduly ambitious programmes for the near future may delay ultimate success rather than bring it nearer.

REFERENCES [l] GELERNTER H. C., and ROCHESTER, N., Intelligent behavior in problem solving machines, IBM Journal of Research and Development, vol. 2,1958, pp. 336-345. [2] ROBINSON, A., On the mechanization of the theory of equations. Bulletin of the Research Council of Israel, vol. 9F, 1960, pp. 41-10.

PROGRAMMING AND THE THEORY OF AUTOMATA1) ARTHUR W. BURKS University of Michigan

This paper will relate some already established results in the theory of growing automata to programming. We begin by reviewing A. M. Turing’s universal simulation result.2) Two different types of computing units are involved. First, a tape consisting of an indefinite number of squares, each capable of storing a single symbol. The tape is indefinitely expansible in one direction, but it is essential that initially only a finite number of squares are “marked”, i.e., are not blank. The second type of unit is ajinite control automaton with a “tape head”, i.e., a finite automaton which can scan and read a square of the tape, erase that square or write in it, and at the next step scan the square to the left or the right of the square just scanned. A Turing machine consists of a finite control automaton connected to a tape. A finite control automaton and an indefinitely expansible tape are very different, the former being finite and active, the latter being potentially infinite and passive. But both are kinds of automata, constructible out of switch and delay elements and passing deterministically from one discrete state to another. The exact relations between them are of interest to the theory of automata and programming. We will study these relations first from the point of view of Turing machines and later on from the point of view of von Neumann’s self-reproducing automaton. Let us establish the following conventions. The addresses of the squares are 0,1,2,. . .The finite control automaton of a Turing machine will scan initially the square which has zero as its address. There is a fixed alphabet

.

l) This is a revised version of a paper presented at the Seminar on the Relationship Between Non-numerical Programming and the Theory of Formal Systems held at the IBM World Trade European Education Center, Blaricum, Netherlands, October 4-6, 1961. This research was supported in part by the United States Army Signal Corps. I wish to thank James Thatcher for his suggestions. 2, “On Computable Numbers, with an Application to the Entscheidungsproblem”. Proceedings of the London Mathematical Society, Series 2, 42 (1936-37) 230-265. “A correction,” ibid., 43 (1937) 544-546. We will not follow Turing’s formulation exactly.

PROGRAMMING AND THE THEORY OF AUTOMATA

101

for the tape, with characters of three types: a blank (the initial state of almost all squares), non-numerical characters, and numerical characters. The finite amount of information recorded on the tape initially(i.e., before the finite control automaton begins to function) will be expressed by alphabetic characters located in the consecutive even-numbered squares 0,2,4,. ..,2n, where n is a nonnegative integer. This information will be called the program and it will never be erased, i.e., we consider only finite control automata which do not alter any part of the program found initially on the tape. It is easy for the finite control automaton to sense the first blank even-numbered square and thereby sense the end of the program. The computed answer will be expressed in numerical characters printed consecutively in the even-numbered squares following those with the program. The odd-numbered squares will be used for “scratch work”. M*T is the Turing machine composed of the finite control automaton M and the tape T. It should be kept in mind that T is just a tape, not a “tape unit”; the circuits for shifting and altering Tare in M . It is clear that the answer computed by a Turing machine is a digital sequence, and that, under the fiction that there is a radix point (e.g., a binary point) to the left, the sequence represents a real number between zero and one. The case where the Turing machine produces no answer (i.e., a null sequence) may be accommodated by speaking of the “null number”. Let q (M*T) be the number computed by M*T. Now let a be the class of numbers computed by all Turing machines, i.e., = {x~(~M)(~T)~(M = *x}. T) We may think of a as being generated by q(M*T) as the variable “M” varies over all finite control automata and the variable “T” varies over all tapes. But it is not necessary to vary both M and T a t the same time to obtain a ; LY may be generated either by varying M while keeping T fixed or by varying T for a suitable fixed M . The first of these results is easily established. Let A be the tape which is initially all blank and let @ be the class of numbers computed by Turing machines with blank tapes :

@

=

{xI(3M)q(M*A) = x}.

We show that a = @. Consider a particular number q(M*T). By definition the program on T is finite; let T‘ be the finite segment of Tcontaining this program and let T“ be the balance of T. Now a finite piece of tape with a

102

ARTHUR W.BURKS

program written on it is itself a finite automaton or is very nearly so. At any rate we can easily design a finite control automaton M’ which incorporates both M and T‘ and which uses T’las its tape. Then q (M’*T”) = q ( M * T ) , and since T“ is totally blank, q(M’*T”) = q(M’*A). Hence a = b. Moreover, since a finite control automaton can sense the end of the program on a tape, it can also erase that program. It follows that for each tape T, a may be generated by varying M over the class of finite control automata, i.e., LY = { ~ 1 ( 3 M ) q(M*T) = x } . It is obvious that M and T do not play dual roles in the generation of a. For example, if MOis a finite automaton which does nothing to the tape, q (Mo*T) is the “null number” for every T. This lack of duality is just what one would expect, for though a tape is an automaton it is a passive one. Turing showed, however, that there is a “universal” finite control automaton Mu which is sufficiently powerful to generate a as T is varied over all tapes. Let y = {XI (3 T ) q(Mu*T) = x } . Turing’s result is, then, a = y. He showed this by defining (hypothetically constructing) Mu so that it has this property: For each finite automaton M there is a tape 9 ( M ) such that

q ( M u * - q M ) ) = q(M*A). The way Mu works is of great interest since it involves the important notion of simulation. Each finite control automaton M has a finite number of states. Its passage from one of these states to the next is determined by the tape symbol scanned; this behavior may be expressed by a finite state table. Each state of the finite control automaton produces a certain effect on the tape; this information likewise can be expressed in a finite table. These two tables constitute the program on 9 ( M ) . Consider now the behavior of M*A. At each moment of time M is in one of its states, and the tape A has a finite sequence of symbols written on it; hence the complete state of M*A can be expressed by a finite sequence of symbols. M u * 9 ( M )simulates M*A in the following way. Mu inspects the program on 9 ( M ) , determines the initial complete state of M*A, and records this complete state on its own tape. Mu then iterates the following step indefinitely: by examining the program on 9 ( M ) and the last computed complete state of M*A it computes the next complete state of M*A. As

PROGRAMMING AND THE THEORY OF AUTOMATA

I03

the digits of q(M*A) are computed, MU*9(M) records these in the evennumbered squares beyond the program. Since Mu can simulate any machine M in this way, M , is a universal simulator. It is easy to see that there are infinitely many finite control automata M’ which satisfy the defining property of Mu, that is, such that for each finite control automaton M there is a tape 9 ( M ) for which

q ( W * 9 ( M ) ) = q(M*A). We will call any such automaton M’ a universal control automaton. To recapitulate: all elements of 01 may be obtained with a blank tape, and all may be obtained with a single universal control automaton. The procedure used earlier to prove 01 = is the reverse of the procedure used later to prove 01 = y. In the former a program was transformed into an automaton, while in the latter an automaton was transformed into a program. The latter transformation is much more significant than the former, partly because it is more difficult to show that it works, but more fundamentally because of the difference between a finite control automaton and a tape. A finite control automaton is active and has the power to interpret a program, while a tape is passive and can do nothing by itself. A practical corollary of this difference is the fact that it is much more difficult to design a computer than it is to prepare a tape. We buy one computer and use it to solve a wide variety of problems by varying the program fed into it. Let us explore further the practical bearings of Turing’s universal simulation result. The first point to note concerns the infinitude of a computation. The tape of a Turing machine is potentially infinite and the computed output is in general an infinite sequence. In contrast, all actual computers and all actual computations are finite. This difference does not, however, really matter in the present instance, for we can restate the universal simulation result for finite computations. Let qf(M*T)refer to the first f digits of q ( M * T ) if they exist, otherwise to q(M*T). The universal control automaton Mu has this property: For each finite automaton M there is a tape 9 ( M )such that for eachf, q f ( M u * 9 ( M ) )= V f(M*A). Thus, in the application we are making of Turing’s universal simulation result, the fact that a Turing machine is potentially infinite whileanactual computation is finite does not matter. It should be noted that there are applications of Turing machine theory where this difference is essential. For example, people often attempt to use certain results about Turing

104

ARTHUR W. BURKS

machines to help answer the question “Is man a machine?”l) Suchresults as the non-existence of certain decision procedures and the fact that no Turing machine can enumerate all mathematical truths 2, are cited as being relevant to the question. But if man is an automaton he is most certainly a finite automaton, so results about Turing machines, which are infinite, do not apply directly to the question “Is man a machine?” There may be indirect connections of importance to this question, but no one has yet shown that there are. Our second point of comparison between a Turing machine and an actual computer concerns the distinction between a special-purposeand a general-purpose computer. It is natural to think of M*A as a specialpurpose computer, since its function is to compute the single number q(M*A), and to think of Mu as a general-purpose computer, since, suitably programmed, it can compute any number q (M*A).But this way of lookingat things can easily be misleading. The most essential differencehere is between a single Turing machine M*Tand an infinite class of such machines obtained by varying M while keeping T fixed or by varying T while keeping M fixed. Moreover, as “special-purpose” and “general-purpose” are normally used by computer people they connote practical rather than theoreticalconcepts. Most so-called special-purposemachinesare universal control automata in the sense defined earlier. That this is so is fairly evident when it is realized that there exists a universal control automaton with eight states operating on tapes having an alphabet of five symbol^.^) Moreover, any actual special-purpose computer is used to solve a number of problems, not just to compute one number q(M*T), and hence is programmed in some sense. A computer is called general-purpose when it is relatively easy to program or use it on any of a wide variety of problems, and it is called special-purpose when in practice it can only be used to solve a relatively narrow class of problems. The distinction between special-purposeand general-purpose computers is thus a bifurcation Turing, op. cit., shows that there is no decision procedure for whether q(M*A) contains a particular symbol, or whether q (M * A)is an infinite sequence. 2, Kurt Godel, “uber formal unentscheidbare Satze der Principia Mathematica und verwandter Syteme I,” Monatshefte fur Mathematik und Physik 38 (1931) 173-198. Godel showed that there is no complete axiomatic system of arithmetic, from which it follows that the set of mathematical truths cannot be enumerated by a Turing machine. s, Shigeru Watanabe, “5-Symbol 8-State and 5-Symbol &State Universal Turing Machines,” Journal of the Association for Computing Machinery 8 (October 1961) 416-483. l)

PROGRAMMING AND THE THEORY OF AUTOMATA

105

of the class of universal control automata based on a practical criterion. This brings us to our third, and last, point of comparison between a Turing machine and a general-purpose computer. We referred to the information placed initially on the tape T as a “program”, but it might just as well have been called the “data”, for we can think of the machine M a s defining a function from Tto q(M*T).To make the usual distinction between program and data we must divide the information placed initially on the tape into two parts, one part to be called the program and the other part the data. We then think of the program as defining a function to be computed by the machine, the data as constituting the argument or arguments of the function, and the computed output as being the function value for those arguments. As so described, the distinction between program and data is purely arbitrary, and this is certainly so from a purely formal point of view. For example, it is arbitrary whether we say a number referred to in a conditional shift of control (branch) command belongs to the program or the data or both, and when a program is the object of computation (e.g., in compiling) this program is the data. Our criterion for distinguishing program from data is an informal, intuitive one: the program is that which, for the most part, directs operations; the data are those items which, for the most part, are operated upon. At this point it is desirable to use a slightly more complex notation for the tape. Let Z be the finite part of T containing the input information. The rest of the tape is blank, and hence is A, so T may be represented by ZnA. In this notation, each finite control automaton M defines a function from Z to q(M*ZnA). The behavior of the universal control automaton Mu may then be stated as: For each finite control automaton M there is a finite description 9 ( M ) such that

q(MU*9(M)-A)= q(M*A). In this new notation “9(M>,’ refers to the finite portion of the tape holding the description of machine M , whereas earlier it referred to the whole tape which contained this description. Mu is a universal simulator of Turing machines with blank tapes. Is there a universal simulator of Turing machines whose tapes are of the form ZnA? It follows from considerations similar to those given earlier that there is. By our earlier convention we considered only tapes in which the squares 0,2,4,. . ,2n, where n is a non-negative integer, were occupied by alphabetic characters (excluding the blank) and all other squares were blank. Let us now also consider tapes in which the squares 2n+4,

.

106

ARTHUR W. BURKS

+

2n 6 , . ..,2121 may also be occupied by alphabetic characters, where n 12 n + 2 . A tape g(M)-Z-A will then consist of a description of machine M on squares 0 , 2 , 4 , . ..,2n followed by the input information Z on squares 2n 4 , 2 n 6 . . .,2n1. A finite control automaton M together with a tape of the form Z-A defines a function from Z to q(M*Z-A). There is a finite control automaton M J which will simulate any such machine, that is, for each M there is a finite tape 9 ( M ) such that q(M:*.9(M)-z-A) = q(M*z-A).

+

+

The basic principle and design of Mul is the same as that of Mu. The foregoing involved the extension of our earlier conventions so as to allow two distinguishable blocks of information on a tape. This extension can be carried on indefinitely, using the squares 0,2,4,...,2n 2n 4,2n 6 , ...,2nl 2n1+ 4,2nl 6 , . ..,2112

+

+

+ +

+

where n l k n 2, n2 2 nl 2, etc. A universal simulator may be designed for each level, the universal simulator using one more block of input information than the machines it simulates. Thus Mu uses one block of input information to simulate all machines M with blank tapes:

q(A!fu*9(M)-A) = q(M*A); M J uses two blocks of information to simulate all machines M with one block of input information : q(Mt*qM)-z-A) = q(M*z-A); Mu2 uses three blocks to simulate all machines M with two blocks of information : q(M:*9(M)-Z-zl-A) = q(M*Z-ll-A); etc., etc., etc. Automatic programming can be analyzed in these terms, and turns out to be a case of Mu2 using three blocks of input information. Let M m be a general-purpose computer (finite automaton) produced by some manufacturer. In the present state of technology Mm will operate with a “machine language” inconvenient for the programmer to use. Let P be the program expressed in this machine language and D the data for a computation. Then a single “run” of the machine computes the finite number qj(Mm*PnD-A), and if the run were extended to infinity and

PROGRAMMING AND THE THEORY OF AUTOMATA

107

the machine was supplied with an unlimited amount of tape it would compute q(Mm*P"D"A). Suppose now you have constructed a "programmer's language" l) (automatic programming language) in which a programmer can very easily write a program and express the input data of a problem. It is theoretically possible to build a machine which will understand this programmer's language directly; call this hypothetical programmer's computer M p . Supplied with program P and data D this automaton will compute q(Mp*PhDhA). Since both M p and Mm are universal control automata they are theoretically equivalent, their differences being practical. The hypothetical computer M p was designed so as to match the human problem-giver well, that is, so as to be easy to program. But M p is impracticable in the present state of technology, the manufacturer's machine Mm being the best machine actually available. Automatic programming seeks to bridge this gap by doing with "software" what cannot be done with "hardware", to use the current computer jargon. Automatic programming works in the following way. We write a coded description 9 ( M p ) of the hypothetical machine M p in the machine language of the manufacturer's computer Mm. In the usual terms 9 ( M p )is the interpretive routine which translates from the programmer's language to the machine language. Then Mm operating under the direction of 9 ( M p ) will compute the same function as M p . That is, for any program P and data D, written in the machine language of M p

q(Mm*9(Mp)"PnD"A)

=

q(Mp*P"D-A).

Since P and D are written in the machine language of M p the programmer need know nothing about the machine Mm on which the computation is carried out. He need only understand the machine language of the hypothetical machine M p . A comparison of the last two equations

~(M~*9(M)"Z"Z'"A) ~j(Mm*9(Mp)"p"D"A)

= 7 (M*Z"Zl"A) =

q(Mp*P"DhA)

shows that an automatic programming system is an instance of Turing's universal simulation result, with Mm playing the role of Mu2, M p of M , P of Z, and D of Z1. But not every such instance of Turing's universal l) Arthur W. Burks, "The Logic of Programming Electronic Digital Computers," Industrial Mathematics 1 (1950) 36-52. See p. 51.

108

ARTHUR W. BURKS

simulation result is a case of automatic programming. What is the differentia? That is, what do we require of a universal control automaton Mu2 and a translation routine 9 ( M ) so that they constitute an automatic programming system. The additional condition is a practical, humanoriented one : automatic programming is an application of universal simulation which makes programming automatic for homo sapiens. Thus it is essential to the idea of automatic programming that the hypothetical programmer’s computer M p be easier for the programmer to use than the actually available manufacturer’s computer Mm. In other words, the computation system M n z * 9 ( M p )must be matched to the human much better than the system M m alone. Let us consider for a moment the role of language in the operation and theory of computers. From the point of view of current automata theory, automata are just devices which jump discretely from state to state under the direction of input symbols. Thus in proving his universal simulation result Turing shows how to express the states and state transitions of a finite automaton M in a sequenceg ( M ) .This requires only a rudimentary language. No concept of a machine language with a complicated structure corresponding to the structure of a machine appears in automata theory. But such languages are needed in working with actual computers. The reason for this is obvious. The number of states assumed by any actual computer in even a short time is much too large for a state-by-state analysis to be of much use in viewing the whole computer, though such a mode of analysis is often valuable in designing small portions of the computer. In planning and operating a computer we must, therefore, organize it into interrelated units or compounds at various levels: halfadders, flip-flops, etc. at the lowest level; registers, memory switches, counters, etc. at the next higher level; arithmetic units, memories, inputoutput units, controls, etc. at a still higher level. The particular organizations of current computers reflect strongly the present state of technology and our current problem interests, and I feel sure that radically new organizations will appear in the future. But in any case we must organize computers into some such hierarchy of units simply because they are too complicated for us to understand otherwise. The organization of the machine provides the basis for the structure of the machine language. Thus a typical machine language is based on commands, one part of which designates an arithmetic or control operation, the other part of which designates memory locations. Now the machine language was designed from the machine’s point of view. This

PROGRAMMING AND THE THEORY OF AUTOMATA

109

is as it should be, but because man and machine are so different the result is that the machine language is not well suited to the human. For that reason we construct a programmer’s language whose structure is much closer to the structure of ordinary language. To do this is, as we noted earlier, tantamount to designing a hypothetical programmer’s computer which is well-matched to the human user. To recapitulate : The organization of a machine into a hierarchy of interconnected units is essential for the understanding, construction, and operation of a complex computer. But none of this organization plays a significant role in automata theory as it is usually conducted. For the most part, current automata theory makes no distinction between a highly organized computer, on the one hand, and any other possible machine which behaves in the same way. Compare, for example, an IBM 7090 with another hypothetical machine which produces the same results but which has no discernible machine structure and no machine language. Most results in automata theory apply equally well and in the same way to these two machines, which differ radically with respect to organization. As a consequence of this lack of theory, the design and instruction of digital computers is an art, the art by which man controls the machine. The late John von Neumann sought a theory of the organization of automata which would be based on “that body of experience which has grown up around the planning, evaluating, and coding of complicated logical and mathematical automata” 1) and which would have applications in the design and programming of digital computers. He outlined the general nature of this proposed automata theory: its materials, some of its problems, what it would be like, and the form of its mathematics. He began a comparative study of artificial and natural automata. And he formulated and partially answered two fundamental questions of automata theory: How can reliable systems be constructed from unreliable components?z) and, What kind of logical organization is sufficient for an automaton to be able to reproduce itself? His discussion of the last question is particularly relevant to programming. Von Neumann has two models of self-reproduction, a kinematic and a 1) The Computer and the Brain, p. 2. See also his “The General and Logical Theory of Automata,” pp. 1-31 of Cerebral Mechanisms in Behavior - The Hixon Symposium (edited by L. A. Jeffress). 2, “Probabilistic Logics and the Synthesis of Reliable Organisms from Unreliable Components,” pp. 43-98 of Automata Studies (edited by C. E. Shannon and J. McCarthy).

110

ARTHUR W. BURKS

cellular one.1) The cellular one is better for our purpose since it abstracts from kinematic problems of motion and enables one to concentrate on organizational and structural problems. The basis of the cellular model is an infinite array of square cells, each consisting of a finite automaton with 29 states. The state of a cell at time t + 1 is a truth-function of its own state and that of its four contiguous neighbors at time t. There is a fiducial state U, called the unexcitable state, and it is stipulated that at time zero all but a finite number of cells are in state U . The unexcitable state Ucorresponds to the blank state of a square of the tape of a Turing machine. As noted earlier, initially all but a finite number of the squares of the tape of a Turing machine are blank. Thus a Turing machine is initially homogeneous except for a finite area on the tape and for the finite control automaton interacting with the tape. Similarly, the basic framework for a self-reproducing automaton is homogeneous, and at time zero all but a finite number of cells are in this homogeneous state. There is not time to develop the details of the construction and destruction processes which may take place in this cellular system, but a few comments will suffice for our purposes. As a first approximation we can think of the signals transmitted through the system as being of two kinds: computation signals of the usual kind, and construction-destruction signals. Construction signals may be used to shift a cell from its unexcitable state into a particular switch-delay state. Destruction signals may also be used for a destructive shift of a cellular finite automaton back into its unexcitable state. A finite complex of cells in a particular switchdelay configuration constitutes a complex finite automaton which can compute in the way an ordinary finite automaton computes. Its output signals may be converted into construction-destruction signals. Consider now the finite area whose cells are initially in some state other than U.It will be a finite automaton; call it the “primary automaton”. Consider next some other area (finite or infinite) of cells that are in state U. The primary automaton can send signals into this area so as to organize it into a “secondary automatoa”. At any time this secondary automaton is finite, but it can get larger from time to time. It is thus possible to The first is described in “The General and Logical Theory of Automata,” loc. cit. The second is described in a manuscript to be published by the University of Illinois Press under the editorship of the present writer. It is also described briefly in C. E. Shannon, “Von Neumann’s Contributions to Automata Theory,” pp. 123-129 of Bulletin of the American Mathematical Society, Vol. 64, No. 3; Part 2, May 1958. l)

PROGRAMMING AND THE THEORY OF AUTOMATA

111

have “special purpose” construction in this cellular system: a primary automaton constructs some particular secondary automaton. A universal Turing machine can be embedded in the cellular system. A finite area is designed so as to be a finite control automaton. An infinite linear array of cells constitutes the tape. The finite control automaton organizes the cells immediately above and below the tape into a reading loop which passes through one cell (square) of the tape. The finite control automaton can extend and contract this loop by means of constructiondestruction signals. Note that in the embedded Turing machine the tape and control both stand still, and reading and writing are done by means of an indefinitely expansible variable length loop, whereas according to our earlier description of a Turing machine the finite control automaton and the tape move relatively to each other. Structurally these two modes of operation are different, but computationally they are equivalent. All that is required in a Turing machine is that the finite control automaton have ultimate access to each square of the tape. Von Neumann showed how to design a universal constructing automaton Me in this system. It is synthesized out of the following three main parts. (I) A finite control automaton which can read any position of an infinite tape. This operates in the same way as the finite control automaton of the universal Turing machine just described. The tape itself consists of an infinite linear array of cells. The tape is used to store the description 9 ( M ) of an arbitrary secondary automaton which is to be constructed by Me. Each automaton M is finite and hence each 9 ( M ) is also finite, but there is no limit to the size of 9 ( M ) . Since the tape is not bounded in size it is not part of M c .(11) A finite automaton which can interpret an arbitrary description 9 ( M ) . Parts (I) and (11) acting conjointly can read and interpret 9 ( M )and send construction-destruction signals to a specifiedsecondary area so as to construct M there. (111) A finite automaton which can reproduce the tape of (I) along with its contents and attach the results to the secondary automaton M . Parts (I), (11) and (111) operating together constitute the universal constructing automaton Me. The finite automaton Mc operates in the following way. When theinitial tape contents of the universal constructing automaton Me consist of a description 9 ( M ) of a finite machine M, the universal constructor will build M and copy 9 ( M )onto the tape of M . This result may be written: M,*9(M) -+ M * 9 ( M ) .

Self-reproduction is obtained by placing a description 9 ( M c ) of the

112

ARTHUR W. BURKS

universal constructing automaton on its own tape. The construction formula just written applies to any finite automaton M , and since Mc is a finite automaton the formula applies to Mc. Substituting “Me” for “My’ in the formula we obtain: Mc*9(Mc) -+ Mc*9(Mc)

which, in a logical sense, is a case of self-reproduction. Let us pause for a moment to compare Turing’s system with von Neumann’s. On the surface they appear very different, but the fundamental logical and information theoretic principles on which they operate are very similar. Both are closed systems with a denumerable number of states, making deterministictransitions between states. Furthermore, both systems are composed of finite automata. Each square of a Turing machine tape is a two-state finite automaton. A square of tape is not a very powerful automaton, to be sure, since it can do nothing by itself but can only interact with the finite control automaton, but it is nevertheless a finite automaton. Thus, initial states of the system aside, Turing’s system consists of an infinite one-dimensional homogeneous array of finite automata with a single more complicated automaton attached to one element of the array. In contrast, von Neumann’s system consists of an infinite two-dimensional homogeneous array of finite automata. In each case the system is used by imposing a finite amount of inhomogeneity on it. This initial inhomogeneity can spread without bound throughout the system as time progresses. The infinitude of the system provides the matrix or background for unlimited growth, and the finite initial configuration of the system controls the pattern of this growth. Thus a Turing machine consists of a finite control automaton plus an unlimited amount of blank tape. Similarly, a von Neumann cellular automaton consists of a finite automaton plus an unlimited number of cells in state U which may be modified by that automaton. With this comparison of a Turing machine and a von Neumann cellular system in mind let us look at Turing’s universal simulation result once more. In the proof that LY = /? a program was transformed into an automaton, while in the proof that LY = y an automaton was transformed into a program. Thus programs and automata are sometimes interchangeable. When designing the ENIAC l) we expressed this as a choice l)

This was the first electronic “general-purpose” digital computer. It is described in

my paper “Electronic Computing Circuits of the ENIAC,” Proceedings of the Znstitute

of Radio Engineers 35 (August, 1947) 756-767.

PROGRAMMING AND THE THEORY OF AUTOMATA

113

between constructing (“wiring in”) an operation (such as division) and instructing (programming) that operation. Today the point would be made by saying that “hardware” and “software” are, within limits, interchangeable. The “within limits” refers, of course, to the fact that there must be a finite control automaton which is active and has the power to interpret a program. Thus when a Turing machine is embedded in a von Neumann cellular system, the finite control automaton must extend and contract the reading loop by which the tape is read as well as direct the computation on the basis of what is recorded on the tape. A program is really a state of an automaton, so the interchangeability of programs and automata is really an interchangeability of state and system. One is reminded here of the analogy between kinetic energy and potential energy. A finite automaton M is active -a form of kinetic energy -while its description 9 ( M ) recorded in a succession of cells is passive - a form of potential energy. And just as passive programs and active automata are interchangeable within limits, so are potential and kinetic energy. In both cases some activity (kinetic energy) is required in the system initially unless it is to remain forever quiescent. Von Neumann was interested in an existence result concerning the logic of self-reproduction: he sought a formal system with a reasonably minimal base in which one can construct an automaton that will reproduce itself in a manner logically similar to actual self-reproduction. Hence his choice of a relatively weak automaton as the occupant of each cell and his restriction that information cannot propagate any faster than one cell per time step. As a consequence, the construction of a self-reproducing automaton in his system is exceedingly involved and complicated. Moreover, practical considerations dictate that it operate serially, because parallel operation results in complicated, ad hoc phasing and interlocking problems. Systems more directly relevant to computer design and programming and the study of adaptive and evolutionary processes can be obtained by strengthening von Neumann’s basis. This is done principally by placing much more powerful finite automata in the cells, and secondarily by relaxing the restrictions on the speed of transmission of information between cells. The second modification also makes parallel operation feasible. Self-reproduction becomes simple in such a system, l) and can, of l) See for example, my “Computation,Behavior and Structure in Fixed and Growing Automata,” Behavioral Science 6 (January 1961) 5-22.

114

ARTHUR W. BURKS

course, become trivial, as when there is a cell statea such that a cell in state a will cause any immediate neighbor in state U to go directly into statea. One type of modification of this form has been made by John Holland, who calls his systems iterative circuit computers. Another modification has been tentatively considered by myself, and I will outline it here. The idea of an automatic programming language is a commonplace now and it is customary to teach this language to the user of a machine, rather than the machine language. As noted earlier, an automatic programming language is the machine language of a hypothetical programmer's machine M p with a certain organization, and this organization is presupposed in the automatic programming language. This suggests that it would be better to teach the potential user of a machine about the hypothetical machine M p in conjunction with its language rather than to teach the automatic programming language in isolation from this hypothetical machine. But what I wish to propose goes further than this. The hypothetical machine M p was designed to solve all problems of a very wide class, and hence does not take advantage of the special properties of a particular problem. This limitation is inherent in the idea of a general-purpose computer. For many problems it is easier to think of the problem in terms of a special-purposecomputer especially designed to solve that problem. In doing this one will not be distorting his natural way of formulating a problem to adapt it to a particular computer. Instead, he can formulate the algorithm for solving his problem by designing a special-purpose computer analogous to the problem. I suggest, then, that instead of always writing a program for a problem one should sometimes design a special-purpose computer for that problem. No doubt this suggestion seems preposterous. But the moral to be drawn from the work of Turing and von Neumann is that programs and computers are, to a large extent, interchangeable. Since this is so there cannot really be such a great difference between writing a (specialpurpose) program and designing a special-purposemachine as it seems at first sight. There appears to be a great chasm between these two types of activities because the comparison between machine design and program writing is usually drawn between the long, involved design procedures which have produced our present general-purpose computers, and the relative ease of writing a program in a given rigorously formulated program language. But this contrast is not the relevant one here. The engineering design of an actual computer involves much more than the

PROGRAMMING AND THE THEORY OF AUTOMATA

115

purely logical design of the computer, and this purely logical design is constrained by these engineering considerations. Moreover, in writing out or diagramming the logical design of a computer one does not have available a rigorously formulated design language comparable in power to the best current automatic programming languages. Hence my proposal involves the development of a framework or language of great expressive power for specifying the logical structure of any computer. Experience in machine design and the use of flow charts for programming suggests that this language be diagrammatic as well as symbolic. Moreover, it is feasible to build a computer which can scan a two-dimensional diagram, so that the design of a machine in this language can be fed directly into the manufacturer's machine Mm.In other words, in designing a machine M one is writing 9 ( M ) in the proposed machine design language. The machine M mmust be instructed how to interpret the expressions written in the machine design language - this calls for an interpretive routine 3. To summarize: when one is interested in a computation q(M*D-A) he writes 9 ( M ) and gives it to the machineprogram complex M m * 9 . The number q (Mm*9"9(M)"D7'l), which equals q(M*D"A), is then produced. Thus, my proposal involves, first, the development of a rigorously formulated machine design language, and second, the development of a routine 3 for the automatic translation of expressions in that language into the machine language of the actual machine Mm.These two steps are, of course, the same as those required for the development of an automatic programming system Mm*9(MP): the machine language corresponding to the programmer's machine MP must be worked out and the interpretive routine 9 ( M P )must be written. Likewise, the use of the automatic system Mm*J is similar to the use of the automatic pro. both cases one is given a problem. To solve the gramming M m * 9 ( M P ) In problem on M m * 3 he writes a description 9 ( M ) of a machine M which is equivalent to that problem. To solve the problem by means of M m * 9 ( M p ) he writes a program P which is equivalent to that problem. The systems M m * 9 and M m * 9 ( M P )operate on different levels of the hierarchy of Turing machines introduced earlier. It will be recalled that the universal machine Mu uses one block of input information to simulate Turing machines with blank tapes, Mul uses two blocks to simulate machines with one block of input information, Mu2 uses three blocks to simulate machines with two blocks, etc., etc. In this hierarchy M m * 9 is a case of Mul, as is shown by the formulas:

116

ARTHUR W. BURKS

and Mm used with 9 ( M p )is a case of Mu2, as is shown by the formulas

q(Mm*9(M,)-P-D-A) q(M:*qM)-Z-zl-A)

=

q (M,*P^D^A)

=

q (M*Z-zl-A).

There are many possible approaches to our proposed machine design language. We will briefly indicate an approach which is suggested by von Neumann's cellular self-reproducingautomaton but which diverges from it in a number of important respects. A finite or growing automaton of any power may be stipulated as the contents of a cell, provided that the specification of the automaton, either directly or via a chain of definitions, is reasonably simple. Thus one cell could store a number, with the understanding that the cell can store as many (finite) digits as the number has. For example, if it stores a ten bit number x to begin with and is to store x2, x3, x4, ... at various stages during the computation, the cell will automatically grow in size so as to accommodate the extra bits that are produced by successive multiplications. In specifying a problem by means of a special-purpose computer one would assume as many serial stores, parallel memories, control units, etc., as was convenient. Data could be organized into blocks in natural ways. The control automata stipulated could direct operations like: sum the series in block A, monotonize the data in blocks B and C, withdraw from memory all sequences having property 4, etc. There would be provision in the machine design language for defining new automata in terms of old ones, so once an automaton is specified others can easily be designed in terms of it. Von Neumann has a fixed crystalline structure for his cells. We propose to allow new cells to spring up between old ones under the control of the computation. Suppose a list of words is stored in bins and at a later date new entries are to be inserted. This change would be conceived as an automatic process of inserting new storage bins between the old ones. This change must, of course, be accompanied by an appropriate change of the switches which connect these bins to the rest of the automaton. In general, storage and computing facilitieswould be created wherever needed and in a form suited to the problem being solved. Hence a batch of information would not be stored in a homogeneous memory, as is the case

PROGRAMMING AND THE THEORY OF AUTOMATA

117

in current computers, but in a memory organized so as to reflect the organization of the information itself. That is, the memory would be divided into categories, sub-categories, etc., in natural and useful ways, cross-switching connections would be assumed where needed, etc. Current computers are organized into large, specialized units such as memories, arithmetic units, and controls. The reasons for this organization are to be found in the nature of the components from which computers are built. Since the special-purposecomputers to be designed in our proposed machine design language are not to be built, there is no reason for organizing them in the conventional way. Rather, they should be organized in whatever way best accommodates the problem at hand. Consider, for example, a two-dimensional partial differential equation. It may be convenient to solve this equation by computing the value of a function at all grid points simultaneously, in which case the specialpurpose computer should be organized so as to do this. It should be clear from the foregoing that in our proposed machine design language one could formulate machine organizations radically different from present ones. In conclusion, let us review briefly how one would use the proposed machine design language. It would be most effective when applied to a problem capable of analog treatment, i.e., whose structure may be paralleled by the structure of a special-purpose computer which will solve the problem. In such a case the mathematical equation describes the behavior of a physical model. To specify the solution of this equation one describes in the machine design language a special-purpose computer which would operate analogously to the given physical model. The description of this special-purpose computer is supplied to a generalpurpose computer which translates it into its own machine language and then solves the problem. Whatever the practical feasibility of this proposed system, I think that the theoretical possibility of it illuminates Turing’s and von Neumann’s results on universal machines.

THE ALGEBRAIC THEORY OF CONTEXT-FREE LANGUAGES*

N. CHOMSKY Massachusetts Institute of Technology AND

M. P. SCHUTZENBERGER Harvard University 1. LINGUISTIC MOTIVATION We will be concerned here with several classes of sentence-generating devices that are closely related, in various ways, to the grammars of both natural languages and artificial languages of various kinds. By a language we will mean simply a set of strings in some finite set V of symbols called the vocabulary of the language. By a grammar we mean a set of rules that give a recursive enumeration of the strings belonging to the language. We will say that the grammar generates these strings. (Thinking of natural languages, we would call the generated strings sentences; in algebraic parlance they would ordinarily be called words and the vocabulary would be called an alphabet; regarding a grammar as specifying a programming language, the strings would be called programs; we will generally use the neutral term strings). For a class of grammars to have linguistic interest, there must be a procedure that assigns to any pair (a, G), where a is a string and G a grammar of this class, a satisfactory structural description of the string a with respect to the grammar G. In particular, the structural description should indicate that the string a is a well-formed sentence of the language L(G) generated by G, where this is the case. If it is, the structural description should contain grammatical information that provides the basis for explaining how a is understood by speakers who have internalized the grammar G; if it is not, the structural description might indicate in what respects u deviates from well-formedness.

* This work was supported in part by the U. S.A r m y Signal Corps, the Air Force Office of Scientific Research, and the Office of Naval Research; and in part by the National Science Foundation; and in part by a grant from the Commonwealth Fund.

119

THE ALGEBRAIC THEORY OF CONTEXT-FREE LANGUAGES

We will be concerned with only one aspect of the structural description of a sentence, namely, its subdivision into phrases belonging to various categories. Thus, for example, a structural description of the English sentence “those torn books are completely worthless” should indicate that those is a Determiner, torn and worthless are Adjectives, books is a Noun, completely is an Adverb, those torn books is a Noun Phrase, completely worthless is an Adjective Phrase, are completely worthless is a Verb Phrase, the whole string is a Sentence, as well as additional details regarding subclassification. This information can be represented by a diagram such as (1) :

7

(1)

.1 -NP-----l

.1

.1

Det

Adj

those

torn

.1

.1

s

-

4 N 4

books

.1 I -pv

.1

are

.1

7 f l 1

4

.1

Adj

D

.1

completely

.1

worthless

or, equivalently, by a labelled bracketing of the string, as in (2): (2)

[s [NPb e t thosel[~djtorn] [N books]] [VPare l a p [ D completely] worthless] ] 1.

[ ~ d j

A major concern of the general theory of natural languages is to define the class of possible strings (by fixing a universal phonetic alphabet); the class of possible grammars ; the class of possible structural descriptions; a procedure for assigning structural descriptions to sentences, given a grammar; and to do all of this in such a way that the structural description assigned to a sentence by the grammar of a natural language will provide the basis for explaining how a speaker of this language would understand this sentence (assuming no limitations of memory, attention, etc.). The grammar, then will represent certain aspects of the linguistic competence of the speaker of the language. We will not be concerned here with the empirical question of adequacy of the structural descriptions or the grammars that we will investigate. In fact, the classes of grammars that we will consider, and the kinds of

120

N. CHOMSKY AND M. P. S C ~ Z E N B E R G E R

structural descriptions that they generate, are undoubtedly too narrow to do justice to real human linguistic competence. Nevertheless, the systems we consider (which, in effect, formalize traditional notions of parsing and immediate constituent analysis) bear certain relations to the kinds of systems that seem empirically adequate, and that are, for the time being, too complex to permit abstract study.l) In the representation (2), we have, aside from brackets, two kinds of symbols: (i) symbols of the generated string (i.e., the six symbols those, torn, books, are, completely, worthless)2); (ii) the symbols S, NP, Det, A&, N , VP, AP, D representing phrase-categories. Symbols of type (i) we will call terminals; symbols of type (ii), non-terminals. We will assume, below, a fixed stock of terminal and non-terminal symbols from which the grammars of all languages are constructed. The set of terminals can be regarded as constituting a potential common vocabulary for all languages. Thinking of spoken language, we can regard the set of terminals as defined by a universal phonetic alphabet (assuming, as is natural, ’an upper bound on the length of morphemesz). Thinking again of natural language, we can regard the fixed set of nonterminals as a universal set of categories from which the phrase types of all languages are drawn. An important and traditional question of general linguistics has to do with the possibility of giving a concrete interpretation of the non-terminals that constitute the categories in terms of which grammars are constructed - is it possible, in other words, to find a general definition, independent of any particular language, of such categories as Noun, Verb, etc., in terms of semantic content or formal properties of grammars? The problem of giving a concrete interpretation to the set of terminals and non-terminals is, of course, like the problem of empirical adequacy of certain categories of grammars, a crucial issue in the science of language; but it is beyond the range of our immediate interests here. We can generate the sentence “those torn books are completely worthless”, with the structural description (2), by the set of rewriting rules: For further discussion of these questions, see Chomsky [lo]. In a linguistically adequate grammar, we would generate not these symbols, but rather a more abstract representationusing the symbols the, demonstrative, plural, tear, participle, book, plural, be, plural, complete, ly, worth, less, in this order. Representation in terms of these symbols (called morphemes) will be converted to phonetic representation by a set of phonological rules which will not concern us at all here. See Chomsky and Miller [15]. We will use actual sentences, such as (2), only for illustrativeexamples, and will therefore not be concerned with such refinements as this. l)

2)

THE ALGEBRAIC THEORY OF CONTEXT-FREE LANGUAGES

(3)

S NP Det Adj Adj N VP AP D

121

+ N P VP -+ Det Adj N + those + torn -+worthless +. books + are AP +DAdj += completely

by a derivation that is constructed in the following way. First, write down the initial symbol S as the first line of the derivation. Form the n 1st line of the derivation by selecting at will an occurrence of a non-terminal (x in the nth line (where this occurrence of (x is not labelling a bracket), and replace it by the string: [=PI, where 01 + y is one of the rules of (3). Continue until the only non-terminals that appear are those that label brackets, at which point, the derivation is terminated. Deleting the brackets of a terminated derivation, with their labels, we have a string containing only terminals. Call this a terminal string. Four different terminal strings can be generated by the grammar (3). We can construct a grammar that generates infinitely many terminal strings, each with a structural description, by permitting recursions, e.g., by adding to (3) the rules

+

(4)

NP -+ that S VP + is AP AP -+ obvious

in which case we can generate, e.g., “that those torn books are completely worthless is obvious”, etc.1). Each of the generated sentences will again have a structural description of the appropriate kind. Grammars of the type (3), (4) we will call context-free (CF) grammars. They are characterized by the property that exactly one non-terminal appears on the left-hand side of each rewriting rule. If this restriction is relaxed, we have systems with entirely different formal properties. It l) In this case, infinitely many non-English sentences will also be generated, e.g., “that those tom books is obvious are completely worthless”, etc. Hence the grammar ((3), (4)) is unacceptable. The difficulty of avoiding empirical inadequacies of this sort can easily be underestimated.We stress again that this is the key issue for both linguistics and psychology, though it will not concern us directly here. For discussion, see Chomsky [13].

122

N. CHOMSKY AND M. P. SCHUTZENBERGER

seems that grammars for natural languages must contain at least some rewriting ryles of this more general form, and some rules that are not rewriting rules at all. Cf. Chomsky [8], [lo], and [12], Chomsky and Miller [15], for further abstract discussion of systems of these sorts, which we will not consider further here. A set of terminal strings that can be generated by some CF grammar we will call a CF language. A CF language may generate a terminal (debracketised) string y with several different structural descriptions. In this case, if the grammar is empirically adequate, y should be structurally ambiguous. Consider, for example, the CF grammar with the rules (5)

S NP VP Verb Adj N

+ N P VP -+ they; NP -+ Adj N; NP -+ N + are NP; VP -+ Verb NP -+ are flying +flying -+ planes

With this grammar we can generate both (6) and (7): (6) (7)

[ S [ N P they1 [

v P [ vare ~~ flying1 ~ [NAN planes1 1 1 I.

[ S [ N P they1 [VPare [NP 14dj flying1 [Nplanes1

11 1

Correspondingly, the terminal string “they are flying planes” is structurally ambiguous; it can mean: “my friends, who are pilots, are flying planes”; or: “those spots on the horizon are flying planes”. Study of structural ambiguity is one of the most instructive ways to determine the empirical adequacy of a grammar. We will see below that there are certain CF languages that are inherently ambiguous, in the sense that each CF grammar that generates them assigns alternative structural descriptions to some of their sentences. Furthermore, we will see that the problem of determining whether a CF grammar is ambiguous is recursively unsolvable, 1) even for extremely simple types of CF grammars. Though CF grammars are far from fully sufficient for natural languages, they are certainly adequate for the description of familiar artificial languages, and apparently for the description of certain, perhaps all, programming languages. In particular, a CF grammar can be written for There is, in other words, no mechanical procedure (algorithm) for determining whether an arbitrary CF grammar assigns more than one structural description to some string that it generates.

THE ALGEBRAIC THEORY OF CONTEXT-FREE LANGUAGES

123

ALGOL[18], and each program in ALGOLwill be one of the terminal strings generated by this grammar. Clearly, a programming language must be unambiguous. Therefore, it is important to determine whether, in fact, a particular programming language meets this condition, or whether a particular infinite set of programs can each be unambiguous, given certain techniques for constructing them (e.g., techniques that can be represented as rules for constructing derivations in a CF grammar). As indicated in the preceding paragraph, these may be rather difficult questions. Suppose that GI and GZ are generative systems that specify certain techniques for constructing computer programs; suppose, in fact, that they are grammars that generate the programming languages L1 and Lz, each of which consists of an infinite number of strings, each string being a possible program. It is often interesting to inquire into the relative power of programming languages. We will see that if GIand GZare CF grammars (as, e.g., in the case of ALGOL), most problems concerning the relation of L1 to LZare recursively unsolvable, in particular, the problem of determining whether L1 and LZ have an empty or an infinite intersection, or whether L1 is contained in LZ [2], or whether there is a finite transducer (a “compiler”) that maps LI onto LZ (Ginsburg and Rose, personal communication). Hence it is possible that general questions concerning the formal properties of CF systems and formal relations between them may have a concrete interpretation in the study of dataprocessing systems, as well as in the study of natural language. This possibility has been pointed out particularly by Ginsburg and Rice [18], Ginsburg and Rose [19]. In considering a grammar as a generative device, we may be concerned with the language (i.e., set of terminal strings) that it generates, or with the set of structural descriptions that it generates (N.B.: each structural description uniquely determines a terminal string, as in (2)). The latter is clearly the much more interesting question. Similarly, in studying generative capacity of a class of grammars (or relative capacity of several such classes, as in evaluating alternative linguistic theories), we may be concerned either with the set of languages that can be generated, or with the set of systems of structural descriptions that can be generated. The latter, again, is a more interesting, but much more difficult question. Investigation of such questions is, altogether, quite recent, and attention has been restricted almost exclusively to generation of languages rather than of systems of structural descriptions. We will consider genera-

124

N. CHOMSKY AND M. P. SCH6TZENBERGER

tion from a point of view intermediate between the two just mentioned. We will consider a representation of a language not as a set of strings and not as a set of structural descriptions, but as a set of pairs (c,n), where c is a string and n expresses its degree of ambiguity; that is, n is the number of different structural descriptions assigned to c by the grammar G generating the language to which it belongs.

2. GRAMMARS AS GENERATORS OF FORMAL POWER

SERIES

2.1. Suppose that we are given a finite vocabulary V partitioned into the sets VT (= terminal vocabulary) and VN (= non-terminal vocabulary). We consider now languages with the vocabulary VT, and grammars that take their non-terminals from VN. Let F(VT) be the free monoid generated by VT, i.e., the set of all strings in the vocabulary VT. A language is, then, a subset of F(VT). Consider a mapping r which assigns to each string f E F(VT) a certain integer . Such a mapping can be represented by a formal power series (denoted also by r ) in the non-commutative variables x of VT.Thus

r

(8)

=

X

ji

d

=

fi

+ fz + ...)

where f i , f z , ... is an enumeration of all strings in VT. We define the support of r (= Sup(r)) as the set of strings with non-zero coefficients in r. Thus

We do not insist that the coefficients of the formal power series r in (8) be positive. If, in fact, for each i, 2 0, then we shall say that r is a positive formal power series. If for each f a E F( VT), the coefficient is either zero or one, we say that r is the characteristic formal power series of its support. 2.2. If r is a formal power series and n an integer, we define the product nr as the formal power series with coefficients , where < r , f > and < r ' , f > are, respectively, the coefficients off in r and r'. We define rr' as the formal

>

+

+

+

>,

THE ALGEBRAIC THEORY OF CONTEXT-FREE LANGUAGES

power series with coefficients = &



where for each n, nnrn is the polynomial formed from rn by replacing all coefficients of strings of length > n by zero. Then the ring of the formal power series becomes an ultrametric, hence topological, ring. With these notions defined, we can turn to the problem of relating the representation of languages in terms of formal power series to the representation of languages by generative processes such as CFgrammars.

2.3. Suppose that G is a generative process generating the language L(G). Each string f E F( VT) is assigned a certain number N(G,f) of structural descriptions by G; N(G,f) > 0 justincase f EL(@.N(G,f)expresses the degree of structural ambiguity offwith respect to G. It is natural to associate with G the formal power series r(G) such that = N(Gf), where is the coefficient offin r(G). Thus r(G) expresses the ambiguity of all terminal strings with respect to the grammar G . The coefficient is zero just in casefis not generated by G; it is one just in case f is generated unambiguously (in one and only one way) by G; it is two just in case there are two different structural descriptions for f, in terms of G; etc. An r(G) associated with a grammar G will, of course, always be positive; and its support Sup(r(G)) will be exactly the language L(G) generated by G. We can regard a formal power-series r with both positive and negative coefficients as being associated with two generative processes

126

N. CHOMSKY AND M. P. SCH~TZENBERGER

G1 and Gz.The coefficient < r f > off in r can be taken as the difference between the number of times that f is generated by G I and by Gz; that is, in this case, = N(G1,f) - N(G2,f). Suppose that G is a CF grammar with non-terminals 011, ...,an,where 011 is the designated initial symbol (i.e., 011 = S, in the example (I), above). We can construct the formal power series r(G) associated with G by a straightforward iterative procedure. To do this, we proceed as follows. Observe, first of all, that G can be written as a system of equations in the variables 011, ..., an. Let ~ ) t , l ,..., q ~ t be , ~ the ~ strings such that 0 1 ~-+ vt,j (1 5 j 5 mg) are rules of G. We then associate with at the polynomial expression ut, (1 1)

at

= vt,l

+ vr,z + ... + v w t

We now associate with the grammar G the set of equations O11 = Ul;

(12)

...;an=an.

Let us assume that the grammar G contains no rules of the form 016 -+ O1(

e

+aj.

It is clear that these assumptions do not affect generative capacity [2]. That is, for every CF grammar containing such rules there is another grammar without any such rules, which generates the same language. We will also explicitly require, henceforth, that if G is a CF grammar and 01 is a non-terminal of G, then there must be terminal strings derivable from 01 - i.e., if G’ contains the rules of G and has 01 as its initial symbol, then the language generated by G‘ must be non-null. Again, this requirement obviously does not affect generative capacity. Returning now to the problem of constructing the power-series that is associated with G and that represents the degree of ambiguity that G assigns to each string, observe that we can regard each equation 014 = at of (12) as defining a mapping yr that carries an n-tuple (rl, ..., rn) of power series into the power series defined by replacing 01jin a6 by rj. This is legitimate because of the closure properties of the ring of power series noted above in 0 2.2. Thus the set of equations (12) defines a mapping y , (14)

y(r1, ..., rn)

= (r’l,

..., rIn), where 1’1 = yt(r1, ..., m).

THE ALGEBRAIC THEORY OF CONTEXT-FREE LANGUAGES

127

Consider now the infinite sequence of n-tuples of power series Po, e l , .. , where

.

(15)

eo = (ro.1, ..., r0.n) = (0,..., 0)

el = (r1,1, ..., e 2 = (r2,1, ..., r2,n)

and where for each i j ( j > 0 ) (16)

rj,t = yt (rj-1,1, ..., r j - ~ , ~ ) ,

and where 0 is the power series in which all coefficients are zero. Each rj,r in (15) has only finitely many non-zero coefficients; it is, in other words, a polynomial. Furthermore, we can show that for each i,j,j’ such thatj’ > j > 0, 1 I i s n, it is the case that (17)

rj,t

= rj’,c (mod degj).

Consequently, as noted in Q 2,2, the limit rm,j of the infinite sequence rI,t, r2,t, ... is well-defined for each i (it is, of course, in general not a polynomial). We will call the n-tuple ( r m , l , ..., roo,,& so defined, the solution to the set of equations (12). Indeed, the n-tuple (roo,l,..., is the only n-tuple within our framework to satisfy the set of equations (12). For this reason we will say that a power series is algebraic [42] if it is one of the terms of a solution to a set of equations such as (12), where there is no restriction on the sign of the numerical coefficients. We will call a power series context-free if the coefficients in the defining equations are all positive. In particular, rm,l, which we will henceforth call the power series generated by the grammar G of (12) with initial symbol L Y is ~ ,the power series associated with G in the manner described at the outset of Q 2.3. Its support is the language L(G) generated by G, and the coefficient < r m , l , f > of a stringf E F( VT)determines the ambiguity of fwith respect to G, in the way described above. Notice that if an algebraic power series is context-free, it is positive, but not necessarily conversely. That is, a power series may be a term of the solution to a set of equations and may have only positive coefficients, but may not be a term of the solution to any set of equations with only positive coefficients.1 ) 1) For example, using notions which will be defined below in 8 3.1, the Hadamard square s 0 s, for s E Ao, has only positive coefficients (and has the same support as s) but it is not, in general, generated by a set of equations with only positive coefficients.

128

N. CHOMSKY AND M. P. SCHUTZENBERGER

2.4. As examples of the process described above, consider the two grammars (18) and (19) :

S + bSS; S + a S -+ SbS; S + a. Each of these grammars has only a single non-terminal; hence the corresponding set of equations will in each case consist of a single equation. Corresponding to (18) we have (20), and corresponding to (19) we have (21).

S S

=a =a

+ bSS + SbS.

The equations (19) and (20) correspond to (12), above, with n = 1. Both (19) and (20) meet the condition (13). Consider first the grammar (18) represented in the form (20). Proceeding in the manner of the preceding section, we regard (20) as defining a mapping ly such that y ( r ) = a brr, where r is a power series. We then (corresponding to (15)) form the infinite sequence eo, el, e 2 , .. as follows :

+

(22)

eo = ro

=0

+ broro = a + bOO = a + brlrl = a + baa = a + brerz = a + b(a + baa)(a + baa) a + baa + babaa + bbaaa + bbaabaa = r4 = a + br3r3

el = rl = a

r2 = r3

e2 = ~3

=a

=

~4

.. . . . .. . . .

Clearly for each j , j ‘ such that j ‘ >j > 0, we have rj = rj’ (mod deg j ) . Consequently the limit r- is well-defined. This power series is the solution to equation (20), and its support is the language generated by the CF grammar (18). Notice that the power series roo, in this case, is characteristic, and its support is the set of well-formedformulas of the “implicational calculus” with one variable in parenthesis-free (Polish) notation (with the symbol a playing the role of propositional variable, and b the role of the operator “conditional”). Consider now the grammar (19) represented in the form (21). We regard (21) as defining a mapping y such that y ( r ) = a rbr, where r is a power series. We now form the infinite sequence eo, el, ~ 3 2 ,.. :

+

.

THE ALGEBRAIC THEORY OF CONTEXT-FREE LANGUAGES

(23)

129

eo = ro = 0 =a e 2 = r2 = a e 3 = r3 = a

+ robro = a + ObO = a + rlbrl = a + aba + r2br2 = a + (a + aba)b(a + aba) = a + aba + 2ababa + abababa = r4 = a + r3br3 = u + aba + a(ab)% + 5 ( ~ b )+ ~ a6(ab)4a+ 6(ab)5a + 4(ab)aa + (ab)7a

el = rl

~4

.. . .. ...

Again, for each j, j‘ such that j‘ >j > 0, we have rj and the limit rm is defined as the power series

(24) rm where

=

= rj’, (mod

deg j ) ,

1 [2n”] n+ l (ab)%a= a + aba + 2(abI2a + 5(abI3a + 1 4 ( ~ b )+ ~ a4 2 ( ~ b ) + ~a

KI-

-2nx2n-1xx..xn+1 1 x 2 x ... x n

The power series rco of (24) is the solution to the equation (21), and its support is the language generated by the grammar (19). It is not, in this case, a characteristic power series. Taking the symbol a again as a propositional variable and b as the sign for “conditional”, the grammar (19) is the set of rules for generating the well-formed formulas of the implicational calculus with one variable in ordinary notation, but with the parentheses omitted. The structural descriptions generated by (19) in the manner described in section 1 (cf. (3)) are of course unambiguous, since brackets are preserved, but the terminal strings formed by debracketization are ambiguous, and the degree of ambiguity of each generated terminal string is exactly its coefficient in r- - thus ababa can be interpreted in two ways, either as (ab(aba))or ((aba)ba),etc. A more general case has been treated by Raney [38] by Lagrange’s inversion formula. In (20) and (21) all coefficients are positive and the solution is therefore a positive power series. Consider, however, the set of equations consisting of the single member

(25)

S = u - SbS.

In this case we have the sequence

130

N. CHOMSKY AND M. P. SCmTZENBERGER

eo = ro = 0 ei=ri=a-robro =a-Obo=a @ 2 = rz = a - rlbrl = a - aba @3 = r3 = a - rz brz = a - (a - aba)b(a - aba) = a - aba 2ababa - abababa

(26)

+

In fact the coefficients in ,or of (26) are exactly those of et of (23) except for sign - the coefficient off in el of (26) is positive just in casefhas an even number of b’s. The power series rm which is the solution to (25) is not positive and it is consequentlynot context-free (though its support happens to be a contextfree language, in this case, in fact, the language with (19) as one of its grammars). We can, however, regard roo as the difference between two context-free power-series roo+ and rm-; and, correspondingly, we can regard its support as the set of strings that are not generated the same number of times by a pair of CF grammars G+ and G- which generate roo+ and rm-, respectively. Suppose we set S = S+ - S-, so that (25) becomes S+ - S- = a - (S+ - S-)b(S+ - S-) = a - (S+bS+- S+bS- - S-bS+ S-bS-) =a S+bS- S-bS+ - (S+bS+- S-6s-).

(27)

+

+

+

Consider now the set of equations (28)

(9

(ii)

+

+

S+ = a S+bSS-bS+ S- = S+bS+ S-bS-.

+

This is a set of positive equations with two variables S+ and S-, and it will have as solution the pair (roo+,roo-), where roo+ is the limit of the sequence ro+, rl+, ... and roo- the limit of the sequence ro-, rl-, ... of (29) :

eo = (ro+, ro-) = (0,O)

ei = (rl+, rl-) = (a, 0 )

ez = (rz+,r2-) = (a, aba).

It is clear that where roo is the solution to (25), roo = rm+ - rm-. But, furthermore, rm+ is the power series generated by the CF grammar G+ with the initial symbol S+ and the grammar (28i); and roo- is the power

THE ALGEBRAIC THEORY OF CONTEXT-FREE LANGUAGES

131

series generated by the CF grammar G- with the initial symbol S- and the grammar (28ii). In a similar manner, any algebraic power series can be represented (in infinitely many different ways) as the difference of two context-free power series, and its support can be regarded, therefore, as the set of strings which are not generated the same number of times by two CF grammars. This is as close as we can come to a concrete interpretation for the general notion of algebraic power series. More generally, the same construction could be carried out for an arbitrary ring of coefficients instead of the ring of natural numbers used above. This is a still unexplored domain. For instance, if the coefficients are taken modulo a prime p (i.e., if we consider as “non-produced” the strings produced a multiple of p times), the formal power series& > ozPn in the single terminal z is algebraic [27], although its support cannot be the support of any of the power series introduced above. 3. FURTHER OPERATIONS ON FORMAL POWER SERIES 3.1. In Q 2.2 we observed that the set of power series is closed under the

operations of addition, product, and multiplication by an integer. We pointed out that the support of r rl is the union of the supports of r and rl, and that the support of rr’ is the set product of the supports of r and rl, provided that the coefficients are non-negative. We will now turn to two other operations under which the set of power series is closed, and consider the corresponding set-theoretic interpretation for the supports. It is standard terminology to say that r is quasi-regular if = 0. Then rn‘ = 0 (mod deg n) for 0 < n < nl and the element r* = limn+ < n1 < n rn‘ is well-defined. Furthermore, r* satisfies the identity

+

co

r

(30)

-+ r*r = r + rr* = r*,

which determines it uniquely. Thus r* is usually called the quasi-inverse of r. This notion relates directly to the more familiar notion of an inverse by the remark that if rl = e - r and rrr = e r*, then rlrl’ = (e - r) (e r*) = e - r r* - rr* = e = r r , that is, rrl = rl-1. Conversely, given r1 such that < I 1 , e> = 1, we can write it as r’ = e - r, where r is quasi-regular, so that e r* is the inverse of r“. Note that by the very definition of r*, this power series has only non-

+

+

I1

+

I

+

132

N. CHOMSKY A N D M. P. SCHUTZENBERGER

negative coefficients if r does, and that Sup r* = (Sup r)*, where on the right side of the equation the star denotes Kleene’s star operation [21]. In particular if V is an arbitrary set of letters and if the power series v is defined by = 1 if x e V = 0 if x 4 V (i.e., if v is the characteristic function of V), e V* (in Kleene’s sense) is the set of all words generated by the letters of V and e v* = (e - v)-1 is the characteristic function of this set. This follows from the fact that any word f e V* appears once and only once in the infinite sum En> 0 Vn. Consequently, when we know the characteristic function r of a set of strings, we are able to write also the characteristic function (1 - VT)-l-r of its complement. It is worth mentioning that in this case the latter has non-negative coefficients and although it is algebraic in the sense defined above, it is not necessarily context-free. The second operation that we define is the Hadamard product, thus generalizing in one of the possible ways the usual notion of classical analysis. The definition that we give differs from the various extensions to the case of several variables that occur in the literature, but it seems to be most natural extension for non-commutative power series. Where r and r’ are two power series, their Hadamard product r 0 r’ will be the power series with coefficients

+



+

=

< r , f >

identically for all stringsf. Hence Sup (r o r’) = (Sup r) n (Sup r’), and r o r‘ is a characteristic function if r and r‘ are. Finally we introduce the following notation: given a string xrlxr2 ...,

xt n - l ~ t n=f ( x t , e V ) we define f (the mirror image off) to be the string

f= f nx4n-1

(32)

* *

c* rr

- x42x41

f’x

Clearly f =f and the relation ff‘=f ” implies f” = Formally this mapping is an involutory anti-automorphism of the ring of power series and it can be proved to be uniquely characterized by this property (up to a permutation of the elements of V).

3.2. The notation just introduced will be used later on for simplifying the description of grammars in the following way. Suppose that a grammar G contains the rules (33)

THE ALGEBRAIC THEORY OF CONTEXT-FREE LANGUAGES

where the nj's are polynomial expressions in V rule implies

(0~1). Then

133

the second

(34) and the rules (33) can be replaced by the simpler rule (35)

011 = n1( 1

-n 4 ) -1n2

+ n3.

We can, in fact, give a linguistic interpretation to this simplified form of description. Thus, for example, a pair of rules of the form a 1 +.flag f2, a 2 +a2012 (that is, a pair which can now be given in the form: a 1 --f fl(1 -a2)-lf2) can be regarded as constituting, in effect, a rule schema: a1 +fiaznf2 (n = (1, 2, ...). With this reinterpretation, the grammar, though still finitely specifled by rule schemata, consists of an infinite number of rules. But now recall the manner in which a structural description (a labelled bracketing) is assigned to a terminal string generated by a CF grammar (see above, !j 1). A grammar specified by the rule schema given above can generate a structural description of the form (36)

---

[a,fiL2P11[a2P21 . - * [ a , ~ n I f 2 1 - - -

for each n, where each P k is derived from a 2 . In the sentence (terminal string) with this structural description, each $k is a phrase of type a 2 , where $k is formed by debracketization of Pk. The successive phrases $1, ...,$n form a "coordinate construction", which, taken together with the strings formed ultimately from f1 and f2, is a construction of the type a 1 . This is the natural way to extend CF grammars to accommodate true coordination, as, e.g., where a string of adjectives of arbitrary length may appear in predicate position with no internal structure defined on them. Cf. Chomsky [lo]. 3.3. Let us try to relate what we have done so far to classical analysis, writing yf = yf' for any two strings f and f'if they contain exactly the same number of each of the letters (terminal or not). Clearly y extends to a mapping of our non-commutative power series onto the ring of the ordinary (commutative) formal power series with integral coefficients,and it is easily seen that q~ is a homomorphism. For example, if a = a baa, we have q~a= ya ybqJaya, and ~ O is L the ordinary power series

+

(37)

+

134

N. CHOMSKY AND M. P. SCHUTZENBERGER

+

in the ordinary variables Fa, pb. (Here if a’ = a a‘ba‘, we would also have yap= pa). Furthermore, it can be shown directly from the way our power series are obtained that the coefficients do not grow faster than an exponential function of the degree (length) of the strings. Thus the image q~ of any one of our power-series is in fact an ordinary convergent Taylor series expansion of an algebraic function. Reciprocally, if we are given (ordinary) variables Z1,..., Z,,, an (ordinary) algebraic function of this quantity? is defined by a polynomial in 9 and the Zt ;and in case 7 admits a development in Taylor series (with integral coefficients) around zero in the %’s, we can associate with it infinitely many formal power series p such that pp = jj and p is defined by formal equations. For instance: starting from the algebraic function jj of i and 6 defined by y 2 6 -9 a = 0, we obtain the two examples given above, and also formal power series

+

(38)

a

=a

+ b a a i-na-aocn

where n is an arbitrary polynomial in a and b. Thus, e.g., take Then (39)

=a a1 = a

010

3t =

b.

+ baa + ba - ab

......

etc.

3.4. Let us conclude by indicating some connections between our con-

siderations and Lyndon’s theory of equations in a free group (Lyndon, 1960). Let {xi} (1 5 i 5 n) be a terminal vocabulary, 6 a non-terminal letter and let w be a product of terms of the form 1 - xt, (1 - x4)-l, 1 - 6,(1 - [)-I. We define deg(w) = d+ -d- where d+ and d- are the number of factors 1 - 5 and (1 - E)-1 in w. Thus, for instance, for w = (1 - xz)(l - xl)(l - 6)(1 - xt)-l(l - &)-l(l -~ $ 1 , one has deg ( w ) = 1 - 1 = 0. As is well-known, the elements 1 -xi generate (by multiplication) a free group G. The relation w = 1 may be considered as an equation in the unknown 5. In our terminology a solution of w = 1 would be a power series 50 in the xt’s such that w = 1 identically when t o is substituted for 6 in w ; t o will be a group solution if, furthermore, 1 - 50 e G; i.e., if 1 - 60 is itself expressible as a product of terms (i - xt)fl. R.C. Lyndon

THE ALGEBRAIC THEORY OF CONTEXT-FREE LANGUAGES

135

has proven the very remarkable result that the totality of the group solutions can be obtained algorithmically. Let us relate part of this question to our remarks in 0 2.3. For this we introduce the new symbols &(l i i Sn), q, and equations

so that

(1 --{)-I

=

1

+

and (1 -[)-I

=

1 + [ +52

+ tq

Substituting these expressions in w = 1 and simplifying, we obtain a relation (deg (WN 5 = P' (2) where p' is a polynomial in the variables xt, &, q having no term of degree less than 2. Hence if deg ( w ) # 0 the system (l), (2) has one and only one solution in power series (the fact that the coefficients are eventually rational instead of integral numbers is irrelevant to the proof in 0 2.3) and since the group solutions are a subset of the power series solutions, we have verified directly that if deg w # 0, the free group equation w = 1 has at most one solution. On the contrary, if deg w = 0 (as for instance for the equation w = (1 - [)(I - x{)(l - 5)-1(1 - xf)-l = 1) our approach entirely collapses and says nothing even about the unrestricted solutions of w = 1. For instance (1 -xr)(l -()( 1 -xt) E (1 - [)-I = 1 has no solution if E # -1 and has an infinity of group solutions if E = -1, viz. 1 -5 = (1 - x@" (n > 0). Indeed, then, the equation can equivalently be written 5x1 = x15 (which has as solutions, in our sense, all the power series in XI). Of course, the case deg w = 0 is precisely that in which, the unknown 1 - 5 disappears when taking the commutative image as in 5 3.3 and it is the non-trivial case from a group theoretic point of view. 4.

TYPES OF

CF GRAMMARS AND THEIR GENERATIVE PROPERTIES

4.1. In terms of conditions on the rules that constitute them, we can define several categories of CF grammars that are of particular interest. In the following we will usem, B, .. for non-terminal symbols;f, g , ...for terminal

.

strings (possibly null); and v, p for arbitrary strings. Recall that we have

136

N. CHOMSKY AND M. P. SCHUTZENBERGER

excluded the possibility of rules of the forma + e or a + /?,remarking that this restriction does not affect generative capacity. We will describe CF grammars in terms of rules or equations, whichever is more convenient. If the grammar G contains no non-terminal a from which it is possible to derive both a stringy and a stringfag, then the terminal languageL(G)generated by G will be finite. In this case, G will be called apolynomialgrammar. Consider now grammatical rules of the following kinds : (40) 0)

(ii) (iii) (iv)

01

+fkl (right-linear)

a + /?f (left-linear)

a -+f/?g (linear) a +f (terminating)

A grammar containing only right-linear and terminating rules or only left-linear and terminating rules will be called a one-sided linear grammar. A grammar containing only rules of the type (40) will be called linear. Suppose that G contains only rules of the type (40) and of the type 011 -+ v, where 011 is the initial symbol of G; and that, furthermore, it contains no rule /? -+ va1y. Thus the defining equation for a1 is a1 = m, where m is a polynomial not involving L Y ~Such . a grammar will be called meta-linear. Given a grammar G (i.e., a set of positive equations) which is polynomial, one-sided linear, linear, meta-linear or context-free, we will say that the power series r which is the principle term of its solution (i.e., which it generates, in the sense defined in 9 2.3) and the language Sup r which it generates are, respectively, polynomial, one-sided linear, linear, meta-linear or context-free. These families of power-series will be designated, respectively, B+,YO+, 9+, Ym+,#+; and for each family 9the family of supports of 9will be designated Sup (9). Notice that Sup (P+) is just the family of finite sets, and that Sup ( g o + ) is the family of regular events, in the sense of Kleene [21] (cf. Chomsky, [7] - note that the class of regular events is closed under reflection). We consider now certain elementary properties of these families of languages. It is, first of all, immediate that the following inclusion relations hold among these families : (41)

SUP(B+)C S U P ( ~ O < + )Sup(Y+) < Sup(9m+) < SUP(#+).

Furthermore, in each of these cases inclusion can be strengthened to proper inclusion. Thus we have :

THE ALGEBRAIC THEORY OF CONTEXT-FREE LANGUAGES

137

PROPERTY 1.

The simplest example of a language in Sup(9+)but not in Sup(90+)is the set of all strings { a n ban} (a, b e VT).This is generated by the grammar: a = aaa b, and is easily shown not to be a regular event. The product of languages in Sup(9f) is always in Sup(pm+),but not in general in Sup(9+). The language LICof our example (1 8) above with the grammar:

+

a =a

(42)

+ baa

and consisting of the set of well-formed formulas of the implicational calculus with one free variable in Polish notation is in Sup(Y+)but not in Sup(9m+). This follows from the fact that LIC contains all strings of the form

bml aml bm2 amz... bmk amk a,

(43)

for each k 2 1, mt 2 1. But each string in LICcontains n occurrences of b and n 1 occurrences of a, for some n 2 1. Consequently, for a fixed integer k, to generate all strings of the form (43), it must be possible to derive from the initial symbol of the grammar of LICa string q~containing k occurrences of non-terminals. Consequently, this grammar cannot be metalinear. For empirical interpretation of the theory of CF grammars, the relation between Sup(Y+)and Sup(90+)is of particular importance, since a finite device incorporating the instructions of a CF grammar G generating L(G) as a representation of its intrinsic competence, will be able to interpret only the sentences of some fixedsubset R e Sup (90. of) L(G) B Sup(Y+) (with fixed supplementary aids). This relation can be described precisely in terms of certain formal features of the structural descriptions (labelled bracketings) generated by CF grammars - cf. 4 1. Let us say that G is a self-embedding grammar if it generates a structural description of the form

+

(44)

tap)tayIXl ..*,

where q~ and x contain non-null terminals, and where p is a properly bracketed expression. Then we have the following result :

138

N. CHOMSKY AND M. P. SCH~~TZENBERGER

THEOREM la. L @ PO+ fi and only fi every CF grammar thatgeneratesL is self-embedding. Chomsky [9]. This result can be extended in the following way. Define the degree of self-embedding of a structural description D as the largest N such that D contains a subconfiguration: [=(pi ...[a(p~+l](p~+z] ...](~zN+I] where each (pi contains non-null terminals. Then there is a one-one effective mapping @ of {(G,n) : G a CF grammar, n 2 I} into the set of one-sided linear grammars and a one-one effective mapping 1y of the set A of structural descriptionsinto d such that: THEOREM Ib. For each L E Sup(#+), there is a CFgrammar G generating L such that for each N , @ ( G , N ) generates f with the structural description D if and only if G generates the terminal string f with the structural description !P(D), where Y(D)has degree of self-embedding 5 N . Chomsky [8]. Thus, intuitively, we can, given G , construct a finite device @(G,N) that will recognize the structure of a string f generated by G just insofar as the degree of self-embedding of a particular structural description off does not exceed N . This fact suggests certain empirical consequences. For discussion, cf. Chomsky [101, Miller and Chomsky 1291. 4.2. We consider now various closure properties of these families of

languages. The families of power series defined above can be given the following algebraic characterization. P+is a semi-ring.l) PO+ is the smallest semiring containing P+ and closed by quasi-inversion of quasi-regular elements. 9+ is a module, and 9 m + is the smallest semi-ring containing it. The full set #+is a semi-ring closed by quasi-inversion of quasi-regular elements. Correspondingly, we have the following properties of the supports : Sup(9) is closed under set union and set product; Sup(Po+)is the smallest set containing the finite sets and closed under the operations of set union, set product, and the star operation described in 9 3.1 [21];Sup(9+) is closed under set union, but not set product; Sup(64,+) is the smallest set 1) The notion of semi-ring generalizes to that of ring in that the additive structure is only a monoid (not necessarily group) structure. A typical semi-ring is the socalled “Boolean ring” with two elements 0 and 1 and the rules ( O = o + 0 = 00 = 01 = 10; 1 = o + 1 = 1 o = 1 1 = 11).

+

+

THE ALGEBRAIC THEORY OF CONTEXT-FREE LANGUAGES

139

containing the sets of Sup(dp+) and closed under set product as well (this is, of course, the motivation behind the construction of $4,+);The full set Sup($+) is closed by union, product and the star operation. These properties are immediate, and it is natural to inquire into closure under the other elementary operations on sets, namely, intersection and complementation. It is obvious that Sup(P+) is closed under intersection, and it is well-known that the class Sup(90f) of regular events is closed under intersection and complementation. For the other families, we have the following results. The family Sup($+) of all CF languages is not closed under intersection and hence (since it is closed under union) not closed under complementation [40], [2]. The example given, in each of these references, consists of a pair of meta-linear languages whose intersection is not context-free. Hence it follows that Sup (2Zm+)is also not closed under intersection or, consequently, complementation. This result can be strengthened to cover linear grammars, in fact (for intersection) even linear grammars with a single non-terminal. To see this, consider the grammars GI and GZdefined as in (45) and (46) respectively: (45) (46)

a a

+ borc + bc = aacc + aab + ab. = aaac

GIand Gz are each linear with a single non-terminal. But the intersection of the languages that they generate is the set of strings [a2nbna2n],which is not context-free. This example (along with the fact that these families are closed under union) establishes that PROPERTY 2. Thefamilies Sup(dp+),Sup(dpm+),Sup($+) are not closed under either intersection or complementation; the intersection of two sets in one of these families may not even be in Sup($+), even when the sets in question are generated by grammars with a single non-terminal.

Presumably the complement of a language of Sup(9+) or of Sup(dpm+) is not context-free (i.e., is not a member of Sup($+)). However, we have no examples to show this. Thus of the classes of languages discussed above, only the regular events (and the finite sets) are closed under formation of intersections. However, the intersection of a regular event and a context-free grammar

140

N. CHOMSKY AND M. P. SCH~TZENBERGER

is again a context-free language [2]. We have in fact, the following stronger result which extends a well-known theorem of classical analysis due to R. Jungen [20]. THEOREM 2. Suppose that rl e lo+. Let U+be one of the families B+,lo+, A+, Am+,$+. Let r l @ r2 be the Hadamardproduct of rl, r2 (CJJ 3.2). Then rl o r2 e U+, for every r2 E U+.Furthermore, ifr2, r3 E ;lo+,then r2 o r3 E lo+.

Cf. Schiitzenberger [46]. It follows that the intersection of a language of Sup(U+) with a regular event is in Sup(U+), for each U+. The proof of this result, which is related to a similar result concerning closure under transduction, will be outlined in 3 8, below. 4.3. The category of linear grammars is of particular interest, as we will

see directly, and we will now make a few preliminary observations concerning it. Notice that if L is a language generated by a linear grammar, we can find a vocabulary V’disjoint from VT,two homomorphicmappings a, a‘ of F(V’) into F(VT), a regular event R in V’, and a finite set C C F(VT) such that L consists of exactly the strings f = a(g)c a’@), where g e R, f is the reflection of g , and c e C. Thus a finite process dealing with a collection of pairs of strings or a pair of coordinated finite processes can, in general, be correlated to a linear grammar and studied in this way. Equivalently, we can characterize a linear language in the following, slightly different way. Let V’ = V+ u V- (V+ = {vi : 0 5 i _< n } ; I/- = { V I : -n 5 i 5 -1}. Where f F(V+),let us define f as the result of substituting v-t for v{ inf, throughout. Then a linear language L is determined by choice of a homomorphic mapping !of F(V’) into F(VT), a regular event R in V+, and a finite set C C F( VT). L is now the set of strings !(f)c!(f), where f e R and c e C. We will use this alternative characterization below. We can now determine special classes of linear languages by imposing further conditions on the underlying regular event R, the mappings a, a’, and the class C. In particular, in applications below we will be concerned with the case in which R is simply a free monoid (a regular event defined by a single-state automaton) and where C contains just C E VT, where a ( f )# ycry # or’(f>. We will call grammars defined by this condition minimal linear grammars.

THE ALGEBRAIC THEORY OF CONTEXT-FREE LANGUAGES

141

A minimal linear grammar contains a single non-terminal symbol S and a single terminating rule S -+ c, and no non-terminating rule S -+ vCy.Thus each string of the language it generates has the designated “central marker” c. This is the simplest set of languages in our framework beyond the regular events, and we will see that they differ markedly from regular events in many formal properties. For later reference, we give now one particular result concerning minimal linear grammars. Let us take V’, VT = W u {c} (c 4 W), a and a‘ as above. Let G be the minimal linear grammar defined by a, a’ and generating L(G). Thus G has the defining equation

(47)

/!I = c + ~{a(v)/!Ia’(v) : v € V’}

where CX,a‘ are mappings of F( V’) into F( W). Then we have: THEOREM 3. Zfa is a monomorphism (isomorphism into), then the complement F( VT)\ L(G) of L(G) with respect to F( VT) is generated by an unambiguous linear grammar. Proof: Let A = a(V‘), F(A) = mF(V’), and for any set F C F(W), let F+ = { f F~ :f # e}. Clearly there is a partition : F( VT)\ L(G) = L U L‘, such that

(48) L‘

= F( W ) u

L = fcf’ :f E F+(A),.f’ € F( W ) ,fcf ’ # L(G); cF( W )u ((F(W )\ F(A)) cF( W )u F( VT)CF( VT)CF( VT).

But L’ is a regular event. Hence it suffices to show that L is generated by an unambiguous linear grammar. Since OL is a monomorphism, there exists an isomorphism & : F(A) -+ F(V’). We extend a’ to F+(A) by defining a’a = a’(&), for a c F+(A). Suppose that acf’ E L. Thus E F+(A),f‘ E F( W), and f’# a’a. By definition there are just three mutually exclusive possibilities for acf‘. (49) (i) f‘E F+(W)a’a (ii) a’a E F+(W)f’ (iii) a = a1a2a3 and f ’ = hwga’al (where a l , a3 e F(A); u ~ E AW ;E W ; ~ , ~ E F ( W ) ; ~ ’ U ~ E F + ( W ) ~ ; ~ ’ U ~ € (49i) is the case in which f’has a’a as a proper right factor; (49ii) is the case in which a’a has f’ as a proper right factor;

142

N. CHOMSKY AND M. P. SCHUTZENBERGER

(49iii) is the case in which a’a and f ‘ have as their common maximal right factor the string ga’al, which is a proper substring of botha‘a andf’. Thus the three cases are mutually exclusive and exhaustive, and we have a partitioning of L into the three subsets L1, Lz, L3, consisting of the strings meeting (i)-(iii), respectively. What we now have to show is that each of L1, Lz, L3 is generated by an unambiguous linear grammar. In the case of L1 and LZ this fact is obvious. Let A = 11 {a : a E A} and = { w : w E W}. Then L1 is generated by the grammar (50) and LZ by the grammar (51) (cf. 0 3.2).

w

/? = Z (a/?a‘a : a E A } + c( 1 -F 7 - 1 /? = Z{aBa’a : a E A } + (1 -A)%

(50)

(51)

Consider now the case of 4.For each a E A, let us denote by B(a) the set of all strings wg ( w E W , g E F( W)) such that &’a B F+( W ) g and a’a 4 F( W)wg. Clearly B(a) is always a finite set, since g is shorter than a’a. We can now generate L3 by the unambiguous linear grammar with the equations: (52)

Z{O/?~N’U : u E A } + Z{&b = c + c(1 - m ) - 1 + (1 -A)%

/?I =

pz

: u E A , b E B(u)} Z{(a/?zw : a E A , w E W } .

+

Verification is straightforward. But now we have given F( VT)\ L(G) as the union of the four disjoint sets L1, Lz, L3, L’, each of which has an unambiguous linear grammar. Consequently, F( VT)\ L(G) itself has an unambiguous linear grammar, as was to be proven. Notice that if we had taken a originally as an “information-lossless transduction” I431 instead of as a monomorphism, we could prove a result differing from Theorem 3 only in that the linear grammar constructed would have bounded ambiguity, rather than no ambiguity. 4.4. We have considered several subfamilies of the class of CF grammars,

classifying on the basis of structural properties of the defining rules. There are other principles of classification that might be considered. Thus, for example, it might be worthwhile to isolate the class of the star grammars (languages) characterized as follows: G is a star grammar if associated with each non-terminal at of G there is a set Zt of non-terminals and three terminal strings6, f ’ t , f ) ’ t , and G contains all and only the rules: at + f ” 4 , at +ftajf ’t (aj E Z),af + OLkCq (q, Olk, at E Ct).These are, in a sense, the most “structureless” CF grammars. The interest of

THE ALGEBRAIC THEORY OF CONTEXT-FREE LANGUAGES

143

these languages lies in the fact that the equations defining the associated power series are expressible using in an essential manner only the quasiinverse and addition, as we have observed in § 3.2. Notice, in particular, that the non-metalinear language Lrc defined by (42) is a star language. We have suggested a linguistic interpretation for the notion “star language” in 9 3.2. Another principle of classification might be in terms of the number of non-terminals in the minimal defining grammar of a certain power series. However, it does not seem likely that interesting properties of language can correlate with a measure so insensitive to structural features of grammars as this (except for the special case of the languages defined by grammars with only one non-terminal), because for monoids, as distinct from groups, the gross numerical parameters do not relate in an interesting way to the fine structure. Notice, incidentally, that for any finite N we can construct a regular event which cannot be generated by a CF grammar with less then N non-terminal symbols. Another principle of classification is suggested by consideration of dependencies among subparts of the grammar. Let us call a CF grammar irreducible if no proper subset of the set of defining equations constitutes a CF grammar (recall that terminal strings must be derivable from each non-initial non-terminal of a CF grammar) ; otherwise, reducible. If a CF grammar is reducible, in this sense, there must be proper subsets XIof its rules and & of its non-terminals, such that only rules of XIare involved in extending derivations to terminated derivations at points where symbols of C.2 appear in lines of derivations. One particular extreme form of reducibility has been studied by Ginsburg and Rice (18). Following them, let us call a CF grammar G sequential if its non-terminals can be ordered as 011, .. ,O1n (where 011 is the initial symbol) in such a way that there is no rule 1x1 +~ 0 1 j yfor j < i. The solution to a sequential grammar is particularly easy to determine by the iterative procedure described in 9 2.3 by successive elimination of variables. Concerning the family Y+of sequential grammars and the family Sup(Y+) of their supports, Ginsburg and Rice establish the following results, paralleling those mentioned above. First, it is clear that Y+, like 4+is a semi-ring closed by quasi-inversion of quasi-regular elements. Correspondingly, Sup(Y+) is closed by union, product, and the star operation. From this fact, and the fact that 8+C Y+, it follows that Sup(90+) C Sup(Y+). Furthermore, the inclusion is proper, as we can

.

144

N. CHOMSKY AND M. P. SCH~~TZENBERGER

see from the grammar (42), which, since it contains a single terminal, is sequential. In fact, we have

; Sup(Y+) ; Sup(Y+).

Sup(Lz%+)

(53)

Ginsburg and Rice show that there is no sequential grammar for the language with the vocabulary (a, b, c, d } and containing the string

(54) (which is symmetrical about c) for each sequence ( k , n l , ..., n z k - 1 ) of positive integers, although this language is generated by the grammar. a

(55)

= adpda

+ acxa + aca

p = bpb + bdadb.

There is no stronger relation than (53) between S u p ( 9 f ) and the families of Property 1, 0 4.1, however. The grammar (55) is in fact linear, though not sequential, so that S u p ( 9 + ) 6 Sup(Y+); and the grammar (42) is sequential but not meta-linear, so that Sup(Y+) 4 Sup(2Zm+). Since the grammars (45)and (46) are sequential, we see that Property 2 (but not Theorem 2 ) can be extended to Sup (9+). For further results on sequential languages, see Ginsburg and Rose [19], Shamir [51].

5. AN ALTERNATIVE CHARACTERIZATION OF FAMILIES OF CF LANGUAGES In this section we will present a rather different approach to the definition of families of languages, and we will show how it interrelates with the classification presented above. We rely here on the two fundamental notions : standard regular event and Dyck language, which we now define. A standard regular event A is given by a finite alphabet X , two subsets J1 and 5 2 of (X, X),and the rule that f E A if and only if (56) 0)

(ii)

f a xF(X) n F(X)x’, where (x, x’) a 51 f t F(X)xx‘F(X), where (x, x’) a J2.

Thus A is the set of all strings that begin and end with prescribed letters, and that contain no pair of consecutive letters belonging to J2. It is, more technically, the intersection of the quasi-ideal determined by J1 with the complement of the two-sided ideal generated by all products

145

THE ALGEBRAIC THEORY OF CONTEXT-FREE LANGUAGES

A is, in particular, what is sometimes called a “l-definite event” [21], [35]. We define the Dyck language Dzn on the 2n letters x+t (1 5 i 5 n) as the set of all stringsf which can be reduced to the empty siring by repeated cancellation of consecutive pairs of letters xjx-j (-n 5 j 5 n). The Dyck language is a very familiar mathematical object: if q~ is the homomorphism of the free monoid generated by {xkt} onto the free group generated by the subset {xt : i > 0} that satisfies identically (vzg)-l= vZ-,, then D2n is the kernel of v, that is, the set of strings f such that vf = 1. Concerning these notions, we have the following results. xx’ ((x, x‘)

E J2).

PROPOSITION 1. For any regular event B C F(Z), we can find a standard regular event A and a homomorphism m : F ( X ) -+ F(Z), such that B = &A. It is worth mentioning that this representation can be chosen in such a way that not only B = &A, but, furthermore, each string f E A has the same degree of ambiguity as the corresponding string mf E B. That is, if B = Sup@),we can find y such that A = Sup(y) and for eachf, =

-

We can generalize Proposition I to CF languages, making use of the following property of Dzn. PROPERTY 1 . Dzn is generated by an unambiguous CF grammar. To obtain an unambiguous grammar of Dzn, we introduce 2n 1 non- (1 5 i 5 n) and #?. Consider now the 2n 1 equations terminals a+

(57) (0

a{ = xi (1 -

x

+

+

mj)-lX-{

g+ 4

#? = (1 - xmp. (ii) (Cf. 9 3.2, for notation). Intuitively, #? can be interpreted as the sum of all strings that can be reduced to the empty string by successive cancellation of two consecutive letters xtx-t. Each mg is the sum of all words in Sup(#?)that begin by xt and have no proper left (or right) factor in Sup(#?).The equation (579 implies that each f E Sup(0rg) has one and only one factorization (58)

f

= Xifif2

...fmx-r

where each jj belongs to a well-defined set Sup(mj) (where j is not -i because we want the initial letter x{ to cancel only with the final letter La).

146

N. CHOMSKY AND M. P. SCHUTZENBERGER

Similarly, each f e Sup(,!?)has one and only one factorization f =f1 fm, where thef3’s belong to u i Sup(m). We now have the following result, analogous to Proposition 1.

...

PROPOSITION 2. Any CF language L C F ( 2 ) is given by an integer n, a standard regular event A on Xzn = {x.( : 1 5 i s n}, a homomorphism p7 : F(Xzn) +F(Z), and the rule L = p7 ( A nDzn). [481, [491, [111, D21. Again, as above, this statement implies that the strings are produced with the appropriate ambiguity. Furthermore, it is possible to choose J1 such that (x,x’) e J1 if x belongs to a certain subset of X (cf.[48]). Special subfamilies of languages such as those considered above can be defined by imposition of conditions on the underlying standard regular event A and the homomorphism 9. Thus suppose that we take the standard regular event A on the alphabet X u Y (where X = {x+i : 1 5 i 5 n}, Y = {y+c - : 1 5 i 5 m } defined by the following conditions on J1 and 52: (59)

{(xi, x j ) : i > 0} JZ = {(xi, x j ) : sign ( i ) # sign U)}u {(yi,y j ) : i < 0 o r j > O}u {(xi, y j ) : i < 0 or j < O } u {(yt, x j ) : i > 0 or j > O}. J1 =

Thus every string has the form fgg‘f‘, whereJf’a F ( X ) ;g , g‘ E F(Y);f, g (respectively,f‘,g’) contain only letters with positive (respectively, negative) indices. If we designate by X + and X - the subsets of X consisting of letters with positive indices and negative indices, respectively (similarly, Y+and Y-), we can describe the permitted and excluded transitions by the matrix (60), where the entry 1 (0) indicates that transition is (is not) permitted from the element labelling to row to that labelling the column, and where U is the matrix with all one’s and 0 the matrix with all zeroes. (60)

X+

X-

Y + O U O Y - 0 0 0

O

x - 0

u

Y+

Y-

u

x + u o u o 0

0

But consider now the set A n DXY (where DXY is the Dyck language

THE ALGEBRAIC THEORY OF CONTEXT-FREE LANGUAGES

147

on the alphabet X u Y). Iffg a A (wherefa F(X+ u Y+),g a F(X- u Y-)) meets the additional condition thatfg a Dxy,then g must be the mirrorimage off(up to a change of the sign of indices). That is, in the notation of the second paragraph of 4 4.3, it must be the case that g = J Clearly, if oc is a homomorphic mapping of F ( X u Y) into F( VT), then a (A n Dxy) is a linear language. Furthermore, if we add the further condition that yi = e for i < 0 and yt = c for each i > 0, where am d F( VT)CF(VT)for any i, then L = a ( A n DXY)is a minimal linear language with c as designated center symbol, and every minimal linear language is given by such a choice of a. This gives an independent characterization of minimal linear languages. Furthermore, by adding additional pairs to 52 we can delimit the defined canonical language A in such a way that {f:f a F(X+) and for some g,fg a A and g a F(Y)F(X)} is an arbitrary regular event (instead of simply the free monoid on X+, as above), so that L = oc(A n Dx-) will be an arbitrary linear language. Thus we have an independent definition of the notion “linear language”. (Notice that these further restrictions on 5 2 affect only the permitted transitions in the matrices along the main diagonal of (60). In much the same way, we can give a general definition of “metalinear language”. Thus, for example, consider the particular metalinear language generated by the grammar with the equations (61)

Er

= tilt2

51 = e

tz

=e

+ X {atlb : a,b a VT} + X {a&$ : a,b e vT}.

In this case, the matrix for the underlying standard regular event would be (62)

XI+ XI- X2+

xz-

u u

0

XI+

u

XI-

0 0 0

xz+

xz-

0 0

0

u u 0

0

u u

Any metalinear language, and only these, will be based on a standard event with a matrix of essentially this kind (with, perhaps, additional restrictions along the main diagonal). Propositions I and 2 thus provide for the possibility of very natural definitions of the full class of CF languages, and various subfamilies of this class, independently of the approach taken in preceding sections.

148

N. CHOMSKY AND M. P. SCHUTZENBERGER

6. UNDECIDABILITY 6.1. In Post [36] it is shown that the following problem, known as the Correspondence problem, is recursively unsolvable. Where Z = {(f l , g l ) , ..., (fn,gn)}is a sequence of pairs of strings, let us say that a sequence Z = ( i l , ..., im) of integers (1 5 ij 2 n ) satisfies C if

fa,

(63)

...fam

= gi,

... gp,.

The correspondence problem is the problem of determining whether, given 2, there is an index sequence that satisfies Z. Notice that either 2 is satisfied by no index sequence or else by infinitely many, since if ( i l , ..., im) satisfies C, then SO does ( i l , ..., im,il, ...., im). Post showed that there is no algorithm for determining, for arbitrary X, whether there is no index sequence satisfying Z, or whether there are infinitely many, these being the only alternatives. We can reformulate the correspondence problem directly in terms of minimal linear grammars. Given C = { ( f l , g l ) ,...,(fn,gn)}, form G( C) with the single non-terminal S and the defining equation:

(64)

S

=a

+jiSgl

+ ... + fnSgn,

where a is a symbol not in any of thefa’s or gt’s. Clearly there is an index sequence satisfying C just in case G( C) generates a string faf Or, to put it differently, let Lm be the “mirror-image’’ language consisting of all strings f a x f e F( VT), and let L ( G ( Z ) ) be the language generated by G . Then either there is no index sequence satisfying C, in which case Lm n L ( G ( C ) ) is empty; or there are infinitely many index sequences satisfying C, in which case Lm n L ( G ( 2 ) ) is infinite. From the unsolvability of the correspondence problem and the fact that Lm is generated by a linear grammar with one non-terminal, we conclude directly that:

UNDECIDABILITY THEOREM 1. There is no algorithm f o r determining, given two CF grammars G I and Gz generating L1 and LZ respectively, whether L1 n L Z is empty or infinite. This is true even where G1 and Gz are minimal linear grammars and where G I is a fixed particular grammar of Lm. The problems of emptiness or finiteness of intersection are easily seen to be solvable for one-sided linear grammars, but we see that for the simplest grammars in our framework that go beyond regular events in generative capacity, these problems are no longer solvable.

THE ALGEBRAIC THEORY OF CONTEXT-FREE LANGUAGES

149

This observation is generalized in Bar-Hillel, Perles, Shamir [2], where many problems concerning CF grammars are shown to be recursively unsolvable. In brief, their method is as follows. Let us limit VT to the set {a, 0, 1). Where = {Cfl,gl), ...,cfn,gn)} is a set of pairs of strings in the vocabulary (0, l} (i.e.,h,gd B F{O, l}), let L ( Z )be the set of all strings

x

(65) where

... 1ot1ah1...hkagj1... g51a~’l1 .. 0’2 1 5 il, ...,i k , j l , ...,j l 5 n. loth

1,

More perspicuously, let us use i = O l t as a code for the number i. Then a string of L( is formed by selecting index sequences I = (il, ...,i k ) and J = 0’1, ...,j l ) and forming

x)

(66)

- ik

-

-

... ilafr, ...hkag5, ... gjlajl ...jl.

L ( Z )now plays the same role as the language generated by (64) in the

foregoing proof of Undecidability Theorem 1. It is clearly a CF language (generated, in fact, by a meta-linear grammar which is an obvious modification of (64). But from Theorem 3, 0 4.3, above, it follows directly that the complement F(VT)\ L ( x ) of L ( Z )with respect to the vocabulary VT is a CF language, and that we can cowtruct its grammar given the grammar of L ( x ) .(Notice, in fact, that we could have used any code, in place of the particular choice i = 016, for defining L(Z)). In place of the mirror-image language Lm used in the proof of Undecidability Theorem I , let us consider the “double-mirror-image’’ language Lam consisting of all strings

(67)

xlaxzaXzaX1, where

XI

and x z are strings in (0, l}.

It is not hard to show that Lam and its complement with respect to VT are both CF languages. Observe that

(68)

L ( x )n Ldm

-

-

... ilafr, ...jikagrk... gi,ail ...ik} -

= {ik

where (il, ...,i k ) satisfies Z (that is, where&

...Ak= g4, ... gtk).

Observe also that an infinite set of strings of the form of (68) cannot constitute a CF language (nor, a fortiori, a regular event). Suppose now that there is a positive solution to the correspondence

150

N. CHOMSKY AND M. P. SCHUTZENBERGER

x.

x;

problem for that is, there is an index sequence satisfying Then, as we have observed, there are infinitely many such sequences. Consequently L ( x ) n Lam is infinite. It is therefore neither a regular event nor a CF language. Suppose, on the other hand, that there is no index sequence satisfying Then L( n Lam is empty, and is therefore both a regular event and a context-free language. But L( and Lam are CF languages; and, with fixed, we can construct their CF grammars G( and Gdm (which are, in fact, meta-linear). Thus if there were an algorithm for determining whether the intersection of the languages generated by two CF grammars GI and G2 is empty, finite, a regular event, or a CF language, this algorithm would also provide a solution to the general correspondence problem. We conclude, then :

x.

x

x)

x)

x)

UNDECIDABILITY THEOREM 2. There is no algorithm for determining, given CF grammars GI and G2, whether the intersection of the languages that they generate is empty, finite, a regular event, or a CF language - in particular, this remains true when both are meta-linear and G2 is a fixed grammar of Lam. Let C a m be the CF grammar that generates the complement Ldm (all complements now are with respect to VT) of Lam. And, given let G( be the CF grammar that generates the complement L(c) of L( as guaranteed by Theorem 3, Q 4.3. Consider now the grammar G generm u L(). Clearly G is CF and can be ting the language L(G) = L dconstructed from Gam and G ( 2 ) . But the complement L(G) of L(G) is just the set Lam u L ( x ) = Lam n L ( Z ) ,and we know by Undecidability Theorem 2 that there is no algorithm for determining, given whether this set is empty, finite, a regular event, or a CF language. But given G is determined as a CF grammar. Therefore we have:

x, x),

x)

x,

x,

UNDECIDABILITY THEOREM 3. There is no algorithm for determining, given the CF grammar G, whether the complement of the language generated by G is empty, finite, a regular event, or a CF language. There is, in particular, no general procedure for determining whether the CF grammar G generates the universal language F(VT), or whether G generates a regular event (since the complement of a regular event is a

THE ALGEBRAIC THEORY OF CONTEXT-FREE LANGUAGES

151

regular event). Consequently, there is no algorithm for determining, given CF languages L1 and L2, whether there is a transducer mapping L1 onto LZsince all and only regular languages can be obtained by transduction from the CF language F(VT) (Ginsburg and Rose, personal communication). There is, furthermore, no general method for determining whether two CF grammars are equivalent, i.e., generate the same language, since if there were such a method, it could be used to determine whether a CF grammar G is equivalent to the grammar GU generating F(VT). It also follows immediately that there is no algorithm for determining, given CF grammars, whether the language generated by one includes the language generated by the other, since this would give a solution for the equivalence problem. These results have been outlined for languages constructed from a three-element vocabulary VT,but it is clear that by appropriate recoding, they still apply to languages in a vocabulary of two or more letters. This is worked out in detail in Bar-Hillel, Perles, Shamir [2].

0 4 that finite processes involving pairs of strings receive a natural formulation in terms of linear grammars. In particular, as we have just seen, the correspondence problem can be described directly as a problem concerning minimal linear grammars. The same is true of a second combinatorial problem, also due to Post, called the “Tag problem”. We can state a generalized form of the Tag problem in the following way. Let W be the set of strings (the free monoid) in some finite vocabulary, and let P be a finite subset of non-null strings of W meeting the condition that no string of W has more than one left factor in P. That is, there are no pl, p2, w1, w2, w3 (pi E P, wt E W) such that pl # p2 and w1 = p1w2 = p 2 W 3 . Let V be the set of strings of W that have no left factor in P - that is, v E V if and only if there is no p E P such that v = pw, for w E W. Clearly V is a recursive, in fact regular, set. Let LX be a mapping of P into W (thus LX defines a set of pairs of strings @, w), where w = ap, p a P,w E W). Define a mapping T on W, where 6.2. We observed in

(69)

Tf =f’ap, iff = pf’ Tf = H,iff a V (Hd W).

Consider the problem :

(70) given a stringf, is there an integer n such that Tnf

=

H?

152

N. CHOMSKY AND M. P. SCHUTZENBERGER

Regarding T as defining the computation of a Turing machine, (70) is the haltingproblem for this Turing machine. It has been shown by Minsky [30 that (70) is a recursively unsolvable problem. The Tag problem as formulated by Post is the special case of (70), above, where T meets the following additional conditions: P is the set of all strings of length k, for some fixed k 2 2; a p depends only on the left-most symbol of p . Even with this restriction, the problem (70) is unsolvable, as Minsky has shown. This is a somewhat surprising result, because of the determinacy (monogenicity) of the generative procedure T. As a step towards reformulating the generalized Tag problem in terms of minimal linear grammars, we observe that it can be stated in the following way. Given W, P, V, a, T, as above, the question (70) has a positive answer just in case (71)

there are strings p1, ...,Pn E P and v E V such that: . .pn v =fapl apn.

PI

I . .

But we can now restate the generalized Tag problem as the following problem concerning linear grammars. Given W, P, V, LY, T, let us define the grammar G generating L(G) with the single equation

where vt E V,pi E P , and c # W is the distinguished central marker. Let us define the language M ( f ) = {fgcg : g E W }(thus M ( f ) =fLm, where L, is the “mirror-image language” defined above). Then the answer to (71) (equivalently, (70)) is positive if and only if the intersection of L(G) with M ( f ) is non-empty. Thus we see that there is no algorithm for determining whether, for fixedf, the language M u ) has a non-empty intersection with a language with a grammar meeting (72) (even for the special case in which P is the set of all strings of length k, for fixed k 2 2, and a p depends only on the left-most letter of P ) . Notice that Undecidability Theorem I , above, also follows directly from unsolvability of the Tag problem. In fact, the Correspondence and Tag problems both concern the cardinality of the intersection of a minimal linear language L with the languages M ( f ) , where f = e and L is arbitrary, for the case of the Correspondence problem, while f is arbitrary and L meets the condition (72), above, for the case of the Tag problem.

THE ALGEBRAIC THEORY OF CONTEXT-FREE LANGUAGES

153

7. AMBIGUITY 7.1. We have defined the power series r to be characteristic just in case each coefficient

{anbmcp : n = m or m = p } .

(73)

In this case, the strings of the form anbncn must have ambiguity at least two in any CF grammar generating (73) (and there is a CF grammar generating (73) in which they have ambiguity exactly two). We do not have examples illustrating the extent of inherent ambiguity in CF languages, or special types of CF languages. Notice that it is an immediate consequence of Undecidability Theorem 1 of 5 6 that there can be no algorithm for determining whether a CF grammar, or even a linear grammar, is ambiguous. Suppose in fact that, as above, Z = {(fi, gl), ..., ($n, g,)} is a sequence of pairs of strings. Select n 1 new symbols xo, ...,xn and construct the grammars Gf with the rules Sf -+ XO, Sf -+ xtSf& (15 i 5 n) and G gwith the rules Sg -+ xo, SB--f xrSg& (1 5 i 5 n). Clearly Gf and G g are unambiguous, and the Correspondence problem for Z has a positive solution if and only if there is a string generated by both Gf and G g ,that is, if and only if the grammar G f gis ambiguous, where G f gcontains the rules of G f , the rules of G g ,and the rules S + S f , S -+ Sg,where S is the initial symbol of G f p Consequently, there can be no procedure for determining, for arbitrary Z, whether the grammar G f gassociated with Z in this way is unambiguous. The grammar G f gis linear with three non-terminals and a designated central marker, and we see that for this class of grammars the ambiguity

+

N.

154

CHOMSKY A N D M. P. S C m Z E N B E R G E R

problem is unsolvable. Presumably, this remark can be generalized to grammars with two non-terminals. It is an interesting open question, however, whether the ambiguity problem remains unsolvable for minimal linear grammars. Summarizing the matter of ambiguity, as it stands at present, we have the following results : AMBIGUITY THEOREM 1. There are inherently ambiguous CF languages. AMBIGUITY THEOREM 2. There is no algorithm for determining whether a CF grammar (which may even be linear with a designated central marker) is ambiguous. 8. FINITE TRANSDUCTION

We want to describe a particularly simple family of transformations from language to language. The first and most essential one is a homomorphism. Let L be any language on a terminal vocabulary Z and assume that for each z e Z we are given a language L, on a second vocabulary X. We denote by 8L the set of all strings (in X) which can be obtained by taking a word g = zilz{, ... ztm e L, and replacing each zt5by an arbitrary ” In fact, word from L . The name ‘ ‘ h o m ~ m ~ r p h i ~ismself-explanatory. ,f5

if we consider the rings A ( 2 ) and A ( X ) of formal power series in the variables z E Z and x B X,and if we denote by 8 the homomorphism of A ( Z ) into A ( X ) that is induced by the mapping 8, = the formal power series associated with L,, then 8L is the support of the image by 8 of the formal power series associated with L. An interpretation within our previous framework can be given if L and the L,’s are CF languages. In this case, suppose that L is produced by the CF grammar G (with non-terminal vocabulary Y) and that each L, is produced by the CF grammar G, (with the set of non-terminals Yz and the initial letter yt,o). We assume that the sets Y, are disjoint and we consider a CF grammar with non-terminals Y u Z u u Y , conZ E Z

sisting of the rules of G and of the G,’s and the rules z +.yZ,o ( z e Z). (More simply we identify each z with y , , ~ ) It . is clear that G produces exactly 8L. We now generalize this construction to the following type of context

THE ALGEBRAIC THEORY OF CONTEXT-FREE LANGUAGES

155

dependency: Let Rg (i E I) and Rv (i’ E 1’) be two finite families of regular events such that every g E F(Z) belongs to one and only one member of each family. Suppose also that for each triple (z e 2, i e I, i‘ E Z’), we have in the vocabulary X. a language Then for any y = zj1zj2 ...zjk we replace each zjk by an arbitrary string from the language L(zjk, i, i’) where i and ‘i are determined by the condition that the string zj1zj2 ...zjk-l is in Rc and the string zjk+l ... zjk+l in Rt‘. It is easily proven that without loss of generality it may be assumed that for any string g belonging to some set Rgl, and for z e 2, the set Rt, which contains gz depends only upon the index il and the letter z. In other words we may assume that we are given a set of states Z,a transition mapping Z x 2 -+ Z and an initial state io e Z such that zj1zj2 ... zjk-l e Rr if and only if i is the state reached from io after reading zjl zj2 ... zjkW1. A similar construction applies to Rt, and for the sake of clarity we write the corresponding mapping as a left multiplication. Given the two mappings I x Z + I and Z x 1’, we denote by ug, for each g = zjl z j 2 . .. zjm E F(X), the sequence of triples Lt,g,{g

(ii, zjl,

(74)

i’m)

(iz, zj2, i’m-1)

... ( i k , Z j k , i’m-k+i) ... (im,zjm, i’i)

where inductively (75)

i 2 = i1zjl, i3 = i2zj2, .I

1 2

.I

.I

...,im = i m - l z j m ,

i k = ik-izjk-l

and

...,

= zjml 1 , 1 3 = ~ j ~ - ~ i ’ Z , i’m = z j 2 i ‘ m - l .

With these notations the transformation we have been describing can be considered as consisting of two steps: (76) (i)

replacement of every g E L by the string ug = (h,zj1,i m) i’) in an alphabet U consisting of triples (i,z, i’);

...

(im,Zjm,

(ii) replacement in ug of every triple (ik,Zjk,i‘m-k+l) by an arbitrary string from the language L(zjk,i k , i’m-k+l). Since step 2 is only a homomorphism, it is sufficient to discuss step 1. For this let U denote the set of all triples (i, z, i’) and consider the language L’ obtained from L by adding to its grammar all the rules zj -+ (i, zj, i‘) with i E I, i’ B I’ arbitrary). Clearly a string of L‘ belongs to the set {ug : g E L } if and only if it satisfies the condition (75) above, or, in other words, if it belongs to the

156

N. CHOMSKY AND M. P. SCH~TZENBERGER

regular event a determined by the condition (75) on the set F(U)of all strings in the alphabet U. Hence step 1 consists only of a homomorphism from L in to the set of all strings on U (which gives L’)followed by the intersection of L‘ with a regular event. Let us now give a final interpretation of what we have done: For each z d 2, let pz denote a matrix whose rows and columns are indexed by pairs (i # I, i’ 4 1’)and whose entries are as follows

(77) pz(z,u)(v p‘)= the triple (i, z, i”’) if i” = 0 otherwise.

= iz

and i’

= zi”’

Then if we compute

...pzjm = pg, it is easily verified that the entry (i, iTm)(im7

i’l)

of

a

pg is precisely ug. From this it follows easily that {og : g 4 L} = L‘ n is also a context-free language. Indeed, p is a homomorphism - we

replace every non-terminal y by a matrix p y whose entries are new nonterminals and we verify that p commutes with the substitutions used for defining the language as the solution of a system of equations. On the other hand identifying the entries one by one in the image p of our equations gives a new set of equations of the usual type that exactly defines L’n a [46]. More simply still we can define p’ as above except that for each non-zero entry we take the formal power series associated with L(q,i, i’), instead of the triple (il, zj, i’). Then the two steps of the construction are telescoped in a single one and the power series associated with the language (on X) obtained by our transformation is simply an entry of

Z{p‘ g : g € L}. This is the basis for the proof of Theorem 2, 9 4, above. 9. CONNECTIONS WITH THE THEORY OF AUTOMATA

We have so far been studying generative processes, the languages and systems of structural descriptions that they define, and finitary mappings on these languages from a completely abstract point of view. To relate these remarks to the theory of automata, it is convenient to introduce a temporal asymmetry into consideration. An automaton M can be regarded as a device consisting of a set of states Z (the memory of M) that accepts (equivalently, produces) a

THE ALGEBRAIC THEORY OF CONTEXT-FREE LANGUAGES

157

sequence of symbols from a vocabulary (alphabet) V in accordance with iixed, finitely statable instructions (which can be given by associating with each v E V a mapping qv of into itself (or into the set of subsets of 2, in the case of a “non-deterministic” automaton). If we designate an initial state and a set of final states, we define a language M(L) consisting of the strings that can be accepted by M as it proceeds in accordance with its instructions from the initial state to a final state, proceeding from s E 22 to S‘ E Z on accepting v just in case yV(s)= S‘ (or S‘ e qV(s), in the non-deterministic case). The size of memory of M, or its rate of growth in the course of computation, provides a certain index of the richness of the language L(M) in terms of which we can compare various families of languages of the kinds we have considered. Given a set of strings L, let us writef- f ’ just in case for all g, f g L if and only iff’g E L. Clearly ’L is an equivalence. Furthermore, it is clear that we can take the equivalence classes defined by ‘L as states of an automaton M(L) that accepts L, since all of the information about f relevant to the further computation of M(L), once it has read f, is given by the equivalence class to which f belongs. Notice that L is the union of certain of these equivalence classes, and that f ’Lf ’ implies that fg-f’g, for all g . Secondly, given L let us write f =f ’ if and only if for all g , g f ’L gf ‘. Clearlyf =f ’ if and only if for all g,g’, gfg’ if and only if gf’g’. Thus = is symmetrical, and it is easy to show thatfi 3f2 andf3 ~ f 4 Thus . 3 is a congruence relation, and the =-classes in the set F( V ) can be multiplied together giving a quotient monoid of F(V). This quotient monoid F’(V) = qF( V ) is such that L = q-lqL - and is canonically associated with LO[41]. This observation relates the present theory to the theory of monoids. The interest of this is that in certain cases, the %-classes (and the quotient monoid) have a simple interpretation that can be translated into the language of automata, and, conversely, that certain algebraic notions (in particular, that of extension), receive a simple interpretation. Returning now to the problem of characterizing families of languages in terms of automata, it is well-known that the sub-family Sup(de+) of CF languages is uniquely characterized by the fact that for each language L E Sup(90+),there is an automaton M(L) with bounded memory that accepts L. Consider now the family Sup(90), that is, the set of supports of power series that are the solutions to systems of “one-sided linear” equations

x

158

N. CHOMSKY AND M. P. S C H ~ Z E N B E R G E R

with positive or negative integral coefficients. As we have observed, L e Sup(90) if and only if L = Sup(r1- r2), where r ~r2, 4 (Lo+).It can now be shown that the following statements are equivalent: (79) (i) L e Sup(20+); (ii) there is a one-one correspondence between the %-classes for L and a finite dimensional space of integral vectors v ( f ) such that for each x e V, v(fx) = v(f)px, where px is a matrix; (iii) F / - is isomorphic to a monoid of finite dimensional integral matrices (i.e., the matrices p of ii); (iv) L is accepted by an automaton M(L) with a finite dimensional space of vectors with integral coordinates as memory and transitions as above in ii. (Schutzenberger, [44] -let the class d of automata be those defined by (79iv)). Consider now the following two restrictions on the class d o f automata. (80) (i) there is an N such that, for allfe F(V), IIv(f)l (ii) for all f,f’,f” e F( V ) and E > 0, lim e-&n [Iv(f%f”)lI =0

I