The Ergodic Theory of Discrete Sample Paths (Graduate Studies in Mathematics 13)

The Ergodic Theory of Discrete Sample Paths

Paul C. Shields

Graduate Studies in Mathematics Volume 13

American Mathematical Society

Editorial Board James E. Humphreys David Sattinger Julius L. Shaneson Lance W. Small, chair 1991 Mathematics Subject Classification. Primary 28D20, 28D05, 94A17; Secondary 60F05, 60G17, 94A24. ABSTRACT. This book is about finite-alphabet stationary processes, which are important in physics, engineering, and data compression. The book is designed for use in graduate courses, seminars or self study for students or faculty with some background in measure theory and probability theory.

Library of Congress Cataloging-in-Publication Data

Shields, Paul C. The ergodic theory of discrete sample paths / Paul C. Shields. p. cm. — (Graduate studies in mathematics, ISSN 1065-7339; v. 13) Includes bibliographical references and index. ISBN 0-8218-0477-4 (alk. paper) 1. Ergodic theory. 2. Measure-preserving transformations. 3. Stochastic processes. I. Title. II. Series. QA313.555 1996 96-20186 519.2'32—dc20 CIP

Copying and reprinting. Individual readers of this publication, and nonprofit libraries acting

for them, are permitted to make fair use of the material, such as to copy a chapter for use in teaching or research. Permission is granted to quote brief passages from this publication in reviews, provided the customary acknowledgment of the source is given. Republication, systematic copying, or multiple reproduction of any material in this publication (including abstracts) is permitted only under license from the American Mathematical Society. Requests for such permission should be addressed to the Assistant to the Publisher, American Mathematical Society, P.O. Box 6248, Providence, Rhode Island 02940-6248. Requests can also be made by e-mail to reprint-permission0ams.org .

C) Copyright 1996 by the American Mathematical Society. All rights reserved. The American Mathematical Society retains all rights except those granted to the United States Government. Printed in the United States of America. ® The paper used in this book is acid-free and falls within the guidelines established to ensure permanence and durability. .. Printed on recycled paper. t.,1 10 9 8 7 6 5 4 3 2 1

01 00 99 98 97 96

Contents ix

Preface I

1 1 13 33 43 51 56 66 79 87 103

Basic concepts. I.1 Stationary processes. 1.2 The ergodic theory model. 1.3 The ergodic theorem. 1.4 Frequencies of finite blocks 1.5 The entropy theorem. 1.6 Entropy as expected value 1.7 Interpretations of entropy 1.8 Stationary coding 1.9 Process topologies. 1.10 Cutting and stacking.

II Entropy-related properties. II.! Entropy and coding 11.2 The Lempel-Ziv algorithm 11.3 Empirical entropy 11.4 Partitions of sample paths. 11.5 Entropy and recurrence times

121 121 131 137 147 154

III Entropy for restricted classes. Ill.! Rates of convergence 111.2 Entropy and joint distributions. 111.3 The d-admissibility problem. 111.4 Blowing-up properties. 111.5 The waiting-time problem

165 165 174 184 194 200

IV B-processes. IV1 Almost block-independence. 1V.2 The finitely determined property. 1V.3 Other B-process characterizations.

211 211 221 232

Bibliography

239

Index

245

vii

Preface This book is about finite-alphabet stationary processes, which are important in physics, engineering, and data compression. The book is designed for use in graduate courses, seminars or self study for students or faculty with some background in measure theory and probability theory. The focus is on the combinatorial properties of typical finite sample paths drawn from a stationary, ergodic process. A primary goal, only partially realized, is to develop a theory based directly on sample path arguments, with minimal appeals to the probability formalism. A secondary goal is to give a careful presentation of the many models for stationary finite-alphabet processes that have been developed in probability theory, ergodic theory, and information theory. The two basic tools for a sample path theory are a packing lemma, which shows how "almost" packings of integer intervals can be extracted from coverings by overlapping subintervals, and a counting lemma, which bounds the number of n-sequences that can be partitioned into long blocks subject to the condition that most of them are drawn from collections of known size. These two simple ideas, introduced by Ornstein and Weiss in 1980, immediately yield the two fundamental theorems of ergodic theory, namely, the ergodic theorem of Birkhoff and the entropy theorem of Shannon, McMillan, and Breiman. The packing and counting ideas yield more than these two classical results, however, for in combination with the ergodic and entropy theorems and further simple combinatorial ideas they provide powerful tools for the study of sample paths. Much of Chapter I and all of Chapter II are devoted to the development of these ideas. The classical process models are based on independence ideas and include the i.i.d. processes, Markov chains, instantaneous functions of Markov chains, and renewal and regenerative processes. An important and simple class of such models is the class of concatenated-block processes, that is, the processes obtained by independently concatenating fixed-length blocks according to some block distribution and randomizing the start. Related models are obtained by block coding and randomizing the start, or by stationary coding, an extension of the instantaneous function concept which allows the function to depend on both past and future. All these models and more are introduced in the first two sections of Chapter I. Further models, including the weak Bernoulli processes and the important class of stationary codings of i.i.d. processes, are discussed in Chapter III and Chapter IV. Of particular note in the discussion of process models is how ergodic theorists think of a stationary process, namely, as a measure-preserving transformation on a probability space, together with a partition of the space. This point of view, introduced in Section 1.2, leads directly to Kakutani's simple geometric representation of a process in terms of a recurrent event, a representation that not only simplifies the discussion of stationary renewal and regenerative processes but generalizes these concepts to the case where times between recurrences are not assumed to be independent, but only stationary. A ix

x

PREFACE

further generalization, given in Section 1.10, leads to a powerful method for constructing examples known as cutting and stacking. The book has four chapters. The first chapter, which is half the book, is devoted to the basic tools, including the Kolmogorov and ergodic theory models for a process, the ergodic theorem and its connection with empirical distributions, the entropy theorem and its interpretations, a method for converting block codes to stationary codes, the weak topology and the even more important d-metric topology, and the cutting and stacking method. Properties related to entropy which hold for every ergodic process are discussed in Chapter II. These include entropy as the almost-sure bound on per-symbol compression, Ziv's proof of asymptotic optimality of the Lempel-Ziv algorithm via his interesting concept of individual sequence entropy, the relation between entropy and partitions of sample paths into fixed-length blocks, or partitions into distinct blocks, or partitions into repeated blocks, and the connection between entropy and recurrence times and entropy and the growth of prefix trees. Properties related to entropy which hold only for restricted classes of processes are discussed in Chapter III, including rates of convergence for frequencies and entropy, the estimation of joint distributions in both the variational metric and the d-metric, a connection between entropy and a-neighborhoods, and a connection between entropy and waiting times. Several characterizations of the class of stationary codings of i.i.d. processes are given in Chapter IV, including the almost block-independence, finitely determined, very weak Bernoulli, and blowing-up characterizations. Some of these date back to the original work of Ornstein and others on the isomorphism problem for Bernoulli shifts, although the almost block-independence and blowing-up ideas are more recent. This book is an outgrowth of the lectures I gave each fall in Budapest from 1989 through 1994, both as special lectures and seminars at the Mathematics Institute of the Hungarian Academy of Sciences and as courses given in the Probability Department of E6tviis Lordnd University. In addition, lectures on parts of the penultimate draft of the book were presented at the Technical University of Delft in the fall of 1995. The audiences included ergodic theorists, information theorists, and probabilists, as well as combinatorialists and people from engineering and other mathematics disciplines, ranging from undergraduate and graduate students through post-docs and junior faculty to senior professors and researchers. Many standard topics from ergodic theory are omitted or given only cursory treatment, in part because the book is already too long and in part because they are not close to the central focus of this book. These topics include topological dynamics, smooth dynamics, random fields, K-processes, combinatorial number theory, general ergodic theorems, and continuous time and/or space theory. Likewise little or nothing is said about such standard information theory topics as rate distortion theory, divergence-rate theory, channel theory, redundancy, algebraic coding, and multi-user theory. Some specific stylistic guidelines were followed in writing this book. Proofs are sketched first, then given in complete detail. With a few exceptions, the sections in Chapters II, III, and IV are approximately independent of each other, conditioned on the material in Chapter I. Theorems and lemmas are given names that include some information about content, for example, the entropy theorem rather than the ShannonMcMillan-Breiman theorem. Likewise, suggestive names are used for concepts, such as

PREFACE

xi

building blocks (a name proposed by Zoli Gytirfi) and column structures (as opposed to gadgets.) Also numbered displays are often (informally) given names similar to those used for LATEX labels. Only those references that seem directly related to the topics discussed are included. Exercises that extend the ideas are given at the end of most sections; these range in difficulty from quite easy to quite hard. I am indebted to many people for assistance with this project. Imre Csiszdr and Katalin Marton not only attended most of my lectures but critically read parts of the manuscript at all stages of its development and discussed many aspects of the book with me. Much of the material in Chapter III as well as the blowing-up ideas in Chapter IV are the result of joint work with Kati. It was Imre's suggestion that led me to include the discussions of renewal and regenerative processes, and his criticisms led to many revisions of the cutting and stacking discussion. In addition I had numerous helpful conversations with Benjy Weiss, Don Ornstein, Aaron Wyner, and Jacob Ziv. Others who contributed ideas and/or read parts of the manuscript include Gdbor TusnAdy, Bob Burton, Jacek Serafin, Dave Neuhoff, Gyiirgy Michaletzky, Gusztdv Morvai, and Nancy Morrison. Last, but far from least, I am much indebted to my two Toledo graduate students, Shaogang Xu and Xuehong Li, who learned ergodic theory by carefully reading almost all of the manuscript at each stage of its development, in the process discovering numerous errors and poor ways of saying things. No project such as this can be free from errors and incompleteness. A list of errata as well as a forum for discussion will be available on the Internet at the following web address. http://www.math.utoledo.edurpshields/ergodic.html I am grateful to the Mathematics Institute of the Hungarian Academy for providing me with many years of space in a comfortable and stimulating environment as well as to the Institute and to the Probability Department of Etitvotis Lordnd University for the many lecture opportunities. My initial lectures in 1989 were supported by a Fulbright lectureship. Much of the work for this project was supported by NSF grants DMS8742630 and DMS-9024240 and by a joint NSF-Hungarian Academy grant MTA-NSF project 37. This book is dedicated to my son, Jeffrey. Paul C. Shields Toledo, Ohio March 22, 1996

Chapter I Basic concepts. Section 1.1

Stationary processes.

of random A (discrete-time, stochastic) process is a sequence X1, X2, ... , X, variables defined on a probability space (X, E, p). The process has alphabet A if the range of each X i is contained in A. In this book the focus is on finite-alphabet processes, so, unless stated otherwise, "process" means a discrete-time finite-alphabet process. Also, unless it is clear from the context or explicitly stated stated otherwise, "measure" will mean "probability measure" and "function" will mean "measurable function" with respect to some appropriate a-algebra on a probability space. The cardinality of a finite set A is denoted by IA I. The sequence a m , am+i, • • • , an, where each ai E A, is denoted by amn . The set of all such amn is denoted by Al, except for m = 1, when An is used. The k-th order joint distribution of the process {X k } is the measure p.k on A k defined by the formula p.,k (alic) = Prob(Xif = c4), alic E Ak . ...

When no confusion will result the subscript k on p.k may be omitted. The set of joint distributions {Ak: k > 1} is called the distribution of the process. The distribution of a process can, of course, also be defined by specifying the start distribution, A i , and the successive conditional distributions k -1 k- 1 k- 1\ (akla 1 ) = Prob(Xk = ak1X 1 = al ) =

til(alk)

ktk-1(a

1) .

The distribution of a process is thus a family of probability distributions, one for each k. The sequence cannot be completely arbitrary, however, for implicit in the definition of process is that the following consistency condition must hold for each k > 1, (1)

111(4) = E P4+1 (a jic+1 ),

al E A k .

ak-1-1

A process is considered to be defined by its joint distributions, that is, the particular space on which the functions Xn are defined is not important; all that really matters in probability theory is the distribution of the process. Thus one is free to choose the underlying space (X, E, p) on which the Xn are defined in any convenient manner, as 1

CHAPTER I. BASIC CONCEPTS.

2

long as the the joint distributions are left unchanged. An important instance of this idea is the Kolmogorov model for a process, which represents it as the sequence of coordinate functions on the space of infinite sequences drawn from the alphabet A, equipped with a Borel measure constructed from the consistency conditions (1). The rigorous construction of the Kolmogorov representation is carried out as follows. Let Aœ denote the set of all infinite sequences X = 1Xil,

Xi

E

A, 1 < i < co.

The cylinder set determined by amn , denoted by [4], is the subset of A' defined by

[4] = {x: x, = ai , m < i < n} . Let C, be the collection of cylinder sets defined by sequences that belong to An, let C = Un C„, and let R. = 'R,,(C,) denote the ring generated by Cn . The sequence {R,} is increasing, and its union R = R(C) is the ring generated by all the cylinder sets. Two important properties are summarized in the following lemma. Lemma 1.1.1

(a) Each set in R, is a finite disjoint union of cylinder sets from C,. (b) If {B„} c R is a decreasing sequence of sets with empty intersection then there is an N such that B„ = 0, n > N.

Proof The proof of (a) is left as an exercise. Part (b) is an application of the finite intersection property for sequences of compact sets, since the space A is compact in the product topology and each set in R. is closed, from (a). Thus the lemma is established. Let E be the a-algebra generated by the cylinder sets C. The collection E can also be defined as the the a -algebra generated by R, or by the compact sets. The members of E are commonly called the Borel sets of A. A is defined by kn (x). xn , x E For each n > 1 the coordinate function in : A' representation theorem states that every process with alphabet A A. The Kolmogorov can be thought of as the coordinate function process {i n }, together with a Borel measure on A". Theorem 1.1.2 (Kolmogorov representation theorem.) If {p k } is a sequence of measures for which the consistency conditions (1) hold, then there is a unique Borel probability measure p. on A" such that, p([4]) = pk (alic), for each k and each 4. In other words, if {X,} is a process with finite alphabet A, there is a unique Borel measure p, on A' for which the sequence of coordinate functions fin } has the same distribution as {X J.

Proof The process {X,} defines a set function p, on the collection C of cylinder sets by the formula (2) p({41) = Prob(X, = a i , 1 _< i < n). The consistency conditions (1) together with Lemma I.1.1(a), imply that p. extends to a finitely additive set function on the ring R. = R(C) generated by the cylinder sets. The finite intersection property, Lemma I.1.1(b), implies that p can be extended to a

3

SECTION Ll. STATIONARY PROCESSES.

unique countably additive measure on the a-algebra E generated by R. Equation (2) translates into the statement that { 5-C,} and {X,} have the saine joint distribution. This proves Theorem 1.1.2. The sequence of coordinate functions { k - n } on the probability space (A', E, it) will be called the Kolmogorov representation of the process {U. The measure will be called the Kolmogorov measure of the process, or the Kolmogorov measure of the sequence Lu l l. Process and measure language will often be used interchangeably; for example, "let be a process," means "let i be the Kolmogorov measure for some process {Xn } ." As noted earlier, a process is considered to be defined by its joint distributions, that is, the particular space on which the functions X„ are defined is not important. The Kolmogorov model is simply one particular way to define a space and a sequence of functions with the given distributions. Another useful model is the complete measure model, that is, one in which subsets of sets of measure 0 are measurable. The Kolmogorov measure on A extends to a complete measure on the completion t of the Borel sets E relative to tt. Furthermore, completion has no effect on joint distributions, for Aga in) = ictk (alic) for all k > 1 and all a, and uniqueness is preserved, that is, different processes have Kolmogorov measures with different completions. Many ideas are easier to express and many results are easier to establish in the framework of complete measures; thus, whenever it is convenient to do so the complete Kolmogorov model will be used in this book, though often without explicitly saying so. In particular, "Kolmogorov measure" is taken to refer to either the measure defined by Theorem 1.1.2, or to its completion, whichever is appropriate to the context. A process is stationary, if the joint distributions do not depend on the choice of time origin, that is,

(3)

Prob(Xi = ai , ni <j < n) = Prob(Xi+i = ai , ni <j < n),

for all m, n, and affin . Stationary finite-alphabet processes serve as models in many settings of interest, including physics, data transmission and storage, and statistics. The phrase "stationary process" will mean a stationary, discrete-time, finite-alphabet process, unless stated otherwise. The statement that a process {X n } with finite alphabet A is stationary translates into the statement that the Kolmogorov measure tt is invariant under the shift transformation T. The (left) shift T: A" 1-*- A" is defined by

(Tx) n= xn+i, x

E

A, n> 1,

which is expressed in symbolic form by

x = (xi, x2, ...) == T x = (x2, x3, . .). The shift transformation is continuous relative to the product topology on A", and defines a set transformation T -1 by the formula, T -1 B = fx: T x

E

BI,

B

CHAPTER I. BASIC CONCEPTS.

4 Note that

T-1 r„n 1 _ ri,n+1 i

L"m-1 — Lum+1.1 ,

bi+1 =

ai, m <j < n,

so that the set transformation T -1 maps a cylinder set onto a cylinder set. It follows that T -1 B is a Borel set for each Borel set B, which means that T is a Borel measurable mapping. The condition that the process be stationary is just the condition that p,([4]) = p.([b 11 ]), that is, A(B) = p(T -1 B) for each cylinder set B; which is, in turn, equivalent to the condition that p(B) = p(T -1 B) for each Borel set B. The condition,

p(B) = p(T -1 B), B E E, is usually summarized by saying that T preserves the measure iu or, alternatively, that ti. is T -invariant. In summary, the shift T is Borel measurable. It preserves a Borel probability measure A if and only if the sequence of coordinate functions {:kn } is a stationary process on the probability space (A", E, p.). The intuitive idea of stationarity is that the mechanism generating the random variables doesn't change in time. It is often convenient to think of a process has having run for a long time before the start of observation, so that only nontransient effects, that is, those that do not depend on the time origin, are seen. Such a model is well interpreted by another model, called the doubly-infinite sequence model or two-sided model. Let Z denote the set of integers and let A z denote the set of doubly-infinite A-valued sequences, that is, the set of all sequences x = {x„}, where each xn E A and n E Z ranges from —oo to oo. The shift T: A z 1-- A z is defined by (Tx) n= xn+i, n E Z. The concept of cylinder set extends immediately to this new setting. Furthermore, if {p4} is a set of measures for which the consistency conditions (1) and the stationarity conditions (3) hold then there is a unique T-invariant Borel measure ii, on A z such that Again) = pk (a1 ), for each k > 1 and each all'. The projection of p, onto il°'° is, of course, the Kolmogorov measure on A' for the process defined by {pa In summary, for stationary processes there are two standard representations, one as the Kolmogorov measure p, on A", the other as the shift-invariant extension to the space Az. The latter model will be called the two-sided Kolmogorov model of a stationary process. Note that the shift T for the two-sided model is invertible, that is, T is one-to-one and onto and T -1 is measure preserving. (The two-sided shift is, in fact, a homeomorphism on the compact space A z .) From a physical point of view it does not matter in the stationary case whether the one-sided or two-sided model is used. Proofs are sometimes simpler if the two-sided model is used, for invertibility of the shift on A z makes things easier; such results will then apply to the one-sided model in the stationary case. In some cases where such a simplification is possible, only the proof for the invertible case will be given. In the stationary case, "Kolmogorov measure" is taken to refer to either the one-sided measure or its completion, or to the two-sided measure or its completion, as appropriate to the context. Remark 1.1.3 Much of the success of modern probability theory is based on use of the Kolmogorov model, for asymptotic properties can often be easily formulated in terms of subsets of the infinite product space A". In addition, the model provides a concrete uniform setting for all processes; different processes with the same alphabet A are distinguished by having

5

SECTION 1.1. STATIONARY PROCESSES.

different Kolmogorov measures on A. When completeness is needed the complete model can be used, and, in the stationary case, when invertibility is useful the two-sided model on A z can be used. As will be shown in later sections, even more can be gained in the stationary case by retaining the abstract concept of a stationary process as a sequence of random functions with a specified joint distribution on an arbitrary probability space. Remark 1.1.4 While this book is primarily concerned with finite-alphabet processes, it is sometimes desirable to consider countable-alphabet processes, or even continuous-alphabet processes. For example, return-time processes associated with finite-alphabet processes are, except in trivial cases, countable-alphabet processes, and functions of uniformly distributed i.i.d. processes will be useful in Chapter 4. The Kolmogorov theorem extends easily to the countable case, as well as to the continuous-alphabet i.i.d. cases that will be considered.

I.1.a

Examples.

Some standard examples of finite-alphabet processes will now be presented. Example 1.1.5 (Independent processes.) The simplest example of a stationary finite-alphabet process is an independent, identically distributed (i.i.d.) process. A sequence of random variables {X„} is independent if Prob(X, = an IX7 -1 = ar I ) = Prob(X„ = an ), holds for all n > 1 and all al' E An. It is identically distributed if Prob(X„ = a) = Prob(Xi = a) holds for all n > 1 and all a E A. An independent process is stationary if and only if it is identically distributed. The i.i.d. condition holds if and only if the product formula k

k

iak (al

)=

n Ai (ai), i= 1

holds. Such a measure is called a product measure defined by Ai. Thus a process is i.i.d. if and only if its Kolmogorov measure is the product measure defined by some distribution on A. Example 1.1.6 (Markov chains.) The simplest examples of finite-alphabet dependent processes are the Markov processes. A sequence of random variables, {X„}, is a Markov chain if (4)

Prob (X n = an I VI -1 = ar l ) = Prob(Xn = an IXn _i =a_1)

holds for all n > 2 and all a7 E An. A Markov chain is said to be homogeneous or to have stationary transitions if Prob(X n = blXn _i = a) does not depend on n, in which case the I AI x IAI matrix M defined by Mab = Mbia = Prob(X„

= bi X„_i = a)

CHAPTER I. BASIC CONCEPTS.

6

is called the transition matrix of the chain. The start distribution is the (row) vector lit = Lai (s)}, where /2i(a) = Prob(X i = a), a E A. The start distribution is called a stationary distribution of a homogeneous chain if the matrix product A I M is equal to the row vector pl. If this holds, then a direct calculation shows that, indeed, the process is stationary. The condition that a Markov chain be a stationary process is that it have stationary transitions and that its start distribution is a stationary distribution. Unless stated otherwise, "stationary Markov process" will mean a Markov chain with stationary transitions for which A I M = tti and, for which iu 1 (a) > 0 for all a E A. The positivity condition rules out transient states and is included so as to avoid trivialities. Example 1.1.7 (Multistep Markov chains.) A generalization of the Markov property allows dependence on k steps in the past. A process is called k-step Markov if (5)

Prob(X, . anix ri .4-1 ) . Prob(X, = an i Xnn :ki = anniki )

holds for all n > k and for all dii. The two conditions for stationarity are, first, that transitions be stationary, that is, A (an I a n

_ ) = Prob(X n = anl, ilv n—i n—k —

n-1\ an—k )

does not depend on n as long as n > k, and, second, that A k M (k) = ,uk , where ii.k (4`) = Prob(X i` = 4), 4 E A " , and M (k) is the I BI x IBI matrix defined by Ai

(k) _

1 r1 ak

bk — l' 1

Prob(Xk ±i = bk I X ki = a k\ il , 0

i• f uLk—I i = a2k otherwise,

and B = fa: p,k (4c) > 01. Note that probabilities for a k-th order stationary chain can be calculated by thinking of it as a first-order stationary chain with alphabet B. Example 1.1.8 (Finite-state processes.) Let A and S be finite sets. An A-valued process {Y,} is said to be a finite-state process with state space S, if (6)

Prob(sn = s, Yn = bisrl ,1T -1 ) = Prob(sn = s, Y,

= biSn-1),

for all n > 1. In other words, {Yn } is a finite-state process if there is a finite-alphabet process {s n } such that the pair process {X, = (sa , Yn )} is a Markov chain. Note that the finite-state process {Yn } is stationary if the chain {X„} is stationary, but {Yn } is not, in general, a Markov chain, see Exercise 2. The Markov chain {X,} is often referred to as the underlying Markov process or hidden Markov process associated with {Yn }. In ergodic theory, any finite-alphabet process is called a finite-state process, and what is here called a finite-state process is called a function (or lumping) of a Markov chain. In information theory, what is here called a finite-state process is sometimes called a Markov source, following the terminology used in [15 ]. Here "Markov" will be reserved for processes that satisfy the Markov property (4) or its more general form (5), and "finite-state process" will be reserved for processes satisfying property (6).

7

SECTION 1.1. STATIONARY PROCESSES.

Example 1.1.9 (Stationary coding.) The concept of finite-state process can be generalized to allow dependence on the past and future, an idea most easily expressed in terms of the two-sided representation of a stationary process. Suppose A and B are finite sets. A Borel measurable mapping F: Az 1-- Bz is a stationary coder if F(TA X) = TB F(x), Vx

Az ,

E

where TA and TB denotes the shifts on the respective two-sided sequence spaces Az and B z . The coder F transports a Borel measure IL on Az into the measure v = it o F -1 , defined for Borel subsets C of Bz by

v(C) = The encoded process v is said to be a stationary coding of ti.„ with coder F. Note that a stationary coding of a stationary process is automatically a stationary process. Define the function f: Az i- B by the formula f (x) = F(x)o, that is, f (x) is just the 0-th coordinate of y = F (x). The function f is called the time-zero coder associated with F. The stationary coder F and its time-zero coder f are connected by the formula = f (rx), x

E

AZ, n

E

Z,

that is, the image y = F(x) is the sequence defined by y, = f (T:14 x). Thus the stationary coding idea can be expressed in terms of the sequence-to-sequence coder F, sometimes called the full coder, or in terms of the sequence-to-symbol coder f, otherwise known as the per-symbol or time-zero coder. The word "coder" or "encoder" may refer to either F or f. A case of particular interest occurs when the time-zero coder f depends on only a finite number of coordinates of the input sequence x, that is, when there is a nonnegative integer w such that = f (x) = f CO . In this case it is common practice to write f (x) = f (xw w ) and say that f (or its associated full coder F) is a finite coder with window half-width w. The encoded sequence y = F(x) is defined by the formula

- ), n Yn = f (x',71 Z

E

Z,

that is, the value y, depends only on xn and the values of its w past and future neighbors. In the special case when w = 0 the encoding is said to be instantaneous, and the encoded process {Yn } defined by Yn = f (X,) is called an instantaneous function or simply a function of the {X n } process. Finite coding is sometimes called sliding-block or sliding-window coding, for one can think of the coding as being done by sliding a window of width 2w ± 1 along x. At time n the window will contain Xn—w, Xn—w+1,

• • • 9

Xn, • • •

1

Xn-f-w

which determines the value y,, through the function f. To determine yn+1 , the window is shifted to the right (that is, the contents of the window are shifted to the left) eliminating xn _ u, and bringing in

CHAPTER I. BASIC CONCEPTS.

8

The key property of finite codes is that the inverse image of a cylinder set of length n is a finite union of cylinder sets of length n 2w, where w is the window half-width of the code. Such a finite coder f defines a mapping, also denoted by f, from An,n41, to B,7, by the formula f (x,nn+-ww ) = ymn , where yi = f

If F: Az

m <j < n.

B z is the full coder defined by f, then F' ([y]) =

[x:t ww], f (x;nitww)=Y ■11

n ,v ) so that, in particular, F -1 ([), 1) is measurable with respect to the a-algebra E(X,n+w generated by the random variables X„,,, Xn,„ + i, , Xn±„,. Furthermore, for any measure ,u, cylinder set probabilities for the encoded measure y = o F -1 are given by the formula y(4) = ii,(F-1 [4]) = (7)

E

f

A process {Yn } is a finite coding of a process {X n } if it is a stationary coding of {Xn } with finite width time-zero coder. Note that a finite-state process is merely a finite coding of a Markov chain in which the window half-width is 0, that is, a function of a Markov chain. A stationary coding of an i.i.d. process is called a B-process. More generally, a finite alphabet process is called a B-process if it is a stationary coding of some i.i.d. process with finite or infinite alphabet. Much more will be said about B-processes in later parts of this book, especially in Chapter 4, where several characterizations will be discussed. Example 1.1.10 (Block codings.) Another type of coding, called block coding, destroys stationarity. It is frequently used in practice, however, and is often easier to analyze than is stationary coding. An N -block code is a function CN: A N B N , where A and B are finite sets. Such a code can be used to map an A-valued process {X n } into a B-valued process {Yn } by applying CN to consecutive nonoverlapping blocks of length N, that is, y(j+1)N jN+1 = CN ()((f+1)N)

j = 0 1, 2,, • .

+1 iN

,

The process {Yn } is called the N-block coding of the process {X n ) defined by the N -block code C N. The Kolmogorov measures, pt and v, of the respective processes, {X,} and {K}, are connected by the formula y = o F -1 , where y = F(x) is the (measurable) mapping defined by (j+1)N r jN+1 =

,

(j+1)N X j1s1+1 )

9

j

0, I, 2, ...

If the process {X,} is stationary, the N-block coding {Yn } is not, in general, stationary, although it is N-stationary, that is, it is invariant under the N-fold shift T N , where T = Tg, the shift on B. There is a simple way to convert an N-stationary 2rocess, such as the encoded process {Yn }, into a stationary process, {Yn }. The process {Yn } is defined bj selecting an integer u E [1, N] according to the uniform distribution and defining i = 1, 2, .... This method is called "randomizing the start." The relation =

9

SECTION I.1. STATIONARY PROCESSES.

between the Kolmogorov measures, y and î, of the respective processes, {Y,,} and { Y, }, is expressed by the formula v ( c)

= E v (T C), i=o

is the average of y, together with its first N — 1 shifts. The method of that is, randomizing the start clearly generalizes to convert any N-stationary process into a stationary process. In summary, an N-block code CN induces a block coding of a stationary process {Xn } onto an N-stationary process MI, which can then be converted to a stationary process, {Y, } , by randomizing the start. In general, the final stationary process {Yn } is not a stationary encoding of the original process {X,} and many useful properties of {X n } may get destroyed. For example, block coding introduces a periodic structure in the encoded process, {Y,I, a structure which is inherited, with a random delay, by the process {41. In Section 1.8, a general procedure will be developed for converting a block code to a stationary code by inserting spacers between blocks so that the resulting process is a stationary coding of the original process, and as such inherits many of its properties. Example 1.1.11 (Concatenated-block processes.) Processes can also be constructed by concatenating N-blocks drawn independently at random according a given distribution, with stationarity produced by randomizing the start. There are several equivalent ways to describe such a process. In random variable terms, suppose X liv = (X1, , X N ) is a random vector with values in AN. Let {Y,} be the A-valued process defined by the requirement that the blocks . } obtained (j-11N+1 be independent and have the distribution of Xr. The process { 171 from the N-stationary process {Yi I by randomizing the start is called the concatenatedblock process defined by the random vector Xr . The concatenated-block process is characterized by the following two conditions.

(i) li;;} is stationary. (ii) There is random variable U uniformly distributed on {1, 2, ... , NI such that, conditioned on U = u, the sequence {Y:±±(iiN oN±i : j = 1, 2, ...} is i.i.d. with distribution t. An alternative formulation of the same idea in terms of measures is obtained by starting with a probability measure p, on A N , forming the product measure If on (A N )", transporting this to an N-stationary measure p,* o 0 -1 on A" via the mapping = x = w(l)w(2) • • • where

JN X (j-1)N+1 = W(i),

j

= 1, 2, ... ,

then averaging to obtain the measure Z defined by the formula N-1

it*(0 -1 (7—i B)), B

N il(B) = — i=o

E E.

CHAPTER I. BASIC CONCEPTS.

10

The measure ii. is called the concatenated-block process defined by the measure p, on AN. The process defined by ii has the properties (i) and (ii), hence this measure terminology is consistent with the random variable terminology. A third formulation, which is quite useful, represents a concatenated-block process as a finite-state process, that is, a function of a Markov chain. To describe this representation fix a measure p, on A N and let { Y„,} be the (stationary) Markov chain with alphabet S = AN x {1, 2, ... , N}, defined by the following transition and starting rules. (a) If i < N , (ar , i) can only go to (ar , i ± 1).

(b) (ar , N) goes to (b Ç', 1) with probability p,(br), for each br (C)

E

AN.

The process is started by selecting (ar ,i) with probability i,(4)/N.

The process {Y,n } defined by setting Y, n = ai, if Y,r, = (ar , j), is the same as the concatenated-block process based on the block distribution A. (See Exercise 7.) Concatenated-block processes will play an important role in many later discussions in this book. Example 1.1.12 (Blocked processes.) Associated with a stationary process {X n } and a positive integer N are two different stationary processes, {Z n } and IKI, defined, for n > 1, by Zn = (X(n-1)N+19 X(n-1)N+29 • • •

Wn = ant X+1, • • •

9

1

XnN);

Xn+N-1)•

The Z process is called the nonoverlapping N-block process determined by {X n }, while the W process is called the overlapping N-block process determined by {X J. Each is stationary if {X,} is stationary. If the process {X,} is i.i.d. then its nonoverlapping blockings are i.i.d., but overlapping clearly destroys independence. The overlapping N-blocking of an i.i.d. or Markov process is Markov, however. In fact, if {X,} is Markov with transition matrix Mab, then the overlapping N-block process is Markov with alphabet B = far: War) > 0} and transition matrix defined by Ma i, b' 1

1

=

{ MaNbN, 0

if bNi—1 = ci 2iv otherwise.

Note also that if {X n} is Markov with transition matrix Mab, then the nonoverlapping N-block process is Markov with alphabet B and transition matrix defined by N-1

1-4a'iv ,tSiv

= MaNbi

ri

Mb,b, +i

i= 1

A related process of interest {Yn }, called the N-th term process, selects every N-th term from the {X,} process, that is, Yn = X (n-1)N+1, n E Z. If {Xn } is Markov with transition matrix M, then {Y,i } will be Markov with transition matrix MN, the N-th power of M.

SECTION 1.1. STATIONARY PROCESSES.

I.l.b

11

Probability tools.

Two elementary results from probability theory will be frequently used, the Markov inequality and the Borel-Cantelli principle. Lemma 1.1.13 (The Markov inequality.) Let f be a nonnegative, integrable function on a probability space (X, E, p..). If f f dp.. < c 8 then f (x) < c, except for a set of measure at most b. Lemma 1.1.14 (The Borel-Cantelli principle.) If {C n } is a sequence of measurable sets in a probability space (X, E, p.) such that E ii(c,i ) < 00 then for almost every x there is an N = N(x) such that x g C„, n > N. In general, a property P is said to be measurable if the set of all x for which P(x) is true is a measurable set. If {Pn } is a sequence of measurable properties then (a) P(x) holds eventually almost surely, if for almost every x there is an N = N(x) such that Pn (x) is true for n > N. (b) P, (x) holds infinitely often, almost surely, if for almost every x there is an increasing sequence {n i } of integers, which may depend on x, such that Pn,(x) is true for i = 1, 2, ....

For example, the Borel-Cantelli principle is often expressed by saying that if E wc,) < oo then x g C,„ eventually almost surely. Almost-sure convergence is often established using the following generalization of the Borel-Cantelli principle. Lemma 1.1.15 (The iterated Borel-Cantelli principle.) Suppose {G„} and {B} are two sequences of measurable sets such that x E G„, eventually almost surely, and x g B, n Gn , eventually almost surely. Then x g Bn , eventually almost surely. The proof of this is left to the exercises. In many applications, the fact that x E B, n Gn, eventually almost surely, is established by showing that E ,u(Bn n Gn ) < oo, in which case the iterated Borel-Cantelli principle is, indeed, just a generalized Borel-Cantelli principle. Frequent use will be made of various equivalent forms of almost sure convergence, summarized as follows. Lemma 1.1.16 The following are equivalent for measurable functions on a probability space. (a) f, —> f, almost surely.

(b) I fn (x) — f (x)I <E, eventually almost surely, for every c > O. (c) Given c > 0, there is an N and a set G of measure at least 1 — c, such that Ifn(x) — f(x)1 < E , X E G, n ? N.

CHAPTER I. BASIC CONCEPTS.

12

As several of the preceding examples suggest, a process is often specified as a function of some other process. Probabilities for the new process can then be calculated by using the inverse image to transfer back to the old process. Sometimes it is useful to go in the opposite direction, e. g., first see what is happening on a set of probability 1 in the old process then transfer to the new process. A complication arises, namely, that Borel images of Borel sets may not be Borel sets. For nice spaces, such as product spaces, this is not a serious problem, for such images are always measurable with respect to the completion of the image measure, [5]. This fact is summarized by the following lemma.

Lemma 1.1.17 (The Borel mapping lemma.) Let F be a Borel function from A" into B", let p. be a Borel measure on A", let v = p. o F-1 and let r) be the completion of v. If X is a Borel set such that p(X) = 1, then F(X) is measurable with respect to the completion i of v and ii(F (X)) = 1. A similar result holds with either A" or B" replaced, respectively, by AZ or B z . Use will also be made of two almost trivial facts about the connection between cardinality and probability, namely, that lower bounds on probability give upper bounds on cardinality, and upper bounds on cardinality "almost" imply lower bounds on probability. For ease of later reference these are stated here as the following lemma.

Lemma 1.1.18 (Cardinality bounds.) Let p. be a probability measure on the finite set A, let B c A, and let a be a positive number (a) If a E B (b) For b

E

p(a) a a, then I BI 5_ 11a. B, p(b) > a/I BI, except for a subset of B of measure at most a.

Deeper results from probability theory, such as the martingale theorem, the central limit theorem, the law of the iterated logarithm, and the renewal theorem, will not play a major role in this book, though they may be used in various examples, and sometimes the martingale theorem is used to simplify an argument.

I.l.c

Exercises.

1. Prove Lemma 1.1.18. 2. Give an example of a function of a finite-alphabet Markov chain that is not Markov of any order. (Include a proof that your example is not Markov of any order.) 3. If ,u([4] n [ci_t) . ,u([4]),u([4]), for all n and k, and all a;' and an+ni±k n+m-1-1 , then a stationary process is said to be m-dependent. Show that a finite coding of an i.i.d. process is m-dependent for some m. (How is m related to window half-width?) 4. Let {tin } be i.i.d. with each U, uniformly distributed on [0, 1]. Define X, = 1, if U„ > Un _i , otherwise X, = O. Show that {X,1 is 1-dependent, and is not a finite coding of a finite-alphabet i.i.d. process. (Hint: show that such a coding would have the property that there is a number c > 0 such that A(4) > c", if p.(x 1') 0 O. Then show that the probability of n consecutive O's is 11(n + 1)!.)

13

SECTION L2. THE ERGODIC THEORY MODEL.

5. Show that the process constructed in the preceding exercise is not Markov of any order. 6. Establish the Kolmogorov representation theorem for countable alphabet processes. 7. Show that the finite-state representation of a concatenated-block process in Example 1.1.11 satisfies the two conditions (i) and (ii) of that example. 8. A measure kt on An defines a measure ,u (N) on AN, where N = Kn+r, 0 < n < r, by the formula

(n

K-1

=

kn -En iU(Xkr1+1)

,Kn-Fr \ 1-4-4 Kn+1 , •

k=0

The measures {,a (N) , N = 1, 2, ...} satisfy the Kolmogorov consistency conditions, hence have a common extension ,tc* to A". Show that the concatenatedblock process Z defined by kt is an average of shifts of , that is, ÎZ(B) = . (1/n) E7=1 kt* (1' B), for each Borel subset B C 9. Prove the iterated Borel-Cantelli principle, Lemma 1.1.15.

Section 1.2 The ergodic theory model. The Kolmogorov measure for a stationary process is preserved by the shift T on the sequence space. This suggests the possibility of using ergodic theory ideas in the study of stationary processes. Ergodic theory is concerned with the orbits x,Tx,T 2x,... of a transformation T: X X on some given space X. In many cases of interest there is a natural probability measure preserved by T, relative to which information about orbit structure can be expressed in probability language. Finite measurements on the space X, which correspond to finite partitions of X, then give rise to stationary processes. This model, called the transformation/partition model for a stationary process, is the subject of this section and is the basis for much of the remainder of this book. The Kolmogorov model for a stationary process implicitly contains the transformation/partition model. The shift T on the sequence space, X = A", (or on the space A z ), is the transformation and the partition is P = {Pa : a E A}, where for each a E A, Pc, = {x: x 1 = a}. The partition P is called the Kolmogorov partition associated with the process. The sequence of random variables and the joint distributions are expressed in terms of the shift and Kolmogorov partition, as follows. First, associate with the partition P the random variable Xp defined by X2 (x) = a if a is the label of member of P to which x belongs, that is, x E Pa . The coordinate functions, {X,}, are given by the formula (1)

X(x) = Xp(Tn -1 x), n > 1.

The process can therefore be described as follows: Pick a point x E X, that is, an infinite sequence, at random according to the Kolmogorov measure, and, for each n, let Xn = X n (x) be the label of the member of P to which 7' 1 x belongs. Since to say that

14

CHAPTER I. BASIC CONCEPTS.

Pa is the same as saying that x E T -n+ 1 Pa , cylinder sets and joint distributions may be expressed by the respective formulas, Tn-1 X E

[4] = n7..„7—i+1 Pai , and (2)

= u([4]) = i (nLI T —i+1 Pa, ) In summary, the coordinate functions, the cylinder sets in the Kolmogorov representation, and the joint distributions can all be expressed directly in terms of the Kolmogorov partition and the shift transformation. The concept of stationary process is formulated in terms of the abstract concepts of measure-preserving transformation and partition, as follows. Let (X, E, A) be a probability space. A mapping T: X X is said to be measurable if T -1 B E E, for all B E E, and measure preserving if it is measurable and if

(T' B) = ,u,(B), B

E

E,

A partition

P = {Pa : a

E

A}

of X is a finite, disjoint collection of measurable sets, indexed by a finite set A, whose union has measure 1, that is, X — U n Pa is a null set. (In some situations, countable partitions, that is, partitions into countably many sets, are useful.) Associated with the partition P = {Pa : a E Al is the random variable Xp defined by Xp(x) = a if x E Pa . The random variable Xp and the measure-preserving transformation T together define a process by the formula

X(x) = Xp(T n-l x), n> 1.

(3)

The k-th order distribution Ak of the process {Xa } is given by the formula

(4) = u (nLiT -i+1 Pai )

(4)

the direct analogue of the sequence space formula, (2). The process {Xa : n > 1} defined by (3), or equivalently, by (4), is called the process defined by the transformation T and partition P, or, more simply, the (T, 2) process. The sequence { xa = X n (x): n > 1} defined for a point x E X by the formula Tn -l x E Pin , is called the (T, 2) name of x. The (T, 2)-process may also be described as follows. Pick x E X at random according to the A-distribution and let X 1 (x) be the label of the set in P to which x belongs. Then apply T to x to obtain Tx and let X2(x) be the label of the set in P to which Tx belongs. Continuing in this manner, the values, -

-

(x), X 2 (x X3(x), ),

X a (x),

tell to which set of the partition the corresponding member of the random orbit

belongs. Of course, in the Kolmogorov representation a random point x is a sequence in 21°' or A z , and the (T, 2)-name of x is the same as x, (or the forward part of x in the two-sided case.)

SECTION 1.2. THE ERGODIC THEORY MODEL.

15

The (T, 2)-process concept is, in essence, just an abstract form of the stationary coding concept, in that a partition P of (X, E, p) gives rise to a measurable function F: X 1-÷ B" which carries p onto the Kolmogorov measure y of the (T, 2)-process, such that F(Tx) = TA F (x), where TB is the shift on 13'. The mapping F extends to a stationary coding to Bz in the case when X = A z and T is the shift. Conversely, a stationary coding F: A z B z carrying it onto y determines a partition P = {Pb: b E BI of Az such that y is the Kolmogorov measure of the (TA, 2)-process. The partition P is defined by Pb = {x: f (x) = 1) } . See Exercise 2 and Exercise 3. In summary, a stationary process can be thought of as a shift-invariant measure on a sequence space, or, equivalently, as a measure-preserving transformation T and partition P of an arbitrary probability space. The ergodic theory point of view starts with the transformation/partition concept, while modern probability theory starts with the sequence space concept.

I.2.a Ergodic processes. It is natural to study the orbits of a transformation by looking at its action on invariant sets, since once an orbit enters an invariant set it never leaves it. In particular, the natural object of study becomes the restriction of the transformation to sets that cannot be split into nontrivial invariant sets. This leads to the concept of ergodic transformation. A measurable set B is said to be T-invariant if TB C B. The space X is Tdecomposable if it can be expressed as the disjoint union X = X1U X2 of two measurable invariant sets, each of positive measure. The condition that TX C Xi, i = 1, 2 translates into the statement that T -1 Xi = Xi, i = 1, 2, hence to say that the space is indecomposable is to say that if T -1 B = B then p(B) is 0 or 1. It is standard practice to use the word "ergodic" to mean that the space is indecomposable. A measure-preserving transformation T is said to be ergodic if

7-1 B = B = (B) = 0 or p(B) = 1.

(5)

The following lemma contains several equivalent formulations of the ergodicity condition. The notation, C AD = (C — D)U(D— C), denotes symmetric difference, and UT denotes the operator on functions defined by (UT f)(x) = f (T x), where the domain can be taken to be any one of the LP-spaces or the space of measurable functions. Lemma 1.2.1 (Ergodicity equivalents.) The following are equivalent for a measure-preserving transformation T on a probability space.

(a) T is ergodic. (b) T -1 B C B, = (c) T -1 B D B, (d) (e) UT

(B) = 0 or kt(B) = 1. ,i(B) = 0 or p(B) = 1.

B LB) = 0 = (B) = 0 or kt(B) = 1.

= f , a.e., implies that f is constant, a.e.

CHAPTER I. BASIC CONCEPTS.

16

Proof The equivalence of the first two follows from the fact that if T -1 B c B and C = n>0T -n B, then T -1 C = C and ,u(C) = ,u(B). The proofs of the other equivalences are left to the reader.

Remark 1.2.2 In the particular case when T is invertible, that is, when T is one-to-one and for each measurable set C the set T C is measurable and has the same measure as C, the conditions for ergodicity can be expressed in terms of the action of T, rather than T -1 . In particular, an invertible T is ergodic if and only if any T-invariant set has measure 0 or 1. Also, note that if T is invertible then UT is a unitary operator on L2 . A stationary process is ergodic, if the shift in the Kolmogorov representation is ergodic relative to the Kolmogorov measure. As will be shown in Section 1.4, to say that a stationary process is ergodic is equivalent to saying that measures of cylinder sets can be determined by counting limiting relative frequencies along a sample path x, for almost every x. Furthermore, a shift-invariant measure shift-invariant measures. Thus the concept of ergodic process, which is natural from the transformation point of view, is equivalent to an important probability concept.

I.2.b

Examples of ergodic processes.

Examples of ergodic processes include i.i.d. processes, irreducible Markov chains and functions thereof, stationary codings of ergodic processes, concatenated-block processes, and some, but not all, processes obtained from a block coding of an ergodic process by randomizing the start. These and other examples will now be discussed.

Example 1.2.3 (The Baker's transformation.) A simple geometric example provides a transformation and partition for which the resulting process is the familiar coin-tossing process. Let X = [0, 1) x [0, 1) denote the unit square and define a transformation T by

T (s, t) =

{ (2s, t/2) (2s — 1, (t ± 1)/2)

if S 1/2.

The transformation T is called the Baker's transformation since its action can be described as follows. (See Figure 1.2.4.) 1. Cut the unit square into two columns of equal width.

2. Squeeze each column down to height 1/2 and stretch it to width 1. 3. Place the right rectangle on top of the left to obtain a square.

• Tw

• Tz

•

Tw

Figure 1.2.4 The Baker's transformation.

• Tz

17

SECTION I.2. THE ERGODIC THEORY MODEL.

The Baker's transformation T preserves Lebesgue measure in the square, for dyadic subsquares (which generate the Borel field) are mapped into rectangles of the same area. To obtain the coin-tossing process define the two-set partition P = {Po, PO by setting Po = {(s, t): s < 1/2}, P1 = {(s, t): s _?: 1/2}.

(6)

To assist in showing that the (T ,P)-process is, indeed, the binary, symmetric, i.i.d. process, some useful partition notation and terminology will be developed. Let P = {Pa : a E A} be a partition of the probability space (X, E, ii). The distribution of P is the probability distribution ftt(Pa), a E Al. The join, P v Q, of two partitions P and Q is their common refinement, that is, the partition P v Q = {Pa n Qb: Pa

E P , Qb E Q} •

The join AP(i) of a finite sequence P (1) , P (2) , ... , P(k) of partitions is their common refinement; defined inductively by APO ) = (vrP (i) ) y p(k). Two partitions P and Q are independent if the distribution of the restriction of P to each Qb E Q does not depend on b, that is, if

[t(Pa n Qb) = 12 (Pa)11 (Qb),Va, b. The sequence of partitions {P (i) : i > 1} is said to be independent if P (k+ i) and v1P (i) independent for each k? 1. are For the Baker's transformation and partition (6), Figure 1.2.5 illustrates the partitions P, TP ,T 2P , 7-1 P , along with the join of these partitions. Note that T 2P partitions each set of T -1 7, v P v TT into exactly the same proportions as it partitions the entire space X. This is precisely the meaning of independence. In particular, it can be shown that the partition TnP is independent of vg -1 PP , for each n > 1, so that {T i P} is an independent sequence. This, together with the fact that each of the two sets P0 , P1 has measure 1/2, implies, of course, that the (T, 2)-process is just the coin-tossing process.

1

T -1 1)1 n Po n T Pi n T2 Po

0 P

Ti'

1 0 1 0

I I

01101

T -I P v P v T'P v T 2P

I I .

T 2p

T -1 P Figure 1.2.5 The join of P, TP, T2P, and 7— '1'.

In general, the i.i.d. process ii, defined by the first-order distribution j 1 (a), a E A, is given by a generalized Baker's transformation, described as follows. (See Figure 1.2.6.)

CHAPTER I. BASIC CONCEPTS.

18

1. Cut the unit square into I AI columns, labeled by the letters of A, such that the column labeled a has width A i (a). 2. For each a E A squeeze the column labeled a down and stretch it to obtain a rectangle of height (a) and width 1. 3. Stack the rectangles to obtain a square. The corresponding Baker's transformation T preserves Lebesgue measure. The partition P into columns defined in part (a) of the definition of T then produces the desired i.i.d. process.

•

Figure 1.2.6 The generalized Baker's transformation. Example 1.2.7 (Ergodicity for i.i.d. processes.) To show that a process is ergodic it is sometimes easier to verify a stronger property, called mixing. A transformation is mixing if (7)

lim ,u(T -nC n D) = ,u(C),u(D), C, D

E

n—,froo

E.

A stationary process is mixing if the shift is mixing for its Kolmogorov measure. Mixing clearly implies ergodicity, for if T -1 C = C then T -ncnD=CnD, for all sets D and positive integers n, so that the mixing property, (7), implies that p, (C nD) = Since this holds for all sets D, the choice D = C gives ,i(C) = ,u(C) 2 and hence ,u(C) is 0 or 1. Suppose it is a product measure on A'. To show that the mixing condition (7) holds for the shift first note that it is enough to establish the condition for any two sets in a generating algebra, hence it is enough to show that it holds for any two cylinder sets. But this is easy, for if C = [a7] and N > 0 then T-N =

A +1

bNA -i = a1,

1 < < n.

Thus, if D = [br] and N > m then T -N C and D depend on values of x i for indices i in disjoint sets of integers. Since the measure is product measure this means that

p,(T -Nc n D) = ,a(T-N C),a(D) = Thus i.i.d. measures satisfy the mixing condition. Example 1.2.8 (Ergodicity for Markov chains.) The location of the zero entries, if any, in the transition matrix M determine whether a Markov process is ergodic. The results are summarized here; the reader may refer to standard probability books or to the extensive discussion in [29] for details. The stochastic matrix M is said to be irreducible, if for any pair i, j there is a sequence io, i 1 , , in with io = i and in = j such that > 0, in = 0, 1,

, n — 1.

SECTION 1.2. THE ERGODIC THEORY MODEL.

19

This is just the assertion that for any pair i, j of states, if the chain is at state i at some time then there is a positive probability that it will be in state j at some later time. If M is irreducible there is a unique probability vector it such that irM = 7r. Furthermore, each entry of it is positive and N liM

(8)

N—oc)

N

n=1

where each row of the k x k limit matrix P is equal to jr. In fact, the limit of the averages of powers can be shown to exist for an arbitrary finite stochastic matrix M. In the irreducible case, there is only one probability vector IT such that 7rM = it and the limit matrix P has it all its rows equal to 7r. The condition 7rM = it shows that the Markov chain with start distribution Ai = it and transition matrix M is stationary. An irreducible Markov chain is ergodic. To prove this it is enough to show that N (9)

C n D) =

LlMoo

for any cylinder sets C and D, hence for any measurable sets C and D. This condition, which is weaker than the mixing condition (7), implies ergodicity, for if T -1 C = C and D = C, then the limit formula (9) gives /..t(C) = 1u(C) 2 , so that C must have measure 0 or 1. To establish (9) for an irreducible chain, let D = [4] and C = [cri and write + T—k—n = [bnn ++ kk-Fr ], where bn_f_k+i = ci, 1 < i < m. Thus when k > 0 the sets T -k-n C and D depend on different coordinates. Summing over all the ways to get from d, to bn+k+1 yields the product

p.,([4] n [btni 1- _- :Vinj) = uvw, where n-1 7r (di)11Md,d, +1 =

V

acin),

(n-Flc-1

=

JJ

Md,d, +1

Mc1,1±kbn+k+i

i=n n+k+m-1

= 11

i=n+k-F1

Note that V, which is the probability of transition from dn to bn+k+1 = Cl in equal to MI ci = [Mk ]d c „ the dn ci term in the k-th power of M, and hence

k

steps, is

.-1 (10)

itt([d]

n [bnn ii

1) = ttacril i) A I

mc,c,+i.

,

1=,

The sequence ML.1 1 converges in the sense of Cesaro to (Ci), by (8), which establishes (9), since ,u(C) = This proves that irreducible Markov chains are ergodic.

CHAPTER I. BASIC CONCEPTS.

20

The converse is also true, at least with the additional assumption being used in this book that every state has positive probability. Indeed, if every state has positive probability and if Pi = {x: xi = j}, then the set B = T -1 Pi U T -2 PJ U has positive probability and satisfies T -1 B D B. Thus if the chain is ergodic then ,u(B) = 1, so that, ,u(B n Pi ) > 0, for any state i. But if it(B n Pi ) > 0, then ,u(T -nP n Pi ) > 0, for some n, which means that transition from i to j in n steps occurs with positive probability, and hence the chain is irreducible. In summary, Proposition 1.2.9 A stationary Markov chain is ergodic if and only if its transition matrix is irreducible. If some power of M is positive (that is, has all positive entries) then M is certainly irreducible. In this case, the Cesaro limit theorem (8) can be strengthened to limN,„ M N = P. The argument used to prove (9) then shows that the shift T must be mixing. The converse is also true, again, assuming that all states have positive probability. Proposition 1.2.10 A stationary Markov chain is mixing if and only if some power of its transition matrix has all positive entries.

Remark 1.2.11 In some probability books, for example, Feller's book, [11], "ergodic" for Markov chains is equivalent to the condition that some power of the transition matrix be positive. To be consistent with the general concept of ergodic process as used in this book, "ergodic" for Markov chains will mean merely irreducible with "mixing Markov" reserved for the additional property that some power of the transition matrix has all positive entries.

Example 1.2.12 (Codings and ergodicity.) Stationary coding preserves the ergodicity property. This follows easily from the definitions, for suppose F: A z B z is a stationary encoder, suppose it is the Kolmogorov measure of an ergodic A-valued process, and suppose v = p. o F -1 is the Kolmogorov measure of the encoded B-valued process. If C is a shift-invariant subset of B z then F-1 C is a shift-invariant subset of Az so that ,u(F-1 C) is 0 or 1. Since v(C) = p,(F-1 C) it follows that v(C) is 0 or 1, and hence that v is the Kolmogorov measure of an ergodic process. It is not important that the domain of F be the sequence space A z ; any probability space will do. Thus, if T is ergodic the (T, 2)-process is ergodic for any finite partition P. It should be noted, however, that the (T, 2)-process can be ergodic even though T is not. (See Exercise 5, below.) A simple extension of the above argument shows that stationary coding also preserves the mixing property, see Exercise 19. A finite-state process is a stationary coding with window width 0 of a Markov chain. Thus, if the chain is ergodic then the finite-state process will also be ergodic. Likewise, a finite-state process for which the underlying Markov chain is mixing must itself be mixing. A concatenated-block process is always ergodic, since it can be represented as a stationary coding of an irreducible Markov chain; see Example 1.1.11. The underlying

21

SECTION 1.2. THE ERGODIC THEORY MODEL.

Markov chain is not generally mixing, however, for it has a periodic structure due to the blocking. Since this periodic structure is inherited, up to a shift, concatenated-block processes are not mixing except in special cases. A transformation T on a probability space (X, E, p.) is said to be totally ergodic if every power TN is ergodic. If T is the shift on a sequence space then T is totally ergodic if and only if for each N, the nonoverlapping N-block process defined by Zn = (X (n-1)N-1-11 X (n-1)N+2, • • •

9

Xn

is ergodic. (See Example 1.1.12.) Since the condition that F(TA X) = TB F (x), for all x, implies that F (Ti x) = T: F(x), for all x and all N, it follows that a stationary coding of a totally ergodic process must be totally ergodic. As noted in Section 1.1, N-block codings destroy stationarity, but a stationary process can be constructed from the encoded process by randomizing the start, see Example 1.1.10. The final randomized-start process may not be ergodic, however, even if the original process was ergodic. For example, let p. give measure 1/2 to each of the two sequences 1010... and 0101 ..., that is, p is the stationary Markov measure with transition matrix M and start distribution 7r given, respectively, by m

ro 1 L o j'

=

rl

1]

i•

Let y be the encoding of p. defined by the 2-block code C(01) = 00, C(10) = 11, so that y is concentrated on the two sequences 000... and 111 .... The measure -17 obtained by randomizing the start is, in this case, the same as y, hence is not an ergodic process. A condition insuring ergodicity of the process obtained by N-block coding and randomizing the start is that the original process be ergodic relative to the N-shift T N . The proof of this is left to the reader. In particular, applying a block code to a totally ergodic process and randomizing the start produces an ergodic process. Example 1.2.13 (Rotation processes.) Let a be a fixed real number and let T be defined on X = [0, 1) by the formula Tx = x ED a,

where ED indicates addition modulo 1. The mapping T is called translation by a. It is also called rotation, since X can be thought of as the unit circle by identifying x with the angle 27rx, so that translation by a becomes rotation by 27ra. A subinterval of the circle corresponds to a subinterval or the complement of a subinterval in X, hence the word "interval" can be used for subsets of X that are connected or whose complements are connected. The transformation T is one-to-one and maps intervals onto intervals of the same length, hence preserves Lebesgue measure p,. (The measure-preserving property is often established by proving, as in this case, that it holds on a family of sets that generates the a-algebra.) Any partition 7, of the interval gives rise to a process; such processes are called translation processes, (or rotation processes if the circle representation is used.) As an example, let P consist of the two intervals P0 = [0, 1/2), P 1 = [1/2, 1), which correspond to the upper and lower halves of the circle in the circle representation. The (T, P)-process {X n } is then described as follows. Pick a point x at random in the unit interval according to the uniform (Lebesgue) measure. The value X(x) = Xp(Tn -1 x) is then 0 or 1, depending on whether Tn - lx = x ED (n — 1)a belongs to P0 or P1 . The following proposition is basic.

CHAPTER I. BASIC CONCEPTS.

22 Proposition 1.2.14 T is ergodic if and only if a is irrational.

Proof It is left to the reader to show that T is not ergodic if a is rational. Assume that a is irrational. Two proofs of ergodicity will be given. The first proof is based on the following classical result of Kronecker.

Proposition 1.2.15 (Kronecker's theorem.) If a is irrational the forward orbit {Tnx: n > 1} is dense in X, for each x. Proof To establish Kronecker's theorem let F be the closure of the forward orbit, {Tx: n > 1 } , and suppose that F is not dense in X. Then there is an interval I of positive length which is maximal with respect to the property of not meeting F. Furthermore, for each positive integer n, TI cannot be the same as I, since a is irrational, and hence the maximality of I implies that n I = 0. It follows that is a disjoint sequence and 00 1 = ,u ( [0, 1))

a.

E n=1

00,

which is a contradiction. This proves Kronecker's theorem.

Proof of Proposition 1.2.14 continued. To proceed with the proof that T is ergodic if a is irrational, suppose A is an invariant set of positive measure. Given c > 0 choose an interval I such that /2(1) < E

and /..t(A

n I) > (1 —

Kronecker's theorem, applied to the end points of I, produces integers n1 1 — 2E. The assumption that i,t(A n I) > (1 — 6) gives it (A)

E

n Trig)

—E)

(1 — 0(1 — 2E).

i= 1

Thus ,u(A) = 1, which completes the proof of Proposition 1.2.14. The preceding proof is typical of proofs of ergodicity for many transformations defined on the unit interval. One first shows that orbits are dense, then that small disjoint intervals can be placed around each point in the orbit. Such a technique does not work in all cases, but does establish ergodicity for many transformations of interest. A generalization of this idea will be used in Section 1.10 to establish ergodicity of a large class of transformations. An alternate proof of ergodicity can be obtained using Fourier series. Suppose f is square integrable and has Fourier series E an e brinx . The Fourier series of g(x) = f (T x)

is E a, e2-n(x+a)

a, e2"ina e 27rinx

SECTION 1.2. THE ERGODIC THEORY MODEL.

23

If g(x) = f(x), a. e., then an = an e 2lri" holds for each integer n. If a is irrational 0 unless n = 0, which means that a, = 0, unless and an = an e 27' then e2lrin" n=0, which, in turn, implies that the function f is constant, a. e. In other words, in the irrational case, the rotation T has no invariant functions and hence is ergodic. In general, translation of a compact group will be ergodic with respect to Haar measure if orbits are dense. For example, consider the torus T, that is, the product of two intervals, T = [0, 1) x [0, 1), with addition mod 1, (or, alternatively, the product of two circles.) The proof of the following is left to the reader.

Proposition 1.2.16 The following are equivalent for the mapping T: (x, y) 1-÷ (x a, y El) p) (i) {Tn(x, y)} is dense for any pair (x, y). (ii) T is ergodic. (iii) a and 13 are rationally independent.

I.2.c The return-time picture. A general return-time concept has been developed in ergodic theory, which, along with a suggestive picture, provides an alternative and often simpler view of two standard probability models, the renewal processes and the regenerative processes. Let T be a measure-preserving transformation on the probability space (X, E, IL) and let B be a measurable subset of X of positive measure. If x E B, let n(x) be the least positive integer n such that Tnx E B, that is, n(x) is the time of first return to B.

Theorem 1.2.17 (The Poincare recurrence theorem.) Return to B is almost certain, that is, n(x) < oc, almost surely. Proof To prove this, define the sets Bn = E B: n(x) = n}, for n > 1, and define = {X E B: Tx B,Vn > O. These sets are clearly disjoint and have union B, and, furthermore, are measurable, for B 1 = B n 7-1 B and

Bn =

(X

—

B)

n B n T -n B, n > 2.

Furthermore, if x E 1100 then Tx g Bc.,, for all n > 1, from which it follows that the sequence of sets {T B} is disjoint. Since they all have the same measure as B and p(X) = 1, it follows that p(B,,o ) = 0. This proves Theorem 1.2.17. To simplify further discussion it will be assumed in the remainder of this subsection that T is invertible. (Some extensions to the noninvertible case are outlined in the exercises.) For each n, the sets

(11)

Bn , TB,

, T n-1 Bn

are disjoint and have the same measure. Furthermore, only the first set Bn in this sequence meets B, while applying T to the last set Tn -1 Bn returns it to B, that is, TBn C B. The return-time picture is obtained by representing the sets (11) as intervals, one above the other. Furthermore, by reassembling within each level it can be assumed that T just moves points directly upwards one level. (See Figure 1.2.18). Points in the top level of each column are moved by T to the base B in some manner unspecified by the picture.

CHAPTER I. BASIC CONCEPTS.

24

Tx

B1

B3

B2

B4

Figure 1.2.18 Return-time picture. Note that the set Util=i Bn is just the union of the column whose base is Bn , so that the picture is a representation of the set (12)

U7=0 TB

This set is T-invariant and has positive measure. If T is ergodic then it must have measure 1, in which case, the picture represents the whole space, modulo a null set, of course. The picture suggests the following terminology. The ordered set

Bn , TB,

, T 11-1 13,

is called the column C = Cn with base Bn, width w(C) = kt(B,z ), height h(C) = n, levels L i = Ti-1 Bn , 1 < i < n, and top T'' B. Note that the measure of column C is just its width times its height, that is, p,(C) = h(C)w(C). Various quantities of interest can be easily expressed in terms of the return-time picture. For example, the return-time distribution is given by Prob(n(x) =nIxEB)—

w (Ca )

Em w(C,n

)

and the expected return time is given by

E(n(x)lx E B) = (13)

h(C,i )Prob(n(x) =nix E B)

h(C)w(C)

=

m w(Cm )

In the ergodic case the latter takes the form

(14)

E(n(x)ix E B) =

1 Em w(Cm )

1

since En h(C)w(C) is the measure of the set (12), which equals 1, when T is ergodic. This formula is due to Kac, [23]. The transformation I' = TB defined on B by the formula fx = Tn(x)x is called the transformation induced by T on the subset B. The basic theorem about induced transformations is due to Kakutani.

25

SECTION I.2. THE ERGODIC THEORY MODEL.

Theorem 1.2.19 (The induced transformation theorem.) If p(B) > 0 the induced transformation preserves the conditional measure p(.IB) and is ergodic if T is ergodic. -

Proof If C c B, then x E f- 1C if and only if there is an n > 1 such that x Tnx E C. This translates into the equation

E

Bn and

00

F-1 C =U(B n n=1

n TAC),

which shows that T-1 C is measurable for any measurable C c B, and also that 00

/2(f—i

n

= E ,u(B,, n T'C), n=1

since Bn n An = 0, if m 0 n, with m, n > 1. More is true, namely, (15)

T n Ba nT ni Bm = 0, m,n > 1, m n,

by the definition of "first return" and the assumption that T is invertible. But this implies 00 E,(T.Bo n=1

co

= E p(Bn ) = AO), n=1

which, together with (15) yields E eu(B, n T'C) = E iu(Tnii n n c) = u(C). Thus the induced transformation f preserves the conditional measure p.(.1B). The induced transformation is ergodic if the original transformation is ergodic. The picture in Figure 1.2.18 shows why this is so. If -f --1 C, then each C n Bn can be pushed upwards along its column to obtain the set ,

c.

oo n-1 Ti(c n Bn)• D=UU n=1 i=0

But p,(T DAD) = 0 and T is invertible, so the measure of D must be 0 or 1, which, in turn, implies that kt(C1B) is 0 or 1, since p,((D n B),LC) = O. This completes the proof of the induced-transformation theorem. El

1.2.c.1 Processes associated with the return-time picture. Several processes of interest are associated with the induced transformation and the return-time picture. It will be assumed throughout this discussion that T is an invertible, ergodic transformation on the probability space (X, E, p,), partitioned into a finite partition P = {Pa : a E A}; that B is a set of positive measure; that n(x) = minfn: Tnx E BI, x E B, is the return-time function; and that f is the transformation induced on B by T. Two simple processes are connected to returns to B. The first of these is {R,}, the (T, B)-process defined by the partition B = { B, X B}, with B labeled by 1 and X — B labeled by 0, in other words, the binary process defined by —

Rn(x) = xB(T n-l x), n?: 1,

CHAPTER I. BASIC CONCEPTS.

26

where xB denotes the indicator function of B. The (T, 5)-process is called the generalized renewal process defined by T and B. This terminology comes from the classical definition of a (stationary) renewal process as a binary process in which the times between occurrences of l's are independent and identically distributed, with a finite expectation, the only difference being that now these times are not required to be i.i.d., but only stationary with finite expected return-time. The process {R,,} is a stationary coding of the (T, 2)-process with time-zero encoder XB, hence it is ergodic. Any ergodic, binary process which is not the all 0 process is, in fact, the generalized renewal process defined by some transformation T and set B; see Exercise 20. The second process connected to returns to B is {Ri } , the (7." , g)-process defined by the (countable) partition

{Bi, B2, • • •}, of B, where B, is the set of points in B whose first return to B occurs at time n. The process {k } is called the return-time process defined by T and B. It takes its values in the positive integers, and has finite expectation given by (14). Also, it is ergodic, since I' is ergodic. Later, it will be shown that any ergodic positive-integer valued process with finite expectation is the return-time process for some transformation and partition; see Theorem 1.2.24. The generalized renewal process {R n } and the return-time process {kJ } are connected, for conditioned on starting in B, the times between successive occurrences of l's in {Rn} are distributed according to the return-time process. In other words, if R 1 = 1 and the sequence {S1 } of successive returns to 1 is defined inductively by setting So = 1 and

= min{n >

Rn = 1},

j > 1,

then the sequence of random variables defined by

(16)

= Si — Sf _ 1 , j> 1

has the distribution of the return-time process. Thus, the return-time process is a function of the generalized renewal process. Except for a random shift, the generalized renewal process {R,,} is a function of the return-time process, for given a random sample path R = {k } for the return-time process, a random sample path R = {R n } for the generalized renewal process can be expressed as a concatenation (17)

R = ii)(1)w(2)w(3) • • •

of blocks, where, for m > 1, w(m) = 01 _ 1 1, that is, a block of O's of length kn -1, followed by a 1. The initial block t-t,(1) is a tail of the block w(1) = 0 /1-1-1 1. The only real problem is how to choose the start position in w(1) in such a way that the process is stationary, in other words how to determine the waiting time r until the first 1 occurs. The start position problem is solved by the return-time picture and the definition of the (T, 2)-process. The (T, 2)-process is defined by selecting x E X at random according to the it-measure, then setting X(x) = a if and only if Tx E Pa . In the ergodic case, the return-time picture is a representation of X, hence selecting x at random is the same as selecting a random point in the return-time picture. But this, in turn, is the same as selecting a column C at random according to column measure

SECTION 1.2. THE ERGODIC THEORY MODEL.

27

then selecting j at random according to the uniform distribution on 11, 2, ... , h (C)}, then selecting x at random in the j-th level of C. In summary,

Theorem 1.2.20 The generalized renewal process {R,} and the return-time process {km } are connected via the successive-returns formula, (16), and the concatenation representation, (17) with initial block th(1) = OT-1 1, where r = R — j 1 and j is uniformly distributed on [1, The distribution of r can be found by noting that

r (x) = jx E from which it follows that 00

Prob(r = i) =E p.,(T 4-1 n=i

=

E w (C ) . ni

The latter sum, however, is equal to Prob(x E B and n(x) > i), which gives the alternative form 1 Prob (r = i) = (18) Prob (n(x) ijx E B) , E(n(x)lx E B) since E(n(x)lx E B) = 1/ p.(B) holds for ergodic processes, by (13). The formula (18) was derived in [11, Section XIII.5] for renewal processes by using generating functions. As the preceding argument shows, it is a very general and quite simple result about ergodic binary processes with finite expected return time. The return-time process keeps track of returns to the base, but may lose information about what is happening outside B. Another process, called the induced process, carries along such information. Let A* be the set all finite sequences drawn from A and define the countable partition 13 = {P: w E A* } of B, where, for w =

=

=

E Bk: T i-1 X E

Pao 1 <j

kl.

The (?,)-process is called the process induced by the (T, P)-process on the set B. The relation between the (T, P)-process { X, } and its induced (T, f5)-process {km } parallels the relationship (17) between the generalized renewal process and the returntime process. Thus, given a random sample path X = {W„,} for the induced process, a random sample path x = {X,} for the (T, P)-process can be expressed as a concatenation

(19)

x = t-v(1)w(2)w(3) • • • ,

into blocks, where, for m > 1, w(m) = X m , and the initial block th(1) is a tail of the block w(1) = X i . Again, the only real problem is how to choose the start position in w(1) in such a way that the process is stationary, in other words how to determine the length of the tail fi)(1). A generalization of the return-time picture provides the answer. The generalized version of the return-time picture is obtained by further partitioning the columns of the return-time picture, Figure 1.2.18, into subcolumns according to the conditional distribution of (T, P)-names. Thus the column with base Bk is partitioned

CHAPTER I. BASIC CONCEPTS.

28

into subcolumns CL, labeled by k-length names w, so that the subcolumn corresponding to w = a k has width (ct, T - i+ 1 Pa n BO, ,

the ith-level being labeled by a,. Furthermore, by reassembling within each level of a subcolumn, it can again be assumed that T moves points directly upwards one level. Points in the top level of each subcolumn are moved by T to the base B in some manner unspecified by the picture. For example, in Figure 1.2.21, the fact that the third subcolumn over B3, which has levels labeled 1, 1, 0, is twice the width of the second subcolumn, indicates that given that a return occurs in three steps, it is twice as likely to visit 1, 1, 0 as it is to visit 1, 0, 0 in its next three steps. o o I 0

I

o .

1

I I

1

I

I I

1o 1

1

I 0

I I 1

1

1Ii

o

10

I 0

I

o i

I I

1

I I 1 I

o

11

o

B3

B4

I

1

B1

I 1 I

o i

I

B2

Figure 1.2.21 The generalized return-time picture. The start position problem is solved by the generalized return-time picture and the definition of the (T, 2)-process. The (T, 2)-process is defined by selecting x E X at random according to the ii-measure, then setting X, (x) = a if and only if 7' 1 x E Pa . In the ergodic case, the generalized return-time picture is a representation of X, hence selecting x at random is the same as selecting a random point in the return-time picture. But this, in turn, is the same as selecting a column Cu, at random according to column measure p.(Cu,), then selecting j at random according to the uniform distribution on {1, 2, ... , h(w), where h(w) = h(C), then selecting x'at random in the j-th level of C. This proves the following theorem. Theorem 1.2.22 The process {X,..,} has the stationary concatenation representation (19) in terms of the induced process {54}, if and only if the initial block is th(1) = w(1)ih. (w(l)) , where j is uniformly distributed on [1, h(w(1))].

A special case of the induced process occurs when B is one of the sets of P, say B = Bb. In this case, the induced process outputs the blocks that occur between successive occurrences of the symbol b. In general case, however, knowledge of where the blocking occurs may require knowledge of the entire past and future, or may even be fully hidden.

I.2.c.2 The tower construction. The return-time construction has an inverse construction, called the tower construction, described as follows. Let T be a measure-preserving mapping on the probability

29

SECTION 1.2. THE ERGODIC THEORY MODEL.

space (X, E, p.), and let f be a measurable mapping from X into the positive integers Ar = { 1, 2, ...}, of finite expected value. Let k- be the subset of the product space X x Ar defined by = {(x , 1 < i f (x)}. A useful picture is obtained by defining Bn = {x: f (x) = n}, and for each n > 1, stacking the sets Bn x {1}, B x {2}, Bn x {n} (20) as a column in order of increasing i, as shown in Figure 1.2.23.

Tx B3 X

B2 X

Bi X {0}

111

{2}

B3 X

B2 X {0}

B3 X {0}

Figure 1.2.23 The tower transformation. The measure it is extended to a measure on X, by thinking of each set in (20) as a copy of Bn with measure A(B), then normalizing to get a probability measure, the normalizing factor being

n ( B n ) = E(f), n=1 the expected value of f. In other words, B C X is defined to be measurable if and only if (x, i) E T8} n Bn E E, n > 1, i < n, with its measure given by 2-, 0

‘

Tt(k) =

,u({x:

, E 131 )

n 13,0 .

E (f ) n=1 i=1

A transformation T is obtained by mapping points upwards, with points in the top of each column sent back to the base according to T. The formal definition is

T (x, i) =

+ 1) i < f (x) i = f (x). (T x , 0)

The transformation T is easily shown to preserve the measure Ti and to be invertible (and ergodic) if T is invertible (and ergodic). The transformation T is called the tower extension of T by the (height) function f. The tower extension T induces the original transformation T on the base B = X x {0}. This and other ways in which inducing and tower extensions are inverse operations are explored in the exercises. As an application of the tower construction it will be shown that return-time processes are really just stationary positive-integer valued processes with finite expected value.

CHAPTER I. BASIC CONCEPTS.

30

Theorem 1.2.24 An ergodic, positive-integer valued process {kJ} with E(R 1 ) < oc is the return-time process for some transformation and set. In particular, fkil defines a stationary, binary process via the concatenation-representation (17), with the distribution of the waiting time r until the first I occurs given by (18). Proof Let p, be the two-sided Kolmogorov measure on the space .Ar z of doubly-infinite positive-integer valued sequences, and note that the shift T on Aîz is invertible and ergodic and the function f (x) =X has finite expectation. Let T be the tower extension of T by the function f and let B = x x {1}. A point (x, 1) returns to B = X x L1} at the point (T x , 1) at time f (x), that is, the return-time process defined by T and B has the same distribution as {k1 }. This proves the theorem. A stationary process {X,} is regenerative if there is a renewal process {R,} such that the future joint process {(Xn, Ra ):n > t} is independent of the past joint process {(X, n , R,n ): n < t}, given that Rt = 1. The induced transformation and the tower construction can be used to prove the following theorem, see Exercise 24. (Recall that if Y, = f (X n ), for all n, then {Yn } is an instantaneous function of {Xn}.)

Theorem 1.2.25 An ergodic process is regenerative if and only if it is an instantaneous function of an induced process whose return-time process is

I.2.d

Exercises.

1. Complete the proof of Lemma 1.2.1. 2. Let T be an invertible measure-preserving transformation and P = {Pa : a E Al a finite partition of the probability space (X, E, /2). Show that there is a measurable mapping F: X 1-± A z which carries p. onto the (two-sided) Kolmogorov measure y of the (T, P)-process, such that F(Tx) = TAF(x), where TA is the shift on A z . (Thus, a partition gives rise to a stationary coding.) 3. Suppose F: A z F--> Bz is a stationary coding which carries p, onto v. Determine a partition P of A z such that y is the Kolmogorov measure of the (TA, P)-process. (Thus, a stationary coding gives rise to a partition.) 4. Let T be a measure-preserving transformation. A nonnegative measurable function f is said to be T-subinvariant, if f(Tx) < f (x). Show that a T-subinvariant function is almost-surely invariant. (Hint: consider g(x) = min{ f (x), N}.) 5. Show that the (T, P)-process can be ergodic even if T is not. (Hint: start with a suitable nonergodic Markov chain and lump states to produce an ergodic chain, then use Exercise 2.) 6. Show that the (T, P)-process can be mixing even if T is only ergodic. 7. Let Zn = ( X (n -1 )N+19 41-1)N+2, , XnN) be the nonoverlapping N-block process, let Wn = (Xn, Xn+19 • • • Xn+N-1) be the overlapping N-block process, and let Yn = X(n-1)N+1 be the N-th term process defined by the (T, P)-process {Xn}. (a) Show that {Zn } is the (TN,

P-IP)-process.

31

SECTION 1.2. THE ERGODIC THEORY MODEL. (b) Show that {147, } is the (T, v7P -1 P)-process. (c) Show that {Y } is the (T N , 'P)-process.

8. Let T be the shift on X = {-1, 1} Z , and let ,u be the product measure on X defined by p,(-1) = p,(1) = 1/2. Let Y=Xx X, with the product measure y = p, x and define S(x, y) = (Tx, Txoy). (a) Show that S preserves v. (b) Let P be the time-zero partition of X and let Q = P x P. Give a formula expressing the (S, Q)-name of (x, y) in terms of the coordinates of x and y. (The (S, Q)-process is called "random walk with random scenery.")

9. Generalize the preceding exercise by showing that if T is a measure-preserving transformation on a probability space (X, p,) and {Tx : x c X) is a family of measure-preserving transformations on a probability space (Y, y), such that the mapping (x, y) i-± (Tx, T ry) is measurable, then S(x , y) = (Tx, Try) is a measure-preserving mapping on the product space (X x Y, x y). S is called a skew product. If T, = R, for every x, then S is called the direct product of T and R.

10. Show that a concatenated-block process is ergodic. 11. Show that the overlapping n-blocking of an ergodic process is ergodic. 12. Show that the nonoverlapping n-blocking of an ergodic process may fail to be ergodic. 13. Insertion of random spacers between blocks can be used to change a concatenatedblock process into a mixing process. Fix a measure p, on AN, a symbol a E A N , and number p E (0, 1). Let {Z,n } be the stationary Markov chain with alphabet S = A N x {0, 1, 2, ... , N}, defined by the following transition rules. (i) If i < N, (ar , i) can only go to (ar , i

1).

(ii) (ar,, N) goes to (br , 0) with probability p ite(br ), for each br E AN. (iii) (ar , N) goes to (br , 1) with probability (1 — p),u(br), for each br E AN. (a) Show that there is indeed a unique stationary Markov chain with these properties. (Hint: the transition matrix is irreducible.) (b) Show that the process {} defined by setting 4, = a if I'm = (ar , 0) and Ym = ai , if Y„, = (ar , j), j > 0, is a mixing finite-state process.

14. Show that if T is mixing then it is totally ergodic. 15. Show that an ergodic rotation is not mixing, but is totally ergodic. 16. Show that if T is mixing then so is the direct product T x T: X x X where X x X is given the measure ,u x ,u.

X x X,

17. Show directly, that is, without using Proposition 6, that if a is irrational then the direct product (x, y) 1-± (x a, y ± a) is not ergodic.

CHAPTER I. BASIC CONCEPTS.

32

18. Prove that even if T is not invertible, the induced transformation is measurable, preserves the conditional measure, and is ergodic if T is ergodic. (Hint: obtain a picture like Figure 1.2.18 for T -1 , then use this to guide a proof.) 19. Prove that a stationary coding of a mixing process is mixing. (Hint: use the formula F-1 (C n T ' D) = F-1 C n TA- i F -1 D.) 20. Show that if {X,} is a binary, ergodic process which is not identically 0, then it is the generalized renewal process for some transformation T and set B. (Hint: let T be the shift in the two-sided Kolmogorov representation and take B = Ix: xo = 1 1.) 21. Prove: a tower i'' over T induces on the base a transformation isomorphic to T. 22. Show that the tower defined by the induced transformation function n(x) is isomorphic to T. ?

I' and return-time

23. Let T be an invertible measure-preserving transformation and P = {Pa : a E A} a finite partition of the probability space (X, E, pt). Let X be the tower over X defined by f, and let S = T be the tower transformation. Show how to extend P to a partition 2 of X, such that the (T, P)-process is an instantaneous coding of the induced (,?, d)-process. 24. Prove Theorem 1.2.25. 25. Define P = {Pa : a E AI and 2 = {Qb: b Ea,b lit(Pa n Qb) — it(Pa)P,(Qb)1 ._ E.

E

BI to be 6-independent if

(a) Show that p and 2 are independent if and only if they are 6-independent for each 6 > 0. (b) Show that if P and 2 are 6-independent then Ea iii(Pa I Qb) — 11 (Pa)i < Ag, except for a set of Qb's of total measure at most ,g. (c) Show that if Ea lia(PaiQb) — ii(Pa)i < E, except for a set of Qb's of total measure at most 6, then P and 2 are 3E-independent.

SECTION 1.3. THE ERGODIC THEOREM.

33

Section 1.3 The ergodic theorem. The ergodic theorem extends the strong law of large numbers from i.i.d. processes to the general class of stationary processes. Theorem 1.3.1 (The ergodic theorem.) If T is a measure-preserving transformation on a probability space (X, E, p,) and if f is integrable then the average (1/n) EL, f (T i-1 x) converges almost surely and in L 1 -norm to a T -invariant function f*(x). The ergodic theorem in the almost-sure form presented here is due to G. D. Birkhoff and is often called Birkhoff's ergodic theorem or the individual ergodic theorem. The L'-convergence implies that f f* d 1a = f f d ia, since

f1- E n f(T i-l x) d,u = 1- E f f(T i-l x)d,u = f f n

n i=1

by the measure-preserving property of T. Thus if T is ergodic then the limit function f*, since it is T-invariant, must be almost-surely equal to the constant value f f d,u,. In particular, if f is taken to be 0,1-valued and T to be ergodic the following form of the ergodic theorem is obtained. Theorem 1.3.2 (Ergodic theorem: binary ergodic form.) If {X n } is a binary ergodic process then the average (X 1 +X2+. ..+Xn )In converges almost surely to the constant value E(X 1 ).This binary version of the ergodic theorem is sufficient for many results in this book, but the more general version will also be needed and is quite useful in many situations not treated in this book. The proof of the ergodic theorem will be based on a rather simple combinatorial result discussed in the next subsection. The combinatorial idea is not a merely step on the way to the ergodic theorem, however, for it is an important tool in its own right and will be used frequently in later parts of this book.

I.3.a Packings from coverings. A general technique for extracting "almost packings" of integer intervals from certain kinds of "coverings" of the natural numbers will be described in this subsection. In this discussion intervals are subsets of the natural numbers Ar = {1, 2, ...}, of the form [n, m] = (j E n < j < m). A strong cover C of Af is defined by an integer-valued function n m(n) for which m(n) > n, and consists of all intervals of the form [n, m(n)], n E H. (The word "strong" is used to indicate that every natural number is required to be the left endpoint of a member of the cover.) A strong cover C has a packing property, namely, there is a subcover C' whose members are disjoint. This is a trivial observation; just set C' = {[n,, m(n)]}, where n1 = 1 and n i±i = 1 + m(n), i > 1. The finite problem has a different character, for, unless the function m(n) is severely restricted, it may not be possible, even asymptotically, to pack an initial segment [1, K] by disjoint subcollections of a given strong cover C of the natural numbers. If it is only required, however, there be a disjoint subcollection that fills most of [1, K], then a positive and useful result is possible.

CHAPTER I. BASIC CONCEPTS.

34

To motivate the positive result, suppose all the intervals of C have the same length, say L, and apply a sequential greedy algorithm, that is, start from the left and select successive disjoint intervals, stopping when within L of the end. This produces the disjoint collection C' = {[iL +1,(i +1)L]: 0 < i < (K — L)IL} which covers all but at most the final L — 1 members of [1, K]. In particular, if 6 > 0 is given and K > LIS then all but at most a 6-fraction is covered. The desired positive result is just an extension of this idea to the case when most of the intervals have length bounded by some L L}1 < 6. A collection of subintervals C' of the interval [1, K] is called a (1 —6)-packing of [1, K] if the intervals in C' are disjoint and their union has cardinality at least (1 — . Lemma 1.3.3 (The packing lemma.) Let C be a strong cover of N and let S > 0 be given. If K > LIS and if [1, K] is (L, 8)-strongly-covered by C, then there is a subcollection C' c C which is a (1 — 26)packing of [1, K]. Proof By hypothesis K > LIS and (1)

iin E

K]: m(n) — n +1 > L11 < SK.

The construction of the (1 — 26)-packing will proceed sequentially from left to right, selecting the first interval of length no more than L that is disjoint from the previous selections, stopping when within L of the end of [1, K]. To carry this out rigorously set m(0) = no = 0 and, by induction, define n i = minfj E [1+ m(n i _i), K — L]: m(j) — j +1 < L}.

The construction stops after I = I (C, K) steps if m(n i ) > K — L or there is no j E[1 ±m(ni), K — L] for which m(j) — j + 1 < L. The claim is that C' = {[n i ,m(n i )]: 1 < i < I} is a (1 — 26)-packing of [1, K]. The intervals are disjoint, by construction, and are contained in [1, K], since m(n i )—n i +1 < L, and hence m(n i ) < K. Thus it is only necessary to show that the union 1,1 of the [ni ,m(n i )] has length at least (1 — 25)K. The interval (K — L, K] has length at most L — 1 < SK, so that I(K —L,K]—U]I< 6K. For the interval [1, K — L], the definition of the n i implies the following fact. (2)

If j E [I, K — L] —Li then m(j) — j + 1 > L.

The (L, 6)-strong-cover assumption, (1), thus guarantees that 1[1, K — L]-1,ll< 6K. This completes the proof of the packing lemma.

35

SECTION 1.3. THE ERGODIC THEOREM.

Remark 1.3.4 In most applications of the packing lemma all that matters is the result, but in the proof of the general ergodic theorem use will be made of the explicit construction given in the proof of Lemma 1.3.3, including the description (2) of those indices that are neither within L of the end of the interval nor are contained in sets of the packing. Some extensions of the packing lemma will be mentioned in the exercises.

I.3.b

The binary, ergodic process proof.

A proof of the ergodic theorem for the binary ergodic process case will be given first, as it illustrates the ideas in simplest form. In this case ,u, is invariant and ergodic with respect to the shift T on {0, 1}, and the goal is to show that i

n

lirn — x i = ,a(1), a.s, n—*øo n i=1

(3)

where A(1) = A{x: x 1 = 1} = E(X1). Suppose (3) is false. Then either the limit superior of the averages is too large on a set of positive measure or the limit inferior is too small on a set of positive measure. Without loss of generality the first of these possibilities can be assumed, and hence there is an E > 0 such that the set {n

1

B = x : lirn sup — E xi > ,u, ( 1 ) n-+oo n i=i

+E

has positive measure. Since lim sup

xi +

+ xn

= =sup

x2 + .. • + Xn+1

n—>oo

n—>oo

the set B is T-invariant and therefore p.(B) = 1. (Here is where the ergodicity assumption is used.) Now suppose x E B. Since B is T-invariant, for each integer n there will be a first integer m(n) > n such that X n Xn+1 + • • • + Xm(n)

m(n) — n + 1

> p,(1) + E.

Thus the collection C(x) = lln, m(n)]: n E JVI is a (random) strong cover of the natural numbers .A'. Furthermore, each interval in C(x) has the property that the average of the xi over that interval is too big by a fixed amount. These intervals overlap, however, so it is not easy to see what happens to the average over a fixed, large interval. The packing lemma can be used to reduce the problem to a nonoverlapping interval problem, for, combined with a simple observation about almost-surely finite variables, it will imply that if K is large enough then with high probability, most of the terms xi , i < K are contained in disjoint intervals over which the average is too big. But, if the average over most such disjoint blocks is too big, then the average over the entire interval [1, K] must also be too big by a somewhat smaller fixed amount. Since this occurs with high probability it implies that the expected value of the average over the entire set A K must be too big, which is a contradiction.

36

CHAPTER I. BASIC CONCEPTS.

A bit of preparation is needed before the packing lemma can be applied. First, since the random variable m(1) is almost-surely finite, it is bounded except for a set of small probability. Thus given 3 > 0 there is a number L such that if D = {x: m(1) > L}, then ,u(D) LI 8 and x E GK then there is a subcollection Cf (x) = {[ni,m(ni)]: i < I (x)} C C(X) which is a (1 - 23)-packing of [1, K]. Since the xi. are nonnegative, by assumption, and since the intervals in C'(x) are disjoint, the sum over the intervals in C'(x) lower bounds the sum over the entire interval, that is, 1(x) m(n)

x; >

(4) j=1

(l

_ 28)K [p,(1) + c].

i=1 j_—n 1

Note that while the collection C(x) and subcollection C'(x) both depend on x, the lower bound is independent of x, as long as x E GK. Thus, taking expected values yields (K E iE =1 •x

> (1 - 26)K ku(1) + El u(GK) > (1 - 26)(1

(5)K [p.(1)+ c] ,

so that 1 K p. (1) = E (—Ex .) > (1 - 23)(1 6)[,a(1) + c], J=1 which cannot be true for all S. This completes the proof of (3), and thereby the proof of Theorem 1.3.2, the binary, ergodic process form of the ergodic theorem.

I.3.c

The proof in the general case.

The preceding argument will now be extended to obtain the general ergodic theorem. The following lemma generalizes the essence of what was actually proved; it will be used to obtain the general theorem. Lemma 1.3.5 Let T be a measure-preserving transformation on (X, E, ,a), let f let a be an arbitrary real number, and define the set 1 n B = {x: lim sup - E f (T i-1 x) > al. n i=i Then fB f (x) d(x) > a g(B).

E

L i (X, E, ,u),

SECTION 1.3. THE ERGODIC THEOREM.

37

Proof Note that in the special case where the process is ergodic, where f is the indicator function of a set C and a = p(C) c, this lemma is essentially what was just proved. The same argument can be used to show that Lemma 1.3.5 is true in the ergodic case for bounded functions. Only a bit more is needed to handle the general case, where f is allowed to be unbounded, though integrable. The lemma is clearly true if p(B) = 0, so it can be assumed that ,i(B) > O. The set B is T-invariant and the restriction of T to it preserves the conditional measure ,u(. 1B). Thus, it is enough to prove

fB f (x) c 1 ii.(x1B) > a. For x E B and n E A1 there is a least integer m(n) > n such that Emi=(nn ) f (T i-1

(5)

m(n)—n+1

> a.

Since B is T-invariant, the collection C(x) = fin, m(n)]: n E AO is a (random) strong cover of the natural numbers Js/. As before, given 3 > 0 there is an L such that if D=

E B: m(1) > L }

then ,u(D1B) LIB then the packing lemma can be applied and the argument used to prove (4) yields a lower bound of the form 1(x) m(ni)

E f( Ti-lx )

R(K , x) +

j=1 (6)

E E f (T i-1 x) i=1 j=ni

>

R(K , + (1 — 28)K a,

where

E

R(K , x) =

f (T x)

jE[1,1q—U[n,,m(n,)]

is the sum of f(Ti -l x) over the indices that do not belong to the [ne,m(n i )] intervals. (Keep in mind that the collection {[ni,m(ni)]} depends on x.) In the bounded case, say 1f(x)1 < M, the sum R(K , x) is bounded from below by —2M KS and the earlier argument produces the desired conclusion. In the unbounded case, a bit more care is needed to control the effect of R(K, x) as well as the effect of integrating over GK in place of X. An integration, together with the lower bound (6), does give 1 v-IC ■

f (x) dia(x1B) = (7)

a'

B

i=i

(1 — 8)a A(G KIB)

II -I- 12 + 13,

CHAPTER I. BASIC CONCEPTS.

38 where

K fB—Gic j=1 12 =

1 ___ K

f (Ti -1 x) c 1 ii,(x1B), E GK jE[1,K— L] —U[ni,M(ri,)]

1

E

— K

f(Ti -l x) dp,(x1B),

GK

f13= (T i-1 x) d,u(xIB).

jE(K—L,K]—U[ni,m(n,)]

The measure-preserving property of T gives

f3-Gic f (T ./ x) clitt(xIB) = fT_J(B-GK) f(x) Since f E L I and since all the sets T -i(B — GK) have the same measure, which is upper bounded by 8, all of the terms in 11, and hence 11 itself, will be small if 8 is small enough. The integral 13 is also easy to bound for it satisfies

f

1

dp,(xiB),

1131 15- k- B jE(K—L,K]

and hence the measure-preserving property of T yields 1131

B

1 fl(x) dp,(xim,

which is small if K is large enough, for any fixed L. To bound 12, recall from the proof of the packing lemma, see (2), that if j E [1, K — L] — m(ni)] then m(j) — j 1 > L. In the current setting this translates into the statement that if j E [1, K — L] — U[ni , m(n)] then Ti -l x E D. Thus 1 f v—N K

1121 5_ --17

. .B

xD(T' - x)1 f (P -1 x)I dwxiB),

and the measure-preserving property of T yields the bound

1121 5_

f D1 f (x)1 clia(x1B).

which is small if L is large enough. In summary, in the inequality

L

f(x) d ii(x B) ?. (1 —

kt(G

+ 11 +

± 13,

see (7), the final three terms can all be made arbitrarily small, so that passage to the appropriate limit shows that indeed fB f (x) dp,(x1B) > a, which completes the proof of Lemma 1.3.5.

39

SECTION 1.3. THE ERGODIC THEOREM.

Proof of the Birkhoff ergodic theorem: general case. To prove almost-sure convergence first note that Lemma 1.3.5 remains true for any T-invariant subset of B. Thus if

E

E

1 n 1 n f (T i x) C = x: lim inf — f (T i x) y > O. For almost every x, there is an N = N (x, r) such that if n > N, then [1, n] has a y -packing C' c C(x, r)

Lemma 1.3.9 (The two-packings lemma.) Let {[ui, vd: i E I} be a disjoint collection of subintervals of [1, K] of total length aK, and let {[si , ti]: j E J} be another disjoint collection of subintervals of [1, K] of total length 13 K . Suppose each [ui , v,] has length m, and each [s1 , ti ] has length at least M, where M > m13. Let I* be the set of indices i E I such that [ui, yi] meets at least one of the [s i , ti ]. Then the disjoint collection {[s i , ti ]: j E J } U flub i E I*1 has total length at least (a + p - 23)K.

I.3.e

Exercises.

1. A (1 — 3)-packing C' of [1, n] is said to be separated if there is at least one integer between each interval in C'. Let p, be an ergodic measure on A', and let r be an almost-surely finite stopping time which is almost-surely bounded below by a positive integer M > 1/8. Show that for almost every x, there is an integer N= x) such that if n > N then [1, n] has a separated (1 — 28)-packing C' c C(x, r). (Hint: define F'(x) = r(x) + 1 and apply the ergodic stopping-time lemma to F.) 2. Show that a separated (1 — S)-packing of [1, n] is determined by its complement. Show that this need not be true if some intervals are not separated by at least one integer. 3. Prove the partial-packing lemma. 4. Formulate the partial-packing lemma as a strictly combinatorial lemma about the natural numbers. 5. Prove the two-packings lemma. (Hint: use the fact that the cardinality of J is at most K I M to estimate the total length of those [u, , yi ] that meet the boundary of some [si, ti].)

CHAPTER I. BASIC CONCEPTS.

42

6. Show that if r is an almost-surely finite stopping time for a stationary process p, and G n (r , 8) is the set of all x such that [1, n ] is (1 — (5)-packed by intervals from oc. C(x, r), then ii,(Gn (r, 8)) 1, as n 7. Show that if ti is ergodic and if f is a nonnegative measurable, but not integrable f (T i-1 x) converge almost surely to oc. function, then the averages (1/N)

E7

f con8. The mean ergodic theorem of von Neumann asserts that (1/n) EL I verges in L2-norm, for any L 2 function f. Assume T is invertible so that UT is unitary. (a) Show that the theorem is true for f E F = If E L 2 : UT f = f} and for f E .A4 = (I — UT)L 2 . (b) Show that .F+M is dense in L 2 . (Hint: show that its orthogonal complement is 0.) (c) Deduce von Neumann's theorem from the two preceding results. (d) Deduce that L' -convergenceholds, if f

E

LI .

(e) Extend the preceding arguments to the case when it is only assumed that T is measure preserving.

9. Assume T is measure preserving and f is integrable, and for each n, let Un be the set of x's for whichEL-01 f (T i x) > 0. Fix N and let E = Un 0. The ideas of the following proof are due to R.Jones, [33]. (a) For bounded f, show that r(x)-1

E i=o

f (T i x)xE(T i x) ?_ 0, for some r(x)
N, show that E Un ,

E f (T i x)xE(T i x) i=o

— (N —

fII.

(Hint: sum from 0 to r(x) — 1, then from r(x) to r(T) — 1, continuing until within N of L.) (c) Show that the theorem holds for bounded functions. (d) Show that the theorem holds for integrable functions. (e) For B = {x: sup E71-01 f (T i x) > a}, show that h f da a,u(B).

10. Assume T is measure preserving and { g,} is a sequence of integrable functions such that gn+,,(x) < g(x) g ni (Tn x). Kingman's subadditive ergodic theorem asserts that gn (x)/ n converges almost surely to an invariant function g(x) > —oc. The ideas of the following proof are due to M.Steele, [80]. (a) Prove the theorem under the additional assumption that gn < 0. (Hint: first show that g(x) = liminf g n (x)In is an invariant function, then apply packing.)

43

SECTION 1.4. FREQUENCIES OF FINITE BLOCKS.

(b) Prove the theorem by reducing it to the case when g„ < O. g 1 (x) = gni (x) — ET-1 gi (T i x ).)

d,u < oc, then convergence is also in L 1 -norm,

(c) Show that if a = inf, f and a = fg d

(d) Show that the same conclusions hold if gn < 0 and gn+8+,n (x) < cc. g„,(Tn+gx) + y8 , where y8 0 as g

Section 1.4

(Hint: let

+

Frequencies of finite blocks.

An important consequence of the ergodic theorem for stationary finite alphabet processes is that relative frequencies of overlapping k-blocks in n-blocks converge almost surely as n —> oc. In the ergodic case, limiting relative frequencies converge almost surely to the corresponding probabilities, a property characterizing ergodic processes. In the nonergodic case, the existence of limiting frequencies allows a stationary process to be represented as an average of ergodic processes. These and related ideas will be discussed in this section.

I.4.a

Frequencies for ergodic processes.

The frequency of the block ail' in the sequence Jet' is defined for n > k by the formula n—k+1 Palic Wiz =

E i=1

x fa

i

l

(T i-1 x),

where x [4] denotes the indicator function of the cylinder set [an. The frequency can also be expressed in the alternate form

f(4x) = I{i

E

[1, n — k + 1]:

=

where I • I denotes cardinality, that is, f(a14) is obtained by sliding a window of length k along x'it and counting the number of times af is seen in the window. The relative frequency is defined by dividing the frequency by the maximum possible number of occurrences, n — k 1, to obtain

Pk(aflx7) =

f (4x)

= I{i E [1, n — k + 1]: xi +k-1 = 4}1

n—k+1

n—k1

If x and k < n are fixed the relative frequency defines a measure pk • 14) on A " , called the empirical distribution of overlapping k-blocks, or the (overlapping) k-type of x. When k is understood, the subscript k on pk(ali( 14) may be omitted. The limiting (relative) frequency of di in the infinite sequence x is defined by

(1)

p(ainx) = lim p(ainx), oo

provided, of course, that this limit exists. A sequence x is said to be (frequency) typical for the process p. if each block cif appears in x with limiting relative frequency equal to 11,(4). Thus, x is typical for tt if for each k the empirical distribution of each k-block converges to its theoretical probability p,(4).

44

CHAPTER I. BASIC CONCEPTS.

Let T(A) denote the set of sequences that are frequency typical for A. A basic result for ergodic processes is that almost every sequence is typical. Thus the entire structure of an ergodic process can be almost surely recovered merely by observing limiting relative frequencies along a single sample path. Theorem 1.4.1 (The typical-sequence theorem.) If A is ergodic then A(T (A)) = 1, that is, for almost every x,

lim p(aIN) = for all k and all a.

Proof In the ergodic case, n—k+1

p(afixiz) -

' —1

X •_k • (T i ' ul'

n - k +1 E

x),

which, by the ergodic theorem, converges almost surely to f x [4] dp, = p(a), for fixed 4. In other words, there is a set B(a) of measure 0, such that p(alicixtiz) -> 1),(4), for x B(a lic), as n -> co. Since there are only a countable number of possible af, the set 00

B(a 1`)

B=U k=1 aki EAk

has measure 0 and convergence holds for all k and all al', for all measure 1. This proves the theorem.

XE

A

- B, a set of [=:]

The converse of the preceding theorem is also true. Theorem 1.4.2 (The typical-sequence converse.) Suppose A is a stationary measure such that for each k and for each block di', the limiting relative frequencies p(ali`Ix) exist and are constant in x, p.-almost surely. Then A is an ergodic measure.

Proof If the limiting relative frequency p(a'nx) exists and is a constant c, with probability 1, then this constant c must equal p,(4), since the ergodic theorem includes L I -convergence of the averages to a function with the same mean. Thus the hypotheses imply that the formula x--, n liM — n

i=1

x8

=

holds for almost all x for any cylinder set B. Multiplying each side of this equation by the indicator function xc (x), where C is an arbitrary measurable set, produces n

liM n—)00

fl

E X B (T i-l x)X c (x)= i=1

and an integration then yields

(2)

E n

lim A(T -i B n i=1

n=

45

SECTION 1.4. FREQUENCIES OF FINITE BLOCKS.

The latter holds for any cylinder set B and measurable set C. The usual approximation argument then shows that formula (2) holds for any measurable sets B and C. But if B = B then the formula gives it(B) = A(B) 2 , so that p(B) must be 0 or 1. This proves Theorem 1.4.2. Note that as part of the preceding proof the following characterizations of ergodicity were established. Theorem 1.4.3 The following are equivalent. (i) T is ergodic. (ii) lima pi,-

x ,(Ti-lx) = tt(E), a.e., for each E in a generating algebra.

(iii) limn

A(T -i A n B) = ,u(A)A(B), for each A and B ma generating algebra.

Some finite forms of the typical sequence theorems will be discussed next. Sequences of length n can be partitioned into two classes, G, and B, so that those in Gn will have "good" k-block frequencies and the "bad" set Bn has low probability when n is large. The precise definition of the "good" set, which depends on k and a measure of error, is G n (k,

(3)

p(afix) — ,L(a)1 < c, a

6) =

E

The "bad" set is the complement, Bn (k, 6) = An — G n (k, E). The members of the "good" set, G ,(k, c), are often called the frequency-(k, c)-typical sequences, or when k and c are understood, just the typical sequences. Theorem 1.4.4 (The "good set" form of ergodicity.) (a) If ji is ergodic then 4 (b) If lima p.n (G n (k, ergodic.

6))

E

Gn (k,

E),

eventually almost surely, for every k and c > O.

= 1, for every integer k > 0 and every c > 0, then p, is

Proof Part (a) is just a finite version of the typical-sequence theorem, Theorem 1.4.1. The condition that A n (G n (k, E)) converge to 1 for any E > 0 is just the condition that p(41x7) converges in probability to ii(4). Since p(an.q) is bounded and converges almost surely to some limit, the limit must be almost surely constant, and hence (b) follows from Theorem 1.4.2. It is also useful to have the following finite-sequence form of the typical-sequence characterization of ergodic processes. Theorem 1.4.5 (The finite form of ergodicity.) A measure IL is ergodic if and only if for each block ail' and E > 0 there is an N = N(a, c) such that if n > N then there is a collection C, C An such that tt(Cn) > 1 —

(ii) If x,

E.

C, then 1/3(41fi') — p(aini)I < c.

CHAPTER I. BASIC CONCEPTS.

46

Proof If it is ergodic, just set Cn = G n (k, E), the "good" set defined by (3), and use Theorem 1.4.4. To establish the converse fix a il' and E > 0, then define G n = ix: IP(a inxi) — P(a inx)I < EI, and use convergence in probability to select n > N(a, c) such that ,u(an) > 1 — Cn satisfies (i) and (ii) put En = {x: 4 E Cn}

E.

If

and note that ,a(a n n e . n ) > 1 — 2E. Thus, 3E, X,

IP(ailx) — P(ailZ)I

E_ _

n

n El.n ,

and hence p(a Ix) is constant almost surely, proving the theorem.

I.4.b

The ergodic theorem and covering properties.

The ergodic theorem is often used to derive covering properties of finite sample paths for ergodic processes. In these applications, a set Bn c An with desired properties is determined in some way, then the ergodic theorem is applied to show that, eventually oc, there are approximately tt(Bn )M indices i E [1, M — n] for almost surely as M which x: +n-1 E Bn . This simple consequence of the ergodic theorem can be stated as either a limit result or as an approximation result.

Theorem 1.4.6 (The covering theorem.) If 12, is an ergodic process with alphabet A, and if Bn is a subset of

An of positive

measure, then liM Pn(BniXi ) M—).co

1-1 (Bn), a.s.

In other words, for any 8 > 0,

(4)

E [1, M — n]:

E 13n}1 — ML(13n)1 1 — then xr is eventually almost surely (1 — (5)-strongly-covered by r3n.

47

SECTION 1.4. FREQUENCIES OF FINITE BLOCKS.

The almost strong-covering idea and a related almost-packing idea are discussed in more detail in Section L7. The following example illustrates one simple use of almost strong-covering. Note that the ergodic theorem is used twice in the example, first to select the set 13n , and then to obtain almost strong-covering by 13n . Such "doubling" is common in applications. Example 1.4.8 Let p. be an ergodic process and fix a symbol a E A of positive probability. Let F (xr) and L(xr) denote the first and last occurrences of a E Xr, that is,

F(xr) = min{i E [1, M]: x i = a} L(x) = max{i E [1, M]: x i = a}, with the convention F(41) = L(41 ) = 0, if no a appears in xr. The almost-covering idea will be used to show that

lim kr,00

(5)

L(41 )— F(41 )

= 1, almost surely.

To prove this, let

Bn = {a7: ai = a, for some i

E

[1, nil.

Given 8 > 0, choose n so large that A(B n ) > 1 — 3. Such an n exists by the ergodic theorem. The almost-covering principle, Theorem 1.4.7, implies that xr is eventually almost surely (1 — 3)-strongly-covered by Bn . Note, however, that if xr is (1 — 6)strongly-covered by 13n , then at least one member of 13n must start within 3M of the beginning of xr and at least one member of 1:3n must start within 8M of the end of xr. But if this is so and M > n/3, then F(41 ) < 23M and L(xr) > (1 — 3)M, so that L(xr) — F(41) > (1 — 3 8 )M. Since 3 is arbitrary, the desired result, (5), follows. It follows easily from (5), together with the ergodic theorem, that the expected waiting time between occurrences of a along a sample path is, in the limit, almost surely equal to 1/p,(a), see Exercise 2. This result also follows easily from the return-time picture discussed in Section I.2.c.

I.4.c

The ergodic decomposition.

In the stationary nonergodic case, limiting relative frequencies of all orders exist almost surely, but the limit measure varies from sequence to sequence. The sequences that produce the same limit measure can be grouped into classes and the process measure can then be represented as an average of the limit measures, almost all of which will, in fact, be ergodic. These ideas will be made precise in this section. The set of all x for which the limit

p(ainx) = lim p(44) exists for all k and all di will be denoted by E. Thus is the set of all sequences for which all limiting relative frequencies exist for all possible blocks of all possible sizes. The set is shift invariant and any shift-invariant measure has its support in This support result, which is a simple consequence of the ergodic theorem, is stated as the following theorem.

CHAPTER I. BASIC CONCEPTS.

48 Theorem 1.4.9 If it is a shift-invariant measure then A ( C) = 1.

Proof Since 1

p (all' 14) = n ..... k

n —k+ 1

± i E x[ak,,(Ti. — ix),

i=1 the ergodic theorem implies that p(aflx) = limn p(aii`ix) exists almost surely, for each k and 4. Since there are only a countable number of the al` , the complement of ,C must El have measure 0 with respect to 1.1,, which establishes the theorem. Theorem 1.4.9 can be strengthened to assert that a shift-invariant measure kt must actually be supported by the sequences for which the limiting frequencies not only exist but define ergodic measures. To make this assertion precise first note that the ergodic theorem gives the formula (6)

,u(alic) = f p(af Ix) dp,(x).

The frequencies p(41.)c) can be thought of as measures, for if x E

r then the formula

,u,x (a) = p(ainx), 4 E Ak defines a measure itx on sequences of length k, for each k, which can be extended, by the Kolmogorov consistency theorem, to a Borel measure i.tx on A. The measure ,ux will be called the (limiting) empirical measure determined by x. Two sequences x and y are said to be frequency-equivalent if /ix = A y . Let n denote the projection onto frequency equivalence classes. The projection x is measurable and transforms the measure ,u onto a measure w = ,u o x -1 on equivalence classes. Next define /27(x) = ktx, which is well-defined on the frequency equivalence class of x. Formula (6) can then be expressed in the form (7)

,u(B) = f 1.1.,(x )(B) dco(7(x)), B E E.

This formula indeed holds for any cylinder set, by (6), and extends by countable additivity to any Borel set B. Let E denote the set of all sequences x E L for which the measure ti,x is ergodic. Of course, each frequency-equivalence class is either entirely contained in E or disjoint from E. Theorem 1.4.9 may now be expressed in the following much stronger form. Theorem 1.4.10 ( The ergodic decomposition theorem.) If p. is shift-invariant then p,(E) = 1 and hence the representation (7) takes the form,

(B)= f p.,( x )(B) da)(7r(x)), BE E.

e In other words, a shift-invariant measure always has its support in the set of sequences whose empirical measures are ergodic. The ergodic measures, ,u, (x), x E E, are called the ergodic components of kt and the formula represents p, as an average of its ergodic components. An excellent discussion of ergodic components and their interpretation in communications theory can be found in [ 1 9 ]. The usual measure theory argument, together with the finite form of ergodicity, Theorem 1.4.5, shows that to prove the ergodic decomposition theorem it is enough to prove the following lemma.

SECTION 1.4. FREQUENCIES OF FINITE BLOCKS.

49

Lemma 1.4.11 Given c > 0 and 4 there is a set X1 with ii,(Xi) > 1 — E, such that for any x E X1 there is an N such that if n > N there is a collection C, c A n , with /L(C) > 1 — c, such that the frequency of occurrence of 4 in any two members of C, differs by no more than E. Proof Fix E > 0, fix a block 4, and fix a sequence y E L, the set of sequences with limiting frequencies of all orders. Let L i = Li(y) denote the set of all x E L such that

(8)

ip(a'nx) — p(41y)l < E/4.

Note, in particular, that the relative frequency of occurrence of al in any two members of Li differ by no more than E/2. oo, hence For each x E Li, the sequence {p(al 14)} converges to p(41x), as n there is an integer N and a set L2 C Li of measure at least (1 — E2 )A(L1) such that

(9)

IP(414) — p(alx)I

< E/4, x E L2, n > N.

Fix n > N and put Cn = {x: x E L2 }. The conditions (9) and (8) guarantee that the relative frequency of occurrence of 4 in any two members x7, itiz of Cn differ by no more E. The set L1 is invariant since p(ainx) = p(4`17' x), for any x E L. Thus the conditional measure it (. ILi) is shift-invariant and

1 ,u(xilLi) = A(CI)

Li

P(xilz) dtt(z), x' iz E An.

In particular, tt(CnICI) =

ti

1 (L1) L1 P(Crilz) d iu(z) > 1 — 6 2

so the Markov inequality yields a set L3 C L2 such that /4L3) > (1 — E)/L(L 1 ) and for which

E p(xz) ?. 1 - E, Z E r3,

.q.c.„

which translates into the statement that p(C) > 1 — E, z E L3. The unit interval [0, 1] is bounded so L can be covered by a finite number of the sets Li(y) and hence the lemma is proved. This completes the proof of the ergodic decomposition theorem, Theorem 1.4.10. 0 Remark 1.4.12 The ergodic decomposition can also be viewed as an application of the ChoquetBishop-deLeeuw theorem of functional analysis, [57]. Indeed, the set Ps of shift-invariant probability measures is compact in the weak*-topology (obtained by thinking of continuous functions as linear functionals on the space of all measures via the mapping it i— f f dit.) The extreme points of Ps correspond exactly to the ergodic measures, see Exercise 1 of Section 1.9.

CHAPTER I. BASIC CONCEPTS.

50

I.4.d

Exercises

1. Let qk • 14) be the empirical distribution of nonoverlapping k-blocks in 4, that is,

qk (alnx;1 ) = I{/ E [0, t): where n = tk

=

r, 0 < r < k.

(a) Show that if T is totally ergodic then qk (ali`14)

,u(x lic), almost surely.

(b) Show that the preceding result is not true if T is not totally ergodic.

2. Combine (5) with the ergodic theorem to establish that for an ergodic process the expected waiting time between occurrences of a symbol a E A is just 111,t(a). (Hint: show that the average time between occurrences of a in 4 is close to 1/pi (a14).) 3. Let {X„: n > 1} be a stationary process. The reversed process {Yn : n > 1} is the , P)-process, where P is the Kolmogorov partition in the two-sided representation of {X,: n > 1}. Show that {Xn } is ergodic if and only its reversed process is ergodic. (Hint: combine Exercise 2 with Theorem 1.4.5.) 4. Use the central limit theorem to show that the "random walk with random scenery" process is ergodic. (Hint: with high probability (x k , z„,m+k ) and (4, zt) are looking at different parts of y. See Exercise 8 in Section 1.2 for a definition of this process.) 5. Let P be the time-0 partition for an ergodic Markov chain with period d = 3. (a) Determine the ergodic components of the (T 3 , P)-process. (b) Determine the ergodic components of the (T 3 , P V TP y T 2P)-process. 6. Show that an ergodic finite-state process is a function of an ergodic Markov chain. (It is an open question whether a mixing finite-state process is a function of a mixing Markov chain.) 7. Let T be an ergodic rotation of the circle and P the two-set partition of the unit circle into upper and lower halves. Describe the ergodic components of the (T x T ,P x P)-process. 8. This exercise illustrates a direct method for transporting a sample-path theorem from an ergodic process to a sample-path theorem for a (nonstationary) function of the process. Fix a E A and let Aa" be the set of all x E [a] such that Tnx E [a], for infinitely R(x) = {R 1 (X), R2(x), .1 by many n. Define the mapping x

(x) = min{m > 1: x,n = a} — 1 R1 (x) =

minfm >

E

(x): x m = a} —

E Rico

.

i=1

(This is just the return-time process associated with the set B = [a].) A measure on Aa" transports to the measure y = ,u o R -1 on Aroo. Assume p. is the conditional measure on [a] defined by an ergodic process. Show that

SECTION 1.5. THE ENTROPY THEOREM. (a) The mapping x

51

R(X) is Borel.

(b) R(T R ' (x ) x) = SR(x), where S is the shift on

(e) If T is the set of the set of frequency-typical sequences for j, then v(R(T)) = 1. (Hint: use the Borel mapping lemma, Lemma 1.1.17.) (d) (1/n) Eni R i -÷ 11 p,(a), v-almost surely. (Hint: use the preceding result.)

Section 1.5 The entropy theorem. For stationary processes, the measure p.(4) is nonincreasing in n, and, except for some interesting cases, has limit 0; see Exercise 4c. The entropy theorem asserts that, for ergodic processes, the decrease is almost surely exponential in n, with a constant exponential rate called the entropy or entropy-rate, of the process, and denoted by h = h(t).

Theorem 1.5.1 (The entropy theorem.) Let it be an ergodic measure for the shift T on the space A', where A is finite. There is a nonnegative number h = h(A) such that

1 lim — log

n oo n

,u,(4)

= h, almost surely.

In the theorem and henceforth, log means the base 2 logarithm, and the natural logarithm will be denoted by ln. The proof of the entropy theorem will use the packing lemma, some counting arguments, and the concept of the entropy of a finite distribution. The entropy of a probability distribution 7(a) on a finite set A is defined by H(z) = —

E 71- (a) log 7(a). aEA

Let

1 1 h n (x) = — log — n ti,(4)

1 log p,(4), n

so that if A n is the measure on An defined by tt then H(t) = nE(hn (x)). The next three lemmas contain the facts about the entropy function and its connection to counting that will be needed to prove the entropy theorem. Lemma 1.5.2 The entropy function H(7) is concave in 7 and attains its maximum value log I AI only for the uniform distribution, z(a) 111AI. Proof An elementary calculus exercise.

0

Lemma 1.5.3 E(17,(x)) _< log I AI. Proof Since 1/(.4) = nE(h n (x)), the lemma follows from the preceding lemma.

CI

CHAPTER I. BASIC CONCEPTS.

52

Lemma 1.5.4 (The combinations bound.) (n ) denotes the number of combinations of n objects taken k at a time and k 3 < 1/2 then n 2nH(3) , k) If

E

k It(x

1 ) and hence h(Tx) < h(x),

that is, h(x) is subinvariant. Thus h(Tx) = h(x), almost surely, since subinvariant functions are almost surely invariant, see Exercise 4, Section 1.2. Since T is assumed to be ergodic the invariant function h(x) must indeed be constant, almost surely. Define h to be the constant such that h = lim inf hn (x), n-).00

almost surely.

The goal is to show that h must be equal to lim sup, hn (x), almost surely. Towards this end, let c be a positive number. The definition of limit inferior implies that for almost

SECTION 1.5. THE ENTROPY THEOREM.

53

all x the inequality h(x) < h + E holds for infinitely many values of n, a fact expressed in exponential form as A(4) >

(1)

infinitely often, almost surely. The goal is to show that "infinitely often, almost surely" can be replaced by "eventually, almost surely," for a suitable multiple of E.

Three ideas are used to complete the proof. The first idea is a packing idea. Eventually almost surely, most of a sample path is filled by disjoint blocks of varying lengths for each of which the inequality (1) holds. This is a simple application of the ergodic stopping-time packing lemma. The second idea is a counting idea. The set of sample paths of length K that can be mostly filled by disjoint, long subblocks for which (1) holds cannot have cardinality exponentially much larger than 2 /"+€) , for large enough K. Indeed, if these subblocks are to be long, then there are not too many ways to specify their locations, and if they mostly fill, then once their locations are specified, there are not too many ways the parts outside the long blocks can be filled. Most important of all is that once locations for these subblocks are specified, then since a location of length n can be filled in at most 2n(h+E) ways if (1) is to hold, there will be a total of at most 2 K(h+E ) ways to fill all the locations. The third idea is a probability idea. If a set of K-length sample paths has cardinality only a bit more than 2K (h+E) , then it is very unlikely that a sample path in the set has probability exponentially much smaller than 2 — K (h+E ) . This is just an application of the fact that upper bounds on cardinality "almost" imply lower bounds on probability, Lemma I.1.18(b).

To fill in the details of the first idea, let S be a positive number and M > 1/B an integer, both to be specified later. For each K > M, let GK(S, M) be the set of all that are (1 — 23)-packed by disjoint blocks of length at least M for which the inequality (1) holds. In other words, xr E G K (8 , M) if and only if there is a collection

xr

S = S(xr) = {[n i , m i [} of disjoint subintervals of [1, K] with the following properties.

(a) m

—

n 1 +1 > M, [ni, m i l E S.

(b) p(x) >

ni+1)(h+c),

[n,, m,] E S.

(c) E [ „,..,,Es (mi — n, + 1) > (1 — 28)K.

An application of the packing lemma produces the following result.

CHAPTER I. BASIC CONCEPTS.

54

Lemma 1.5.5

E

GK

(3, M), eventually almost surely.

Proof Define T. (x) to be the first time n > M such that p,([xrii]) >n(h+E) Since r is measurable and (1) holds infinitely often, almost surely, r is an almost surely finite stopping time. An application of the ergodic stopping-time lemma, Lemma 1.3.7, then yields the lemma. LI The second idea, the counting idea, is expressed as the following lemma.

Lemma 1.5.6 There is a 3 > 0 and an M> 116 such that IG large K.

, M)1 M, for each i, and if it covers all but a 26-fraction of [1, K], that is, (2)

E (mi _ ni ± 1) >_ (1 — 23)K. [n,,m,JES

A sequence xr is said to be compatible with such a skeleton S if 7i ) > 2-(m, --ni+i)(h+e) kt(x n for each i. The bound of the lemma will be obtained by first upper bounding the number of possible skeletons, then upper bounding the number of sequences xr that are compatible with a given skeleton. The product of these two numbers is an upper bound for the cardinality of G K (6, M) and a suitable choice of 6 will then establish the lemma. First note that the requirement that each member of a skeleton S have length at least M, means that Si < KIM, and hence there are at most KIM ways to choose the starting points of the intervals in S. Thus the number of possible skeletons is upper bounded by (3)

E

Kk

H(1/111)

k

must hold for each [ni , m i ] E S. For a given [ni , m i ] the number of ways xnm , can be chosen so that the compatibility condition (4) holds, is upper bounded by 2(m1—n's +1)(h+E) , by the principle that lower bounds on probability imply upper bounds on cardinality, Lemma I.1.18(a). Thus, the number of ways x j can be chosen so that j E Ui [n„ and so that the compatibility conditions hold is upper bounded by

(fl

nK h-1-€) L

55

SECTION 1.5. THE ENTROPY THEOREM.

Outside the union of the [ni , mi] there are no conditions on xi . Since, however, there are fewer than 28K such j these positions can be filled in at most IA 1 26K ways. Thus, there are at most IAl23K 2K (h+E)

sequences compatible with a given skeleton S = {[n i , m i ]}. Combining this with the bound, (3), on the number of possible skeletons yields

1G K(s, m)i

(5)

< 2. 1(- H(1lm) oi ncs 2 K(h+E)

Since the binary entropy function H (1 M) approaches 0 as M 00 and since IA I is finite, the numbers 8 > 0 and M > 1/8 can indeed be chosen so that IGK (8, M) < 0 2K(1 +2' ) , for all sufficiently large K. This completes the proof of Lemma 1.5.6. Fix 6 > 0 and M > 1/6 for which Lemma 1.5.6 holds, put GK = Gl{(8, M), and let BK be the set of all xr for which ,u(x(c ) < 2 -1C(h+36) . Then tt(Bic

n GK) < IGKI 2—K(h+3E)

5 2—ICE

,

holds for all sufficiently large K. Thus, xr g BK n GK, eventually almost surely, by the Borel-Cantelli principle. Since xt E GK, eventually almost surely, the iterated almost-sure principle, Lemma 1.1.15, implies that xr g BK, eventually almost surely, that is,

lim sup h K (x) < h + 3e, a.s. K—>oo

In summary, for each e > 0, h = lim inf h K (x) < lim sup h K (x) < h 3e, a.s., K—*co

which completes the proof of the entropy theorem, since c is arbitrary.

0

Remark 1.5.7 The entropy theorem was first proved for Markov processes by Shannon, with convergence in probability established by McMillan and almost-sure convergence later obtained by Breiman, see [4] for references to these results. In information theory the entropy theorem is called the asymptotic equipartition property, or AEP. In ergodic theory it has been traditionally known as the Shannon-McMillan-Breiman theorem. The more descriptive name "entropy theorem" is used in this book. The proof given is due to Ornstein and Weiss, [51], and appeared as part of their extension of ergodic theory ideas to random fields and general amenable group actions. A slight variant of their proof, based on the separated packing idea discussed in Exercise 1, Section 1.3, appeared in [68].

I.5.b Exercises. 1. Prove the entropy theorem for the i.i.d. case by using the product formula on then taking the logarithm and using the strong law of large numbers. This yields the formula h = — Ea ,u(a) log it(a). 2. Use the idea suggested by the preceding exercise to prove the entropy theorem for ergodic Markov chains. What does it give for the value of h?

CHAPTER I. BASIC CONCEPTS.

56

3. Suppose for each k, Tk is a subset of Ac of cardinality at most 2k". A sequence 4 is said to be (K, 8, {T})-packed if it can be expressed as the concatenation 4 = w(1) . w(t), such that the sum of the lengths of the w(i) which belong to Ur_K 'Tk is at least (1 —8)n. Let G„ be the set of all (K, 8, {T})-packed sequences 4 and let E be positive number. Show that if K is large enough, if S is small enough, and if n is large enough relative to K and S, then IG n I < 2n(a+E) . 4. Assume p is ergodic and define c(x) = lima

,44), x

E

A°°.

(a) Show that c(x) is almost surely a constant c. (b) Show that if c > 0 then p. is concentrated on a finite set. (c) Show that if p. is mixing then c(x) = 0 for every x.

Section 1.6

Entropy as expected value.

Entropy for ergodic processes, as defined by the entropy theorem, is given by the almost-sure limit 1 1 — log h= n—>oo n Entropy can also be thought of as the limit of the expected value of the random quantity —(1/n) log A(4). The expected value formulation of entropy will be developed in this section.

I.6.a

The entropy of a random variable.

Let X be a finite-valued random variable with distribution defined by p(x) = X E A. The entropy of X is defined as the expected value of the random variable — log p(X), that is,

Prob(X = x),

H(X) =

E p(x) log -1-=— p(x)

xEA

p(x) log p(x). xEA

The logarithm base is 2 and the conventions Olog 0 = 0 and log0 = —oo are used. If p is the distribution of X, then H(p) may be used in place of H(X). For a pair (X, Y) of random variables with a joint distribution p(x, y) = Prob(X = x, Y = y), the notation H(X,Y)-= — p(x , y) log p(x, y) will be used, a notation which extends to random vectors. Most of the useful properties of entropy depend on the concavity of the logarithm function. One way to organize the concavity idea is expressed as follows.

Lemma 1.6.1 If p and q are probability k-vectors then

_E pi log p, < — with equality if and only if p = q.

pi log qi ,

57

SECTION 1.6. ENTROPY AS EXPECTED VALUE.

Proof The natural logarithm is strictly concave so that, ln x < x — 1, with equality if and only if x = 0. Thus qj

with equality if and only if qi = pi , log x = (In x)/(1n 2).

E,

1 < i 5_ k.

_

= 0,

This proves the lemma, since

The proof only requires that q be a sub-probability vector, that is nonnegative with qi 5_ 1. The sum

Dcplio =

pi in

PI qi

is called the (informational) divergence, or cross-entropy, and the preceding lemma is expressed in the following form.

Lemma 1.6.2 (The divergence inequality.) If p is a probability k-vector and q is a sub-probability k-vector then D(pliq) > 0, with equality if and only if p = q. A further generalization of the lemma, called the log-sum inequality, is included in the exercises. The basic inequalities for entropy are summarized in the following theorem.

Theorem 1.6.3 (Entropy inequalities.) (a) Positivity. H(X) > 0, with equality if and only if X is constant.

(b) Boundedness. If X has k values then H(X) < log k, with equality if and only if each p(x) = 11k.

(c) Subadditivity. H(X, Y) < H(X) H(Y), with equality if and only if X and Y are independent. Proof Positivity is easy to prove, while boundedness is obtained from Lemma 1.6.1 by setting Px = P(x), qx ----- 1/k. To establish subadditivity note that H(X,Y).—

Ep(x, y) log p(x, y), X.

y

then replace p(x , y) in the logarithm factor by p(x)p(y), and use Lemma 1.6.1 to obtain the inequality H(X,Y) < H(X)-}- H(Y), with equality if and only if p(x, y) p(x)p(y), that is, if and only if X and Y are independent. This completes the proof of the theorem. El The concept of conditional entropy provides a convenient tool for organizing further results. If p(x , y) is a given joint distribution, with corresponding conditional distribution p(xly) = p(x, y)1 p(y), then H ain

=_

p(x, y) log p(xiy) = —

p(x, y) log

p(x, y)

POO

CHAPTER I. BASIC CONCEPTS.

58

is called the conditional entropy of X, given Y. (Note that this is a slight variation on standard probability language which would call — Ex p(xIy) log p(xly) the conditional entropy. In information theory, however, the common practice is to take expected values with respect to the marginal, p(y), as is done here.) The key identity for conditional entropy is the following addition law.

(1)

H(X, Y) = H(Y)+ H(X1Y).

This is easily proved using the additive property of the logarithm, log ab = log a+log b. The previous unconditional inequalities extend to conditional entropy as follows. (The proofs are left to the reader.)

Theorem 1.6.4 (Conditional entropy inequalities.) (a) Positivity. H (X IY) > 0, with equality if and only if X is a function of Y. (b) Boundedness. H(XIY) < H (X) with equality if and only if X and Y are independent. (c) Subadditivity. H((X, Y)IZ) < H(XIZ)+ H(YIZ), and Y are conditionally independent given Z.

with equality if and only if X

A useful fact is that conditional entropy H(X I Y) increases as more is known about the first variable and decreases as more is known about the second variable, that is, for any functions f and g,

Lemma 1.6.5 H(f(X)In < H(XIY) < 1-1 (X1g(n). The proof follows from the concavity of the logarithm function. This can be done directly (left to the reader), or using the partition formulation of entropy which is developed in the following paragraphs. The entropy of a random variable X really depends only on the partition Px = {Pa: a E A} defined by Pa = fx: X(x) = a), which is called the partition defined by X. The entropy H(1)) is defined as H (X), where X is any random variable such that Px = P. Note that the join Px V Py is just the partition defined by the vector (X, Y) so that H(Px v Pr) = H(X, Y). The conditional entropy of P relative to Q is then defined by H(PIQ) = H(1) V Q) — H (Q). The partition point of view provides a useful geometric framework for interpretation of the inequalities in Lemma 1.6.5, because the partition Px is a refinement of the partition Pf(x), since each atom of Pf(x) is a union of atoms of P. The inequalities in Lemma 1.6.5 are expressed in partition form as follows.

Lemma 1.6.6 (a) If P refines Q then H(QIR) < H(PIR). (b) If R. refines S then H(PIS) > 11(P 1R).

SECTION I.6. ENTROPY AS EXPECTED VALUE.

59

Proof The proof of (a) is accomplished by manipulating with entropy formulas, as follows.

H (P IR)

(i)

(P v 21 1Z) H(QITZ) + H(PIQ v 1Z) H (OR).

The equality (i) follows from the fact that P refines Q, so that P = Pv Q; the equality (ii) is just the general addition law; and the inequality (iii) uses the fact that H (P I VR,) > O. Let Pc denote the partition of the set C defined by restricting the sets in P to C, where the conditional measure, it (. IC) is used on C. To prove (h) it is enough to consider the case when R. is obtained from S by splitting one atom of S into two pieces, say Ra = Sa , a 0 b; Sb = Rb U Rb, • The quantity H(P IR.) can be expressed as

-EEtt(Pt n R a ) log

n Ra) +

it(Pt

(4)11 (Psb liZsb )

t ab

Ewpt n Ra ) log It(Pt n Ra) + it(Sb)1-1(Psb ). II(Ra)

t a0b

The latter is the same as H(PIS), which establishes inequality (b).

I.6.b

The entropy of a process.

The entropy of a process is defined by a suitable passage to the limit. The n-th order entropy of a sequence {X1, X2, ...} of A-valued random variables is defined by

H(r)

= E p(x)log 1 ATEAn

E

=— p(x'iL)log p(4). 11) P(x1 xnEan

The process entropy is defined by passing to a limit, namely,

(2)

1 1 1 H ({X n }) = lim sup — H((X7) = lim sup — p(4) log n n p(4) . n n xn

In the stationary case the limit superior is a limit. This is a consequence of the following basic subadditivity property of nonnegative sequences. Lemma 1.6.7 (The subadditivity lemma.) If {an} is a sequence of nonnegative numbers which is subadditive, that is, an-Fm < an + am , then limn an ln exists and equals infn an /n. Proof Let a = infn an /n. Given c > 0 choose n so that an < n(a ± 6). If m > n write m = np +.r, 0 < r < n. Subadditivity gives anp < pan , so that if b = sup ,n a, then another use of subadditivity yields am < anp + a,. < pan + b.

CHAPTER I. BASIC CONCEPTS.

60

Division by m, and the fact that np/m —> 1, as in —> pc, gives lim sup ani /m < a + E. CI This proves the lemma. The subadditivity property for entropy, Theorem I.6.3(c), gives

H(X7 +m) < H(X7)+ H(rnTin). If the process is stationary then H(X n + -FT) = H(r) so the subadditivity lemma with an = H(X7) implies that the limit superior in (2) is a limit. An alternative formula for the process entropy in the stationary case is obtained by using the general addition law, H(X, Y) = H(Y)± H(X1Y), to produce the formula

H(X ° n )— H(X1,1 )= H(X0IX:n1), n > 0. The right-hand side is decreasing in n, from Lemma 1.6.5, and the simple fact from analysis that if an > 0 and an+ i — an decreases to b then an /n —> b, can then be applied to give H({X n })= lim H (X01X1 nI ) , (3) n--4co

a formula often expressed in the suggestive form

(4)

H({X n }) = H (X01X_1 0 ) .

The next goal is to show that for ergodic processes, the process entropy H is the same as the entropy-rate h of the entropy theorem. Towards this end, the following lemma will be useful in controlling entropy on small sets.

Lemma 1.6.8 (The subset entropy-bound.)

_ E p(a) log p(a) _< p(B) log IBI — p(B)log p(B), aEB

for any B c A. Proof Let p(. 1B) denote the conditional measure on B and use the trivial bound to obtain P(a1B)log p(aIB) ._- _ log IBIaEB —

E

The left-hand side is the same as

1

E P(a)log p(a) + log p(B)) ,

( p(B)

aEB

0

from which the result follows.

Let p, be an ergodic process with alphabet A, and process entropy H, and let h be the entropy-rate of A as given by the entropy theorem, that is

1

1

n

,u (x '0

urn—log n

= h, a.e.

61

SECTION 1.6. ENTROPY AS EXPECTED VALUE.

Theorem 1.6.9 Process entropy H and entropy-rate h are the same for ergodic processes. Proof First assume that h > 0 and fix E such that 0 < E < h. Define G, = {4: 2—n(h+E) < 1,1 (4) < 2—n(h—E)}

and let B, = A n — G. Then

E „,(x ) log ,u(xtil) A" = — E ,u (4 ) log ,u(xliz) — E ,u(4) log p.(4).

H (X7) = —

G,

B,,

The two sums will be estimated separately. The subset entropy-bound, Lemma 1.6.8, gives

(5)

—

E A (x) log ii(x) < np,(Bn )log IA 1 — ,i(B)log(B), 8„

since I B I < IAIn . After division by n, this part must go to 0 as n —> oc, since the entropy theorem implies that ti(B) goes to O. On the set G, the following holds

n(h

—

e) _< — log ,u(4) co the measure of G, approaches 1, from the entropy theorem, so the sum converges to H({X„}). Thus

h—

E

which proves the theorem in the case when h > O. If h = 0 then define G, = {4: ,u(4) > 2 —ne . The bound (5) still holds, while D only the upper bound in (6) matters and the theorem again follows. }

Remark 1.6.10 Since for ergodic processes, the process entropy H({X,}) is the same as the entropyrate h of the entropy theorem, both are often simply called the entropy. A detailed mathematical exposition and philosophical interpretation of entropy as a measure of information can be found in Billingsley's book, [4]. The Csiszdr-Körner book, [7], contains an excellent discussion of combinatorial and communication theory aspects of entropy, in the i.i.d. case. The recent books by Gray, [18], and Cover and Thomas, [6], discuss many information-theoretic aspects of entropy for the general ergodic process.

CHAPTER I. BASIC CONCEPTS.

62

I.6.c

The entropy of i.i.d. and Markov processes.

Entropy formulas for i.i.d. and Markov processes can be derived from the entropy theorem by using the ergodic theorem to directly estimate p.(4), see Exercises 1 and 2 in Section 1.5. Here they will be derived from the definition of process entropy. Let {X, } be an i.i.d. process and let p(x) = Prob(Xi = x). The additivity of entropy for independent random variables, Theorem 1.6.3(c), gives

H (XI') = H(Xi)

H (X2) +

H(X n ) = nH (X i ),

which yields the entropy formula

(7)

H({X,}) = H(X i ) = —

p(x) log p(x).

An alternate proof can be given by using the conditional limit formula, (3), and the fact that H(Xl Y) = H(X) if X and Y are independent, to obtain

H({X,}) = lip H (X0IX1,!) = H(X0)• Now suppose {X n } is a Markov chain with stationary vector p and transition matrix M. The Markov property implies that

Prob (X0 = alX) = Prob (X0 = alX_i) , a

E

A,

from which it follows that

H (X 0 1

= H(X01 X -1)•

Thus the conditional entropy formula for the entropy of a process, (3), along with a direct calculation of H(X0IX_ 1 ) yields the following formula for the entropy of a Markov chain.

(8)

H({X n }) = H(X0IX_ i ) = —

E 04 log Mii . ,

Recall that a process {Xn } is Markov of order k if

Prob (X0 = aiX) = Prob (X0 = alX:k1 ) , a

E

A.

The argument used in the preceding paragraph shows that if {X,} is Markov of order k then

1/({X}) = H(X0IX:1) =

n > k.

This condition actually implies that the process is Markov of order k.

Theorem 1.6.11 (The Markov order theorem.) A stationary process {X n } is Markov of order k if and only if H(X0IX:) = H(X0IX: n1 ), n > k. Proof The conditional addition law gives

H((X o , Xink+1 )1X1) = H(X: nk+1 1X11) H(X 0 IXI kl

,

Xink+1).

SECTION 1.6. ENTROPY AS EXPECTED VALUE.

63

The second term on the right can be replaced by H(X01X: k1 ), provided H(X01X1 k1 ) = H(X0 1X1n1 ), for n > k. The equality condition of the subadditivity principle, Theorem I.6.4(c), can then be used to conclude that X0 and X:_ nk+1 are conditionally independent given XI/. If this is true for every n > k then the process must be Markov of order k. This completes the proof of the theorem. In general the conditional entropy function Hk = H(X0IX: k1 ) is nonincreasing in k. To say that the process is Markov of some order is to say that Hk is eventually constant. If the true order is k*, then Hks_i > fik* = Hk , k > k*. This fact can be used to estimate the order of a Markov chain from observation of a sample path.

I.6.d

Entropy and types.

The entropy of an empirical distribution gives the exponent for a bound on the number of sequences that could have produced that empirical distribution. This fact is useful in large deviations theory, and will be useful in some of the deeper interpretations of entropy to be given in later chapters. The empirical distribution or type of a sequence x E A" is the probability distribution Pi = pl (. 14) on A defined by the relative frequency of occurrence of each symbol in the sequence 4, that is, n

= an ,

Ifi:

Pi(alxi) =

a E A.

Two sequences xi' and y'11 are said to be type-equivalent if they have the same type, that is, if each symbol a appears in x'; the same number of times it appears in Type-equivalence classes are called type classes. The type class of xi' will be denoted by T WO or by 7, where the latter stresses the type p i = pi (. 14), rather than a particular sequence that defines the type. Thus, x'1' E T; if and only if each symbol a appears in xi' exactly n pi(a) times. The empirical (first-order) entropy of xi' is the entropy of p i (- 14), that is,

17(130= ii(pl(. 14)) = -

Pi(aixi)log Pi(aI4). a

The following purely combinatorial result bounds the size of a type class in terms of the empirical entropy of the type.

Theorem 1.6.12 (The type class bound.) < 2r1(131), for any type p i . -

Proof First note that if Qn is a product measure on A n then (9)

Q" (x)= ji Q(xi) i=1

= fl

War P1(a) , x

E

rn

aEA

since np i (a) is the number of times the symbol a appears in a given 4 particular, a product measure is always constant on each type class.

E

Tpn,. In

CHAPTER 1. BASIC CONCEPTS.

64

A type pi defines a product measure Pn on An by the formula

Pn (z?) =

pi (zi), z E A'

.

Replacing Qn by Pi in the product formula, (9), produces pi (a)nPl(a) ,

Pn(xiz) =

E rpni ,

aEA which, after taking the logarithm and rewriting, yields

Pn(x7) = 2—n17(PI) ,

x

E

In other words, Pn (4) has the constant value 2' -17(P' ) on the type class of x7, and hence = ypn, 12— n 11(”v 1 ) But Pi is a probability distribution so that Pi (Tpni ) < 1. This establishes Theorem 1.6.12. The bound 2' 17, while not tight, is the correct asymptotic exponential bound for the size of a type class, as shown in the Csiszdr-Körner book, [7]. The upper bound is all that will be used in this book. Later use will also be made of the fact that there are only polynomially many type classes, as stated in the following theorem. Theorem 1.6.13 (The number-of-types bound.) The number of possible types is at most (n + 1)1A I.

Proof This follows from the fact that for each a E A, the only possible values for , n. npi (alfii) are 0, 1, The concept of type extends to the empirical distribution of overlapping k-blocks. The empirical overlapping k-block distribution or k-type of a sequence x7 E A' is the probability distribution Pk = pk(' 14) on A k defined by the relative frequency of occurrence of each k-block in thesen [i quenkce+ii'i th_ai±t ki_S,1 =

I

E

-41

n— k

1

E

A k.

Two sequences x;' and yç' are said to be k-type-equivalent if they have the same k-type, and k-type-equivalence classes are called k-type classes. The k-type class of x i" will be denoted by Tpnk , where Pk = Pk ( Ix), that is, 4 E T, if and only if each block all' appears in xÇ exactly (n — k + 1) pk (alic) times. The bound on the number of types, Theorem 1.6.13, extends immediately to k-types. Theorem 1.6.14 (The number of k-types bound.) The number of possible k-types is at most (n — k + Note, in particular, that if k is fixed the number of possible k-types grows polynomially in n, while if k is also growing, but satisfies k < a logiAi n, with a < 1, then the number of k-types is of lower order than IA In.

SECTION 1.6. ENTROPY AS EXPECTED VALUE.

65

Estimating the size of a k-type class is a bit trickier, since the k-type measures the frequency of overlapping k-blocks. A suitable bound can be obtained, however, by considering the (k — 1)-st order Markov measure defined by the k-type plc ( 14), that is, the stationary (k — 1)-st order Markov chain ri 0c -1) with the empirical transition function

(ak 14 -1 ) =

Pk(ainxii ) „ f k-1 „ir‘ uk IA 1 ) E bk pk

and with the stationary distribution given by The Markov chain ri(k-1) has entropy (k

-

1)

"Aar') =

Ebk Pk(ar i

-E 1.-j(a k Or 1 )Ti(a ki -1 ) log 75(akiar) E Pk(akil4) log Ti(ak lari ), ak

ak

by formula (8). The entropy FP") is called the empirical (k — 1)-st order Markov entropy of 4. Note that it is constant on the k-type class 'Tk(xiii) = Tp k of all sequences that have the same k-type Pk as xrii. Theorem 1.6.15 (The k-type-class bound.) I7k (fil)1 < (n — Proof First consider the case k = 2. If Q is Markov a direct calculation yields the formula

Q(4)

= Q(xi)n Q(bi a)(n—l)p2(ab)

x rit c Tp2.

a,b

This formula, with Q replaced b y (2-1) = rf,(1) and after suitable rewrite using the definition of W I) becomes

1 TV I) (Xlii ) = Ti-(1) (X02—(n-1)1 ,

x

E

and hence P

2- (n- 1>ii(1 ) IT P2 I — 1n

since ii:(1) (xi) > 1/(n — 1). This yields the desired bound for the first-order case, since P1) (4) is a probability measure on A n . The proof for general k is obtained by an obvious extension of the argument.

I.6.e

Exercises.

1. Prove the log-sum inequality: If a, > 0, b, > 0, i = 1, 2, E ai , b = E bi , then E ai log(ai I bi ) ?_ a log(a/b).

, n, and a

2. Let P be the time-0 partition for an ergodic Markov chain with period d = 3 and entropy H > O. (a) Determine the entropies of the ergodic components of the (T3 , 2)-process.

CHAPTER I. BASIC CONCEPTS.

66

(b) Determine the entropies of the ergodic components of the (T 3 , p V Tp T22)-process. 3. Let p be an ergodic process and let a be a positive number. Show there is an N = N(a) such that if n > N then there is a set Fn , measurable with respect to xn oc , such that ,u(Fn ) > 1 — a and so that if xn E Fn then 27" 11, (4) < t(xIx)

2",i(4).

(Hint: apply the ergodic theorem to f (x) = — log //,(xi Ix), and use the equality of process entropy and entropy-rate.)

4. Let T be the shift on X = {-1,1} z , and let ji be the product measure on X defined by ,u0(-1) = p,(1) = 1/2. Let Y = X x X, with the product measure y = j x it, and define S(x, y) = (T x , Toy). Let p be the time-zero partition of X and let Q = P x P. Find the entropy of the (S, Q)-process, which is the "random walk with random scenery" process discussed in Exercise 8 of Section 1.2. (Hint: recurrence of simple random walk implies that all the sites of y have been almost surely visited by the past of the walk.) 5. Prove that the entropy of an n-stationary process {Yn } is the same as the entropy the stationary process {X„} obtained by randomizing the start. (Hint: let S be uniformly distributed on [1, n] and note that H (X I v , S) = H H (SIXr) = H(S)+ H(X liv IS) for any N> n, so H(Xr)1N H(Xr IS)1N. Then show that I-1(r IS = s) is the same as the unshifted entropy of Xs/v+1 .) 6. The divergence for k-set partitions is D(P Q) =

Ei It(Pi)log(p,(Pi)1 A(Q i )).

(a) Show that D (P v 7Z I Q v > Q) for any partition R.. (b) Prove Pinsker's inequality, namely, that D (P Il Q) > (1 / 2 ln 2)IP — Q1 2 , where jp - QI 1,u(Pi ) - bt 1 a2012 . (Hint: use part (a) to reduce to the two set case, then use calculus.)

7. Show that the process entropy of a stationary process and its reversed process are the same. (Hint: use stationarity.)

Section 1.7

Interpretations of entropy.

Two simple, useful interpretations of entropy will be discussed in this section. The first is an expression of the entropy theorem in exponential form, which leads to the concept of entropy-typical sequence, and to the related building-block concept. The second interpretation is the connection between entropy and expected code length for the special class of prefix codes.

I.7.a

Entropy-typical sequences.

Let p. be an ergodic process of entropy h. For E > 0 and n > 1 define En(E) 1 4: 2—n(h+E) < tt (x in) < 2—n(h-6)} The set T(E) is called the set of (n, €)-entropy-typical sequences, or simply the set of entropy-typical sequences, if n and c are understood. The entropy theorem may be expressed by saying that xril is eventually almost surely entropy-typical, that is,

SECTION 1.7. INTERPRETATIONS OF ENTROPY.

67

Theorem 1.7.1 (The typical-sequence form of entropy.) For each E > 0, x E T(e), eventually almost surely.

The convergence in probability form, limn ,u(Tn (E)) = 1, is known as the asymptotic equipartition property or AEP, in information theory. The phrase "typical sequence," has different meaning in different contexts. Sometimes it means entropy-typical sequences, as defined here, sometimes it means frequencytypical sequences, as defined in Section 1.4, sometimes it means sequences that are both frequency-typical and entropy-typical, and sometimes it is just shorthand for those sequences that are likely to occur. The context usually makes clear the notion of typicality being used. Here the focus is on the entropy-typical idea. The members of the entropy-typical set T(e) all have the lower bound 2 —n(h+f) on their probabilities. Since the total probability is at most 1, this fact yields an upper bound on the cardinality of Tn , namely, Theorem 1.7.2 (The entropy-typical cardinality bound.) The set of entropy-typical sequences satisfies 17-n (01 < 2n (h+E)

Thus, even though there are I A In possible sequences of length n, the measure is eventually mostly concentrated on a set of sequences of the (generally) much smaller cardinality 2n(h+6) • This fact is of key importance in information theory and plays a major role in many applications and interpretations of the entropy concept. The preceding theorem provides an upper bound on the cardinality of the set of typical sequences and depends on the fact that typical sequences have a lower bound on their probabilities. Typical sequences also have an upper bound on their probabilities, which leads to the fact that too-small sets cannot be visited too often. Theorem 1.7.3 (The too-small set principle.) < 2n(h--€), n > 1, then .7q If C, c A n and ICI _

Ci', eventually almost surely.

Proof Since .X1' E 'T,(E12), eventually almost surely, it is enough to show that x

g C, n'T,i (e/ 2), eventually almost surely. The cardinality bound on Cn and the probability upper bound 2 2) on members of Tn (E/2) combine to give the bound (Cn

Thus ,u(Cn

n Tn(E/2))
2n(h-E)a(i 6),

fl

which proves the theorem.

The covering exponent idea is quite useful as a tool for entropy estimation, see, for example, the proof in Section I.7.e, below, that a rotation process has entropy O. The connection between coverings and entropy also has as a useful approximate form, which will be discussed in the following paragraphs. Let d(a, b) be the (discrete) metric on A, defined by 0 d (a , b) = J 1 1

if a = b otherwise,

and extend to the metric d on A', defined by

1 dn (4 , yri`) = — n i=1

yi)•

The metric cin is also known as the per-letter Hamming distance. The distance dn (x, S) from 4 to a set S C An is defined by

dn (x' , S) = min{dn (4,

y;' E S},

and the 6-neighborhood or 6-blowup of S is defined by

[S ], =

S) < 81.

It is important that a small blowup does not increase size by more than a small exponential factor. To state this precisely let H(8) = —6 log 6 (1— 6) log(1 — 8) denote the binary entropy function.

69

SECTION 1.7. INTERPRETATIONS OF ENTROPY. Lemma 1.7.5 (The blowup bound.) The 6-blowup of S satisfies

I[S]81

ISI2 n11(6) (1A1 —

1) n3 .

In particular, given c > 0 there is a S > 0 such that Ir1n(c)131

< n(h+2€) 2

for all n.

Proof Given x, there are at most nS positions i in which xi can be changed to create a member of [ { x7 } 13. Each such position can be changed in IA — 1 ways. Thus, the _ 2nH(3) (IAI — 1) n8 , which combinations bound, Lemma 1.5.4, implies that I[{4}]51 < yields the stated bound. < 2n(h-1-€) since (IAI — 1)n8 < 20 log IAI and since Slog IA I and H(8) Since IT(e)1 _ < 2n (h+26) 0, it follows that if 8 is small enough, then Irrn(E)ls1 _ both go to 0 as El will hold, for all n. This proves the blowup-bound lemma.

The blowup form of the covering number is defined by

Arn (a, 8) = min{ICI: C c

(1)

An,

and //([C]3 )

a},

that is, it is the minimum size of an n-set for which the 8-blowup covers an a-fraction of the probability. It is left to the reader to prove that the limit of (1/n) log Nen (a, 8) oc. An application of Lemma 1.7.5, together with Theorem 1.7.4 then exists as n establishes that 1 lim lim — log Afn (a, 8) = h. 6—>0 n—> oo

I.7.b

n

The building-block concept.

An application of the ergodic theorem shows that frequency-typical sequences must consist mostly of n-blocks which are entropy-typical, provided only that n is not too small. In particular, the entropy-typical n-sequences can be thought of as the "building blocks," from which longer sequences are made by concatenating typical sequences, with occasional spacers inserted between the blocks. This idea and several useful consequences of it will now be developed. Fix a collection 5, c An , thought of as the set of building blocks. Also fix an integer M > n and 8 > O. A sequence xr is said to be (1 — 0-built-up from the building blocks B, if there is an integer I = I(41 ) and a collection llni, mi]: i < 1), of disjoint n-length subintervals of [1, M], such that

(a) Eil.=1 (mi — n i + 1) > (1 — (b)

E

I3,,

i c I.

In the special case when 8 = 0 and M is a multiple of n, to say that 41 is 1-built-up from Bn is the same as saying that xr is a concatenation of blocks from B„. If 8 > 0 then the notion of (1-0-built-up requires that xr be a concatenation of blocks from 13,, with spacers allowed between the blocks, subject only to the requirement that the total length of the spacers be at most SM. Both the number I and the intervals [ni , m i ] for which x , is required to be a member of 13,, are allowed to depend on the sequence xr. The reader should also note the concepts of blowup and built-up are quite different. The

CHAPTER I. BASIC CONCEPTS.

70

blowup concept focuses on creating sequences by making a small density of otherwise arbitrary changes, while the built-up concept only allows changes in the spaces between the building blocks, but allows arbitrary selection of the blocks from a fixed collection. An important fact about the building-block concept is that if S is small and n is large, then the set of M-sequences that can be (1 — 6)-built-up from a given set 8„ C An of building blocks cannot be exponentially much larger in cardinality than the set of all sequences that can be formed by selecting M I n sequences from Bn and concatenating them without spacers. The proof of this fact, which is stated as the following lemma, is similar in spirit to the proof of the key bound used to establish the entropy theorem, Lemma 1.5.6, though simpler since now the blocks all have a fixed length. As usual, H(8) = —8 log6 — (1 —8) log(1 — S) denotes the binary entropy function. Lemma 1.7.6 (The built-up set bound.) Let DM be the set of all sequences xr that can be (1 — 6)-built-up from a given collection B„ c An. Then

IB.1 2mH (0 1 A im s.

I Dm I

In particular, if B, = 7,(c), the set of entropy-typical n-sequences relative to c, and if

is small enough then 11)m 1 < 2m(h +26) . Proof The number of ways to select a family {[n i , m i ]} of disjoint n-length subintervals that cover all but a 6-fraction of [1, M] is upper bounded by the number of ways to select at most SM points from a set with M members, namely,

EY1
m i _ i such that ric+n--1 E 13„, stopping when "" Ic (1 — 3/2)-strongly-covered by

within n of the end of xr. The assumption that xr is 13, implies there are at most 6M/2 indices j < M — n 1 which are not contained in one of the [ni , m i ], while the condition that M > 2n18 implies there at most 3M/2 indices j E [M — n 1, M] which are not contained in one of the [ni , mi]. The lemma is therefore established.

Remark 1.7.8 The preceding two lemmas are strictly combinatorial. In combination with the ergodic and entropy theorems they provide powerful tools for analyzing the combinatorial properties of partitions of sample paths, a subject to be discussed in the next chapter. In particular, if B, = Tn (e), and n is large enough, then ,u,(7,,(€)) 1. The almost-covering principle, Theorem 1.4.7, implies that eventually almost surely xr is almost stronglycovered by sequences from T(e), hence eventually almost surely mostly built-up from members of 7; (e). A suitable application of the built-up set lemma, Lemma 1.7.6, implies that the set Dm of sequences that are mostly built-up from the building blocks T(e) will eventually have cardinality not exponentially much larger than 2/4(h+6) • The sequences in Dm are not necessarily entropy-typical, for no upper and lower bounds on their probabilities are given; all that is known is that the members of Dm are almost built-up from entropy-typical n-blocks, where n is a fixed large number, and that xr is eventually in DM. This result, which can be viewed as the entropy version of the finite sequence form of the frequency-typical sequence characterization of ergodic processes, Theorem 1.4.5, will be surprisingly useful later.

I.7.c

Entropy and prefix codes.

In a typical data compression problem a given finite sequence 4, drawn from some finite alphabet A, is to be mapped into a binary sequence bf = b1, b2, , b.c, whose length r may depend on 4 , in such a way that the source sequence x'1' is recoverable from knowledge of the encoded sequence bf.. The goal is to use as little storage space as possible, that is, to make the code length as short as possible, at least in some average sense. Of course, there are many possible source sequences, often of varying lengths, so that a typical code must be designed to encode a large number of different sequences and accurately decode them. A standard model is to think of a source sequence as a (finite) sample path drawn from some ergodic process. For this reason, in information theory a stationary, ergodic process with finite alphabet A is usually called a source. In this section the focus will be on a special class of codes, known as prefix codes, for which there is a close connection between code length and source entropy. In the following discussion B* denotes the set of all finite-length binary sequences and t(w) denotes the length of a member w E B*. B*. A code is said to be faithful or A (binary) code on A is a mapping C: A noiseless if it is one-to-one. An image C (a) is called a codeword and the range of C

CHAPTER I. BASIC CONCEPTS.

72

is called the codebook. The function that assigns to each a E A the length of the code word C (a) is called the length function of the code and denoted by £(IC) or by r if C is understood. Formally, it is the function defined by ,C(aIC) = £(C (a)), a E C. The expected length of a code C relative to a probability distribution a on A is E(L) = Ea ,C(aIC)p,(a). The entropy of p, is defined by the formula

H (A) =

E p(a) log

1

aEA

that is, H(j) is just the expected value of the random variable — log ,u(X), where X has the distribution A. (As usual, base 2 logarithms are used.) Without further assumptions about the code there is little connection between entropy and expected code length. For codes that satisfy a simple prefix condition, however, entropy is a lower bound to code length, a lower bound which is almost tight. To develop the prefix code idea some notation and terminology and terminology will be introduced. For u = u7 E B* and u = vr E B*, the concatenation uv is the sequence of length n m defined by

(uv) i =

ui

1 x, and note that

E 2— C (a) < E a

2logg(a)

a

E a

Lemma 1.7.10 produces a prefix code C(a) = w(a), a Since, however, ,C(a) < 1— log p(a), it follows that = a

1.

E

A, whose length function is L.

E(1- log tt(a))p,(a) a

1—

E p(a) log p(a) =1+ H(p). a

Thus part (ii) is proved, which completes the proof of the theorem. Next consider n-codes, that is, mappings Cn : A n 1—> B* from source sequences of length n to binary words. In this case, a faithful n-code Cn : An B* is a prefix n-code if and only if its range is prefix free. Of interest for n-codes is per-symbol code length ,C(a71Cn )/n and expected per-symbol code length E(C(.lCn ))1n. Let /in be a probability measure on A n with per-symbol entropy 1-1,(A n ) = H(i)/n. Theorem I.7.11(i) takes the form 1 (4) H(iin)

for any prefix n-code Cn , while Theorem I.7.11(ii) asserts the existence of a prefix n-code Cn such that 1 1 (5) — E (reIC n)) 5- H n (P, n) — . n " n A sequence {Cn } , such that, for each n, Cn is a prefix n-code with length function ,C(4) = ,C(4 IQ, will be called a prefix-code sequence. The (asymptotic) rate of such a code sequence, relative to a process {X n } with Kolmogorov measure IL is defined by = liM sup

E (C(IC n))

n—co

The two results, (4) and (5) then yield the following asymptotic results. Theorem 1.7.12 (Process entropy and prefix-codes.) If p, is a stationary process with process entropy H then (a) There is a prefix code sequence {C n } such that 1-4({G}) < H. (b) There is no prefix code sequence {C n } such that 1Z 4 ({C n }) < H. Thus "good" codes exist, that is, it is possible to compress as well as process entropy in the limit, and "too-good" codes do not exist, that is, no sequence of codes can asymptotically compress more than process entropy. In the next chapter, almost-sure versions of these two results will be obtained for ergodic processes.

SECTION 1.7. INTERPRETATIONS OF ENTROPY.

75

Remark 1.7.13 A prefix code with length function ,C(a) = F— log ,u(a)1 will be called a Shannon code. Shannon's theorem implies that a Shannon code is within 1 of being the best possible prefix code in the sense of minimizing expected code length. A somewhat more complicated coding procedure, called Huffman coding, produces prefix codes that minimize expected code length. Huffman codes have considerable practical significance, but for n-codes the per-letter difference in expected code length between a Shannon code and a Huffman code is at most 1/n, which is asymptotically negligible. For this reason no use will be made of Huffman coding in this book, since, for Shannon codes, the code length function L(a) = 1— log (a)1 is closely related to the function — log p,(a), whose expected value is entropy.

I.7.d

Converting faithful codes to prefix codes.

Next it will be shown that, as far as asymptotic results are concerned, there is no loss of generality in assuming that a faithful code is a prefix code. The key to this is the fact that a faithful n-code can always be converted to a prefix code by adding a header (i.e, a prefix) to each codeword to specify its length length. This can be done in such a way that the length of the header is (asymptotically) negligible compared to total codeword length, so that asymptotic results about prefix codes automatically apply to the weaker concept of faithful code. The following is due to Elias, [8].

Lemma 1.7.14 (The Elias-code lemma.) There is a prefix code £:11,2, B*, such that i(e(n)) = log n + o(log n). Any prefix code with this property is called an Elias code. Proof The code word assigned to n is a concatenation of three binary sequences,

C(n) = u(n)v(n)w(n).

The third part w(n) is the usual binary representation of n, so that, for example, w(12) = 1100. The second part v(n) is the binary representation of the length of w(n), so that, for example, v(12) = 100. The first part u(n) is just a sequence of O's of length equal to the length of v(n), so that, for example, u(12)=000. Thus E(12) = 0001001100. The code is a prefix code, for if u(n)v(n)w(n) = u(m)v(in)w(m)w', then, u(n) = u(m), since both consist only of O's and the first bit of both v(n) and v(m) is a 1. But then v(n) = v(m), since both have length equal to the length of u(n). This means that w(n) = w(m), since both have length specified by v(n), so that w' is empty and n = m. The length of w(n) is flog(n +1)1, the length of the binary representation of n, while both u(n) and v(n) have length equal to

f log(1 + Flog(n

1 )1)1,

the length of the binary representation of Flog(n + 1)1. The desired bound f(E(n)) = log n + o(log n) follows easily, so the lemma is established. CI

CHAPTER I. BASIC CONCEPTS.

76

Given a faithful n-code Cn with length function Le ICn ), a prefix n-code Cn* is obtained by the formula

C: (4 ) = ECC(.4 I Cn ))Cn (4), x lii E An ,

(6)

where e is an Elias prefix code on the integers. For example, if C,(4) = 001001100111, a word of length 12, then

C:(4) = 0001001100, 001001100111, where, for ease of reading, a comma was inserted between the header information, e (C(4 ICn )), and the code word C, (4). The decoder reads through the header information and learns where C, (4) starts and that it is 12 bits long. This enables the decoder to determine the preimage .x11 , since Cn was assumed to be faithful. The code CnK, which is clearly a prefix code, will be called an Elias extension of C. In the example, the header information is almost as long as Cn (4), but header length becomes negligible relative to the length of Cn (4) as codeword length grows, by Lemma 1.7.14. Thus Theorem 1.7.12(b) extends to the following somewhat sharper form. (The definition of faithful-code sequence is obtained by replacing the word "prefix" by the word "faithful," in the definition of prefix-code sequence.)

Theorem 1.7.15 If ,u is a stationary process with entropy-rate H there is no faithful-code sequence such that R IL ({C n }) < H.

Remark 1.7.16 Another application of the Elias prefix idea converts a prefix-code sequence {C n } into a single prefix code C: A* i where C is defined by

C(4) = e(n)C(4),

X tiz E An,

n = 1, 2, ...

The header tells the decoder which codebook to apply to decode the received message.

I.7.e

Rotation processes have entropy O.

As an application of the covering-exponent interpretation of entropy it will be shown that ergodic rotation processes have entropy 0. Let T be the transformation defined by T:xi--> x ea, x E [0, 1), where a is irrational and ED is addition modulo 1, and let P be a finite partition of the unit interval [0, 1).

Proposition 1.7.17 The (T, P)-process has entropy O. The proof for the special two set partition P = (Po, PO, where Po = [0, 0.5), and P1 = [0.5, 1), will be given in detail. The argument generalizes easily to arbitrary partitions into subintervals, and, with a bit more effort, to general partitions (by approximating by partitions into intervals). The proof will be based on direct counting arguments rather than the entropy formalism. Geometric arguments will be used to estimate the number of possible (T, P)-names of length n. These will produce an upper bound of the form 2nEn , where En —> 0 as

77

SECTION 1.7. INTERPRETATIONS OF ENTROPY.

n oo. The result then follows from the covering-exponent interpretation of entropy, Theorem 1.7.4. The key to the proof is the following uniform distribution property of irrational rotations.

Proposition 1.7.18 (The uniform distribution property.) Inn

E [1, tz]: Tlx E

=

for any interval I c [0, 1) and any x E [0, 1), where A denotes Lebesgue measure. Ergodicity asserts the truth of the proposition for almost every x. The fact that rotations are rigid motions allows the almost-sure result to be extended to a result that holds for every x. The proof is left as an exercise.

The key to the proof to the entropy 0 result is that, given z and x, some power y = Tx of x must be close to z. But translation is a rigid motion so that Tmy and Trnz will be close for all m, and hence the (T-P)-names of y and z will agree most of the time. Thus every name is obtained by changing the name of some fixed z in a small fraction of places and shifting a bounded amount. In particular, there cannot be exponentially very many names. To make the preceding argument precise define the metric Ix - yli = minflx - y1,11

x - y11, x, y E [0, 1),

and the pseudometric dn(x, y) = dn(xç ,

1 n d(xi, Yi), n

=-

where d(a, b) = 0, if a = b, and d(a,b) = 1, if a 0 b, and {x n } and {yn} denote the (T, P)-names of x, y E [0, 1), respectively. The first lemma shows that cln (x, y) is continuous relative to Ix - Y 1 1.

Lemma 1.7.19 E, if Ix - Y11 < 6/4 and n > N. Proof Suppose lx - yli < €14. Consider the case when Ix Y Ii = lx - yi. Without loss of generality it can be assumed that x < y. Let I = [x, y]. The names of x and y disagree at time j if and only if Tx and T y belong to different atoms of the partition P, which occurs when and only when either T --1 0 E I or (0.5) E I. The uniform distribution property, Proposition 1.7.18, can be applied to T -1 , which is rotation by -a, to provide an N = N(x, y) such that if n > N then Given

E

> 0 there is an N such that dn (x, y)
N. The case when lx - yll = 11 + x - yl can be treated in a similar manner. Compactness then shows that N can be chosen to be independent of x and y. The next lemma is just the formal statement that the name of any point x can be shifted by a bounded amount to obtain a sequence close to the name of z = 0 in the sense of the pseudometric dn.

CHAPTER I. BASIC CONCEPTS.

78 Lemma 1.7.20

Given E > 0, there is an integer M and an integer K > M such that if x E [0, 1) there is a j = j(x) E [1, M] such that if Z = 0 and y = Tx then dk (zki, yk ) < 69 k > K.

Given E > 0, choose N from the preceding lemma so that if n > N and lx — yli < 6/4 then dn (x , y) < E. Given x, apply Kronecker's theorem, Proposition 1.2.15, to obtain a least positive integer j = j(x) such that IT ] x — Ok < 6/4. Since small changes in x do not change j(x) there is a number M such that j(x) < M for all x. With K = M N the lemma follows. Proof

Now for the final details of the proof that the entropy of the (T, P)-process is 0. Given E > 0, determine M and K > M from the preceding lemma, let n > K+M and let z = O. The unit interval X can be partitioned into measurable sets Xi = Ix : j(x) = j}, j < M, where j(x) is given by the preceding lemma. Note that dn—j(Z,T i X) < E, X E X.

The number of binary sequences of length n — j that can be obtained by changing the (n — j)-name of z in at most c(n — j) places is upper bounded by 2 (n— i )h(E) , where

h(6) = —doge —(1 — 6) log(1 — 6), is the binary entropy function, according to the combinations bound, Lemma 1.5.4. There are at most 2i possible values for the first j places of any name. Thus an upper bound on the number of possible names of length n for members of Xi is

2(n— Dh(e) 114 : x E Xill < 2 i

(7)

IA lnh(6)

Since there only M possible values for j, the number of possible rotation names of 0 length n is upper bounded by M2m2nh(€), for n > K + M. Since h(€) —> 0 as E 1=1 this completes the proof that the (T, 2)-process has entropy 0.

I.7.f Exercises. 1. Give a direct proof of the covering-exponent theorem, Theorem 1.7.4, based on the algorithm (Al ,A2) given for the computation of the covering number. (Hint: what does the entropy theorem say about the first part of the list?) 2. Show that limn (1/n) log.Arn (a, 6) exists, where

Nn (a, 6)

is defined in (1).

3. Let g be the concatenated-block process defined by a measure it on A n . Let H(A) denote the entropy of p,. (a) Show that Ann ( '2) < (P,) ± log n. + (b) Show that H„,n (g) > mH(A). (Hint: there is at least one way and at most n1441 2n ways to express a given 4 in the form uw(l)w(2)... w(k — 1)v, where each w(i) E An and u and v are the tail and head, respectively, of words w(0), w(k) E An.)

79

SECTION 1.8. STATIONARY CODING.

4. Use the covering exponent concept to show that the entropy of an ergodic nstationary process is the same as the entropy of the stationary process obtained by randomizing its start. 5. Let p, a shift-invariant ergodic measure on A z and let B c A z be a measurable set of positive measure. Show, using the entropy theorem, that the entropy of the induced process is 1/(A)//t(B). 6. Suppose the waiting times between occurrences of "1" for a binary process it are independent and identically distributed. Find an expression for the entropy of it, in terms of the entropy of the waiting-time distribution. 7. Let {Ifn } be the process obtained from the stationary ergodic process {X} by applying the block code CN and randomizing the start. Suppose {1',„} is ergodic. Show that H({Y„}) < 8. Show that if C„: A n H B* is a prefix code with length function L then there is a prefix code e: A" 1—* B* with length function E such that (a) Î(x) N, there is a set Cn c Ani +ww of measure at least 1/2 and cardinality at most 2 (n±2w+1)(h( A )+6) • The image f (C n ) is a subset of 13 1 of measure at least 1/2, by formula (2). Furthermore, because mappings cannot increase cardinality, the set f (C n ) has cardinality at most 2(n+21D+1)(h(4)+E) . Thus 1 - log I f (Cn)i h(11) + 6 + 8n, n where Bn —>. 0, as n -> oo, since w is fixed. It follows that h(v) < h(tt), since entropy equals the asymptotic covering rate, Theorem 1.7.4. In the general case, given c > 0 there is a finite time-zero coder f such that A (ix: f (x) 0 f (x)}) < E. Let F and F denote the sample path encoders and let y = it o F -1 and 7/ = ki o F -1 denote the Kolmogorov measures defined by f and f, respectively. It has already been established that finite codes do not increase entropy, so that h(71) < h(i). Thus, there is an n and a collection 'e" c Bn such that 71(E) > 1 - E and II Let C = [C], be the 6-blowup of C, that is, Proof

C = { y: dn(Yi, 3711 ) 5_

E,

for some

3-,T;

el

E '

The blowup bound, Lemma 1.7.5, implies that

ICI < where

3(E) = 0(E).

9

Let

15 = {x: T(x)7

E

El} and D = {x: F(x)7

E

CI

be the respective pull-backs to AZ. Since ki (ix: f (x) 0 7(x)}) 1 — 2E, since ,u(15) = 75(e) > 1 — E. By definition of G and D, however, Gniic D, so that

e > 1 — 2c. v(C) = g(D) > p(G n ij) The bound ICI < 2n (h(A )+E+3(E )) , and the fact that entropy equals the asymptotic covering rate, Theorem 1.7.4, then imply that h(v) < h(u). This completes the proof of Theorem 1.8.3.

Example 1.8.4 (Stationary coding preserves mixing.) A simple argument, see Exercise 19, can be used to show that stationary coding preserves mixing. Here a proof based on approximation by finite coders will be given. While not as simple as the earlier proof, it gives more direct insight into why stationary coding preserves mixing and is a nice application of coder approximation ideas. The a-field generated by the cylinders [4], for m and n fixed, that is, the cr-field generated by the random variables Xm , Xm+ i, , X n , will be denoted by E(Xmn ). As noted earlier, the i-fold shift of the cylinder set [cinin ] is the cylinder set T[a] = [c.:Ln ], where ci +i = ai, m < j 1,

p,

F(x)7

P(x)7, or F(x)mm -Vi

• P(x)21:71) < 2n8,

83

SECTION 1.8. STATIONARY CODING. so that (4)

Ivaan n T - m[b7]) — j)([4] n T -m[b71)1

6/3,

provided only that 8 is small enough, uniformly in m for all ail and b7. Since 1'5 is mixing, being a finite coding of ,u, it is true that

n T -m[bn) — i)([4])Dab7DI < E/3, provided only that m is large enough, and hence, combining this with (3) and (4), and using the triangle inequality yields I vga7] n T -ni[b7]) — vga7])v([14])1 < c, for all sufficiently large m, provided only that 8 is small enough. Thus, indeed, stationary coding preserves the mixing property.

I.8.b

From block to stationary codes.

As noted in Example 1.1.10, an N-block code CN: A N B N can be used to map an A-valued process {X„} into a B-valued process {Y„} by applying CN to consecutive nonoverlapping blocks of length N, v (f-1-1)N

i iN+1

-

ii(j+1)N) iN+1

j = 0, 1, 2, ...

If {X,} is stationary, then a stationary process {4} is obtained by randomizing the start, i.e., by selecting an integer U E [1, N] according to the uniform distribution and defining = Y+-i, j = 1, 2, .... The final process {Yn } is stationary, but it is not, except in rare cases, a stationary coding of {X„}, and nice properties of {X,}, such as mixing or even ergodicity, may get destroyed. A method for producing stationary codes from block codes will now be described. The basic idea is to use an event of small probability as a signal to start using the block code. The block code is then applied to successive N-blocks until within N of the next occurrence of the event. If the event has small enough probability, sample paths will be mostly covered by nonoverlapping blocks of length exactly N to which the block code is applied.

Lemma 1.8.5 (Block-to-stationary construction.) Let 1,1, be an ergodic measure on A z and let C: A N B N be an N-block code. Given c > 0 there is a stationary code F = Fc: A z A z such that for almost every X E A Z there is an increasing sequence { ni :i E Z}, which depends on x, such that Z=

ni+i) and

(i) ni+1—ni < N, i E Z. (ii) If .I„ is the set of indices i such that [ni , +1) c [—n, n] and ni+i — n i < N, then ihn supn (1/2n)E iEjn ni+ 1 — ni < c, almost surely. (iii) If n i+1 — ni = N then y: nn +'- ' = C(xn:±'), n -1 where y = F(x).

CHAPTER I. BASIC CONCEPTS.

84

Proof Let D be a cylinder set such that 0 < ,a(D) < 6/N, and let G be the set of all x E A z for which Tmx E D for infinitely many positive and negative values of m. The set G is measurable and has measure 1, by the ergodic theorem. For x E G, define mo = mo(x) to be the least nonnegative integer m such that Tx E D, then extend to obtain an increasing sequence mi = m i (x), j E Z such that Tinx E D if and only if m = mi, for some j. The next step is to split each interval [mi, m 1+1) into nonoverlapping blocks of length N, starting with mi, plus a final remainder block of length shorter than N, in case mi+i — mi is not exactly divisible by N. In other words, for each j let qi and ri be nonnegative integers such that mi +i—mi = qiN d-ri, 0 0. Later it will be shown that the class of mixing processes is a-closed, hence the nonmixing process v cannot be the d-limit of any sequence of mixing processes.

1.9.b.1

The joining definition of d.

The definition of d-distance given by Theorem 1.9.2 serves as a useful guide to intuition, but it is not always easy to use in proofs and it doesn't apply to nonergodic istance as a processes. To remedy these defects an alternative formulation of the d-d limit of a distance dn un , vn ) between the corresponding n - th order distributions will be developed. This new process distance will also be denoted by d(A, v). In a later subsection it will be shown that this new process distance is the constant given by Theorem 1.9.2. (

A key concept used in the definition of the 4-metric is the "join" of two measures. A joining of two probability measures A and y on a finite set A is a measure A. on A x A that has ,u and y as marginals, that is,

(1)

,a(a) =

E X(a, b); v(b) = E X(a, b). b

a

A simple geometric model is useful in thinking about the joining concept. First each of the two measures, kt and y, is represented by a partition of the unit square into horizontal

91

SECTION 1.9. PROCESS TOPOLOGIES.

strips, with the width of a strip equal to the measure of the symbol it represents. The joining condition ,u(a) = Eb X(a, 6) means that, for each a, the ,u-rectangle corresponding to a can be partitioned into horizontal rectangles {R(a,b):b E A} such that R(a,b) has area X(a, b). The second joining condition, v(b) = Ea X.(a, 6), means that the total mass of the rectangles {R(a, b): a E A} is exactly the y-measure of the rectangle corresponding to b. (See Figure 1.9.5.) In other words, a joining is just a rule for cutting each A-rectangle into subrectangles and reassembling them to obtain the y-rectangles.

0

R(0,a)

R(0,a)

a R(0,c)

R(1,a)

R(1,a) R(2,a)

1 R(1b)

R(1,b) R(2a)

2

R(2,b) R(0,c) R(2,c)

R(2,b) R(2c)

Figure 1.9.5 Joining as reassembly of subrectangles. Of course, one can also think of cutting up the y-rectangles and reassembling to give the A-rectangles. Also, the use of the unit square and rectangles is for simplicity only, for a finite distribution can always be represented as a partition of any nonatomic probability space into sets whose masses are given by the distribution. (A space is nonatomic if any subset of positive measure contains subsets of any smaller measure.) A representation of p, on a nonatomic probability space (Z, a) is just a measure-preserving mapping ct. from (Z, a) to (A, A), that is, a measurable mapping such that a(0 -1 (a)) = p(a), a E A. A joining X can then be represented as a pair of measure-preserving mappings, (/) from (Z, a) to (A, A), and ik from (Z, a) to (A, y), such that X is given by the formula

(2)

X(a i , ai )

a (q5-1 (ai )

n ( - 1 (c)).

Turning now to the definition of y), where p, and v are probability measures on An , let J(11, y) denote the set of joinings of p, and v. The 4-metric is defined by cin(ti, V) =

min

Ejdn(4 ,

where ex denotes expectation with respect to X. The minimum is attained, since expectation is continuous with respect to the distributional distance IX — 5,1In, relative to which Jn(Itt, V) is a compact subset of the space 7'(A x A n ). A measure X on A n x A n is said to realize v) if it is a joining of A and y such that E Vd n (x7, yti')) = cin v). The function 4(A, y) satisfies the triangle inequality and is strictly positive if p, y, hence is a metric on the class P(Afl) of measures on A", see Exercise 2.

CHAPTER I. BASIC CONCEPTS.

92

The d-distance between two processes is defined for the class P(A) of probability measures on A" by passing to the limit, that is,

d(it, Y) = lirn sup an(An, un). n—).co

Note that d(p,, y) is a pseudometric on the class P(A), since d, is a metric on and hence d defines a topology on P(A"). In the stationary case, the limit superior is, in fact, both the limit and the supremum.

Theorem 1.9.6 If i and v are stationary then C1 (1-1 , v) = SUP dn (An, vn) = limiin(An.vn)•

Proof The proof depends on a superadditivity inequality, which is, in turn, a consequence of the definition of dn as an minimum, namely,

(3)

n

vilz)± m

(p,:tr,, v'nV) < ( n + m) di±,n (1t7+m , q+m),

where pi denotes the measure induced by p, on the set AL. The proof of this is left to Exercise 3. In the stationary case, p: +41 = pr, so the preceding inequality takes the form nd + mdm < n + m)dn+m, (

which implies that the limsup is in fact both a limit and the supremum. This establishes Theorem 1.9.6. LI A consequence of the preceding result is that the d-pseudometric is actually a metric on the class P(A) of stationary measures. Indeed, if d(p,, v) = 0 and p and y are stationary, then, since in the stationary case the limit superior is a supremum, dn (p,n , un ) must be 0, for all n, which implies that A„ = vn for all n, since iin(A, y) is a metric, Exercise 2, which, in turn, implies that p = v. The d-distance between stationary processes is defined as a limit of n-th order dndistance, which is, in turn, defined as a minimization of expected per-letter Hamming distance over joinings. It is useful to know that the d-distance can be defined directly in terms of stationary joining measures on the product space A" x A". A joining A of p and v is just a measure on A x A" with p, and y as marginals, that is,

p(B) = A.(B x

v(B) = X(A" x B), B

E

E.

The set of all stationary joinings of p and y will be denoted by Js(p,, y).

Theorem 1.9.7 (Stationary joinings and d-distance.) If p, and v are stationary then (4)

cl(p, v) =

min

ex(d(xi, yl)).

SECTION 1.9. PROCESS TOPOLOGIES.

93

Proof The existence of the minimum follows from the fact that the set .1s(p., y) of stationary joinings is weakly closed and hence compact, and from the fact that e ,k (d(x i , Yi)) is weakly continuous, for it depends only on the first order distribution. It is also easy to see that d cannot exceed the minimum of Ex (d(x l , yi)), for if A E Js(, y) then stationarity implies that

ex(d(xi, yl)) = exn (dn (4, y rit )), and the latter dominates an (IL, y), since A n belongs to -In(un, va). It takes a bit more effort to show that d cannot be less than the right-hand side in (4). Towards this end, for each n choose A n E Jn(tin, va), such that an(An, vn) = ex(c1,(4 ,

The goal is to construct a stationary measure A E y) from the collection {A}, such that ex (d(Xi , yi ) = lima Cin (tt. v). This is done by forming, for each n, the concatenatedblock process defined by A n , then taking a weakly convergent subsequence to obtain the desired limit process À. The details are given in the following paragraphs. For each n, let ),-.01) denote the concatenated-block process defined by An. As noted in Exercise 8 of Section 1.1, the measure 7,(n) is given by the formula i "(n) (B) = —

(5)

n

n 1=1

where A. (n) is the measure on (A x A )° defined by A((4, yr)) = K-1 (

H

(t vknn -l"n kk"kn-I-1

.kn-Fn

t( y Kn+r

Ykn+1 //

"n kk""Kn-1-1

)1(n+1 , / ,

k=0

if N = Kn+r, 0 el) 5_ E.

I.9.b.4 Ergodicity, entropy, mixing, and d limits. In this section it will be shown that the d-limit of ergodic (mixing) processes is ergodic (mixing) and that entropy is d-continuous. The key to these and many other limit results is that do -closeness implies that sets of large measure for one distribution must have large blowup with respect to the other distribution, which is just Lemma 1.9.14. As before, for 3 > 0, let [B]s = {y7: dn(y ri' , B) 3), denote the 3-neighborhood or 3-blow-up of B c A". Theorem 1.9.15 (Ergodicity and d-limits.) The a-limit of ergodic processes is ergodic. Proof Suppose ,u is the d-limit of ergodic processes. By the ergodic theorem the relative

frequency p(eil`ix) converges almost surely to a limit p(anx). By Theorem 1.4.2 it is enough to show that p(aii`lx) is constant in x, almost surely, for every a. This follows from the fact that if v is ergodic and close to i, then a small blowup of the set of vtypical sequences must have large ti-measure, and small blowups don't change empirical frequencies very much. To show that small blowups don't change frequencies very much, fix k and note that if )7 4-k---1 = all' and 4 +k-1 0 al', then yi 0 x i for at least one j E [i, i + k — 1], and hence (14) (n — k + 1)p(41),11 ) < (n — k + 1)p(a in.q) + knd,(4 , )1), where the extra factor of k on the last term is to account for the fact that a j for which yi 0 x i can produce as many as k indices i for which y :+ k -1 _ — ak1 and X tj. ±1c-1 0 ak1' Let c be a positive number and use (14) to choose a positive number S < c, and N1 so that if n > N1 then (15)

d( x,

11

)

3

I p(44) — p(any)1 < c.

CHAPTER I. BASIC CONCEPTS.

100

Let y be an ergodic measure such that cl(A, y) < 3 2 and hence an (11,n , vn ) < 82, n > Because y is ergodic N1 , since for stationary measures d is the supremum of the there is an N2 > Ni such that if n > N2, the set

an .

Bn = fx ril : I P(414) — v(4)1 < has vn measure at least 1-6. Fix n > N2 and apply Lemma 1.9.14 to obtain A n ([13, ] 5 ) > 1 — 2 8 . Note, however, that if y E [A]s then there is a sequence x E Bn such that dn (fi! , 1,

y: .- .;1 })

- iz X({(xr +n , y'1 ): 4 0 yrii or x: V X({(xi, yi): xi 0 4 } )

X({(x,nm F-7,

m+n

xm _11

ymm:++:rill)

,n

28,

so that if 8 1. Since u is mixing, however,

Iv(a7)v(b7) — vgan

n T - m[b7])1

6/3,

for all sufficiently large m, and hence, the triangle inequality yields

— ii([4]

n T -m[brill)1

6

,

for all sufficiently large m, provided only that 6 is small enough. Thus p, must be mixing, which establishes the theorem. This section will be closed with an example, which shows, in particular, that the

ii-topology is nonseparable. Example 1.9.18 (Rotation processes are generally d-far-apart.) Let S and T be the transformations of X = [0, 1) defined, respectively, by Sx = x

where, as before,

e denotes

a, T x = x

fi,

addition modulo 1.

Proposition 1.9.19 Let A be an (S x T)-invariant measure on X x X which has Lebesgue measure as marginal on each factor If a and fi are rationally independent then A must be the product measure. Proof "Rationally independent" means that ka ± mfi is irrational for any two rationals k and m with (k, m) (0, 0). Let C and D be measurable subsets of X. The goal is to • show that A.(C x D) = It is enough to prove this when C and D are intervals and p.(C) = 1/N, where N is an integer. Given E > 0, let C 1 be a subinterval of C of length (1 — E)1N and let

E = C i x D, F = X x D.

CHAPTER I. BASIC CONCEPTS.

102

Since a and fi are rationally independent, the two-dimensional version of Kronecker's theorem, Proposition 1.2.16, can be applied, yielding integers m 1 , m2, , rnN such that if V denotes the transformation S x T then Vms E

n vrni

E = 0, if i j,

and X(FAÉ) < 2E,

where F' = U 1 Vrn1E .

It follows that X(E) = X(P)/N is within 26/N of A.(F)/ N = 1,t(C)A(D). Let obtain X(C x D) =

E -±

0 to

El

This completes the proof of Proposition 1.9.19.

Now let P be the partition of the unit interval that consists of the two intervals Pc, = [0, 1/2), Pi = [1/2, 1). It is easy to see that the mapping that carries x into its (T, 2)-name {x} is an invertible mapping of the unit interval onto the space A z the Kolmogorov measure. This fact, together withwhicaresLbgumont Proposition 1.9.19, implies that the only joining of the (T, 2)-process and the (S, 2)process is the product joining, and this, in turn, implies that the d-distance between these two processes is 1/2. This shows in particular that the class of ergodic processes is not separable, for, in fact, even the translation (rotation) subclass is not separable. It can be shown that the class of all processes that are stationary codings of i.i.d. processes is d-separable, see Exercise 3 in Section W.2.

I.9.c

Exercises.

1. A measure ,u E P(X) is extremal if it cannot be expressed in the form p = Ai 0 ia2. (a) Show that if p is ergodic then p, is extremal. (Hint: if a = ta1 -F(1 — t)A2, apply the Radon-Nikodym theorem to obtain gi (B) = fB fi diu and show that each fi is T-invariant.) (b) Show that if p is extremal, then it must be ergodic. (Hint: if T -1 B = B then p, is a convex sum of the T-invariant conditional measures p(.iB) and peiX — B).) 2. Show that

l') is a complete metric by showing that

(a) The triangle inequality holds. (Hint: if X joins X and /7, and X* joins 17 and Zri', then E,T yi)A*(z7ly'll) joins X and 4.) (b) If 24(X?, ri) = 0 then Xi; and 1111 have the same distribution. (c) The metric d(X7, 17) is complete. 3. Prove the superadditivity inequality (3). 4. Let p, and y be the binary Markov chains with the respective transition matrices m [

P —p

1—p p 1'

Let fi be the Markov process defined by M2.

N [0 1 1 o]'

1

SECTION 1.10. CUTTING AND STACKING.

103

(a) Show that if x is typical for it, and vv n = X2n, n = 1, 2, ..., then, almost surely, y is typical for rt. (b) Use the result of part (a) to show d(,u, y) = 1/2, if 0 < p < 1. 5. Use Lemma 1.9.11(f) to show that if it. and y are i.i.d. then d(P., y) = (1/2)1A — v I i. (This is a different method for obtaining the d-distance for i.i.d. processes than the one outlined in Example 1.9.3.) 6. Suppose y is ergodic and rt, is the concatenated-block process defined by A n on A n . ii ) (Hint: g is concentrated on shifts of sequences Show that d(v ,g) = --n.-(pn, ,.--n,• that are typical for the product measure on (An )° ° defined by lin .)

a.

7. Prove property (d) of Lemma 1.9.11. 8. Show there is a joining A.* of ,u and y such that Xn* (a7 , a7) = min(p,(a7), v(a)). 9. Prove that ikt — v in = 2 — 2 Ea7 min(i.t(a7), v(a)). 10. Two sets C, D C A k are a-separated if C n [D], = 0. Show that if the supports of 121 and yk are a-separated then dk(Ilk.vk) > a. 11. Suppose ,u, and y are ergodic and d(i.t, for sufficiently large n there are subsets v(Dn ) > 1 — E, and dn (x7, y7) > a Pk('IX 11`) ' Ak and Pk('IY1 c ) ' Vk, and k much smaller than a, by (7))

y) > a. Show that if E > 0 is given, then Cn and Dn of A n such that /L(C) > 1 —e, — e, for x7 E Cn , yli' E D. (Hint: if is large enough, then d(4 , y ) cannot be

12. Let (Y, y) be a nonatomic probability space and suppose *: Y 1-4- A n is a measurable mapping such that dn (un , y o * -1 ) < S. Show that there is a measurable mapping q5: y H A n such that A n = V curl and such that Ev(dn(O(Y), fr(y))) 1 is left as an exercise. 0 Next the question of the ergodicity of the final process will be addressed. One way to make certain that the process defined by a complete sequence is ergodic is to make sure that the transformation defined by the sequence is ergodic relative to Lebesgue measure. A condition for this, which is sufficient for most purposes, is that later stage structures become almost independent of earlier structures. These ideas are developed in the following paragraphs. In the following discussion C denotes either a column or its support. For example, A(B I C) denotes the conditional measure X(B n c)/x(c) of the intersection of the set B with the support of the column C. The column structures S and S' are 6-independent if

(3)

E E Ige n D) — X(C)À(D)I

6.

CES DES'

In other words, two column structures S and S' are 6-independent if and only if the partition into columns defined by S and the partition into columns defined by S' are 6-independent. The sequence {S(m)} of column structures is asymptotically independent if for each m and each e > 0, there is a k > 1 such that S(m) and S(m k) are 6-independent. Note, by the way, that these independence concepts do not depend on how the columns are labeled, but only the column distributions. Related concepts are discussed in the exercises.

SECTION I10. CUTTING AND STACKING.

113

Theorem 1.10.6 (Complete sequences and ergodicity.) A complete asymptotically independent sequence defines an ergodic process.

Proof Let T be the Lebesgue measure-preserving transformation of the unit interval defined by the complete, asymptotically independent sequence { 8(m)}. It will be shown that T is ergodic, which implies that the (T, P)-process is ergodic for any partition P. Let B be a measurable set of positive measure such that T -1 B = B. The goal is to show that A(B) = 1. Towards this end note that since the top of S(m) shrinks to 0, the widths of its intervals must also shrink to 0, and hence the collection of all the levels of all the columns in all the S(m) generates the a-algebra. Thus, given E > 0 there is an m and a level L, say level j, of some C E 8(M) such that

X(B n L) > 0 This implies that the entire column C is filled to within (1 — E 2 ) by B, that is,

À(B n c)

(4)

> 1 - E 2 )x(c), (

since T k (B n L) = B n TkL and T k L sweeps out the entire column as k ranges from —j + 1 to aW) — j. Fix C and choose M so large that S(m) and S(M) are EX(C)-independent, so that, in particular, the following holds.

(5)

E

igvi c) — x(D)1

E-

DES(M)

Let .T be the set of all D E S(M) for which X(B1 C n D) > 1 — 6. The set C n D is a union of levels of D, since S(M) is built by cutting and stacking from S(m), so if D E .F, then there must be at least one level of D which is at least (1 — E) filled by B. The argument used to prove (4) then shows that the entire column D must be (1 — c) filled by B. In summary,

(6)

X(B n D) > (1 — c)X(D), D e .F.

Since X(B I C) = E-D A.(B1 C X(B I C) ?.. (1 — 6 2 ), imply that

n v)x(vi C), the Markov inequality and the fact that

E x (7, 1 C)

El

DO'

which together with the condition, (5), implies

E X(D) 1 — 3E. This shows that X(B) = 1, and hence T must be ergodic. This completes the proof of Theorem 1.10.6. El If is often easier to make successive stages approximately independent than it is to force asymptotic independence, hence the following stronger form of the complete sequences and ergodicity theorem is quite useful.

CHAPTER I. BASIC CONCEPTS.

114

Theorem 1.10.7 (Complete sequences and ergodicity: strong form.) If {S(m)} is a complete sequence such that for each m, S(m) and S(m 1) are Em -independent, where Em 0, as m OC, then {S(m)} defines an ergodic process.

Proof Assume T -1 B = B and X(B) > O. The only real modification that needs to be made in the preceding proof is to note that if {Li} is a disjoint collection of column levels such that X (BC n (UL)) = BciL i , x ( L i) < E 28, (7)

Eg

where BC is the complement of B, then by the Markov inequality there is a subcollection with total measure at least 3 for which

x(B n L i ) > (1 —

E 2 )À(L i ).

Let S' be the set of all columns C for which some level has this property. The support of S' has measure at least 3, and the sweeping-out argument used to prove (4) shows that (8) X(B n > ( 1 — E2 E SI

)x(c), c

.

Thus, taking 3 = ,u(B)/2, and m so large that Em is smaller than both E 2 and (ju(B)/2) 2 so that there is a set of levels of columns in S(m) such that (7) holds, it follows that and there must be at least one C E S(m) for which (8) holds and for which

E

Ix(Dic)

—

x(D)i
1 — 3E. This proves Theorem 1.10.7. The freedom in building ergodic processes via cutting and stacking lies in the arbitrary nature of the cutting and stacking rules. The user is free to vary which columns are to be cut and in what order they are to be stacked, as well as how substructures are to become well-distributed in later substructures. There are practical limitations, of course, in the complexity of the description needed to go from one stage to the next. A bewildering array of examples have been constructed, however, by using only a few simple techniques for going from one stage to the next. A few of these constructions will be described in later chapters of this book. The next subsection will focus on a simple form of cutting and stacking, known as independent cutting and stacking, which, in spite of its simplicity, can produce a variety of counterexamples when applied to substructures.

I.10.d

Independent cutting and stacking.

Independent cutting and stacking is the geometric version of the product measure idea. In this discussion, as earlier, the same notation will be used for a column or its support, with the context making clear which meaning is intended. A column structure S can be stacked independently on top of a labeled column C of the same width, provided that they have disjoint supports. This gives a new column structure denoted by C * S and defined as follows. (i) Partition C into subcolumns {Ci } so that r (C1) = (Di ), where Di is the ith column of S.

115

SECTION 1.10. CUTTING AND STACKING.

(ii) Stack Di on top of Ci to obtain the new column, Ci *Di . The new column structure C * S consists of all the columns C, * V. It is called the independent stacking of S onto C. (See Figure 1.10.8.)

C* S Figure 1.10.8 Stacking a column structure independently onto a column. To say precisely what it means to independently cut and stack one column structure on top of another column structure the concept of copy is useful. A column structure S is said to be a copy of size a of a column structure S' if there is a one-to-one correspondence between columns such that corresponding columns have the same height and the same labeling, and the ratio of the width of a column of S to the width of its corresponding column in S' is a. In other words, a scaling of one structure is isomorphic to the other, where two column structures S and S' are said to be isomorphic if there is a one-to-one correspondence between columns such that corresponding columns have the same height, width, and name. Note, by the way, that a copy has the same width distribution as the original. A column structure S can be cut into copies {S i : i E n according to a distribution on /, by partitioning each column of S according to the distribution 7r, and letting Si be the column structure that consists of the i-th subcolumn of each column of S. Let S' and S be disjoint column structures with the same width. The independent cutting and stacking of S' onto S is denoted by S * S' and is defined for S = {Ci } as follows. (See Figure 1.10.9.) (i) Cut S' into copies {S; so that r(S:)= .1- (Ci ). (ii) For each i, stack S; independently onto Ci , obtaining Ci * S;. The column structure S * S' is the union of the column structures Ci *

C2

S'

Figure 1.10.9 Stacking a structure independently onto a structure. An alternative description of the columns of S * S' may given as follows.

CHAPTER I. BASIC CONCEPTS.

116 (i) Cut each C each Ci E S.

E

S' into subcolumns {C» } such that r(C) = (C)T- (Cii ) I r (S') for

(ii) Cut each column Ci

E

S into subcolumns {CO, such that r(Cii ). -r(Ci').

The column structure S *S' consists of all the C,J*Cii . In particular, in the finite column case, the number of columns of S * S' is the product of the number of columns of S and the number of columns of S'. The key property of independent cutting and stacking, however, is that width distributions multiply, that is, r (Cii *C)

r (Ci )

r (S * S')

r (S)

r(C)

since r (S*S') = r(S). This formula expresses the probabilistic meaning of independent cutting and stacking: cut and stack so that width distributions multiply. A column structure can be cut into copies of itself and these copies stacked to form a new column structure. The M-fold independent cutting and stacking of a column structure S is defined by cutting S into M copies {Sm : m = 1, 2, ... , M} of itself of equal width and successively independently cutting and stacking them to obtain Si

*52 * • • ' * SM

where the latter is defined inductively by Si *. • • *Sm = (S1* • • • *Sm_i)*Sm. Two-fold independent cutting and stacking is indicated in Figure 1.10.10.

Si

S2

SI

* S2

Figure 1.10.10 Two-fold independent cutting and stacking. Note that (Si * • • • * S) = t (S)/M, and, in the case when S has only finitely many columns, the number of columns of Si * • • • * Sm is the M-th power of the number of columns of S. The columns of Si * • • • * Sm have names that are M-fold concatenations of column names of the initial column structure S. The independent cutting and stacking construction contains more than just this concatenation information, for it carries with it the information about the distribution of the concatenations, namely, that they are independently concatenated according to the width distribution of S. Successive applications of repeated independent cutting and stacking, starting with a column structure S produces the sequence {S(m)}, where 5 (1) = S, and

S(m + 1) = S(m) * S (m), m > 1. The sequence {S(m)} is called the sequence built (or generated) from S by repeated independent cutting and stacking. Note that S(m) is isomorphic to the 2m-fold independent

117

SECTION I.10. CUTTING AND STACKING.

cutting and stacking of S, so that its columns are just concatenations of 2' subcolumns of the columns of S, selected independently according to the width distribution of S. In the case when X(S) = 1, the (T, P)-process defined by the sequence generated from S by repeated independent cutting and stacking is called the process built from S by repeated independent cutting and stacking. This is, of course, just another way to define the regenerative processes, for they are the precisely the processes built by repeated independent cutting and stacking from the columnar representation of their firstorder block-structure distributions, Exercise 8. Ergodicity is guaranteed by the following theorem. Theorem 1.10.11 (Repeated independent cutting and stacking.) If {S(m)} is built from S by repeated independent cutting and stacking, then given > 0, there is an m such that S and S(m) are E-independent. In particular, if X(S) = 1 then the process built from S by repeated independent cutting and stacking is ergodic. Proof The proof for the case when S has only a finite number of columns will be given here, the countable case is left to Exercise 2. The notation D = C = C1 C2 .. • Ck, will mean the column D was formed by taking, for each i, a subcolumn of Ci E S, and stacking these in order of increasing i. For k = 2', D = C1C2. • • Ck E SOTO, and C E S, define E [1, k]:C = p(CICO I = so that, in particular, kp(CICf) is the total number of occurrences of C in the sequence C. By the law of large numbers, (9)

p(CIC) —>

and (10)

1 1 — i(D) = — k k

E £(0)

E(t(S)),

i=1

oc, where E(t(S)) is the expected height of the columns both with probability 1, as k of S, with respect to the width distribution. Division of X(C n D) = kp(C1Cbt(C)r(D) by the product of X(C) = i(C)r(C) and À(D) = i(D)t(D) yields

x(c n D) X(C)À(D)

kp(CIC I ) i(D)r (C)

1,

with probability 1, as k —> cc, by (9) and (10), since r(S)E(t(S)) = X(S) = 1, by assumption. This establishes that indeed S and S(m) are eventually c-independent. Note that for each mo, {S(m): m > mol is built from S(rno) by repeated independent cutting and stacking, and hence the proof shows that {S(m)} is asymptotically independent, so that if X(S) = 1 then the process defined by {S(m)} is ergodic. This completes the proof of Theorem 1.10.11. The following examples indicate how some standard processes can be built by cutting and stacking. More interesting examples of cutting and stacking constructions are given in Chapter 3. The simplest of these, an example of a process with an arbitrary rate of

118

CHAPTER I. BASIC CONCEPTS.

convergence for frequencies, see III.! .c, is recommended as a starting point for the reader unfamiliar with cutting and stacking constructions. It shows how repeated independent cutting and stacking on separate substructures can be used to make a process "look like another process on part of the space," so as to approximate the desired property, and how to "mix one part slowly into another," so as to guarantee ergodicity for the final process.

Example 1.10.12 (I.i.d. processes.) Let S be the column structure that consists of two columns each of height 1 and width 1/2, labeled as '0' and '1'. The process built from S by repeated independent cutting and stacking is just the coin-tossing process, that is, the binary i.i.d. process in which O's and l's are equally likely. By starting with a partition of the unit interval into disjoint intervals labeled by the finite set A, repeated independent cutting and stacking produces the A-valued i.i.d. process with the probability of a equal to the length of the interval assigned to a. Example 1.10.13 (Markov processes.) The easiest way to construct an ergodic Markov chain {X„} via cutting and stacking is to think of it as the regenerative process defined by a recurrent state, say a. Let S be the set of all aki for which ai a,1 < i < k, and ak = a. Define Prob(X i` = alic1X0 = a),

all'

E

S,

and let pt* be the product measure on S defined by p,T. The process built from the first-order columnar representation of S by repeated independent cutting and stacking is the same as the Markov process {X„}. Example 1.10.14 (Hidden Markov chains.) A hidden Markov process {X,} is a function X, = f (Y,) of a Markov chain. (These also known as finite-state processes.) If {KJ is ergodic, then {X,} is regenerative, hence can be represented as the process built by repeated independent cutting and stacking. In fact, one can first represent the ergodic Markov chain {(Xn , Yn )} as a process built by repeated independent cutting and stacking, then drop the first coordinates of each label. Example 1.10.15 (Finite initial structures.) If the initial structure S has only a finite number of columns, the process {X,} built from it by repeated independent cutting and stacking is a hidden Markov chain. To see why this is so, assume that the labeling set A does not contain the symbols 0 and 1 and relabel the column levels by changing the label A (L) of level L to (A (L), 0) if L is not the top of its column, and to (.Ar(L), 1) if L is the top of its column. It is easy to see that the process {KJ built from this new column structure by repeated independent cutting and stacking is Markov of order no more than the maximum length of its columns. Since dropping the second coordinates of each new label produces the old labels, it follows that X, is a function of Y. Note, in particular, that if the original structure has two columns with heights differing by 1, then {Y, i } is mixing Markov, and hence {X,} is a function of a mixing Markov chain. Remark 1.10.16 The cutting and stacking ideas originated with von Neumann and Kakutani. For a discussion of this and other early work see [12]. They have since been used to construct

SECTION 1.10. CUTTING AND STACKING.

119

numerous counterexamples, some of which are mentioned [73]. The latter also includes several of the results presented here, most of which have long been part of the folklore of the subject. A sufficient condition for a cutting and stacking construction to produce a stationary coding of an i.i.d. process is given in [75].

I.10.e Exercises 1. Show that if T is the upward map defined by a column C, then a subcolumn has the form (D, T D,T 2 D, ... , re(c)-1 D), where D is a subset of the base B. 2. Prove Theorem 1.10.11 for the case when S has countably many columns. 3. Show that formula (2) holds for k > 1. (Hint: the tower with base transformation Sn defines the same process Ti. Then apply the k = 1 argument.) 4. A column C E S is (1 —6)-well-distributed in S' if EDES' lx(vi C)—X(7))1 < E. A sequence {S(m)} is asymptotically well-distributed if for each m, each C E S(m), and each E > 0, there is an M = M(m,C, E) such that C is (1 —6)-well-distributed in S(n) for n > M. (a) Suppose S' is built by cutting and stacking from S. Show that for C E S and D E S', the conditional probability X(DI C) is the fraction of C that was cut into slices to put into D. (b) Show that the asymptotic independence property is equivalent to the asymptotic well-distribution property. (c) Show that if S and S' are 6-independent, then E CES IX(CI D) - - - À(D)i except for a set of D E S I of total measure at most Nfj. (d) Show that if EcEs IX(C1 D) — X(D)I < E, except for a set of D E SI of total measure at most E, then S' and S are 26-independent. 5. Let pt" be a block-structure measure on Soe, where S C A*, and let {S(m)} be the standard cutting and stacking representation of the stationary A-valued process defined by (S, p*). Show that {S(m)} is asymptotically independent if and only if it* is totally ergodic. (Hint: for S = S(m), Exercise 4c implies that A* is m-ergodic.) 6. Suppose 1(m) c S(m), for each m, and X(1 (m)) -->- 1 as m —> oc. (a) Show that {S(m)} is asymptotically independent if and only if {r(m)} is asymptotically independent. (b) Suppose that for each m there exists R(m) C S(m), disjoint from 1(m), and an integer 111,n such that T(m + 1) is built by An -fold independent cutting and stacking of 1(m) U R.(m). Suppose also that {S(m} is complete. Show that if {Mm } increases fast enough then the process defined by {S(m)} is ergodic. 7. Suppose S has measure 1 and only finitely many columns. (a) Show that if the top of each column is labeled '1', with all other levels labeled '0', then the process built from S by repeated independent cutting and stacking is Markov of some order.

120

CHAPTER I. BASIC CONCEPTS.

(b) Show that if two columns of S have heights differing by 1, then the process built from S by repeated independent cutting and stacking is a mixing finitestate process. (c) Verify that the process constructed in Example 1.10.13 is indeed the same as the Markov process {X„}. 8. Show that process is regenerative if and only if it is the process built from a column structure S of measure 1 by repeated independent cutting and stacking.

Chapter II Entropy-related properties. Section 11.1 Entropy and coding. As noted in Section I.7.c, entropy provides an asymptotic lower bound on expected per-symbol code length for prefix-code sequences and faithful-code sequences, a lower bound which is "almost" tight, at least asymptotically in source word length n, see Theorem 1.7.12 and Theorem 1.7.15. Two issues left open by these results will be addressed in this section. The first issue is the universality problem. The sequence of Shannon codes compresses to entropy in the limit, but its construction depends on knowledge of the process, knowledge which may not be available in practice. It will be shown that there are universal codes, that is, code sequences which compress to the entropy in the limit, for almost every sample path from any ergodic process. The second issue is the almost-sure question. The entropy lower bound is an expected-value result, hence it does not preclude the possibility that there might be code sequences that beat entropy infinitely often on a set of positive measure. It will be shown that this cannot happen. An n-code is a mapping Cn : A n F-* 10, 1}*, where {0, 1}* is the set of finite-length binary sequences. The code length function Le Icn ) of the code is the function that assigns to each .7q , the length i(Cn (4)) of the code word Cn (4). When Cn is understood, r (4) will be used instead of r (x rii ICn ). A code sequence is a sequence {Cn : n = 1, 2, ...), where each Cn is an n-code. If each Cn is one-to-one, the code sequence is called a faithful-code sequence, while if each C, is a prefix code it is called a prefix-code sequence. The Shannon code construction provides a prefix-code sequence {C n } for which L(xrii) = r—log,u(4)1, so that, in particular, L(4)In h, almost surely, by the entropy theorem. In general, however, if {C n } is a Shannon-code sequence for A, and y is some other ergodic process, then L(4)1 n may fail to converge on a set of positive y-measure, see Exercise 1. A code sequence {C n } is said to be universally asymptotically optimal or, more simply, universal if

L(4) < h(p), almost surely, n n—>co

lim sup

for every ergodic process ,u, where h(p) denotes the entropy of A.

121

CHAPTER II. ENTROPY-RELATED PROPERTIES.

122

Theorem 11.1.1 (Universal codes exist.) There is a prefix-code sequence {C„} such that limsup n ,C(fii)1 n < h(a), almost surely, for any ergodic process ,u. Theorem 11.1.2 (Too-good codes do not exist.) If {C n } is a faithful-code sequence and it is an ergodic measure with entropy h, then lim inf, L(x 11 )/n > h, almost surely. In other words, it is possible to (universally) compress to entropy in the limit, almost surely for any ergodic process, but for no ergodic process is it possible to beat entropy infinitely often on a set of positive measure. A counting argument, together with some results about entropy for Markov chains, will be used to establish the universal code existence theorem, Theorem 11.1.1. The nonexistence theorem, Theorem 11.1.2, will follow from the entropy theorem, together with a surprisingly simple lower bound on prefix-code word length. A second proof of the existence theorem, based on entropy estimation ideas of Ornstein and Weiss, will be discussed in Section 11.3.d. A third proof of the existence theorem, based on the Lempel-Ziv coding algorithm, will be given in Section 11.2. Of interest, also, is the fact that both the existence and nonexistence theorems can be established using only the existence of process entropy, whose existence depends only on subadditivity of entropy as expected value, while the existence of decay-rate entropy is given by the much deeper entropy theorem. The proof of the existence theorem to be given actually shows the existence of codes that achieve at least the process entropy. A direct coding argument, which is based on a packing idea and is similar in spirit to some later proofs, will then be used to show that it is not possible to beat process entropy in the limit on a set of positive measure. These two results also provide an alternative proof of the entropy theorem. While no simpler than the earlier proof, they show clearly that the basic existence and nonexistence theorems for codes are, in essence, together equivalent to the entropy theorem, thus further sharpening the connection between entropy and coding. In addition, the direct coding construction extends to the case of semifaithful codes, for which a controlled amount of distortion is allowed, as shown in [49].

II.1.a

Universal codes exist.

A prefix-code sequence {C} will be constructed such that for any ergodic it,

lim sup ,C(4)/n < H, a.s., n

where H = 114 denotes the process entropy of pt. Since process entropy H is equal to the decay-rate h given by the entropy theorem, this will show that good codes exist. The code C, utilizes the empirical distribution of k-blocks, that is, the k-type, for suitable choice of k as a function of n, with code performance established by using the type counting results discussed in Section I.6.d. The code is a two-part code, that is, the codeword Cn (4) is a concatenation of two binary blocks. The first block gives the index of the k-type of the sequence 4, relative to some enumeration of possible k-types; it has fixed length which depends only on the number of type classes. The second block gives the index of the particular sequence x'iz in its k-type class, relative to some enumeration of the k-type class; its length depends on the size of the type class. Distinct words, 4 and yril, either have different k-types, in which case the first blocks of Cn (fi') and Cn (Yil)

SECTION 11.1. ENTROPY AND CODING.

123

will be different, or they have the same k-types but different indices in their common k-type class, hence the second blocks of Cn (4) and Cn (y;`) will differ. Since the first block has fixed length, the code Cn is a prefix code. If k does not grow too rapidly with n, say, k -- (1/2) log IAI n, then the log of the number of possible k-types is negligible relative to n, so asymptotic code performance depends only on the asymptotic behavior of the cardinality of the k-type class of x. An entropy argument shows that the log of this class size cannot be asymptotically larger than nH . A slight modification of the definition of k-type, called the circular k-type, will be used as it simplifies the final entropy argument, yet has negligible effect on code performance. Given a sequence 4 and an integer k < n, let ril+k-1 = xx ki -1 , the concatenation of xtiz with x ki -1 , that is, 4 is extended periodically for k — 1 more terms. The circular k-type is the measure Pk = ilk(.IX II ) on il k defined by the relative frequency of occurrence of each k-block in the sequence .4+k-1 , that is,

lic 14) — Pk(a •

1 ti E [1, rd: 5.,,:+ k -1 = ak 11 , aik E A k . 1f1

n

The circular k-type is just the usual k-type of the sequence .7irk-1 , so that the bounds, Theorem 1.6.14 and Theorem 1.6.15, on the number of k-types and on the size of a k-type class yield the following bounds (1) (2)

(k, n) < (n ± olAl k , ITi(4)1 < (n — 1)2 (11-1)11- "7

on the number /V (k, n) of circular k-types that can be produced by sequences of length n, and on the cardinalityj7k (x)i of the set of all n-sequences that have the same circular k-type as 4, where Hk_1, x 7 denotes the entropy of the (k — 1)-order Markov chain defined by Pk (.14). The gain in using the circular k-type is compatibility in k, that is,

Pk-1(4 -1 14) =---"

EFkcal,c14), ak

which, in turn, implies the entropy inequality

(3)

ilk-Le, < iii,f,' , i 0, choose K such that HK_i _< H ± E, where, in this discussion, HK-1 = H (X IX iK -1 ) denotes the process entropy of the Markov chain of order K — 1 defined by the conditional probabilities ,u(ar)/,u(ar-1 ). Since K is fixed, the ergodic theorem implies that PK (ar lx7) oo. In almost surely, as n ,u(4), almost surely, and hence HK _ Lx? integer N = N(x, E) such that almost every x there is an particular, for e■••

N.

For all sufficiently large n, k(n) will exceed K. Once this happens the entropy inequality (3) combines with the preceding inequality to yield ilk(n)-1,x?
H, a.s., n n and completes this proof of the entropy theorem.

C3

II. Le Exercises 1. This exercise explores what happens when the Shannon code for one process is used on the sample paths of another process. (a) For each n let p,,, be the projection of the Kolmogorov measure of unbiased coin tossing onto A n and let Cn be a Shannon code for /in . Show that if y 0 p, is ergodic then C(4)In cannot converge in probability to the entropy of v. (b) Let D (A n il v pi ) = E7x /1,1(4) log(pn (4)/vn (4)) be the divergence of An from yn , and let £(4) be the length function of a Shannon code with respect to vn . Show that the expected value of £(4) with respect to p,,, is 1-1(pn )± D (An II vn). (C) Show that if p, is the all 0 process, then there is a renewal process v such that lim supn D (gn II vn )/n = oc and lim infn D(pn II vn )/n = 0.

2. Let L (xi') denote the length of the longest string that appears twice in 4. Show that if bt is i.i.d. then there is a constant C such that lim sup,, L(4)/ log n < C, almost surely. (Hint: code the second occurrence of the longest string by telling how long it is, where it starts, and where it occurred earlier. Code the remainder by using the Shannon code. Add suitable headers to each part to make the entire code into a prefix code and apply Barron's lemma, Lemma 5.) 3. Suppose Cn* is a one-to-one function defined on a subset S of An , whose range is prefix free. Show that there is prefix n-code Cn whose length function satisfies £(4) < Kn on the complement of S, where K is a constant independent of n, and such that C,,(4) = 1C:(4), for X'il E S. Such a code Cn is called a bounded extension of Cn* to An.

SECTION 11.2. THE LEMPEL-ZIV ALGORITHM

131

Section 11.2 The Lempel-Ziv algorithm. An important coding algorithm was invented by Lempel and Ziv in 1975, [91]. In finite versions it is the basis for many popular data compression packages, and it has been extensively analyzed in various finite and limiting forms. Ziv's proof, [90], that the Lempel-Ziv (LZ) algorithm compresses to entropy in the limit will be given in this section. A second proof, which is due to Ornstein and Weiss, [53], and uses several ideas of independent interest, is given in Section 11.4. The LZ algorithm is based on a parsing procedure, that is, a way to express an infinite sequence x as a concatenation x = w(1)w(2) . • of variable-length blocks, called words. In its simplest form, called here (simple) LZ parsing, the word-formation rule can be summarized by saying: The next word is the shortest new word. To be precise, x is parsed inductively according to the following rules. (a) The first word w(1) consists of the single letter x1. (b) Suppose w(1) ... w(j) = (i) If xn 1 +1 SI {w(1), ... , w(j)} then w(j ± 1) consists of the single letter xn ro. (ii) Otherwise, w(j + 1) = xn7 + 1 , where m is the least integer larger than n 1 such that xmn .1 +1 E {WM, ... , W(j)} and xm+ 1 g {w(1), ... , WW1. Thus, for example, 11001010001000100... parses into 1, 10, 0, 101, 00, 01, 000, 100, ... where, for ease of reading, the words are separated by commas. Note that the parsing is sequential, that is, later words have no effect on earlier words, and therefore an initial segment of length n can be expressed as

(1)

xi! = w(1)w(2)... w(C) y,

where the final block y is either empty or is equal to some w(j), for j < C. The parsing defines a prefix n-code Cn , called the (simple) Lempel-Ziv (LZ) code by noting that each new word is really only new because of its final symbol, hence it is specified by giving a pointer to where the part before its final symbol occurred earlier, together with a description of its final symbol. To describe the LZ code Cn precisely, let 1- .1 denote the least integer function and let f: {0, 1, 2, ... , n) i-- Brl0gn1 and g: A 1--÷ BflogiAll be fixed one-to-one functions. If fiz is parsed as in (1), the LZ code maps it into the concatenation Cn (eii) = b(1)b(2)... b(C)b(C + 1) of the binary words b(1), b(2), ... , b(C), b(C + 1), defined according to the following rules.

132

CHAPTER II. ENTROPY-RELATED PROPERTIES.

(a) If j < C and w(j) has length 1, then b(j) = 0 g(w(j)). (b) If j < C and i < j is the least integer for which w(j) = w(i)a, a E A, then b(j) = 1 f (i)0 g(a).

1) is empty, otherwise b(C (c) If y is empty, then b(C the least integer such that y = w(i).

1) = 1 f (i), where i is

Part (a) requires at most IA l(Flog IA + 1) bits, part (b) requires at most Calog n1 ± Flog I A + 2) bits, and part (c) requires at most Flog ni + 1 bits, so total code length £(..eiz) is upper bounded by

(C+1)logn-FaC-1- 16, where a and /3 are constants. The dominant term is C log n, so to establish universality it is enough to prove the following theorem of Ziv, [90], in which C1(4) now denotes the number of new words in the simple parsing (1).

Theorem 11.2.1 (The LZ convergence theorem.) If p is an ergodic process with entropy h, then (11n)C(4) log n —> h, almost surely. Of course, it is enough to prove that entropy is an upper bound, since no code can beat entropy in the limit, by Theorem 11.1.2. Ziv's proof that entropy is the correct almost-sure upper bound is based on an interesting extension of the entropy idea to individual sequences, together with a proof that this individual sequence entropy is an asymptotic upper bound on (1/n)C(4) log n, for every sequence, together with a proof that, for ergodic processes, individual sequence entropy almost surely upper bounds the entropy given by the entropy theorem.

Ziv's concept of entropy for an individual sequence begins with a simpler idea, called topological entropy, which is the growth rate of the number of observed strings of length k, as k oc. The k-block universe of x is the set tik(x) of all a l' that appear as a block of consecutive symbols in x, that is,

= {a•ki• x

1' 1 = all for some i _>

The topological entropy of x is defined by

1 h(x) = lim — log 114 (x)I k a limit which exists, by subadditivity, Lemma 1.6.7, since I/44k (x)i < Vim (x)I • Pk (x)1. (The topological entropy of x, as defined here, is the same as the usual topological entropy of the orbit closure of x, see [84, 34] for discussions of this more general concept.) Topological entropy takes into account only the number of k-strings that occur in x, but gives no information about frequencies of occurrence. For example, if x is a typical sequence for the binary i.i.d. process with p equal to the probability of a 1 then every finite string occurs with positive probability, hence its topological entropy h(x) is log 2 = 1. The entropy given by the entropy theorem is h(p) = —p log p — (1 — p) log(1 — p), which depends strongly on the value of p, for it takes into account the frequency with which strings occur.

SECTION 11.2. THE LEMPEL-ZIV ALGORITHM.

133

The concept of Ziv-entropy is based on the observation that strings with too small frequency of occurrence can be eliminated by making a small limiting density of changes. The natural distance concept for this idea is cl(x, y), defined as before by

c-1(x , y) = lim sup d,i (xii , )1), where

d(4 , )7;1 ) =

1 ,t--n■

n

is per-letter Hamming distance. The Ziv-entropy of a sequence x is denoted by H(x) and is defined by

H(x) = lim inf h(y). Ziv established the LZ convergence theorem by proving the following two theorems, which also have independent interest. Theorem 11.2.2 (The LZ upper bound.)

lim sup

C(xti') log n

H(x), x

E A c°

.

Theorem 11.2.3 (The Ziv-entropy theorem.) If is an ergodic process with entropy h then H(x) < h, almost surely. These two theorems show that lim supn (l/n)C(x7) log n < h, almost surely, which, combined with the fact it is not possible to beat entropy in the limit, Theorem 11.1.2, leads immediately to the LZ convergence theorem, Theorem 11.2.1. The proof of the upper bound, Theorem 11.2.2, will be carried out via three lemmas. The first gives a simple bound (which is useful in its own right), the second gives topological entropy as an upper bound, and the third establishes a ii-perturbation bound. The first lemma obtains a crude upper bound by a simple worst-case analysis. Lemma 11.2.4 (The crude bound.) There exists Bn —O. 0 such that

C(x) log n n oo

Proof Define the word-length function by t(w) = j, for w E Ai. Fix S > 0 and suppose d(x, y) < S. Let x = w(1)w(2) ... be the LZ parsing of x and parse y as y = v(1)v(2) ..., where t(v(i)) = t(w(i)), i = 1, 2, .... The word w(i) will be called well-matched if

dt(u, ()) (w(i), v(i))
1- E, n—>oo

CHAPTER II. ENTROPY-RELATED PROPERTIES.

136

which can be expressed by saying that x is the concatenation

x = uw(1)w(2)..., where the initial block u has length s, each w(i) has length k, and

(4)

liminf n—,00

Ri < n: w(i) E TxII > 1 — E. — n

Fix a E A and let ak denote the sequence of length k, all of whose members are a. The sequence y is defined as the concatenation

y = uv(1)v(2)..., where

w(i) ak

if w(i) E 'Tk otherwise.

Condition (4) and the definition of y guarantee that ii(x, y) <E. If M is large relative to k, a member of the m-block universe Um (y) of y has the form qv(j)v(j + D.. • v(j + in — 1)r, where q and r have length at most k, and (M — 2k)/ k < m < M/k. Since each v(i) E U {a l } , a set of cardinality at most 1 + 2k(h +E) , the built-up set bound, Lemma 1.7.6, implies that h m (y) < h + 6 ± 3m, Tic

where 3A1 ---* 0, and hence h(y) < h + E. This completes the proof of Lemma 11.2.7, 0 and, thereby, the proof of the Ziv-entropy theorem, Theorem 11.2.3. Remark 11.2.8 Variations in the parsing rules lead to different forms of the LZ algorithm. One version, which has also been extensively analyzed, defines a word to be new only if it has been seen nowhere in the past (recall that in simple parsing, "new" meant "not seen as a word in the past.") In this alternative version, new words tend to be longer, which improves code performance, but more bookkeeping is required, for now the location of both the start of the initial old word, as well as its length must be encoded. Nevertheless, the rate of growth in the number of new words still determines limiting code length and the upper bounding theorem, Theorem 11.2.2, remains true, see Exercise 1. Remark 11.2.9 Another variation, also discussed in the exercises, includes the final symbol of the new word in the next word. This modification, known as the LZW algorithm, produces slightly more rapid convergence and allows some simplification in the design of practical coding algorithms. It is used in several data compression packages. Remark 11.2.10 For computer implementation of LZ-type algorithms some bound on memory growth is needed. One method is to use some form of the LZ algorithm until a fixed number, say Co, of words have been seen, after which the next word is defined to be the longest block that appears in the list of previously seen words. Another version, called the slidingwindow version, starts in much the same way, until it reaches the no-th term, then defines

137

SECTION 11.3. EMPIRICAL ENTROPY.

the next word as the longest block that appears somewhere in the past no terms. Such algorithms cannot achieve entropy for every ergodic process, but do perform well for processes which have constraints on memory growth, and can be shown to approach optimality as window size or tree storage approach infinity, see [87, 88].

II.2.a

Exercises

1. Show that the version of LZ discussed in Remark 11.2.8 achieves entropy in the limit. (Hint: if the successive words lengths are , t2., . • • , ic, then E log Li < C log(n/C).) 2. Show that the LZW version discussed in Remark 11.2.9 achieves entropy in the limit. 3. Show that simple LZ parsing defines a binary tree in which the number of words corresponds to the number of nodes in the tree.

Section 11.3 Empirical entropy. In many coding procedures a sample path 4 is partitioned into blocks, then each block is coded separately. The blocks may have fixed length, which can depend on n, or variable length. Furthermore, the coding of a block may depend only on the block, or it may depend on the past or even the entire sample path, e. g., through the empirical distribution of blocks in the sample path. In this section, a beautiful and deep result due to Ornstein and Weiss, [52], about the covering exponent of the empirical distribution of fixed-length blocks will be established. The theorem, here called the empirical-entropy theorem, suggests an entropy-estimation algorithm and a specific coding technique, both of which will also be discussed. Furthermore, the method of proof leads to useful techniques for analyzing variable-length codes, such as the Lempel-Ziv code, as well as other entropy-related ideas. These will be discussed in later sections. The empirical-entropy theorem has a nonoverlapping-block form which will be discussed here, and an overlapping-block form, see Exercise 1. The nonoverlapping k-block distribution, is defined by

qic(cli

=

Ri

E

[0,

rit

— 1]:

X: ;cc+ -E

=a

}

I

in

where n = km + r, 0 < r < k. The theorem is concerned with the number of k-blocks it takes to cover a large fraction of a sample path, asserting that eventually almost surely only a small exponential factor more than 2kh such k-blocks is enough, and that a small > 21th exponential factor less than 2kh is not enough, provided only that n _ .

CHAPTER II. ENTROPY-RELATED PROPERTIES.

138

Theorem 11.3.1 (The empirical-entropy theorem.) Let ii be an ergodic measure with entropy h > O. For each c > 0 and each k there is a set 'Tk (e) c A" for which l'Tk (e)I < 2k(h +E) , such that for almost every x there is a K = K(e, x) such that if k > K and n > 2kh , then (a) qk(Z(E)14)) > 1 — E. (b) qk(BI4)) <E, for any B c A" for which IBI < 2k( h — e) .

The theorem is, in essence, an empirical distribution form of entropy as covering exponent, Theorem 1.7.4, for the latter says that for given c and all k large enough, there < 2k(h+f) and pl (Tk(E)) > 1 — e, and there is no is a set T(E) c A k such that l'Tk(E)I _ set B C A k such that IB I _< 2k(h--E) and ,a(B) > E. The ergodic theorem implies that for fixed k and mixing kt, the empirical distribution qk(.14) is eventually almost surely close to the true distribution, ,uk, hence, in particular, for fixed k, the empirical and true distributions will eventually almost surely have approximately the same covering properties. The surprising aspect of the empirical-entropy theorem is its assertion that even if k is allowed to grow with n, subject only to the condition that k < (11 h) log n, the empirical measure qk(.I4) eventually almost surely has the same covering properties as ktk, in spite of the fact that it may not otherwise be close to iik in variational or even cik -distance. For example, for unbiased coin-tossing the measure qk (.14) is not close to kt k in variational distance for the case n = k2", see Exercise 3.

II.3.a

Strong-packing.

Part (a) of the empirical-entropy theorem is a consequence of a very general result about parsings of "typical" sample paths. Eventually almost surely, if the short words cover a small enough fraction, then most of the path is covered by words that come from fixed collections of exponential size almost determined by entropy. A fixed-length version of the result is all that is needed for part (a) of the entropy theorem, but the more general result about parsings into variable-length blocks will be developed here, as it will be useful in later sections. As before, a finite sequence w is called a word, word length is denoted by £(w) and a parsing of 4 is an ordered collection P = {w(1), w(2), ... , w(t)} of words for which 4 = w(1)w(2) • • • w(t). Fix a sequence of sets {E}, such that Tk c A k , for k > 1. A parsing X ni = w(l)w(2) • • • w(t) is (1 — e)-built-up from {'Tk}, or (1 — e)-packed by {TO, if the words that belong to U'rk cover at least a (1 — E)-fraction of n, that is,

E

f(w(i)) a. (1 — O n.

w(i)Eurk

A sequence 4 is (K, c)-strongly-packed by MI if any parsing .elz = w(1)w(2) • • • w(t) for which awci» < --En ,

E

t(w(I)) K, a set 'Tk = 'Tk(E) C A k , such that both of the following hold.

< 2k(h+E) for k > K. (a) ITkI_ (b) x is eventually almost surely (K, 0-strongly-packed by {7 } . Proof The idea of the proof is quite simple. The entropy theorem provides an integer m

and a set Cm c Am of measure close to 1 and cardinality at most 2m (h +6) • By the ergodic theorem, eventually almost surely most indices in x7 are starting places of m-blocks from Cm . But if such an x is partitioned into words then, by the Markov inequality, most of 4 must be covered by those words which themselves have the property that most of their indices are starting places of members of Cm . If a word is long enough, however, and most of its indices are starting places of members of Cm , then the word is mostly built-up from C m , by the packing lemma. The collection 'Tk of words of length k that are mostly built-up from Cm has cardinality only a small exponential factor more than 2"") , by the built-up set lemma. To make the outline into a precise proof, fix c > 0, and let 8 be a positive number to be specified later. The entropy theorem provides an m for which the set Cm = {c4n : p,(ar) > 2 -m(h+ 5)} has measure at least 1 — 3 2 /4. For k > m, let 'Tk be the set of sequences of length k that are (1 — 3)-built-up from Cm . By the built-up set lemma, Lemma 1.7.6, it can be supposed that 6 is small enough to guarantee that iTk i < 2k(n+E) , k > in. By making 8 smaller, if necessary, it can be supposed that 3 < 12. It remains to show that eventually almost surely x'IL is (K, 0-strongly-packed by {Tk for a suitable K. This is a consequence of the following three observations. (i) The ergodic theorem implies that for almost every x there is an N = N(x) such that for n > N, the sequence 4 has the property that X: +171-1 E Cm for at least (1 — 3 2 /2)n indices i E [1, n — m + 1], that is, 4 is (1 — 3 2 /2)-strongly-covered by Cm . (ii) If x is (1 —3 2 /2)-strongly-covered by C m and parsed as 4 = w(1)w(2) • • w(t), then, by the Markov inequality, the words w(i) that are not (1 — 3/2)-stronglycovered by Cm cannot have total length more than Sn < cn12. (iii) If w(i) is (1 — 3/2)-strongly-covered by C m , and if f(w(i)) ?_ 2/3, then, by the packing lemma, Lemma 1.3.3, w(i) is (1-0-packed by C m , that is, w(i) E Te(w(j)).

140

CHAPTER II. ENTROPY-RELATED PROPERTIES.

From the preceding it is enough to take K > 2/8, for if this is so, if n > N(x), and if the parsing x 11 = w(1)w(2) • • • w(t) satisfies

t(w(i)) K and w(i) E U Tk must have total length at El least (1 — On. This completes the proof of Lemma 11.3.2.

II.3.b

Proof of the empirical-entropy theorem.

Proof of part (a). Fix c > 0 and let S < 1 be a positive number to be specified later. The strongpacking lemma provides K and 'Tk c A " , such that Irk I .- 2k(h+') for k a K, and such that x;' is (K, 8)-strongly-packed by {Tk}, eventually almost surely. Fix x such that 4 is (K, 8)-strongly-packed by {T} for n a N(x), and choose K (x) > K such that if k> K(x) and n> 2kh , then n > N(x) and k < Sn. Fix k > K(x) and n a 2kh , and suppose xril = w(1)w(2) • • • w(t)w(t + 1), where t(w(i)) = k, i < t, and t(w(t + 1)) < k. All the blocks, except possibly the last one, are longer than K, while the last one has length less than Sn, since k < Sn.. The definition of strong-packing implies that

(n — k + 1)qk('Tk I4) a- (1 — 2 8 )n, since the left side is just the fraction of 4 that is covered by members of Tk . Thus, dividing by n — k + 1 produces

qk(Tkixriz ) a (1 — 28)(1 —8) > (1 — c), for suitable choice of S. This completes the proof of part (a) of the empirical-entropy 0 theorem. Proof of part (b). An informal description of the proof will be given first. Suppose xtiz is too-well covered by a too-small collection B of k-blocks, for some k. A simple two-part code can be constructed that takes advantage of the existence of such sets B. The first part is an encoding of some listing of B; this contributes asymptotically negligible length if I BI is exponentially smaller than 2kh and n > 2kh . The second part encodes successive kblocks by giving their index in the listing of B, if the block belongs to B, or by applying a fixed good k-block code if the block does not belong to B. If B is exponentially smaller than 2" then fewer than kh bits are needed to code each block in B, so that if B covers too much the code will beat entropy. The good k-code used on the blocks that are not in B comes from part (a) of the empirical-entropy theorem, which supplies for each k a set 'Tk c A k of cardinality roughly 2, such that eventually almost surely most of the k-blocks in an n-length sample path belong to Tk , provided only that n > 2kh . If a block belongs to Tk — B then its index in some ordering of Tk is transmitted; this requires roughly hk-bits. If the block does

141

SECTION 11.3. EMPIRICAL ENTROPY.

not belong to Tk U B then it is transmitted term by term using some fixed 1-block code; since such blocks cover at most a small fraction of the sample path this contributes little to overall code length. To proceed with the rigorous proof, fix c > 0 and let 8 be a positive number to be specified later. Part (a) of the theorem provides a set Tk c ilk of cardinality at most 2k(h+3) , for each k, and, for almost every x, an integer K(x) such that if k > K(x) and n > 2" then qk(TkIx) > 1 - 8. Let K be a positive integer to be specified later and for n > 2/(4 let B(n) be the set of 4 for which there is some k in the interval [K, (log n)/h] for which there is a set B c A k such that the following two properties hold. (i) qic(Tk14) ?_ 1 - 8. (ii) I BI E.

By using the suggested coding argument it will be shown that if 8 is small enough and K large enough then x ni ,_, B(n), eventually almost surely.

This fact implies part (b) of the empirical-entropy theorem. Indeed, the definition of {rk } implies that for almost every x there is an integer K (x) > K such that qk(Tk14) > 1 - 3, for K(x) < k
c, and 4 is parsed as 4 = w(l)w(2) • • • w(t)w(t + 1),

where w(i) has length k for i < t, and w(t + 1) has length r = [0, k). The code is defined as the concatenation

0

C(4) = 10k 1E(1B1)0k (B)v( 1)v(2) • • • v(t)v(t + 1), where

(2)

v(i) =

I

Mk.B(w(i)) OlG k (w(i)) 11 Fk(w(i)) 11Fr (w(t +1)

w(i) E B W(i) E Tk - B

w(i) g Tk UB,i 0 as K —> oc, and qk = clk(Blx'1 2 ), for the block length k and set B c itic used to define C(.4). The lemma is sufficient to complete the proof of part (b) of the empirical-entropy theorem. This is because tkq k (h — c) + tk(1 — q k )(h + 3) < n(h + 3 — 6(6 + S)),

since qk >

E,

for 4 E B(n), so that if K is large enough and 8 small enough then n(h — c 2 12), 4 E B(n),

for all sufficiently large n. But this implies that xriz V B(n), eventually almost surely, since no sequence of codes can beat entropy infinitely often on a set of positive measure, and as noted earlier, this, in turn, implies part (b) of the empirical-entropy theorem. 1=1

SECTION 11.3. EMPIRICAL ENTROPY.

143

Proof of Lemma 11.3.3. Ignoring for the moment the contribution of the headers used to describe k and B and to tell which code is being applied to each k-block, as well as the extra bits that might be needed to round k(h — E) and k(h +3) up to integers, the number of bits used to encode the blocks in Tk U B is given by tqk (Bix)k(h — E) + (t — tq k (BI4)) k(h +3), since there are tqk(Bl.q) blocks that belong to B, each of which requires k(h — E) bits, and (t — tqk(B 14)) blocks in Tk — B, each of which requires k(h +3) bits. This gives the dominant terms in (3). The header 10k le(I BD that describes k and the size of B has length (4)

2 +k+logIBI o(logIBI)

which is certainly o(n), since log I BI < k(h — E) and k < (log n)I n. It takes at most k(1 + log I Ai) bits to encode each w(i) Tk U B as well as to encode the final r-block if r > 0, and there are at most 1 +3t such blocks, hence at most 80(n) bits are needed to encode them. Adding this to the o(n) bound of (4) then yields the 30(n) term of the lemma. The encoding Øk(B) of B takes k Flog IAMBI < k(1+ log I A1)2 k(h- E ) bits, which is, in turn, upper bounded by K (1+ loglA1)2. -KEn, (5) provided K > (E ln 2) - ', since k > K and n > 2". There are at most t + 1 blocks, each requiring a two-bit header to tell which code is being used, as well as a possible extra bit to round up k(h — E) and k(h +8) to integers; together these contribute at most 3(t + 1) bits to total length, a quantity which is at most 6n/K, since t < nlk, and n> k > K. Thus with 6 ax = K(1 + log IA1)2 -KE — = o(K), K the encoding of 4k(B) and the headers contributes at most naK to total length. This completes completes the proof of Lemma 11.3.3, thereby establishing part (b) of the empirical-entropy theorem. El

Remark 11.3.4 The original proof, [ 52], of part (b) of the empirical-entropy theorem used a counting argument to show that the cardinality of the set B(n) must eventually be smaller than 2nh by a fixed exponential factor. The argument parallels the one used to prove the entropy theorem and depends on a count of the number of ways to select the bad set B. The coding proof given here is simpler and fits in nicely with the general idea that coding cannot beat entropy in the limit.

II.3.c Entropy estimation. A problem of interest is the entropy-estimation problem. Given a sample path Xl, X2, , xn from an unknown ergodic process it, the goal is to estimate the entropy h of II. A simple procedure is to determine the empirical distribution of nonoverlapping k-blocks, and then take H (qk )/ k as an estimate of h. If k is fixed and n oc, then H(qk)1 k will converge almost surely to H(k)/k, which, in turn, converges to h as

144

CHAPTER II. ENTROPY-RELATED PROPERTIES.

k —> oc, at least if it is assumed that it is totally ergodic. Thus, at least for a totally ergodic ,u, there is some choice of k = k(n) as a function of n for which the estimate will converge almost surely to h. At first glance, the choice of k(n) would appear to be very dependent on the measure bt, because, for example, there is no universal choice of k = k(n) for which k(n) —> oc for which the empirical distribution qi,(.14) is close to the true distribution bak for every mixing process A. The empirical entropy theorem does imply a universal choice for the entropy-estimation problem, for example, the choice k(n) — log n works for any binary

ergodic process. The general result may be stated as follows. Theorem 11.3.5 (The entropy-estimation theorem.) If it is an ergodic measure of entropy h > 0, if k(n) —> oc, as n -- oc, and if k(n) < (1/h)log n, then 1 lim — H(q00(.14)) = h, a.s.

(6)

n->C0 k(n)

In particular, if k(n) -- log iAl n then (6) holds for any ergodic measure p, with alphabet A, while if k(n) — log log n, then it holds for any finite-alphabet ergodic process. Proof Fix c > 0 and let {Tk(c) c A k : n > 1) satisfy part (a) of the empirical-entropy theorem. Let Uk be the set of all cif E Tk(E) for which qk (ali` 'xi' ) < 2 -k(h+2E) , so that for all large enough k and all n > 2kh , (7)

qk(Uklxi) _< Irk(E)12-

ch+2E) < 2k(h+E)2 -k(h+2E) _ "-Ek —z.

Vk be the set of all alic for which qk (cif 14) > 2 -k(h-26) , and note that 2/(/1-26) . Part (b) of the empirical-entropy theorem implies that for almost every

Next let I Vkl
K (x), n _>

This bound, combined with the bound (7) and part (a) of the empirical-entropy theorem, implies that for almost every x, there is a Ki (x) such that

qk(GkIx) > 1 — 36, for k > Ki(x), n > 2kh , where G k = Tk(E) — Uk — Vk. In summary, the set of al' for which ,(k1n — iikkai 'x i

) < 2-0h-2E)

has qk (. lx) measure at least 1 — 3e. The same argument that was used to show that entropy-rate is the same as decay-rate entropy, Theorem 1.6.9, can now be applied to complete the proof of Theorem 11.3.5. CI Remark 11.3.6 The entropy-estimation theorem is also true with the overlapping block distribution in place of the nonoverlapping block distribution, see Exercise 2.

SECTION 11.3. EMPIRICAL ENTROPY

II.3.d

145

Universal coding.

A universal code for the class of ergodic processes is a faithful code sequence {C n } such that for any ergodic process p.

lirn

“.42 )

= h, almost surely,

71

where h is the entropy of A. Such a code sequence was constructed in Section II. La, see the proof of Theorem 11.1.1, and it was also shown that the Lempel-Ziv algorithm gives a universal code. The empirical-entropy theorem provides another way to construct a universal code. The steps of the code are as follows.

Step 1. Partition xi: into blocks of length k = k(n) — (1/2) log AI n. Step 2. Transmit a list rence in 4.

of these k-blocks in order of decreasing frequency of occur-

Step 3. Encode successive k-blocks in 4 by giving the index of the block in the code book E.

Step 4. Encode the final block, if k does not divide n, with a per-symbol code. The number of bits needed to transmit the list ,C is short, relative to n, since k -(1/2) log ik n. High frequency blocks appear near the front of the code book and therefore have small indices, so that the empirical-entropy theorem guarantees good performance, since the number of bits needed to transmit an index is of the order of magnitude of the logarithm of the index. To make this sketch into a rigorous construction, fix {k(n)} for which k(n) — (1/2) log n. (For simplicity it is assumed that the alphabet is binary.) The code word Cn (4) is a concatenation of two binary sequences, a header of fixed length m = k2k , where k = k(n), followed by a sequence whose length depends on x. The header br is a concatenation of all the members of {0, 1} k , subject only to the rule that af precedes 5k whenever k n -k n) qk(ailX1) > qk( all X 1 •

The sequence {v(j) = j = 0, 1, , 2k — 1) is called the code book. Suppose n = tk r, r E [0, k) and 4 = w(1)w(2)•• • w(t)v, where each w(i) has length k. For 1 < i < t and 0 < j < 2k, define the address function by the rule

A(w(i)) = j, if w(i) = that is, w(i) is the j-th word in the code book. Let E be a prefix code on the natural numbers such that the length of the word E(j) is log j o(log j), that is, an Elias code. Define b2++Li to be the concatenation of the E(A(w(i))) in order of increasing i, where L is the sum of the lengths of the e(A(w(i))). Finally, define b:+1 +-1 to be x4+1 • The code Cn (4) is defined as the concatenation

Cn (x7) = br • b:V' • b7,JF- iiri I . The code length £(4) = ni+ L +r, depends on 4, since L depends on the distribution qk (.lxiii), but the use of the Elias code insures that C, is a prefix code.

146

CHAPTER II. ENTROPY-RELATED PROPERTIES.

The empirical-entropy theorem will be used to show that for any ergodic tt,

lim

(8)

n-*

f(x n ) 1 = h, a. s., n

where h is the entropy of it. To establish this, let {7-k(E) c Ac} be the sequence given by the empirical-entropy theorem. For each n, let Gk denote the first 2k(h +E) members of the code book, where k = k(n). Note that IGk I > l'Tk(E)I and, furthermore, qk(GkIx t0

qk(Tk(E)Ixi),

since Gk is a set of k-sequences of largest qke probability whose cardinality is at most 2k(h +E ) . The empirical-entropy theorem provides, for almost every x, an integer K(x) such that qk (GkI4) > qk (7k (c)14) > 1 — E for k > K(x). Thus, for k > K(x), at least a (1 — E)-fraction of the binary addresses e(A(w(i))) will refer to members of Gk and thus have lengths bounded above by k(h + c)+ o(log n). For those w(i) Gk, the crude bound k + o(log n) will do, and hence

< (1 — On(h + c) + En + o(log n), for k > K (x). This proves that

,C(4) lim sup — < h, a.

S.

The reverse inequality follows immediately from the fact that too-good codes do not exist, Theorem 11.1.2, but it is instructive to note that it follows from the second part of the empirical-entropy theorem. In fact, let Bk be the first 2k(h-E) sequences in the code book for x. The empirical-entropy theorem guarantees that eventually almost surely, there are at most Et indices i for which w(i) E Bk. These are the only k-blocks that have addresses shorter than k(h — 6) + o(log n), so that

lirninf' > (h — c)(1 — c), a. s.

Remark 11.3.7 The universal code construction is drawn from [49], which also includes universal coding results for coding in which some distortion is allowed. A second, and in some ways, even simpler algorithm which does not require that the code book be listed in any specific order was obtained in [43], and is described in Exercise 4. Another application of the empirical-entropy theorem will be given in the next chapter, in the context of the problem of estimating the measure II from observation of a finite sample path, see Section 111.3.

II.3.e Exercises 1. Show that the empirical-entropy theorem is true with the overlapping block distribution, Pk(a in4)

E [1, n — k + 1]: 444-1 = aDi

n—k+ 1 in place of the nonoverlapping block distribution qk(.lfiz). (Hint: reduce to the nonoverlapping case for some small shift of the sequence.)

SECTION 11.4. PARTITIONS OF SAMPLE PATHS.

147

2. Show that the entropy-estimation theorem is true with the overlapping block distribution in place of the nonoverlapping block distribution. 3. Show that the variational distance between qk (.14) and ,u k is asymptotically almost surely lower bounded by (1 — e -i ) for the case when n = k2k and ,u is unbiased coin-tossing. (Hint: if M balls are thrown at random into M boxes, then the expected fraction of empty boxes is asymptotic to (1 — 4. Another simple universal coding procedure is suggested by the empirical-entropy theorem. For each k < n construct a code Ck m as follows. Express x'12 as the concatenation w(l)w(2) • • • w(q)v, of k-blocks {w(i)}, plus a possible final block y of length less than k. Make a list in some order of the k-blocks that occur. Transmit k and the list, then code successive w(i) by using a fixed-length code; such a code requires at most k(1 + log IA I) bits per word. Append some coding of the final block y. Call this C k m (Xil ) and let £k(X) denote the length of Ck ,,,(x;'). The final code C(xil) transmits the shortest of the codes {Ck,,,(4)}, along with a header to specify the value of k. In other words, let km,,, be the first value of k E [ 1, n] at which rk(x'11 ) achieves its minimum. The code C(4) is the concatenation of e(kiii,n ) and Cknun ,n (x). Show that {C,,} is a universal prefix-code sequence.

Section 11.4 Partitions of sample paths. An interesting connection between entropy and partitions of sample paths into variable-length blocks was established by Ornstein and Weiss, [53]. They show that eventually almost surely, for any partition into distinct words, most of the sample path is covered by words that are not much shorter than (log n)I h, and for any partition into words that have been seen in the past, most of the sample path is covered by words that are not much longer than (log n)I h. Their results were motivated by an attempt to better understand the Lempel-Ziv algorithm, which partitions into distinct words, except possibly for the final word, such that all but the last symbol of each word has been seen before. As in earlier discussions, a word w is a finite sequence of symbols drawn from the alphabet A and the length of w is denoted by i(w). A sequence Jell is said to be parsed (or partitioned) into the (ordered) set of words {w(1), w(2), , w(t)} if it is the concatenation (1) 4 = w(1)w(2)... w(t). If w(i) w(j), for i words. For example,

j, then (1) is called a parsing (or partition) into distinct

000110110100 = [000] [110] [1101] [00] is a partition into distinct words, while 000110110100 = [00] [0110] [1101] [00] is not. Partitions into distinct words have the asymptotic property that most of the sample path must be contained in those words that are not too short relative to entropy.

148

CHAPTER II. ENTROPY-RELATED PROPERTIES.

Theorem 11.4.1 (The distinct-words theorem.) Let be ergodic with entropy h > 0, and let E > 0 be given. For almost every w(t) is a x E A" there is an N = N(E, x) such that if n > N and xri' = w(1)w(2) partition into distinct words, then t ( o )) < En. ow(0) 1. Et _ The general case follows since it is enough to estimate the sum over the too-short words that belong to G. This proves the lemma.

To continue with the proof of the distinct-words theorem, let 8 be a positive number to be specified later. The strong-packing lemma, Lemma 11.3.2, yields an integer K and { rk C A": k> 1} such that I7j < 21c(h+S) , k > K,

CHAPTER II. ENTROPY-RELATED PROPERTIES.

150

and such that eventually almost surely xrii is (K, 3)-strongly-packed by MI, i. e.,

E

t (o ))

a ( l-3)n,

w(i)Euciz k rk

for any parsing P of xriz for which

E

(4)

Sn atv(i)) — . 2

Suppose xr,' is (K, 3)-strongly-packed by {'Tk }. Suppose also that n is so large that, by Lemma 11.4.3, property (4) holds for any parsing of .x. rit into distinct words. Thus, w(t) into distinct words, strong-packing implies that given a parsing x = w(l)w(2)

t (o)) a _3)n,

E w(i)EU

Klj

and therefore Lemma 11.4.4 can be applied with a = h + 3 to yield aw(i)) < 25n, f(w(i))(1-15)(logn)/(h+S)

provided only that n is large enough. The distinct-words theorem follows, since 8 could have been chosen in advance to be so small that log n (1 — 2S) log n h + c 5h+3 •

I1.4.b

1:3

Proof of the repeated-words theorem.

It will be shown that if the repeated-words theorem is false, then a prefix-code sequence can be constructed which beats entropy by a fixed amount infinitely often on a set of positive measure. The idea is to code each too-long word of a repeated-word parsing by telling where it occurs in the past and how long it is (this is, of course, the basic idea of the version of the Lempel-Ziv code suggested in Remark 11.2.8.) If a good code is used on the complement of the too-long words and if the too-long words cover too much, then overall code length will be shorter than entropy allows. As in the proof of the empirical-entropy theorem, the existence of good codes is guaranteed by the strong-packing lemma, this time applied to variable-length parsing. Throughout this discussion p, will be a fixed ergodic process with positive entropy h and E < h will be a given positive number. A block of consecutive symbols in xiit of length more (h — E) -1 log n will be said to be too long. The first idea is to merge the words that are between the too-long words. Suppose = w(l)w(2) w(t) is some given parsing, for which s of the w(i) are too long. Label these too-long words in increasing order of appearance as

V(1), V(2),

V(s).

Let u1 be the concatenation of all the words that precede V(1), for 1 < i < s + 1, let ui be the concatenation of all the words that come between V(i — 1) and V(i), and let

SECTION 11.4. PARTITIONS OF SAMPLE PATHS.

151

u s+ 1 be the concatenation of all the words that follow V(s). In this way, x'iz is expressed as the concatenation x Çz = u i V(1)u2V(2) us V(s)u s±i .

Such a representation is called a too-long representation of 4, with the too-long words {V(1), ..., V(s)} and fillers lui, Let B(n) be the set of all sequences x for which there is an s and a too-long representation x = uiV (1)u2 V(2) ... us V(s)u s+i with the following two properties.

(i) Ei f ( V (D)

>En.

(ii) Each V(j) has been seen in the past. To say V(j) = xnn i +± imi has been seen in the past is to say that there is an index i E [0, ni ) Since the start set .T is a fixed finite set, it can be supposed such that V(j) that when n is large enough no too-long word belongs to F, and therefore to prove the repeated-words theorem, it is enough to prove that x g B(n),

eventually almost surely.

The idea is to code sequences in B(n) by telling where the too-long words occurred earlier, but to make such a code compress too much a good way to compress the fillers is needed. The strong-packing lemma provides the good codes to be used on the fillers. Let 3 be a positive number to be specified later. An application of the strong-packing lemma, Lemma 11.3.2, provides an integer K and for a each k > K, a set 'Tk c A k of cardinality at most 2k(h+8) , such that eventually almost surely xril is (K, 3)-stronglypacked by {'Tk }. Let G(n) be the set of all xri' that are (K, 8)-strongly-packed by {Tk } . Since .q E G(n) eventually almost surely, to complete the proof of the repeated-words theorem it is enough to prove that if K is large enough, then B(n) n G(n), eventually almost surely. A code C, is constructed as follows. Sequences not in B(n) n G(n) are coded using some fixed single-letter code on each letter separately. If x E B(n) n G(n), a too-long representation = u1V(1)u2V(2)...u s V(s)u s+i, is determined for which each too-long V(j) is seen somewhere in its past and the total length of the {V(j)} is at least En. The words in the too-long representation are coded sequentially using the following rules. (a) Each filler u 1 E U K Tk is coded by specifying its length and giving its index in the set Tk to which it belongs. (b) Each filler u1 g U K rk is coded by specifying its length and applying a fixed single-letter code to each letter separately. (c) Each too-long V(j) is coded by specifying its length and the start position of its earlier occurrence.

152

CHAPTER II. ENTROPY-RELATED PROPERTIES.

An Elias code is used to encode the block lengths, that is, a prefix code S on the natural numbers such that the length of the code word assign to j is log j + o(log j). Two bit headers are appended to each block to specify which of the three types of code is being used. With a one bit header to tell whether or not xri` belongs to B(n) n G(n), the code becomes a prefix code. For 4 E B(n) n G(n), the principal contribution to total code length E(xriz) .C(fi` IC n ) comes from telling where each V(j) occurs in the past, which requires Flog ni bits for each of the s too-long words, and from specifying the index of each u i E UlciK rk, which requires £(u) [h + 81 bits per word. This fact is stated in the form needed as the following lemma.

Lemma 11.4.5 If log K < 6 K , then .C(x'11 ) = s log n + (

(5)

E au ., ) (h + 6) + 60(n),

ti J ELJA7=Kii uniformly for 4

E

B(n) n G(n).

The lemma is sufficient to show that xrii B(n) n G(n), eventually almost surely. The key to this, as well as to the lemma itself, is that the number s of too-long words, while it depends on x, must satisfy the bound,

av(i))

s

(6)

—

log n

(h — e),

since, by definition, a word is too long if its length is at least (h — E) -1 log n. By assumption, tw(i) > en, so that (5) and (6) yield the code-length bound

< n(h + — e(e + 6)) + 0(n), for

X Fil E

B(n) n G(n), and hence if 6 is small enough then

£(4) < n(h — € 2 12), xÇ2 E B(n) n G(n), for all sufficiently large n, which, since it is not possible to beat entropy by a fixed amount infinitely often on a set of positive measure, shows that, indeed, "ell B(n) n G(n), eventually almost surely, completing the proof of the repeated-words theorem.

Proof of Lemma 11.4.5. It is enough to show that the encoding of the fillers that do not belong to Urk , as well as the encoding of the lengths of all the words, plus the two bit headers needed to tell which code is being used and the extra bits that might be needed to round up log n and h + 6 to integers, require a total of at most 60(n) bits. For the fillers that do not belong to U71, first note that there are at most s + 1 fillers so the bound Es. n s < i="' (h — e) < — (h — 6), — log n — log n implies that

E au ., ) (log n)/ (h — 6), is upper bounded by exp((h — c) Et; + o(n)). This is equivalent to showing that the coding of the too-long repeated blocks by telling where they occurred earlier and their lengths requires at most (h — c) E ti + o(n) bits. The coding proof is given here as fits in nicely with the general idea that coding cannot beat entropy in the limit.

II.4.c

Exercises

1. Extend the repeated-words theorem to the case when it is only required that all but the last k symbols appeared earlier. 2. Extend the distinct-words and repeated-words theorems to the case where agreement in all but a fixed number of places is required.

CHAPTER II. ENTROPY-RELATED PROPERTIES.

154

3. Carry out the details of the proof that (2) follows from the distinct-words theorem. 4. Carry out the details of the proof that (3) follows from the repeated-words theorem. 5. What can be said about distinct and repeated words in the entropy 0 case? 6. Let ,u, be the Kolmogorov measure of the binary, equiprobable, i.i.d. process (i. e., unbiased coin-tossing.) (a) Show that eventually almost surely, no parsing of 4 into repeated words contains a word longer than 4 log n. (Hint: use Barron's lemma, Lemma 11.1.3.) (b) Show that eventually almost surely, there are fewer than n 1-612 words longer than (1 + c) log n in a parsing of 4 into repeated words. (c) Show that eventually almost surely, the words longer than (1 + c) log n in a parsing of 4 into repeated words have total length at most n 1- E12 log n.

Section 11.5

Entropy and recurrence times.

An interesting connection between entropy and recurrence times for ergodic processes was discovered by Wyner and Ziv, [86], see also the earlier work of Willems, [85]. Wyner and Ziv showed that the logarithm of the waiting time until the first n terms of a sequence x occurs again in x is asymptotic to nh, in probability. This is a sharpening of the fact that the average recurrence time is 1/,u(x'11 ), whose logarithm is asymptotic to nh, almost surely. An almost-sure form of the Wyner-Ziv result was established by Ornstein and Weiss, [53]. These results, along with an application to a prefix-tree problem will be discussed in this section. The definition of the recurrence-time function is Rn (x) = min{rn > 1: x m+ m+n1 — X n } — .

i

.

The theorem to be proved is Theorem 11.5.1 (The recurrence-time theorem.) For any ergodic process p, with entropy h, lim 1 log Rn (x) = h, almost surely. n

n--÷oo

Some preliminary results are easy to establish. Define the upper and lower limits, 1 = lim sup - log R„(x), n 1 r(x) = lim inf - log Rn (x). n—woo n T(x)

Since Rn _1(T x) < R n (x), both the upper and lower limits are subinvariant, that is, F(Tx)