Abstract Methods in Information Theory

ABSTRACT METHODS IN INFORMATION THEORY Yftichiro Kakihara Department ofMathematics University of California, Riverside...

Author: Yuichiro Kakihara

69 downloads 1035 Views 15MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

ABSTRACT METHODS IN

INFORMATION THEORY

Yftichiro Kakihara Department ofMathematics University of California, Riverside

USA

b World Scientific

III

'

Singapore· New Jersey· London- Hong Kong

Published by

World Scientific Publishing Co. Pte. Ltd. POBox 128, Farrer Road, Singapore 912805 USA office: Suite IB, 1060 Main Street, River Edge, NJ 07661 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE

Library of Congress Cataloging-in-Publication Data Kakihara, Yiiichiro. Abstract methods in information theory I Yfiichirf Kakihara. p. em. -- (Series on multivariate analysis ; v. 4) Includes bibliographical references. ISBN 9810237111 (alk. paper) 1. Information theory. 2. Functional analysis. I. Title. II. Series. Q360.K35 1999 003'.54--dc21 99-31711 CIP

British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library.

Copyright © 1999 by World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher.

For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher.

Printed in Singapore by Uto-Print.

PREFACE

Half a century has passed since C. E. Sharman published his epoch-making paper entitled "A mathematical theory of communication" in 1948. Thereafter the socalled "information theory" began to grow and has now established a firm and broad field of study. Viewing from a mathematical angle, information theory might be thought of having the following four parts: (1) the mathematical structure of inforrnation sources, (2) the theory of entropy as amounts of information" (3) the theory of inforrnation channels, and (4) the theory of coding. Probabilistic and algebraic rnethods have mainly been used to develop information theory. Since the early stage of the expansion of information theory, however, measure theoretic and functional analysis methods have also been applied and are providing a powerful tool to obtain rigorous results in this theory. The purpose of this book is to present the first three parts of information theory, rnentioned above, in the environment of functional analysis, in addition to probability theory. Here are a couple of examples in each of which functional analysis played a crucial role obtaining important results in information theory. The coincidence of the ergodic capacity C; and the stationary capacity C, for a certain channel was one of the rnost irnportant problems in the late 1950s. L. Breiman (1960) showed that for a finite rnemory channel the equality C; = C; holds and, moreover, C; is attained by some ergodic input source (= measure) invoking Krein-Milman's theorem to the weak* compact convex set Ps(X) of all stationary input sources. Another such example appeared in a characterization of ergodic channels. In the late 1960s H. Umegaki and Y. Nakamura independently proved that a stationary channel is ergodic if and only if it is an extreme point of the convex set of all stationary channels. Umegaki observed a one-to-one correspondence between the set of channels and a set of certain averaging operators from the set of bounded measurable functions on the compound space to the set of those functions on the input. Then a channel is identified with an operator, called a channel operator, and hence we can make a full use of functional analysis in studying channels. In this book, readers will find how functional analysis helps to describe information theory, especially the mathernatical structure of information souces and channels, in an effective way. Here is a brief summary of this book. In Chapter I, entropy is considered as the amount of information, Shannon's entropy for finite schema is defined and its basic properties are exarnined together with its axioms. After collecting fundarnental propvii

viii

Preface

erties of conditional expectation and probability, Kolmogorov-Sinai's entropy is then obtained for a measure preserving transformation. Some fundamental properties of the Kolmogorov-Sinai entropy are presented along with the Kolmogorov-Sinai theorem. Algebraic models are introduced to describe probability measures and measure preserving transformations. Sorne conjugacy problems are studied using algebraic models. When we fix a measurable transformation and a finite partition, we can consider Kolmogorov-Sinai's entropy as a functional on the set of invariant (with respect to the transformation) probability measures, called an entropy functional. This functional is extended to be the one defined on the set of all complex valued invariant measures, and its integral representation is obtained. Relative entropy and Kullback-Leibler information are also studied in connection with sufficiency which is one of the most important notions in statistics, and with hypothesis testing. In Chapter II, information sources are considered. Using an alphabet message space as a rnodel, we describe information sources on a compact Hausdorff space. Mean and Pointwise Ergodic Theorems are stated and proved. Ergodicity is one of the important concepts and its characterization is presented in detail. Also strong and weak rnixing properties are examined in some detail. Among the nonstationary sources, AMS (== asyrnptotically mean stationary) sources are of interest and the structure of this class is studied. Shannon-McMillan-Breiman Theorem is then formulated for a stationary and an AMS source, which is regarded as the ergodic theorem in information theory. Ergodic decomposition of a stationary source is established and is applied to obtain another type of integral representation of an entropy functional. Chapter III, the main part of this book, is devoted to the information channels. After defining channels, a one-to-one correspondence between a set of channels and a set of certain averaging operators is established, as rnentioned before. Strongly and weakly rnixing channels are defined as a generalization of finite dependent channels and their basic properties are obtained. Ergodicity of stationary channels is discussed and various necessary and sufficient conditions for it are given. For AMS channels, absolute continuity plays a special role in characterizing ergodicity. Capacity and transmission rate are defined for stationary channels. Coincidence of ergodic and stationary capacities is proved under certain conditions. Finally, Shannon's coding theorems are stated and proved. Special topics on channels are considered in Chapter IV. When a channel has a noise source, some properties of such a channel are studied. If we regard a channel to be a vector (or measure) valued function on the input space, then its measurabilities are clarified. Some approximation problems of channels are treated. When the output space is a (locally) compact abelian group, a harmonic analysis method can be applied to channel theory. Some aspects of this viewpoint are presented in detail. Finally, a noncorumutative channel theory is introduced.. We use a C*-algebra approach to formulate channel operators as well as other aspects of noncommutative

Preface

ix

extension. Another purpose of this book is to present contributions of Professor Hisaharu Umegaki and his school on information theory. His selected papers is published under the title of "Operator Algebras and Mathematical Information Theory," Kaigai, Tokyo in 1985. As one of his students, the author is pleased to have a chance to write this rnonograph. In the text 111.4.5 denotes the fifth item in Section 4 of Chapter III. In a given chapter, only the section and item number are used, and in a given section, only the item number is used. The author is grateful to Professor M. M. Rao at University of California, Riverside (UCR) for reading the manuscript and for the valuable suggestions. UCR has provided the author with a very fine environment, where he could prepare this monograph. He is also grateful for the hospitality of UCR.

Yiiichiri) Kakihara Riverside, California April, 1999

CONTENTS

vii

Preface Chapter I. Entropy

1

1.1. The Shannon entropy 1.2. Conditional expectations 1.3. The Kolmogorov-Sinai entropy 1.4. Algebraic models 1.5. Entropy functionals 1.6. Relative entropy and Kullback-Leibler information Bibliographical notes

Chapter II. Information Sources

1 11 16 29 41 49 63

67

2.1. Alphabet message spaces and information sources. . . . . . . . . . . . . . . . . . . .. 2.2. Ergodic theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 2.3. Ergodic and mixing properties 2.4. AMS sources 2.5. Shannon-McMillan-Breiman theorem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 2.6. Ergodic decompositions 2.7. Entropy functionals, revisited Bibliographical notes

67 71 75 91 99 106 110 119

Chapter III. Information Channels

121

3.1. Inforrnation channels 3.2. Channel operators 3.3. Mixing channels 3.4. Ergodic channels 3.5. AMS channels 3.6. Capacity and transmission rate 3.7. Coding theorems Bibliographical notes

121 129 136 147 156 166 178 187

Chapter IV. Special Topics

189

4.1. Channels with a noise source

189 xi

xii

Contents

4.2. Measurability of a channel 4.3. Approximation of channels 4.4. Harmonic analysis for channels 4.5. ~oncouauautative channels Bibliographical notes

196 202 207 214 222

References

225

Indices Notation index Author index Subject index

239 239 244 247

CHAPTER I

ENTROPY

In this chapter, basic ideas of entropy are presented from the works of Shannon and Kolmogorov-Sinai. The first one is defined for finite schema and the second is for measure preserving transformations. Conjugacy between two measure preserving transformations is considered in terms of their algebraic models. When a transformation is fixed, the entropy is defined for all transformation invariant probability measures. In this case, it is called an entropy functional. An integral representation for this functional is given. Relative entropy and Kullback-Leibler information are studied in connection with sufficient statistics and hypothesis testing.

1.1. The Shannon entropy We consider basic properties and axioms of Shannon's entropy. Let n E N (the set of all positive integers) and X = {Xl, ... ,xn } be a finite set with a probability distribution P

= (PI, ... ,Pn), i.e., Pj = p(Xj)

n

~

L Pj = 1, where

0,1 :::; j :::; nand

j=l

p(.) denotes the probability. We usually denote this as (X,p) and call or the Shannon

it2:.J~~~::!.~

n

H(X) = - LPj logpj,

(1.1)

j=l

where "log" is the natural logarithm and we regard 0 log 0 = 0 log = o. We also say that H(X) is Justification of these terminologies will be clarified later in this section. Since the RHS (= right hand side) of (1.1) depends only on the probability distribution P = (PI, ... ,Pn) we may also write n

H(X) = H(p) = H(PI, ... ,Pn) = - LPj log pj , j=l

1

Chapter I: Entropy

2

We need sorne notations. For n E N, ~n denotes the set of all probability distributions P == (PI, ... ,Pn), i.e.,

Let Y == {Yl, ... ,Yrn} be another finite set. The probability of (Xj, Yk) and the conditional probability of Xj given Yk are respectively denoted by p(Xj, Yk) and

p~jl~)==P~) Up(~»O. Then~~~~~~~~~~~:Q_L~L~~g~~ is defined by

H(XIY) ==

L

L

p(y)p(xly) logp(xly)·

(1.2)

yEYxEX

If we define H(Xly),

H(Xly) == - L

p(xIY) logp(xIY),

xEX

then (1.2) is interpreted as the average of these conditional entropies over Y'. The qua~tlyl(X,Y)definedbelow~called=~=~~~~~~~_~~ __~__ ~~~_~_~~

tcx, Y)

==

H(X)

H(XIY)

since we can easily verify that

It X, Y) == [f(Y) - H(YIX) == H(X) + H(Y) - H(X, Y) ~ p(x,y) = L."p(x, y) log (x) ( ) ~ 0, P

x,y

(1.3)

PY

where

H(X, Y)

- LP(x, y) logp(x, y)

is ~~e e!ltr.QPY ()f~ll.~~gc,~.~~!!!E~~"~~~~~~~!!!~{(~'E),.{Y~9)) (cf. Theorern 1 below). If we consider two probability distributions P, q E ~n of X, of w.r.t. with is by n

n

j=1

j=1

H(plq) == LPj(1ogPj -logqj) == LPj log p~. ~

If Pi > 0 and qj == 0 for some j, then we define H(plq) == 00. Observe the difference between H(XjY) and H(plq). Relative entropy will be discussed in detail in a later section.

1.1. The Shannon entropy

3

The next two theorerns give basic properties of entropies. Theorem 1. Consider entropies on ~n' (1) H(plq) ~ 0 for P, q E ~n, and H(plq) == 0 iff if and only if) p == q. (2) Let p E ~n and A == (ajk)j,k be an n by n doubly stochastic matrix, i.e., n

ajk ~ 0,

n

2:: ajk == 2:: ajk ==

1 for 1 ::; j, k ::; n. Then, q == Ap E ~n and j=l k=l H (p). The equality holds iff qk == P1r(k) , 1 ::; k ::; n for sorne permutation 1f

H (q) ~ of{I, ... ,n}. (3) H(X, Y) == H(X) + H(YIX) ==H(Y) + H(XIY). (4) H(X, Y) < H(X) + H(Y). The equality holds iff X and Yare independent. (5) H(XIY) < H(X). The equality holds iff X and Yare independent. Proof. (1) Assume H(plq)
0," we get

for j == 1, ... ,n. Hence n

n

H(plq) == LPj log P~ ~ L(Pj - qj)== O. j=l qJ j=l The statement about the equality follows from the fact that t log t == t - 1 iff t == 1. (2) q == Ap E ~n is clear. Since the function ¢(t) == -t log t is concave (== concave downward) for t > 0, we have H(q)

n

n (n

= ~¢(qj) = ~¢ (;a jkPk n

n

)

n

~ LLajk¢(Pk) == L¢(Pk) == H(p). j=lk=l

¢( k=l t ajkPk)

k=l

=

t

¢(Pk) for 1 :::; j :::; n iff for each j = k=l 1, ... ,n, ajk == 1 for some k and == 0 otherwise iff qk == P1r(k) , 1 ::; k ::; n for some permutation 1f of {I, ... ,n}. (3) Observe the following computations: The equality holds iff

H(X, Y) == - LP(x, y) logp(x, y) x,y

Chapter I: Entropy

4

= - LP(x, y) logp(x)p(Ylx) x,Y

= - LP(x, y) logp(x) - LP(x, y) logp(ylx) x,Y

= H(X)

x,Y

+ H(YIX),

giving the first equality and, similarly, H(X, Y) = H(Y) (4) is derived as follows:

H(X)

+ H(Y) = -

+ H(XIY).

LP(x) logp(x) - LP(y) logp(y) x

y

= LP(x, y) logp(x)p(y) x,y.

~ - LP(x, y) logp(x, y),

by (1),

x,y

= H(X,Y). By (1) the equality holds iff p(x, y) are independent. (5) is clear from (3) and (4). Let lR = (-00,00), lR+ Theorem 2. Let p

= p(x)p(y) for x

E

X and y E Y, i.e., X and Y

= [0,00) and lR+ = [0,00].

= (Pj), q = (qj)

E

U ~n.

n=2

(1) (Positivity)H (p) ~ O. (2) (Continuity) H: U ~n -+ lR+ is continuous. n2::2 (3) (Monotonicity) fen) == H(~, . . . ,~) is an increasing function of nand

H(pl, ... ,Pn) ::; H(~, ...

,~) == f(n).

(4) (Extendability) H(Pl,'" ,Pn) = H(Pl,'" .v«, 0). (5) (Symmetry) H(Pl,.' . ,Pn) = H(P1r(l) , ... 'P1r(n)) for every permutation {I, ... ,n}. (6) (Additivity)

H(Plql, ... ,Plqm,P2ql,··· ,P2Qm,··· ,PnQl,··· ,PnQm) = H(Pl,'.' ,Pn) + H(ql, ... ,qm)'

1T

of

5


(7) (Subadditivity) If rjk ~ 0,

(8) (Concavity) For p, q H(ap

E rjk = 1, E rjk = Pj, E rjk = qk, j,k

k

E ~n and a E

+ (1 - a)q)

then

j

(0,1) it holds that

~ aH(p)

+ (1 -

a)H(q).

Proof. (1),(2),(4) and (5) are obvious.

(3) f (n) = log n for n ~ 1, so that f is an increasing function. As to the second staternent, without loss of generality we can assume Pj > 0 for all j. Then

or H(Pl, ... ,Pn) ~ H(~, ... , ~). (6) follows from the following computation: n

H(Plql,."

m

,Plqm,'" ,Pnql,'" ,Pnqm) = - LLPjqklogPjqk j=lk=l

= - LPjqk logpj - LPjqk logqk j,k j,k

= H(Pl, ... ,Pn)

+ H(qI, ... , qrn).

(7) is a reformulation of Theorem 1 (4). (8) Since ¢(t) = -t logt is concave for t > 0 we have 1

< j < n.

Summing w.r.t. j we obtain the desired inequality. Before characterizing the .Shannon entropy we consider the function f (n) H (~, ... , ~), n ~ 1. f (n) stands for the entropy or uncertainty or information that

Chapter I: Entropy

6

a finite scheme (X, (~, ... ,~)) has. We impose some conditions on the function f (n). In the case where n == 1, there is no uncertainty, so that we have

1°) f(l) ==

o.

If n ~ m, then p ==

2°) f(n)

~

(*, . .. , *) has more uncertainty than

f(m) if n

~

q

== (~, ... , ~). Hence,

m, i.e., f is nondecreasing.

, ~)) are two independent schema, then the comIf (X, (*, . . . , *)) and (Y, (~, pound scheme is (X x Y, (n~' , n~)). In this case, the uncertainty of X x Y should be equal to the sum of those of X and Y, i.e., 3°) f(nm) == f(n)

+ f(m).

Under these conditions we can characterize

f.

Proposition 3. Let f : N -+ lR+ be a function satisfying conditions 1°),2°) and 3°). Then there is some A > 0 such that

f(n) == Alogn,

nEN.

Proof. This is well-known in functional equation theory. For the sake of completeness we sketch the proof. By 3°) we have f(n 2 ) == 2f(n) and, in general, n,r E N,

(1.4)

which can be verified by mathematical induction. Now let r, s, n E N be such that r, s ~ 2. Choose mEN so that

Then m log r ::; n log s < (rn + 1) log r and hence

m

logs

m

1

-< --< -+-. n - logr n n

(1.5)

On the other hand, by 2°) we get

and hence by (1.4)

rnf(r) ::; nf(s) ::; (rn + l)f(r), so that

m

f(s)

m

1

-n < - - < -+-. - f(r) - n n

(1.6)

7


'rhus (1.5) and (1.6) give

f(s) f(r) I

logs I < ~ logr - n'

n 2: 1,

which implies that

f(s)

f(r)

log s

logr'

Since r, s 2: 2 are arbitrary, it follows that for some A >

f(n) == A logn,

n

°

2: 2,

and by 1°) the above equaity is true for n == 1 too. (PI,'" ,Pn) E ~n' If p(Xj) Pj == Consider a finite scheme (X,p) with p log ~ as information or entropy, which is justified by Proposition 3. This suggests that each Xj has inforrnation of -logpj and H(X) == ~, then Xj has log n == n

- I: Pj log Pj

is the average information that X == {Xl,... ,xn } has, giving a good

j=l

reason to define the entropy of (X, p) by (1.1). To characterize the Shannon entropy we consider two axioms.

The Shannon-Khintchine Axiom (SKA). (1°) H: U ~n --+ lR+ is continuous and, for every n 2: 2, n~2

H ( ~, ... , ~)

= max { H (p) : p

E

~n}'

mj

(3°) If p == (PI, ... ,Pn) E ~n, Pj ==

I: qjk

for 1 :::; j :::; nand qjk 2: 0, then

k=I

The Faddeev Axiom (FA). [1°] f(p) == II(p, 1-p) : [0,1] --+ lR is continuous and f(po) >

°for some Po

E [0,1].

Chapter I: Entropy

8

[2°] H(Pl, tation

,Pn)

of {I, [3°] If (PI, 1f

=

H(P1r(I), ... 'P1r(n)) for every (PI, . . . ,Pn) E ~n and permu-

,n}. ,Pn) E ~n and Pn = q + r

> 0 with q, r

~

0, then

(FA) is an improvement of (SKA) since [1°] and [3°] are simpler than (1°) and (3°), and [2°] is very natural. These two axioms are equivalent and they imply the Shannon entropy to within a positive constant multiple as is seen in the following theorem.

Theorem 4. The following statements are equivalent to each other: (1) H(·): U ~n -+ JR+ satisfies (SKA). n~2

(2) H(·):

U ~n

n~2

-+ JR+ satisfies (FA).

(3) There is some ,\ > 0 such that n

H(PI, ... ,Pn) =

-x LPj logpj,

(1.7)

j=I

Proof. (1) =} (2). [1°] follows from (10). [2°] is derived as follows. If PI, ... ,Pn are positive rationals, then Pi = ~ for some .e I , ... ,.en, mEN. Hence

Thus, for any perrnutation 1f of {I, ... ,n}, H(PI, ... ,Pn) = H(P1r(I), ... 'P1r(n)). The case where Pj'S are not necessarily rational follows from the continuity of H ((1°)) and the approximation by sequences of rational numbers. [3°]. It follows from (2°), (3°) and [2°] that

HG, D= HG, ~,o,o) = HG,O, ~,o) 1 1)

= H ( 2' 2

1 1 + 2H (1, 0) + 2H (1, 0),

9


implying H(I, 0)

= O. Hence n-1

= H(Pb . . . ,Pn) + I>;H(l, 0) + PnH( .!L, .!-) Pn Pn

j=l

= H(P1,' .. ,Pn) + PnH( .!L, .!-), Pn Pn

i.e., [3°] holds. (2) =? (3). Using [3°], we have for any P, q ~ 0, r H(p,q,r)

> 0 with P + q + r = 1

= H(p,q+r) + (q+r)H(-q-, _r_) q+r q+r

= H(q,p+r) + (p+r)H(-P-, _r_). p+r p+r

If we set f(p)

= H(p, 1 f(p)

Letting p

p), then the second of the above equalities becomes

+ (1- P)f(l ~ p) =

f(q)

+ (1- q)f(l ~ q).

(1.8)

= 0 and 0 < q < 1, we get f(O) = H(O, 1) = O.

Integrating (1.8) w.r.t. q from 0 to 1

p gives

r» io

(1 - p)f(P) + (1 - p)2

P

dt = [1- f(t) dt + p2

io

t f~) dt.

i

(1.9)

t

p

Since f (p) is continuous and hence all terms except the first on the LHS (= left hand side) of (1.9) are differentiable, we see that f(p) is also differentiable on (0,1). By differentiating (1.9) we obtain

1 1

(1 - p)f I (p) - f(p) - 2(1 - p)

f(t) dt = - f(1 - p)

1

+ 2p

1

f(t) dt - f(p).

-3-

o p t

P

We can simplify the above by using f(p) = f(1 - p) to get (1 - p)f'(P)

= 2(1 - p) [1 f(t) dt + 2p [1 f~) dt _ f(P).

io

i

p

t

P

(1.10)

Chapter I: Entropy

10

It follows that f'(p) is also differentiable on (0,1). By differentiating (1.10) we have

~ p)

rep) = - p(l

1 1

(1.11)

O
Basic properties of conditional expectations and probabilities needed for our later work are given below. In the rest of this section, we assume that all the functions are in L 1 (:£) and that equalities and inequalities are in the u-a.e. sense. (1) (Linearity) E('I~) : L 1(.x) -+ L1(~) is a linear operator, i.e.,

Eio] + fJgl~) == aE(fl~) In fact, the RHS is

i

+ fJE(gl~),

~-rneasurable

(af + (3g) dJL =

a,fJ E lR, i.s E L 1 (.x) .

and the equality

i

{aEUI!D) + (3E(gl!D)} dJL

12

Chapter I: Entropy

holds for every A E

~.

I 2 0 ~ E(/I~) 2 o. In fact, 0 < fA I dj.L == fA E(/I~) dj.L for (2) (Positivity)

(3) IE(/I~)I

A E ~ and E(/I~) is ~-measurable.

< E(I/II~)·

This follows from (2) and

-III ~ I

III·

~

(4) (Boundedness) IIE(/I~)lll ~ IIfllb 11·111 being. the Ll-norm. This is immediate from (3).

(5) (Idempotency) E(E(/I~)I~) == E(/I~)· For, observe that the RHS is already

~-measurable.

(6) E(·I~) : Ll(X) -+ Ll(~) is a projection of norm 1. This follows from (1), (4) and (5).

(7) ~l

< ~2 ~ E(E(/I~2)1~1) == E(/I~l).

In fact, the RHS is

i

~l-measurable

I dp, =

and

i E(JI~h) i E(JI~h) dp, =

(8) E(112) == E(/) == fx smallest a-subalgebra of X.

I du;

A

dp"

the expectation of

I,

E~l.

== {0, X}, the

where 2

For, E(/12) is a constant.

(9) [I E LOO(X), 9 E Ll(~)] or [I E Ll(X), 9 E LOO(~)] implies E(gll~) == gE(/I~)·

In fact, both sides are ~-measurable and the equality holds for a ~-simple function g. The general case follows from a suitable approximation by a sequence of ~-simple functions. (10) (Dominated Convergence Theorem)

Ifni

~

g, In -+ I u-a.e.

~ E(lnl~)

E(/I~) u-a.e. and in L 1 .

For, observe that for every A E

i

I

E(Jnl!D) dp. -

i

~

E(JI!D) dP,1

i 0 be arbitrary. Since

L 1(X), there exist k ~ 1 and g E L1(~k) such that ~k ~ ~n and E(gl~n) = g, so that

III -

U Ll(~n)

is dense in

n=l gill < e. Then, for n ~ k,

Chapter I: Entropy

16

::; 211f -

gl/1 < 2£.

This shows that E(fl~n) -+ E(fl~) in L 1 . As to u-a.e. convergence, we proceed as follows. For any e > 0

JL ([ li~-:~p IEUI!Dn) - fl >

vie])

< JL( [ ~-:~p {lEU - gl!Dn) - U - g)1 + IE(gl!Dn) - gl} > vie])

~ JL( [li~~p {lEU -

gl!Dn)1 + If - gl} >

vie])

~ JL( [li~S~P lEU - gl!Dn)1 > ~]) + JL( [If - gl > ~]) (p(S-lAIS- 1q)))) ,

by (3.2),

AE2t

=

L E(¢>(P(AIq)) S)), L E(¢>(P(AIq)))), by 0

by (16) in Section 1.2,

AE2t

=

jl

0

s:' = u,

AE2t

=

H(2tIq))·

(11) is obvious. Remark 2. (1) Theorern 1 holds if H(2t) and H(2tIq)) are replaced ,by I(2t) and I(2tIq)), respectively, where (10) and (11) of Theorem 1 are then read as: (10) I(S-12tIS- 1q)) = I(2tIq)) 0 S. (11) I(S-l2t) = I(2t) 0 S.

(2) In Theorem 1 (2), if 2t ~ ~ and q) = ~, then 2t V ~ = ~ and Q( V ~ = ~, so that H(2tI~) = O. This means that ~ contains all the information that 2t has, and hence the conditional entropy of 2t given ~ equals o. In particular, H(2tIX) = 0 for every 2t E P(X). Definition 3. Let 2t E P{X). H(8 1) = H(82). That is, the Kolmogorov-Sinai entropy of measure preserving transformations is invariant under isomorphism. Proof. Let ip : Xl -+ X 2 be the isomorphism between two systems. Let 2t1 E P(X1) and observe that cp(2t1 ) = {cp(A) : A E 2(1} E P(X2). Moreover, H(2t1, 8 1 ) = H(CP(2(l) , 8 2 ) since J.l1(A) = J.l2(cp(A)) for A E 2(1. Hence, H(81 ) ~ H(82). The converse H(8 1) 2: H(82) is also true. Thus H(8 1 ) = H(82). It trivially follows frorn the above theorem that if H(Sl) =1= H(82 ) , then 8 1 '1- 8 2 . In the next exarnple we show how to compute the entropy of Bernoulli shifts.

1.3. The Kolmogorov-Sinai entropy

27

Example 14 (Bernoulli shifts). Let (Xo,p) be a finite scheme, where X o = {al,'" ,all and p = (PI, ... ,Pi) E ill, so that p(aj) = Pj, 1 :::; i : t. Consider the infinite Cartesian product

x = X~ = {x = (... where Z

,X-I,XO,XI,"') : Xk

E

X o, k E Z},

= {O, ±1, ±2, ... }, and the shift S on X given by

A cylinder set is defined by

and let

J.to([X?' .. x~]) = p(x?) ... p(x~). Extend J.to to the a-algebra X generated by all cylinder sets, denoted by u: Note that S is measure-preserving w.r.t. J.t and hence (X, X, u; S) is a dynarnical system. The shift S is called a (PI, . . . ,Pl)-Bernoulli shift. Since 2t = {[xo = all, ... , [xo = all} is a finite partition of X and

00

V

-

n=-oo

sn2t = X by definition, we have by Theorem 10

and Lemma 5 (2) that

H(S) = H(2t,S) = lim

n-+oo

!:H(nyl S-k2t). n k=O

n-l

Now V S-k2t = {[xo ... Xn-l] : Xj E X o, 0 :::; j :::; n - I} and hence k=O

J.t([xo ... Xn-l]) log J.t([xo ... Xn-l])

Xo,··· ,Xn-l EXo

= -

L

J.t([xo]) log J.t([xo]) - ... -

xoEXo

=

since J.t([aj])

L xn-lEXo

nH(2t)

= p(aj) = Pj for

1 :::; j :::; n. This implies that l

H(S) = H(2t) = - LPj logpj. j=l

J.t([Xn-I]) log J.t([Xn-I])

Chapter I: Entropy

28

Thus (~, ~) -Bernoulli shift and ( ~, ~, ~) -Bernoulli shift are not isornorphic. (PI, ... ,Pn)- and (ql, ... ,qm)-Bernoulli shifts have the same entropy, i.e.,

If

m

n

- LPjlogpj == - Lqklogqk, j=l

k=l

then are these isomorphic? This was affirmatively solved by Ornsterin [1](1970) as: Theorem 15. Two Bernoulli shifts with the same entropy are isomorphic. We say that the entropy is a complete invariant among Bernoulli shifts. The proof rnay also be found in standard textbooks such as Brown [3], Ornstein [3] and Walters [1]. Example 16 (Markov shifts). Consider a finite scheme (Xo,p) and the inifinite product space X == X~ with the shift S as in Example 14. Let M == (mij) be an f x f £

stochastic matrix, Le.,

mij

2: 0,

E rnij == 1 for 1 ~ i,j ~ f,

and m == (ml, ...

,m£)

j=l

£

E mimij == mj for 1 < j

~ f. For each i, i, i=l mij indicates the transition probability from the state a; to the state aj and the row vector m is fixed by M' in the sense that mM == m. We always assume that m; > 0 for every i 1, ... ,f. Now we define J-lo on 911, the set of all cylinder sets, by

be a probability distribution such that

11.0

r"

{[a· ... a· ]) -~o

~n

m·~o rn·~O~l . . . . m·~n-l ~n . .

J-lo is uniquely extended to a measure J-l on X which is S-invariant. The shift S is called an (M, m)-Markov shift. To cornpute the entropy of an (M, m)-Markov shift S consider a partition Q{ ==

{[xo

==

al], ... ,[xo == all}

E

P(X), which satisfies

V

n=-oo

sn§( == X. As in Example

14, we see that H

nV - l S-kQ{ ) ==( k=O

Xo,··· ,Xn-l EXo

£

L io,···

£

mio mioi1

•••

mi n-2 in-l

£

== - ' " m·~o logm·~o - (n - 1) L....J '" L....J io=l

log mio rnioil

,in-l=l

i,j=l

m·m· ~ ~J·logm·· ~J

... min-2in-l

1.4. Algebraic models £

since n

L: mimij

i=l -t 00 we

29

£

= mj and

L: mij

= 1 for 1

< i, j < t.

By dividing n and letting

j=l

get £

H(S) = -

L

mirr~ijlogmij.

i,j=l

1.4. Algebraic models In the previous section, we have defined the Kolmogorov-Sinai entropy for measure preserving transforrnations and have seen that two isomorphic dynarnical systems have the .same entropy. In this section, we relax isomorphism to conjugacy, which still has the property that two conjugate dynamical systems have the same entropy, Algebraic models of dynamical systems are good tools to examine conjugacy arnong these systems and these will be fully studied in this section. Let (X, X, J.L, S) be a dynamical system. Two sets A, B E X are said to be J.Lequivalent, denoted A ~ B, if J.L(A~B) = 0, where A~B = (A U B) - (A n B), the symmetric difference. Let A. = {B Ex: A ~ B}, the equivalence class containing A E X, and ~I.£ = {A. : A E X}, the set of all equivalence classes of x, called the measure . algebra of J.L. The measure J.L on ~ 1.£ is of course defined by

J.L(A.) = J.L(A), and is strictly positive in the sense that J.L(A) > 0 if A. :f= 0. For now we consider conjugacy between a pair of probability rneasures.

Definition 1. Let (Xj , Xj, J.Lj) (j = 1,2) be two probability measure spaces with the associated rneasure algebras ~l and ~2, respectively. Then J.Ll and J.L2 are said to be conjugate, denoted J.Ll ~ J.L2, if there exists a measure preserving onto isomorphism T : ~l -t ~2, i.e., T satisfies that BE

~l,

Conjugacy between a pair of probability measures can be phrased in terms of a unitary operator between L 2-spaces as follows.

Chapter I: Entropy

30

Proposition 2. Let (Xj, Xj, J-tj) (j = 1,2) be a pair of probability measure spaces. Then, J-tl c::: J-t2 iff there exists a unitary operator U : L 2(J-tl) -+ L 2(J-t2) such that ULoo(J-tl) ~ Loo(J-t2) and U(fg) = Uf· Ug,

Proof. To prove the "only if' part, assume J-tl c::: J-t2 and let T : Q)l -+ Q)2 be an onto measure preserving isomorphism, where Q)j is the measure algebra of J-tj (j = 1,2). Define an operator U : L 2(J-tl) -+ L 2(J-t2) by B E Q)l.

Note that IIU1BII2 = JJ-t2(TB) = JJ-tl(B) = II1BII2 since T is measure preserving. Then, U can be linearly extended to the set of all Q)l-simple functions and is an isometry. Thus U becomes a unitary operator since T is onto. n

Now, let f

= L:: a j1B j,

j=l disjoint. Then, it holds that

where aj's are complex and B j E Q)l (1

Uf2 = U(

r r

~ajlBj J

~

j ~ n) are

= u(~ajaklBjlBk) J,k

= Lajak1TBjnTBk = La;lTBj' j,k

(Uf)2 =

(~ajlTBj J

=

= ~ajaklTBjlTBk

L aj ak 1T(BjnB

k)

j,k

j

J,k

= L a;lTBj' j

and hence Uf2 = (Uf)2. This implies by the polarization identity that U(fg) = U 1 . U 9 for all Q)l-simple functions I, 9 and then for all i, 9 E LOO (J-tl) by a suitable approximation. Conversely suppose U satisfies the conditions mentioned. If f is an indicator function corresponding to B l E Q)l, then 1 2 = ! and U1 = U1 2 = (U1)2, so that U1 is also an indicator function corresponding to B~ E Q)2, say. Define T : Q)l -+ Q)2 by TB l = B~ for B; E Q)l. Since U is onto, T is also onto. Since U is a unitary operator, T B l = 0 implies B l = 0. To see that T is a measure preserving isomorphism, note that J-tl(Bl) = Ilf"~ = "UI"~ = J-t2(TB l) for B l E Q)l, where 1 is the corresponding element to B l as above. Now T(B lnB2) = TB lnTB2 follows from U(lg) = UI·Ug,

1.4. Algebraic models

31

where 9 corresponds to B 2. Moreover, T(B 1 U B 2) = TB 1 U TB 2 follows from the fact that f + 9 - fg corresponds to B 1 U B 2, and hence

fL1(B1 U B 2) = Ilf + 9 - fgll~ = IIU(f + 9 - fg)ll~

= IIUf + Ug - Uf·

TCQl

nQl

fL2CQl TB

That Bn ) = TBn and verified by a suitable approximation.

UglI~ = fL2(TB1 U TB 2). n)

=

fLl CQl B

n)

for {Bn } s.;; SJ3 1 are

Now let (X, X, fL) be a probability measure space and r(fL) be the set of (equivalence classes of) functions f E L 00 (fL) with If I == 1. Note that I' (fL) is a group with the product of pointwise multiplication, where the complex conjugate 1 is the inverse of f E r(fL). We identify the circle group C = {z E C : Izi = I} with the constant functions of T (fL), so that C c r (fL), where C is the cornplex humber field. For convenience we assume that r(fL) is a group of X-measurable functions f : X -+ C such that f,g E r(fL) and f = 9 u-a.e. imply f(x) = g(x) for every x E X. This is due to the existence of a lifting on Loo(fL) (cf. Tulcea and Tulcea [1]). Let us define a function 0 we can choose

oS

ma

say.

(5.8)

2: 1 such that

J(rn, n) S J(rn,O) < e,

m 2: ma, n 2: 1.

Hence 1

k

L

~(A) log~(A) < I(k, m, n)

+ 10,

m

2: rna, n 2: 1.

AE2{V 2{k-l

If we let k -+ 00, then I(k, m, n) -+ H((" 2l(m) , S) and the LHS of (5.8)-+ H ((" 2l, S). Thus we have H((" 2l, S) S H((" 2l(m) , S)

Since {H((" 2l(m) , S)}

:=1

+ c,

m 2: ma, n 2: 1.

(5.9)

is monotonely nondecreasing by Remark 7 (2), lim H((" 2l(m) , S)

m-+oo

= He

exists. This and (5.9) imply that

H((" S) S He S H((" S), so that (5.7) holds. In the rest of the section we consider an integral representation of the entropy functionaL Take a /-L E Ps(X) and let MJ-t(X) be the set of all measures (, E M(X) such that (, ~ /-L. MJ-t(X) is a Banach space with the total variation norm II . II and is identified with L 1 (/-L) by I == /-L/ for I E L 1 (/-L), where P,f(A) Note that II/-L/II =

11/11t for I

E

=

L

L 1 (/-L).

f dp"

AEX.

S denotes the operator on M(X)

defined by

A E X, (, E M(X).

A linear functional F on MJ-t(X) is said to be S-stationary if F(S(,) = F((,) for (, E

MJ-t(X).

Let 2l E P(X) be fixed and consider an entropy functional H(·, 2l, S) on Ms(X) n MJ-t(X). For each (, E Ms(X), H((" 2l, S) is represented as

H(~, 21, S) =

Ix h~(x) ~(dx),

(5.10)

48

Chapter I: Entropy

he == -

L {Pel (AI2l

oo )

log Pel (AI2l oo )

Pe2 (AI2loo ) log Pe2 (AI2l oo )

-

AE~

+ iPe (AI2too ) log Pe (AI2too ) 3

3

-

iPe4 (AI2loo ) log Pe4 (AI2loo ) }

.

Our goal is to find a universal function h on X such that (5.10) holds instead of he. The following theorem is a partial solution to this.

Theorem 9. The entropy functional H(·, m, S) on Ms(X) n MJL(X) is extended to a functional H(.,m, S) on M JL (X) so that it is an S-stationary nonnegative bounded linear functional. Moreover, it has an integral representation

ll(("

Q(, S) =

L

h(x) (,(dx),

by some S-invariant nonnegative measurable function h on X. Proof. Let 'JJL be the a-subalgebra of X consisting of S-invariant (mod p,) sets, i.e., 'JJL == {A EX: J-L(S-l A~A) == O}. EJL(·I'JJL) denotes the conditional expectation relative to 'JJL under J-L. We define a functional H(., m, S) on MJL(X) by

ll(("

Q(, S)

= H ( E p (:~

IJ

p)

du, Q(, S),

(-f

Then this functional is well-defined since EJL I'JJL)djl E Ms(X). Moreover, it is linear. The boundedness of H(., m, S) follows from

Ill(("

Q(, S)

I= IH(e; (:~ IJp )djL, Q(, S) I < 1Q(I"Ep(:~IJp)djLll ~ Imlll~11

for ~ E MJL(X). To see that

H(., m, S) is S-stationary, observe that

1.6. Relative entropy and K ullback-Leibler information

49

since d 0) is convex, we have for A E X with v(A) > 0 and for a measurable function f ~ 0 that

by Jensen's Inequality (cf. Section 1.2 (12)). If, in particular, JL(A) j1.(A) log v(A) :S

It then holds that for any

2{ E

To show the reverse inequality, let k

=

!fIf;, then

P(X)

Taking the supremum on the LHS, we see that by (6.1)

n,

=

dJL) 1fA (dJL· dv log dv du.

""' JL(A) ""' f (dJL dJL) LJ j1.(A) log v(A) :S ALJ lA dv log dv du = AEm Em

A

f

['5.- < dJL < k + n - du

dJL . ] A n ,n 2 = [ dv ~ n .

1]

n'

lxf

(dJL dJL) dv log dv du.

51


Then {An,k : 0 ::; k ::; n 2 } E P(X) for each n 2:: 1. Moreover, by the definition of

An,k'S k -v(A n k) ::;

n

for 0 ::; k ::; n

2

'

!

X

lA

k

n,

dJ.L dv::; --v(A k+ 1 -d n k) n

V

'

1 and n 2:: 1. This implies

-

k +1 -k < J.L(An,k) < -. n - V(An,k) -

n

Let K 1 = {k: ~ ::; ~} and K 2 = {k: ~ 2:: ~}. Since 'lj;(t) on (0, ~) and increasing on (~, 1), we have that k + II k + 1 - og - - < J.L(An'k) oIg J.L(An,k) n n - V(An,k) V(An,k) ,

-k Iog -k < J.L(An'k) oIg J.L (An ,k) n n - V(An,k) V(An,k) ,

= tlogt is decreasing

k E K 1,

k E K 2•

Define a function b« for n 2:: 1 by

In(x),

x E

hn(x) = { ( ) 9n x ,

xE

n2- 1

U

An k

U

An k '

kEK2 kEK l

'

'

n2- 1

where In = '" '" ~ klA n n, k and 9n = ~ tl!IA n n, k' It then follows that for n 2:: 1 k=O

k=O

which implies that

H(p,lv) 2: On the other hand, let

I

Ln;

log n; du.

= ~ and observe that for x E X and n 2:: 1

o::; 9n(x) o::; I(x) -

1 n

In (x) ::; -, 1 n

In(x) ::; 9n(X) - In(x) ::; -,

Chapter I: Entropy

52

Then we see that

o :s; If -

hn I :s;

If - f n 11 kEUK2 An ' k + If - gn 11 kEKl U An k , '

n

~

1,

and hence

{ flog f dv = lim ( h n log h n du n~oo lx

lx

< H(/1lv).

Thus the reverse inequality is proved. Therefore (6.2) is true. Let us examine some properties of the relative entropy, which are collected in the following theorem. Theorem 3. (1) Let ~n (n ~ 1) and ~ be a-subolqebras of eX and /1, v E P(X). If ~n t~, then H fD n (/1lv) t HfD(/1l v). (2) Let 0 :s; a < 1 and /1j, Vj E P(X) (j = 1, 2). Then,

II(a/11

+ (1 -

a)/12I a Vl

+ (1

(3) If II/1n - vnll -+ 0 and Ilvn -

a)v2):S; aH(/11Ivl)

vII

+ (1 -

-+ 0 with {/1n, /1, u«,

t/ :

a)H(/12Iv2).

(6.3)

n ~ I} ~ P(X), then

(6.4) (4) If /1, v

E

P(X), then

Proof. (1) By Lemrna 1, {H fD n (/1lv)} is a monotone nondecreasing sequence. only have to show the convergence: H fD n (/1lv) -+ H~ (/1lv). Suppose first /1 ~ t/, For n ~ 1 let /1n = /11~n and V n = vl~n' the restrictions of /1 and ~n, respectively. Then /1 ~ v implies /1n ~ V n for n ~ 1. If we let fn = ~~: f = ~~, then it follows from Theorem 2 that for n ~ 1

Since {fn : n ~ I} is a martingale in L1(v), we have fn -+ f t« log fn -+ flog f u-a.e. Then, Fatou's lemma implies that

HV(fLlv) = [ flogfdv:::; liminf [ fn10gfndv lx n~oo lx

We that v to and

v-a.e. and hence

1.6. Relative entropy and K ullback- Leibler information

53

from which we conclude that H!Dn(JLlv) -+ H!D(JLlv). Now assume that JL 1:- u, Then there is an A E ~ such that v(A) = 0 and JL(A) > O. Let e > 0 be such that JL(A) > 2c. Since ~n t ~, there exist n 2: 1 and B E ~n such that v(A~B) < c. JL(A~B) < c, Now observe that lim H!Dn (JLlv) 2: H!Dn (JLlv) 2: H~(JLlv)

n-+oo

JL( B)

JL( BC)

c

= M(B) log v(B) + M(B ) log v(Bc) ,

(6.5)

where ~ = {B, B C } E P(~n). Note that

v(B) < v(A)

+e=

c.

Thus one has

Therefore, by (6.5) we have lim II9J n (Mlv) 2': !M(A) log M(B) + M(BC ) 2 e -+ 00 as e -+ O.

-

v(B C )

n-+oo

That is, (2) We can assume 0 < a < 1. If JLl 1:- VI or JL2 both sides of (6.3) become 00. So we assume JLl « prove (6.3) it suffices to show that for any 2t E P(X)

1:- V2, VI

then (6.3) is true since and JL2 «V2. In order to

(6.6)

Chapter I: Entropy

54

Let

Cl, C2,

d1 , d2 be nonnegative constants and consider a function sp defined by

cp(x) =

{ XCI

XCI

+ (1 -

X)C2 }

log d X

+ (1-x)c2

1+ (1 - X )d· 2

Then we see that

for

X

E [0,1] since

Cj,

dj 2:: 0 for j = 1,2. Thus,

ip

is convex and hence

0 and "=" holds iff t = 1, we have

j(t)(logg(t) -logj(t)) = Jet) log ~~:~ -::;. get) where we define ~

= 00 (a > 0) and

0·00

Jet) a.e.,

= O. This implies that

1

(J(t) log Jet) - Jet) logg(t)) dt

-::;.1 (g(t) -

Jet)) dt = 0,

or (6.8) is true. Obviously, the equality holds iff f(t) = g(t) a.e.

Lemma 6. Let ~ and 'fJ be a pair of real random variables with probability distributions J.le and J.l"" respectively. If J.le and J.l", have probability density functions f and g, respectively, then

Proof. If H(J.lelJ.lTJ) < 00, then J-te I(17)

= = =

«

J.lTJ by Theorem 2. Hence

1 1

(J(t) logj(t)~ Jet) logg(t)) dt

Jet) log ~gj dt

t v« log dJ.le/ dtdt dt

JJR dt

= f d/-L 0] for J.-l E VJt. A subset K E eX is called a chunk if there is some J.-l E VJt such that K ~ Kp, and J-L(K) > o. A disjoint union of chunks is called a chain. Since A(X) < 00 and A(K) > 0 for every chunk K, we see that every chain is a countable disjoint union of chunks. We also see that a union of two disjoint chains is a chain and, hence, a countable union of (not necessarily disjoint) chains is a chain. Let a = sup { A(0) : 0 is a chain}. Then, we can find an increasing sequence 00

{On} of chains such that lim A(On) = a. If we let 0 = U On, then 0 is a n-+oo n=l chain and A(O) = a. Moreover, there exists a sequence {Kn } of chunks such that c = n=l U Knwith a sequence of measures {J-Ln} ~ 9J1 such that K n ~ Kp,n and J-Ln(Kn) > 0 for n ~ 1. Let SJt = {J-Ln : n ~ I}. Obviously SJt ~ VJt since SJt ~ VJt. We shall show VJt ~ SJt. Suppose that A E eX satisfies J-Ln(A) = 0 for n ~ 1. Take an arbitrary J.-l E VJt. Since J-L(A\KJ-£) = 0 by the definition of Kp" we can assume that A ~ KJ-£. If J-L(A\O) > 0, then A(A\O) > 0 and hence Au 0 is a chain with A(A U 0) > A(O) = a. This contradicts the maximality of O. Thus J-L(A\C) = o.

Chapter I: Entropy

58

Now observe that

00

A{A n C) ==

:E A{A n K n) == 0 n=l

since 0 == JLn{A) == JL{A n K n) == JAnK n fJ-tn dA and K; ~ KJ-tn imply A{A n K n) == 0 for n 2: 1. Therefore, JL{A) == JL{A\C} + JL{A n C) == O. This means VJt· ~ SJt. We now introduce sufficiency. Definition 10. Let ~ be a a-subalgebra of X and 9J1 ~ P{X). ~ is said to be sufficient for 9J1 if, for any A E X, there exists a ~-measurable function hA such that JL E 9J1. That is, ~ is sufficient for 9J1 if there exists a conditional probability function common to every JL E rot. The following is a fundamental characterization of sufficiency. Theorem 11. Let 9J1 ~ P{X) be a dominated set and ~ be a a-subalqebra of X. Then, ~ is sufficient for 9J1 iff there exists A E P{ X) such that 9J1 ~ {A} and ~ is ~ -measurable JL-a. e. for JL E 9J1.

Proof. Suppose that n 2: 1}. Let

~

is sufficient for 9J1. By Lemma 9 we assume that VJt == {JLn : 1

00

A(A)

= :E 2nJLn(A),

AEX.

n=l

Then, A E P{X) and VJt ~ {A}. Since ~ is sufficient for 9J1, for each A E X there exists a ~-measurable function hA such that

A(A n B)

=

f 2~ 1

Ep,n (lAI!D) dJLn

n=l

=

B

1·

hA

a»,

B

Hence EA{1AI~) == hA A-a.e. Take any JL E 9J1 and let 9 == ~. Then, for any A E X we have

L = L~~ = = Ix Ix = Ix = Ix = Ix = Ix gdA

dA

=

hA dJL

JL(A)

Ep,(lAI!D)dJL

E.x(lAI!D) dJL

E.x(1A 1!D)E.x (gl!D) dA

E.x(lAI!D)gdA

E.x(lAE.x(gl!D)I!D) o.


59

which implies that 9 = EA(gl~) and 9 is ~-measurable A-a.e. Conversely, suppose that there exists A E P(X) such that rot ~ {A} and ~ is ~-measurable J.-L-a.e. for J.-L E rot. Let J.-L E rot and A E X be arbitrary and let 9 = ~, which is ~-measurable u-a.e. Then, for B E ~ it holds that

L

E),(lAI!D) dtt =

= Since B E

Hence

~

~

L

E),(lAI!D)gd>'

L

lAg o. =

L

E,,(lAI!D) du.

is arbitrary, we see that

is sufficient for rot.

Remark 12. Let rot ~ P(X) and ~ be a a-subalgebra of X. rot is said to be homogeneous if J.-L ~ v for J.-L, lJ E rot. If rot is homogeneous, then ~ is sufficient foroot iff ~~ is ~- measurable for J.-L, lJ E rot. This is seen as follows. Let A be any measure in 001. Then 001 is dominated by A and Theorem 11 applies. We introduce another type of sufficiency.

Definition 13. So let rot pairwise sufficient for m1 if

~ ~

P(X) and ~ ~ X, a a-subalgebra. ~ is said to be is sufficient for every pair {J.-L, lJ} ~ 001.

It is clear from the definition that sufficiency implies pairwise sufficiency. To consider the converse implication, we need the following lemma.

Lemma 14. Let 001 ~ P(X) and ~ ~ X, a a-subalgebra. Then, ~ is pairwise sufficient for 001 iff, for any pair {J.-L, lJ} ~ rot, d(:~V) is ~-measurable (J.-L + lJ)-a.e. Proof. Assume that ~ is pairwise sufficient for 001. Let J.-L, lJ E 001 be arbitrary and let A = ~ E P(X). Since ~ is sufficient for {J.-L, lJ}, we have that ~ and ~~ are ~-measurable by the argument similar to the one given in the proof of Theorem 11. Thus d(:~v) is also ~-measurable. The converse is obvious in view of Theorem 11. Theorem 15. Let rot ~ P(X) be dominated and ~ ~ X, a a-subalgebra. Then, is sufficient for 001 iff ~ is pairwise sufficient for 001.

~

Chapter I: Entropy

60

Proof. The "only if' part was noted before. To prove the "if' part, suppose that is pairwise sufficient for oot. We can assurne that oot = {J.Ln : n 2: I} in view of Lemma 9. Let 00 1 AEX. .\(A) = 2nJln(A), ~

L

n=l

It follows from Lemma 14 that d(~~A) is ~-measurable for J.L E oot. Consequently, for J.L E oot d d(Jl: .\)

)-1

is

~-measurable.

Theorem 11 concludes the proof.

Now sufficiency can be phrased in terms of relative entropies. Theorem 16. Let J.L, v E P(X) and ~ ~ X, a a-subalqebra. (1) If ~ is sufficient for {J.L, v}, then H~ (J.Llv) = H(J.Llv). (2) If H~ (J.Llv) = H(J.Llv) < 00, then ~ is sufficient for {J.L, v}. Proof. We only have to show the case where J.L ~ u, Let J.Lo the restrictions of J.L and v to ~, respectively. Then, H(Jllv) - H!l)(Jllv)

= }fx (dJ.L log dv =

=

J.LI~

and Vo

=

vl~,

dJ.LO)

- log dvo dJl

r dJl/ dv log dJ.LodJl// dvdvo . dJlo du dvo

}x dJ.Lo / dvo

=

Ix

flog f d("

where f = d~j:o and d~ = ~dv. Let 'ljJ(t) = tlogt, t > O. Then, since and hence 0 < f < 00 u-a.e., we can write ¢(J(x)

= ¢(1) + {I(x) = f(x)

- 1+

- 1}¢'(1) + ~ {I (x) - 1}2¢" (g(x)

{f(x) - I} () 2g x

2

u-a.e.,

where g(x) E (1, f(x)) or (f(x), 1) and hence 0 < g
0 with O! + 13 = 1 and 'fJ, ~ E P; (X) imply that

JL

= 'fJ =~.

(3) The operator S on M(X) is continuous in the weak* topology if S is a continuous transformation on X. To see this, first we note that S is measurable. Let f E C(X). Then 8f E C(X) since 8f(·) = f(S·) and S is continuous. If C E X is compact, then there is a sequence {fn}~=1 ~ C(X) such that l-. t Ie as n --t 00 since X is compact and Hausdorff. Thus, le(S·) 81e(·) is Baire measurable, i.e., S-1C E X. Therefore, S is measurable. Now let JLn --t JL (weak*), i.e., JLn(f) --t JL(f) for f E C(X). Then, we have for f E C(X)

SJLn(f)

Ix = Ix =

Ix

f(x) SJLn(dx)

=

f(Sx) JLn(dx)

= JLn(SJ) --+ JL(SJ),

f(x) JLn(dS-1x)

since 8f E C(X), implying SJLn --t SJL (weak*). Therefore, S is continuous in the weak* topology.

2.2. Ergodic theorems Two celebrated Ergodic Theorems of Birkhoff and von Neumann will be stated and proved in this section. We begin with Birkhoff's ergodic theorem, where the operators 8 n 's are defined by (1.3).

Chapter II: Information Sources

72

Theorem 1 (Birkhoff Pointwise Ergodic Theorem). Let j-t E Ps(X) and E LI(X, j-t). Then there exists a unique fs E LI(X, j-t) such that (1) fs = lim Snf j-t-a.e.;

f

n-+oo

(2) Sis = Is u-a.e.; (3)

L L f dp. =

fs df-t for every S-invariant A E X;

(4) IISnl - Is II I,ll- -+ 0 as n -+ 00, II·"I,1l- being the norm in LI(X,j-t). II, in particular, j-t is ergodic, then Isis constant j-t-a. e. Proof. We only have to consider nonnegative I I(x) = lim sup (Snf)(x), n-+oo

E

LI(X, j-t). Let

f(x) = liminf (Snf)(x),

-

xEX.

n-+oo

To prove (1) it suffices to show that

since this implies that

I= I

p-a.e. and (1). Let M > 0 and e > 0 be fixed and

I M(X) = min {lex), M},

xEX.

Define n(x) to be the least integer n 2:: 1 such that n-I

1M(X) ~ (SnJ) (X) + e = !ti :E f(Si x ) + c,

xEX.

j=O

Note that n(x) is finite for each x E X. Since

1 and 1M are

S-invariant, we have

n(x)-I n(x)IM(x):s n(x) [(Sn(x)IM)(X) +e] =

:E

j(sjx) +n(x)e,

j=O

Choose a large enough N 2:: 1 such that

e j-t(A) < M Now we define

with

A = {x EX: n(x) > N}.

j and ii by

lex) =

{ I(x),

x~A

0,

XEA'

ii(x) = { n(x), 1,

x~A

xEA

x E X.

(2.1)

2.2. Ergodic theorems

73

Then we see that for all x E X

n(x)

~

N,

n(x)-l L IM(sjx) j=O

r},

r

E JR.

2.3. Ergodic and mixing properties

77

Then A r E X is S-invariant and hence J-l(A r ) = 0 or 1 for every r E lR by (1). This means I = const J-l-a.e. (5) => (6). Let I E Ll(X,J-l). Then, Is is measurable and S-invariant J-l-a.e. by Theorem 2.1. By (5) Is = const J-l-a.e. Hence, Is = Is dJ-l I dJ-l J-l-a.e. 2(X, (6) => (7). Let I, 9 E L J-l). Then, by (6), Is = I dJ-l J-l-a.e. and the Mean Ergodic Theorem implies

Ix Ix

=Ix

(7) => (8) => (9) are obvious since C(X) ~ B(X) ~ L 2(X, J-l) and (9) => (7) can be verified by a simple approximation argument since C(X) is dense in L 2(X, J-l). (8) => (10). Take I = lA and 9 = IB in (8). (10) => (11) is obvious. (11) => (1). Let A E X be S-invariant. Then (11) implies that J-l(A) = J-l(A) 2 , so that J-l(A) = 0 or 1. Hence (1) holds.

Remark 3. (1) Recall that a semialgebra of subsets of X is a set Xo such that (i) 0 E Xo ; (ii) A, B E X o => An B E X o; n (iii) A E X o => AC = .U B j with disjoint B l, . . . ,Bn E X O• J=l

As we have seen in Section 2.1, in an alphabet message space X~, the set VJt of all messages is a semialgebra. Another such example is the set X x ~ of all rectangles, where (Y,~) is another measurable space. (2) Let J-l E P(X) and X o be a semialgebra generating X, i.e., u(Xo) = X. If J-l is S-invariant on Xo, i.e., J-l(S-lA) = J-l(A) for A E X o, then J-l E Ps(X). In fact, let

Then, clearly Xo ~ Xl' It is not hard to see that each set in the algebra A(Xo) generated by X o isa finite disjoint union of sets in X o. Hence A(Xo) ~ Xl. Also it is not hard to see that Xl is a monotone class, i.e., {An}~=l ~ Xl and An t (or An L) imply

U An

n=l

E Xl (or

n An E Xl)'

n=l

Since the o-algebra generated by A(Xo) is

the monotone class generated by A(Xo), we have that X = u(A(Xo)) = Xl' Thus J-l E Ps(X). (3) In view of (2) above, we can replace X in conditions (10) and (11) of Theorem 2 by a semialgebra X o generating X. In fact, suppose that the equality in (10) of


78

Theorem 2 holds for A, B E Xo. Then it also holds for A, B E A(Xo) since each A E A(Xo) can be written as a finite disjoint union of some AI, ... ,An E X o. Now let e > 0 and A, B E X, and choose A o, B o E A(Xo) such that J.t(A~Ao) < e and J.t(B~Bo) < E; Note that for j 2:: 0

(S-j A n B)~(S-j A o n B o) ~ (S-j A~S-jA o) U

(B~Bo)

(S-j(A~Ao)) U (B~Bo)

and hence

since J.t is S-invariant. This implies that j

2::

o.

(3.1)

Moreover, we have that

1J.t(S-jA n B) - J.t(A)J.t(B)

I

::; 1J.t(S-jA n B) - J.t(S-j A o n B o)I + 1J.t(S-jA o n B o) - J.t(Ao)J.t(Bo)I

+ 1J.t(Ao)J.t(Bo) - J.t(A)J.t(Bo)I + 1J.t(A)J.t(Bo) < 4e + 1J.t(S-j A o n B o) - J.t(Ao)J.t(Bo)I,

J.t(A)J.t(B) I (3.2)

which is irrelevant for ergodicity but for mixing properties in Theorem 6 and Remark 11 below. Consequently by (3.1) it holds that 1 1n

I: j=O

/-£(S-j A n B)

/-£(A)/-£(B)/

< /~ ~ {/-£(S-jA n B) -/-£(S-jAo n Bo)}! +

I!I: n

/-£(S-j Ao n Bo)

+ 1J.t(Ao)J.t(Bo) < 4e +

I

/-£(Ao)/-£(Bo)

j=O

J.t(A)J.t(B) I

I~ ~/-£(S-jAo n Bo) -/-£(Ao)/-£(Bo)l,

where the second term on the RHS can be made < e for large enough n. This means that (10) of Theorem 2 holds. (4) Condition (11) of Theorem 2 suggests that the following conditions are equivalent to anyone of (1) - (11) of Theorem 2:


(7') nl~ (Snf, fh,1-' =

79

IU, Ih'1-'1

= J-L(f)2 (9') lim 1£( (Snf)f) = J-L(f)2 n-+oo (8') lim 1£( (Snf)f) n-+oo

2

for every f E L 2(X,p,);

for every f E B(X); for every f E C(X).

(5) J denotes the er-subalgebra consisting of S-invariant sets in X, i.e., J = {A X: S-lA = A}. For 1£ E P(X) let

E

the set of all J-L-a.e. S-invariant or S-invariant (mod 1£) sets in X. Clearly J ~ J J.L. Then, we can show that 1£ E Ps(X) is ergodic iff J-L(A) = 0 or 1 for every A E JJ.L. In fact, the "if' part is obvious. To prove the "only if' part, let A EaJ.L. First we note that n 2:: o. For, if n 2:: 1, then

s:»A~A ~

n-1

n-1

j=O

j=O

U (S-(j+1) A~S-j A) = Us! is:' A~A)

and hence

Now let A oo

00

= n

00

.U

n=O J=n

.

S-J A = lim sup S"?' A. Then we also note that J-L(A oo ) = J-L(A) n-+oo

since

p,(Aoo~A) = p,( C~OjQn s-j A )~A ) < p,( CQn s~j A)~A) 00

~

E J-L(S-j A~A) = o. j=n

Finally, we note that S-l A oo = A oo , since

n Us-j A = n=Oj=n n US-(j+1)A = n=Oj=n+1 n U s-j A = A n=Oj=n 00

s:' A oo = s:'

00

00

00

00

It follows from ergodicity of 1£ that J-L(A oo ) = J-L(A) = 0 or 1.

00

oo .


80

Example 4. Consider an (M,m)-Markov source J.L, where M = (mij) is an f x f stochastic rnatrix and m is a row probability vector such that m = mM with tti; > 0, 1 ~ i ~ f. If we write Mk = (m~;)) for k 2 1, then this gives the k-step transition probabilities, i.e., 1

< i,j < f.

M or J.L is said to be irreducible if for each i, j there is some k 2 1 such that rn~;) We first claim that

> o.

n-l

N

exists and N

=

=

lim.!.

n-+oo

n

LMk

k=O

(nij) is a stochastic matrix such that

NM=MN=N=N 2 •

In fact, let Ai = [xo = ail, 1 ~ i ~ f and apply the Pointwise Ergodic Theorem to f = 1Ai· Then we have that

exists u-a.e.x and

~ rfs(x)lA-(x)/-L(dx)=~ lim .!.~/-L(S-kAinAj) m; } x m; n-+oo n k=O 3

n-l

= n-+oo lim .!. " m~k) = nij n 'L....J '£)

(3.3)

k=O

for 1 ~ i, j ~ f. Thus N = (nij) is well-defined. The other properties of N are clear. Under this preparation we can characterize ergodicity of Markov sources. For an (M, m)-Markov source J.L the following conditions are equivalent: (1) J.L is ergodic. (2) nij = rnj for every i, j. (3) nij > 0 for every i, j. (4) J.L is irreducible. (1) => (2). By (3.3) we see that for each i, j

81


while the ergodicity of J.L implies the RHS is also equal to mimj' Hence nij =mj' (2) => (3) is clear since we are assuming m; > 0 for every i. 1 n-l (3) => (4). For any i,j, lim - E m~7) = nij > O. This irnplies that we can find n-too

n k=O

some k ~ 1 such that m~7) > O. That is, J.L is irreducible. It is not hard to show the implications (4) => (3) => (2) => (1) and we leave it to the reader. We now consider mixing properties for stationary sources which are stronger than ergodicity.

Definition 5. A stationary source J.L E Ps(X) is said to be strongly mixing (SM) if A,BEX, and to be weakly mixing (WM) if

A,BEX.

It follows from the definition and Theorem 2 that strong mixing => weak mixing => ergodicity. First we characterize strong mixing.

Theorem 6. For a stationary source J.LE Ps(X) the following conditions are equivalent to each other: (1) J.L is strongly mixing. (2) nl~~(snl,g)2'11 = (I, 1)2,11(1,g)2,11 for every I,g E L 2(X,J.L). That is, snl-+

L

f dp, weakly in L 2(X, p,) for every f E L 2(X, p,).

(3) nl~~(snf,fh,,,=

IU,lh,,,1 2

for every fEL 2(X,p,).

(4) lim J.L(s-nA n A) = J.L(A)2 for every AE X. n-too

(5) lim J.L(s-nA n-too

n A)

= J.L(A)2 for every A E :to, a generating semialgebra.

Proof. (2) => (1) is seen by considering 1 = 1A and 9 = lB. (2) => (3) => (4) => (5) is clear. (5) => (4) follows from (3.1) and (3.2) with A = Band A o = B«.


82

(1) => (2). Let A, B E

.x.

Then by (1) we have

lim (sn1A, 1B)2 p, = lim /-L(s-n A n B) = /-L(A)/-L(B) = (lA,

n-+oo

'

n-+oo

/,

If I =

I:

1)2 ' p,(1, 1n)2,p,.

m

aj

1Aj and g =

j=l

I:

{3k1Bk' simple functions, then

k=l

=

Ej,k aj~k(lAj' 1)2,p,(1, 1Bk)2,JL

Hence the equality in (2) is true for all simple functions I, g. Now let I, g E L 2(X, /-L) and E > o. Choose simple functions 10 and go such that III - loll2,JL < e and Ilg - goI12,p, < e. Also choose an integer no ~ 1 such that

n Then we see that for n

I(sn l,g)2,JL

~

~

no.

no

(/,1)2,JL(1,g)2,p,1

::; I(snl,g)2,p, - (snI0,g)2,p,1 + l(snlo,g)2,p, + I(sn 10,gO)2,p, - (/0, 1)2,p,(1, gO)2,JLI + 1(/0, 1)2,p,(1, gO)2,p, - (I, 1)2,p,(1, gO)2,p, I

(snI0,gO)2,p,1

+ I(I, 1)2,p,(1, gO)2,p, - (I, 1)2,JL(1, g)2,p,1 ::; l(sn(1 - 10),g)2,p,1 + I(snlo,g gO)2,p,1 +e + 1(1 - 10, 1)2,p,11(1,go)2,p,1 + 1(/, 1)2,p,11(1,g - gO)2,p,1 ::; III - 10112,p,lIgI12,JL + 1I/0112,p,lIg - goll2,JL + E + III - loll2,p,llgoll2,p, + 11/112,p,llg - goll2,JL ::; ellgI12,p, + II/0112,p,e + e + ellgoI12,p, + 11/1I2,p,e ::; ellgI12,p, + (11/112,P, + e)e + E + e(llgI12,JL + c) + ell/ll2,p,. It follows that nli~ (sn I,

g)2,p, = (I, 1)2,p,(1, g)2,JL.

(4) => (3) is derived by a similar argument as in the proof of (1) => (2) above.


(3) => (2). Take any

f

E

83

L 2(X, JL) and let

1-£ = 6 [S" t, c : c E C, n 2: O}, the closed subspace of L 2(X, JL) generated by the constant functions and S" t, n 2: Now consider the set

o.

Clearly 1-£1 is a closed subspace of L 2(X, JL) which contains f and constant functions, and is S-invariant. Hence 1-£1 contains 1-£. To see that 1-£1 = L 2(X, JL) let g E 1-£.1., the orthogonal complement of 1-£ in L 2(X, JL). Then we have

so that g E 1-£1. Thus 1-£.1. ~ 1-£1. Therefore 1-£1

= L 2(X, JL), i.e.,

(2) holds.

In Theorem 6 (2) and (3), L 2(X, JL) can be replaced by B(X) or C(X).

Example 7. Every Bernoulli source is strongly mixing. To see this let JL be a (PI, ... ,Pt)- Bernoulli source on X = X~. Let A = [x?· .. xJ], B = [y~ . . . y~] E VR. Then it is clear that since for a large enough n 2: 1 we have n+i

> t. By Theorem 6 JL is strongly mixing.

In order to characterize weak mixing we need the following definition and lernma.

Definition 8. A subset J ~ Z+ == {O, 1,2, ... } is said to be of density zero if lim

n.-+oo

where I n

= {O, 1,2, ... ,n -

.!.IJ n Jnl = n

I} (n 2: 1) and

Lemma 9. For a bounded sequence

0,

IJn Jnl is the cardinality of J n I n .

{an}~=l of real nurnbers the following conditions

are equivalent: 1 n-l (1) lim - L: lajl = 0; n.-+oo

n

j=O

1 n-l (2) lim - L: lajl2 = 0; n.-+oo

n

j=O

(3) There is a set J ~

z+

of density zero such that

lim

JtJ n.-+oo

an =

o.


84

Proof. If we can show (1)

(3), then (2)

¢}

lim an J'tJ n-+-oo

¢}

= 0 ¢}

(3) is easily verified by noting that

lim a~ J"tJ n-+-oo

= O.

~},

k

So we prove (1) ¢} (3). (1) =} (3). Suppose (1) is true and let

Ek Observe that E 1

~

E2

{n

=

~ •..

E

Z+ : lanl :?:

~

1.

and each Ek has density zero since

as n -+ 00 by (1). Hence for each k = 1,2, ... we can find an integer jk > 0 such that 1 = jo < i, < j2 < ... and (3.4) Now we set J =

jk-l

~

n < jk, then

U (Ek n [jk-l,jk)).

k=l

In I n

We first show that J has density zero. If

[In [O,jk-l)] U [In [jk-l,n)] < [Ek n [O,jk-l)] U [Ek+1 n [O,n)]

=

and hence by (3.4)

This implies that ~IJ n Jnl -+ 0 as n -+ Secondly, we show that lim an = J"tJ n-+-oo lanl < k~l. This gives the conclusion.

00,

o.

i.e., J has density zero. If n > jk and n tf. J, then n

tf. E k

and


85

(3) ::::} (1). Suppose the existence of a set J ~ Z+ and let e

> O. Then,

Since {an} is bounded and J has density zero, the first term can be made < E for large enough n. Since an -+ 0 as n -+ 00 and n t/. J, the second term can also be made < e for large enough n. Therefore (1) holds. Theorem 10. For a stationary source J-t E Ps(X) the following conditions are equivalent to each other: (1) J-t is weakly mixing. (2) For any A, B E X there is a set J ~ z+ of density zero such that

1 n-l 2 (3) lim - I: 1J-t(S-jA n B) - J-t(A)J-t(B) I = 0 for every A, B E X. n-+oo n j=O 1 n-l (4) lim - I: 1(Sk t, g)2 /1- - (f, 1)2 /1-(1, g)2 /1-1 = 0 for every t, g E L 2(X, J-t). n-+oo n k=O

'

,

,

(5) J-t x J-t is weakly mixing relative to S x S, where J-t x J-t is the product measure on (X x X,XQ9X). (6) J-t x rj is ergodic relative to S x T, where (Y,~, n, T) is an ergodic dynarnical system, i. e., rj E P s e (Y) . (7) J-t x J-t is ergodic relative to S x S. Remark 11. In Theorem 10 above, conditions (2), (3) and (4) may be replaced by (2'),(2"),(3'),(3") and (4'),(4") below, respectively, where X o is a semialgebra generating X: (2') For any A, B E Xo there exists a set J ~ Z+ of density zero such that lim J-t(s-n A n B) = J-t(A)J-t(B). J1:J n-+oo

(2") For any A E X o there exists a set J ~ Z+ of density zero such that lim J-t(s-n A n A) = J-t(A)2.

J1:J n-+oo

I:

1 2 (3') lim ..!:. IJL(S-iA n A) - JL(A)21 = 0 for every A E Xo. n-+oo n j=O 1 n-l . (3") lim - I: 1J-t(S-J A n A) - J-t(A)21 = 0 for every A E X o. n-+oo n j=O


86

Proof of Theorem 10. (1)

¢:>

(2)

¢:>

(3) follows from Lemma 9 with

(1) :::} (4) can be verified first for simple functions and then for L 2 -funct ions by a suitable approximation as in the proof of (1) :::} (2) of Theorem 6. (4):::} (1) is trivial. (2) :::} (5). Let A, B, C, D E X and choose J1, J 2 ~ Z+ of density zero such that

It follows that

(/-L x /-L)((S x s)-n(A x C)

lim JIUJ2f;) n-+oo

==

lim

n (B

x D))

/-L(s-n A n B)/-L(s-nc n D)

J1UJ2f;) n-+oo

== /-L(A)/-L(B)/-L(C)/-L(D) == (/-L x /-L)(A x C)(/-L x /-L)(B x D). Since X x X == {A x B : A, B E X} is a semialgebra and generates X ® X, and J 1 U J 2 ~ Z+ is of density zero, we invoke Lemma 9 and Remark 11 to see that /-L x /-L is weakly mixing. (5) :::} (6). Suppose /-Lx/-L is weakly mixing and (Y,~, T, 'T/) is an ergodic dynarnical system. First we note that (5) implies (2) and hence /-L itself is weakly mixing. Let A,B E X and C,D E~. Then n-1

.!:. :L(/t x 1])(S x T)-i(A x C) n (B x D)) n

j=O

1 n-1

== n

.

:L /-L(A)/-L(B)'T/(T-JC n D) j=O

n-1

+ .!:. :L {/t(S-i A n B) n

j=O

/t(A)/t(B) }7J(T-iC n D).

(3.5)

87


The first term on the RHS of (3.5) converges to

since

'f}

is ergodic. The second term on the RHS of (3.5) tends to 0 (n -+ (0) since I

~ ~ {JL(S-j A n B) n-l

S;

.!- E n

JL(A)JL(B)}1J(T-

jC

n D)!

IJL(S-j A n B) - JL(A)JL(B) I ~ 0 (n

~ (0),

j=O

since J-L is weakly mixing. Thus J.L x 'f} is ergodic since X x ~ is a semialgebra generating X®~.

(6) =} (7) is trivial. (7)

(3). Let A, B E X and observe that

=}

1

n-l.

n

j=O

1

n-l

.

- E JL(S-J A n B) = ;;: E(JL x JL)((S x S)-J(A x X) n (B x X)) j=O

-+ (J.L x J.L)(A x X)(J.L x J.L)(B x X),

by (7),

= J-L(A)J-L(B), n-l

n-l

.!- E JL(S-jAn B)2 = .!- E(JL x JL)((S x S)-j(A x A) n (B n B)) n

n j=O -+ (J-L x J.L)(A x A)(J.L x J-L)(B x B),

j=O

by (7),

2

= J-L(A) J-L(B)2.

Combining these two, we get n-l

.!- EIJL(S-j A n B) n

2

JL(A)JL(B) 1

j=O

n-l

=

.!- E {JL(S-j An B)2 n

2JL(S-j An B)JL(A)JL(B) + JL(A)2JL(B)2}

j=O

-+ J-L(A)2J.L(B)2

2J.L(A)J.L(B)J-L(A)J.L(B) + J.L(A)2J.L(B)2 =

o.

Thus (3) holds. Example 12. Let J-L be an (M,m)-Markov source on X = X~. J-L or M is said to be aperiodic if there is some n-o ~ 1 such that Mno has no zero entries. Then, the following statements are equivalent:


88

(1) J-L is strongly mixing. (2) J-L is weakly mixing. (3) J-L is irreducible and aperiodic. · m (n) = mj £or every 2,. J.. (4) 1rm n-+oo

ij

The proof may be found in Walters [1, p. 51]. In the rest of this section we use relative entropy to characterize ergodicity and mixing properties. Recall that for a pair of information sources J-L and v the relative entropy H(vlJ-L) of v w.r.t. 1£ was obtained by

~ v(A) log :~~~ : 2l E P(X) }

H(vIJL) = sup { = {

i

log

~: du,

if v

«

JL,

otherwise

00,

(cf. (1.6.1) and Theorem 1.6.2). The following lemma is necessary.

Lemma 13. Let J-Ln (n ~ 1), J-L E P(X). Suppose J.Ln :::; aJ.L for n ~ 1, where a > 0 is a constant. Then, lim J.Ln(A) = J.L(A) uniformly in A E X iff lim H(J-LnIJ.L) = o. n-+oo

n-+oo

Proof. The "if" part follows from Theorem 1.6.3 (4). To see the "only if" part, observe that {~} is uniformly bounded and converges to 1 in probability (w.r.t. 1£) by assumption. Since

t

~

0,

we have {~log ~} converges to 0 in probability. Thus, since {~log ~ } is uniformly bounded, we also have

lim H(JLnIJL) n-+oo

= lim n-+oo

r ddJ.Ln log ddJ.LnJ-L dJ.L

Jx

1£

=

We introduce some notations. Let J-L E P(X). For each n Mn on X Q9 X by 1

Mn(A x B) = n

n-l

L

j=O

o. ~

1 define a rneasure

.

J.L(8- J A n B),

A,BEX.


89

For a finite partition 2l E P(X) of X, J-L2{ denotes the restriction of J-L to 21 = o-(2l), i.e., J-LS)l == J-Ll m. For 2l, Q) E P(X) denote by A(2l x Q)) the algebra generated by the set {A x B : A E 2l, B E ~} of rectangles. We also let

- (

Il.; !!, Q3

)

=

,,_ ( ) iln(A x B) L..J {tn A x B log (A) (B)' AE2{,BE~ J-L J-L

Hjin(2l x~) = -

E

iln(A x B)logiln(A x B),

AE2{,BE~

HJ1-xJ1-(2l x~) = -

E

J-L(A)logJ-L(A) AES)l = HJ1-(2l) + HJ1-(Q)) , say.

E J-L(B)logJ-L(B) BE~

Now we have the following.

Proposition 14. For a stationary source p E Ps(X) the following statements are equivalent to each other: (1) J-L is ergodic. (2) lim Hn(2l,~) = 0 for every 2l, Q) E P(X). n-+oo

(3) lim Hjin (2l x Q)) = HJ1-xJ1-(2l x ~) for every 2l, ~ E P(X). n-+oo

Proof. (1) {:} (2). By Theorem 2 J-L is ergodic iff

lim iln(A x B)

n-+oo

= J-L(A)J-L(B),

A E 2l, B E

Q),

2l, Q) E P(X).

If we fix 2l, ~ E P(X), then the convergence above is uniform on A(2l x Q)) and

where il~'~ = ilnIA(2{X~), the restriction of iln to A(2l x ~). Thus by Lemma 13 (1) {:} (2) holds. (2) {:} (3) is clear since for n 2: 1 and 2l, Q) E P(X)

To characterize rnixing properties we let for n 2: 1 A,BEX,

90


Hn(2!,~) = H JLn(21

L

JLn(A

AE~,BE~

x~) = -

XB) log JL(~ X(:!, {L

L

{Ln(A X

21, ~ E P(X),

{L

B) log {Ln(A

X

B),

21, ~

E

P(X).

AE~,BE~

Then the following proposition is derived from Proposition 14 by replacing fIn and Hjln by u; and H JLn, respectively.

Proposition 15. For a stationary source J.L E Ps(X) the following statements are equivalent to each other: (1) J.L is strongly mixing. (2) lim Hn(2t,~) = 0 for every 21, ~ E P(X). n-+oo

(3) lim

n-+oo

H JLn (2t X ~) = H JL xl-£(2t X ~)

for every 21, ~ E P(X).

Finally, weak mixing is characterized as follows:

Proposition 16. For a stationary source J.L E Ps(X) the following statements are equivalent to each other: (1) J.L is weakly mixing. 1

(2) lim

n-l

I:

n-+oo n j=O

1

(3) lim n-+oo

n

Hj(2t,~) =

n-l

I:

HJLj

j=O

(21 x

0 for every 21, ~

~) = HJLxJL(21,~)

Proof. (1) ¢:> (2). For any 21, ~ E P(X) and n 1 2a

1

P(X).

E

for every 21, ~ E P(X). ~

1 it holds that

1 n-l.

2

L (A) J.L (B)·;:; L IJL(S-J A n B) - JL(A)JL(B) I AE~,BE~ J.L j=O

JL(A)JL(B)¥=O

1

0 there is an integer no 2 1 such that n 2 no, A E X.

Thus for A E X and P, q 2 1 we have that

Since J.lno is AMS we can choose an integer Po 2 1 such that the third term of the RHS of the above expression can be made < £ for p, q 2 Po. Consequently it follows that for p, q 2 Po, the LHS of the above expression is < 3£. Therefore, the limit in (4.1) exists for every A E X, so that J.l E Pa(X). (2) Let us set

Ma(X) == {aJ.l + f3rJ : a, f3

E

C, u; "l E Pa(X)},

Ms(X) == {aJ.l + (3"l : a, f3

E

C, u, "l E Ps(X)}.

Note that Ma(X) is the set of all measures J.l E M(X) for which the limit in (4.1) exists, and Ms(X) is the set of all S-invariant J.l E M(X). Define an operator 'I'o : Pa(X) -+ Ps(X) by 'I'oJ.l == ii, and extend it linearly to an operator 'I' on Ma(X) onto Ms(X). Then we see that 'I' is a bounded linear operator of norm 1, since Ma(X) is a norrn closed subspace of M(X) by (1). Hence (2) follows irnmediately.

93

2.4. AMS sources

Definition 4. Let J.L, 'TJ E P(X). That n asymptotically dominates J.L, denoted J.L ~ 'TJ, means that 'TJ(A) = 0 implies lim J.L(s-nA) = o. n-"7OO

The usual dominance implies the asymptotic dominance in the sense that, if J.L E P(X), 'TJ E Ps(X) and J.L ~ rJ, then J.L ~ rJ. In fact, if rJ(A) = 0, then rJ(s-n A) = rJ(A) = 0, which implies that J.L(s-nA) = 0 by J.L ~ rJ for every n E N. Thus lim J.L(s-nA) = 0, implying J.L ~ rJ. Although ~ is not transitive, one has that if n-"7OO a a a J.L« ~«rJ or J.L~~ ~ rJ, then J.L~'TJ. After the next lemma, we characterize AMS sources.

Lemma 5. Let J.L,'TJ E P(X) and for n 2: 0 let SnJ.L = (SnJ.L)a + (SnJ.L)s be the Lebesgue decomposition of SnJ.L w.r.t. n, where (SnJ.L)a is the absolutely continuous part and (snp,)s is the singular part. If In = d(S;:)a is the Radon-Nikodsim a

derivative and J.L« 'TJ, then it holds that lim { sup.IJ.L(s-n A) n-"7OO

AEX

{ fn

JA

drJl} = o.

(4.2)

Proof. For each n = 0,1,2, ... let B n E X be such that rJ(Bn) = 0 and AEX.

Let B

= n=O UBn .

Then we see that rJ(B) = 0 and for any A E X

o < p,(s-n A) -

L

fn d17 == p,(s-n(A n B n ))

::; J.L(s-n(A n B)) as n -+

ex)

< J.L(s-nB) -+ 0

by J.L~rJ. Taking the supremum over A E X, we have (4.2).

Theorem 6. For J.L E P(X), the following conditions are equivalent: (1) J.L E Pa(X), i.e., J.L is AMB. (2) There exists some rJ E Ps(X) such that J.L ~ rl·

(3) There exists some rJ E Ps(X) such that rl(A) =0 and A E Xoo ==

en

n=O

s-nx

imply J.L(A) = O. (4) There exists some rJ E Ps(X) such that rJ(A) = 0 and S-lA = A imply J.L(A) = O.


94

(5) lim Snf = fs J-t-a.e. for f E B(X), where fs is an S-invariant function. n-+oo (6) lim J-t(Snf) exists for f E B(X). n-+oo If one (and hence all) of the above conditions holds, then the stationary mean ji of J-t satisfies: (4.3) f E B(X). ji(f) = lim J-t(Snf) = J-t(fs), n-+oo

Proof. (1) =} (2). Suppose (1) is true and let rJ = ji E Ps(X). Assume ji(A) = 0 and let B = lim sup s:» A = S-k A. Then n-+oo n=l k=n

nu

ji(B) = lim n-+oo

ii( k=n U S-k A) ~ ii( k= U S-k A) 1

00

::; L ii(S-k A) = 0 k=l since ji(S-k A) = ji(A) = 0 for every k 2:: 1. This implies J-t(B) S-invariant. Now we see that lim sup JL(s-n A) ::; JL( lim sup s:»A) n-+oo n-+oo

= 0 since B

is clearly

= JL(B) = 0

by Fatou's lemma. Thus lim J-t(s-nA) = 0, and therefore (2) holds. n-+oo (2) =} (3). Assume that rJ E Ps(X) satisfies J-t~rJ. Take any A E oXoo such that rJ(A) = 0 and find a sequence {An}~l ~ oX with S':" An = A for n 2:: 1. It then follows from Lemma 5 that

where

In =

d(S;:;)a. Since rJ is stationary and '11(A)

= 0, we have

so that J-t(A) = O. Thus (3) holds. (3) =} (4) is immediate since A E oXoo if A E oX is S~invariant. (4) =} (2). Let rJ E Ps(X) satisfy the condition in (4). Take an A E oX with rJ(A) = 0 and let B = lim sup S':" A. Then we see that B is S-invariant and rJ(B) = n-+oo O. That lim J-t(s-nA) = 0 can be shown in the same fashion as in the proof of n-+oo (1) =} (2) above.

95

2.4. AMS sources

(2) => (5). Suppose that J.L ~ 'TJ with r/ E Ps(X), and let f E B(X) be arbitrary. Then the set A == {x EX: (Snf)(x) converges} is S-invariant and 'TJ(A) == 1 by the Pointwise Ergodic Theorern. Thus, that AC is S-invariant and 'TJ(AC) == 0 imply lim J.L(s-nAC) == J.L(AC) == 0 by J.L ~ 'TJ. Consequently, J.L(A) == 1, and lim Snf == fs n-+oo exists and is S-invariant u-a.e. (5) => (6). Let f E B(X) and observe that {Snf}~=l is a bounded sequence in B(X) ~ L 1(X, J.L) such that Snf -+ fs u-a.e. by (5). Then the Bounded Convergence Theorem implies that J.L(Snf) -+ J.L(fs). (6) => (1). We only have to take f == 1A in (6). The equality (4.3) is almost clear. n-+oo

a

Remark 7. (1) Note that J.L« Ii holds for J.L E Pa(X). This follows from the proof of (1) => (2) in the above theorem. (2) If J.L E P(X) and J.L «: 'TJ for some 'TJ E Ps(X), then J.L is AMS. (3) The Pointwise Ergodic Theorem holds for J.L E P(X) iff J.L E Pa(X). More precisely, the following staternents are equivalent for J.L E P(X): (i) J.L E Pa(X). (ii) For any f E B(X) there exists some S-invariant function fs E B(X) such that Snf -+ fs u-a.e. In this case, for f E L 1 (X ,ii), Snf -+ fs ti-a.e., ii-a.e. and in L 1 (X ,ii), and fs == Ej.£(fIJ) == Eji(fIJ) u-a.e. and ii-a.e., where J == {A EX: s:' A == A} is a a-algebra. (4) In (5) and (6) of Theorenl 6, B(X) can be replaced by C(X). (5) Pa(X) is weak* compact with the identification of Pa(X) c M(X) == C(X)*. When S is invertible, we can have some more characterizations of AMS sources. Proposition 8. Suppose that S is invertible. Then, for J.L E P(X) the following conditions are equivalent: (1) J.L E Pa(X). (2) There exists some 'TJ E Ps(X) such that J.L «: 'TJ. (3) There exists some 'TJ E Pa (X) such that J.L «: 'TJ.

Proof. (1) => (2). Let 11 == ii E Ps(X). Since S is invertible, we have SX == X == S-lX and hence Xoo == s-nx == X. Thus the implication (1) => (3) in Theorem 6 n=O implies J.L «: 'TJ == ii. (2) => (3) is immediate. (3) => (1). If'TJ E Pa(X), then 11 «: fj by the proof of (1) => (2). Hence, if J.L «: 'TJ,

n


96 a

then J-t ~ fj. This implies J-t ~ fj as was remarked in the paragraph before Lemma 5. Thus Theorem 6 concludes that (1) is true. Remark 9. In the case where S is invertible, note that the asymptotic dominance implies the usual dominance in the sense that, if J-t E P(X), 'fJ E Ps(X) and J-t ~ n, then J-t «'fJ. This immediately follows from the above proof. Also note again that J-t ~ ji for J-t E Pa(X). Example 10. (1) Take a stationary source 'fJ E Ps(X). By Remark 7 (2), any J-t E P(X) such that J-t ~ 'fJ is AMS. Hence, if I E L1(X, 'fJ) is nonnegative with norm 1, then J-t defined by

JL(A) =

L

AEX

f dn,

is AMS. In this case the stationary rnean ji of J-t is given by 1 ji(A) == lim n-+oo n

n-l

L J-t(S-k A)

k=O n-l

=

.!. L f f d'f/ n-+oo n k=O JS-k A lim

n-l

== lim n-+oo

==

= where

Is

== lim

n-+oo

particular, if J-t

~

j

jA .!.n k=O L f(Skx) 'f/(dx)

lim Snl d'fJ

A n -+ oo

L

fsd'f/,

A E X,

(4.4)

Snl, by the Pointwise Ergodic Theorem since 'fJ is stationary. In ii, then

where Jji == {A EX: ji(S-l A~A) == O}. (2) Take a stationary source 'fJ E Ps(X) and a set B E X such that 'fJ(B) Then, J-t defined by J-t

n B) == (AlB) (A) == 'fJ(A 'fJ(B) 'fJ ,

> O.

2.4. AMS sources

97

is AMS since fL «: "l. In fact, this is a special case of (1) since fL(A) = fA 71(B) d"l for A E X. Similarly, take an AMS source "l E Pa(X) with the stationary mean fj and a set B E X with "l(B) > O. Then the conditional probability fL(') = "l(·IB) of"l given B is also AMS. For, if fj(A) = 0, then "l(B)fL(limsupS- nA) 5: "l(lirnsupS- n A) = 0 n-too

n-too

by tt ~ fj. Hence lirn fL(s-n A) = O. This means that fL ~ fj. Theorem 6 deduce that n-too fL is AMS. It follows that AMS property remains in conditioning while stationarity is lost by conditioning, in general.

DefinUion 11. ~~~~~~~~~~~~~~~~~~~fL(A) =Oor 1 for every S-invariant AMS sources. We have several equivalence conditions of ergodicity for an AMS source.

Theorem 12. For fL E Pa(X) with the stationary mean Ji E Ps(X) the following conditions are equivalent: (1) fL E .Pae(X). (2) Ji E Pse(X).

(3) There exists some "l E Pse(X) such that fL ~ "l.

(4) Is(x) == lim (Snl)(x) = ( I dji Ii-a.e. and u-a.e. lor I

l»

n-too

E

L 1(X,ji).

(5) n-too lim ( s, f, g) 2 ,11- = (f, 1)2 ' ti (1, g) 2 ,11- for t, g E L 2 (X, fL) n L 2 (X, Ji). (6) lim fL((Snf)g) = Ji(f)fL(g) for every f,g

E B(X).

(7) lim fL((Snf)g) = Ji(f)fL(g) for every t,s

E C(X).

n-+oo n-+oo

1

n-l

n

k=O

1

n-l

(8) lim n-+oo

(9) lim -

L

L

n-+oo n k=O

fL(S-k A n B) = Ji(A)fL(B) for every A, B E X. fL(S-k A n A) = Ji(A)fL(A) for every A E X.

Proof. (1) {::} (2) is clear from the definition. (1), (2) =? (3) follows from Remark 7 (1) by taking "l = ii.

be such that fL ~ n. If A E X is S-invariant, then "l(A) = 0 or 1. If "l(A) = 0, then fL(A) = fL(s-n A) ~ 0 by fL ~ 'fJ, i.e., fL(A) = O. Sirnilarly, if "l(A) = 1, then we have fL(A) = 1. Thus fL E Pae(X). The implications (1) =? (4) =? (5) =? (6) =? (7) =? (8) =? (9) =? (1) are shown in much the same way as in the proof of Theorem 3.2.

(3)

=?

(1). Let "l

E Pse(X)


98

Remark 13. In (8) and (9) of Theorem 12, X can be replaced by a semialgebra Xo that generates X. Also in (5), (6) and (7) of Theorem 12, we can take 9 = f. Theorem 14. (1) If J-L E exPa(X), then J-L E Pae(X). That is, exPa(X) ~ Pae(X). (2) If Pse(X) ¥- 0, then the above set inclusion is proper. That is, there is a J-L E Pae(X) such that J-L ~ exPa(X). Proof. (1) This can be verified in exactly the same manner as in the proof of (4) => (1) of Theorem 3.2. (2) Let J-L E Pae(X) be such that J-L ¥- Ii. The existence of such a J-L is seen as follows. Take any stationary and ergodic ~ E Pse(X) (¥- 0) and any nonnegative f E Ll(X,~) with norm 1 which is not S-invariant on a set of positive ~ measure. Define J-L by

JL(A) =

1i d~,

AEX.

We see that J-L is AMS by Example 10 (1) and ergodic because ~ is so. Clearly J-L is not stationary. Hence J-L ¥- Ii. Also note that {i = ~ since for A E X

(i(A) =

1is d~

= ~(A)

by (4.4) andfs = 1 ~-a.e. because of the ergodicity of~. Then 'T/ = ~(J-L + {i) is a proper convex combination of two distinct AMS sources and 'T/(A) = 0 or 1 for S-invariant A E X. Thus 'T/ ~ exPa(X) and 'T/ E Pae(X). Again, if S is invertible, ergodicity of AMS sources is characterized as follows, which is similar to Proposition 8: Proposition 15. If S is invertible, then the following conditions are equivalent for

Pa(X): (1) J-L E Pae(X). (2) There exists some 'T/ (3) There exists some 'T/

J-L E

(4) There exists some 'T/

Pse(X) such that Pae(X) such that

J-L ~ 'T/.

E E

Pae(X) such that

J-L ~ 'TJ.

E

J-L

«

'TJ.

Proof. (1) => (2). Take 'T/ = {i E Pse(X), then J-L ~ {i = 'TJ by Remark 9. (2) => (3) is clear. (3) => (4). Let 'TJ E Pae(X) be such that J-L ~ 'TJ. Then fj E Pse(X) and 'TJ ~ fj. a Hence J-L « fj and J-L ~ fj since fj is stationary.

(4) => (1). Let fJ E Pae(X) be such that J-L~'TJ. Then 'TJ' ~ fj and J-L~fj. Since Pse(X), Theorem 12 concludes the proof.

fj E

99

2.5. Shannon-McMillan-Breiman Theorem

Ergodicity and mixing properties may be defined for nonstationary sources. Let tt E P(X). tt is said to be ergodic if tt(A) = 0 or 1 for every S-invariant A E X. tt is said to be weakly mixing if n-l

! '" IJL(S-jA n B) n-+oo n L...J lim

I = 0,

JL(S-jA)JL(B)

A,BEX

j=O

and strongly mixing if A,BEX.

Clearly, these definitions are consistent with the ones for stationary sources and it holds that strong mixing implies both weak mixing and ergodicity. We can show that if tt is AM8 and weakly mixing, then tt is ergodic. In fact, the condition (8) or (9) of Theorem 12 may be easily verified.

2.5. Shannon-McMillan-Breiman Theorem An ergodic theorem in information theory, the so-called 8hannon-McMillanBreiman Theorem (8MB Theorem), is proved in this section. First we briefly describe practical interpretation of entropy (or information) in an alphabet message space. Then we formulate the 8MB Theorem in a general setting. Let us consider an alphabet message space (X, X, S), where X = X~, X o = {aI, ... ,at} and S is a shift. Let tt be a stationary information source, where we sometimes denote it by [X, ttl. For an integer n ;::: 1, ootn denotes the set of all messages of length n of the form [X~k)

••• x~k~ll,

Le., the messages of length n starting at time O. Note that ootn is a finite partition -

of X, i.e., ootn E P(X), and mtn

=

n-l

.

.V S-Joot1 . Hence the entropy of

J=O

ootn under the

measure tt is given by

where we write the dependence of the entropy on u. If oot~ denotes the set of all messages of length n starting at time i = ±1, ±2, ... ,

100


then the entropy of VJt~ is the same as that of VJtn VJt~ since f.-L is stationary. Hence the information (or entropy) per letter (or symbol) in messages of length n input from the stationary information source [X, f.-L] is

and if we let n -t

00,

then the limit

exists by Lemma 1.3.5, which represents tt!&~~~.!!!!Q!1~ll21~~l~~J2!J!!~ information source This is a interpretation Kolmogorov-Sinai entropy of the shift S, 00 denoted H(f.-L, S), is equal to H(f.-L) by Theorem 1.3.10 since V snVJt1 = X'. n=-oo

To formulate the 5MB Theorem we are concerned with entropy functions. Let (X, X, S) be an abstract measurable space with a measurable transformation Sand f.-L E Ps(X). Recall that for a finite partition 2t E P(X) and a a-subalgebra q) of X', the entropy function 1p,(2t) of 2t and the conditional entropy function 1p,(2t1q)) are defined respectively by Ip,(2t)(·) =

E 1A(·) log f.-L(A) , AEQ(

1p,(2t1q))(·) = -

E 1A(·)logPp,(AIq))(·), AEQ(

where Pp,(·Iq)) is the conditional probability relative to q) under the measure f.-L (cf. Section 1.3). These functions enjoy the following properties: for 2t,~,

,

(5.2)

1x9dP,.

If, in particular, Jl is ergodic, then

Ix

h = H(p" 21, S)

Proof. Since

9 dp, u-a. e.

V S-k21T k=l V S-k21 we can apply Lemma 1.3.9 (1) to obtain

k=l

gn

--+ 9 u-a.e. and in L 1 .

(5.3)

It follows from the Pointwise Ergodic Theorem (Theorem 2.1) that there exists an h E L 1(X, Jl) such that Sh = h u-a.e. and 1 n-1

-n E sn-k-1 g = Sng -+ h

Jl-a.e. and in L 1.

(5.4)

k=O

Now observe that

II ~ In - hll l ,1J = 1/ ~ ~sn-k~lgk - hl/

,1J l

~ II~ E(sn-k-l gk -

sn-k-lg)11

: :; ~ E

II ~

k=O

IIgk

glll,1J +

k=O

-+0

1,1-£

I:

+ II~ Esn-k-l g - hll k=O

sn-k-l g -

k=O

1,1-£

hll 1,1-£

asn-+oo

by (5.3) and (5.4). This establishes the L1-convergence. To prove u-a.e. convergence we proceed as follows. We have frorn the above computation that

~ In - hi < ~ Esn-k-ll gk -

I

k=O

gl +

IE k=O

sn-k-l g -

hi.·

103


The second term tends to 0 J-L-a.e. by the Pointwise Ergodic Theorem. So we have to show 1 n-l (5.5) lim sup sn-k- 119k - 91 = 0 u-a.e.

E

n k=O

n-+oo

For N = 1,2, ... let G N = sup 19k k~N

91·

Then, GN 4.- 0 u-a.e. since 9k -+ 9 u-a.e., and

o::; Go ::; sup 9n + 9 E L 1(X, J-L) n~l

by Lemma 1.3.8. Let N 2 1 be fixed. If n

~

I:

sn-k-1I gk -

> N, then

(I: + I:) < .! I: sn-k-1Go+.! ~

gl =

k=O

~

sn-k-1I gk -

k=O

n k=O

=

~

gl

k=N

sn-k-1G N

n k=N

N-l

n-N-l

k=O

J=o

""' sn-k- 1 G o + n - N . _1__ nL.-J n n-N

""' SiGN. ~

Letting n -+ 00, we see that the first terrn on the RHS tends to 0 J-L-a.e. and the second term converges to some S-invariant function GN,S E L 1(X, J-L) J-L-a.e. by the Pointwise Ergodic Theorem. Thus we have 1 n-l lim sup sn-k- 119k -

E

n-+oo

Finally, note that G N,S 4.- as N -+

L

91::; GN,S

u-a.e.

n k=O

GN,sd/L

=

00

and by the Monotone Convergence Theorem

L

GNd/L -+ 0

as N -+

00

since G N 4.- 0, which implies that GN,S -+ 0 p-a.e. Therefore (5.5) holds and J-L-a.e. convergence is obtained. (5.2) is clear from (10) above and (5.1). Corollary 2. Let X = X~ be an alphabet rnessage space with a shift Sand J-L E Ps (X) be a stationary source. Then there exists an S-invariant junction h E L1(X, J-L) such that lim

n-+oo

{_.!n

"'" L.-J

XO, •••

,xn-lEXo

1["'0"'''' n -d log /L([xo ... xn-1J)} = h

u-a.e. and in L

1

.


104

If, in particular, J1 is ergodic, then h == H (J1, S) J1-a. e. Proof. Take 2t == rot 1 E P(X) and observe that for n

~

1

i« == t; (nvl S-k2t) == Ip,(rotn), k=O

== -

L

1A log J1(A)

AEVJt n

L

l[xo···x n - d

log J1([xo ... Xn-l])'

xo,··· ,xn-1EXo

Thus the corollary follows from Theorem 1. The following corollary is sometimes called an entropy equipartition property.

Corollary 3. Let X == X~ be an alphabet message space with a shift Sand J1 E

Pse(X) be a stationary ergodic source. For any c > 0 and 8 > 0 there is an integer no

~

1 such that

Hence, for n ~ no, the set rotn of messages of length n starting at time 0 can be divided into two disjoint subsets rotn,g and VJtn,b such that (1) e~nH(p"S)-c < J1(M) < e-nH(p"S)+c for every message M E VJtn,g.

(2)

JL(

U MEVJtn,g

(3)

JL(

U MEVJtn,b

M) ~ 1 M) < O.

s.

Proof. Since a.e. convergence implies convergence in probability the corollary follows immediately from Theorem 1. Remark 4. (1) In Corollary 3, for a large enough n (n ~ no) each message M E VJtn,g has a probability approximately equal to e-nH(p"S) and hence the number of messages in VJtn,g is approximately enH(p"S). Since the number of messages in VJt n is /!n == enlogl and H(J1,S) S log z, it holds that IVJtn,gl ~ IVJtn,bl, i.e., the number of elements in VJtn,g is much larger than that of those in VJtn,b' (2) Another consequence of Corollary 3 is that if we receive a message long enough, then the entropy per letter in the message almost equals the entropy of the information source. A generalized version of Theorem 1 is formulated as follows:


105

Corollary 5. Let (X, X) be a measurable space with a measurable transformation S : X -+ X. Let J-L E Ps(X) be a stationary source and 2{ E P(X) be a finite partition of X. Assume that V is a a-subalqebra of X such that S-lV = V and let

In = I~ (:~~ S-k2lI!D), 90

= I~(2lI!D), 9n = I~ (2l1 k21 S-k2i V!D),

9

= I~ (2l1 k~l S-k2i V!D)

for n ~ 1. Then there exists an S-invariant function h E L 1(X, J-L) such that

lirn

n-+oo

If, in particular,

!n I«

1 lim -11£

n-+oo

n

(n-1V S-k2{I) V = h k=O

V ~ 2loo == V S-k21, k=l

H (p"

u-a.e. and in L 1 .

(5.6)

then

2l, S) =

L

h du.

(5.7)

Proof. (5.6) can be verified in a similar rnanner as in the proof of Theorem 1. As to (5.7) assurne that V ~ 2loo . Then we have

H(J-L, 2{, S) = lirn HJ.£(QtI2ln) , n-+oo

where

2{n =

.\1

J=l

s-jQt and

=

HJ.£(QtI21oo )

=

HJ.£(QtI21oo

2tn

=

V

V),

by Lemma 1.3.5,

byassurnption,

u(Qtn ) as before. Thus we get

Since the function h in Corollary 5 depends on J-t E Ps(X) we should denote it by

hJ.£' so that H(p"

2l, S) =

Ln,

dp"

(5.8)

106


This will be applied to obtain an integral representation of the entropy functional in Section 2.7.

2.6. Ergodic decompositions In this section, ergodic decomposition of a stationary source is studied. Roughly speaking, if a measurable space (X, X, S) with a rneasurable transformation S : X -+ X is given, then there is a measurable family {JLx}xEX of stationary ergodic sources, which does not depend on any stationary source, such that each stationary source JL is written as a mixture of this family:

J.t(A) =

L

AEX.

J.tx(A) J.t(dx),

This is, in fact, a consequence of the Krein-Milman Theorem: P, (X) = co Ps e (X) (cf. Section 2.1). Our setting here is that X is a compact metric space, so that X is the Baire (= Borel) a-algebra and the Banach space C(X) is separable. Let {fn}~=l c C(X) be a fixed countable 'dense subset. S is a measurable transformation on X into X as before. Recall the notation Sn (n 2: 1): for any function f on X 1

(Snf)(x) = n

n-l

.

E f(SJ x),

xEX.

j=O

Definition 1. For n 2: 1 and x E X consider a functional Mn,x on B(X) given by

Mn,x(f) = (Snf)(x),

f

E

B(X).

A point x E X is said to be quasi-regular, denoted x E Q, if

exists for every f E C(X). A measurable set A E X is said to have invariant measure 1 if JL(A) = 1 for every JL E Ps(X).

Lemma 2. For each quasi-regular point x E Q there is a unique stationary source JLx E Ps(X) such that JLsx = JLx and

Mx(j) =

L

fey) J.tx(dy),

f

E

C(X).

(6.1)

107

2.6. Ergodic decompositions

Moreover, Q is an S-invariant Baire set of invariant measure 1. Proof. Let x E Q. It is not hard to see that M x ( · ) is a positive linear functional of norm 1 on C(X) since

IIMn,x(·)11 =

sup IMn,x(f)1 IIfll~1

sup I(Snf)(x)1 ~ sup

=

IIfll~1

IIfll~1

Ilfll =

1

and Snf = f for 8-invariant f E C(X). Thus, by Riesz-Markov-Kakutani Theorem there exists a unique source J-Lx E P(X) such that (6.1) holds. To see that J-Lx is stationary, we note that Mx(Sf) = Mx(f) for f E C(X). Hence for f E C(X)

L

f(y) /l-x(dy) = Mx(f) = Mx(Sf)

=

L

f(Sy) /l-x(dy) =

L

f(y) /l-x(dS-1y).

This implies that J-Lx = /-Lx 0 8- 1 or J-Lx is stationary. That J-Lsx = J-Lx is also derived from Msx(f) = Mx(Sf) = Mx(f) for f E C(X). Let {fk}k=1 C C(X) be dense and

Ak =

{x EX:

lim (Snfk)(X) = lim M n x(fk) exists},

n--+-oo

n--+-oo'

k 2:: 1.

00

Then we note that Q = n A k since {fk} is dense in C(X). Also note that A k is an

k=1

Fo-Ii set in X for k 2:: 1 and J-L(A k ) = 1 for every J-L E Ps(X) by the Pointwise Ergodic

Theorem. Now we see that Q

00

= n A k is a Baire set such that J-L(Q) = 1 for every k=1

J-L E Ps(X), or Q has invariant measure 1.

From now on we assume that {J-Lx}xEQ is a family of stationary sources obtained in Lemma 2. Denote by B(X, 8) the set of all 8-invariant functions in B(X). Clearly B(X,8) is a closed subspace of the Banach space B(X). For f E B(X) let

f~(x) = {.

L

f(y) /l-x(dy) ,

x

E

Q,

(6.2)

x¢:. Q.

0,

If f E C(X), then f~ E B(X) since f~(x) = Mx(f) for x E Q and Q E X. For a general f E B(X) choose a sequence {gn}~1 C C(X) such that lim f gn(Y) J-Lx(dy) n--+-oolx = lim Mx(gn) lim g~(x),

t,( f(y) J-Lx(dy) =

n--+-oo

n--+-oo

XEQ.

108


Hence

f Q E B(X). Now we have:

Lemma 3. If f E B(X), then f Q E B(X,B). The mapping A : f I---t jQ is a 1. projection of norm 1 from B(X) onto B(X,B), so that A2 == A and IIAII Moreover it holds that for J-L EPs(X) and f E B(X)

L f(x) p,(dx)

=

k

(L f(y) P"AdY)) p,(dx)

=

kf~

(x) p,(dx).

(6.3)

Proof. Since J-Lsx == J-Lx for x E Q by Lemma 2, we have that

f~(Sx) = L

f(y) p,sx(dy) = L f(y) P,x(dy)

= f~(x),

x

E

Q

and fQ(Bx) == 0 == fQ(x) for x t/. Q. Hence f Q E B(X, B). If f E B(X, B), then clearly f Q == f. Thus A is a projection of norm 1. To see (6.3) we proceed as follows. Let J-L E Ps(X). Then, since J-L(Q) == 1 by Lemma 2 we see that for f E C(X)

( f(x) J-L(dx) == ( lim (Snf)(x) J-L(dx) ,

Jx

Jx n-+oo

=

by the Pointwise Ergodic Theorem,

k

Mx(J) p,(dx)

= k(Lf(y)p,x(dY))P,(dX)

=

kf~(x)

p,(dx).

Let B == {f E B(X) : (6.3) holds for fl. Then B contains C(X) and is a monotone class, which is easily verified. Thus B == B(X) and (6.3) is true for every f E B(X). The following corollary is immediate. Corollary 4. Let J-L E Ps(X). Then, for j E L l (X , J-L),

f Q == Ep,(fIJ) u-a.e., where f Q is defined by (6.2), J == {A EX: B- 1 A == A} and Ep,('IJ) is the conditional expectation w.r.t. J under the measure J-L. Definition 5. A point x E Q is said to be regular if the corresponding stationary source J-Lx is ergodic, i.e., J-Lx E Pse(X). R denotes the set of all regular points. Let J-L* E Pse(X) be a fixed stationary ergodic source and define x

rt R.

109

2.6. Ergodic decompositions

The set {JLx} xEX is called an ergodic decomposition relative to S. We summarize our discussion in the following form.

Theorem 6. Let X be a compact metric space, X the Baire a-alqebra and S : X -+ X a measurable transformation. Then the set R of all regular points is an S-invariant Baire set. Let {JLX}XEX be an ergodic decomposition relative to S. Then, for any stationary source JL E P; (X) the following holds: (1) JL(R) = 1. (2) For f E L 1(X, JL) let f Q be defined by (6.2) and f#(x} = f(y) JLx(dy) if x E Rand = 0 otherwise. Then,

Ix

lim (Snf)(x)

n-+oo

t"

i

f(x) p,(dx) =

= n-+oo' lim M n x(f) = Ell-x (f) J-L-a.e.,

(6.4)

= f Q = EIl-(fIJ)

(6.5)

L

f(x) p,(dx) =

p-a.e.,

L(i

f(y) P,.,(dY))P,(dX).

(6.6)

In particular, if f = 1A (A E X), then

p,(A) =

L

(6.7)

p,.,(A) p,(dx).

Proof. Since JLsx(A) = JLx(A) for x E Q and A E X by Lemma 2, we see that R is S-invariant. Note that by Remark 3.3 (4) R

= {x

E Q: lim J-Lx((Snf)f) n-+oo

= J-Lx(f)2, f

= {x E Q: lim JLx((Snfk)fk) n-+oo

n 00

=

{x

E Q:

E C(X)}

= J-Lx(fk)2, k 2: I}

}~~P,.,(Snfk)!k) = P,.,Uk)2}.

k=l

Ix

Since for each f E C(X), JL(.)(f) = fQ(y) J-L(.) (dy) is X-measurable, R is an Xmeasurable set. Let JL E Ps(X). We only have to verify (1). Observe the following two-sided implications:

JL(R)

= 1 ~ J-Lx ~

E

Pse(X) u-a.e.x

lim (Snfk)(Y) = JLx(fk) J-Lx-a.e.y, u-a.e.x, k 2: 1

n-+oo

110

Chapter II: Information Sources 2

-¢==>

L Ifk,S(Y) - JLx(ik) 1 JLx(dy) = 0 p-a.e.x, k where fk , s(y)

-¢==>

Now for each k

~

~ 1,

= n--+oo lim (Snfk)(Y), Y E X, 2

JLx(fk) 1 JLx(dY))JL(dX) = 0, k

L (L Ifk,S(Y)

~ 1.

1 it holds that 2

L (Llfk'S(Y) - JLx(ik) 1 JLx(dy) )JL(dX)

= ~ (L Ifk,S(Y)1 JLx(dy) -IJLx(fk) 1 ) JL(dx) 2

2

= ~ E/L (Ifk,sl 2IJ) (x) JL(dx)

~ IL

ik dJLxr JL(dx)

= ~ lik,s(x) 1 JL(dx) -

~ If~(x)12 JL(dx)

= ~ Ifk,S(x)1 JL(dx) -

~ Ifk,S(x)1 2 JL(dx) ,

2

2

since f~(x) =0.

= Mx(!k) = !k,S(X) for x E Q,

We should remark that if (X, X) is an abstract measurable space with a countable generator, then for any measurable transformation S : X -+ X we can find an ergodic (6.7) hold. For a detailed decomposition {J-tX}XEX ~ Pse(X) for which (6.4) treatment see e.g. Gray [1].

2.7. Entropy functionals, revisited In Section 1.5 we considered an entropy functional H(·, 2(, S) on the set of all S-invariant C-valued measures on a measurable space (X, X), where 2( E P(X) is a fixed finite partition of X and S is an automorphism on X, and we derived an integral representation of H (., 2(, S) under certain conditions. In this section, we shall show an integral representation of the entropy functional by a universal function by two approaches: functional and measure theoretic ones. Also we shall extend the entropy functional to the space Ma(X) of all AMSmeasures. We begin with a functional approach using ergodic decompositions. Our setting is as follows. Let X be a totally disconnected compact Hausdorff space, S a fixed

111

2. 7. Entropy functionals, revisited

homeomorphism and X the Baire o-algebra, Since an alphabet message space X~ is compact and totally disconnected, the above conditions are fairly general. Take any clopen partition 2( E P(X) consisting of disjoint clopen sets in X. As in Section 1.5, we use the following notations: n

a, =

V s-j2( E P(X), j=l

for n 2: 1. P(X), Ps(X), M(X) and Ms(X) are as before. Since we write the entropy of J-L E Ps(X) relative to 2( and S by

2(

and S are fixed,

H(J-L) = H(J-L, 2(, S)

=-

E Jxr Pp(AIQ{oo) logPp(AIQ{oo) dp.

(7.1)

AE2{

= -

lim 1

n-too

n

'"'" L...J

J-L(A) log J-L(A) ,

AE2{V2{n-l

where Pp,(AI~) is the conditional probability of A relative to a zr-subalgebra ~ of X under the probability measure J-L. To reach our goal we need several lemmas.

Lemma 1. If J-L, 'fJ E Ps(X) and 'fJ« J-L, then for each A E

2(

(7.2) Proof. Let ~ .=

0- ( { sj 2( :

j E Z}). Since ~ has a countable generator, for each

J-L E P, (X), every S- invariant B E ~ belongs to 2100 (mod J-L), i.e., there exists a set B' E 2100 such that J-L(BllB') = O. Let J-L' = J-LI~, the restriction of J-L to ~ for each J-L E Ps(X). Then, J-L' is S-invariant and AE2(

on Cp" the support of J-L. S-invariance of Cp, implies that Cp, E 2l00 • Now let J-L,'fJ E Ps(X) be such that 'fJ ~ J-L. Then, 'fJ'« J-L' and ~ is S-invariant and ~-measurable, hence 2loo -measurable (mod u). Thus, we have that for A E 2{ and B E 2100

112


so that (7.2) is true. Lemma 2. Let J-t E Ps(X). Then there is a bounded, upper semicontinuous and 2t00 -measurable function hl-£ such that

hl-£ == -

L

PI-£(AI2too ) 10gPI-£(AI2t00 ) J-t-a.e.,

(7.3)

AE~

(7.4)

Proof. Note that PI-£(AI2tn) E C(X) for A E 2t. since each B E 2t.n is clopen. For n 2: 1 let (7.5) hl-£,n == PI-£(AI2tn) 10gPI-£(AI2tn)

L

AE~

and observe that hl-£,n E C(X) and hl-£,n 4- by Jensen's Inequality (cf. (12) in Section 1.2). If we let hl-£ == n~oo lim hl-£ ' n, then hl-£ is upper semicontinuous since each hl-£ ,n is continuous, and is 2too-measurable since each hl-£,n is 2tn-measurable. Moreover, {hl-£,n}~=l forms a submartingale relative to {2tn} on (X, X, J-t) and (7.3) is obtained by the Submartingale Convergence Theorern (cf. (25) in Section 1.2).(7.4) follows from (7.3) and (7.1). Lemma 3. The functional H(·) on Ps(X) is weak* upper semicontinuous, where we identify Ps(X) c M(X) == C(X)*.

Proof. For J-t E Ps(X) and n 2: 1 let Hn(p,)

=

L

hp"n dp"

where hl-£,n is defined by (7.5) in the proof of Lemma 2. Since Hn(J-t) -+ H(J-t) as n -+ ex> for each J-t E Ps(X), it suffices to show the weak* continuity of H n(·) on Ps(X), which immediately follows from that Hn(p,)

===

L { lAlogPp,(AI2ln)dp,

AE~JX

L L

AE~BE~n

{J-t(A n B) log J-t(E) - J-t(A n B) log J-t(A n B)}

2. 7. Entropy junctionals, revisited

113

and that A E 2t. and B E 2t.n are clopen. In order to use the ergodic decomposition of 8-invariant probability measures developed in the previous section, we need to introduce a metric structure in X. 00 . Denote by Q3 0 the algebra of clopen sets generated by . u 8 J 2t. and let Q3 = a(Q3o), J=-OO

the a-algebra generated by Q3o. In the Banach space C(X), let C(X, 2t.) be the closed subspace spanned by {IA : A E ~o} ~ C(X). Note that C(X,2t.) has a countable dense subset {jn}~=l since Q3 0 is countable. Define a quasi-metric d on X by x,yEX

and an equivalence relation X

Then the quotient space by

rv

rv

y

by ¢:=>

d(x, y) = 0,

x,yE X.

X = X/ rv becomes a metric space d(x, ii)

= d(x, y),

with the metric d defined

x,yE X,

where x {z EX: Z rv X }, the equivalence class containing x. Moreover, (X,d) is a compact metric space, the canonical mapping x I-t x from X onto X is continuous, and C(X,2t.) is isometrically isomorphic to C(X) with an isometric isomorphism C(X) 3 j I-t j E C(X), where j(x) = j(y),

Hence ~ = {B : B E Q3} with mapping § on X given by

B = {x : x

ss =

S»,

x E X,y E E

x.

B} is the Baire a-algebra of

X.

The

xEX

is well-defined and a homeomorphism. Therefore the triple (X,~, 8) consists of a compact metric space X with the Baire a-algebra ~ and a homeomorphism 8.

Lemma 4. For a positive linear junctional ;\ on C(X,2t.) oj norm 1 there is a probability measure J-LA E P(X, Q3) such that j E C(X,2t.),

where P(X, Q3) is the set of all probability measures on (X, Q3). Moreover, if ;\ is S-invariant, i.e., ;\(8j) = ;\(j) for j E C(X,2t.), then J-LA is S-invariant.

114


Proof. Let A(f) = A(f) for f E C(X,21). Then.:x is a positive linear functional of norm 1 on C(X), and hence there is a probability measure J.LX on (X, ~) such that

i E C(X). By letting J.L>.(B) = J-tx (iJ) for B E 23, we have the desired measure J-t>.. Recall some notations and terminologies from the previous section. For x E X, f E C(X) and n ~ 1 we denote n-l

= (Snf)(x) = ~ E f(Si x) = Mn,x(J).

Mn,x(f)

j=O

Let

Qand R be the sets of quasi-regular and regular points in X, respectively.

Then

Q and R have invariant measure 1, i.e., jj(Q) = jj(R) = 1 for jj E Ps(X) by Lemma 6.2 and Theorem 6.6. If Q = {x EX: x E Q} and R = {x EX: x E R}, then Q and R have invariant measure 1, i.e., J-t(Q) = J-t(R) = 1 for J-t E Ps(X), where Q and R are called the sets of quasi-regular and regular points in X relative to C(X,21), respectively. Thus, for each x E Q, Mx(f) = lim M n x(f) exists for n-+oo ' f E C(X, 21), M x ( ' ) is a positive linear functional of norm 1 on C(X, 21), and there is an S-invariant probability measure J-tx = tiu; on (X,23) such that Mx(f)

=

L

f

f(y) /lx(dy),

E C(X,

21).

Lemma 5. For a bounded 23-measurable function f on X

f~(r) =

L

f(x) /lr(dx) ,

r E

R,j

E

C(X, 21)

is a bounded, 23-measurable and S-invariant function on R and satisfies that

for every J.L E Ps(X). Proof. Let f be bounded and 23-rneasurable on X. By Theorem 6.6 we see that the function 9 on R defined by

9(1')

=

fx lUi;)

/If(dx)

= f~(r),

r ER

115

2.7. Entropy functionals, revisited

is ~- measurable and satisfies

k

g(1') {t(d1') =

S-invariance of

fx

j(i) {t(di),

f Q follows from that of /-Lx for x

E

Q.

Under these preparations, we are now able to prove the integral representation of the entropy functional.

Theorem 6. Let X be a totally disconnected compact Hausdorff space with the Baire cr-algebra X and a homeomorphism S. If 2( E P(X) is a clopen partition of X, then the entropy functional H (.) = H (', 2(, S) on P, (X) has an integral representation with a universal bounded nonnegative S-invariant Baire function h on X:

H(p,) =

Ix

(7.6)

h(x) p,(dx) ,

h(x) = hp.(x) u-a.e.,

/-L E

(7.7)

Ps(X),

where hl-£ is given by (7.3) and h is unique in Ps(X)-a.e. sense. Proof. Let us denote by Ps(X,~) the set of all S-invariant measures in P(X, ~). Define a function h on X by if r E R, if r rJ. R. Clearly h is nonnegative. We shall show that h is bounded, S-invariant and ~ rneasurable. Note that each hl-£r '(r E R) is obtained by the limit of {hl-£r,n}~=l (cf. (7.5)) as was seen in the proof of Lemma 2: hl-£r,n 4- hl-£r' Let

gn(r)

=

Ix

hl-'r>n(x) P,r(dx)

L L

/-Lr(AnB){log/-Lr(AnB)-log/-Lr(B)}

AEQlBE~n

for r E R. Since /-Lr(O) = Mr(lc) (0 E ~o) is a ~-measurable function of r on R, gn(') is also ~-measurable on R. Hence h is ~-measurable on R since hl-£r,n 4- hl-£r and h(r) = f hl-'r (x) P,r(dx) = lim gn(r), r E R.

l»

n-+oo

This and the definition of h imply that h is bounded and S-invariance of h follows from that of /-Lr'

~-measurable

on X.

116


To show (7.6) let A E 21, B E that on one hand

'I7(A n B)

=

l

=

l (l

2Ioo

and J-t,1] E Ps(X) with

P'1(Alsi(",) d'17 =

l

1]

«: u. Then it holds

PJ.'(AI2too ) d'l7,

by Lemma 1,

PI'(AI2too)(x) J-tr(dX)) 'I7(dr),

by Lemma 5,

and on the other hand

'I7(A n B) = = =

L

1AnB d'17

l l(l

= ll~nB d'17

J-tr(A n B) '17 (dr) ,

by Lemma 5,

Pl'r(A I2too)(X)J-tr(dX)) 'I7(dr).

It follows from these equalities and S-invarianceof J-tr (r E R) that for each A E 21

(7.8) J-tr-a.e. for u-a.e.r, We can now derive (7.6) as follows: for J-t E Ps(X)

H(J-t)

= = = = =

L =l l (L l (L l L hJ.'dJ-t

ht(r) J-t(dr) ,

by Lemma 5,

hl'(x) J-tr(dX)) J-t(dr), hl'r(x) J-tr(dX)) J-t(dr),

by Lemma 5, by (7.8),

h(r) J-t(dr) ,

by the definition of h,

h(x) J-t(dx) ,

by the definition of h.

As to (7.7), let u; rJ E Ps(X) be such that rJ «J-t. Then by Lemmas 1,2 and (7.6)

H('I7)

=

L =L L h'1 d'17

hl'd'l7=

h dn,

and hence for every S-invariant f E L 1(X, J-t)

L

h(x)f(x) J-t(dx)

=

L

hl'(x)f(x) J-t(dx).

117

2.7. Entropy functionals, revisited

Since hand hJ.t are S-invariant, (7.7) holds. The function h obtained above is called a universal entropy function associated with the clopen partition 2l and the homeomorphism S. Of course the functional H(·) is extended to Ms(X) with the same universal function h. Next we formulate an integral representation of the entropy functional by a measure theoretic method. So let (X, X, S) be an abstract measurable space with a measurable transformation S : X -+ X.

Theorem 7. Let 2l E P(X) be a fixed partition. Then there exists an S-invariant nonnegative measurable function h on X such that

(7.9)

(7.10)

h = hJ.t j.l-a.e., where hJ.t is given by (5.8). Proof. With the notation of 2ln

=

.v S-j2l,2{n = a(2l

n)

J=1

and 2{00

= a(

U 2{n) let

n=1

and observe that ~ is a a-subalgebra with S-1~ = ~ and ~ ~ 2{00. Hence by Corollary 5.5, for each j.l E Ps(X), there exists a function hJ.t such that

Let

J-L E

Ps(X) be fixed. Then for any A E 2l V 2ln -

1

we have that

1 k-1 . PJt(AI!D)(x) = lim sup k lA(SJ X ) p,-a.e.

L

k~oo

since the RHS is

~-rneasurable

j=O

and for any B E

~

k-l

k-1

f lim sup ~ L lA(Si x) p,(dx) = f lim sup ~ L IB(x)lA(Si x) p,(dx) JB k-+oo j=O Jx k-e-co j=O = f

lim sup ~

Jx k~oo

= j.l(AnB),

k-1

L lAnB(Six) p,(dx) j=O

(7.11)

118


where we have used the S-invariance of B and the Pointwise Ergodic Theorem. Let for A E X 1 k-l . xEX. fA(X) = lim sup k lA(SJ X),

E

k-+oo

j=O

It then follows from (7.11) that, with the notation in Corollary 5.5,

Since the LHS -t hl-£ I-£-a.e., we see that lim sup ..!:. n-+oo

n

E

lA(X) log fA(X) == h(x) j.L-a.e.

AE2(V2(n-l

Note that h is defined on X, is independent of 1-£, and satisfies (7.9) and (7.10). Moreover, his S-invariant mod 1-£ for 1-£ E Ps(X) since hl-£ is so. Thus we can redefine h so that h is actually S-invariant on X. Let 2{ E P(X) be a fixed partition and h be the S-invariant measurable function obtained in Theorem 7. Then h is called a universal entropy function. Finally we want to extend the entropy functional H(·) = H(·, 2{, S) to Pa(X), the space of all AMS sources, and hence to Ma(X) = {al-£+(3rJ : a, (3 E C, 1-£, rJ E Pa(X)}.

Proposition 8. Assume that (X, X, S) is an abstract measurable space with a measurable invertible transformation S : X -t X. Let 2{ E P (X) be a fixed partition. Then the entropy junctional H (., 2(., S) with a universal entropy junction h can be extended to a junctional H(., 2(., S) on Ma(X) with the same entropy junction h such that

1l(t;,,21,S)

= H(e, 21, S) =

L

hdt;"

Proof. Let ~ E Ma(X) and ~ E Ms(X) be the stationary mean. Since ~ « ~, ~ E Me(X) == {rJ E M(X) : rJ ~ ~} ~ Ma(X). Hence, by Theorem 1.5.9, the

functional H(·, 2{, S) can be extended to a functional H(., 2{, S) on Me(X) with the same entropy function h. But then, since h is invariant we see that

1l(t;,,21,S)

=

L = Lhd~ = H(~,21,S), hdt;,

Bibliographical notes

119

by Remark 4.2. This completes the proof. Results obtained in this section will be applied to derive an integral representation of the transrnission rate of a stationary channel in Section 3.6.

Bibliographical notes 2.1. Alphabet message spaces and information sources. Alphabet message spaces were introduced to formulate information sources and channels by McMillan [1](1953). Theorem 1.1 is shown in Umegaki [4](1964) where he proved that an alphabet message space is not hyper Stonean. 2.2. Ergodic theorems. Birkhoff [1](1931) proved the Pointwise Ergodic Theorem. The proof given here is due to Katznelson and Weiss [1](1982), which does not use the maximal ergodic theorem. von Neumann [1](1932) proved the Mean Ergodic Theorem. See also Akcoglu [1](1975).

2.3. Ergodic and mixing properties. (4) of Theorem 3.2 is obtained by Breiman [2](1960) and Blum and Hanson [1](1960) (see also Farrel [1](1962)). (4) of Theorem 3.6 is proved by Renyi [1](1958). Lemma 3.9 is due to Koopman and von Neumann [1](1932). An example of a measurable transformation that is weakly mixing but not strongly mixing is given by Kakutani [1](1973). Characterization of ergodic and mixing properties by the relative entropy is obtained by Oishi [1](1965) (Lemma 3.13 through Proposition 3.16). Related topics are seen in Rudolfer [1](1969).

2.4. AMS sources. The idea of AMS sources goes back to Dowker [2](1951) (see also [1, 3](1947, 1955)). Jacobs [1](1959) introduced almost periodic sources, which are essentially the same as AMS sources. Lemma 4.3 is proved in Kakihara [4](1991). Lemma 4.5 is due to Gray and Kieffer [1](1980). In Theorem 4.6, (2) and (5) are due to Rechard [1](1956), (3) and (4) to Gray and Kieffer [1] and (6) to Gray and Saadat [1](1984). Proposition 4.8 is shown by Fontana, Gray and Kieffer [1](1981) and Kakihara [4]. In Theorem 4.12, (2) is proved in Gray [1](1988), (8) is given by Ding and Yi [1](1965) (for almost periodic sources), and others are noted here. (1) of Theorem 4.14 is obtained in Kakihara [4] and (2) is in Kakihara [5]. (2) and (3) of Proposition 4.15 are in Kakihara [4] and (4) is in Kakihara [5]. 2.5. Shannon-McMillan-Breiman Theorem. The ergodic theorern in information theory is established in this section. Shannon's original form is Corollary 5.3 given in Shannon [1](1948). McMillan [1] obtained the L1-convergence in the alphabet rnessage space (Corollary 5.2). Breirnan [1](1957, 1960) showed the a.e. convergence. Corollary 5.5 is due to Nakamura [1](1969). There are various types of formulations and generalizations of 5MB Theorem. We refer to Algoet and Cover [1](1988), Barron [1](1985), Chung [1](1961), Gray and Kieffer [1], Jacobs [1], [3](1962), Ki-

120


effer [1, 3](1974, 1975), Moy [1](1960), [2, 3](1961), Ornstein and Weiss [1](1983), Parthasarathy [2](1964), Perez [1, 2](1959, 1964) and Tulcea [1](1960). 2.6. Ergodic decompositions. Ergodic decomposition is obtained by Kryloff and Bogoliouboff [1](1937). Oxtoby [1](1952) gave its comprehensive treatment. The content of this section is mainly taken from these two articles. See also Gray and Davisson [1](1974). 2. 7. Entropy functionals, revisited. The integral representation of an entropy functional by a universal entropy function in the alphabet rnessage space was obtained by Parthasarathy [1](1961) (see also Jacobs [5](1963)). For a totally disconnected compact Hausdorff space Umegaki [4] proved such a representation. Lemma 7.1 through Theorem 7.6 are due to Urnegaki [4]. Nakamura [1] derived an integral representation in a measure theoretic setting without using ergodic decompositions (Theorem 7.7). Proposition 7.8 is noted here. Extension of the entropy functional H(·) to almost periodic sources is shown by Ding and Yi [1] using the result of Jacobs

[1].

CHAPTER III

INFORMATION CHANNELS

In this chapter, information channels are extensively studied in a general setting. Using alphabet message spaces as models for input and output, we formulate various types of channels such as stationary, continuous, weakly mixing, strongly mixing, AMS and ergodic ones. Continuous channels are discussed in some detail. Each channel induces an operator, called the channel operator, which is a bounded linear operator between two function spaces and completely determines properties of the channel. One of the main parts of this chapter is a characterization of ergodic channels. Some equivalence conditions for ergodicity of stationary channels are given. It is recognized that many of these conditions are similar to those for ergodicity of stationary sources. AMS channels are considered as a generalization of stationary channels. Ergodicity of these channels is also characterized. Transmission rate is introduced as the mutual information between input and output sources. Its integral representation is obtained. Stationary and ergodic capacities of a stationary channel are defined and their coincidence for a stationary ergodic channel is shown. Finally, Shannon's first and second coding theorems are stated and proved based on Feinstein's fundamental lemma.

3.1. Information channels Definitions of certain types of channels are given and some basic properties are proved. As in Section 2.1 let X o = {al, ... ,ap } be a finite set,· construct a doubly infinite product space X = X~, and consider a shift transformation S on X. If X is the Baire a-algebra of X, then (X, X, S) is our_input space. Similarly, we consider an output space (Y, ID, T), whereY = Y oZ for another finite set Yo = {b I , ••. ,b q } , ~ is the Baire a-algebra of Y, and T is the shift on Y. The compound space (X x Y, X ®~, S x T) is also constructed, where X ® ~ is the a-algebra generated by {A xC: A E X, C E ~} and is the same as the Baire a-algebra of X x Y. We use notations p(n), Pa(n), ps(n), C(n), B(n), ... etc. for 0. = X, Y or X x Y. 121

Chapter III: Inforrnation Channels

122

Definition 1. A channel with input X = X~ and output Y for which the function v : X x q) -+ [0, 1] satisfies

= YoZ is a triple [X, t/, Y]

(cl ) v(x,') E P(Y) for every x E X; (c2) v(·, C) E B(X) for every CEq). In this case v is called a channel distribution or simply a channel. C(X, Y) denotes the set of all channels with input X and output Y. The condition (cl) says that if an input x E X is given, then we have a probability distribution on the output, and we can know the conditional probability v(x, [Yi' .. Yj]) of a particular message [Yi' .. Yj] received when x is sent. The technical condition (c2) is needed for mathematical analysis. to be if (c3) v(Sx, C) = v(x, T-1C) for every x E X and CEq), which is equivalent to (c3') v(Sx, Ex) = v(x, T- 1 Ex) for every x E X and E E XQ9q), where Ex Y : (x, y) E E}, the x-section of E.

= {y E

Since, in this case, Sand T are invertible, we may write the condition (c3) as (c3") v(Sx, TC) = v(x, C) for x E X and CEq). Cs(X, Y) denotes the set of all stationary channels v E C(X, V). Note that C(X, Y) and Cs(X, Y) are convex, where the convex combination is defined by

x EX, CEq). Let p(bk laj) be the conditional probability of bk received under the condition that aj is sent, where 1 ~ j < p and 1 < k < q. If a channel v is defined by j

(c4) V(X'[Yi"'Yj]) = TIp(y£lx£), where x = (Xl) E X and [Yi"'Yj] C Y is a £=i message, then it is said to be memoryless. The p x q matrix P = (P(bklaj))j,k is called Clearly every memoryless channel is stationary. On the 'V~Utl~~~~'v.L v E C X ,Y is said to have a nite memor r to be an (c5) There exists a positive integer m such that for any message V = [Yi'" Yj] with i ~ . j it holds that v(x, V)

= v(x', V),

x

= (Xk), x' =

(x~) E X

with Xk

= x~ (i - m

~ k ~

j).

In some literature, a finite memory channel is defined to satisfy (c5) above and the finite dependence condition A weaker condition than (c5) is as follows:.~~~~~~~~~~!£J~~~~~~

123

9.1. Information channels

(c5') v{·, [Yi·· ·Yj]) E C{X) for every message [Yi·· ·Yj] C Y, which is equivalent to

(c5") [f(·, y)v(o,dy) E G(X) for every f

E

G(X x Y).

The equivalence will be proved in Proposition 5 below.._------~ (c6) There exists some

'TJ E

P{Y) such that v{x,·)

~ 'TJ

_

for every x E X.

Now let v E C{X, Y) and J.L E P{X), which is called an input source. Then the output source J.LV E P{Y) and the compound source J.L®v E P{X x Y) are respectively defined by

I.LV(G)

=

p, ® v(E)

=

L

v(x, G) p,(dx),

L

v(x, Ex) p,(dx),

GE

!D,

E E X ®!D,

(1.1) (1.2)

where Ex is the x-section of E. (1.2) can also be written as P, ®

v(A x G) = Lv(x, G) p,(dx),

A E X,C E~.

Note that J.L ® v{X x C) = J.Lv{C) and J.L ® v{A x Y) = J.L{A) for A E X and C E ~. All of the above definitions (except (c4), (c5) and (c5')) can be made for a pair of general compact Hausdorff spaces X, Y with Baire a-algebras X, ~ and (not necessarily invertible) measurable transformations S, T, respectively. Or more generally, (cl)-- (c3) and (c6) are considered for channels with input and output of measurable spaces (X, X, S), (Y,~, T). In this case, we consider "abstract" channels v with input X and output Y. In what follows, unless otherwise stated, X and Y stand for general measurable spaces. Note that any output source 'TJ E P{Y) can be viewed as a "constant channel" by letting x E X,C E~. So we may write P{Y) C C{X, Y). In this case,

J.L E P{X). Thus, if 'TJ is stationary, the channel v", is stationary. Consequently, we may write Ps{Y) C Cs{X, Y). A simple and important consequence of the above definitions is:

124

Chapter III: Information Channels

Proposition 2. If v E Cs(X, Y) and J-L E Ps(X), then J-tv E Ps(Y) and J-L Q9 v E P, (X x Y). That is, a stationary channel transforms stationary input sources into stationary output sources and such inputs into stationary compound sources.

Proof. Let v E Cs(X, Y), J-t E Ps(X), A E X and CEq). Then we have J-LQ9 v((S x T)-I(A x C))

= J-tQ9 v(S-IA x T- IC) v(x,T- IC)J-L(dx)

= f

JS-IA

= = where

SJ-L = J-t 0

r

v(Sx, C) p,(dx) =

JS-IA

L

rvex, C) 8p,(dx)

JA

vex, C) p,(dx) = p, ® v(A x C),

S-I. This is enough to deduce the conclusion.

A type of converse of Proposition 2 is obtained as follows:

Proposition 3. Assume that S is invertible and q) has a countable generator q)o, and let J-L E P(X) and v E C(X, Y). If tL Q9 v E Ps(X x Y), then J-L E Ps(X) and u E Cs(X, Y) J-L-a.e. in the sense that there is some stationary VI E Cs(X, Y) such that v(x,·) = Vl(X,·) u-a.e.x E X.

Proof. Since J-t Q9 v is stationary, we have for A E X and CEq) p, ® v(A

x C) =

L

vex, C) p,(dx)

x T)-I(A x C)) = J-L Q9 v(S-1 A x T- IC). = J-L Q9 v((S

If C

= Y,

then (1.3) reduces to AEX,

Le., J-L E Ps(X). Using this, (1.3) can be rewritten as p, e v(S-l A x T-1C) =

=

=

r

JS-I A

L L

vex, T-1C) p,(dx)

V(S-lx, T-1C) 8p,(dx) v(S-lX, T-1C) p,(dx)

(1.3)

125


for every A E X and C E~. Hence v(x, C) = v(S-lx, T-1C) u-a.e.x for every C E~. Let ~o = {C1,C2 , ... } and X n = {x EX: v(x,Cn ) = v(S-lx,T-1Cn ) } for n 2: 1. Then X*

00

= n=l n Xn

E

..

v(x, .) = v(S-'-lx, T- 1.) on t/ is stationary u-a.e.

~o

X is such that /-L(X*)

= 1 and,

for every x E X*,

and hence on ~ since ~o is a generator of~. Thus

The following proposition may be viewed as an ergodic theorem for a stationary channel. Proposition 4. If v E Cs(X, Y) and f-L E Ps(X), then for every E, F E X ® ~ the following limit exists: n-l

lim ! "v(x, [(8 n4-OO

n L...t

X

T)-k En Flo:) u-a.e.x.

k=O ~

In particular, for every C, D E

the following limit exists:

n-l

lim ! n4-OO n

L

v(x, T-kC n D) u-a.e.x,

k=O

Proof. Let J = {E E X ® ~ : (S x T)-l E = E} and E, F E X ® ~ be arbitrary. Since f-L ® u E Ps(X x Y) we have by the Pointwise Ergodic Theorem that n-l

lim !LIE((8xT)k(X,y)) = lim [(S@T)n1E](x,y) n4-OO n n4-OO k=O

= EJL®v(lEIJ)(x, y) f-L ® u-a.e.,

(1.4)

where [(8®T)f](x,y) = f(Sx,Ty) for f E B(XxY), x EX, Y E Yand (8®T)n = ~

n-l

l:: (8

k=O

® T)k. Let Z

= {(x,y)

E X x Y : Limit in (1.4) exists for (x,y)}. Then

Ix

f-L® v(Z) = v(x, Zx) f-L(dx) = 1, so that v(x, Zx) = 1 u-a.e.x, Hence the following limit exists u-a.e.x by the Bounded Convergence Theorem: n-l

lim! "v(x, [(8 n4-OO

n L...t

X

T)-k En Flo:)

k=O n-l

= lim (IF(x,y)! LIE((8xT)k(x,y))II(x,dy) n4-OO }y n k=O


126

=[

IF(x, y)EI'®v(lEIJ)(x, y) v(x, dy).

We need a tensor product Banach space with the least crossnorm A (cf. Schatten [1]). Let E and F be two Banach spaces and £. 0 F be the algebraic tensor product. n

For j 0 'ljJj

the least crossnorm A(* (4) )'ljJ* ('ljJ). The completion of £. 0 F w.r. t. A is denoted by £.®.xF and called the injective tensor product Banach space of E and F. Suppose that X, Yare compact Hausdorff spaces. If E = C(X), then C(X) 0 F is identified with the function space consisting of F-valued functions on X: xEX.

Moreover, it is true that

C(X) ®.x F

= C(X ;F),

the Banach space of all F-valued (norm) continuous functions on X with the sup norm. In particular, we have that

C(X) ®.x C(Y) =C(X x Y), where we identify (a0b)(x, y)

= a(x)b(y) for a E C(X), b E C(Y)

Proposition 5. Consider alphabetmessage spaces X v : X x ~ --+ [0,1]. Then: (1) (c5) =>(c5') {::} (c5"). (2) (c5'), (c6) =>(c2).

and x E X, Y E Y.

= X~, Y = YoZ and a junction

Proof. (1) (c5)=>(c5') is clear and (c5'){::}(c5") follows from the fact that each message [Yi ... Yj] is a clopen set and the set rot of all messages forms a topological basis for Y, and the fact that C(X x Y) = C(X) ®.x C(Y), where the algebraic tensor product C(X) 0 C(Y) is dense in it as is noted above. (2) Let b E C(Y). Then there exists a sequence {Sn}~=l of simple functions of kn

the form Sn =

E

O!n,k 1A n ,k ' An,k Erot, n

~ 1 such that

k=l

sup /Sn(Y) - b(y)/ yEY

= /lsn - b/l --+ 0

as n --+

00.

127


This implies that

[ Sn(y) v(x, dy) ----t [b(Y) v(x, dy) uniformly in x,

Jy

Jy

Since sn(Y) v(·, dy) E C(X) by (c5'), we have b(y) v(·, dy) E C(X). Now (c6) implies that the RN-derivative k(x,·) == v~(d:») exists in L 1 (y , rJ) for every fixed x E X and

v(x, C)

=

fc

k(x, y) 'T/(dy) ,

x E X, C E~.

Let C E ~ be given and find a sequence {bn}~=l ~ C(Y) such that

[ Ibn(y) - lc(y)1 'T/(dy) ----t O. Then we have that, for x E X,

v(x, C) = [ lc(y)k(x, y) 'T/(dy)

r lim r bn(y) v(x, dy) l-

== n-+oo lim bn(y)k(x, y) rJ(dy) }y ==

n-+oo

by the Dominated Convergence Theorem. Therefore, v(·, C) is a limit of a sequence of continuous functions, and hence is measurable on X. That is, (c2) holds. For a channel v E C(X, Y) define an operator K; : B(Y)

(Kvb)(x) = [b(Y) v(x, dy),

~

B(X) by

bE B(Y),

(1.5)

where B(X), B(Y} are spaces of all bounded measurable functions on X, Y, respectively. In fact, if b ==

n

E

G-j lej' a simple function on Y, then obviously Kvb E B (X). j=l For a general b E B (Y) we can choose a sequence {bn } of simple functions converging to b pointwise with Ibnl :::; Ibl for n 2: 1. Then the Dominated Convergence Theorem applies to conclude that Kvb E B(X). Now we can characterize continuous channels between alphabet message spaces using the operator K v .

Proposition 6. Consider alphabet message spaces X == X~ and Y == JlQz and a stationary channel t/ E Cs (X, Y). Then the following conditions are equivalent:


128

(1) V is (2) The (3) The M(Y) and weak*.

continuous, i.e., t/ satisfies (c5'); operator K v defined by (1.5) is a linear operator from C(Y) into C(X); dual operator K~ of K; is from C(X)* into C(Y)*, i.e., K~ : M(X) -+ is sequentially weak * continuous, i.e., ~n -+ ~ weak* implies K~en -+ K~~

(4) v(-, k~l[Y}:) ... Y;~)]) E C(X) for n

~

1 and messages [Yik ... Yik]

0, then there exists an ergodic source 'TJ E Pse(X) such that 'TJ(A) > o. As was remarked before (cf. Section 2.2), if S is a continuous transformation on X, then S is measurable, Ps(X) -=I 0 and Pse(X) -=I 0. Proposition 9. Suppose that S is a continuous transformation on the compact Hausdorff space X into X. Then, Ps(X) is complete for ergodicity.

Proof. Suppose that J-L(A) > 0 for some j-t E Ps(X) and A E X. Since J-L is regular, there is a compact set C E X such that C ~ A and J-L( C) > O. Then we can choose a sequence {fn}~=l ~ C(X) such that t« t 1e as n -t 00 since X is compact and Hausdorff. If 0 ::; a ::; 1, then the set Pa == {'TJ E Ps(X) : 'TJ(C) 2: a} is a weak* compact convex set, since

r: == n {'TJ E Ps(X): 'TJ(fn) 2: a} 00

n=l

and Ps(X) is so. Let ao == sup {a : P« -=I 0}. Then, ao 2: j-t(C) > 0 and Pao is a nonempty weak* compact convex set. Hence, there exists some 'flo E ex P ao such that 'TJo(C) == ao > O. We claim that 'flo E exPs(X) == Pse(X). If this is not the case, there exist J-Ll, J-L2 E Ps(X) and (3 E (0,1) such that 'TJo == {3J-Ll + (1 - (3)J-L2 and J-Ll -=I J-L2· Note that by the definition of ao we have J-Ll (C), J-L2 (C) ::; ao. Since ao == 'TJo(C) == (3J-Ll(C) + (1 - (3)J-L2(C), we must have J-Ll(C) == J-L2(C) == ao and hence J-Ll, J-L2 E Pao' This is a contradiction to the extremality of 'flo in Pao' Thus 'TJo E Pse(X). For this 'TJo it holds that 'TJo(A) 2: 'TJo(C) > O. Proposition 10. Suppose that Ps(X) is complete for ergodicity and let Cs (X, Y). Then the following conditions are equivalent:

(1) Vl == V2 (modPs(X)). (2) Vl == V2 (modPse(X)). (3) Vl(X, Ex) == V2(X, Ex) Ps(X)-a.e.xfor every E E XQ9~. (4) vl(x,Ex) == V2(X, Ex) Pse(X)-a.e.xfor every E E XQ9~. (5) J-L Q9 Vl == J-L Q9 V2 for every J-L E P; (X). (6) J-L Q9 Vl == J-L Q9 V2 for every J-L E Pse(X). Proof. (1)

=}

(2), (3)

=}

(4), (5)

=}

(6), (1) {:} (5) and (2) {:} (6) are clear.

Vb

V2

E


136

(1) and (4) =} (3) follow from the completeness for ergodicity of Ps(X). (4). Assume (4) is not true. Then there is some E E X ® q) and some M E Pse(X) such that M({X EX: VI(X, Ex) i= v2(x,Ex)}) > O. We may suppose M({X EX: vI(x,Ex) > v2(x,Ex)}) > O. Then (2) (6)

=}

=}

L.

V

l (x,E x ) P,(dX) >

L.

V

2(x,E x ) P,(dX),

i = 1,2,

it holds that

M® VI (E n (A x which implies that M®

VI

i= M® V2,

Y)) > M® V2 (E n (A x Y)), a contradiction to (6).

To close this section we characterize continuous channels once more in terrns of channel operators, which is an extension of Proposition 1.6.

Proposition 11. Let V E C(X, Y) be a channel and K E JC(X, Y) and A E A(X, Y) be the corresponding channel operators. Then the following conditions are equivalent: (1) v is continuous, i.e., u satisfies (c5"). (2) K : C(Y) -+ C(X) is a linear operator. (3) A : C(X x Y) -+ C(X) is a linear operator. (4) K* : M(X) -+ M(Y) is sequentially weak* continuous. (5) A* : M(X) -+ M(X x Y) is sequentially weak* continuous. Proof. This follows from Proposition 1.6, Remark 3 and Corollary 6. The results obtained in this section will be applied to characterize ergodicity of stationary channels in Section 3.4 and to discuss channel capacity in Section 3.7.

3.3. Mixing channels Finite dependence was defined for channels between alphabet message spaces. As a generalization, asymptotic independence (or strong mixing) and weak mixing are introduced, which are similar to those for stationary information sources. Also ergodicity and semiergodicity are defined for stationary channels and relations among these notions are clarified.

137

3.3. Mixing channels

Consider a pair of abstract measurable spaces (X, X) and (Y,~) with measurable transformations Sand T, respectively. We begin with a notion of ergodicity.

Definition 1. A

n+""'+'i'"\.·.....~......."If ..

if

"".L............L......."".L

i.e., if a stationary ergo_d_i.__~_rce is the input, then the compound source rnust also denotes the set of all stationary ergodic channels. be stationary ergodic. To obtain conditions for ergodicity we first consider channels between alphabet message spaces X = X~ and Y = Yl~, where X o and Yo are finite sets. We have the following with the notation in Section 3.1.

Proposition 2. If

t/

E Cs(X~, YoZ ) is a memoryless channel, then v is ergodic.

Proof. Let P = (p(bla))a,b be the channel matrix of v (cf. Definition 1.1, (c4)). Let J-t E Pse(X) and A = [xo = a],B = [xo = a'] C X and C = [Yo = b],D = [Yo = b'] c Y. Then one has 1

n-l

- LJ-tQ9v((SxT)-k(AxC)n(BxD)) n k=O 1

n-l

= - LJ-tQ9v((S-kAnB) x (T-kCnD)) n

k=O n-l

=

L

1 /L @ V([Xk n k=O 1

= a, Xo = a'] x

[Yk

= b, Yo = b'D

n-l

= -n L J-t([Xk = a, Xo = a'])p(bla)p(b'la') k=O

---+ J-t([xo = a])J-t([xo = a'])p(bla)p(b'la'),

= J-t Q9 v(A

by ergodicity of u,

x C)J-t Q9 v(B x D).

It is not hard to verify that the above holds for general messages A, B c X~ and C, D c Yoz . Hence J-t Q9 u is ergodic by Theorem 11.3.2. Therefore v is ergodic.

Definition 3. A channel v E C(XZ, J:':Z) is said to be finitely dependent or maeuenaeni if n

(c8) There exists a positive integer mEN such that for any n, r, s, tEN with r ~ S ~ t and s- r > m it holds that

~


138

for every x E X and every message Cn,r = [Yn ... Yr], Cs,t

= [Ys ... ytl c Y.

Then we record the following theorem, which immediately follows from Theorem 7 below.

Theorem 4. If a stationary channel v E Cs(X~, yoZ) is finitely dependent, then v is ergodic. Example 5. Consider alphabet message spaces X = X~ and Y = yoZ with messages IDly in Y. Let m 2: 1 be an integer and for each (Xl,... ,Xm ) E X a probability distribution p((XI, ... ,xm)l· ) on Yom is given. Define Vo : X x IDly ~ [0,1] by

o

t-s vo{x,[Ys···Yt])

= IIp((Xs+k-m, ...

,Xs+k-I)I(Ys+k-m, ... ,Ys+k-l))

k=O and extend Vo to v : X X ~ ~ [0,1], so that v becomes a channel. Then, this u is stationary and finitely dependent, and has a finite mernory. This is easily verified and the proof is left to the reader. Finite dependence can be generalized as follows. Consider a general pair of abstract measurable spaces (X, X) and (Y,~) with measurable transformations Sand T as before. First recall that a stationary source JL E P; (X) is said to be strongly mixing (8M) if A,BEX,

and to be weakly mixing (WM) if n-l

.!. L n-too n

IJL(S-kA n B) - JL(A)JL(B)

lim

I = 0,

A,BEX.

k=O

Also recall that JL E Ps (X) is ergodic iff n-l

! ~ JL(S-kA n B) = JL(A)JL(B), n-too n Z:: lim

A,BEX

k=O

(cf. Theorem 11.3.2). It was noted that for stationary sources strong mixing :::} weak mixing :::} ergodicity. As an analogy to these formulations we have the following definition.

I

J

139


Definition 6. A channel v E C(X, Y) is said to be asymptotically independent or strongly "mixing (8M) if (c9) For every C,D E

~

lim {vex, T-nc n D) - v{x, T-nC)v(x, D)} = 0 Ps{X)-a.e.x.,

n-+oo

to be weakly mixing (WM) if (c10) For every C,D E

~

1 n-l

lim

n-+oo n

E Iv{x, T-kC n D) -

vex, T-kC}v{x, D)I = 0 Ps(X)-a.e.x,

k=O

and to be semiergodic (8E) if

(ell) For every C,D E 1

lim -

n-+oon

~

n-l

E {vex, T-kC n D) - vex, T-kC)v(x, D)} = 0

Ps(X)-a.e.x.

k=O

Note that if (X, X, S) is complete for ergodicity (cf. the previous section), then in (c9), (c10) and (3.1) below we can replace Ps(X)-a.e. by Pse(X)-a.e. Consider the following condition that looks slightly stronger than (ell): for every C,D E ~ n-l

n-l

lim .!. "v(x, T-kC n D) = lim.!. "v{x, T-kC)v(x, D) Ps{X)-a.e.x. n-+oo n Z:: n-+oo n Z:: k=O k=O

(3.1)

In view of Proposition 1.4, the LHS of (3.1) exists for a stationary channel. Hence (ell) and (3.1) are equivalent for stationary channels. If (3.1) holds for every x E X, then the existence of the RHS means that v(x,') E Pa(Y) and (3.1) itself means that v(x,') E Pae(Y) for every x E X (cf. Theorem 11.4.12). Also we have for (stationary) channels strong mixing ::::} weak mixing ::::} ergodicity ::::} semiergodicity, where the first implication is obvious, the second is seen in Theorem 7 below, and the last will be verified later. Actually, (3.1) is a necessary (and not sufficient) condition for ergodicity of a stationary channel (cf. Theorem 13 and Theorem 4.2). Now we can give a basic result regarding these concepts.

Theorem 7. Let

t/

E

Cs(X, Y) and f-t E Ps(X).

140


(1) If J-L strongly or (2) If J-L (3) If J-L

is ergodic and v is weakly mixing, then J-L ® v is ergodic. Hence, every weakly mixing stationary channel is ergodic. and v are weakly mixing, then J-L ® v is also weakly mixing. and v are strongly mixing, then J-L ® v is also strongly mixing.

Proof. (1) Let A, B E X and C, D E q). Then we have that n-l

r

.!.E n

v(x, T-kC)v(x, D) p,(dx)

k=O}s-kAnB

1 = .!. E 1 n-l

= .!. E

v(x, T-kC)lA(SkX)V(X, D)lB(x) p,(dx)

n k=O X n-l

v(Sk x, C)lA(Skx)v(x, D)lB(x) p,(dx),

n k=O

x

(3.2)

since v is stationary,

L

-+

vex, C)lA(x) p,(dx)

L

vex, D)lB(x) p,(dx),

since J-L is stationary and ergodic, == J-L ® v(A x C)J-L ® v(B x D).

On the other hand, since 1

t/

n-l

is weakly mixing, it holds that

-n E Iv(x, T-kC n D) -

I

v(x, T-kC)v(x, D) ~ 0 u-a.e.x E X

k=O

as n -+

00.

By the Bounded Convergence Theorem we have

n-l

.!.n E

r

I

Iv(x, T-kC n D) - vex, T-kC)v(x, D) p,(dx)

k=O}S-kAnB

1.!. E I n-l

< as n -+

00.

(3.3)

k=O

Cornbining (3.2) and (3.3), we get

I~ ~ p,0 v((S X T)-k(A x C) n (B x D)) =

I

vex, T-kC n D) - vex, T-kC)v(x, D) p,(dx) -+ 0

x n

p,0 v(A x C)p,0 v(B x D)I

I~ ~ p,0 v((S-k A n B) x (T-kC n D)) -

p,0 v(A x C)p, 0 v(B x D)I

141


==

I.!.n ~ [ vex, T-kC n D) p,(dx) k=O}S-kAnB

p, 0 v(A

X

C)p, 0 v(B

X

D)

I

n-l

: :; .!. L .[

Iv(x, T-kC n D) - vex, T-kC)v(x, D)I p,(dx)

n k=O}S-kAnB

+ I.!.

E[

v(x,T-kC)v(x,D) p,(dx) - p,0v(A

X

C)p,0v(B

X

D)!

n k=O}S-k AnB

-+ 0 as n -+

00.

Thus, by Theroern 11.3.2 I-" ® v is ergodic. (2) and (3) can be verified in a similar manner as above. In the above theorem we obtained sufficient conditions for ergodicity of a stationary channel, namely, strong mixing and weak mixing. Key expressions there are (3.2) and (3.3). In particular, if'f/ E Ps(Y) is WM, then the constant channel Vn is stationary ergodic. For, if I-" E Pse(X), then I-" ® v"., == I-" x 'f/ is ergodic by Theorem 11.3.10. In the rest of this section, we consider semiergodic channels together with averages of channels. We shall show that there exists a semiergodic channel for which, if a weak mixing source is input, then the compound source is not ergodic. This irnplies that semiergodicity of a channel is weaker than ergodicity.

(c12) v*(Sx,C)

(c13)

L

== v*(x;C) for x

vex,C) p,(dx)

=

L

E X and C E~;

v*(x, C) p,(dx) for A

E

J, C E

!D

and p, E Ps(X).

(c12) means that for each C E ~ the function v*(·,C) is S-invariant, while (c13) means that v*(·, C) == EM (v(., C)13)(.) u-a.e. (3.4) for C E ~ and I-" E Ps(X), where E M(·13) is the conditional expectation w.r.t. 3 under the measure 1-". A sufficient condition for the existence of the average of a given stationary channel will be obtained (cf. Theorem 13). We need a series of lemmas.

Lemma 9. If v

E

Cs(X, Y) has an average u", then for C v*(·, C)

= n-too lim .!. n

E ~

n-l

L v(Sk., C)

k=O

Ps(X)-a.e.

(3.5)

142


Proof. In fact, this follows from (3.4) and the Pointwise Ergodic Theorem. Lemma 10. Let v E Cs(X, Y) with an average t/", If J-L E Pse(X) is such that J-L ® v =f. J-L x (J-Lv), the product measure, then J-L ® v =f. J-L ® u" . Proof. Observe that for A E X and C E

JL ® v*(A x C)

=

i

~

v*(x, C) JL(dx) n-I

= [ lim .! L v(Skx, C) JL(dx), JA n-too n k=O

by (3.5),

n-I

lim .! L [ v(Skx, C)lA(X) JL(dx) n-too n k=O Jx

L

v(x, C) JL(dx)

L

lA(X) JL(dx),

since JL is ergodic,

= J-L(A)J-Lv(C).

Thus, J-L ® v

=f. J-L ® t/"

by assumption.

v;,

Lemma 11. Let VI, V2 E Cs(X, Y) be semiergodic with averages vi, respectively. If vi == v; (modPs(X)) (cf. Definition 2.4), then Vo = !(VI +V2) is also semiergodic. Proof. Since

vi (i = 1,2) exists, we see that for C, D E ~ n-I

lim.! LVi(X,T-kC)Vi(X,D) n-too n k=O

= v;(x,C)vi(x,D) Ps(X)-a.e.x

by Lemma 9 and hence, by semiergodicity of Vi, n-I

lim 1 L Vi(X, T-kC n D) n-too n k=O

= V;(x, C)Vi(X, D) Ps(X)-a.e.x.

Consequently, we have that 1 n-I

lim -

n-toon

L

k=O

{vo(x, T-kC n D) - vo(x, T-kC)vo(x, D)}

1 [1 n-I

= n-too lim - '"" n L..J k=O

-2 VI (x, T-kC n D)

1

+ -V2(X, T-kC n D) 2

(3.6)

143


since

VI, V2

are stationary,

= ~{vnx, C)VI(X, D) + vi(x, C)V2(X, D)}

- ~ {vi (x, C) + vi(x, C)} {VI (x, D) + V2(X, D)}, by (3.6), stationarity of

VI, V2,

and the definition of the average,

= ~{vi(x,C) -vi(X,C)}{VI(X,D) -v2(x,D)} == 0

by

Ps(X)-a.e.x,

vi == v 2 (rnodPs(X)).

Lemma 12. If semiergodic.

V

E

Thus Vo is serniergodic.

Cs(X, Y) is semiergodic and has an average u", then u" is also

Proof. Let C, D E ~ and J.L E Ps(X). Then, n-I

n-I

lim 1 "'v*(x,T-kCnD) n-.+oo n L-J k=O

= n-.+oo lim.!.. "'EI'(v(.,T-kCnD)!J)(x) n L-J k=O

= }~~ EI' (~ ~ v(.,T-kC n D)IJ) (x) == E~(v*(.,

by (3.6) with

V

==

Vi

>

C)v(·, D)IJ)(x),

and (10) in Section 1.2, == v*(x,C)E~(v(.,D)IJ)(x) ==

t/"

(x, C)v* (x, D) n-I

= by (cI2), which yields that

t/"

lim .!..

n-.+oo

n

L v*(x, T-kC)v*(x, D)

u-a.e.x,

k=O

is semiergodic.

Now the existence of an average channel of a given stationary channel is considered under SOUle topological conditions.


144

Theorem 13. Let (X, X, S) be an abstract measurable space with a measurable transformation Sand Y be a compact metric space with the Borel a-algebra ~ and a homeomorphism T. Then every stationary channel v E Cs(X, Y) has an average v* E

Cs(X,Y).

Proof. Denote by Mt(Y) c M(Y) = C(Y)* the positive part of the unit sphere and by B(Y) the space of bounded Borel functions on Y. Let v E Cs(X, Y) be given. For each x E X define a functional V x on B(Y) by

v",(J) = [1(y)v(x,dY),

f

E

B(Y).

If we restrict V x to C(Y), then we see that V x E Mt (Y) for each x EX. Let 1) be a countable dense subspace of C (Y) with the scalar multiplication of rational complex numbers and let n-l

A",(J) = lim .!:. n-+oo

n

L VSk",(J),

fE1)

(3.7)

k=O

for each x E X. Since, for each f E C(Y), the function v(o)(f) is bounded and measurable on X, (3.7) is well-defined u-a.e. by the Pointwise Ergodic Theorem for each JL E Ps(X). Let

Xn

= {x EX:

(3.7) exists for all f E 1)}.

It is easily seen that X n E X, Xn is S-invariant, and J-L(Xn ) = 1 for JL E Ps(X) since 1) is countable. Note that, for each x E X, A x ( · ) is a positive bounded linear functional on 1) since for f E 1) IAx(f)1 =

~

I

~

lim .!:. VSk",(J) I n L....t

n-+oo

k=O

n-l

lim n-+oo

.!:. L n

( I/(y)1 v(Skx,dy) < 11111,

k=O}Y

so that Ax (.) can be uniquely extended to a positive bounded linear functional Ax (.) of norm 1 on C(Y). Let us examine some properties of Ax. Ax satisfies that

x E X n , f E C(Y)

(3.8)

since Asx(f) = Ax(f) for x E X n and f E 1). For each f E 1), A(o)(f) is Xmeasurable on Xn, which follows from (3.7). For a general f E C(Y) there exists a sequence {fn}~=l C 1) such that "fn - fll ~ 0, so that


145

implying the measurability of A(o)(/) on X1). Moreover, for each x E X1), one can find a probability measure 'l7x on ~ such that

I

Ax(f) = [f(Y) 1Jx(dy),

E C(Y)

by the Riesz-Markov-Kakutani Theorem. One can also verify that rlx is T-invariant, i.e., '17 E Ps(Y) for x E X1), which follows from (3.7) and stationarity of t/. Consider the set

Eo = {f E B(Y) : [ f d1Jx is X-measurable on X:o}. Then we see that C(Y) ~ 8 0 and 8 0 is a monotone class, i.e., if {In}~=l ~ 8 0 and In .,t. I, then I E 8 0 , Hence one has 8 0 = B(Y). Denote by the same syrnbol Ax the functional extended to B(Y). Take any '17 E Ps(Y) and define u" by if x E X1)

v*(x, C) = { 'l7x(C) '17 (C)

if x E X

n

for C E~. We shall show that t/" is the desired average of t/, First note that u" is a stationary channel, u" E Cs(X, Y), by'l7x (x E X1»),'17 E Ps(Y) and (3.8). Now we verify (c12) and (c13) (Definition 8). If x E X1), then Sx E X1) and

v*(x, C) = 'l7sx(C) = 'l7x(C) = v*(x, C), and if x E X

n, then Sx E X n and v*(Sx, C)

= 'I7(C) = v*(x, C),

CE~,

so that (c12) is satisfied. To see (c13), let I-t E Ps(X) be fixed and observe that

L[

fey) vex, dy)p,(dx) =

for I E 11. For, if g(x)

= Jy I(y) v(x, dy)

gs(x) == lim (Sng)(x) n-+oo

and

L

L

[f(Y) v*(x, dy)p,(dx)

(x E X), then

= }y [ fey)

g(x) p,(dx) =

L

v*(x, dy) u-a.e.x

gs(x) p,(dx)

(3.9)

146


by the Pointwise Ergodic Theorem, which gives (3.9). Moreover, (3.9) holds for f E C(Y) since ~ is dense in C(Y). If G E X is S-invariant (GE J) with /-L(G) > 0, then the measure /-La defined by AEX,

is an S-invariant probability measure. We have that for CEq)

fa

v(x, C) p,(dx) = p,(G)

= p,(G)

=

fa

L L

v(x, C) p,a(dx) v*(x, C) p,a(dx)

v*(x, C) p,(dx).

If G E J is such that /-L(G) = 0, then the equality in (c13) also holds. Thus u" satisfies (c13). Therefore t/" is the desired average of u, The following theorem is our main result of this section. Theorem 14. There is a semiergodic channel Va and a strongly mixing input /-La for which the compound source /-La ® Va is not ergodic. Hence, a semiergodic channel is not ergodic. Proof. Consider an alphabet message space X = {O, l}Z with the Baire a-algebra X and the shift S. Let V E C(X, X) be a memoryless channel with the channel matrix p=

(I t)

(cf. Definition 1.1). Then one can directly verify that u is stationary and semiergodic. i/" by Theorem 13 and t/" is semiergodic by Lemma 12, so that

v has an average

1

1 *

Va = "2 v +"2 v

is also semiergodic by Lemma 11 since (z-") * = t/", Now let /-La be a (!' ~)-Bernoulli source (cf. Example II.1.2), which is strongly mixing (cf. Example 11.3.7). Then /-La ® Va is not a direct product measure, so that

by Lemma 10. Since 1

1.

/-La ® Va = "2/-La ® v + "2/-La ® u

*

147

3.4. Ergodic channels

is a proper convex combination, J-to Q9 Vo is not ergodic by Theorem 11.3.2. This completes the proof.

3.4. Ergodic channels In the previous section sufficient conditions for ergodicity were given. In this section, ergodicity of stationary channels is characterized by finding several necessary and sufficient conditions. We shall notice that many of these conditions are similar to those for a stationary source to be ergodic. Also we use channel operators to obtain equivalence conditions for ergodicity. In particular, we realize that there is a close relation between ergodicity and extremality. Let (X, X, S) and (Y,~, T) be a pair of abstract measurable spaces with measurable transformations Sand T, respectively. Recall that~~a",_~,~~,~,~~~~~~J:"e"e~~:~~~~ v is said to be if

(c7) J-t

E Pse(X)

=> J-t Q9 v

E Pse(X

x Y).

Definition 1. Let P be a subset of P(X). A stationary channel v E Cs(X, Y) is said to be extremal in Cs(X, Y) mod P, denoted v E exCs(X, Y) (mod P), if Vl, v2 E Cs(X, Y) and a E (0,1) are such that v == aVl + (1 - a)v2 (mod P), then Vl == V2 (mod P). The following theorem gives some necessary and sufficient conditions for ergodicity of a stationary channel, which are very similar to those for ergodicity of a stationary source (cf. Theorem II. 3.2) .

Theorem 2. For a stationary channel u E Cs(X, Y) the following conditions are equivalent: (1) v E Cse(X, Y), i.e., v is ergodic. (2) If E E XQ9~ is S x T-invariant, then v(x, Ex) = 0 or 1 Pse(X)-a.e.x. (3) There exists an ergodic channel Vl E Cse(X, Y) such that v(x,') ~ Vl(X,·) Pse(X)-a.e.x. (4) If a stationary channel Vo E Cs(X, Y) is such that vo(x, .) « v(x,·) Pse(X)a.e.z, then Vo == v (modPse(X)). (5) v E exCs(X, Y) (modPse(X)). (6) For E, F E X Q9 ~ and J-t E Pse(X) it holds that n-l

lim 1 n-+oo n

L v(x, [(8

k=O

n-l

X

T)-k En F]a:)

.!. L v(x, [(8 X T)-k E]a:)v(x,Fa:) n-+oo n

= lim

k=O

= J-t Q9 v(E)v(x, F x)

u-a.e.x,


148

(7) For A, B E X, C, D E

~

and J-t E Pse(X) it holds that

n-I

lim.!:. n-+oo n

L r.

{v(x, T-kC n D) - v(x, T-kC)v(x, D)} JL(dx)

= O.

k=OJS-k AnB

Proof. (1) =} (2). Let J-t E Pse(X). Then J-t ® v E Pse(X x Y) by (1). Hence, if E E X ® ~ is S x T-invariant, then J-t ® v(E) = 0 or 1, i.e.,

L

v(x, Ex) JL(dx)

=0

or 1.

This implies v(x, Ex) = 0 u-a.e.x or v(x, Ex) = 1 u-a.e.x, Thus (2) holds. (2) =} (1). Let E E X ® ~ be S x T-invariant, J-t E Pse(X) and

A o = {x EX: v(x, Ex)

= o},

Al = {x EX: v(x,Ex) = I}. Note that T-IEsx = Ex since

T- I Es x = =

{Y E Y : Ty E {z

E Y : (Sx, z) E E}}

{y

E

E

Y : (Sx, Ty)

= {y E Y: (x,y) E (S

E}

x T)-IE

E} = Ex.

(4.1)

Henee A o is S-invariant since

S-IA o = {x EX: Sx E A o} = {x EX: v(Sx, Es x) = n} = {x EX: v(x, T-IESx) = O}, because v is stationary,

= {x

EX: v(x, Ex)

= O} = A o.

Similarly, we see that Al is S-invariant. Thus, J-t(A o) = 0 or 1, and J-t(A I) sinee J-t is ergodic. Consequently, we have that

JL @ v(E) =

= 0 or

1

L

v(x, Ex) JL(dx) = 0 or 1.

Therefore, J-t ® v is ergodic, (1) =} (3) is trivial since we can take 11.3.2 and Corollary 2.6 (1).

VI

=v

and (3)

=}

(1) follows from Theorem


149

(1) => (5). Suppose that (5) is false. Then for some Vi, V2 E Cs(X, Y) with Vi t= V2 (modPse(X)) and a E (0,1) we have V == aVl + (1 - a)v2 (modPse(X)). Hence J-l ® Vi :f. J-l ® V2 for some J-l E Pse(X) and p, 0 v(E)

= =

L L[

v(x, Ex) p,(dx) a vl (X,Ex) + (1- a)v2(X'Ex)] p,(dx)

= au Q9 vl(E) + (1 - a)J-l Q9 v2(E ),

E E X Q9~.

Thus J-l ® V = ap. ® Vi + (1 - a) J-l Q9 V2 is a nontrivial convex combination of two distinct stationary sources. Therfore, J-l Q9 u is not ergodic, which contradicts the ergodicity of u, (5) => (2). Assume that (2) is false. Then, there are some J-l E Pse(X) and some S x T-invariant E E XQ9~ such that J-l(A) > 0, where A = {x EX: v(x, Ex) #= 0, 1}. Define Vi and V2 by

v(x, C n Ex) v(x, Ex) , { v(x, C), v(x, C n E~) V2(X, C) = v(x, Ei) , { v(x, C),

Vi(x, C) =

xEA,

xEA,

where C E~. Clearly, Vi and V2 are channels. As in the proof of (2) => (1) we see that Ex = T-1Esx by (4.1) and hence

v(x, Ex) implying that

s:' A =

= v(x, T- l E s x) = v(Sx, Es x),

xE

A,

(4.2)

A. Thus, for x E A and C E ~

_ v(Sx,CnEs x) _ v(x,T-l(CnEsx)) Vi (S x, C) v(x, Ex) v(Sx, Es x) v(x, T- 1C n T- l Es x) v(x, T- 1C n Ex) v(x, Ex) v(x, Ex) = Vi(x, T- 1C). It follows that Vi is stationary. Similarly, one can verify that V2 is stationary. Moreover, we see that

xEA,


150

and hence VI "t V2 (modPse(X)). Note that for x E X and C E ~

Let B = {x EX: v(x,EaJ ~ ~} E

x and define V3 and V4 by xEB, x E BC, xEB, x E BC,

where C E ~. Obviously, V3, V4 E C(X, Y). Observe that S-1 B == Band S-1 BC = B C by (4.2). Then it is easily verified that V3 and V4 are stationary by S-invariance of Band BC, stationarity of VI and V2, and (4.2). Furthermore, we have by the definition of V3 and V4, and by (4.3) that

V4(X, Ex)

{2v(x, Ex) - 1 }Vl (x, Ex) + {2 - 2v(x, Ex)} V2(X, Ex) = 2v(x, Ex) - 1 < 1.

If x E A n BC, then

V3(X, Ex)

= {I - 2v(x, Ex) }V2(X, Ex) + 2v(x, E x)Vl(X, Ex) = 2v(x, Ex) > 0,

while V4(X, Ex) = V2(X, Ex} = o. Thus, V3(X, Ex) -=1= V4(X, Ex) for x E A. This means that V3 "t V4 (modPse(X)) and so V ~ exCs(X, Y) (modPse(X)). Therefore, (5) is not true, (4) =} (2). Suppose (2) is false. Then for some J-t E Pse(X) and E E x®~ with (S x T)-l E = E it holds that J-t(A) > 0, where A = {x EX: 0 < v(x, Ex) < I}. Define Vo by v(x,CnEx) x E A, C E~, vo(x, C) = v(x, Ex) , { lJ(x,C), x E AC, C E ~. Then we see that Vo E Cs(X, Y) and V "t lJo (rnodPse(X)) in a similar way as before. Moreover, we have lJo(x,·) ~ v(x,·) Pse(X)-a.e.x since D E ~ and v(x, D) = 0 imply Vo (x, D) = 0 for x EX. This contradicts (4).

151


(1), (2), (5) =} (4). Thus far we have proved (1) {:} (2) {:} (3) {:} (5). Assume (4) is false. Then there exists some Vo E Cs(X, Y) such that vo(x,·) ~ v(x,·) Pse(X)-a.e.x,

Vo

"# v

(modPse(X)).

Let fL E Pse(X) be arbitrary. By (2), if E E X ® ~ is 8 x T-invariant, then v(x, Ex) = 0 or 1 u-a.e.x.

Let VI = ~v + ~vo. Then, vI(x,Ex) = 0 or 1 u-a.e.x since v(x, Ex) = 0 u-a.e.x implies vo(x, Ex) = 0 u-a.e.x, and v(x, Ex) = 1 u-a.e.x implies vo(x, Ex) = 1 u-a.e.x, Thus VI is ergodic by (2), which contradicts (5). (1) =} (6). Suppose V is ergodic and let fL E Pse(X), Then fL ® v E Pse(X x Y) and hence for every E, F' E X ® ~ n-I

!L n-+oo n lim

p, @ v((8

X

T)-k En F ')

= p, @ II(E)p, @ v(F').

(4.4)

k=O

If we take F ' = F n (A x Y) for F E X ® ~ and A E X, then on the one hand n-I

lim! n-+oo

L Jfx v(x, [(8

X

T)-k En F n (A x Y)]",) p,(dx)

n k=O

= n-+oo lim ! n

n-I

Lp,@v((8 x T)-kEnFn (A x Y)) k=O

=fL®v(E)fL®V(Fn(AxY)),

= p, @ v(E)

L

by (4.4),

v(x,F",) p,(dx),

(4.5)

and on the other hand by Proposition 1.4

!

LHS of (4.4) = lim n-+oo

=

n-I

L

j

v(x, [(8

X

T)-k En F]x) fL(dx)

L v(x, [(8

X

T)-k En F]",) p,(dx).

n k=O A n-I

j

lim!

(4.6)

A n-+oo n k=O

Since (4.5) and (4.6) are equal for every A E X, one has the equation in (6). (6) =} (7). If E = A x C and F = B x D, then the equation in (6) reduces to n-I

! Lls-kAnB(X)V(x, T-kC n D) n-+oo n lim

k=O

== n-+oo lim ! n

n-I

L

k=O

lS-kAnB(X)V(x, T- kC)II(X, D) Pse(X)-a.e.x.

152


Integrating both sides w.r.t. It E Pse(X) over X, we get the equation in (7). (7) =* (1). Let It E Pse(X), A, B E X and C, D E ~. Then n-I

~E

{Jt ® v((S x T)-k(A x C) n (B x D)) - Jt e v(A x C)Jt ® v(B x D)}

k=O n-I

=

.!. E ti

{

k=O JS-k AnB

+~~

{vex, T-kC n D) - vex, T-kC)v(x, D)} Jt(dx)

{L lA(SkX)V(Sk x, C)lB(x)v(x,

-L

lA(X)V(X, C) Jt(dx)

D) Jt(dx)

L

IB(x)v(x, D) Jt(dX)}

-+0 (asn-+oo) by the assumption (7) and the ergodicity of It. Thus It ® v is ergodic. In (7) of Theorem 2, if we replace E = X x C and F =X x D, then (7) reduces to the condition (3.1). We now consider ergodicity of a stationary channel in terms of the channel operators it associates. So assume that X and Yare a pair of compact Hausdorff spaces with homeomorphisms Sand T, respectively.

Definition 3. Let K E JCs(X, Y), A E As(X, Y) and P ~ P(X). Then K is said to be extremal in JCs(X, Y) mod P, denoted K E exJCs(X, Y) (mod P), provided that, if K == aK I + (1 - a)K 2 (mod P) for some K I , K 2 E JCs(X, Y) and a E (0,1), then K I == K 2 (mod P). Similarly, A is said to be extremal in As(X, Y) mod P, denoted A E ex As (X, Y) (mod P), provided that, if A == aA I + (1- a)A 2 (mod P) for some AI, A 2 E As (X, Y) and a E (0,1), then Al == A 2 (mod P). Under these preparations, extremal operators A E As (X, Y) are characterized as follows:

Theorem 4. Let A E As (X, Y). Then the following conditions are equivalent: (1) For every i.s E C(X x Y) lim A(fng)(x) = lim Afn(x)Ag(x) Pse(X)-a.e.x,

n-+oo

n-+oo

1 n-I

where I« = (S®T)nf = - L (S®T)kf for n k=O (2) It E Pse(X) =* A*1t E Pse(X x Y).

ti

2:: 1.

(4.7)


(3) A

E

Proof. (1)

153

ex As (X, Y) (modPse(X)). =}

(2). Let J-L E Pse(X) and f, 9 E C(X x Y). Then

lim A*J-L(fng)

n-+oo

= n-+oo lim J-t(A(fng))

= n-+oo lim J-t(Afn' Ag),

by (1),

= n-+oo lim J-L(Af)nAg) , since A(S ® T) = SA, = J-t(Af)J-t(Ag),

since J-L is ergodic,

= A*J-L(f)A*J-t(g),

where (Af)n = Sn(Af),n ~ 1 (cf. Theorem 11.3.2). Thus A*J-L E Pse(X x Y) by Theorem 11.3.2. (2) =} (1). Suppose that (1) is false. Then it holds that for some J-L E Pse(X) and i.s E C(X x Y)

J-L({x EX: lim A(fng)(x) n-+oo

=I n-+oo lim Afn(x)Ag(x)}) > 0

or, equivalently, for some h E C(X)

(4.8) Since A*J-L is ergodic by (2), we have lim J-L(A(fng)· h)

n-+oo

= n-+oo lim J-t(A(fngh)) ,

by (al),

= lim A*J-t(fngh) n-+oo

= lim A*J-t(fn)A*J-L(gh) n-+oo

= lim J-t(Afn)J-L(Ag· h). n-+oo

(4.9)

Moreover, since J-L is ergodic, we also have

(4.10) (4.9) and (4.10) contradict (4.8). Thus (4.7) holds. (2) =} (3). Suppose A == aA 1 + (1 - a)A 2 (modPse(X)) for sorne AI, A 2 As (X, Y) and a E (0, 1). For J-t E Pse(X)

E


154

is also ergodic by (2). Then by Theorem 11.3.2, AiJL = A 2JL = A * JL E Pse(X x Y). Since this is true for every JL E Pse(X), we have Al == A 2 == A (modPse(X)) by Proposition 2.7. Thus (3) holds. (3) =} (2). Assume that (2) is not true. Then there exists a JL E Pse(X) such that A * JL is not ergodic. Hence there is some S x T-invariant set E E X ® ~ such that 0 < A * JL(E) < 1. Let Al = A * JL(E) and A2 = 1 - AI, and take 'Y so that o < 'Y < rnin{AI' A2}. Let Qi = (i = 1,2) and define operators AI, A 2 on B(X x Y) by

t

Alf = QIA(fIE) + (1

QIAIE)Af,

A 2f = Q2 A(fIEc) + (1 - Q2A1Ec)Af,

f f

E E

B(X xY), B(X x Y).

We shall show AI, A 2 E A(X, Y). All = 1 is clear. AI(fAIg) = (AIf)(AIg) for f, 9 E B(X x Y) is seen from the following computation:

+ (1- QIA1E)Ag}] = QIA[f{QIA(gIE) + (1- QIAIE)Ag}IE] + (1 QIAIE)A[f{QIA(gIE) + (1- QIAIE)Ag}] = Q~A[fIEA(gIE)] + QIA[f IE(l- QIAIE)Ag] + QI(I- QIAIE)A[fA(glE)] + (1- QIAIE)A[f(I- QIAIE)Ag] = Q~A(fIE)A(gIE) + QIA[f 1E(I- QIAIE)]Ag + QI(I QIAIE)Af· A(gIE) + (1- QIAIE)A[f(I- QIAIE)]Ag

AI(fAIg) = Al [f{Q IA(glE)

= Q~A(fIE)A(gIE)

+ QI(I -

+ QIA(f1E)(I - QIAIE)Ag

QIAIE)Af· A(gIE)

+ (1 -

QIAIE)2Af' Ag = [QIA(fIE) + (1- QIAIE)Af] [QIA(gIE) + (1 - QIAIE)Ag] = (AIf)(AIg), where we used A(fAg) = (Af)(Ag). Similarly, we can verify (al) for A 2. Moreover, (a3) is clearly satisfied by AI, A 2, which is seen from their definitions. Thus AI, A 2 E A(X, Y). Furthermore, AI, A 2 E As (X, Y) since for f E B(X x Y) we have AI(S ® T)f

= QIA(((S ® T)f)IE) + (1

QIA1E)A(S ® T)f

= QIA((S ® T)(fIE)) + (1- QIAIE)SAf, since A(S ® T) = SA and E is S x T-invariant, = QISA(fIE) = SAlf

+ (1 -

QIAIE)SAf


155

and similarly A 2 (S ® T)f = SA 2f. Now we show that A 1 t:. A 2 (modPse(X)). In fact, we have

Jlt(A 21Ec - A 11Ec) = Jlt(a2A1Ec

+ (1- a2A1Ec)A1Ec

- a1A(lEc IE) - (1 - a1A1E )A1Ec )

+ A1Ec(a1A1E - a2 A 1Ec)) = a2A*Jlt(EC) + Jlt{A1Ec(a1 A 1E - a2 A 1Ec)) = a2A2 + Jlt(A1Ec )Jlt(a1 A 1E - a2 A 1Ec), since Sn(A1Ec) = AlEc and Jlt is ergodic, = Jlt(a2A1Ec

+ A2(a1A1 - a2 A2) ')' + A2(')' - ')') > O.

= a2A2

=

(4.11)

Finally we see that A1A1 + A2A2 = A since for f E B(X x Y)

A1 A1f + A2A2f = ')'A(f1E) + (A1 - ')'A1E)Af = ')'Af + (1 - ')')Af = Af.

+ ')'A(f1Ec) + (A2

- ')'A1Ec )Af (4.12)

These results imply that A ~ exAs(X, Y) (modPse(X)), a contradiction. As a corollary we have: Corollary 5. Let v E Cs(X, Y) be a stationary channel and K E Ks(X, Y) and A E As(X, Y) be the corresponding channel operators to t/, Then the following

statements are equivalent: (1) v is ergodic. (2) u E exCs(X, Y) (modPse(X)). (3) K E exKs(X, Y) (modPse(X)). (4) A E ex As(X, Y) (modPse(X)). (5) For f, 9 E C(X x Y) it holds that lim A(fng)(x) = lim Afn(x)Ag(x) Pse(X)-a.e.x,

n-too

n-too

where fn = (S ® T)nf, n ~ 1. (6) For t, 9 E B(X x Y) the equality in (5) holds. (7) For E, F E X ® ~ it holds that for Pse(X)-a.e.x n-1

n-1

k=O

k=O

.!. '" v(x, [(8 X T)-k En F]x) = n-too lim .!. '" v(x, [(8 X T)-k E]x)v(x, Fx). n-too n Z:: n L....t lim


156

Proof. (1) ¢:> (2) ¢:> (3) ¢:> (4) ¢:> (5) follow from Theorems 2.1, 2.2, 2 and 4. (5) => (6) is proved by a suitable approximation. (6) => (7) is derived by taking f = IE and 9 IF for E, F E X ®~. (7) => (6) follows from the fact that {IE: E E X ®~} spans B(X x Y). (6) => (5) is trivial. Remark 6. Observe that each of the following conditions is not sufficient for ergodicity of a stationary channel v E Cs(X, Y), where (X, X, S) and (Y,~, T) are a pair of abstract measurable spaces with measurable transformations: (1) v(x,') E Pae(Y) Pse(X)-a.e.x.

(2) v(x,·) E Pse(Y) Pse(X)-a.e.x. (3) v = v", for rJ E Pse(Y), v", being the constant channel determined by n. In fact, if X = Y, rJ E Pse(X) is not WM, and v", is ergodic, then rJ ® v", = rJ x rJ is ergodic, which implies rJ is WM by Theorem 11.3.10, a contradiction.

3.5. AMS channels AMS sources were considered as an extension of stationary sources in Section 2.4. In this section, AMS channels are defined and studied as a generalization of stationary channels. A characterization of AMS channels is obtained as well as that of ergodic AMS channels. Absolute continuity of measures plays an important role in this section. Let X, Y be a pair of compact Hausdorff spaces with Baire a-algebras X, ~ and homeomorphisms S, T, respectively. The invertibility assumption is crucial. Assume that ~ has a countable generator ~o, i.e., ~o is countable and a(Vo) =~. Recall that Pa(O) and Pae(O) stand for the set of all AMS sources in P(!~) and the set of all AMS ergodic sources in Pa(!~) for 0 = X, Y and X x Y, respectively.

Definition 1. A channel v E C(X, Y) is said to be asymptotically mean stationary (AMS) if (cI4) I-t E Pa(X) =>

V

® I-t E Pa(X x Y).

That is, if the input source is AMS, then the compound source is also AMS. Ca(X, Y) denotes the set of all AMS channels. First we need the following two lemmas.

Lemma 2. A channel u E C(X, Y) is AMS iff I-t E Ps(X) => I-t ® v E Pa(X x Y).

Proof. The "only if' part is obvious. As to the "if' part, let I-t

E

Pa(X). Then

157

3.5. AMS channels

Ii

E

Ps(X) and J-L

~

Ii by Remark 11.4.9. Hence, J-L Q!) v

« Ii Q!) v ~ Ii Q!) u,

where the first « is clear and the second follows from Remark 11.4.9 and the assumption since Ii E Ps(X). Thus J-L Q!) v E Pa(X x Y) by Proposition 11.4.8.

Lemma 3. Let u E C(X, Y) and J-L E P(X) be such that J-LQ!)v E Pa(X x Y). Then: (1) J-L E Pa(X) with the stationary mean Ii(') = J-L Q!) v(· x Y) E Ps(X). (2) J-Lv EPa(Y) with the stationary mean J-Lv(,) = J-L Q!) v(X x .) E Ps(Y). (3) v(x,·) ~ J-LV u-o:e.x.

x

Proof. (1) Observe that for A E n-I

~L n-+oo n lim

k=O

n-I

It(S-k A)

= n-+oo lim ~ " It @ v(S n L..,;

x T)-k(A x Y))

k=O

= J-L Q!) v(A x Y). Thus J-L is AMS with the desired stationary mean. (2) is verified similarly. (3) Suppose that J-Lv(C) = O. Then J-Lv(C) = 0 since J-LV «J-Lv. Hence Itv(C)

=

i

v(x, C) It(dx)

= O.

This irnplies that v(x, C) = 0 p-a.e.x, completing the proof. An important consequence of these two lemmas is: Corollary 4. For a channel u E C(X, Y) the following conditions are equivalent: (1) v is AMS, i.e., u E Ca(X, Y). (2) For each stationary J-L E Ps(X) there exists a stationary channel VI E Cs(X, Y) such that (5.1) V(x,·) « VI(X,') u-a.e.x.

(3) For each stationary J-L E Ps(X) there exists an AMS channel such that (5.1) holds.

VI

E

Ca(X, Y)

Proof. (1):::} (2). Suppose v is AMS and let J-L E Ps(X). Then J-LV E Pa(Y) by Lemma 3 (2) since J-L Q!) u EPa(X x Y). If we let VI (x, C) = J-Lv(C). for x E X and C E !D, then VI is a constant stationary channel and (5.1) is true by Lemma 3 (3). (2) :::} (3) is immediate.


158

(3) :::::> (1). Let u E Ps(X) and suppose the existence of VI mentioned. Then VI E Pa(X x Y) and J-t ® V « J-t ® VI by (5.1). Hence J-t ® V E Pa(X x Y) by Proposition 11.4.8. Thus u is AMS by Lemma 2.

J-t ®

Now we want to consider an analogy of stationary means of AMS channels. Suppose that v E Ca(X, Y) and J-t E Ps(X). Then J-t ® v is AMS. Observe the following computation: for A E X and C E ~ n-l

n-l

1 ' " P, ® v((S x T)-k(A x C)) n LJ k=O

= .!.n L p, ® v(S-k A x T-kC) k=O

n-l

= .!. L n

(

vex, T-kC) p,(dx)

k=O}S-kA

n-l

=.!. L

n k=O

1

1

v(S-kx,T-kC)p,(d(S-kx))

A

n-l

= .!. L v(S-k x, T-kC) p,(dx) A

=

n

L

k=O

vn(x, C) p,(dx),

say,

== J-t® vn(A x C) -+ J-t ® v(A x C) (n -+ 00). Clearly, each V n (n 2:: 1) is a channel, but not necessarily stationary. The sequence {v n } is expected to converge to some stationary channel fJ, to be called again a "stationary mean" of u, We shall prove the following.

Proposition 5. A channel v E C(X, Y) is AMS iff for each stationary input source J-t E Ps(X) there exists a stationary channel v E Cs(X, Y) such that for every C E ~ 1 v(x, C) == lim n-+oo n

n-l

L

v(S-k x, T-kC)

u-a.e.x.

(~.2)

k=O

In this case it holds that (5.3)

Proof. Assume that v E Ca(X, Y) and J-t E Ps(X). Let "1(.) == J-tv(·). Then, by Lemma 3, "1 E Ps(Y) and v(x,') «"1 u-a.e.x, Let Xl == {x EX: v(x,·) « "1}, so that J-t(X I ) == 1. Moreover, if X* n S" Xl, then X* is S-invariant and J-t(X*) == 1 nEZ

159

3.5. AMS channels

since 1-£ is stationary. Let

k(x, y)

=

{

v(x,dy) 'f/(dy)'

x E X*,y E Y,

0,

x ¢ X*, Y E Y.

Then we have that k E L 1(X X Y, 1-£ ® v) by Theorem IV.2.5 and Remark IV.2.6, which are independent of this proposition, since 1-£ ® v « 1-£ x "7, and

vex,C)

=

Ie

k(x, y) 'f/(dy) ,

Now one has for x E X* and C E

-1 L

n-1

n

v(S-k x , T-kC)

~

1 n-1 = -

L1

n

k=O

k=O

n-1

= .!:. L

XEX*,CE~.

k(S-k x , y) "7(dy)

T-kC

1

k(S-kx, T-ky) 'f/(dy),

n k=O

since 'f/ is stationary,

C

n-1

= 11c(Y).!:. n

y

L k(S-kx, T-ky) 'f/(dy)

k=O

= [lc(y)[(S-l @T-1)nk](x,Y)'f/(dy).

By the Pointwise Ergodic Theorem there is an S x T-invariant function

(8- 1 ® T-1)nk -+

k*

k*

such that

1-£ ® u-a.e.

since k is jointly measurable. Hence we see that n-1

lim

.!:. L

n--+oo n k=O

since "7(.)

= 1-£ ® v(X

v(S-k x, T-kC)

=

rlc(y)k*(x, y) 'f/(dy)

u-a.e.x

}Y

x .). Take a stationary channel u" E Cs(X, Y) and define D by

D(x,C) = {

Ie

k*(x, y) 'f/(dy) ,

XEX*,CE~,

x¢X*,CE~.

v*(x, C),

Then, clearly Dis a stationary channel and (5.2) holds. By the Bounded Convergence Theorem we have from (5.2) that n-1

p, @ v(A x C)

= n--+oo lim .!:. '" p, @ v((S x T)-k(A x C)) n L.....t k=O


160

for A E X and C E~. Thus, since j.L Q9 1) is stationary, j.L Q9 v is AMS with the stationary mean j.L Q9 1), i.e., (5.3) is established. In Proposition 5, the stationary channel 1) depends on the given AMS channel v and the stationary input source u: We would like to have a single stationary channel independent of input sources such that (5.2) and (5.3) are true for all stationary j.L E Ps(X). This will be obtained using the countable generator ~o of~.

Theorem 6. For any AMS channel v E Ca(X, Y) there is a stationary channel YJ E Cs(X, Y) such that for any stationary input source j.L E Ps(X) 1

YJ(x, C)

= n-too lim n

n-l

L v(S-k x, T-kC)

u-a.e.x, C E ~,

(5.4)

k=O

(5.5)

j.LQ9v = j.LQ9YJ, v(x,·) «YJ(x,·)

u-a.e.x,

(5.6)

Proof. Let v E Ca(X, Y) and X(C)

=

{

x EX: lim

n-too

} .!.n n-l ~ v(S-kx, T-kC) exists L..J

,

CE~O,

k=O

X(v)

=

n

X(C).

CE~o

Then for any stationary j.L E Ps(X) we have j.L(X(v)) = 1

since j.L (X(C)) = 1 for C· E ~o by Proposition 5 and ~o is countable. Take a stationary channel v* E Cs(X, Y) and define a set function YJ on X x ~o by I

_() v x, C

=

n-l

lim - I: v(S-k x, T-kC), n-too n k=O { v*(x, C),

x E X(v),C E ~o, x fJ. X(v),C E ~o.

It is evident that YJ can be extended to X x ~ and becomes a stationary channel. Thus (5.4) is satisfied. (5.5) is shown by the Bounded Convergence Theorem and (5.6) is almost obvious.

Definition 7.. The stationary channel YJ constructed above"is called a stationary mean of the AMS channel v E Ca(X, Y).

161

3.5. AMS channels

The following is a collection of equivalence conditions for a channel to be AMS. Recall the notations s; and (8 ® T)n'

Theorem 8. For a channel u E C(X, Y) the following conditions are equivalent: (1) v E Ca(X, Y), i.e., u is AMB. (2) J-L E Ps(X) => J-L ® v E Pa(X x Y). (3) There is a stationary VI E Cs(X, Y) such that v(x,·) ~ Vl(X,') Ps(X)-a.e.x. (4) There is an AMB VI E Ca(X, Y) such that v(x,·) ~ VI(X,') Ps(X)-a.e.x. (5) There is a stationary Ii E Cs(X, Y) such that for C E ~ 1 n-l lim - "v(S-k x, T-kC) = vi», C) n4-OO n L....J

Ps(X)-a.e.x.

k=O

(6) For f E B(X x Y) and lim n4-OO

J-L E

Ps(X) the following limit exists:

r

l» [Av(S Q9 T)nf] (x) JL(dx).

If any (and hence all) of the above is true, then it holds that

(7) J-L ® v = J-L ® Ii for J-L E Ps(X). (8) v(x,·) ~ Ii(x,') Ps(X)-a.e.x. (9) For J-L E Ps(X) and f E B(X x Y) lim n4-OO

r [Av(S

Jx

Q9 T)nf]

(x) JL(dx)

=

r

Jx (Avf)(x) JL(dx).

Proof. (1) {::} (2) was proved in Lemma 2 and (1) {::} (5) follows from Theorem 6. By taking VI = Ii, (3) is derived from (5). (3) => (4) is immediate and (4) => (1) is proved in Corollary 4. Thus we proved (1) {::} (2) {::} (3) {::} (4) {::} (5). (2) {::} (6). Observe that for f E B(X x Y) and J-L E Ps(X)

L

[Av(S

Q9

T)nf] (x) JL(dx) = =

L[(S j"{

Q9

T)nf(x, y) v(x, dY)JL(dx)

(8 ® T)nf(x, y)

J-L ®

v(dx, dy)

JXXY

by Lemrna 2.5. The equivalence follows from Theorem 11.4.6. (7) and (8) are already noted in Theorem 6. As to (9) we proceed as follows: for f E B(X x Y) and J-L E Ps(X)

L

(Avf) (x) JL(dx)

=

Ll>

y)IJ(x, dY)JL(dx)


162

=

j"( jr(

f(x, y) Jj ® v(dx, dy),

by Lemma 2.5,

f(x, y) p,(ji'y v(dx, dy),

by (7),

JXXY

=

JXXY

= lim n---too

jf"{ (8 ® T)nf(x, y) Jj ® v(dx, dy) JXXY

= n---tooJx lim ( [Av(S Q9 T)nf](x) tL(dx). Example 9. (1) As was rnentioned before, each probability measure 'fJ E P(Y) can be regarded as a channel by letting v'TJ(x,C) = 'fJ(C) for x E X and C E~. If 'fJ E Pa(X), then v'TJ is AMS. In fact, 'fJ «rj since T is invertible, so that v'TJ(x,·) = 'fJ «rj = vi](x,·) for x E X. Moreover, vi] E Cs(X, Y) implies that v'TJ E Ca(X, Y) by Theorem 8 (3). In this case we have v'TJ = vi]. (2) If a channel u E C(X, Y) satisfies that v(x,·) «'fJ

Ps(X)-a.e.x

for some AMS 'fJ E Pa(Y), then v is AMS by Theorem 8 (4) and (1) above. Let

k(

x,y

) = vex, dy)

(x,y) E X x Y,

rj(dy) ,

where rj E P; (Y) is the stationary mean of n. Then v can be written as

v(x,C) and its stationary mean

= [ k(x,y)'fj(dy),

~

v as

v(x, C) = [k*(X, y) 'fj(dy) where k*(x, y)

x E X,C E

= n---too lim (8- 1 ® T-1)nk(x, y)

P.(X)-a.e.x, C Jj

E !l),

® v-a.e. (x, y) for

Jj E

Ps(X).

(3) A channel v E C(X, Y) satisfying the following conditions is AMS: (i) v is dominated, i.e., v(x,·) « 1/ (x E X) for some 'fJ E P(Y); (ii) v(x,·) E Pa(Y) for every x E X. In fact, u is strongly measurable by Corollary IV.2.3 since v is dominated and ~ has a countable generator. Hence v has a separable range in M(Y), so that {v(x n , · ) : n ~ I} is dense in its range for some {x n } ~ X. Let

(.) =

t n=l

v(~:, .).

163

3.5. AMS channels

Then we see that € E Pa(Y) by (ii) and Lemma 11.4.3(1). Thus v is AMS by (2) above since vex, .) ~ € (x EX).

Definition 10. An AMS channel v E Ca(X, Y) is said to be ergodic if (cI5)

J-L E

Pae(X) =>

J-L

® v E Pae(X x Y).

Cae (X, Y) denotes the set of all ergodic AMS channels in Ca(X, Y). After giving a lemma we have the following characterization of AMS ergodic channels. Sorne of the equivalence conditions are similar to those of AMS ergodic sources and some are to those of stationary ergodic channels.

Lemma 11. Let v E Ca(X, Y) be AMS and J-L E Ps(X) be stationary. Then for every E, F E X ® q) the following limit exists p-a.e.x: n-I

lim

n.-+-oo

.!.n "'v(x,[(8xT)-kEnF]x). L....J k=O

Proof. Since v is AMS and J-L is stationary, J-L ® v is AMS with proof parallels that of Proposition 1.4 and we have

J-L

v=

J-L

® 17. The

n-I

lim

n.-+-oo

.!.n E v(x, [(8 X T)-k En F]x) k=O

=

where J

= {E E X® q)

i

IF(X, y)EI'0v(lEIJ)(x, y) vex, dy) u-a.e.x,

: (8 x T)-IE

= E}.

Theorem 12. Let v E Ca(X, Y) with the stationary mean 17 E Cs(X, Y). Then the following conditions are equivalent: (1) v E Cae (X, Y), i.e., v is ergodic. (2) J-L E Pse(X) => J-L ® v E Pae(X x Y). (3) 17 E Cse(X, Y). (4) There is a stationary ergodic VI E Cse(X, Y) such that

v(x,·) «VI(X,')

Pse(X)-a.e.x.

(5.7)

(5) There is an AMS ergodic VI E Cae (X, Y) such that (5.7) is true. (6) If E E X®q) is 8 x T-invariant, then v(x,Ex ) = 0 or 1 Pse(X)-a.e.x.


164

(7) For E, F

E

X Q9 q) and J-t

E

Pse(X)

1 n-l lim - """ v(x, [(8 x T)-kEnF]x) n4-OO n L...J

=

k=O

1 n-l lirn - L17(x, [(8 x T)-kE]x)v(x,Fx) n4-OO n k=O

= J-t Q917(E)v(x, Fx)

u-a.e.x,

Proof. (1)

=} (2) is clear. (2) =} (3). Suppose (2) is true and let J-t E Pse(X). Then J-t Q9 u E Pae(X x Y). Hence J-t ® 17 = J-t v E Pse(X x Y) by Theorem 11.4.12 and Theorem 8 (7). Thus 17 is ergodic. (3) =} (4). Take VI = 17 and invoke Theorem 8. (4) =} (5) is immediate. (5) =} (6). Assume (5) is true. Let E E X Q9 q) be 8 x T-invariant and take J-t E Pse(X). Then J-t ® VI E Pae(X x Y) and J-t Q9 u ~ J-t Q9 VI· If J-t Q9 vl(E) = 0, then 0 = J-t Q9 v(E) = Ix v(x, Ex) J-t(dx) , so that v(x, Ex) = 0 u-a.e.x, Similarly, if J-t ® VI(E) = 1, then we can show that v(x, Ex) = 1 u-a.e.x. (6) =} (1). Let J-t E Pae(X). Suppose that E E X Q9 q) is 8 x T-invariant and v(x, Ex) = 0 ii-a.e.x and hence u-a.e.x, Then

p, @ veE) =

Ix

v(x, Ex) p,(dx)

= o.

If v(x, Ex) = 1 ii-a.e.x, then we have J-tQ9v(E) = 1. Thus J-tQ9V is ergodic. Therefore v is ergodic. (1) =} (7). This is shown in a similar manner as the proof of (1) =} (6) of Theorem 4.2 using Lemma 11. (7) =} (1). Let J-t E Pse(X) and E, F E X Q9 q). By integrating both sides of the equation in (7) w.r.t. J-t over X, we get

Hence J-t Q9 v is AMS ergodic. Therefore v is AMS ergodic. We noted that exPa(X) ~ Pae(X) and the set inclusion is proper (cf. Theorem 11.4.14). Similarly we can prove the following. Theorem 13. (1) If v E exCa(X, Y) (modPse(X)), then v E Cae(X, Y). Thus ex c; (X, Y) ~ Cae (X, Y).

165

3.5. AMS channels

(2) If there exists a weakly mixing source in Pse(Y), then the above set inclusion is proper, in that there exists some AMS ergodic channel v E Cae (X, Y) such that v

rt exCa(X, Y)

(modPse(X)).

Proof. (1) Let v E Ca(X, Y), I1 E Cs(X, Y) be its stationary mean and u B A E A(X, Y) be the corresponding channel operator. Suppose that v rt Cae (X, Y). Then there is some J-L E Pse(X) such that A * J-L rt Pae(X X Y). Hence there is some S x Tinvariant set E E X Q9 ~ such that 0 < Al == A * J-L(E) < 1. Letting A2 = 1 - AI, take ~ > 0 so that 0 < ~ < Inin{Al, A2}. Let ai = (i = 1,2) and define operators AI, A 2 on B(X x Y) by

Xi

= 0!1A(f1E) + (1 - a lA1E)Af, A 2f = a2 A(f1Ec) + (1 - a2A1Ec )Af, Alf

f f

x Y), B(X x Y).

E B(X E

Then as in the proof of (3) => (2) of Theorem 4.4 we see that AI, A 2 E A(X, Y). It follows from (4.11) that Al 1=. A 2 (modPse(X)). Moreover, A is a proper convex combination of Al and A 2 :

which follows from (4.12). Now we want to show that VI and Observe that for f E B(X x Y)

L

V2

are AMS channels, where

Vi B

Ai (i = 1,2).

[A1(S Q9 T)nf](x) /-L(dx)

= =

L

[a1(A(S Q9 T)nf 1E) + (1 ~ a 1 A 1E)A (S Q9 T)nf] (x) /-L(dx)

L

[a1A(S

Q9

T)n(j1E)

because E is S x T-invariant. Since

r,«

+ (1 -

a 1 A 1E)A (S Q9 T)nf] (x) /-L(dx)

v is AMS, n-+oo lim Ix alA(S Q9 T)n(f1E) dJ-L exists

by Theorem 8 (6). Also lim a lA1E)A(S Q9 T)nf dJ-L exists by Theorem 8 (6) n-+oo and the Bounded Convergence Theorem. Thus, we proved that

exists for every f E B(X x Y) and hence VI is AMS by Theorem 8 (6).. Similarly, is AMS. Consequently we see that u rt exCa(X, Y) (modPse(X)).

V2


166

(2) Take an

1]

E

Pse(Y) that is WM and define

~(C) = L9dT/,

eby

CE~,

where 9 E L l(Y,1]) is nonnegative with norm 1 which is not T-invariant on a set of positive 1] measure. Then, as in the proof of Theorem 11.4.14 we see that e E Pae(Y) , i= 1], ~ = 1] and ( == ~(e + 1]) E Pae(Y) isa proper convex combination of two distinct AMS sources. Hence

e

"c ¢ exCa(X, Y)

(modPse(X))

since "c = ~(ve + v.,,), ve, v." E Ca(X, Y) and iJe i= v.". We need to show "c E Cae (X, Y). Clearly v( = v, = v." E Cse(X, Y) since J-t ® Vn = J-t x 1] E Pse(X x Y) for J-t E Pse(X) by Theorem 11.3.10. Thus "c E Cae (X, Y) by Theorem 12. If the output space is an alphabet message space Y = Yl~ with a shift transformation T, then there exists a Bernoulli source that is strongly mixing (cf. Example 11.3.7), so the assumption in Theorem 13 (2) is satisfied.

3.6. Capacity and transmission rate

For a stationary channel we define the transmission rate functional and the stationary and ergodic capacities. An integral representation of the transmission rate functional is given. For a stationary ergodic channel the coincidence of two capacities is shown. First we deal with the alphabet message space case and then the general case. Consider alphabet message spaces X = X~ and Y = Y oZ with shifts Sand T, respectively, where X o = {at, ... ,ap } and Yo = {b l , ... ,b q } . For each n ~ 1, ~(X) denotes the set of all messages in X of length n starting at time i of the form [X~k) .•. X~~n-l], 1 ~ k ~ pn. Similarly, we denote by VJt~ (Y) and VJt~ (X x Y) the sets of all messages in Y and X x

Y of length n starting at time i, respectively. Note that

.

n-l..

~(X) = .V

3=0

S-3VJti(X)

E

P(X) for n ~ 1 and i E Z. Let a channel u E C(X, Y) be given. For each input source J-t E P(X) we associate the output source J-tV E P(Y) and the compound source J-t ® v E P(X x Y). The mutual information In (X, Y) = In(J-t; v) between the two finite schema (VJt~(X), J-t) and (VJt~(Y),J-tv) is given by In (X, Y) = In(J-t; v)


167

(cf. (1.1.2)), where

HJL(VJt~(X)) = -

L

M(A) logM(A),

AEVJt~(X)

etc. Then, ~ In (M ; v) is considered as an average information of one symbol (or letter) when messages of length n in the input source [X, J-L] are sent through the channel u, If the limit

since

V

n=-oo

with a

snVJt~(X) = X, etc. In this case note that I(· ; v) is affine: for a, (3 ~ 0

+ {3 =

1 and M, rJ E Ps(X)

I(aM + {3rJ; v)

= aI(M; v) + {3I(rJ; v),

which follows from Theorem 1.5.6 (or Lemma 1.5.1). The,.. stationary capacity and the are respectively defined by

Cs(v) = sup {I(M; v) : ME Ps(X)}, Ce(v) = sup {I(J-L; v) : ME Pse(X) and MQ9 v

Pse(X x Y)},

E

where if MQ9 v ¢ Pse(X x Y) for every ME Pse(X), then let Ce(v) = O. Let us mention the weak* upper semicontinuity of the entropy functional. For n ~ 1 let Hn(M) = HJL(VJt~(X)). Then we have that for each M E Ps(X)

n n lim Hn(p,) n

n~oo

~

1, k

= H (8). JL

~

1,

~

1,

(6.1)


168

In particular, (6.1) implies that

Hkn(J-L) < Hn(J-L) kn n '

n

2:: 1,k 2:: 1,

so that

H ( ) > H 2(J-L) > H 22(J-L) > ... > H 2n(J-L) > ... , 1 J-L 2 22 2n i.e, H 2;} p,) t Hp,(S). This shows that H(.)(S) is a weak* upper semicontinuous function on Ps(X) c C(X)* since each H n(·) is so. Although this was proved in Lemma II. 7.3, we shall use this method to prove the part (2) (i) of the following theorem, which summarizes the fundamental results regarding transmission rates and capacities in the alphabet message space case.

Theorem 1. Let v E Cs(X~, Yl~) be a stationary channel. (1) 0 ~ Ce(v) ~ Cs(v) ~ logpq, where p = IXol and q = IYol. (2) If v has a finite memory (cf. Definition 1.1 (c5)) and is finitely dependent (cf. Definition 3.3 (c8)), then: (i) 1(· ; v) is upper semicontinuous in the weak* topology of Ps(X) C C(X)*; (ii) Cs(v) = Ce(v); (iii) There exists a stationary ergodic input source J-L* E Pse(X) such that Ce(v) = I(J-L*;v). (3) Cs(v) = Ce(v) if v is ergodic.

Proof. All the statements of Theorem 1 (except (2)(i)) follow from the discussion below (see Theorem 8), where we consider a more general setting. So we prove (2)(i). Since v has finite memory, it is continuous (Definition 3.1, (c5')), so that Hp,v(T) is a weak* upper semicontinuous function of J-L by Proposition 1.6 and Lemma 11.7.3. Hence, it is sufficient to show that Hp,(S) - Hp,®v(S X T) is a weak* upper semicontinuous function of J-L on Ps(X). Since v has finite memory and is finitely dependent, there exists a positive integer m such that (c5) and (c8) hold. By (c5) we have that for any message C = [Ym+l ... Yn](m + 1 ~ n) C Y

v(X, C) = v(x', C),

x,x'

E

A = [Xl'· ·Xn ] .

We denote this common value by v(A, C). For n 2:: 1 and J-L E Ps(X) let

fn(J-L) =

-!. [L J-L ® v(A x C) log J-L ® v(A x C) n

L J-L(A) log J-L(A)] ,

A,C

A

where the sum is taken for A E 9R~(X) and C E 9R:!~(Y). Observe that

fn(P,) =

~ [LV(A,C)P,(A)lOgV(A,C)P,(A) A,C

LP,(A)lOgP,(A)] A

(6.2)

169


1

=-

2: v(A, C)J.L(A) logv(A, C).

(6.3)

n A,C

= [Xl" 'X n] E VJt~(X) let A' = [Xm+l" .xn] E VJt~!~(X) and for C = [Ym+l" 'Yn] E VJt~!~(Y) let C' = [y~ .. 'y!mYm+I" 'Yn] E VJt~(Y). Then one has for J.L E Ps(X) For A

J.L ® v(A xC') :::; J.L ® v(A x C) :::; J.L ® v(A' x C). It follows that for n ~ m

+ 1 and J.L E Ps(X)

2: J.L ® v(A xC') log rz ® v(A xC') : :; 2: J.L ® v(A xC') logJ.L ® v(A x C)

-Hn(J.L ® v) =

A,C'

A,C'

=

2: J.L ® v(A x C) log J.L ® v(A x C) A,C

= Gn(J.L),

(6.4)

say,

< 2: J.L e v(A x C) log®v(A' x C) A,C

=

2: J.L ® v(A' x C) log u ® v(A' x C) 2: J.L ® v(smA' x TmC) log J.L ® v(smA' x TmC)

A',C A',C

= -Hn-m(J.L ® v).

(6.5)

Hence (6.2) and (6.4) imply

fn(P,)

= !n (G n (p,) + Hn(p,» ~ n1 (Hn(p,) - H n (p, ® lI»,

while (6.2) and (6.5) imply

These two yield that liminf fn(P,) n-+oo

~ n-+oo lim !(Hn(p,) n

Hn(p, ® lI»

= Hp,(S) - HJL®v(S X T)

170


lim sup fn(tt) n-+oo

~ n-+oo lim ! (Hn(tt) n

Hn-m(tt ® v))

= lim ~Hn(J-t) - lim n

n = HJ.t(S) n-+oo

n-+oo

HJ.t®v(S

X

n

m. _ l - Hn_ m (J-t ® v) n- m

T).

So we conclude that lim fn(J-t)·= HJ.t(S) - HJ.t®v(S x T),

n-+oo

Note that fn(·) is a weak* continuous function on Ps(X) for each n ~ 1. To prove the weak* upper semicontinuity of HJ.t(S) - Hp,®v(S X T) it suffices to show that {fn(J-t)} contains a monotonely decreasing subsequence. Let f ~ 1 be arbitrary and n = 2(m + f). Denote a message A = [Xl··· Xn] E 9Jt~(X) by A = [Xl· .. Xm+l] n [X m+l+1 ... Xn] = Al n A 2 , say. Similarly, we write C = [Ym+1

Yn]

Ym+l] n [Ym+i+1 ... Y2m+l] n [Y2m+l+1 ... Y2(m+l)] n c' n C 2 , say.

= [Ym+1 = C1

Since v has a finite memory and is finitely dependent, one has

v(A,O) = v(A, 0 1 n Of n O2)

< v(A, C 1 n O2 ) = v(A, 01)v(A, O2), where V(A,Ol n C 2) stands for the common value of v(x, 0 1 n O2) for X E A. Now from (6.3) and the above we see that

fn(J-t) = f2(m+l)(J-t) 1

L L

< 2(m + l)

v(A 1 , C 1)v(A2 , C 2)tt(A1 n A 2 ) logv(A 1 , C1)v(A2 , C 2 )

A1,C1 A 2,C2

=

1 m+.f..

--0

L

v(A 1, C 1 )J-t(A 1 ) logv(A 1, 0 1 )

A1,C1

= fm+l(J-t), so that fm+l(J-t) .~ f2(m+l)(J-t). Since this holds for every f = 1,2, ... , we can choose a monotonely decreasing subsequence {fnk(J-t)}k=l for each J-t E Ps(X).

171


Now we assume that X and Yare totally disconnected compact Hausdorff spaces with the bases X o and ~o of clopen sets and homeomorphisms Sand T, respectively. As before X and ~ stand for the Baire o-algebras of X and Y, respectively. Let 2t E P(Xo) and ~ E P(~o) be fixed clopen partitions of X and Y, respectively. Q: = 2t x ~ denotes the clopen partition {A x B : A E 21,B E ~} of X x Y. We consider three entropy functionals: H 1 (1t) = H(It, 21, B),

H 2 ( ", ) = H("" H3(~)

~,T),

= H(~, c,S x T),

It E Ms(X), '" E ~ E

Ms(Y), Ms(X x Y).

(6.6)

By Theorem 11.7.6 there are B-, T- and B x T-invariant nonnegative measurable functions hi on X, h 2 on Y and h3 on X x Y, respectively such that

H 1(p,)

=

L

h1(x) p,(dx),

H2(11) =

1, 2(y) 11(dy),

H3{~) =

jr(lxxY h3(X,y) ~(dx, dy),

h

~ E

Ms(X x Y).

Definition 2. Let v E Cs(X, Y) be a stationary channel and It E Ps(X) be a stationary source. Then, the transmission rate v:t It; v = v:t It; v, 21, ~ of the channel v w.r.t. by

where HI, H 2 and H 3 are defined by (6.6). Hence v:t(. ; v) is called the transmission rate junctional of v on Ps(X) or on Ms(X). Hereafter, we shall use the letter v:t for the transmission rate functional. We can obtain an integral representation of the transmission rate functional

v:t(. ; v). Proposition 3. Let v E Cs(X, Y) be a stationary channel. Then the transmission rate junctional v:t(. ; It) is a bounded positive linear junctional on M; (X) and there is a universal B-invariant bounded Baire junction t{·) on X such that (6.7)

172


Proof. Observe that for J-t E Ms(X)

H 1(J-t) + H 2(J-tv) - H 3(J-t ® v)

~(J-t; v) =

=

r h(x) p,(dx) + Jyrh(y) P,V(dy)- j JXXY r h(x, y) p, @v(dx ,dy) Jx r

1

i

2

h1(x) p,(dx) +

i[

3

h2(y) v(x, dy)p,(dx)

-i[h (x ,y) v(x ,dY)P,(dX) = l, [h (X) + [h (y) v(x, dY) - [h (x, y) v(x, dY)] P,(dX) 3

1

2

3

by Lemrna 2.5. Hence, letting XEX,

(6.8)

we have the desired integral representation (6.7). S-invariance of 1: follows from S-invariance of h-i, T-invariance of h2 , S x T-invariance of h3 and stationarity of u, Clearly ~(.; v) is a bounded linear functional on Ms(X) since 1: is bounded. We show that ~(. ; v) is nonnegative. Note that

(6.9)

9l(J-t; v) = H 3(J-t x J-tv) - H 3(J-t ® v)

= n-+oo lim 1 n

L [(p, x p,v)(A x B) log(p, x p,v)(A x B) A,B

- J-t ® v(A x B) log u ® v(A x B)]

= n-+oo lim .!.. '" p, @ v(A n L....J

x B)[log(p, x p,v)(A x B) -log p, @ v(A x B)]

A,B

~O

by Theorem 1.1 (1), where the sum is taken over A E 2l. V 2l.n and B E ~ V Q3n, and we have used the following computation:

L A,B

(1-£ xJ-tv)(A xB) log(J-t x J-tv)(A x B)


173

L J-L(A) log J-L(A) + L J-Lv(B) log J-Lv(B) == L J-L ® v(A B) logJ-L(A) + L J-L ® v(A A,B A,B == L J-L ® v(A xB) log(J-L J-Lv)(A B). ==

A

B

X

X

X

B) logJ-Lv(B)

X

A,B

For a general J-L E M;-(X) we can similarly show 9t(J-L; v) 2:: O. The following corollary immediately follows from (6.9).

Corollary 4. For a stationary channel v E Cs(X, Y), the transmission rate functional 9\(. ; v) is wri·tten as

where (1-£ == J-L

X

J-LV - J-L ® u for J-L E M; (X).

To consider the Parthasarathy-type integral representation of transmission rates we use results in Sections 2.6 and 2.7. As in Section 2.7 denote by 2)0 the algebra of 00 . clopen sets generated by . U S-JSll and let 2)x == a(2)o), the a-algebra generated J=-OO

by 2)0' Let C(X, Sll) be the closed subspace of C(X) spanned by {IA : A E 2)0}. Ps(X,2)x) stands for the set of all stationary sources on (X, 2)x), and Pse(X, 2)x) for the set of all ergodic elements from Ps(X, 2)x). As usual, for J-L E P(X), J-L1~x denotes the restriction of J-L to 2)x .

Lemma 5. With the notation mentioned above, (1) Ps(X, Q3x) == {J-LI~x : J-L E Ps(X)}. (2)Pse(X,2)x) == {J-LI~x : J-L E Pse(X)}.

Proof. (1) Let J-LI E Ps(X,2)x) be given. Then J-LI is regarded as a positive linear functional of norm 1 on C(X, Sll) by the Riesz-Markov-Kakutani Theorem, By the Hahn-Banach Theorem there exists an extension of J-LI onto C(X) of norm 1. Let PI be the set of all such extensions. Since PI is weak* compact and convex, the fixed point theorem implies that there is some S-invariant J-L E P(X), i.e., J-L E Ps(X) and J-L1~x == J-LI· (2) Suppose that J-LI E Pse(X,2)x) is ergodic and let PI be as in (1). If J-L E ex [ps(X) n PI]' then J-L is ergodic. For, if J-L == O',rJ + (3€ with a, (3 > 0, 0',+(3 == 1 and n, € E Ps(X), then as functionals J-L == O',rJ + (3€ == J-LI on C(X, 2l). Since J-LI is ergodic, we see that J-L == rJ == € on C(X,2(.) and hence 11, € E Ps(X) n Pl' Thus J-L rJ == € on C(X) because J-L is extremal in Ps(X) n Pl' Therefore J-L is ergodic and J-L1~x == J-LI.

174


It follows from Lemma 5 (1) that

Hence the entropy functional H I ( · ) on P s(X,2() is unambiguously defined by

Recall that R denotes the set of all regular points in X (cf. Section 2.7). For J-t E Ps(X) and r E R, J-tr denotes the stationary ergodic source corresponding to r. By Lemma 5 (2) there is some Pr E Pse(X) such that J-tr = Pr Imx and we define

for a stationary channel u, For partitions ~ and 0 there exist positive integers n = n(e) and N = N(e), messages UI , ... ,UN E mt~~n(X) and measurable sets Vb ... ,VN E A(mt~(Y)), the algebra generated by mt~ (Y), such that

(1) VinVj=0 (i#j); (2) u (Ui , Vi) > 1 - e, 1 :::; i :::; N; (3) N > en(C-e) . Proof. By the definition of C e there exists an ergodic source JL E P s e (X) such that e 9l(JL ; v) > C - 2".

In fact, we rnay assume 9l(JL;v) = C in view of Theorem 6.1 (2). Since v is stationary and ergodic by Theorem 3.4, JLV and JLQ9v are also stationary and ergodic. For n ~ 1 we denote

A = [x- mX- m+l · · ·Xn-I]

E VJt~~n(X),

D = [YOYI·· ·Yn-I] E VJt~(Y).

(7.1) (7.2)

Then, by 5MB Theorem (Theorern 11.5.1) 1 --logJL(A), n

1 1 --logJLv(D), --logJLQ9v(AxD) n n converge to HI-£(S) , Hl-£v(T) and HI-£®v(S x T) a.e. and hence in probability, respectively. Hence

..!:.log v(A, D) = ..!:.lo /l Q9 v(A x D) n

JLv(D)

n

g JL(A)JLv(D)

3. 7. Coding theorems

179

tends to Hj.£(S) + Hj.£v(T) - Hj.£Q9v(S x T) = 9t(JL; v) in probability (JL we can choose a large enough n = n(c) ~ 1 for which 1 v(A,D) It Q9 u ([ ;;: log Itv(D)

>C

~]) 2

Q9

v). Thus

> 1- ~2'

(7.3)

where [... ] indicates the set

For each A E oot~+n(X) let

VA =

D) Itv(D) > C U{D E 9.n (Y ) : ;;:1 log v(A, 0

n

e}

2 .

Then we get LHS of (7.3)

= JL Q9 v (U(A

x VA))

=L

A

L JL(A)v(A, VA) > 1 A

JL Q9 v(A x VA)

A

~

(7.4)

2

If A and D are such that ~log: JLv(D)en(C-~).

Now choose a UI E oot~+n(X) such that

v(U I , Vu1 ) > 1 - e, which is possible in view of (7.4), and let VI = VUl. If there is a U2 E oot~+n(X) such that v(U2 , VU2 - VI) > 1 c, then we let V2 = VU2 - VI. If, moreover, there is a U3 E

mt~+n(X)

such that

then we let V3 = VUa - (VI U V2 ) . This procedure terminates in a finite number of steps, and we write the so obtained sets as


180

It follows from the construction of these sets that (1) and (2) are satisfied. Furthermore, we have for A E 9Jl;;;'~n(X)

which implies N

V(A,VA) S V(A, }dll~ + V(A, VA -

)

jld N

Vj )

~V(A,UVj)+1-e. J=l

Taking the average over

9Jl;;;'~n(X)

w.r.t. J-L, we get

~ M(A)v(A,VA) S /],V

(jld N

Vj )

+1-

e,

which, together with (7.4), yields that (7.6) Since Vj ~ VUj (1

< j < N), we have by (7.5) that

Thus, invoking (7.6), we see that

if we choose n

= n(e)

~ 1 such that

n(e) ~ ~ log:. Thus (3) holds.

An immediate consequence of the above lemma is: Corollary 2. Let u E Cs(X, Y) be a stationary, nonanticipatory, m-memory and m-dependent channel with the ergodic capacity C = Ceo Then, for any sequence

181

3. 7. Coding theorems

{cn}~=m+l such that

Cn

+0,

there exist a sequence of positive integers {N =

N(n)}~=m+l' a family offinite sets of messages {ui n ) , ..• ,u];)} c m;;"+n(X) (n ~ m + 1) and a family of finite sets of measurable sets {V1(n), . . . , vir)} c A(m~ (Y)) (n ~ m

+ 1)

such that for n ~ m

n ujn)

+1

0 (i =I j); (2) v (Ui(n) , Vi(n)) > 1 - Cn, 1 ::; i (3) N > en(C-c

(1) Ui(n)

=

::; N;

n ) .

To formulate Shannon's coding theorems, we consider another alphabet Xb {a~, ... ,a~} and the alphabet message space (X', X', 8 '), where X' = Xb'L and 8 ' is the shift on X'. A code is a one-to-one measurable rnapping ip : X' -+ X. If we let

vcp (x', A) = 1A ( 0 be given. Choose an Xo E X such that

which is possible by (3.1). Define a functional A on B(Y) by

f

E B(Y).

Then A is bounded and linear with norm II VI (Xo, .) - V2(XO, ')11· Clearly IIK v 1 K V 2 11 2:: \IAII, so that IIK v 1 -K V 2 11 > p(Vl, v2)-e. Therefore IIK v 1 -K v 2 11 2:: P(VI, V2)' Lemma 3. Let V n (n 2:: 1), v E C(X, Y) and p(v n , v) -+ 0 as n -+ any J-L E P(X) it holds that

00.

Then, for

Proof. Let J-L E P(X). It suffices to show the second convergence. For n 2:: 1 and E E oX Q9 ~ choose En,l, E n ,2 E oX Q9 ~ such that En,l n E n ,2 = 0, En,l U E n ,2 =X and \IJ-LQ9 V n

J-LQ9

vii

= 1J-LQ9

(vn

-

= J-L Q9 (vn -

v)I(X x Y) V)(En,l) - J-L ® (vn

V)(En ,2)'

Then the RHS of the above is estimated as J-L Q9 (vn - V)(En,l)

=

i

J-L ® (vn - V) (En ,2)

{v n (x, (En,l).,) - v(x, (En,d.,) }JL(dx)

-i

{v n (x, (En,2).,)

v(x, (E n,2).,) }JL(dx)

204

Chapter IV: Special Topics

~

SUp

xEX

Ivn(x, (En,l)X) - V(X, (En,l)X) I

+ SUp IVn (x, (En,2)x) - V(X, (En,2)x) 1

xEX ~ 2 sup II vn(x , .) - v(x, ·)11 xEX =2p(vn,v)-+O (n-+oo), where (En,i)x is the x-section of En,i for i follows.

= 1,2 and n 2

1. Therefore the conclusion

Proposition 4. Consider the metric p on C(X, Y). (1) (C (X, Y), p) is a complete metric space. (2) (Cs(X, Y), p) is a complete metric space. (3) (Cse(X, Y), p) is a complete metric space. (4) rc, (X, Y), p) is a complete metric space. (5) (Cae (X, Y), p) is a complete metric space.

Proof. (1) Let {vn } C C(X, Y) be a Cauchy sequence, i.e., p(vn, vm ) -+ 0 as n, m -+ Let x E X be fixed. Then,

00.

Since vn(x, .) E P{Y) for n 2 1 and P(Y) is norm closed, there is some v(x, .) E P(Y) such that Ilvn(x,·) - v(x, ·)11 -+ o. Hence v(·,·) satisfies (c1). To see that v(·,·) satisfies (c2), let C E ~ be arbitrary. Since vn ( · , C) is X-measurable for n 2 1 and xEX,

we see that v(·, C) is also X-measurable. Hence (c2) is satisfied. Thus v is a channel. For any c > 0 there is some no 2 1 such that p(vn, vm ) < e for n, rn 2 no. Letting m -+ 00 we obtain p(vn, v) ~ e for n 2 no. This means that p(vn, v) -+ 0 as n -+ 00. Therefore (C(X, Y), p) is complete. (2) Let {v n } < Cs(X, Y) be a Cauchy sequence. Then by (1) there is some v E C(X, Y) such that p(vn, v) -+ O. To see that v is stationary let x E X and C E ~. It follows that

Iv(Sx,C)

v(x,T-1C)1 ~ Iv(Sx,C) - vn(Sx,C)1 + Ivn(Sx,C) - vn(x,T-1C)1

+ Ivn(x,T-1C) -

v(x, T-1C)1

~ II v (Sx, .) - vn(Sx, ·)11 ~2p(vn,v)-+O

+ Ilvn(x,.)

(n-+oo).

- v(x, ·)11

4.3. Approximation of channels

205

Hence v(Sx, C) == v(x, T-1C). Thus v is stationary. (3) Let {vn } ~ Cse(X,Y) be a Cauchy sequence. Then there is some stationary v E Cs(X, Y) such that p(vn , v) -+ 0 by (2). To see that v is ergodic let It E Pse(X). Since It®vn is ergodic for n ~ 1, 111t@vn - It@vll -+ 0 by Lemma 3, and Pse(X x Y) is norm closed, one has that It @ v is also ergodic. Thus v is ergodic. (4) and (5) can be proved similarly. Let us now consider channels [X, t/, X], where the input and the output are identical. Let rJ E P(X) and consider the Hilbert space L 2(X, rJ). If an operator L : L 2(X, rJ) -+ L 2(X, rJ) is defined by

(Lf)(x)

=

L

k(x, y) TJ(dy) ,

where k(·,·) is a suitable kernel function on X x X, then L is said to be of HilbertSchmidt type if k E L 2(X X X, rJ x 'TJ). Similarly we have:

Definition 5. Assume that a channel v E C(X, X) is strongly measurable, so that there is some rJ E P(X) such that v(x, .) «: rJ for x E X and

==

k( x, y ) -

v(x, dy) rJ(dy)

is jointly rneasurable. In this case, v is said to be of ('TJ) Hilbert-Schmidt type if k E L 2(X X X, rJ x rJ).

Theorem 6. Let v E C(X, X) be a strongly measurable channel with a dominating measure rJ E P(X). For any e (0 < e < 1) there exist a pair of channels VI, v2 E C(X, X) and a A (0 < A < e) such that

(3.2) _

Vl(X,

dy)

_

V2(X,

dy)

k1(x,y)= k2(x,y)

Proof. Let k(x, y) constant c > 0 let

= TJ(dy)

== v~(d:» for k~(x, y)

TJ(dy)

== k~(x, y) ==

1

)

(3.3)

EL(XxX,TJXTJ, EL

2(

)

(3.4)

X X X,TJ x TJ .

(x, y) E X x X. Then clearly k E L1(X

k(x, y)l[k~c](x, y),

x,yEX,

k(x, y)l[k 0 for which

o< Then define

VI, V2

IIk~lll,77x71

1 - e :::; IIk~IIt,77x71

< C,

< c < 1) be given.

< 1.

by XEX, CE.x,

x EX, CE.x, and let A = Ilk~IIt,71x71. It follows that (3.2)

Theorem 7. Let

V

E C(X,

(3.4) hold.

X) be a channel. If v is strongly measurable, then:

(cI9) There is an 'fJ E P(X) such that for any e > 0 there exists an ('fJ) HilbertSchmidt channel V g E C(X, X) for which p(v,vg ) < c. If (X,.x) is separable, then the converse is true, i.e., (cI9) implies the strong measurability of v ((cI7)). Proof. Let 'fJ E P(X) be a dominating measure for t/. Theorem 6 implies that for any e (0 < c < 1) there are channels VI, V2 E C(X, X) and a A > 0 such that e

0< A
Chapter IV: Special Topics

218

= ip ® 1/; (0-(1 ® 13(b l =

)

)0-(1 ® b2))

cp ® 1/;( a ® ,8(0-(1 ® bl))o-(l ® b2)).

since cp is WM, so that

A* cp (13(b 1)b2) = cp @1/; (a @,8(0-(1 @bl ) ) ) cp @1/; (0-(1 ® b2)) =

A*(13(b l))A*cp(b 2)

since the algebraic tensor product A0lffi is dense in A@lffi. This means A* cp E WM (,8). Therefore A* is WM. (3) In (2), if 1/; E K(,8), then A* = [(1 ® 1/;) 00- 0 * is KMS. For, let sp E K(a). Then cp®1/; E K(a®,8). Now for Q,R E A®lffi there exists a C-valued function fQ,R on C that is analytic on D = {z E C : 0 < Im z < I}, and bounded and continuous on D such that

jJ

fQ,R(t) = cp ® 1/;(at ® ,8t(Q)R) , fQ,R(t Letting Q

= 0-(1 ® bl )

+ i) =

and R

ip

® 1/;(Rat @ ,8t(Q)),

= 0-(1 ® b2 )

t E JR, t E JR.

for bb b2 E lffi, we see that for t E JR

fQ,R(t) == fb1,b2 (t) = A *cp(,8t(b l)b2) , fQ,R(t

+ i) == fbl,b2(t + i)

= A*cp(b 2,8t(b l

)) .

Hence A*cp is KMS. Therefore A* is KMS. Definition 6. Let S ~ 6(A) and ab a2 E A. We say that al == a2 (modS) if cp(al) = cp(a2) for cp E S. If Ai and A2 are channels, then we say that Ai == A2 (modS) provided that Aicp = A 2CP for cp E S. Proposition 7. If A* is a stationary ergodic channel, then A* is extremal in Cs(A,lffi) modI(a), i.e., if A* = AAi + (1 - A)A2 with 0 < A < 1 and Ai,A 2 E Cs(A,lffi), then Ai == A2 (modI(a)).

Proof. Suppose that A* = AAi+(l A)A2 for some A E (0,1) and Ai,A 2 E Cs(A,lffi). Then we have A*ep(b) = AAi cp(b) + (1 - A)A~cp(b) for b E lffi and cp E exI(a). It is easily verified that for cp E exI(a) b E lffi, t E JR, k = 1,2,

4.5. Noncommutative channels

219

so that Ak

Abstract Methods in Information Theory

Abstract Methods in Information Theory

Information-spectrum methods in information theory

Information-Spectrum Methods in Information Theory

Abstract Measurement Theory

Abstract Systems Theory

Abstract Set Theory

Abstract Measurement Theory

Methods in module theory

Abstract Algebra: Theory and Applications

Information-Spectrum Method in Information Theory

Information theory

Information Theory

Information Theory

Information Theory

Abstract algebra: Theory and applications

Methods of Information Geometry

Methods of Information Geometry

Qualitative methods in quantum theory

Elementary Methods in Number Theory

Mathematical methods in risk theory

Mathematical methods in risk theory

Methods in Banach space theory

Topological Methods in Group Theory

Topological methods in group theory

Qualitative methods in quantum theory

Abstract homotopy and simple homotopy theory

Abstract Analytic Function Theory And Hardy Algebras

Information theory and statistics

Coding and information theory

Network Information Theory

Abstract Methods in Information Theory