Remarks concerning hyperbolic sets

Journal of Mathematical Sciences, Vol. 78, No. 5, 1996

REMARKS CONCERNING

H Y P E R B O L I C SETS

D. V. A n o s o v

UDC 517.987.5, 517.938, 515.168.3

1. The terminology and notations used in this article, as well as the background material, correspond to that in [1]. Naturally, all this can be found in many other papers and articles on the theory of smooth dynamical systems dealing with hyperbolic sets. Only for one detail can the reader encounter in the literature two variants that are in equal circulation. For flows, our stable and neutrally stable manifolds WS(x), W'~(x) are sometimes said to be strongly stable and stable and are denoted by W~(x) and W'(z) respectively. The same refers to unstable manifolds. We shall speak of a hyperbolic set A of the smooth dynamical system {g~}, a flow or a cascade, defined on some smooth manifold M. The questions of interest to us are semilocal and refer, in the final analysis, only to A itself and to what goes on in its neighborhood. Since we are not interested in what goes on far from A, M may be not compact and gtx may be defined, for certain x, not for all t. In the case of a cascade {g'~}, the mapping g may be defined only on some open s e t / / C M. In the case of a flow, the phase velocity vector field v in meant to be definite (and smooth) throughout M, but if M is not compact, then it is quite possible that for certain Xo E M the maximal interval of existence of tile solution x(t) = gtxo of the system = v(x) with the initial value x(0) = x0 may differ from R. The pairs (z, t) for which gtx has sense form an open subset :D C M x • that contains M x 0 but may differ from M x R for a noncompact M. However, by definition, gtx must have sense for all t E Z or t E R (and, of course, must belong to A) for all points x of the invariant, to be more precise, bilaterally invariant, set A (no matter whether it is hyperbolic or not). In the case of a cascade, it is sufficient to require that A C Lt n gu, gA C A, g-lA C A. In the case of a flow, what we have said means that A • R C :D. Under the additional condition of compactness of A, it is sufficient to require that for every x E A there should be r > 0 such that gtx be defined for all Itl < r and that gtx E A for these t. In the case of a hyperbolic A in which we are interested, it also follows from x E A that gty is defined for y E W~(x) for all t _> 0 and, when y e W~(z), for all t < 0. When, in this article, we speak of a stable or unstable manifold W~(x) or W"(x), we always have in mind that z e A. As is known, the simplest example of a hyperbolic set is a set consisting of a finite number of hyperbolic periodic trajectories. But it stands to reason that this concept was introduced not for the sake of this trivial example, but for the sake of objects like Smale's horseshoe in which the number of trajectories is infinite and even has the cardinality of continuum. It is well known that in these "real" hyperbolic sets the dynamics is much more complicated. This article adds a certain contrast of the properties of hyperbolic sets consisting of an infinite number of trajectories to the properties of the trivial example we have cited. In Sec. 3 we briefly discuss the "intermediate" case of a hyperbolic set A consisting of an infinite but countable number of trajectories. If such A is locally maximal, then its structure is very simple and in this respect the present case is certainly "not true." Now if A is not locally maximal, then its structure may be more complicated. I do not deal here with a detailed analysis of these A (although, in principle, such an investigation could have been carried out; perhaps it could have even yielded some results of a classificational nature). But it seems to me that the remarks given in this article present a sufficient answer to the natural question as to the role played by the "intermediate" case, showing that this case is not "true" either as concerns its complexity. 2. We say that the stable manifold Ws(z) is periodic with period r if g~W~(x) C Ws(Z). If, for any y E W~(x), gty are defined for all t, then the periodicity of W'(x) is equivalent to the fact that g~W'(x) = W'(z). Here we must check that if W'(z) is periodic and y e Ws(z), then g-*y e W'(z). It is clear from Translated from Itogi Nauki i Tekhniki, Seriya Sovremennaya Matematika i Ee Prilozheniya. Tematicheskie Obzory. Vol. 13, Dinamicheskie Sistemy-1, 1994. 1072-3374/96/7805-0497515.00 9

Plenum Publishing Corporation

497

the definition of periodicity that g~'x E W'(x), and therefore the distance

p(gty,gtg'~x) --* 0 as t --~ or This means that

p(gtg-~'y,

g'x) =

p(g'-*y, gt-*g'~x) --* 0

as t ~

oo,

so that, indeed, g-~y E W'(x). The periodicity of the unstable manifold W"(z) is defined as follrws: g-*W"(x) C W"(x). If, for all y e W~(x), g*y are defined for all t, then the periodicity of W"(x) is equivalent to the fact that g*W"(x) = W~(x). (We first make sure that g-~W~(x) = W~(x). This is similar to the statement we have made above and can even be reduced to it by means of time reversal.) If x is a periodic point, then the manifold WS(x) is periodic (with the same period). The converse is certainly not true, but the following theorem is valid.

Suppose that A is a hyperbolic set, x E A and W~(x) (or W"(x)) is periodic with period r. Then WS(x) (W~(x) respectively) passes through a periodic point y e A which has the same period v (of course, in this case, W'(x) = WS(y) or W"(x) = W"(y)). Theorem

1 . ( O n a p e r i o d i c s t a b l e or u n s t a b l e m a n i f o l d )

In the trivial cases of a hyperbolicset A consisting of a finite number of hyperbolic periodic trajectories,

W'(x) are different for different x E A, and the same is true of W ". Indeed, two distinct periodic points x and y cannot lie on the same stable (unstable) manifold: if we have, s~y, y E W'(x), then it would follow that limt_.~ p(g*x,gty) = 0, so that the point gty would not be able to return periodically to the initial position. The situation proves to be different in all other cases. T h e o r e m 2. If the hyperbolic set A consists of an infinite number of trajectories or if it consists of a finite number of trajectories but they are not all periodic, then there are trajectories L~, L2 C A such that (W'(LI) \ L1) M A ~ g, (W"(L2) \ L2) M A ~ ~. Remark.

It is obvious that if we take the point

y e (W'(L,) \ Lx)N A, in the same notations, we shall find that it lies on some W'(x), x E L1. Thus, the stable manifolds of two different points x, y E A coincide. Conversely, suppose it is known that there are two points z ~ y in A such that W'(x) = WS(y). Can we infer that the stable smooth manifold W'(L~) of a certain trajectory from A also contains another trajectory L ~ C A? Clearly, it is obviously so if one of the points x, y (say, x) is periodic, since then the other point, y, cannot be periodic and its trajectory L' is just the trajectory from A lying in W'(LI), where L~ = {gtx}, with L' ~ L1. Let us consider the case when neither x nor y is periodic. We denote by L1 and L r the trajectories of these points. It is clear that L ~ C W'(L1) as before, but this time it is not inconceivable that L ~ = L1. However, if L ~ = L1, then y = g*x or'z = g*y with a certain r > 0, and then the stable manifold W'(x) = W'(y) is periodic. According to Theorem 1 it passes through some periodic point z E A. Consequently, L~ = L' C W'({gtz}). In this case, the trajectory L~ = L' is not periodic and {gtz} is a periodic trajectory and so these trajectories are different. Thus the inference remains valid. What we have said remains true for unstable manifolds as well. 3. If the hyperbolic set A consists of an infinite number of trajectories, then it contains a nonperiodic trajectory. Theorem

This theorem follows from Theorem 2 since, as was mentioned, if L is a periodic trajectory, then neither W'(L) nor W~(L) can contain any other periodic trajectories except for L. This means that if A consists of periodic trajectories, then

(W'(L1) \ L1) N A = z, 498

(W~'(L2) \ L2) N A = o

for any of its trajectories L1, L2. However, we shall prove Theorem 3 before we complete the proof of Theorem 2 and, conversely, use Theorem 3 when completing the proof of Theorem 2. Theorems 1, 2, 3 will be proved in See. 8. Somewhat earlier,, in Sees. 4-7, we give some auxiliary "technical" material. A considerable part of this material is more or less well known, but I think it expedient to present the material successively, proceeding from universally known facts. Not mentioning the completeness of the exposition, I believe it to be more convenient for the reader than a number of references to papers in which, explicitly or implicitly, some more or less equivalent considerations appear in passing. Incidentally, what is said in Sees. 4, 6, 7 may be not without some new elements as concerns the methods of exposition. The treatment of the behavior of trajectories in the neighborhood of a fixed'trajectory {gtz} from the hyperbolic set A given there (with due account of the "uniformity" with respect to all {gtx} C A) differs from the treatment I gave earlier [2] and seems to be more convenient, at least for the purposes of the present article. (Maybe not only for them. However, in this article the exposition is constructed so as to achieve these purposes only.) It also seems to me that the construction of the Lyapunov metric (See. 5) for A y~ M, in the case of a flow, is formally new. 3. The set of trajectories lying in the hyperbolic set A is either finite or countably infinite, or has the cardinality of continuum. (And this is certainly also true in a considerably more general situation.) As to periodic trajectories, it contains a countable number of them (it follows from the property of separation of trajectories and compactness that the number of periodic trajectories with period _< T is finite for any T). This means that if the set of trajectories lying in A has a cardinality of continuum, then the "majority" of trajectories are nonperiodic. We do not need Theorem 3 in this case. And what happens when A consists of an infinite countable number of trajectories? According to Theorem 3, in that case there is at least one nonperiodic trajectory. The following example shows that we cannot state anything more. The example is based on the use of symbolic dynamics. It speaks directly of a closed invariant subset a of the Bernoulli topological cascade {~ri} acting on the-space f/2 of two-sided infinite sequences {a(i)} of symbois 0, 1. As is known, the Bernoulli topological cascade is realized as a hyperbolic set of a certain smooth cascade (this is done, for instance, with the aid of Smale's horseshoe). "Thus, closed invariant subsets of the Bernoulli topological cascade also give some examples of hyperbolic sets. In the example, A consists of the trajectories of the points a l , . . . , a,,,.., and b:. al = {al(i)}, where all at(i) = 0; a,, = {a,~(i)}, where a,(i) = 1 if i is divisible by n and a,(i) = 0 otherwise, b = {b(i)}, Where b(0) = 1, and all other b(i) = O. A contains an infinite number of periodic trajectories (each of the points a,~ is periodic with period n). In addition, A includes only one nonperiodic trajectory, the trajectory of the point b, which tends to al in both directions with respect to time. It should also be pointed out that periodic trajectories are dense everywhere in A so that the center (in the Birkhoff sense) of the cascade {ailA) (i.e., the closure of the set formed by the Poisson stable trajectories) coincides with the whole A. This example also shows that in the conclusion of Theorem 2 the trajectories Lx and L2 may coincide. In this example, the hyperbolic set A is not locally maximal. If A is a locally maximal hyperbolic set consisting of a countable number of trajectories, then it contains only a finite number of periodic trajectories. Indeed, if there were an infinite number of such trajectories, then, for any e > 0, there would be two different periodic trajectories L1 and L2 the minimum distance between whose points would be smaller than e. Then we can form an e-trajectory l = {x(t)) which coincides with L~ for t < 0, then "passes" on to L2 and revolves ones along it, and then again "passes" on to L1 and coincides with La further on its way. According to the theorem on the family of e-trajectories (actually, in this case the family reduces to only one e-trajectory), there is a "true" trajectory L = {g~x} close to this e-trajectory. (The proximity in the case of a cascade means the smallness of the distance p(gtx, x(t)) for all t and in the case of a flow the smallness of the Fr~chet distance between l and L as parametrized curves.) Since A is locally maximal, L C A. A certain negative 499

and a certain positive semitrajectory of the trajectory L remain close to L1 all the time. As is known, it follows that L C W"(L1) N W'(L~). (Incidentally, this also follows from what will be said in Sec. 7 of this article.) In this case, L ~ L1 since the point gtx, moving along L, recedes for some time from L1 moving close to L2. (Here again we speak not of the Hausdorff distance between L and L1 as sets of points - - it may be small (say, this is obviously so when L1 and L2 are "sufficiently dense" in A) - - but of p(gtxl,gtx) with some Xl E L1 in the case of a cascade and of the corresponding Fr~chet distance in the case of a flow.) Thus A contains a (transversal) homoclinic trajectory L. (The transversality of the intersection of W'(L1) and W"(LI) along L follows from the fact that W"(L~) = W"(L) and W~(L1) = W'(L) and so we speak of the transversality of the intersection of W"(L) and W'(L) along the trajectory L C A.) As is known, in that case, an arbitrarily small neighborhood of the closure L contains a hyperbolic set B which has a well-known symbolic description; from the latter we can see, in particular, that B contains a continuum of trajectories. But B C A since A is locally maximal. It follows furthermore that for a locally maximal hyperbolic set A consisting of a countable number of trajectories all nonwandering trajectories of the dynamical system {gtlA } are periodic. Indeed, we know that the periodic trajectories are dense everywhere in the set of nonwandering points of this system, and there is a finite number of trajectories of this kind. Finally, the theorem on the spectral decomposition "degenerates" into a statement that every trajectory L C A tends to a certain periodic trajectory Li C A as t --* - o o (i.e., L C W"(L~)) and to a certain periodic trajectory nj C A as t ---* oo (i.e., L C W'(Lj)), and the binary relation "Li > Lj, provided that there exists a trajectory L C A n W"(Li) fl W'(Li) '' is a strict-order relation on the set of the periodic trajectories from A. We can see that the structure of the locally maximal hyperbolic sets A consisting of a countable set of trajectories is very simple, essentially, as simple as in the case of a finite number of trajectories. Therefore, it is expedient to give an example showing that the number of trajectories in such an A can be infinite. We again give an example in terms of symbolic dynamics. In this example, the closed invariant set A C f~2 consists of the trajectories of the following points a, b, c, p, q, r, 8 1 , . . . , s,,,...: a = {a(i)), where all a(i) = O, b = {b(i)}, where b(i) = 1 for even i and b(i) = 0 for odd i, c= {c(i)), where all c(i) = 1, p = {p(i)}, where p(i) = 1 for even i > 0 and all other p(i) = O, q = {q(i)}, where q(i) = 0 for even i _< 0 and all other q(i) = 1, r = {r(i)}, where r(i) = 0 for i < 0 and r(i) = 1 for i _> 0, 8,, = {s,,(i)}, where s,(i) = 0 for i < 0 and for odd i = 1,... ,2n - 1 and all other a,(i) = 1. In- a hyperbolic set A which consists of a countable, even finite, number of trajectories but is not locally maximal, the limiting behavior of trajectories may be more complicated, namely, they (not necessarily) all tend to periodic trajectories. Here are some examples, "symbolic" again. Let us consider in f/2 points a0 = {a0(i)},..., a, = {an(i)} defined as follows. All ao(i) = 0; al(0) = 1 and all other ax(i) = 0. Furthermore, let k = 2 , . . . , n . If i < 0, then all ak(i) = 0. The symbols ak(i) with i > 0 written out in succession are divided into blocks, ak(0)ak(1).., ak(i)... = Bk(1)Bk(2)... Bk(j)..., which are defined as follows. We denote the block consisting of k symbols c by c k. Then B2(j) = 0~10j

(j = 1,2,...),

and if the blocks Bk-l(j) have already been defined, then Bk(1) = Bk-l(1), 500

Bk(2) = B k - l ( 1 ) B k - l ( 2 ) , . . . ,

Bk(j) ----Bk-l(1)Bk-l(2)... Bk-l(j), . . . . We denote by Ak (k = 0 , . . . , n) a set formed by the trajectories of the points n o , . . . , a,, under the action of the Bernoulli topological cascade {crl}. The point ao is fixed, and therefore the a- and w-limit sets of fts trajectory coincide with {a0} = Ao. Now if k > 0, then, as we can easily verify, the a-limit set of the trajectory {ainu} also reduces to Ao, whereas its w-limit set coincides with Ak-1. Moreover, denoting by NW({gt}) the set of nonwandering points of the dynamical system {gt }, we have

N W ( { f l A o } ) = Ao,

N W ( { f l A k } ) = Ak-1

(k = 1,...,n).

Recall the definition of the (Birkhof]) center C({gt}) of the dynamical system {gt}. We set

Ox = N W ( { g t } ) ,

~2 = N W ( { f ] ~ x } ) , . . . , =

N

~,, = N W ( { g ' l f l , - a } ) , . . . ,

=

k<w

(transfinite induction). If A is a metric compact set, then there is a least transfinite number a (not more than countable) for which fl= = ~2~+x . . . . . This set is precisely C({gt}) and a is the depth of the center. (In old papers a less expressive name "the ordinal number of central trajectories" was used. The center admits of other, equivalent, definitions [3].) In our example, gtj = A,_ i for j < n, C({ailA~}) = A0, and the depth of the center is n. In a hyperbolic set consisting of a countable number of trajectories the depth of the center may also be infinite. Here is an example of the corresponding subset in f/2- Let us consider the following points hi, h2, hi, b2, e l , C 2 , . . . , C n , - . . : al = {al(i)}, where all al(i) = 0; a2 = {a2(i)}, where all a2(i) = 1; b~ = {bl(i)}, where bl(i) = 0 for i < 0 and b~(i) = 1 for i > 1; b2 = {b2(i)}, where b2(i) = 1 for i < 0 and b~(i) = 0 for i > 1. Finally, for i < 0 all ck(i) = 0 and the symbols ck(i) with i _> 0 written out in succession are divided into blocks,

c,(i) . . . .

cdl)cd2).., c d j ) . . . ,

which are constructed as follows: Ok(l) = 1k Ck(2) = 0Ck+,(1),

Ck(j + 1)

=

(k = 1,2,...);

Ck(3) = 02Ck+l(l)Ck+l(2),...,

OJCk+l(1)Ck+l (2)... Ck+~(j), ....

Let A be the set {al,a2}, B be obtained by attaching the trajectories of the points bl, b2 to A, and Ok, k = 1, 2,..., be obtained by attaching the trajectories of the points cj, Mth j > k, to B. Then NW({a~[Ck}) = Ck+1 so that for the cascade {cri[Ca}

~'~1 = C 2 , . . . ,

~~j = C j + I , . . . ,

~~w = B,

~"~a+l =

A = C({ai[C~}).

The depth of the center is w + 1. These examples have something in common with Maier's examples of flows with an infinite depth of the center in R s. (See [4, 5]; a part of Maier's constructions is reproduced in [6]. In fact, Maler himself realized some of his examples not in R3 but in the space of unit tangent vectors of a closed surface of negative curvature9 It is indicated in [7] that they can also be realized in R 3 (and with smoothness of class C ~ at that)). To be more precise, they have something in common with the "symbolic part" of Maier's constructions 501

(we have naturally nothing to do with a witty technique which he invented to pass to a flow in ~3). The "symbolic" examples given above correspond only to the simplest Maier's examples. I do not doubt that the "symbolic part" of his more complicated examples (in which the depth of the center exceeds the specified countable transfinite number or which refer to other (nonequivalent) variants of the depth of the center) can also be presented as that referring to closed invariant subsets of the Bernoulli topological cascade consisting of a countable number of trajectories. The first example given in this section also testifies to some complication of the dynamics that may occur when we abandon the local maximality of A (and retain the condition that this hyperbolic set should consist of a countable number of trajectories), namely, there is a nonperiodic trajectory in the Birkhoff center of the system {gt[A}. But in another (and, I think, more important) respect, the dynamics in this case remains if not very simple, then not very complicated: only periodic trajectories can be Poisson stable (one-sided and two-sided) in A. This is not connected with hyperbolicity. Suppose that {gt} is a topological flow or cascade in a complete metric space and L = {9tx} is a Poisson stable (one-sided and two-sided) trajectory. If its closure L consists of a countable number of trajectories, then L is a periodic trajectory. The rephrasing of the well-known arguments going back to Birkhoff (who presented them in connection with some other case) may serve as a proof. To make the exposition more complete and convenient for the reader, I shall give the proof (especially since I do not know whether there are similar arguments in the literature given in the form we now need). From the fact that L is nonperiodic but Poisson stable, it follows that any finite arc of any trajectory in L is nowhere dense. Indeed, let us suppose that the arc I = {9'Y; ~ e [a, b]} is dense somewhere in L, i.e., that there is a ball v =

e L; p(z, u)
N and It,,, - t.I > b - a. Then it follows from

gt,, x that

gt'x

=

gs=~,

gt,~x

__

gS,~t

= g ~ ' u = g ~ ' - ~ " g ~ " u = g~"-~"gt'=x = g t " + ~ ' - ~ " x ,

gtm-t"+sn-smX

=

X.

But and we find that {gtz} is a periodic trajectory. If L now consists of a countable number of trajectories, say, of the trajectories of the points z,~, then L coincides with the union of a countable number of finite arcs =

Itl _ m}.

But L is a complete metric space and, according to the Baire theorem, cannot be represented as the union of a countable number of subsets not dense anywhere. 4. This section and the next are not interconnected. They are both preparatory for Sec. 6 whose results are necessary for what will follow. In this section, the set A is assumed to be compact and invariant; its hyperbolicity is not necessary for the time being. (However, in the final analysis the results of this section are-used only for hyperbolic A.) 502

By analogy with ordinary functions, the map f : X ~ Y between metric spaces (with metric p) is known as a Lipschitzian map (satisfying the Lipschitz condition) if there is C such that p(fz, fy) < Cp(x, y) for all x, y E X. C is known as a Lipschitz constant for f. (A Lipschitz constant for f is not uniquely defined, i.e., if C is a Lipschitz constant for f, then any C' > C is also a Lipschitz constant.) A Lipschitz constant for several maps fi is a C which is a Lipschitz constant for every fi. f is said to be locally Lipschitzian if every point x has a neighborhood U= such that the restriction fJU= is Lipschitzian. If X is compact, then the locally Lipschitzian map f : X ~ Y is Lipschitzian. Indeed, then there is a finite open covering {Ui} of the space X such that all flUi are Lipschitzian with constants Ci, and this means that they have a common Lipschitz constant C = max Ci. According to the well-known Lebesgue lemma, there is r > 0 such that if p(x,y) < r, then there is a Ui containing x and y. Then p(fx, fy) < Cp(z,y). On the other hand, there is C' such that p(fx, fy) < C' for any x, y since the image f X is a compact and, hence, a bounded subset of Y. Now it is clear that

p(x, y) 0. Then, as before, we can define F~,t as in (4.10), and then the resulting map Br+~(B~(A)) • ( - 1 - ~, 1 + ~) ~ BcC,+~)(Bc~(A)) (4.15) will be (2-smooth. True enough, the map (x, ,, ~) ~ Tg~(x)v will be only ( l - s m o o t h , but in this case the differentiation with respect to v does not decrease the smoothness, and so the map

(x, v, t) ~, n , f ( x , v, t) = n,E~,,(v) - Tg~x is a ( l - s m o o t h map

X --* L(TX, TY),

(4.16)

where X is the left-hand side and Y is the right-hand side of (4~15). One difference from the case v E C 1 is that in the last case we state not the smoothness of the mapping (4.16) (which, of course, can also be considered for v E C 1 since the extension of the domain of F and f indicated above does not require a high smoothness of v) but, so to speak, only the existence of a derivative with respect to one of the arguments (and its continuity). I would like to infer now that by virtue of (4.14), D~D,f(x, 0~, t) = 0 and therefore

IID,D.f(x,o=, t)ll ~(1r

2

(5.4) r e Z~".

(5.5)

The construction of the Lyapunov metric as well as many other things are given in [2] as applied to the case A = M. Usually in a situation like this, I do not" think it is necessary to repeat myself if, as compared to [2], I only have to make small obvious changes. But, in this article, I decided to act differently. In the present section, I give the construction of the Lyapunov metric, dwelling on two circumstances. The first is that A ~t M of which I naturally could n o t speak in [2] because of the character of that book. The other circumstance was actually taken into account in [2] (it is not connected with whether A is coincident with M), but it deserves to be given special attention, w h i c h was not done in [2]. It will be indicated as the exposition goes on. Dwelling especially on these circumstances, I conformed to Littlewood's remark: usually the reader can easily fill in one gap in the proof, but "two missed trivialities taken together could form an insurmountable obstacle." We shall first prove that in a certain neighborhood N of the set A there is a continuous function ~ : TN --* Ir whose restriction on every T=N is a quadratic form (in such cases I shall speak of a continuous quadratic form on a vector bundle) possessing t h e following properties. It is positive definite (as a quadratic

T=N). In the case of a flow, for all w E TN, there is a derivative d t=o~(Tgtw) which is a continuous quadratic form on TN. (In tensor language, 9 is a certain twice covariant tensor and we speak of

form on every

its Lie derivative along the vector field of the phase velocity of the flow.) For all x E A, ~ E E:, ~/E E~ we have (5.5) r _< c~(~?)-i1~11~, r ii~ll~ d~

(5.7)

in the case of a flow, and, in addition, ~(v(x)) = 1, where v is the phase velocity vector field of the flow. In (5.6) and (5.7) ]]. I] is the initial Riemannian metric on M with which we begin in order to arrive at the Lyapunov metric ] 9 [. We can somewhat simplify the reasoning if we assume that a certain neighborhood N1 D A is smoothly embedded into R '~ (for the neighborhood of the set A the embedding can be easily constructed by means of local coordinates [8, 9]; as is known, we can embed the whole M, but if it is not compact, it is much more difficult to do, whereas it is sufficient for us to embed only a certain neighborhood N1 D A) and if we also assume that ]]. ]] is a restriction on TN1 of a standard metric in R" which we also denote by [I" IILet z E A C R '~, X q R". Then we can represent X as X=~+r/+v

or

X=~+r/+~'+v,

where v.l.T=U, ~ 6 E=', r / e E~ and (in the case of a flow).~" e E•. We set UI(x,X) = H~ll2, v ~ ( z , x ) = Hr/]l2 and (in the case of a flow) W1 (x, X) = []r 2. These are quadratic forms on R '~ whose coefficients are continuous functions of z e A. These functions admit of a continuation up to the continuous functions R = --* R (a special case of the well-known theorem of Urysohn; this case was discovered by Brauer earlier and admits of a very simple proof). Continuing in this way the coefficients of the forms U1, V1, W1, we obtain some continuations U2, V~, W2 of these forms. We denote the restrictions of 0"2, V2, W2 on TN1 by U3, Va, W3. The latter are defined for the tangent vectors w e TN1, namely, u~(~) = v~(~, ~),

v~(w) = y ~ ( ~ , w ) ,

w~(w) = w 2 ( ~ , . , ) ,

where x is the point of N1 for which w e T=N1. These are continuous quadratic forms on TN1, and when

9 eA,

weT=g,,

~=~+r

or ~ = ~ + r 1 6 2

~eE;,

~eE;,

feE

2,

then

v~(w) = iir ~,

v~ = I[~ll ~,

w~ = [lr ~

We shall first consider the case of a cascade. We take an integer r > 1 (we shall later elucidate the choice of this parameter of the construction). We denote by N~ the neighborhood of the set A for which giN,. C N1 for all Iil -< T, and set r

,.-1

Oh(w) = 2 ~ U3(Tgiw) + 2 ~ Va(Tg-lw) i=0

i=0

for z e N,., w e T~N,.. Clearly, it is a continuous quadratic form on the vector bundle TN,.. At the points A it (i.e., its restriction on T=N,.) is positive definite, and therefore it is also positive definite when z lies in some neighborhood N; of the set A. If

zEA,

wET=M,

w =~+r/,

~EE~,

r / e E~,

then

~(w) = ~(~) + ~(~), ,-1 4,(~) = 2 ~ I[T9'@ 2,

~-I (~(,1) = 2 ~ [ITg-'~[] 2.

o

(5.8)

o

In particular, as can be seen from (5.8), E=~ and E~ are orthogonal in the Euclidean metric r Furthermore, it is easy to see that

r

- (I)(~) = 211Tg,.~H2 - 211~11=, (I)(Tg-ir/)- ~(rt) = 2llTg-,.r/I] 2 - 2l]r[[]2.

(5.9)

But the hyperbolicity of A means that estimates of the form

IlTgi~ll 0 is so small that 1 - 26 > 0 and 1-~s > ~ , then, in accordance with (5.6), qt(Tg~) < ~(Tg~)+~ll~ll ~ _< ~(~)-I1r

~+~11r ~ _< ~(~)+2gll~llZ-IIr ~- _< ~(~)

1 - 26

~ ~ ~(~) _< ( 1 - ~ ) ~ ( ~ ) 511

for ( E T=M, x E A. Similarly, ~(Tg-'7) [~x(0) -- ~2(0)1, 16(0) -- 6(0)1, since the last two differences are zero. Lemma 1 guarantees an exponential growth of I~l(t) - ~2(t)l, whereas rt2(t)l < 66. In order to prove (7.7) we have to show that if

gtyl, gty 2 remain in D35(gtz), and therefore I r h ( t ) -

g'yl e Ds(gtz)

for all

t > 0

lim(vtz,gtyl) = 0,

and

y~ E W~(x). We have shown that the first of these conditions ensures that Yx E W~n(x) so that !t~ E W~s(g~x) with some r E [-26, 25]. Then limt--.oo p(gt+*z, gtyl) 0. So it turns out that limt--.oo p ( g ' + ' z , g'z) =

then

=

0. This is possible for small r only when r = 0 (since there is no equilibrium point in A). Hence yx E W~6(x) whereas yx e Ds(x), and, consequently, yl e W~(z). Relations (7.6) and (7.8) result from (7.5) and (7.7) upon time reversal. P r o o f o f L e m m a 3. We make sure, first of all, that I~(T)} < 6. We apply L e m m a 1 with k = 1 to Yl = z, y2 = Y. In this case,

(~,(t),,llCt),r

= (0,0,0),

(~2(t),,72(t),r

= (r162

1~2(T) - ~a(T)I = I~(T)I = 6 > I~(T)I = 1,7~(T) - ~ , ( T ) I ,

522

I~2(T) - ~,(T)I >_ iG(T) - (x(T)l, and, consequently,

I~(0)1 = 16(0) - 6(0)1 > ~ T I 6 ( T ) - 6(T)I = ~"r6, contrary to the condition that y E Ds(x). It [s more difficult to show that I((T)I < 6 (the specificity of flows as compared to cascades is connected just with (). We assume that ]((T)I = 6. Let z = exp~,,:(~g,=(r/(T), ((T)), ~?(T), r so that z 6 W~'~(gTx). (It is not inconceivable yet that z = gTy.) Then, according to (7.2) (to be m o r e precise, according to the analog of (7.2) for W " , W"; in what follows I do not specify this, considering it to be obvious), z 6 W~5(g~gTx) with some r E [-26, 25], and, according to (7.4), Iv - 6{ < 5/4. This means that gt-W z 6 W~6(g~gtz) C Das(gtz) for t 6 [0, T] (see (7.5) and (7.3)) and we can represent gt-Tz as g ' - r z = exp~,A6Ct) + ~ ( t ) + 6 ( 0 )

for a small 6. We have {(~(0) - r I < 6/4 according to (7.4), and therefore Ir > 6/2. This already excludes the possibility admitted earlier that z = g~ since the ( = ('(0) coordinate for y is such that I([ < 5/4. Let us now apply Lemma 1 with k = 10 to yl = g-Wz, Y2 = Y. Since

1,7,(T) - ,7~(T)I = 0,

{G(T) - el(T)[ = 0,

gty # z,

it foLlows that ~2(T) # [~(T) and therefore }5,(T)

(,(T)I _> 101,72(T) -,7,(T)I,

IOIG(T)-

G(T)I.

Consequently, if 35 < 6(10), then 26___ 16(0)-6(0)l > I016(0)-6(0)l. But I(2(0){> 6/2, {(1(0){< 6/4, {(2(0)- (i(0){ > 5/4, and it turns out Chat 26 > 106/4, which is not true. L e m m a 4. Let a > O, b > O, a + b < 1. There exists 5 = 6(a, b) > 0 such that for 0 < 5 < 5(a, b) it follows from the fact that x 6 A and y 6 A N W~6(x ) that W:~(y) 6 W~(x). (Of course, 6(a, b) also depends on the dynamical system be!ng studied, on A, and on the metric used, but these objects are understood to be fixed, and therefore the dependence of 6 on these quantities is not explicitly indicated in the notations.) Proof. For any e > 0 there is 6(e) > 0 such that if 0 < 5 < 6(e) and z E A, ~ E E~, I~1 -< 6, then {'M~){ -< ~l~l or 1'/~(~)1 --< e{~{, IG(~){ - (~ -

~)l~l,

and, in the case of a flow, we have

(i 2e)[(I -

_< 1~ + ,~=(()+ G((){ -< (I + 2e)l~l.

I f X ~ T~M and IXI < r, then p(x,exp=X) = IXl, Therefore, if~ 6 E~, I~1 _< 6, then

(1 -

2e)l({ < p(x, exp~(~ + ~/=(~))_< (I + 2~){~{ 523

in the case of a cascade (actually, we could have written 1 4- ~ in this case raok~r than 1 + 2e) and (1 - 2e)l~] _< p(x,exp~(~ + ~ ( ~ ) + (~(~))) _< (1 + 2e)l~[ in the case of a flow. Let us take e > 0 such that ~1-2~ ( a + b) = 1. We shall show that ~(a, b) = ~(e) possesses the required properties. In what follows, 0 < 6 < ~(a, b). In particular, then it follows from y E W~(x) that p(x, y) < (1 + 2r Let z E W~8(y) so that z = expy(~' + r/,(~'))

or

z = expy(~' + r/~(~') + ~,(~')),

where f' e E~, [f'] < a& We must prove that z E W~(x). Let us consider a curve [0,11 - .

where

w(O) = exp~(0~' + r/y(0~')) or w(O) = expy(0~' + r/y(0~') + r It starts at the point y E W~6(z) and connects it with z. Let us prove that it does not leave W~(x). We assume that this is not so. Then there is 01 E (0, 1) such that w([0,0a]) C W;(x),

w(O~)e OW;(x),

where OW~(x) is the boundary of W~(z) as a topological ball of the corresponding size or as a domain on W'(x) (in the topology of an immersed submanifold). As long as the point w(O) remains in W~(x), the local coordinates (~, r/) or (~, r/, r are defined for it, with 7/= r/~(~) or r/=.r/~(~), ~ = ~'~(~). Suppose that

w(O) = exp=({(O) + r/.({(e)))

or

w(O) = exp=(~(O) + r/=(~(O)) + ~'.(~(0)))

in these terms for all 0 E (0, 0~). Since w(O~) e OW~(x), it follows that ~(01) = & But

p(y,w(o))

= 1o ' +

or

1o '+

+

p(y, w(O)) 0, k,~ --* cr such that

gix,, e D6(gix)

gk~+lx,~q~D6(gk~+xx).

for i e [0, k,,],

Suppose that

gixn=expgi~(~,~(i)(y)+~ln(i)(t))

for

i e [O, kn]

in terms of the (~, r/) coordinates in D6(g~x). According to Lemma 2,

(8.4)

I# (kn)l ___c6 with some c > 0 independent of z and x~. Let us prove that

-

0, then g-tWO(L) C W['(L), and so the negative semi-trajectory of any point from W~'(L) lies entirely in W~'(L), and therefore it is impossible to leave W~(L) and then return there.) This means that'z,, r W~'(z) in the case of a cascade and z,, r W~"(z) in the case of a flow (for sufficiently large n). If there are z,, in this case, with arbitrarily large n, for which condition (8.2) or (8.3) is fulfilled (with z and z replaced by x,, and z,,), then we have case 1 again. Now if all z,, with sufficiently large n lie on W~(z) or W~'~(z) (we have seen that they cannot lie on W~'(z) or W~"(z)), then it foUows that L C W'(L1) n W"(LI), with L # L~, since the trajectory Lx is periodic and L is nonperiodic. (d) It remains to consider only the case (case 3) where, for sufficiently large n, all x,~ E W~(y), if we speak of a cascade, and z~ E W~'~(y), if we speak of a flow. In the case of a flow, we can slightly "shift" z~ along the trajectory L to make z,, also belong to W~(y). As at the beginning of (c), here the manifold W'(y) is periodic, contains a periodic point z E A, and, if L1 is the trajectory of the latter, then L C Ws(L1). In this case, L # L1 since the trajectory L1 is periodic and L is not. This proves one of the two inferences of Theorem 2, namely, that in which we speak of W'. (e) In order to prove the second inference of the theorem in case 3, we consider a(L). As it was for w, a(L) # L. We take yx E a(L) \ L. There is a sequence t,, --* - c r such that the points z,, = gt"z tend to yx(We no longer need the t,, and z,, used before, and so I use the same letters here.) Now case 3 is divided into three subcases, for two of which we do not need the result given in the preceding paragraph, but we use it in the third subcase. If (8.2) or (8.3) is satisfied with yl substituted for x, then everything is proved, since the time reversal leads to case 1 with yl instead of y. If the infinite number x,, E W~(yl) (when we speak of a cascade) or zn 6 W;r*(yl) (when we speak of a flow), then, by means of time reversal, we reduce the situation to case 2, i.e., the a-limit set becomes the w-limit, the roles of W ~ and W s are switched. Now the statement proved in (c) means that either everything reduces to Case 1 or there is a periodic trajectory L2 C A for which L2 # L and L C W'(L2) f3 W"(L2). Finally, if all z,, E W~'(yx) in the case of a flow or all z,, E W~"(y~) in the case of a cascade, then we carry out time reversal and refer the reader to (d). The statement proved there means now that L lies on a nonstable manifold W~'(L~) of some periodic trajectory L2 C A. This proves the second inference of Theorem 2.

LITERATURE

CITED

. D. V. Anosov and V. V. Solodov, "Hyperbolic sets," In: Sovremennye Problemy Matematiki. Fundamental'aye Napravleniya, Vol. 66, Dynamicheskie Systemy-9, Itogi Nauki i Tekhn., All-Union Institute for Scientific and Technical Information (VINITI), Akad. Nauk SSSR, Moscow (1991), pp. 12-99. . D. V. Anosov, "Geodesic flows on closed Riemannian manifolds of negative curvature," Tr. Mat. Inst. Akad. Nauk SSSR, 90 (1967). .

528

D. V. Anosov and I. U. Bronshtein, "Topological dynamics," In: Sovremenrtye Problemy Matematiki. Fundarnental'nye Napravleniya, VoI. 1, Dynamicheskie Systemy-1, Itogi Nauki i Tekhn., All-Union Institute for Scientific and Technical Information (VINITI), Akad. Nauk SSSR, Moscow (1985), pp. 204-229.

4. A. G. Maier, "On the ordinal number of central trajectories," Dokl. Akad. Nauk SSSR, 59, No. 8, 1393-1396 (1948). 5. A. G. Maier, "On central trajectories and Birkhoff's problem," Mat. Sb., 26, No. 2, 265-290 (1950). 6. V. V. Nemytskii and V. V. Stepanov, The Qualitative Theory"of DifferentialEquations [In Russian], Gostekhizdat, Moscow-Leningrad (1949). 7. L. P. Shilnikov, "Concerning Maier's papers on central motions," Mat. Zametki, 5, No. 3, 335-339 (1969). 8. L. S. Pontryagin, Smooth Manifolds and Their Applications in t[~e Theory of Homotopy [In Russian], Nauka, Moscow (1976). 9. G. de Rham, Varigtgs Diffgrentiables, Hermann, Paris (1955).

529