Statistical Foundations of Econometric Modelling

llII(IIl1!j!ItIIIj!!

Contents

Foreword

xi

by David Hendry

XV

Preface Acknowledgements List of symbols and abbreviations

lntroduction

Part I

Econometric modelling, a preliminary view 1 1.1 Econometries - a brief historical overview 1.2 Econometric modelling - a sketch of a methodology Looking ahead

2

2.1 2.2 2.3

study of data Histograms and their numerical Frequency curves Looking ahead

DescriptiYe

Probability

Part 11 3 3.1 3.2 3.3

characteristics

15 22 23 23 27 29

theory

Probability The notion of probability The axiomatic approach Conditional probability

33 34 37 43

Contents

and probability distributions Random Yariables The concept of a random variable The distribution and density functions The notion of a probability model distributions Some univariate Numerical characteristics of random variables

47 48 55 60 62 68

5 5. 1 5.2 5.3 5.4

and their distributions Random vtors Joint distribution and density functions Some bivariate distributions Marginal distributions Conditional distributions

78 78 83 85 89

6 6. 1 6.2* 6.3

Functions of random Yariables Functions of one random variable Functions of several random variables Functions of normally distributed random

,' 4 4.1 4.2 4.3 4.4 4.5

a

96 96

99 variables,

108 109

summary

Looking ahead Appendix 6.1 The normal

distributions

-

and

related 110

7 7. 1 7.2 7.3

The general notion of expectation Expectation of a function of random variables Conditional expectation Looking ahead

116 116 121 127

8* 8. 1 8.2

Stochastic processes The concept of a stochastic process of a stochastic Restricting the time-heterogeneity

130 131

process

Restricting the memory of a stochastic process Some special stochastic processes

8.3 8.4 8.5 9 9. 1 i?2 . k) but subject to (observation)error u. This formulation was used to model a situation such that: =

it is suspected that for settings linear relation: k )., jj;1 pix; =

xl

,

xc,

.

.

xk there is a value y related

,

.

by a

,

i

=

where p is unknown. A number T of observations on y can be made, corresponding to F different sets of (x: xkl,i.e. we obtain a data set (yf, xfk), r 1s2, xf : xta, 'Cbut the readings on y, are subject to error. (See Heyde and Seneta (1977).) uzzpi)

,

,

.

.

.

,

.

.

.

,

.p

=

.

.

.

,

The problem as seen at the time was one of interpolation (approximation), that is, to the value of #. The solution proposed came in the form of the least-squares approximation of p based on the minimisation of Gapproximate'

u'u=ty

-x#)'(y

-X#),

whichlead to /= (X'X)--1X'y

( 1.4)

A brief historical oveniew

(see Seal (1967:. The problem, as well as the solution, had nothing to do with probability theory as such. The probabilistic arguments entered the problem as an afterthought in the attempts of Gauss and Laplace to justify T are assumed the method of least-squares. lf the error terms lk1, t 1,2, to be independent and identically distributed (IlD) according to the normal distribution, i.e. =

c2),

.::(.0,

ld, x.

t

1, 2,

=

.

.

.

,

.

.

.

,

(1.5)

T)

optimal solution' from a probabilistic then j- in (4) can be justified as viewpoint (see Heyde and Seneta (1977),Seal (1967), Maistrov (1974)). The Gauss linear model was later given a very different interpretation in the context of probability theory by Galton, Pearson, and Yule, which gave rise to what is nowadays called the linear regression model. The model given in (2)is now interpreted as based wholly on probabilistic arguments, y, and Xf are assumed to bejointly normally distributed random variables and X/ is viewed as the conditional expectations of pf given that X! xf (Xr takes the Tt.i.e. value xp) for r 1, 2, kthe

=

=

.

.E()rry/Xt xf ) =

=

.

.

,

#'xt,

t

1, 2,

=

.

.

(1.6)

'C

,

.

with the error term u, defined by lz, yf - F(y/ Xr further details). The linear regression model =

.pf .Et-p,Xf xl) + =

=

u,,

t

=

1, 2,

.

.

.

,

=

xf)

(see Chapter 19 for

T

can be written in matrix form as in (2) and the two models become indistinguishable in terms of notation. From the modelling viewpoint, however, the two models are very different. The Gauss linear model relationship where the xffs are known constants. On describes a the other hand, the linear regression model refers to a relationship where yf is related to the observed values of the random vector Xl (forfurther discussion see Chapter 19). This important difference went largely unnoticed by Galton, Pearson and the early twentieth-century applied econometricians. Galton in particular used the linear regression causal relationships in support of his theories model to establish of heredity in the then newly established discipline of eugenics. The Gauss linear model was initially developed by astronomers in their relationships for planetary orbits, using a attempt to determine large number of observations with less than totally accurate instruments. The nature of their problem was such as to enable them to assume that their theories could account for all the information in the data apart from a vhfrc-ntpfsp(see Chapter 8) error term uf. The situation being modelled design' situation because of the relative resembles an of constancy the phenomena in question with nature playing the role of the 'law-like'

'predictive-like'

'law-like'

'law-like'

'experimental

8

A preliminary

view

experimenter. Later, Fisher extended the applicability of the Gauss linear phenomena using the idea of randomisation model to (see Fisher ( 1958:. Similarly, the linear regression model. firmly based on expectation, was later extended by Pearson to the the idea of conditional stochastic of regressors (see Seal ( 1967)). case ln the context of the Gauss linear and linear regression models the convergence of descriptive statistics and the calculus of probability became a reality, with Galton (1822-1911.q, Edgeworth g1815- 1926q, Pearson g1857-19361 and Yule g1871-195 1q being the muin protagonists. ln the hands of Fisher (1890-1962jthe convergence was completed and a new One of the most important modelling paradigm was proposed. contributing factors to these developments in the early twentieth century was the availability of more complete and reliable data towards the end of the nineteenth century. Another important development contributing to the convergence of the descriptive study of data and the calculus of probability came in the form of Pearson's family of frequency curves which provided the basis for the transition from histograms to probability density functions (seeChapter 2). Moreover, the various concepts and techniques and provide the developed in descriptive statistics were to be reinterpreted basis for the probability theory framework. The frequency curves as used in descriptive statistics provide convenient for the observed data at hand. On the other hand, probability density functions were postulated as imodels' of the population giving rise to the data with the latter viewed as a representative sample from the former. The change from the descripti,e statistics to the probability theor approach in statistical modelling went almost unnoticed until the mid-1930s when the latter approach formalised by Fisher dominated the scene. During the period 1890-1920 the distinction between the population from where the observed data constitute a sample and the sample itself was blurred by the applied statisticians. This was mainly because the paradigm tacitly used, as formulated by Pearson, was firmly rooted in the descriptive statistics tradition where the modelling proceeds from the observed data in hand to the frequency (probablity)model and no distinction between the population and the sample is needed. In a sense the population consists of the data in hand. In the context of the Fisher paradigm, however, the modelling of a probability model is postulated as a generalised description of the actual data generation prpcess (DGP), or the population and the observed data are viewed as a realisation of a sample from the process. The transition from the Pearson to the Fisher paradigm was rather slow and went largely unnoticed even by the protagonists. ln the exchanges between Fisher and Pearson about the superiority of the maximum likelihood estimation over the method of moments on efficiency grounds, Pearson 'experimental-like'

'models'

1.1

A brief historical oveniew

9

never pointed out that his method of moments was developed for a different statistical paradigm where the probability model is not postulated a priori (see Chapter 13). The distinction between the population and the sample was initially raised during the last decade of the nineteenth century and of the early twentieth century in relation to biqher t?l'J:'r approximations limit theorem (CLT) results emanating from Bernouli, Moivre De central and Laplace. These limit theorems were sharpened considerably by the Russian school (Chebyshev (1821-941, Liapounov (1857-1922q,

Markov g1856-1922q, Kolmogorov g19031 (seeMaistov ( 1974)) and this period. Edgeworth and Charlier, among extensively used during expansions which could be used to improve the others, proposed asymptotic offered by the CLT given sample size T (see Cramer approximation for a The of development ( 1972)). a formal distribution theory based on a fixed sample size 'T)however, began with Gosset's (Studenfs) t and Fisher's F. distributions (see Kendal and Stuart ( 1969)). These results provided the basis of modern statistical theory based on the Fisher paradigm. The transition from the Pearson to the Fisher paradigm became apparent in the 1930s when the theory of estimation and testing as we know it today was formulated. lt was also the time when probability theory itself was given its axiomatic foundations by Kolmogorov ( 1933) and tirmly established as part of mathematics proper. By the late 1930s probability theory as well as statistical inference as we know them today were firmly established. The Gauss linear and linear regression models were appropriate for Yule ( 1926) discussed the modelling essentially static phenomena. difficulties raised when time series data are used in the context of the linear regression model and gave an insightful discussion of regressions' (seeHendry and Morgan ( 1986)). ln an attempt to circumvent these problems, Yule (1927) proposed the linear autorenressive model (AR(m)) where the xffs are replaced by the lagged ).fS, i.e. 'non-sense

An alternative model for time-series data was suggested by Slutsky ( 1927) in such data using weighted his discussion of the dangers in showed that weighted He averaging of a white-noise by averaging. process with produce periodicities. Hence, somebody looking for a data series ut can cyclic behaviour can be easily fooled when the data series have been smoothed. His discussion gave rise to the other important family of timeseries models, subsequently called the moinq tkptalv/gcp m()(leI(MA(p)): ksmoothing'

A preliminary

view

Wold ( 1938) provided the foundations for time series modelling by relatinp the above models to the mathematical theory of probability establshed bj Kolmogorov ( 1933). These developments in time series modelling were to have only a marginal effect on mainstream econometric modelling until the mid-7os when a slow but sure convergence of the two methodologies began. One of the main aims of the present boc)k is to complete this convergence in methodology, the context of a reformulated With the above dcvelopments in probability theory and statistical inference in mind, let us consider the history of econometric modelling in the early twentieth eentury. The marginalist revolution of the 187s, with Walras and Jevons the protagonists, began to take root and with it a change of attitude towards mathematical and statistical techniques and their role in studying the economy. ln classical ekronomics ebserved data were used mainly to tendencies in support of theoretical arguments or as kfacts' to be explained. The mathematisation of economic theory brought about by the marginalist revolution contributed towards a purposeful attempt to quantify theoretical relationships using observed data. The theoretical relationships formulated in terms of equations such as demand and supply funetions seemed to offer themselves for quantification using the newly established techniques of correlation and regression. The early literature in econometric modelling concentrated mostly on two general areas, business cycles and demand curves (see Stigler (1954:. This can be explained by the availability of data and th influence of the marginalist revolution. The statistical analysis of business cycles took the form of applying correlation as a tool to separate long-term secular movements, periodic movements and short-run oscillations (seeHooker (1905), Moore (1914)inter alia). The empirical studies in demand theory concentratd mostly on estimating demand curves using the Gauss linear model disguised as regression analysis. The estimation of such curves was fitting' with any probabilistic treated as being arguments coincidental. Numerous studies of empirical demand schedules, mostly of agricultural products, were published during the period 1910-30 (see Stigler ( 1954), Morgan ( 1982), Hendry and Morgan (1986:, seeking to of demand'. These studies establish an empirical foundation for the purported to estimate demand schedules of the simple form 'establish'

tcurve

'law

qD= l

(1.10)

f/() +J1n,

OD

refers to quantities demanded at time l (intentions on behalf of where economic agents to buy a certain quantity of a commodity) corresponding to a range of hypothetical prices (q. By adopting the Gauss linear model line through the these studies tried to approximate' (10) by fitting the of where usually referred to 1, T fq- t ) t 2, t scatter diagram 'best'

/7

'

=

.

.

.

,

1.1

A brief historical overview

11

quantities transacted (or produced) and the corresponding 1. That is, they would estimate

L

=

b1

-1-

b1

,

l

=

1 2s ,

.

.

.

r

prices

f

at time

(1 11)

X

.

using least-squares or some other interpolation method and interpret the estimated coefficients h and /)1as estimates of the theoretical parameters?o and f1j respectively, if the signs and vaiues were consistent wth the law of demand'. This simplistic modelling approach, however, ran into difficulties immediately. Moore (19l4) estimated ( l 1) using data on pig-iron (raw steel) production and price and (or so he thought) a positively sloping demand schedule (Jj > 0). This result attracted considerable criticism from the applied econometricians of the time such as Lehfeldt and issue in Wright (see Stigler ( 1962)) and raised the most important econometrie modelling; te connccrft?n bpfwtzpn be estimated equations pt/ullf/tvlt?l %'cctpntpl?c' rs/ng observed data tznJ the theovetical l-elationships argued that 1915), commenting Moore*s Lehfeldt theory. ( on equation estimated supply demand but the a curve. Several was not a applied econometricians argued that Moore's estimated equation was a mixture of demand and supply. Others, taking a more extreme view, raised the issue of whether estimated equations represent statistical artifacts or genuine empirical demand or supply curves. lt might surprise the reader to learn that the same issue remains largely unresolved to this day. Several 'solutions' have been suggested since then but no satisfactory answer has emerged. During the next two decades (1910-30) the applied econometricians struggled with the problem and proposed several ingenious ways to 'resolve' some of the problems raised by the estimated versus theoretical relationships issue. Their attempts were mainly directed towards specifying theoreteal models and attempting to rid the observed data more of information'. For example, the scenario that the demand and supply curves simultaneously shifting allowing us to observe only their intersection points received considerable attention (see Working (1927) incr ('/fll. The time dimension of time-series data proved particularly given that the theoretical model was commonly static. difticult to Hence the observed data the data was a popular way to In order to bring them closer to the theoretical concepts purporting to estimated-theoretical As argued Morgan the below (1982/. measure (see tssue raises numerous problems which, given the state of the art as far as statistical inference is concerned, could not have been resolved in any Attisfactory way. ln modern terminology these problems can be xummarised under the following headings: theoretical variables versus observed data; : Sdiscovered'

'discovery',

4realistic'

kirrelevant

'solve'

Sdetrending'

'

kpurify'

A preliminary

(ii) (iii) (iv) (v)

statistical

N'iew

model specificalion'. misspecification testing;

statistical

specification

testing, reparametrisation, identification; theoretical models. By the late 1920: there was a deeply felt need for a more organised effort to face the problems raised by the early applied econometricians such as Moore. Mitchell, Schultz. Ctark- Working, Wallace, Wright, fnlcr (Ilftl. This led to the creation of the Econometric Society in 1930. Frisch, Tinbergen of international society' and Fisher (Irving) initiated the establishment //- r/lt, adt,ancement (?/' economic l/:t?t??'). in its rt?lation to statistics and Anfl/lpnlrkrs. The decade immediately after the creation of the Econometric Society can be characterised as the period during which the foundations of modern econometrics were laid mainly by posing some important and insightful questions. An important attempt to resolve some of the problems raised by the estimated theoretical distinction was mae by Frisch ( 1928) (1934). Arguing from the Gauss linear model viewpoint Frisch suggested the so-called errors-in-variables formulation where the theoretical relationships defined Jtkf) in terms of theoretical variables pt EEEfy 1 ,, defined by the system of k linear equations: empirical

versus

'an

bare

.

A'p f and

,

5

,

=

.

=0

the observed data yt BE (.v11 yg

.

pt + ct

.

.

.

,.J.!k2 )'

are related

to pt

via

,

where cr are errors of measurement. This formulation emphasises the distinction between theoretical variables and observed data with the measurement equations ( 13) relating the two. The problem as seen by Frisch was one of approximation (interpolation) in the context of linear algebra in the same way as the Gauss linear model was viewed. Frisch, however, with his coqjluence analq,sis offered no. proper solution to the problem. A complete solution to the simplest case was only recently provided, 50 years later, by Kalman ( 1982). lt is fair to say that although Frisch understood the problems raised by the empirical theoretical the quotation distinction as below testifies, his formulation of the problem turned out to be rather unsuccessful in this respect. Commenting on Tinbergen's 'A statistical test of business cycle theories'. Frisch argued that: The qucstion of what connection there is between relations we work with in theory and those we get by fitting ctlrves to actual statistical data is a very delicatc onc. Tinbergen in his work hardly mentions it. He more or Iess takes it for granted that the relatons he has found are in their nature

1.1

A brief historical overview

the same as of this sort,

the theory This is, in my opinion, unsatisfactory. In a work and bctween slatisifL'fll the connection l,b.ltll'et'il'al l-ta/f-/ri?ns must be thoroughly understood and the nature of the intbrmation which the statistical relations furnish - although they are not identical with the theoretical rclations - should be clearly brought out. (Sce Frisch ( 1938), pp. 2 - 3.) .

.

.

As mentioned above, by the late 1930s the Fisher paradigm of statistical inference was formtllated into a coherent body of knowledge with a firm foundatin in probability theory. The first important attempt to introduce this paradigm into econometrics was made by Koopmans ( 1937). He formulation in the proposed a resetting of Frisch's errors-in-variables context of the Fisher paradigm and related the least-squares method to that of maximum likelihood, arguing that the latter paradigm provides us with additional insight as to the nature of the problem posed and its (estimation). Seven years later Haavelmo (a student of Frsch) published his celebrated monograph on b'T'heprobability approach in econometrics' (see Haavelmo ( 1944)) where he argued that the probability approach (the bsolution'

Fisher

paradigm)

was

the most promising

approach

to econometric

modelling (see Morgan ( 19841). His argument in a nutshell was that if statistical inference (estimation, testing and prediction) are to be used systematically we need to accept the framework in the context of which these results become available. This entails formulating theoretical propositions in the context of a well-defined statistical model. ln the same monograph Haavelmo exemplified a methodological awareness far ahead of his time. ln relation to the above discussion of the appropriateness of the Gauss linear model in modelling economic phenomena he distinguished between observed data resulting from: ( 1) experiments that we should like to make to see if ccrtain real economic intlucnces' would phenomena - when artificially isolatcd from verify certain hypotheses, and 'other

(2) the stream own enormous observers

of experiments

that Nature is steadily turning out from her wc mcrely watch as passive

laboratory, and which

He went on to argue:

ln the first case we can make the agreement or disagreement between theory and facts depend upon two things: the facts we choose to consider, ln the second casc we can only try to as well as our theory about them adjust our theories to reality as it appears before us. And what is the meaningof a design of experiments in this case'? lt is this: Wc try to choose a theory and a design of experiments to go with it. in such a way that thc .

.

.

.

A preliminary

view

resulting data would be those which we get by passive observation reality.And to the extent that we succeed in doing so, we become master reality - by passive agreement.

t)! 0:

Now if we examine current economic theories, we see that a great many of them, in particular the more profound ones, require experiments of the first type mentioned above. On the other hand, the kind of economic data that we actually have belong mostly to the second type. (See Haavelmo (1944).)

Unfortunately for econometrics, Haavelmo's views on the methodology of econometric modelling had much lesser influence than his formulation of a statistical model thought to be tailor made for econometrics; the so-called simultaneous equations model. ln an attempt to capture the interdependence of economic relationships Haavelmo ( 1943) proposed an alternative to Frisch's errors-in-variables and formulation where no distinction between theoretical variables observed data is made. The simultaneous equation formulation was specified by the system F'y f + A'xf + tt

=

0,

where yr refers to the variables whose behaviour this system purports to explain (endogenous)and x/ to the explanatory (extraneous) variables whose behaviour lies outside the intended scope of the theory underlying ( 14) and zt is the error term (seeChapter 25). The statistical analysis of (14) provided the agenda for a group of distinguished statisticians and econometricians assembled in Chicago in 1945. This group known as the Cowles Foundation Group, introduced the newly developed techniques of estimation (maximum likelihood) and testing into econometrics via the simultaneous equation model. Their results, published in two monographs (see Koopmans (1950)and Hood and Koopmans (1953)) were to provide the main research agenda in econometrics for the next 25 years. lt is important to note that despite Haavelmo's stated intentions in his discussion of the methodology of econometric modelling (seeHaavelmo (1944)), the simultaneous equation model was later viewed in the Gauss linear model tradition where the theory is assumed to account for a11 the information in the data apart from some non-systematic (white-noise) errors. lndeed the research in econometric theory for the next 25-30 years was dominated by the Gauss linear model and its misspecification analysis equations model and its identification and and the simultaneous estimation. The initial optimism about the potential of the simultaneous equations model and its appropriateness for econometric modelling was not fulfilled. The problems related to the isstle of estimated versus

A sketcb of a metbodology

mentioned above were largely ignored because of theoretical relationships this initial optimism. By the late 1970s the experience with large equations macroeconometric models based on the simultaneous formulation called into queson the whole approach to econometric modelling (seeSims (1980),Malinvaud (1982),fnl!r t:!ia). The inability of large macroeconometric models to compete with BoxJenkins ARIMA models, which have no economic theory content, on prediction grounds (see Cooper (.1972)) renewed the interest of econometricians to the issue of sttzlic tllptlry tlcrsus Jynclmfcrfmpseries Jcltl raised in the 1920s and 30s. Granger and Newbold ( 1974) questioned the conventional econometric approach of paying little attention to the time series features of economic data', the result of specifying statistical models using only the information provided by economic theory. By the late 1970s it was clear that the simultaneous equations model, although very useful, was not a panacea for al1 econometric modelling problems. The whole in view of the econometric methodology needed a reconsideration experience of the three decades since the Cowles Foundation. The purpose of the next section is to consider an outline of a particular approact to econometric modezing wtch takes account of some of the problems raised above. lt is only an outline because in order to formulate the methodology in any detail we need to use concepts and resultswhich are developed in the rest of the book. In particular an important feature of the proposed methodology is the recasting of statistical models of interest in econometrics in the Fisherian mould where the probabilistic assumptions are made directly in terms of the observable random variables giving rise to the observed data and not some unobservable error term. The concepts and ideas involved in this recasting are developed in Parts 1l-IV. Hence, a more detailed discussion of the proposed methodology is given in the epilogue.

1.2

Econometric modelling - a sketch of a methodology

In order to motivate the methodology

of econometric

modelling adopted

below let us consider a simplistic view of a commonly propounded methodology as given in Fig. 1.1 (for similar diagrams see lntriligator ( 1978), Koutsoyiannis (1977),inter 4?/fJ), ln order to explain the procedure represented by the diagram let us consider an extensively researched theoretical relationship, the transactions demand for money. There is a proliferation of theories related to the demand for money which are beyond the scope of the present discussion (fora survey see Fisher (1978:. For our purposes it suffices to consider a simple theory where the transactions demand for money depends on income. the price level and

A preliminary

view

Data Prediction Econometric model

Theoretical

Theory

model

(f Orecasting)

Estimation testing

policy evaluation Statistical inference

Fig. 1.1. The

interest

'textbook'

approach

to econometric

modelling.

rate, i.e. MB

=./'(Ft #, 1).

(1.15)

in some Most theories of the demand for money can be accommodated variation of (15) by attributing different interpretations to F. The theoretical model is a mathematical formulation of a theory. ln the present case we expressed the theory directly in the functional form (15) in an attempt to keep the discussion to a minimum. Let the theoretical model be an explicit functional form for ( 15), say MB

or

ln

=

zlyvt'ac.fza

MB

=

atj + rzj ln F + xz ln # + aa ln 1

in log-linear form

with tzll ln being a constant. The next step in the methodological scheme represented ,4

=

by Fig. 1.1 is to transform the theoretical model ( 17) into an econometric model. This is commonly achieved in an interrelated sequence of steps which is rarely explicitly stated. Firstly, certain data series, assumed to represent measurements of the theoretical variables involved, are chosen. Secondly, the theoretical variables are assumed to coincide with the variables giving rise to the observed data chosen. This enables us to respecify (17) in terms of these observable variables, say, V,, f,, Ft and I-t ln Vt

=

txtl +

a1 ln 1-,)+ a2 ln #f + aafr,

(1.18)

The last step is to turn (18) into an econometric (statistical) model by attaching an error term t1t which is commonly assumed to be a normally distributed random variable of the form

A sketch of a methodology

l.2

of the exeluded variables.

the effects n

t

=

x () + x 1 .f?+ a at + u

3l

Adding this error term onto + at

,

t

6

( 18) yields (1

.20)

T,

u here small letters represent the logarithm of the corresponding capital lctters. Equation (20) is now viewed as a Gauss linear model with the cstimation, testing and prediction techniques related to this at our disposal transactions demand for money'. :o analyse next qate ''l'e t%to esttmate 2% ustttg tte statistkal results retated to the Gauss linear model and test the postulated assumptions for the error term. lf any of the assumptions are invalid we correct by respecifying the error term and then we proceed to test the a priori restrictions suggested by the theory such as, tz1 :>: 1, aa cls 1, 1 < (s ??. can only

lead

The question which naturally arises at this stage is whether we can tackle some of the problems raised above in the context of an alternative methodological framework. ln view of the apparent limitations of the framework should be flexible textbook methodology any alternative enough so as to allow the modeller to ask some of the questions raised above even though readily available answers might not always be forthcoming. With this in mind such a methodological framework should attribute an important role to the actual DGP in order to widen the modelling. lndeed. the estimable model intended scope of econometric should be interpreted as an approximation to the aclual DGP. This brings the nature of the observed data at the centre of the scene with the statistical model being defined directly in terms of the random variables giving rise to the data and not the error term. The statistical model should be specified as a generalised description of the mechanism giving rise to thedata. in view of the estimable model- because the latter is going to be analysed in its context. A sketch of such a methodological framework is given in Fig. 1 An important feature of this framework is that it can include the textbook methodology as a speeial case under certain conditions. When the actual DGP is to resemble the conditions assumed by the theory in question (Haavelmo type one observed data) then the theoretical and estilnable models could coincide and the statistical model could differ from these by a white-noise error. In general, however. we need to distinguish between them even though the estimable model might not be readily available in some cases such as the case of the transactions demand for money (see Chapter 23). In order to be able to turn the above skeleton of a methodology into a ftllly fleshed framework we need to formulate some of the concepts involved in more detail and discuss its implementation at length. Hence, a more detailed discussion of this methodology is considered in the epilogue where the various components shown in Fig. 1 are properly defined and their role explained. In the meantime the following w'orking definitions will stlffice for the discussion which follows: .2.

-designed-

,2

Tlle()l-

'.'

a conceptual

constrtlct

provid ing an ideal ised description of the us to seek

phenomena within its intended scope which will enable cxplanations and predictions related to the actual DGP.

A sketch of a methodology

l.2

Actual data generating

Theory

Theoretical

Observed data

model

-

-

r'-

l l

l Statistical model

Estimable model

I

l

I

1 1

l

l

Estimation Misspecification Reparametrisation Model selection

l l I

I I l l 1 l

I I 1 I

l l

I I

l

Empirical econometric prediction,

1

model

poI icy evaluation

I l

1 l

l

potentitlly chosen.

l

1 I I

l l l

Estilnable

3

zyt?t/cp/..a particular form of the theo retical model which is estimable in view of the actual DG P and the observed data

a probabilistic formulation purporting to provide a generalised description of the actual DGP with a view of analysing thtt estimable model in its context. Sttltivstil'tll

Enlpirical

??kt)t/(>/.'

trvtlrltpnt-zr-.

motlel:

a

reformulation

(reparametrisation/'

restriction) of a well-defined estimated statistical model in view of the estimable model which can be used for description. explanation or/and prediction.

A preliminary

view

Looking ahead its main aim is the statistical As the title of the book exemplifies. of modelling. econometric In relation to Fig. 1.2 the book foundations mainly The concentrates on the part u ithin the dotted rectangle. statistical model in lerms of the variables giving rise to the of specification a observed data as well as the related statistictl inference results will be the subject matter of Parts 11and 111.ln Part 15' N'arious statistical models of interest in econometric modelling and the related statistical inference results will be considered in some detail. Special attention will be given to the procedure from the specification of the stttistical model to 1he dcmand for money of the empirical econometric model. The transactions example considered above will be tlsed th roughout Part 1V in an attempt to awaiting the tlnaware il1 the context of the textbook illustrate the well with the alternative methodology methodology as as compare this formalised in the present book. modelling Parts 11 and lll form an integral part of econometric and viewed of and the concepts should not be as providing a summary definitions to be used in Part lV, A sound background in probability theory and statistical inference is crtlcial for the implementation of the approach adopted in the present book. This is mainly becatlse the modeller is required statistical model taking into consideration the to specify the nature of the data in hand as well as the estimable model. This entails making decisions about characteristics of the random variables which gave independence, rise to the observed data chosen such as normalitys stationarity- mixing, before any estimation is even attempted. This is one of the most crucial decisions in the context of econometric modelling because model renders choice of the statistical the related an inappropriate reader advised statistical inference concltlsions invalid. Hence. the is to view of Parts 11 and 11I as an integral part econometric modelling and not as reference appendices. ln Part IV the reader is encouraged to view econometric modelling as a thinking person's activity and not as a sequence of technique recipes. Chapter 2 provides a very brief introduction to the Pearson paradigm in an attempt to motivate 1he Fisher paradigm which is the subject matter of Parts 11 and 111. tdesign'

'dangers'

'appropriate'

Additional references

C H AP T E R 2

Descriptive study of data

2.1

Histograms

and their numerical characteristics

By descriptive study of data we refer to the summarisation and exposition of observed data as well as (tabulation, grouping, graphical representation) such as measures of location, the derivation of numerical characteristics dispersion and shape. Although the descriptive study of data is an important facet of modelling with real data in itself, in the present study it is mainly used to motivate the need for probability theory and statistical inference proper.

In order to make the discussion more specific let us consider the after-tax personal income data of 23 000 households for 1979-80 in the UK. These data in raw form constitute 23 000 numbers between f 1000 and f 50000. This presents us with a formidable task in attempting to understand how income is distributed among the 23 000 households represented in the data. The purpose of descriptive statistics is to help us make some sense of such data. A natural way to proceed is to summarise the data by allocating the The number of intervals is chosen a priori numbers into classes (intervals). and it depends on the degree of summarisation needed. In the present case the income data are allocated into 15 intervals, as shown in Table 2. 1 below (see National lncome (CH(/ Expenditul'e ( 1983)). The first column of the table shows the income intervals, the second column shows the number of incomes falling into each interval and the third column the relative frequency for each interval. The relative frequency is calculated by dividing the number of observations in each interval by the total number of observations. Summarising the data in Table 2.1 enables us to get some idea of how income is distributed among the various classes. lf we plot the relative frequencies in a bar graph we get what is known as the histogram,

23

Descriptive study of data

0. 16 0. 15 0. 14 0.13 U'0. 12 11 . E 0. # 0.10

,/ 0.09 1 0.08 g o o7

a

0.06

c.c5

0.04 0.03 0.02 0,01

0

1

2

3

4

5

6

7

8

9

10

11

12 13 14

15

l ncome Fig. 2.1. The histogram and frequency polygon of the personal

income

data.

shown in Fig. 2. 1. The pictorial representation of the relative frequencies gives us a more vivid impression of the distribution of income. Looking at the histogram we can see that most households earn less than E4500 and in some sense we can separate them into two larger groups: those earning between f 1000 and f 4500 and those above f4500. The first impression is

and their numerical characteristies

Histograms

that the distribution rather smilar.

of income inside these two larger groups appears to be

For further information on the distribution of income we could caiculate location, charaeteristcs describing the hstogram's varous numerical dispersion and shape. Such measures can be caleulated directly in terms of the raw data. Howeqrer, in the present case it is more convenient for expositional purposes to use the grouped data. The main reason for this is to introduce variotls concepts which will be reinterpreted in the context of probability theory in Part ll. The n'lpt/l' as a measure of location takes the form 15

()j

.- =

..u

=1

-. '...

L

=

?

...

'y

.

,1

where $: and zi refer to the relative frequency and the midpoint of interval I'. ' Tbe rnott? as a measure of location refers to the value of income that occurs most frequentl) in the data set. ln the present case the mode belongs to the first interval f 1.0- 1.5. Another measure of location is the mtalan referring to when incomes al'e arranged in an the value of ineome in thc luiddle ascenling (01' descending) order according to the size of income. The best cIftnulatit'v qvaph which way to calculate the median is to plot the such consrenient answering questions -Ho5v is more for as many observations fall below a particular value of income?' (see Fig. 2.2). From the cumulative frequency graph we can see that the median belongs to the interval f 3.0-3.5. Comparing the three measures of location we can see that .//gt/l/.eznc)'

$'

1.0 0.9 $

0.8 ' o7 g .

# o.6

+-

y

0.5

';

'

'B 0.4 E

c=

0,3 1

0.2 0. 1

l 1

2

3

4

5

6

Fig. 2.2. The cumulative data.

7

8 9 lncome

10

1 1 12

13

14

15

histogram and ogive of the personal income

Descriptive study of data

mode < median histogram.


0, #f, such that 1 ch Aj ,Z (#.j), isjzzz1, 2, Then #4.) where .P(.,4)refers to the probability assigned to the event z' 1 P(.,4) )J,z 1 a > 1 (seebelow), which is an absurd probability- being p') greater than one; similar inconsistencies arise when S is uncountable. Apart from these inconsistencies sometimes we are not interested in a1Ithe subsets of S. Hence, we need to define ,LF independently of the power set by structure which ensures that no endowing it with a mathematical inconsistencies arise. This is achieved by requiring that .LF has a special mathematical structure, it is a c-field related to S. .,41

z42,

=

U,i)-

zztf

=

.

.

.

,

,

.

.

.

=

=

-4.

=

=

4

Dnron

lf : is called a J-field Let ..F be a set q subsets t#' S. lnt/l/complementation: (f) e: r/ltpnWG .F - closure zzlfl zzlf 1* 1 2, clllsure then ( I-% 6: ,F g () 1 ,.t+-

.4

.:)

U

.kj

=

.

,

.

,

.

utlioll. ctprfflrlh/t.?

Note that

-

t.'/??J ullder

(i) and (ii)taken together imply the following: S e .J; because

(iii) (iv) (v )

.z'1-=

.,4

k.p

S;

,F (from(iii) V= .(J (E

and ,5.) .p1. 1 2 t h en ((-) I i ) G .:F These suggest that a c-field is a set of subsets of S which is closed under complementation- and countable unions and intersections. That is, any of these operations on the elements of will give rise to an element of lt can be checked that the power set of S is indeed a c-field, and so is the set .f3'

.#-),.

(EE .k9j

-4

j

G

i

=

,

,

.

.

,

.

..9

-#'

=

but the set C What we can

)HH), (T/f), (FT)),

(.t(ST)),

.?A)

=

Z,

.),

t(fT),( TH)( is not because ZIC, S'#C, t(ST), (FS) )#C. t do, however, in the latter case is to start from C and construct j

the minimal cn#e'/J generated by its elements. This can be achieved by extending C to include all the events generated by set theoretic operations (unions, intersections, complementations) on the elements of C. Tlaen the minimal c-field generated by C is Z, ((J-1F), (FS)), t(SS), (FT))) c(C). and we denote it by This way of constructing a c-field can be very useful in cases where the events of interest are fewer than the onesgiven by the power set in the case of each H or F a finite S. For example. if we are interested in events w'ith one of c-field and to be the power set, can do as there is no point in defining the ...kL. well with fewer events to attribute probabilites to. The usefulness of this method of constructing c-fields is much greater in the case where S is either in such cases this method is indispensable. Let us infinite or uncountable; .%

=

.6

=

ts',

3.2

The axiomatic

eonsider an example such a c-field.

approach

where S is uncountable

and discuss the construction

of

Example Let S be the real line R

=

be J

t6BxL

=

x c 2)

tx:

-

:c)

x
x (x, :7- ) e c( J4 Taking countable unions of Bx : UJ- 1 ( :f- x (1/))j ( :y- x) s c(./). These imply that c(.f) is indeed a c-field. ln order to see how large a collection c(J') is we can show that events of the form (x, ), gx, also belong to c(J), using set theoretie operations as (x, z) for x < c, and follows'. '

=

-

.

t

.7

=

=

-

.

,

-

,

.:ys),

.cc

tx)

(x, (y.)

=

(

gx, :y:. ) ( =

(x, z) fx )

=

=

(

'ctp -

,

-

'L

I

z4)

#(z4I

-

z'4,4,ra #4z1)

=

In order to illustrate this formula let then since #(-4j) #(.,4) J, .P(z41fo =t,

.

/ll

=

zz

#a(X1)

=

P(XI

j,l 4

I -4)

=

t(ST)) and #( t(1-JT')))

.,4

=

t(/fT), (HH4)

-

=.t,

.4)

=

1

=j=-,

2

as above. Note that .P4-4)>0 for the conditional Using the above rule of conditional

PlAl

l

.?121'--P(,41

f--

,4cl

I

'

probabilities to be delined. probability we can deduce that

Pfz4cl

z11) #4d1), = .P(.,42 .

(3.8) for

.,41,

zzlc

e'

(3.9)

,?>

This is called the mullp/fccllfon rule. Moreover, when knowing that occurred does not change the original probability of z4c, i.e.

-42

has

t

uIX.,4l l -42)-

we say that

ztj

Independence

and

#tz'1ll. zlc

are independent.

r

r

is very different from mutual t?xc/l/sllt?ne'ss in the sense that but Ptz4j ..41rn # .!X.41)and vice versa can both arise. Z lndependence is a probabilistic statement which ensures that the occurrence of one event does not influence the occurrence (or nonoccurrence) of the other event. On the other hand, mutual exclusiveness is a statement which refers to the events (sets) themselves not the associated probabilities. Two events are said to be mutually exclusive when they cannot occur together (see exercise 4). The careful reader would have noticed that the axiomatic approach to probability does not provide us with ways to calculate probabilities for individual events unlike the dassical or frequency approaches. What it provides us with are relationships between the probabilities of certain events when the events themselves are related in some way. This is a feature of the axiomatic approach which allows us to construct a probability modcl without knowing the numerical values of the probabilities but still lets us deduce them from empirical evidence. .42

I

-42)

=

Conditional probability

45

Impovtant concepts

random experiment; classical, frequency and subjective definitions of probability; sample space, eiementary events's c-field. minimal c-field generated by eventss Borel field; probability set function, probability space (S, P( )); conditional probability, independent events, mutually exclusive events. e?6

'

Questions Why do we need probability theory in analysing observed data? What is the role of a mathematical model in attempting to explain real P henomena'? Compare and contrast the classieal and frequency definitions of probability. How do they differ from the axiomatic definition'? Explain how the axiomatic approach formalises the concept of a random experiment 4 to that of a probability space (S, )). Why do we need the coneept of a c-field in the axiomatisation of probability? Explain the concept intuitively. Explain the concept of the minimal c-field generated by some events using the half-closed intervals ( uo, xj, x (E R on the real line as an t%p

.

-

example.

Explain intuitively the continuity property of the probability set function #( ). Discuss the concept of conditional probability and show that #( 1 for some e: is a proper probability set funetion. '

'

.,1

z4)

.t#'

Exerdses Consider the random experiment of throwing a dice and you stand to lose money if the number of dots is odd. Derive a c-field which will enable you to consider your interests probabilistically. Explain your choice. Consider the random experiment of tossing two indistinguishable fair coins and observing the faces turning up. Derive the sample space S, the c-field of the power set L.F'and (i) define the probability set function P( ). Derive the c-field generated by the events )SS) and (T'l). lf you stand to lose a pound every time a coin turns up what is the c-field of interest'? '

'heads'

Probability

46

Consider the effect on S, P ))when knowing that event $at least one F' has occurred and dene the new conditional #a( )).Confirm that for the event probability space (,$x,,.#r4, tails, - two .6

.4

'

-41

'

P,4(z41) =

8.4

c5

z11) .

17(X)

Consider the events IHH). and (FT) and show whether they are mutually exclusive or and independent. Consider the random experiment of tossing a coin until it turns up theads'. Define the sample space and discuss the question of detining a c-field associated with it. Consider the random experiment of selecting a card at random from an ordinary deck of 52 cards. Find the probability of (i) .41 - the card is an ace;

and

.,42- the card is a diamond. Knowing that the card is a diamond show how the original (S, #( )) changes and calculate the probability of ,d

'

yla - the card is the ace of diamonds. Find 174..11 ro

z4cl

derived in (ii).

and compare

Define two events which are: mutually exclusive and (a) mutually exclusive but (b) mutually exclusive not (c) (d) not mutually exclusive

it with the probability

of

..43

independent; not independent;

but independent; and and not independent.

Additional references Barnett (1976).

Giri ( 1974); Mood, Graybill (1973);

and Boes ( 1974); Pfeiffer ( 1978/ Rohatgi

CHAPTER

4

Random variables and probability

distributions

In the previous chapter the axiomatic approach provided us with a mathematical model based on the triplet (S, P( )) which we called a probability space, comprising a sample space S, an event space + (c-field) and a probability set function P( ). The mathematical model was not developed much further than stating eertain properties of P ) and introducing the idea of conditional probability. This is because the model based on (S, Pq ))does not provide us with a flexible enough framework. ,#t

.

.

'

-f6

'

The main purpose of this section

is to change this probability

space

by

mapping it into a much more flexible one using the concept of a random varable. The basic idea underlying the construction of S, #( ))was to set up a framework for studying probabilities of events as a prelude to analysing problems involving uncertainty. The probability space was proposed as a formalisation of the concept of a random experiment & One facet of tf' which can help us suggest a more flexible probability space is the fact that when the experiment is performed the outcome is often considered in relation to somc quantisable attribute; i.e. an attribute which can be represented by numbers. Real world outcomes are more often than not expressed in numbers. lt turns out that assigning numbers to qualitative outcomes makes possible a much more flexible formulation of probability theory. This suggests that if we could find a consistent way to assign numbers to outcomes we might be able to change (,S, #( ))to something more easily handled. The concept of a random variable is designed to do just that without changing the underlying probabilistic structure of (S, % P( )). ,#7k

'

,%

'

'

Random variables

and probability

distributions

4 LHHi

j(J?s)) 1(Fr)l

(/?F)

1(8r) (88) (r/?)l 14F8) (/?r), (rrlt (rr)# 148/-/), jjsr), (rsjj ,

(r8) (rr)

Fig. 4. 1. The relationship probability set function.

4.1

,

!

,

l

I

I 1 1 1 1 I

l

I

0 0.2 0.4 0.6 0.8 1.O

between

sample

space,

c-field

and

The concept of a random uriable

Fig. 4. 1 illustrates the mathematical model (,$', #( )) for the coin-tossing example discussed in Chapter 3 with the c-tield of interest being .F= (S, Z, )(TT)), l(HH,(TT))l(TH),(11T')), 't(1.T),(TH),(H.H)l, ((1.f1.f)), )(SF),(T'S),(FF))). The probability set function #(') is defined on .F and 1j, i.e. #(.) assigns probabilities to the events takes values in the interval in ,F. As can be seen. various combinations of the elementary events in S define the c-field .F (ensure that it is a c-fieldl) and the probability set function #(.) assigns probabilities to the elements of .F. The main problem with the mathematical model (S, #( )) is that the general nature of S and .F being defined as arbitrary sets makes the of #( ) N'ery difficult; its domain being a c-field mathematical manipulation of arbitrary sets. For example, in order to define #( ) we will often have to derive all the elements of .F and tabulate it (a daunting task for large or infinite to say nothing about the differentiation or integration of such a r9;

'

r0,

,@

'

'

.

./-s),

set function.

Let us consider the possibility of defining a function Ar( ) which maps S directly into the real Iine R, that is, '

A'(

.

): S

-+

Rx,

assigning a real number xl to eaeh sl in S by xl xl g R. sl e S. For example, in the coin-tossing experiment we could define the function A' the number of heads'. This maps all the elements of S onto the set Rx .t0, 1, 2) see Fig. 4.2. -tsll,

=

-

=

,

The question arises as to whether

every function from S to R will provide

4.1

The concept of a random

Fig. 4.2. The random example.

variable

variable

Ar-number of

49

Sheads'

in the coin-tossing

us with a consistent way of attaching numbers to elementary events', consistent in the sense of preserving the event structure of the probability space (S, t'f #( )). The answer, unsurprisingly, is certainly not. This is because, although is a function defined on S, probabilities are and in qF the issue we have to face is how to dene the assigned to events values taken by X for the different elements of S in a way which preserves the ln order to illustrate this let us return to the earlier event structure of To value of X, equal to 0, 1 and 2 there correspond some each example. of i.e. S, subset '

x

.?/f

0

'r))

-...

t(T' (z?'z')), tt-rffl, ,

1 2 .:(1-1.r.f)), -+

-.+

and we denote it by -Y-

i(0)

=

t(TT)),

A--

14

1)= t(TS), (f1T)),

X-

1(2)::=,

)(Sf1)),

'(

used by abuse of mathematical using the inverse mapping ) (sinverse language). What we require from A' - 1( ) (or .Y) is to provide us with a correspondence between Rx and S which reflects the event structure of that is, it preserves unions, intersections and complements. ln other words, x'

-

.#1

Random variables

and probability distributions

i for each subset N of Rx the inverse image X - (N) must be an event ' in ,F. Looking at X as defined above we can see that X - (0)G,?A, k..p 1 X - 1 (2)c X - 1( X - 1 (/t.0 J1 t..p f( 2 ))g X - 1( 1) G ))(E 1(( 1) t.p g A' that is, -Y( ) does indeed preserve the event structure of Ry defined by F(tff1F) )= ,X On the other hand, the function Y'( ): S YI..fLHH)) 1, y(.t TH j ) F( TT)) 0 does nOt preserve the event structure 140) 1( of .Lt- since F # F - 1) ( ,i/J This prompts us to define a random variable A' to be any such function satisfying this event prpst?rnfng condition in relation to some c-field defined on Rx; for generality we always take the .%

to)

cz

.t2))

.t#',

'

.ft

.%

-+

'

t

=

=

.#t

Borel field

,?d

.%

=

on R.

Dhnition

l

A random variable X is a p-,(?l valued function S to R wllfc B G 4 on E, tbe set satjles the c'(?nlll't?n that jr ptktr/? Borel X - 1(/) in s.' .Y(.$)e:B, s g s') is an .#(?rl1

.$t?r

=

t

.?8

'gt?rlr

Three important features of this definition are worth emphasising. A random variable is always defined relative to some specific c-

(i)

field R is a ln deciding whether some function F( ) : S (ii) variable we proceed from the elements of the Borel field of the c-field rF and not the other way around. variable'. (iii) A random variable is neither nor Let us consider these important features in some more detail in of the concept of a random enhance our understanding undoubtedly the most important concept in the present book. ,%'

-+

'

..,d

trandom'

random

to those

'a

order to variable)

The question is X( ) : S -+ R a random variable?' does not make any sense unless some c-field ..F is also specified. ln the case of the function Xnumber of heads, in the coin-tossing example we see that it is a random variable relative to the c-field as defined in Fig. 4. 1. On the other hand, F, variable relative to ..R This, however, does random above, is nt?r defined a as preclude from variable with respect to some other crandom F being not a 'y (S,Z,)(S1'1),(f1F)), ((Ff.f),(TT)) ) lntuition field ,.Fy; for instance valued real function .Y( S R we should be able to that for suggests any ): variable. that random ln the previous such Ar define a c-field S is on a section we considered the c-field generated by some set of events C. Similarly, we can generate tr-fields by functions A-( ): S -+ R which turn Indeed above is the nlfnfrnf?/ o'zfleld Ar( ) into a random variable. generated ?v F, denoted by c(F). The way to generate such a minimal c-field 14 is to start from the set of events of the inverse mapping F - ), i.e. 1(0) 1 J-field .t(.fF), (HH); by and generate a FJ' - ( 1) and t(TS), (FT)) taking unions, intersections and complements. ln the same way we can see '

,%

=

.

.

-+

rh

'

.%

'

'

=

=

The concept of a random variable

4.1

that the minimal c-field generated by .Y - the number of heads, c(Ar) coincides with the c-field .LFof Fig. 4.2,. verify this assertion. ln general, however, the c-field .F associated with S on which a random variable X is defined does not necessarily coincide with c(Ar). Consider the function X :(

): S

'

(R

-+

Xl( klSfflll

=

A' !('t(Tf1)))= X1('t(1T)l)

Xl('t(TT)))

1,

=

140)

'(

0

=

'

(4.2) )(TF)) iE

1)= t(ffJ1), (Tf1), (ffT)) iE ,,F (see Fig. 4.2), X( tj ) s (E x, is a random variable on with respect to the c1 s x(ST), (TS))) # indeed field ,.'F c(i-). But c(X:) (S, Z, (41-1f1),

since i(

z%-1

.%

=

,#,

tot, .:

-

.%

=

=

c(A-1) cu ..'F

=

c(aY).

The above example is a special case of an important general result where A',, are random variables on the same probability space -Y:, Xz, (S, P( )) and we define the new random variables .

.

,

.

.%

'

Fz

=

X : + A'a + X s

,

(4.5)

c4Y;,)form an increasing sequence of c-fields in ln the above i.e. c(Yj), example we can see that if we define a new random variable ,Yc( ): S R by ,.'k

.

.

.

,

'

-Yztttf1'flll=

1,

A-2()(f1T)))= Xa()(Tf1)))

=

-+

A-2('t(TT)))=0,

+ X1 (seeTable 4. 1) is also a random variable relative to c(X); then X X is defined as the number of J'ls (see Table 4.1). =

x'j

Note that

z1

is defined as

:at

least one H' and Xz as

Etwo

Jls'.

generated by random variables will prove very useful in the discussion of conditional expectation and martingales (see Chapters 7 and 8). The concept of a c-field generated by a random variable enables us to concentrate on particular aspects of an experiment without having to consider everything associated with the experiment at the same time. Hence, when we choose to define a r.v. and the associated c-field we make an implicit choice about the features of the random experiment we are The above concept of c-fields

interested in. il-low do we decide that some function .X( ): S R is a random variable relative to a given c-field ,i.F?9From the above discussion of the concept of a .

-+

and probability distributions

Random uriables

random variable it seems that if we want to decide whether a function X is a random variable with respect to .F we have to consider the Borel field on R or at least the Borel field on Jx'. a daunting task, lt turns out, however. that this is not necessary. From the discussion of the c-field c(J) generated by the set J x e: J@ ) where Bx ( 'Lt, x(l we know that .t4 c(J) and if .Y( ) is such that .4

./dx

.t#.:

=

=

=

-

,

'

1

X '' (( -

v.

xj

,

)

=

ft

.Y( -)

.'

:

G

(-

xj

py.- s

,!;

,

e:S

j!

g

.kT'

for a11 ( -

then

1

A-- (B4

=

f

(s

:

-Y(s)e: B, s (F S 'j c .F

v-

.

x)Js

.??,

for a1l B g .A

ln other words, when we want to establish that A- is a random variable or define #xt ) we have to look no further than the half-closed intervals (x(Iand the c-field c(..f)ft they generate, whatever the range Rx. Let us g( yt xj, s 6E to use the shorthand notation .YtylGxlj. instead of f(s: number the above in the of of Hs. with argument A' - the consider case in Fig. 4.2. respect to '

':y-

,

'(x)

-

,

,

.t7-

1 x< 2 :;

,(3 ,

=

f#

( TH (T T) ) -

(H

t

1

'r)-

0 .A.J' < 1

,

(HH)

)

(4.9)

1 % )',

,

and thus F -- (( - v- ).q) # .F for ). 0, y 1, i.e. F ( ) is not a random however, f(s : F(s) ,6Ll.j1 G .k) fo r variable with respect to With respect to all )' e: R and thus it is a random variable. The tenn random variable is rather unfortunate because as can be seen h?t?/- (1 vtll-l'tlble'', it i s a real from the above definition A- is tleib t?lvalued function and the notion of probability does not enter its definition. =

=

,

'

.../'),,

..

.

wrandotnn

Probability an attempt

-

enters the picture after the random variable has been defined in model induced by X. to complete the malhematical

Tbe concept of a random variable

Table 4. 1

A' relative to .- maps S into a subset of the real line, on 2 plays now the role of .k ln order to complete the Common assign probabilities to the elements B of the assignment of probabilities to the events B (E @ must the probabilities assigned to the corresponding events need to define a set function #xl ): E0,11 such that

variable

A random

and the Borel field model we need to sense suggests that be consistent with in Formally, we .ut

.?4

.?4

.?#

-->

'

for all B G in the case illustrated

For example, p

i

).

() x ( J

(

)

Px l (h0 l)

=

4.

=

1.4

4

*

,

p x ( tf j J) j

-1-

=

Px , ( 111)

2' =

3-4 .

(4.10)

,.#.

in Table 4. 1 p x (ft 2,))

=

-l4

Px ( )0 l k.p j

5

p x ( j()'j

h11)

=

k.p ( t

1 Px ,

j J).j

=

J- ? 4

etc.,

( ft0 ) rn t 1)) 0. =

j

The question which arises is whether, in order to define the set function #x( ), we need to consider al1 the elements of the Borel field 4. The answer is that we do not need to do that because, as argued above, any such element of can be expressed in terms of the semi-closed intervals ( :s, .xq. This we can implies that by choosing such semi-closed intervals define #xt ) with the minimum of effort. For example, Px( ) fOr x, as defined in Table 4. 1, can be defined as follows: '

..#

-

Sintelligently',

'

'

As we can see, the semi-closed intervals were chosen to divide the real line at the points corresponding to the values taken by X. This way of defining the semi-closed intervals is clearly non-unique but it wll prove very convenient in the next section. The discerning reader will have noted that since we introduced the concept of a random variable A'( ) on (S, .k P( )) we have in effect '

'

Random variables

54

and probability

distributions

developed an alternative but equivalent probability space (R, Px )) induced by X. The event and probability structure of (S, #4 )) is #xt ))and the latter has a preserved in the induced probability space (R, much to handle' mathematical structure; we traded S, a set of arbitrary elements, for R, the real line, ,F' a c-field of subsets of S with 2..d, the Borel field on the real line- and #( ) a set function defined on arbitrary sets with #x( ),a set function on semi-closed intervals of the real line. ln order to P( )) to illustrate the transition from the probability space (S, (Rx, Pxt )) let us return to Fig. 4. 1 and consider the probability space of heads, defined above. As can induced by the random variable z-number variable Fig. 4.3, the random A'( ) maps S into k0,1, 2). be seen from 1q, ( 'Js, 21 we can intervals semi-closed the Choosing ( vs, %, ( which of Borel #xl R field on forms the domain generate a ).The concept of a random variable enables us to assign numbers to arbitrary elements of a as set (S) and we choose to assign semi-closed intervals to events in induced by X. By defining #xt ) over these semi-closed intervals we complete the procedure of assigning probabilities which is consistent with the one used in Fig. 4. 1. The important advantage of the latter procedure is Px )) is a that the mathematical structure of the probability space (R. lot more flexible as a framework for developing a probability model. The purpose of what follows in this part of the book is to develop such a tlexible mathematical framework. lt must be stressed, however, that the original probability space (S, #( )) has a role to play in the new mathematical framework both as a reference point and as the basis of the probability model we propose to build. Any new concept to be introduced has to be related to (S, P( )) to ensure that it makes sense in its context. ..%

'

.%

'

.?d,

'

Ceasier

'

'

.%

'

..%

'

'

':yo,

-

'

.t7-

'

.@,

'

-i/')

'

-%

'

/

s

1(8H), ( rrll

1(/./s)) j(rrlk

CHHL

,

(TH3 t8f) Fr)

s 1(rJ?)

(/./r)) (/.fr) ( 1 r/.?),(r7')t 1(8r) (THb (/-/8)1 ,

-

(

,

,

!

0 !

.-

(

I

!

0

1

2

s

O

1

I

k

1

!

i

i

.

1

o o.2 o.4 0.6.-0.8 I

L

I

1

l

I

!

1

2

.))

to (Rx,?#,#x(

'

1

1

.0

0.2 O.4 0.6 0,8 1.O

Rx Fig. 4.3. The change from (.S,,#-,17(

L

)) induced

by X.

I

I

The distribution and density functions

4.2

The distribution and density functions

4.2

ln the previous section the introduction of the concept of a random variable (r.v.), X, enabled us to trade the probability space (,$', #( )) for .gi

'

(R, #xt )) which has a much more convenient mathematical structure. The latter probability space, however, is not as yet simple enough because Px( ) is still a set function albeit on real line intervals. ln order to simplify it we need to transform it into a point function (a function from a point to a point) with which we are so familiar. The first step in transforming #xl ) into a point function comes in the form of the result discussed in the previous section, that #xt ) need only be defined on semi-closed intervals ( - cc, x g R, because the Borel field 4 viewed c-field the minimal generated by such intervals. With this as can be view of the fact that a1l such proceed in mind in to argue that we can starting intervals have a common ( ) we could conceivably define point function a .%

'

'

'

'

.x(1,

'point-

E:c,

-

F( ): ER--+ .

g0,11,

which is, seemingly, only a function of x. In effect, however, this function will do exactly the same job as Px ). Heuristically, this is achieved by defining F( ) as a point function by '

.

for all x c R, and assigning the value zero to F( - :y.). Moreover, given that as increases the interval it implicitly represents becomes bigger we need to ensure that F(x) is a non-decreasing function with one being its maximum value (i.e. .x

F(x1) % F(x2) if xl :t xc and limx- . F(.x) 1). For mathematical also require F( ) to be continuous from the right.

reasons we

=

'

Dlflnition 2 Let ZYbe a ,-.!?. tljlned I0. 11 dehned tv F(x) #x(( =

(.,

(pll

x1)

'wy-

,

#(

.k',i

=

.

)).The ptpnr

Pr(X G x),

is cf;l//t?J tbe distribution function (1)F) ///t?.'fng pl-operties: (f)

F(x) is rltpn-f/fs?c-tv/.sfng''

( )

F( -

i1*

:y.-

) li rn.x =

-+

-

.,

F( x ) 0 =

,

.jr

t#'

./ntrlforl

all x

A- and

F( 6

R ustkrs/'s

.

)..R

-+

(4.14) the

56

Random variables

and proability

distributions

It can be shown (seeChung ( 1974)) that this defines a unique point function for every set function #x( ). The great advantage of F( ) over #( ) and #xt ) is that the former is a point function and can be represented in the form of an algebraic formula) the kind of functions we are so familiar with in elementary mathematics. '

'

'

'

This will provide us with a very convenient way of attributing probabilities to events. Fig. 4.4 represents the graph of the DF of the r.&'.X in the coin-tossing example discussed in the previous section, illustrating its properties in the of Hs. case of a discrete 1-.1,. --number

Drlrft'?n

3

z4random of l/?t? set

variable

is called discrete if its rc/nfyer Izx is stplot? subset 0 k 1 + 2, Z integers (?J ). -

=

,

.

.

.

ln this book we shall restrict ourselves to only two types of variables, namely, discrete and (absolutely)continuous.

random

Dehnition 4 X is called (absolutelyj continuous i.f its F(x) is continuous .Jtpr alI x iE R and there t.?--?'xr. real tbe that Iine .J( ) on

-4 random lwr/?/g (Iistribution (1non-neqative ,lrlcrft'?n

-s?.,/c/?

.//ntrrft?l?

'

X

F(x)

=

(l) d r/,

4.2

The distribution and density functions

It must be stressed that for A- to be continuous is not enough for the distribution function F(x) to be continuous. The above definition postulates that F(x) must also be derivable by integrating some non-negative function /'(x). So far the examples used to illustrate the various concepts referred to discrete random variables. From now on, however, emphasis will be placed almost exclusively on continuous random variables. The reason for this is that continuous random variables (r.v,-s)are stlsceptible to a more flexible and this helps in the mathematical treatment than discrete r.N construction of probability models and facilitates the mathematical and statistical analysis. r.N we introdtlced the function In defining the concept of a contnuous /'(x) which is directly related to F(x). .'s

.

Dehni rtpn 5

Y e: r'.'i 't'

-

F(x)

=

()

ld A

i.

t't'?? t 1*?.7 ut?l?.s

(4.20)

vx G Er,!l - (Iis'l-ee

./'(u),

.v

(probability)density function ( pt/./') q/' X. A,y, .1-4. example, 1) $-sand (see Fig. 4. 5). I n

saitl t() /?t?tll(?

ln the coin-tossing for a discrete with those of a continuous r.v. order to compare F(x) and where consder the Alet us takes values n the interval lk, ?q and a1l case attributed of the values z are same probability', we express this by saying unljrnll is Adistributed in the interval (k, l and we write Athat t7tl, !?).The DF of Ar takes the form ,/-(0)

./'(

=

./'(2)

=

=

,/'(x)

.

'v

X < (1

(see Fig. 4.6). The corresponding

pdf of X is given b)'

elsewhere.

Random uriables

and probability

distributions

Comparing Figs. 4.4 and 4.5 with 4.6 and 4.7 we can see that in the case of random variable the DF is a step function and the density discrete a function attributes probabilities at discrete points. On the other hand, for a continuous r.v. the density function cannot be interpreted as attributing probabilities because, by definition, if X is a continuous r.v. P(X .x) 0 for a1l x 6 R. This can be seen from the detinition of /'(x) at every continuity =

=

4.2

59

The distribution and density functions

Fig. 4.7. Thedensity

function of a uniformly distributed

random

variable.

(4.23) i.e. .J( ): R .

-+

r0,

v-,l.

(4.24)

Although we can use the distribution function F(x) as the fundamental concept of our probability model we prefer to sacrifice some generality and adopt the density function ,J(x)instead, because what we lose in generality we gain in simplicity and added intuition. lt enhances intuition to view density functions as distributing probability mass over the range of .Y. The density function satisfies the following properties:

(4.25) (4.26) (4.27) (iv)

d

.J(x)=.uF(x),

at every point where the DF is continuous.

(4.28)

Random variables

60

and probability

distributions

Properties (ii)and (iii)can be translated for discrete r.v.'s by substituting t) for j- dx' It must be noted that a continuous r.v. is not one with a continuous DF F( ). Conlfnufrv refers to the condition that also requires the existence of a non-negative function /'( ) such that '

'

*

x

'

.

(4.29) ln cases where the distribution function F(x) is continuous but no integrating function .J(x)existss i.e. (d/dx)F(x) 0 for some x e: J2,then F(x) is sqid to be a sinqular f/f.$r?'l'?l./lf(?n. Singular distributions are beyond the scope of this book (see Chung ( 1974)). =

4.3

The notion of a probability model

Let us summarise the discussion so far in order to put it in perspective. The axiomatic approach to probability formalising the concept of a random #( )), where S experiment J' proposed the probability space (., of all possible of ,F the is the set events and #( ) set outcomes, rcpresents assigns probabilities to events in The uncertainty relating to the outcome is formalised in P( ). The concept of a of a particular performance of random variable A-enabled us to map S into the real line Ii and construct an equivalent probability space induced by X. (R, Px( )),which has a much teasier to handle' mathematical structure, being defined on the real line. Although #xt ) is simpler than #( ) it is still a set funiion albeit on the Borel field Using the idea of c-fields generated by particular sets of #xt ) on semi-closed intervals of the fonn ( vs, xl and defined we events managed to define the point function F( ), the three being related by .:/t

'

'

.k

'

'

.@,

'

'

'

,.?d.

'

'

Pfs: -Y(y)6

(-

:y.,

x(1,

us

6

S)

=

#x( -

.,.'y,

xl

=

(4.30)

F(.x).

The distribution function F(x) was simplified even further by introducing jXvia F(x) du. This introduced further the density function w is definable in closed flexibility into the probability model because algebraic form. This enables us to transform the original uncertainty related to J to uncertainty related to unknown parameters 0 of /'('); in order to emphasise this we write the pdf as $. We are now in a position to define ./?'ld'/J' ()f t/pnsl'r model probabilit parametric in form the of J.' jilnctions a our which we denote by ./'4lk)

,/'(x)

=

./'(x)

./'(x;

'

(l)

=

).J(.x',04, 0 G O ) .

* represents a set of density functions indexed by the unknown parameter (.) (usuallya multiple of 0 which are assumed to belong to a parameterspace the real line). In order to illustrate these concepts let us consider an example

The notion of a probability

model

Fig. 4.8. The density function of a Parcto distributed random of the parameter.

variable

for

differentvalues

family of density functions, the Pareto distribution:

of a parametric (Lp=

ftx.,(?)

=

.

-

p

x

x -9. x

t?+ l ,

x > 0, ()(E (.)

,

xll - a known number O r1+- the positive real line. For each value in 0. /'(.x;p)represents a different density (hencethe term parametric family) as can be seen from Fig. 4.8. When such a probability model is postulated it is intended as a description of the chance mechanism generating the observed data. For example, the model in Fig. 4.8 is commonly postulated in modelling personal incomes exceeding a certain level x(). lf we compare the above graph with the histogram of personal income data in Chapter 2 for incomes over E4500 we can see that postulating a Pareto probability density seems to be a reasonable model. In practice there are numerous such parametric families of densities we can choose from, some of which will be considered in the next section, The choice of one such family, when modelling a particular real phenomenon, is usually determined by previous experience in modelling similar phenomena or by a preliminary study of the data. When a particular parametric family of densities * is chosen, as the appropriate probability model for modelling a real phenomenon, we are in effect assuming that the observed data available were generated by the 'chance mechanism' described by one of those densities in *. The original uncertainty relating to the outcome of a particular trial of the experiment =

,

62

Random variables

and probability

distributions

has now been transformed into the uncertainty relating to the choice of one 0 in 6), say 0*, which determines uniquely the one density, that is, tx,'p*), which gave rise to the observed data. The task of determining 0* or testing some hypothesis about 0* using the observed data lies with statistical inference in Part 111. ln the meantime, however, we need to formulate a mathematical framework in the context of which the probability model (l) can be analysed and extended. This involves not only considering a number of different parametric families of densities, appropriate for modelling different real phenomena but also developing a mathematical apparatus which enables us to describe, compare, analyse and extend such models. The reader should keep this in mind when reading the following chapters to enable him her not to lose sight of the woods for the trees. The woods comprise the above formulation of the probability model and its various generalisations and extensions, the trees are the various concepts and techniques which enable us to describe and analyse the probability model in its various formulations. 4.4

uniYariate

Some

distributionst

ln the previous section we discussed how the concept of a random variable (r.v.) X defined on the probability space (S, P( )) enabled us to construct a general probability model in the form of a parametric family of densities (31).This is intended to be an appropriate mathematical model purporting of real phenomena in a stochastic to provide a good approximation (probabilistic) environment. ln practice we need a menu of densities to describe different real phenomena and the purpose of this section is to consider a sample of such densities and briefly consider their applicability to such phenomena. For a complete menu and a thorough discussion see Johnson and Kotz ( 1969), (1970),(1972). .%

'

(1)

Discrete distributions

(i) Bernoulli distribution experiment J's where there are only two possible and for convenience, that is, S tfailure'). vafiable X by lf we define on S the random 1) p 1, A-tfailure) 0 and postulate the probabilities Przr 0) 1 p we can deduce that the density function of X takes the

Consider a random outcomes, we call (Esuccess',

Asuccess) and Pr(X

=

=

'success'

kfailure'

=

=

=

=

=

-

'!- The term probability distribution is used to denote a set of probabilities complete system (a c-field) of events.

on a

4.4 Some

.

/'(x', p)

=

pXl

distributions

univariate

1 p) -

1 -

fo r x

'N '

=

0, 1

otherwise. and the probability

ln practice p is unknown

takes the form

model

(4.34) Such a probability model might be appropriate in modelling the sex of a newborn baby, boy or girl, or whether the next president of the USA will be a Democrat or a Republican.

(ii) Tlle binomial distributiov The binomial distribution is unquestionably the most important discrete distribution. lt represents a direct extension of the Bernoulli distribution in the sense that the random experiment Js is repeated n times and we define in rl trials. If we the random variable F to be the number of al1 21 trials the density of probability is in that the the same assume of Y takes the form tsuccesses-

isuccess'

otherwise. and

denote this by writing J'

we

'v

#(n, p):

&

' 'w

Sdistributed

reads

as'.

Note that

n .'

n! -(n y) ! .p!

/(!

,

k

'

(/

1)

'

(/( 2)

.

.

.

2 1. .

The relationship between the Bernoulli and binomial distributions is of considerable interest. lf the Bernoulli r.v. at the th trial is denoted by Xi, i + X,,; + ,Ya + 1, 2, n, then Y is the summation of the Arfs,i.e. L This is used emphasise subscript because Y' dependence in is its the to on n. which implies that the Arfstake the value 1for a success' and 0 for a ))- j Xi represents the number of successes' in n trials. The interest in this relationship arises because the 'is generate a sequence of increasing c-fields cu c(L); c(Ff) represents the c-field generated of the fonn c(Fj) cz c('L)c by the r.f. Y).This is the property, known as martinqale condition (seeSection 8.4),that underlies a remarkable theorem known as the De Moivre-luaplace =

.

.

.

=

,

tfailure',

'

'

'

'

-1

'

'

64

Random variables and probability distributions

cvntral Iimit rtrtpl-?rrl.De Moivre and Laplace, back in the eighteenth n) for a century, realised that in order to calculate the probabilities large n the formula given above was rather impractical. ln their attempt to find an easier way to calculate such probabilities they derived a very important approximation to the formula by showing that, for large n, ./'()';

where j. ,g

-u-

=

g

.jy

&

x'7.t,7/.,( -

-- . - .

,

1 J?)q

5

rx

reads approximately

equal.

on the RHS of the equality was much easier. This the density function of the most celebrated of a1l has a bell-shaped symmetric curve, the nortnal. Fig. 4.9 of a binomial density for a variety of values for 11 and p. As we can see, as n increases the density function becomes more and more bell-shape like- especially when the value of # is around 1y. This result gave rise to one of the most important and elegant chapters in probability theory, the so-called limit theorems to be considered in Chapter 9.

Using the formula formula represents distributions which represents the graph

(2)

Continuous distributions (i)

-'b

e ntp/vzlt//

tlist'lnibut

k't?/'?

The normal distribtltion is by far the most important distribution in both probability theory and statistical inference. As seen above, De Moivre and Laplace regarded the distribution only as a convenient approximation to the binomial distribution. By the beginning of the nineteenth century, however, the work of Laplace, Legendre and Gauss on the theory of placed the normal distribution at the centre of probability theory. lt was found to be the most appropriate distribution for modelling a large number situations in astronomy, physics and eugkmics. Moreoverof experimental Markov, Lyapounov and the work of the Russian School (ChebysheN, 'errors'

limit theorems, relating to the behaviour of certain standardised sums of random variables, ensured a central role for the Kolmogorov)

on

normal distribution.

A random variable

zYis normally

distributed if its probability

function is given by j'q

x y .

''

.

5

!

(y

2

j

=

1 ex j) cx (.27r) -

-s .-. ...

1 -

.

2c

j

2 (x - g j .

y

density

4.4

Some univariate distributions

0.8

0.8

0.7

0. 7

65

0,6

0.6 0.6

n R

< 04

5 0'0S

=

=

0.5

n V

< 04

0.3

0.3

0.2

0,2

0.1

0.1 0 12 3 4 5 6

0.5

0 1 2 34 56

X

X

0.6

0.6

0.5

0.5 0.4

n #'

< 03

S

s

=

=

1()

=

0.05

=

0.4 < 03

0.2

0.2

0. 1

0. 1

lo 0. 5

0 1 2 3 4 5 6 7 8 9 10

X

X

0.6

0.5

0. 5

0.4

=

=

0 12 3 4 56 7 0.6

n /7

< 03

n P

ac

=

0.05

=

0.4 < 03

0.2

0.2

0. 1

0. 1 0 12 34 5 6 7 X

=

z

a

P

=

.

.

pc 0'5

.

.

l

I

j

jI

I

-

-

.

0 1 2 3 4 5 6 7 8 9 10 12 14 1 1 13 15

Fig. 4.9. The density function of a binomially distributed variable for different values of the parameters n and p.

random

this by Ar N(p, c2). The parameters p and c2 will be studied in more detail when we consider mathematical expectation. At this stage we will treat them as the parameters determining the location and flatness of the density. For a fixed c2 the normal density for three different values of p is given in Fig. 4. 10. 'v

and probability distributions

Random variables

0

-4.0

Jz

g

=

=

Jl

4.0

=

0.40 0.30

:i o,2o


=

-

=

=

=

0.5987.

(ii) Expon

(.w tial

.?'l'll'/-J'

t?/' (listributions

Some of the most important distributions in probability theory, including the Bernoulli, binomial and normal distributions, belong to the exponential family of distributions. The exponential family is of considerable interest in statistical inference because several results in estimation and testing (see Chapters 11- l4) depend crucially on the assumption that the underlying probability model is defined in terms of density functions belonging to this family; scc Barndorff-Nielscn ( 1978). 4.5

characteristics

Numerical

of random

Yariables

ln modelling real phenomena using probability models of the form *= p), 0 g (.)). we need to be able to postulate such models having only a f general quantitative description of the random variable in question at our Such information comes in the form of certain numerical disposal a characteristics of random variables such as the mean, the variance. the skewness and kurtosis coefficients and higher moments. lndeed, sometimes such numerical characteristics actually determine the type of probability density in *. Moreover, the analysis of density functions is usually undertaken in terms of these numerical characteristics. ./'(x;

'priori.

(1)

Mathematial

expectation

(r.v.) on (S, P )) with F(x) and .J(x)its distribution function (DF) and (probability) density function (pdf) vafiable

Let Ar be a random

respectively. The F(.Y)

mean .ylxl

=

n%

'

of A- denoted by f)A-) is defined by dx - for a continuous

r.v.

Characteristics

4.5

69

of random vayiables

and F(A-)

=

j

xf./'t-Yf

i

)

-

fOr

a discrete r.v.,

when the integral and sum exist. is over all possible values of X. The integral in the definition of mathematical expectation for a continuous random variable can be interpreted as an improper Riemann integral. lf a unifying approach to both discrete and continuous r.v.'s is required the integral (see Clarke ( 1975)) concept of an improper Riemann-stieltjes

Note that the summation

(4.47) used. We sacrifice a certain generality by not going directly to the Lebesque integral which is tailor-made for probability theory. This is done, however, to moderate the mathematical difficulty of the book. The mean can be interpreted as the centre q/' gravitv of the unit mass as distributed by the density function. lf we denote the mass located at a from the origin by m(xf) then the centre of gravity is distance x, i 1, 2,

can be

=

located

.

.

.

at -vfnyt-vj)

j

Zf-

.-

(4.48)

.

?Fl(.Y)

1. If we identify nt-vsl with ptxf) then f)A-)= jf xptxf), given that f Jx'f)= provides a measure of location (orcentral In this sense the mean of the r.v. tendency) for the density function of X. '

If A- is a Bernoulli distributed

r.v. (X

'v

b 1, p)) then

0

A'

/'(x) (1 - p) lf X

'v

distributed

/?), i.e. Ar is a uniformly

U((I,

'(A-)

=

,x?

h

.Vtxldx

=

j x

-

b-

(1

dx

1 =

-.

then

r.v.,

b

j

x2

---

2 b

-

a

=

a

a +. /)

2

.

70

and probability distributions

Random uriables

lf X

Np, c2), i.e. X is a normally distributed r.v., then

'v

F(Ar)

t

=

-

(2,c) +. p ) e .-ya (ja (2zr)

.x;

p

'

--

-

1

cc

2

1 x u 2 c -

exo

c

(Jz

x

=

1

x

e

for

e-izz

(jz

J

X

-

=

(r

:c

a

.

dx,

(2z:)

z

-

+

.

-.jc2 (j Z

(27:)

= 0 + p 1 p, since the first term is an odd function, i.e. h - x)= - /1(x).Thus, the P arameter p for X Np, c2) represents its mean. .

=

'w

ln the above examples the mean of the nv. A- existed. The condition which guarantees the existence of '(Ar) is that X

dx < Ixl.(x)

cc.

-

< w. )(gIxfl/txf)

or

vo

(4.49)

i

One example where the mean does not exist is the case of a Cauchy distributed r.v. with a pdf given by flxs

=

1 zr(1 + x 2 )

,

R.

.X 6

In order to show this let us consider the above condition:

x, -X

dx Ixl/txl

1

1

r

lxl1 + x

=-

z: - x

a

=- olimzc 2 zr -+

Ctl

1 --

zr

x,

x

2

j

o

oxc

dx

by synlnxetry

1

=

dx c

o

1 x dx=- lim logell : a . 1+ x a

+J2)

-+

.

That is, '(Ar) does not exist for the Cauchy distribution.

Some properties of the expectation (E1) (.E2)

c, (' c is a constant. ftzArj + bxz) tkEtxjl + lxEl-fzl for Jn-p '(c)

=

=

lwt? r.,.'s

ArI and

p

of random

4.5 Cllaracteristics

variables

Xz whose means exist and a, b are real constants. For example, (J Xi b 1, p), i 1, 2, n, i.e. Xi represents the Bernoulli r.r. of the ith trial, then for F;, S(Fl, P) f (Y;,) 1 E (Xf) np. 1 Xfl 1 p That is, the mean of a binomiallv distributed r.r. equals the number of trials multiplied b)' the probability' oj' and Properties E1 E2 desne F(.) as a linear transformation. Prx > ;.E(X)) % 1/2 jr a positive r.t,. X and ;.> 0,' this is ('3) the so-called Markov inmuality. Although the mean of a r.v. X, .E(Ar) is by far the most widely used measure of location two other measures are sometimes useful. The first is themode defined to be the value of X for which the density function achieves its maximum. The second is the median, xm of X defined to be the value of X =

'w

.

.

.

,

=

(Z)1-

::::>'

'v'

=

Z)'=

=

Z7=

=

bsuccess'.

such that

It is obvious that if the density function of A- is sq?mmetric then '(.Y) x..

(4.51)

=

If it is both symmetric and unimodal

mean

=

median

(i.e.it has

only one mode) then

=mode,

assuming that the mean exists. On the other hand, if the pdf is not unimodal this result is not necessarily valid, as Fig. 4. 13 exemplifies. ln contrast to the mean the median always exists, in particular for the Cauchy distribution x,n =0.

I I

l l

I

I

I

1

I

I I I

I I I 1

1

I j 1 1 I I I

'G

mode

Fig. 4.13. A symmetric

mode.

mean

median

mode

r

x

density whose mean and median differ from the

(2)

and probability distributions

variables

Random

The varlance

When a measure of location for a nv. is available, it is often required to get widely the values of .,Y are spread around the location an idea as to how is, measure, that a measure of dispersion tor spread). Related to the mean as variance and a measure of location is the dispersion measure called the defined by '

Vart#l

SIIA- f'tA-ljz

=

-

(x - f(-))2/(x)

=

F(.Y))2/'(xf)

= il (xf-

(4.52)

dx - continuous

(4.53)

- discrete.

t#' inertia of the mass can be interpreted as the moment through the mean. axis distribution with respect to the perpendicular

The variance

is referred to as

Note: the square root of the variance

deviation.

standard

Exkrnp/t?s (i)

Let A' ??(1,p); it x,

was shown above that F(Xl -p2)(

VarlA-l

=

(0

Sf

Var(A-)

X-

=

p, thus

=

-p)2p=p(

1

a+

2

1

+ (1

-p)

b

-p).

-f.'/)2

2

1

(? dx =

b

-a

-

-

12

(verify).

An equality which turns out to be convenient for deriving Var(-Y) is given by E'(A-2) (E'(.Y)12, Vart-l =

-

where v2./'(x)dx.

for z

=

x

p

-

-..-

c

Characteristics of random

4.5

Yariables

.fr

( PrI)

fkny c'tpnsltknr ('. Vartt?l 0 2 (1 constant. Varll X) a VartXl, #-((xY E(xY)l> k) :$ gVar(X)q,/k2 Chebyshev's inmuality the fr /( > 0. This ntx?-fllflr lives (1 relation and JC#nt?l probabilitv q.f by dispersion as variance te such tA- E(A-)1p: k, pl-onpng in //t'?cr bound an upper for =

lS2)

.jr

( ( Iz'3)

=

-

-lt?lwtlt?n

probabilities. (3)

Higher moments

Continuing

the analogy with the various concepts from mechanics we define the moments of inertia from x 0 to be the so-called rlw' moments: =

the l-th raw moment, if it exists, with p'a 1 and yj usually denoted by Jt. Similarly, the rth moment around central nlt?rnt?nl, is defined (ifit exists) by pvEEE/I..Y -

plr

(-x-- p)r./-(x)dx,

=

r

=

2, 3,

F(A-); the mean is x p, called the rth

H

=

=

.

.

.

.

c2. These higher ;tz a E'(- - p)2 is the variance, usually denoted by moments are sometimes useful in providing us with further information relating to the distribution and density functions of r.v.'s. In particular, the in the form: 3rd and 4th central moments, when standardised X3

=

14

=

:3

--

G

3

and lt *

-:

(T

are referred to as measures of skpwncss and kljrtosis and provide us with measures of asymmetry and flatness of peak, respectively. Deriving the raw moments first is usually easier and then the central moments can be derived via (see Kendall and Stuart ( 1969:: r

pr- j=1 (

l .

-

1/

i

p'ipr-j. ,

(4.58)

An important tool in the derivation of the raw moments is the characteristic

74

and probability distributions

Random variables

function defined by J

Eteilx)

kx-

eirxdz-txl,

=

Using the power series form of

/x(l)

=

v''- 1.

(4.59)

eA

we can express it in the form

., (jrlr

p'r. F. --r!

1+

=

i

X

-

r

(4.60)

1

=

This implies that we can derive y'r via t

Fr

=

dr/

x (jjr

(r) =

A function related to Chapter 10) is

loge/xtr)

()

/xtrl of

1+

=

(4.6j)

1

t

interest in asymptotic theory

particular

x) (ir)r&r r 1 r!

)2

(see

(4.62)

=

where

sr, r= 1, 2,

.

.

.

are called the cumulants.

Example Let A- Np, c2), the characteristic

function takes the form

'w

-r2c2),

/

1d/xlll 'i- dl

(M=expirp

X

1

= o 1-

,-

1 d24 (r) 7 dr ,

c

-.ir

exptirp

X

()

-

a = p +

Gz =

pa

=

c

J,

)(ip

(r a

-

rc c)

p,

=

()

,-

.

Similarly we can show that ps pzb 3/4, ps /z6 15c6, etc. c2, x,=0, Kendall and Stuart rb 3, aa 0, a,yc::z3', see ( 1969). &72 =0,

=0,

=

=

l1

=p,

=

=

the various numerical characteristics of r.v.'s as related under certain to their distribution, it is natural to ask whether 1, 2, Jz'r, knowing the moments we can determine the circumstances r DF F(x). The answer is that F(x) is uniquely determined by its moments /t;, if and only if r= 1, 2, Having considered

=

.

.

.

.

.

.

X

l (/t'c,)-r -

r

=

(4.63)

vs.

1

This is known as Carleman's

condition.

4.5

Important Random

of random variables

Characteristics concepts

the probability

variable,

by a r.v., a c-field

space induced

generated by a r.v., an increasing sequence of c-fields, the minimal Borel field generated by half-closed intervals ( r xl x e R, distribution function,density function,discrete and continuous r.v.'s, probability model, parametlic family of densities, unknown parameters, normal distribution, expectation and variance of a r.v., skewness and kurtosis, higher raw and central moments, characteristic function, cumulants. -

,

Questions Since we can build the whole of probability theory on (S, % P ))why do we need to introduce the concept of a random variable? Define the concept of a r.v. and explain the role of the Borel field generated by the half-closed intervals ( vz, x(l in deciding whether a function .Xt ): S R is a r.v. 'Although any function Art ); S Ii can be defined to be a nv. relative c-field valuable information if we do not stand lose to we to some define the nv. with care'. Discuss. Explain the relationship between #( ), #xt ) and Fx( ). Discuss the relationship between Fx( ) and .J( ) for both discrete and continllous r.v.'s. What properties do density functions satisfy? Explain the idea of a probability model * ).J(.x; 0j, 0 (E 6)) and discuss its relationship with the idea of a random experiment $ as well as the real phenomenon to be modelled. Give an example of a real phenomenon for which each of the following distributions might be appropriate: Bernoulli; (i) binomial', (ii) normal. (iii) '

-

-+

.

'

--+

,.'h

'

'

'

'

'

=

Explain your choice. Explain why we need the concepts of mean, variance and higher moments in the context of modelling real phenomena using the probability model * 0), 0 6 O). B'hat features of the density function do the following numerical characteristics purport to measure? =

ttx;

mean, median, mode, variance, skewness, kurtosis. ixplain the difference between these and the concepts with the same names in the context of the descriptive study of data. iixplain Markov's and Chebyshev's inequalities.

76

Random variables and probability

distributions

12. Compare the properties of the mean with those of the variance. do the moments characterise the 13. Under what circumstances distribution function of a r.v.? Exercises b, c, J) and 84/) #(h) Consider a random experiment with S (tp, #(C) P(Is l'. Derive the c-eld of the power set (i) l/J, ?) say ..W). Derive the minimal c-field generated by (ii) S: Consider the following function defined as

=t,

=

=

=-),

=

,?/.'

,

-tfzl

-Y(c) -Y(J)

0,

.(?)

=

=

J'(/))

=

=

F((.')

=

1,

=

7 405 926,t

J'(J)

2.

=

Show that .Y and F are both nv.'s relative to but F is not. is a r.v. relative to Show that Find the minimal c-field generated by F, .?>:

(iii) (iv) (v) (vi) (vii)

,t7j

-

,#(.

1) k.p g1, 2:. Find #y(.t0) ), #y((0. 1q), #y,((- cfs 11), Py(I)0, Derive the distribution and density functions F(y), ,

.(y)

and

plot them. Calculate E(F), Var(1'), az(F) and a4(J'l. (viii) The distlibution function o the exponential distl-ibution is F(x)

=

1

-

exp

-

x

'

,

and plot its graph. Derive the density function Derive the characteristic function /(1) 'Derive /J(Ar), VartxYl, aa(,Y) and a4(Ar). .(x)

(i) (ii) (iii)

'(ei'x).

=

Note:

'# If the reader is wondering about the significance of thfs number it fs the number of demons inhabiting the earth as calculated by German physician Weirus in the sixteenth centtlry (see Jaslrow (1962:.

Characteristics

4.5

of random

variables

lndicate which of the following functions functions and explain your answer: (i) /'(x) kx2, 0 < x < 2,'

represent

proper

density

=

2(1 -x)2, x > 1,' 341 x y 1,e .J(x) < x < 2,' 1), + 0 .(x3 (iv) /'(x) 3, J(.Y) iE R. x (v) -lx Prove that Vart-Yl E(X2) - gE(xY)q2. lf for the nv. X, E(-Y) 2, F(xY2) 4, find F(3X + 4), Var(4X). Let Ar N(0. 1). Use the tables to calculate the probabilities F(2.5); (i) F(0. 15),(ii) 1 - F(2.0). (iii) and compare them with the boundsfrom Chebyshev's inequality. What is the percentage of error in the three cases? ,f(x)

(ii)

=

(iii)

=

''E).

-

=

=

=

=

=

x

Additional references Bickel and Doksum ( 1977)., Chung ( 1974)., Cramer ( 1946)., Dudewicz ( 1976),. Giri t 1974)) Mood, Graybill and Boes (1974); Pfeiffer ( 1978); Rohatgi ( 1976),

C H AP T E R 5

Random vectors and their distributions

The probability model formulated in the previous chapter was in the form of a parametric family of densities associated with a random variable (r.v.) 0), 0 s O). ln practice, however, there are many observable X: * phenomena where the outcome comes in the form of several quantitative attributes. For example, data on personal income might be related to number of children, social class, type of occupation, age class, etc. ln order to be able to model such real phenomena we need to extend the above framework for a single r.v. to one for multidimensional r.v.'s or random vectors, that is, =

t/tx',

X

=

(A'1 Xz,

.

X',)'.

,

.

.

where each Xf, i 1, 2, n measures a particular quantifiable attribute of experiment's random (J) outcomes. the For expositional purposes we shall restrict attention to the twodimensional (bivariate)case, which is quite adequate for a proper understanding of the concepts involved, giving only scanty references to the n-dimensional random vector case (just for notational purposes). ln the next section we consider the concept of a random vector and its joint distribution and density functions in direct analogy to the random variable case. ln Sections 5.2 and 5.3 we consider two very important forms of the joint density function, the marginal and conditional densities respectively. These forms of the joint density function will play a very important role in Part 1V. =

.

.

.

,

Joint distribution and density functions

Consider the random experiment 78

of tossing a fair coin twice. The sample

Joint distribution and density functions

5.1

space takes the form S t(ST), CTH), CHH), (TT)). Define the function Both and -Y2( ) to be the number of .lt ) to be the number of of these functions map S into the real line (!4in the form =

Stails'.

'heads'

'

'

(.X'1( ),X2( ))1)(FS)l '

'

(-''i( ),.X'2( ))1)(.fT)l '

'

=

(1,1), i.e. (.X'1(Tf1), X2(TS))= (1,1), (1,1),

),.Yc( ))((f.f.f)l (2,0),

(A-1(

=

'

'

(.Y1(

=

'

),

-'2( -

))l(TT)l

=

(.0,2).

R2 is a twoThis is shown in Fig. 5. 1. The function (xY:( ), A-2(. )): S dimensional vector function which assigns to each element s of S, the pair of ordered numbers (x1,x2) where xj xYj(s), xc Xc(s). As in the onedimensional case, for the vector function to define a random vector it has to satisfy certain conditions which ensure that the probabilistic and event structure of (S, P( )) is preserved. ln direct analogy with the single s ariable case we say that the mapping -+

'

=

=

..%

.

X(

'

)H (X1( ), X2( )): S '

-+

222

vector if for each event in the Borel #a), the event defined by

definesa random .22.say B H (:j, X- 1(B)

belongsto

'

=

(s: xYI(s)

e #1, -Y2(.s)e #c, s e

tield product

.@

x

,)

.@

M

(5.2)

..%

S Xa

(88) (HP

a

( F8) (rr)

1

@

0

1

Fig. 5.1. A bivariate random vector.

2

xl

80

Random vectors and their distributions

Extending

can be profitably seen as being the c-field generated by half-closed intervals of the fonn ( :yz to the case of the direct product x we can show that the random vector X( ) satisfying the result that

,#

,xj

-

.@

,?d

.

1

X-

((

w xq)

-

,

g

.F for a1l x

g

/2

implies X - 1(B) (E .F for all B G 41. This allows us to define a random vector Dhnition

1

A random

Vtor

Ar(

'

)..is

a vector

as follows:

.jnction

-

cc


w.

(0 0)

A

'--

x

z

w

R

N >.

-

X1

normal density.

Fig. 5.4. The density function of a standard

lt is interesting to note that the expression inside the square brackets expressed in the form of X1

#1

-

c

2

--

1

X1

- 2p

#1 c1 -

p2

X2 -

+

(r 2

X2

- p2 o'z

2 m

ca

defines a sequence of ellipses of points with equal probability which can be x2) represented in Fig. 5.4. viewed as map-like contours of the graph of .f(xl,

0 (k =

(3)

,

a l a a) ,

.

Bikariate binomial dltrl'bution

/'(x 1 x c) . ,

!

/1 ... - ! J?1'P)2, a -.j 1 .v 2

=

-Y1

+ xc

=

1

,

p 1 + I.'z

=

1 .

(5.20)

The extension of the concept of a random variable A- to that of a random (.YI .Y,,) enables us to generalise the probability model X1, vector X =

,

.

.

.

,

85

5.3 Marginal distributions *

.f

=

/ (x ; 0) 0 6 O ) '

t.

,

family of

to that of a parametric *

,/'(

l

=

x

!

xa

.

,

.

.

.

,

joint density functions

x,,; 0) 0 6 (9 ,

(5.22)

)

.

generalisation since in most applied disciplines, the real phenomena to be modelled are usually including econometrics, multidimensional in the sense that there is more than one quantifiable feature to be considered.

This is a very important

5.3

Marginal distributions

Let X EB (xY1 Xal be a bivariate random vector defined on (,. ...Fi#( ))with a joint distribution function F(x1, xa). The question which naturally arises is whether we could separate A-l and Xz and consider them as individual random variables. The answer to this question leads us to the concept of a marginal J.srrf/pfklfon. The marginal distribution funetions of A-1and Xz are defined by '

,

F1 (x1)

=

and F c(x 2 )

lim F(x1 xc) ,

X

=

z

-+

J-

l im F(x : x 2) ,

. 1

''*

.

ik

Having separated .,Y1and Xz we need to see whether they can be considered defining a random as single r.v.'s defined on the same probability space. In that condition the vector we imposed

$ts : A-1 (s) < x l

-

,

2(#

:%

x2

)G

(5.25)

.:''

The definition of the marginal distribution function we used the event s : A-1 (s)

:$

2.( x 1 A- s) < vs ,

lj

(5 6)

.2

,

t which we know belongs to .K This event, however, can be written as the intersection of two sets of the form

u hich implies that .ts:

.,1

(-$)% xl X 2(s) ,


fC1 >0,

=

C-

X2 '

f?xzl - *1 exp

1 + $x 1 )(1 + ()4

2)t2 ;.lk + 1)(r.,1 1 (1

xc >az

distribution

,

>0.

-1-

l $4/./

2x 1

+ a 1x a -

k- x ( 1 + 0x a) l 21

(1 1 (1

1

c )-

.

(2

+

'

density functions There are two things to note in relation to conditional above examples'. brought out by the the conditional density is a proper density function- i.e. (a)

both properties can be verified in the above examples. lf we vary Xz. that is, allow Xz to take all its values in Rxz, we get different conditional density for each Xz .x2 of the form

a

=

/x' vct-xl /x2),

xlgR.y,

,

(5.54)

xzeRxa

reflection suggests that knowledge of al1 these xa), a densities is equivalent to knowledge of conditional equality brought general relationship out by the A moment's

./'(xj

,

(x1 x2) -

e' R2.

5.4 Conditional distributions model tp .t.J'(x x,:; 0), 0 e (.)) because it offers us a general way to I xc, decompose the joint density function. It can be seen as a generalisation of holding when A-I and Xz are xa) the equality (x1) independent, considered in the previous section; (55)-(56)being valid for any joint density function. lndeed, we can use the condition which makes the two equalities coincide as an alternative definition of independence, i.e. .;t-: and Xz are independent if =

,

.

.

,

.

'./'2(xa),

=./)

./'(xj

,

/)

x24-Y1

,'''xz)

(.Y1

=l

(5.57)

),

This definition of independence can be viewed as saying that the information relating to X2 is irrelevant in attributing probabilities to XI. Looking back at the way (x1) was derived from the bivariate normal density we can see that the expression inside the integrl in (37)was the conditional density of X; given Arj xl. It can be verified directly that in this case =

/'(xl . .

.x2)

-

=

-), ( x

=

1

exp

j azrr.):

-j

x,(.x'2, ( /'1(x1))(-/x' c

The marginal and conditional

1 vi -

'

.:1)).

distributions

2

t'/2 cj

2) -

( 1 - /? cax,

.

(2z:) --

(5.59) in this case are denoted by

(5.60) (5.6 1) (5.62)

C a Se :

(5.64)

Random vectors

and their distributions

A sequence of r.v.'s -Yl, Xc, any x G R, Fj(x)

=

F2(x)

=

'

'

'

.

=

.

.

,

Ar,,is said to be identically distributed if,for .65)

(f

F,,(x).

The concept of conditioninq enables us to simplify the probability model in the form of a parametric family of multivariate density function (1) x,,; p), 0 i5 e) in two related ways: t/'txl xa, (i) to decompose the joint density function into a product of conditional densities which can make the manipulation much easier; and information in some in the case where the stochastic (probabilistic) of the r.v.'s is not of interest we can use the conditional density with respect to some observed values for these r.v.'s. For instance in the case of the income-age example if we were to consider the question of poverty in relation to age we would concentrate on a:(X2/'X1 exclusively. 1) 2 . =

,

.

.

.

,

=

.

lmportant concepts Px( )); random vector, the induced probability space (Rn, the joint distribution and density functions', marginal distribution and density functions, marginal normal density; independent r.v.'s; identically distributed r.v.'s; conditional distribution and density functions, conditional normal density. .@'',

'

Questions Why do we need to extend the concept of a random variable to that of a random vector defined on (S, P( ))? Explain how we can extend the definition of a r.v. to that of a random vector stressing the role of half-closed intervals ( vz xq. Explain the relationships between #( ), #xl ) and Fx x2( ). ! Compare the concepts of marginalisation and conditloning. Define the concepts of marginal and conditional density functions and discuss their properties. Define the concept of independence among r.v.'s via both marginal as well as conditional density functions. Define the concept of identically distributed r.v.'s. .%

'

-

,

'

'

.

'

,

6.

Exercises

Consider the random experiment of tossing a fair coin three times.

5.4 Conditional distributions Derive the sample space S. Define the random variables A-1- the number of Xz lnumber of number of Derive the c-fields generated by c(-Y1), Xz, c(A-a) and cl-Yl, A-2). Define the joint distribution and density functions F(x1, xc), x2). /'(xj x2) and plot Derive the marginal density function /'j(x1)and ,J2(x2). Derive the conditional densities theads',

=

'tails'l.

eheads'

-

(iii)

-4-1

,

,/'(x1

,

,

/'l(x1.7l),

3) .J'1(x1

2),

./(x2

./'clxc

0),

,

For the income-age example discussed above l), Plot the graph of .J(x: xc), .J'1(.x (i) ://2), c,/'3) (ii) Derive /'ltx/ 1), .J:(x fzqx and compare their graphs with that of Jttx ). be bivariate normal 3. Let the joint distribution of Arj and ,/'24x2).

,

.:-2

X1 Xz

p1 (i) (ii) (iii)

=

t7'il

p1

'w N

PG 1 (r 2 -

,

cl

pclcc

/:2

1

,

Derive Derive

(x1) and L (x2). jx' xa(.Y1/'.X2), fOr x2 j

=

0, 1, 2.

Under what circumstances are X3 and X1 independent'? Let the joint density function of A'j and X1 be /'(x 1 x2) ,

=

Derive Derive f. 1, 2, 10.

x l expt - xj( 1 + xzlj'

and

./-(x1)

v-tx,

'x

,)

fctxz). for x

,

=

,

x 1 > 0,

1, 2, 10, and

x2

>

0.

for (xc,/'x1) /'xz.a.j

x:

=

XG4itional derences Bickel and Doksum (1977); Chung (1974)',Clarke (1975);Cramer (.1946/ Dudewicz 1974); Pfeiffer ( 19781.,Rohatgi (1976), (1976); Giri ( 1974); Mood, Graybill and Boes (

6.1

of one random

Functions

variable

CHAPT ER 6

Functions

of random

variables Fig. 6. 1. A Borel function of a random

One of the most important problems in probability theory and statistical A',,) when inference is to derive the distribution of a function I1lZYl A-a, This known. randoln X,, ) is vecto r X (A-l the distribtltion of the problem is important for at least two reasons'. it is often the case that in modelling observable phenomena we are (i) primarily interested in functions of random variables; and inference the quantities ot- primary interest are in statistical colnmonly functions of random variables. It is no exaggeration to say that the whole of statistical inference is based on of various l'unctions of r.v.'s. l n the fi rst ou r abil ity to deri ve the distribution consider the distribution of functions of a single subsection we art! going to and then consider the case of functions of random vecto rs. r.v. .

.

=

6.1

Functions of one random

.

.

.

.

.

.

,

,

considered a function from S to R and the above ensures that the composite function (X): S R is indeed a random variable, i.e. the set /l(Ar)(s)e:.B,)s,.i/-for any Bkt?d (seeFig. 6. 1). Let us denote the r.v. hX) by Ft then F induces a probability set function #yt ) such that PyBh)= #x(#x) JN.,4),in order to preserve the probability structure of the original (S, P )).Note that the reason we need ( ) to be a Borel function is to preserve the event structure of Having ensured that the function ( ) of the r.v. A' is itself a r.v. F hzrj we want to derive the distribution of F when the distribution of X is known. Let us consider the discrete case first. When A' is a discrete r.v. the F hzr) is again a discrete r.v. and a1l we need to do is to give the set of values of F and the corresponding probabilities. Consider the coin-tossing example where X is the r.v. defined by A' (number of H - number of F), then since .4

-.+

=

'

ts: =

'%

.

'

,..d.

'

=

=

=

variable

S

P Let A' be a r.v. on the probability space (,, Suppose -.+ function S. real valued R, i.e. -Yis a on S function with at most a 11 is a continuous (liscontinuities. More formally we need /1( ) to be ..#7t

'

)). By definition, A-( ): '

'

that hl

.

countable

): R

=

)HT TH, HH, FF), XIHT)

XITH)

=0,

X(SS)

2, A'IFFI -2 and =

=

the probability

function is A- x - 2 0 2 .,t J

Rs where number of --+

a B()reI function.

=

=

t

'

22 4 with the same &'&'' I.et F Xl, then F takes values ( 2)2 4, (0)2 probabilities as X but since 4 occurs twice we add the probabilities, i.e. =

Dpnit 1011l

variable.

=0,

=

F

y

=

#r(F

=

y)

0

4

1.

-t2

2

'

ll1 gencral. the distribution 1-3( )')

!1t- I't. t I1t- i1)

=

'$

'

P(

,s

t. 1' st.

:

l

1'-(s) lI

.1

t'


0 and x < 0 separately to get (d.') =

which implies that Edtxlq/tdx) 1/c> and F Let X N(p, 1(y)q/(dy) 1(y) > o.y +y and Ed 0 for a1l x c R since tr 0 by definition; /lc. Thus since tr2)

'w

Fig. 6.2. Thc function F= ..Y2where X is normally

to the

refer

l

-jy)

expt

+

for y>o

((a) expt .-jyjjy-.i

-y)

That is, fytl,'lis the so-called gamma density, where F( is the gamma fnction (F(n) jJ' t??1e-r d$. A gamma r.v.1 denoted by F.v G(r, p) has a density of the form .J(y) g.p/F(r)q(p#)'- expt p)1, > 0. The above 1.2) and is known distribution; an distribution is G(1y, as the chi-square important distribution in statistical inference; see Appendix 6.1. .)

=

,y

=

-

=

'v

Fy(y)

=

#r@(x)

= #r(

-

:%

y)

=

#r(x

6F

h

-

1(

-

),

xj) -

Functions of several random variables

As in the case of a single r.v. for a Borel function h ): R'1 R and a random xY,,), (X) is a random variable. Let us consider vector X (.11,Xz, used functions of random variables concentrating on the certain commonly variables case for convenience of exposition. two '

=

Fx( Vy) Uy% A' %Qyl Fx(Uy) =

6.2*

-

.

.

.

,

-

Fig. 6.3. The function

(l )

F

=

.Y1 + Xz.

The distribution of X 1 + Ar2

By definition

the distribution

function of F

=

..Yj +

.:-.,.,

(sey Fig. 6.3) is Example 4 U( - 1, 1), 1 1, 2 (uniformlydistributed). t 7sing Fig. 6.3 we can show that I

.et

'j

'v

=

and define F

=

A-j + X1.

=

ln particular,

if .,Y1and .Y2 are independent,

Iy'(..v) =

by X1

symmetry. -

A' 2

./'1

(.p -

x.z)./2'(x2)dx2

Using an analogous

=

argument

then ,1

(xj )./a* (-'

-vl -

) dx 1

,

we can show that for F

=

(2hl -#/1

:6:

zc

1L .

This density function is shown below (seeFig. 6.5) and ascan

be seen it is not

1():!

6.2*

of random variables

Functions

Functions

of several random variables

Fig. 6.6. The function F= Xb Xz for F Fig. 6.4. The density function of F uniformly distributed.

X3 + Xz where

=

and

A-I

l03

.::0

and F

>0.

A'c are

1$k t-s t he lbrl'rl ./'txl

F y (.p)

fv (y)

=

,

0

x2) dx1 dxc

- ccl

0.5 .;''r'

) :

i -1

-p

1

o

Fig. 6.5. The density function of F= uniformly distributed.

.11

+ Xz + X5 where

2

. ).

B

Xi, i

=

1, 2, 3, X

are

.

only continuous but also differentiable everywhere. The shape of the curve is very much like the normal density. This is a general result which states uniformly distributed independent r.v.'s, that for Xi U( 1, 1), i 1, 2, which is closer to a normal distribution the L Z)'-l Xi has a distribution particular of value case of the central limit theorem (see greater the n; a Chapter 9). 'v

=

-

.

.

.

=

II1 tilt. t'tstt

:.

The distrihution

(.

A' 1 and Xz are independent

.

'

this becomes

.

I., tl'l;I' t j

(.1,.v2) '.fL(x2)dxc, I.vzI./'l

#

.

.

'q

of A-1 A'2

where

:

.E.

(2)

xa) dxc, jxajytyxa,

y,j ,y,.,(

1 1- t .1! t

.$ (t l)e

(6.8)

htllnatical manipulations

lllat

are not ilnportantl)

2

.N(t ). l ) :1 11 tl A c y 4tl ) chi-sq u a re wi t h 11 deg rees of freedorn, .' 1 t. 1l t j? l I ) k lt- I I 1t It- l l I 'lt-t'il t- 1 lt ) lt n d let u s d e ri ve its j . x ( j t l I 1 l l 1 1 l t I 1 I l l t ( Ik l 1 s l ! l't I I ) t' l i( ) I ( .3#' l I1t- (.It. ! 1( I 11il : t t (. 1' l ) is i t't r'l 4 N ( .

,

x.

-

I

i

,

l

'

.1

'')t-

''

.'

.'

'1

..!

.'

(

'(

', 1'1sitl

t- 1'

t u-(

-s ',

l'.

'$ .

.'

I

:1 1,1(.1

.'

?

(.1

Ct1'1

It'!t

'i

'

'

'

.

'$

.1

'k

'1

'k

.....'''1

.j?.

--

..:

'$'

Functions of random

l() l

Ix. (Z) . Since

.J'(x1,::)

=

n

variables

n/ 2 Zn

ag(u,a)jjj-jus)

,

=/'1(x:)

,

-

*

nz

eXP -

2

a

,

it takes values only for z > 0, which implies that

'.gz),

1

' .x

=

.-

--

Jr G j G 2

=

This is the density of Sfut/pnr's

g

j)

()

u-) z V

S( G 2

'

Y

2

'y.

:-

- 2-

2

y +.

()' 1

j-

V1

)

($ .- u

d ly

z2(nj)and

'v

()

l-t/fsll-fhuffon.

F

'

/y( . Aj

=

xz2(na)

Xz

be two independent

=

.-.

.

xz/nz)

'c

.

,?,

n -!.

=

The distribution

v2

.

n1

/ 1 -na ,

-

n2

exp F

/'y (

.r.)

n 1 + nz 2

--

j-

n j

1

2

n2

..2

.

y,

y))

nl

,

dxa nj /2

-n

),'c(,,j

-

ajyaj

2 j + 1.4,1

m,

-

j'n

h (x g ) dx .c .

.v

tj1( la 1+

xa

:=

and define

(X1 /n1) nz Xl

. --

0

.

r.v.'S

1+

u 1 '

112

y?

)/21

,, ...?

,

d xa

y

W Uere bl =

,

Example 6 Let Arl

1

.

G2

x

V 1 CT2 .--

xp

'

of F

=

mint A' 1 A-c) ,

'

v %

--

2 2

2

!

.

2

j

u! ..y -V ---j G1

G2

106

Functions

of random

6.2*

uriables

of several random variables

Functions

'rhese assumptions

enable

to deduce that

us

(6. 12) Example 9

l

.ct

Xi

N(0, 1), i

'v

1'1 =

1, 2 be two independent

=

ll1(.. 1 X2) ,

X1 + X2,

=

nv.'s and

1'2 /12(X1. X2)

A' 1

=

=

-

.

Ar2

sillce X

l'' 1 0 1( 1 -

.172)

-

,

'1.F2

j+

'

.y,

2

.'j!

Fig. 6.7. The function F

=

min tXj

,

i

X2).

J= det

consider them together. Let (#1, X2, joint probability density function transformation'. .(x1,

.

Ar,,) be a random vector with a x,,) and define the one-to-one x2, .

.

.

.

1 1+ )'c

t lllskilnplies that

,

.

+ ),c

I.'''l

,

/'(y1 .

1

-p2)

,

=

2,:

exp

1

-j

jy 3....1 (yy-).4)

(y 1 .J.'2 ) + y f (1 + yc) c 2

...

,

1 ( 1 -I 1 J.c)2 .J,f -j exp + ( j .y y,a) ( 1 27: .rc)

(6. 10) l l1t. Illltin

drawback of this approach

.

is well demonstrated by the above

'N.$l1:I'le. The method provides us with a way to derive the joint density 1(,11 of the Ffs and not the marginal density functions. These can be t 1$l,1k 1It.I 15 t'tl by integrating out the other variables. For instance,

j .. E

.I1l

F

111

t 11cltbove

example

these take the form

Assume:

(i) (ii) (iii)

hi( ) and gi ) are continuous; the partial derivatives Jxf/pyf, i,j 1, 2, continuous', and the Jacobian of the inverse transformation '

'

=

.

.

.

,

bn exist and

are

(. atlchy density. 'f! .

,:@

108

Functions

6.3

Functions of normally distributed random

of random variables

Looking ahead

6.4 variables,

109

a summary

The above examples on functions of random variables show clearly that A',,) when deriving the distribution of hz-, x,,) is known is not an easy exercise. Indeed this is one of the most difficult problems in probability theory as argued below. Some of the above results, although involved (as far as mathematical manipulations are concerned), have been included because they play a very important role in statist'al /ld/-pnct?. Because of their importance generalisations of these results will be summarised below for reference purposes. ,(x1,

,

.

.

,

.

.

.

.

,

Lgnlrntt 6 l .

1/' Xi

(Yt'1

I

Nqyi c/).

x.

i

,

z'b'ij

Letnlna 6

1 2,

=

,

.

N(Yt,'-pi, Y'f'-

-

1

1

.

.

11 tgr'(?

,

c/)

-

normal.

independen r

?-.t?.*.!f

then

.2

//' X,. .N(0, 1),i 1 2, z2(rl) chi-square with =

x.

indepvndent n Jtx' rees q/' jeedom. -.l?.5y

,

.

.

.

n

,

g/-g

tbe

()-)'.1 .Y,2.)

x

6.2* f--pnklrlt?

1/' Xi

()-'1 i 1

N(/tf c/), i x g2(j7.

'v

z

=

1 2,

=

,

2/.c2) i i

,

1'-'l'i/i fl

=

j

..

.

.

,

j

'G 2 i

Fig. 6.8. The normal

independent r'.!7.'-$ then chi-square wr non-centralit p

n are

) ..-non-central

'

,

p(.l /-(1111(.4l Q9l* ,

.

I-cmma 6.5

N(0, 1), ..Y1 x. 2(n) arv A' : .,Yz j?7(/t?/?(.?nJ?n t ?-.t).'s then z with ?? f Student's 'A j l)x,'' X Jt.yf://-t.afz?-s )1 t 2, n ( (rl).q.Jjeedoln. -

'

distributions.

,

Ll,nltntl 6.-3

1.1%X 1

and related

x.

,

../

.

.

x,

Lt.zrrlnkt/6

.-3

N(p, c2), Xz 1/' X 1(?; J) .,:-1,,/'(x'/(X2,/:)1 t.??J p/'c. ptlratnet j

among the distributions referred to in these lemmas are in Fig. 6.8. For a summary of these distributions see Appendix 6. 1 1,4.11 lt'r tt more extensive discussion see the excellent book by Johnson II,lt $ K t ' t z ( I970). I llt.

'v

'v

'v

I'cl:ttionships

lt'ki

L'1'It

*

,y$

c2g2(n),

.

.Yj Xz indepgndgnt ,-.j?.'y thgn non-central t with ntr?n-trt.?nl?-./// )' ,

=

Lemma 6.4 ?-.!,:.*s A' 1 ztln1 )s X1 z2(nc), A-j Xz independent thetl (xYl n j )/(A-2/n2) F(n l na) - Fisher's F with n j and na degrees t?/'

lf

'v

'v

,

'v

,

jeedom.

Lenma 6.4*

.k. .t *4 '

I

II,

lllx

.*, II1

? q 1) .(.)

I :

,

,1

,1

'

'lk's

y (.t

ahead

.tltpking

w'e considered

of functions of random AIt Iltltlgh thtl are in general rather l 1t is kt ve 1'),' iInp() rtant facet of probability theo ry for two reasons: I ( t 'l't t.I1 k'cctl I's i1) prltct icl?t hat the probability model is not defined I 11 t t. l I 11 s $ ) l. l llt! ( ) l'i g i11:1l I'. N' s btl t i 11 sorne fu nctions of these. %l ; t 1I s l I k': 1 I I I 1 t't-I't- I 1k't.- isk c I'tlt. i t lIy tlttpentlell t o 11 t he d ist ribution of l 1 1 l 1 t I l , s . l l ; 1! 1k 1( ', I 1) y : t !' 1kl l7 It- s I s ! iI1 1 t t ( I's t 11(.l t t.ls t s t l.t t is t ic s .trtt .

k'llitlltel'

the distribution

manipulations

l'nathematical

.

.

1dtI

k

( '

(

.,

Functions

of random

Appendix 6.1

variables f(B)

A-,,)and the distribution functions of r.v.'s of the form h(X3, Xz, of such functions is the basis of any inference related to the unknown parameters 0. From the above discussion it is obvious that determining the distribution Ar,,)is by no means a trivial exercise. lt turns out that more of /1(.-1, X1, often than not we cannot determine the distribution exactly. Because of the importance of the problem, however, we are forced to develop approximations', the subject matter of Chapter 10. lt is no exaggeration to say that most of the results derived in the context of the various statistical models in econometrics, discussed in Part IV, in Section 6.3 above. depend crucially on the results summarised Estimation, testing and prediction in the context of these models is based on the results related to functions of normally distributed random variables and the normal, Student's t, Fisher's F and chi-square distributions are used extensively in Part IV. .

.

.

.

.

.

.

f (/; 1)

,

Appendix 6.1 The normal Univariate

normal

.- Ar

#lxl

1

=

.

.. .

Gx/

l

;.- .....-

/-(x) --,V-,2 exp c ( z) ' x F(.Y)

=

.

=

y,

exp

(27:)

-

Var(.Y)

=

1 u- p - -2 (5'

1 xy - 2 .--..c skvwnvss

s)

p'

Chi-square distribution - F

'v

z2(n)

=

=

2

d r/,'

Reproductit'e

prtpptrrly

2 ,

=

as

=

0,

kurtosis

=

a4

=

distribution

chi-square

3

()

7. ,

,

n)

=

1

---.;.,-

Vn

Higher moments:

x

r t?7 el

2-

cc

Zch

k

-

tN''2)

exp

z2(n,.(i)

- 15 'v

y!fH.''

.j 2 ), + J)1. t - -.-41

f

.

2) - 1

)J)(,?'/'

is

Some prtppt?rres

F( F)

=

n+

(),)kuk +. j.)

(2k)! 1- k +

,

)

.

n

,

>0

Cumulants

chi-square.

.E'(.p) n (thedegrees q/' freedomb,Vart)l) 2/. I ilt! density function is illustrated for several values of n in Fig. 6.9.

Non-central c2,

,

Fig. 6.9. The density functions of a central and non-central

distributions

and related

/. (z ; 6

0

N(p, c2)

'v

.Y

(y; 6 )

Var(F)

=

>

n

=

12 ,

,

.

.

.

2(n + 2).

E:

;

'E

'

.) ''m. :

is that the rtant difference with the centra j c jaj-square ftl llct itln is shifted to the right and the variance increases.

Ilctlt.t-. l Ile iIllpo

deltsl l

11t

-;'''(

?t/

.

.

11(

+1

l 1'(:

/)?'f

)'

Functions

of random

variables f tx)

z (w; 4 ) Nxw

z

-..V'''''- f (x)

z-

N

tzvj

exp

x2 -

nz nz -2

.E'(LJ)

=

nz > 2,

,

r

=

I l1ccentral and non-central

>

2

1

3

x

t's /-J-ifr?-//7l.flk'(?l71.1z' r(N)

F-distribution

/'( &; n 1

'v

-

F-distribution

l ig. 6. 11 for purposes of comparison. Non-central

e-

,

n 2,'

y .. 1+

k

=

)

0

.. U

2k) - 1

N1

nz

j-'

f-distribution

F(n1 rl2,' J), ,

>

0

u

+2:) n 1 -i''t -..n.2

n 1 + n z + 2/ -.... 2

yn, +nz

n 1 4. These moments show that for large n the

'v

density functions are shown in

=

ku J(n,+

Gt'

+2k)

F

nz

n2

2

F

...... Fl 1 +

gk

... . ,

2

>2s

is very close to the

normal (see Fig. 6. 10).

t (t.?)

Non-central

t-distrib. u/ ion - Hz' r(n; J), J

'

w; n

>

x

N .,

'''

ir

/'(

(jj ,

=

:;2!

.--z

(.ti

C .....(

iEl

-.-::?2

.-

0

)1

- .-y wzliV' Y /n r(n/2)(n + --

j

f (tz; n , n z )

) , a-j

,

'

)'.

);'.

tit

x

y'gs(.l-+k -1--ljtjt k=0

2

2w2

b-y .

jk/l

n + u(r,

,

ws

f (fz; n j n a ; ) ,

u

.

'. )il)1/., '''' ::.

1201-ft,lp-f/n g

() :'

'

kr( 1.12)

---

'i;

,

-

N

0

St l-ff/cri

+rl2 2#lJ(n1 Vart (p) n1(n2 2) c (n2 -4) -2)

2

N

A

A

1

=

&

u s ()

,

Functions

of random variables

Important

Appendix 6.1

tl'tlaccr/.:

Additional

Borel functions, distribution of a Borel function of a r.v., normal and related distributions, Student's t, chi-square, Fisher's F and Cauchy distributions.

'larke

( ( l t?78);

Questions Why should

be interested

we

in Borel functions

of r.v.'s and their

distributions? :A Borel function is nothing more than a r.v. relative to the Borel field on the real line.' Discuss. Explain intuitively why a Borel function of a r.v. is a r.v. itself. Explain the relationships between the normal, chi-square, Student's !, Fisher's F and Cauchy distributions. What is the difference between central and non-central chi-square and F-distributions'? .td

Exercises Let zYkbe a nv. with density function

f

A-1

0

(.X

*2

1

)

Derive the density functions (i)

X X A'

(ii) (iii)

of

2*

A' 17

=

e'Yi

=

;

l0 + 2%-21 Let the density function of the r.v. X be /'(x) e distribution of F logc X. Let the joint density function of X3 and Xz be =

.

=

-X,

x > 0. Find the

=

),t

Derive the distribution of F X21 + X2'25 (i) F m in (A-j .Ya ) (ii) ( .'(() 1) kl t.-I-i t..'yt I1t-tl is t 1'i13t I t i( 1. t

j *)

=

=

-

.

.'

.e

.x.

'$.'

.

.

',

1) (

.'

1'1'

references

(1975)., Cramer ( 1946),, Giri (1974),. Mood, Graybill Rao ( 1973),' Rohatgi

( 1976).

and Boes

( 1974)*,Pfeiffer

The general notion

of expectation

expectation

In Chapter 4 we considered the notion of mathematical context of the simple probability model

m

)

(7.1)

0j, 0 e:O

./'(x;

=

single random as a useful characteristic of density functions of a model probability generalised the to Since then we

*

f

=

(

/ (x 1

,

x 2s

variable.

J'

'

..

in the

.

.

,

.

x , , ; 0 ) 0 iE (). ,

and put forward a framework in the context of whichjoint density functions luarginalisation, conditioning and functions can be analysed. This included section is variables of this The to consider the of random (r.v.'s). purpose general framework. of this more notion of expectation in the context Expectation

7.1

of a function of random

A'ariables

ln the one-dimensional case of a single r.v. we considered many numerical characteristics of density functions, namely. E'(-Y), S(,Yr), .E'I-Y A(A-))2, which contain summary information concerning '(.Y - JJ(x))r, r= 2, 3. the nature of the distribution of zY.lt is important to note that each of these characteristics is the expectation of some function of X, that is, F(/;(xY)); /)4zY) X, /?(') Xr, etc. This provides us with the natural way to proceed in the present section, having diseussed the idea of a Borel functioll ( ): For (r which preserves the event structure of the Borel field R'' where n 2. simplicity of exposition we consider the case their joint Let (.Y1 z'c) be a bivariate random vector with density function and let h ): 22 -+ R be a Borel function. Define F (A'1 Xcl and consider its expectation. This can be dened in two -

.

.

.

,

=

=

'

;..d.

--+

=

.vc)

.(.x1

,

,

'

,

=

7.1

of a function of random

Expectation

variables

equivalent ways:

:X.

J

(ii) E(ll('

:,

A-z))

/1(.x1 x2)./'(x1

=

,

,

dxj dxa.

xz)

and it is usually The choice between (i) and (iij is a matter of convenience of of Y. diffkulty deriving by the degree in the distribution determined

Let Xi Using

'v

N(0, 1), i

,

X( + X(.

=

(ii), E(X 21 + X 2) 2

1 /',(J.') 2,)-.( . =

j

----

-

(x( +

=(v2j

the other hand, Y freedom, that is,

x

2aj

-

2/

exp

)

-

jx ( + x 2aj) dx ( dx c

z2(2.j- chi-square

+ -Y2a) '.w

exp

t

cc

r

=

on

and

r.v.'s and /1(Xl X a)

1. 2 be two independent

=

=

l

.

with two degrees of

t 1-v) -

we know that

E(Y)

)') -.'Jy',(

=

-v

2

=

equals the number of degrees of freedom (see Appendix 6.1). Before we consider particular forms of pltAr(. .Ya) let us consider some properties of E(ll(A-: .Xa)). ,

Properties

of expectation fkE(lll(A-l Xa)) + ?E(/12(X1, A'2)); avd :( ), b a( ) are Borel are 122 rtp R. .1n particular

EIJ/k1(X1. X2) + bhzx

Iinearity, u/ntl'l ons

u'lltpl-t? a .//4)m

1,

,

'

'

11

aiz-i 1* .=

=

t'onsrlnr.s

/1

E x

Ar2)1

/nt

1

-

i

Z aikjz-i). =

(7.5)

1

I/' X 1 and X a are independent

-.p.'s,

.ll-

tvyt?ry

Bovel jnction h

(')

The general notion of expectation

and Jl2( ),'R

R,

-.+

'

f(/'l1(A-1)2(A'c))

Eht

=

(.,1))

/'l2(..Y2)),

'

(7.6)

given that the above expectations exist. This is botb a nessary as well as suflkient condition for independence. particular of interest is when /lj(xY1) XL and /lc(aY2) Xz, One case =

=

.E'(A-1X2) F(.X1) &.zY2). =

This is in some sense Iinear independence independence. Moreover, given that CovtA'l

,

(7.7)

'

which is much weaker than

.Y2) F(Ar:A-c) - f)-Y1) .E't-tcl

(7.8)

'

=

(see below), linear independence is equivalent to uncorrelatedness since it implies that Covt-rj A'a) 0. A special case of linear independence of particular interest in what follows is when =

,

=0,

S(.Y1A'2)

and we say that A-l and Xz are orthogonal, writing X3 -1-Xz. A case between F(.Y1), that independence and uncorrelatedness is that in which E(k3/Xz) is, the conditional and unconditional expectations of .11 coincide. The analogous case for orthogonality is =

E(XL,/X1)

when .E'(A-:) 0.

=0,

(7.10)

=

This will prove to be a useful property in the context of limit theorems in Chapter 9, where the condition EzY',/xn

.',,

.Y'l) 0 - c, plays an important role. For reasons that will become apparent sequel we call this property martingale orthogonality'. 1,

-

Forms of hX

j

.

,

.

.

=

,

A-cl ofparticular

in the

interest

lllA'l -Y2) A-tA-22, 1,k > 0, ,

=

then

!

p;k =

'(.YtA%)

=

X)

xtxt-fxlxz)

dxl dx2

are calledytpfnl mw moments of order I + k; this is a direct generalisation the concept for one random variable. For (-Y1,.-2)=(*-1

-&A-1)/(-Y2 -'(A-2))k,

of

of a function of random variables

Expectation

7.1

E'(-Y1)

EEEE

/t1

are called joint central moments of order I + k. Two especially interesting joint central moments are the variance and covariance; () Covariance.. l k =

Covtxl,

1

=

(7.14)

-/-t2)).

-Y2)- .E'((A'1-p1)(A-c

provides a measure of the linear lrlwt?t?nfwt? random variables. With a direct multiplication above formula becomes :

relationship

The covariance

Covt-l

Ar2)

E(A-1) F(A-a). '(Ar1.Y2) independent then using E2 wtl deduce tat =

,

I/' zYl and Xz are

'

=0.

to note that the

tlt?rv important

(7.15) (7.16)

Covlx'j, Ar2) 1t is

the

is not true.

conYerse

(iij Variance l 2, k 0 =

=

-p1)2,

'Varl-Yl) For a linear

E(A'1

EEE

functionjf

var jl aiuri

aixi the tlariance

=)

i

is of the

form

a,? Var(A-i). i

Using the variance wc could dejlne the standardised formof a covariance kntpwnas rt? correlation coefficient and dehned by

Corrt-'l, X2)=

Covtx'l A'c) ,

gvartA'jl

'

VartAralq

.

(7.20)

The general notion of exptation

Properties (C/) (CJ)

(4)7.)

of CorrtA'l

,

-Y2)

Corr(A'1, aY2) 0 for A-j and .Y2 independent r.p.'.s. 1 % Corrt.r 1 A'al % 1. Corrtfj A'2) 1 for a + bX1, a, b being real constants. =

-

,

-:-1

=

=

,

Example 2 Let pcgljczj), (*'x1a) xllyLljafc, -

Covtfj, X1) X

*

(x1

-/.tc)T(x1,xc) dx1 dx2

(-Y1

1

-/z1)(xa

'X

=

cjcc

=,

x

.

-/t1) cl

xz (2r)

(2a)

exp

1

exp

-j

-p2)-.

(1

.%1

1

1

(1

z

2

dx1

xa -/.ta

-

-p

-Jt1 cl

)

o.,

.-ptxlts-/jtjztj (txa

-0.,a,v.

1

Hence, Corrtztj A-a) It should be noted that p in the case of normality does imply independence, as can be easily verified directly that p 0 o.J'(x1 x2) 'X(xz). =p.

=0

,

=/(x1)

=

,

Nlta that correlation measures the linear dependence between r.v.'s but if we consider only the first two moments that is the only dependence we can analyse.

7.2 Conditional

expectation

Conditional expectation

7.2

The concept of conditional expectation plays a very important role both in x,,; 0), 0 (E 0) to time extending the probability model *= ).(x1, x2, and in the specification of variables process) random dependent (stochastic models Part in lV. Iinear of Arj for Xz x2 was ln Section 5.4 the conditional distribution function exists) limit the defined to be (if .

.

.

,

=

Fx: Aatxl

xc)

).

1im #r(.Y1 Gxj/xa - h %Xz %xa +

=

0 0, xl

For a fxL

.fnction

of x2.

is a proper density . fxj xatxl, xc) dx1 1. .rfrlcrft?rl

EER

and

J'''-

wl

=

The conditional expectation of A-j given that Xz takes a particular value .vc (A'a xc) is defined by =

The general notion of expectation

ln general for any Borel function hl ) whose expectation '

Properties

of the conditional

exists

expectation

Let X, Arl and Xz be random variables ()n (s%, P )), tben: x) '(tz1:(A-1)+ a1h(Xz)jX x) + azkhzrzlj J1 Eh(X3),''X X x), tJ1 az constants. (CE2) lf Arj > X1, '(.Y1/-Y x) )v:ELXI/X x). (CE3) E(l1(A-j -Ya) Xz x2) E(l1(X1 xa),,''X2 xa). ICfVI f((A'1) X1 x2) f(ll(-Yt)) if A-1 al)(I are independent. outside tbe square (CF5) J)/l(-Y1)) 'gS((-1) Xz xaq, tbe pxptvltzrr?rl brackets being wfr respect t() X1. The conditional expectation EIXL/XZ x2) is a non-stochastic function of -+ f(.Y1 Rx, The graph i.e. @. ): (x2, EIXL x2)) is called the regression x2, .%

'

=

=

=

=

,

=

,

=

=

=

,

=

-:-2

=

=

=

=

=

'

(7&rPf?.

Example 3

ln the case of the bivariate normal distribution,

The graph of this linear

reqression

fncrftan

is shown in Fig. 7.1.

Conditional expectation

f (xlexa)

M

e'

-

.-

l I

.-

I I

X1 1 1 l

Xz

I

+

>

7z .z.

.f

''-o

Fig. 7. 1. The regression

.

.

;v

2w

7e

eJ

curve of a bivariate normal density.

As in the case of ordinary expectation, moments:

we can define higher conditional

Rflw conditional moments:

(7.28) Central conditional moments: A'EIA-I

Of particular skedasticity'

-EX,/Xz

=x2))'/-Y2

interest is the conditional

Vart-Yl/-'rc

=

x2)

=

'E(Arj Exljjxz

-

-

(7.29)

x21,

variance,

EIXL/XZ

=

sometimes

x2))2/Ara

=

called

x 2.

xc) g.E'(Arj Xz xalq = In the above example of the bivariate normal distribution we can show that VartArl/Arc x2) c21(1 In the case where the conditional variance is free of the conditioning variables it is said to be homoskedastic; otherwise it is called heteroskedastic. -p2).

=

=

=

-

=

The general notion of exptation

Example 4 Let

?6 This

is the distribution distribution

function of Gumbel's

bivariate

E0,1(l .

exponential

since

'(X:/X2

.X2)

=

r! ( 1 + pxa + r+ (1 + pxc)

=

Var(.Y1,.'A-a xc) =

=

l

and -2p2

+pxa)2

(l +p

--

r0)

,f.--.--

( 1 + pxc)

For a bivariate exponential distribution the regression and the conditional variance is heteroskedastic.

curve is non-linear

Example 5 The

joint density of a bivariate Pareto distribution takes the form

'./'(x1xc). ,

-

ltkcx,

ptp-h 1)(4?1t7c)+

+

',x.a

-:71Ja)-''+''

xl The marginal

> fzj >0,

x2 >az

>0.

density function of X1 is 2

(.X

2

)

=

PJt? '-tt'-b 11 , 2x 2

2

Var(.;Y'1/,Y2 xa) =

=

t?l (p+ 1) xj c az 0 0 + 1)

-

ln the case of the Pareto distribution the conditionalvariance is heteroskedastic.

regression curve is linear but the

7.2

Conditional

expectation

Example 6

joint density of a bivariate Iogistic distribution is

The

.Jtxj xa) ,

E(X3,,'Xz

=

x2)

'Xz

=

xa)

Vart-

1

2E1 +

zll -

3

expt

xc)),

- (x1+ 1 - logt 1 + e-'2) - non-linear in x2

=

=

=.1yzrz -

e-rt + e-

1

- homoskedastic

xa) is a non-stochastic function of x2 and for a particular .Q)is interpreted as the average value of .Y1 given Xz k1. The temptation at this stage is to extend this concept by considering the conditional expectation '(-Y1 Xz

As argued above value

ECXL,/XI

.fa,

=

=

=

(7.31) The problem, however, is that we will be very hard pressed to explain its meaning. What does it mean to say average value of A-j given the --+ variable E'? The only meaning we can attach to such a -Y2( ): S random conditional expectation is 'the

.

StA- 1 c(-Ya)),

where c(A'c) represents the c-field generated by Xz, since what we condition on must be an event. Now, however, E'(-Yj c('c)) is no longer a nonstochastic function, being evaluated at the r.v. Xz. lndeed, .E'tArl/ct.Ycll is a random tlariable with respect to the c-field c(A-a) c: .F which satisfies certain properties. The discerning reader would have noticed that we have already used this generalised concept of conditional expectation in stating CE5, '(/l(A'1)/ where we took the expected value of the conditional expectation 'c xc). What we had in mind there was '(llj(.Y1) c(-2)) since otherwise the conditional expectation is a constant. The way we defined '(A-j/c(-Y2)) as a direct extension of E(X !yA-a x2), one would hope that the similarity between the two concepts wlll not end there. lt turns out, not surprisingly, that f'taj /c(A'2)) satisfies certain properties analogous to CEI-CES: =

=

LSCE/) '(J1(A-1) + t72g(A-2)/c(A-)) fk11)ll(-Y!) t7'(xY)) + t72S(#(-Y2) t4aY)). $SCE2) 1./*-Y1y Xz. J).Yl c(-)) > E(A-a c4x)). (SCS-?) SEll(A'1) #(zY2)1 FEg(-Y2).E'((A-l) c(X2))1. (SCE4) &(zY1),/c(xYa)) JJ(&(-Y1))#' zk-k and A-a are independent. '((aYI) g(xYc)f)(zYI), (SC'5) g(-Y2) c(-2)) c(A-c)). This implies that the two conditional expectation concepts are directly related but one is non-stochastic and the other is a random variable. =

'

=

=

.

=

The general notion

of expectation

Example 7

ln the case of the bivariate normal distribution considered above, '(-Yl/c(A'a))

-i-p --1 (A'a

=pl

-p2),

G2

and it is a Xz

random

variable. Note that Vartx''l 'c(A'2))

(homoskedastic).

ln relation to the conditional variance the variance exists, Vart.Yll

=

we can show

c2j( 1

=

-p2)

is free of

that if '(A-f)
n;. That is, would for practice > ln independent expect to be able to find a we rrl. )lj enough' able approximate 'large m so as to be to any asymptotically m-dependent This is equivalent to independent process by an process. small able > assuming that j(z) for z m are so to equate them to zero. as to be An alternative way to express the weakening of the dependence between .Y(l1) and A'(l2) as !lj ra( increases is in tenms of the autocorrelation function which is a measure of linear dependence (see Chapter 7). ,4

,t7-

.

=

=

,

-+

.

.

.

,

.

.

.

-+

,

-(r1)

lcJ

-

-

Dejlnition J

.4 stochastic process (A'(l), r (E T) is said to be asymptotically uncorrelated #' tere exists t; sequence of constants kfp(z), z y 1) dehned by d1, t +z)

(t'(l)r(f+z))''

G P(T),

./r

aIl r 6 V,

sucb tat

0 %p(z)
1 define only upper bounds for the two measures of dependence given that when equality is used in their definition they will depend on (11, Svell 1,,) as as z. r2, general formulation of asymptotic independence can be achieved A more using the concept of a c-field generated by a random vector (seeChapters 4 where (.Y(r), and 7). Let denote the c-field generated by ..Y(1), T) is a stochastic process. A measure of the dependence among the tc elements of the stochastic process can be defined in terms of the events (E and B c Mt by ,

.

.

.

,

,@j

.:-(1)

.

.,4

.

.

,

.@y

,

a(z) suplf'l-gt =

J

c B4 - #(.4)/'4:)1,

(8.25)

Dejlnition 6

,-1stocbastic process (z) 0 as z --.

kf

( c T) .Y(M,

is said to be

(strongly)mixing

1)/-

bz'.

-+

of the asymptotic As we can see, this is a direct generalisation independence concept which is defined in terms of particular events and B related to the definition of thejoint distribution function. In the case where )A'(r), l G T) is an independent process a(z) for z > 1. Another interesting special case defined above of a mixing process is the m-dependent process where a(z) 0 for z > m. In this sense an independent process is a zerodependent process. The usefulness of the concept of an m-dependent process stems from the fact that commonly in practice any asymptotically independent (ormixing) process can be approximated by such a process for ilarge enough' ?'n. A stronger form of mixing, sometimes called unlfrm mixing, can be defined in terms of the following measure of dependence: ,4

=0

=

tp(T)

=

sup

(8,4/,)-#(-4)t,

PB)

>0.

(8.26)

Restricting

#.3

143

memory

Dehnition

,4 stochastic process (Ar(r), t c T) is said to be uniformly mixing :f 0 as

(P(T)

'r

'--.

(f

-+

.

Looking at the two delinitions of mixing we can see that a(z) and @(z)define absolute and relative measures of temporal dependence, respectively. The formeris based on the definition of dependence between two events and B separated by z periods using the absolute measure ,4

E#(z4ro #) 8-4) -

'

#(#)) 10

and the latter the relative measure EPIA/.B)

-

#(X)) > 0.

stochastic stationary processes ln the context of second-order asymptotic uncorrelatedness can be defined more intuitively in terms of the temporal covariance as follows: Xt CovtAXll,

+z))

=

t?(z)

0

-+

A weaker form of such memory restriction is the so-called ergodicity property. Ergodicity can be viewed as a condition which ensures that the p(z) by averaging over memory of the process as measured by time' kweakens

Dehnition 8

-4second-order

stationary

1 r lim -T Zl p(z) zv-vx

process (-Y(l), r e T) is said to be ergodic

=0.

(8.28)

If wecompare (28)with (25)wecan deduce that in the case of a second-order stationary process strong mixing implies ergodicity. The weaker form of temporal dependence in (28),however, is achieved at the expense of a very In modelling we need restrictive form of time-homogeneity (stationarity). and off between them (see there is often a trade both type of restrictions Domowitz and White (1982:. Memory restrictions enable us to model the temporal dependence of a stochastic process using a finite set of parameters in the form of temporal moments or some parametric process (see Section 4). This is necessary in order to enable us to construct operational probability models for modelling dynamic phenomena. The same time-heterogeneity and memory restrictions enable us to derive asymptotic results which are crucial for

Stochastic processes

statistical inference purposes. For example one of the most attractive features of mixing processes is that any Borel function of them is also mixing. This implies that the limit theorems for mixing processes (see Section 9.4) can be used to derive asymptotic results for estimators and test statistics which are functions of the process. The intuition underlying these results is that because of stationarity the restriction on the memory enables us to argue that the observed realisation of the process is typical (in a certain sense) of the underlying stochastic process and thus the time averages of the corresponding probability constitutee reliable estimates expectations.

8.4

Some spial

stochastic processes

The purpose of this section is to consider briefly several special stochastic processes which play an important role in econometric modelling (seePart IV). Thcse stochastic processes will be divided into parametric and nonparametric processes. The non-parametric processes are defined in terms of their joint distribution functions or the first few joint moments. On the other hand, parametric processes are defined in terms of a generating mechanism which is commonly a functional form based on a nonparametric process.

(1)

Non-parametric

processes

The concept of conditional expectation discussed in Chapter 7 provides us with an ideal link between the theory of random variables discussed in Chapters 4-7 and that of stochastic processes, the subject matter of the present chapter. This is because the notion of conditional expectation enables us to formalise the temporal dependence in a stochastic process (.:-(1),t e:T) in terms of the conditional expectation of the process at time t, .:-(1)('the present') given (-(l 1), A'(l -2, (ithe past). One important application of conditional expectation in such a context is in connection with a stochastic process which forms a martingale. .)

-

.

(1)

.

Martingales Dehnition 9 (lfned #( )) and Let )A'(r), t G T) be a stocastic on (S, process Jt oj' oujlelds G T increasing 9$. l e: T) sequence r (f ,, ) an tc the conditions: satisfyinq followinq ,.%

'

.@

,#7t

8.4 Some

special stochastic

Ar(l) is a random

() (ff) (fi)

processes lmriable

zR1x(r)l) (.c. its

(r.t?.)relative rtl

mean is bounde X(l 1), for aII t 6 T. '(-Y(l) 9. t - j) Then (A'(l), t 6 T) is said rtp be a martingale ).f4, t 6 T) and wtl wrflp tAr(l), 4, t e T). < c/s

=

.f/?.

t

for aII r e: T.

for all t c T,. and

-

wirll

respect

to

Several aspects of this definition need commenting on. Firstly, a martingale is a relative concept; a stochastic process relative to an increasing sequence of c-elds. That is, c-fields such that 91 c cLhcu L'it, and each -Y(l) is a r.v. relative to %, t G T. A natural choice for c Vt cu c-fields X(1)), t 6 T. Secondly, the 1), will be gtt c(Ar(l), Xt such value bounded all of xY(l) expected for must be t c T. This, however, implies because Z'(X(l))= stochastic has constant that the mean process jq all c-CE7 of conditional E(.Y(l 1)) 'gF(xY(t))/.@ T for t c by property 1.expectations (see Section 7.2). Thirdly, (iii)implies that .

.

.

.

.

,

.

=

=

-

.

.

.

,

-

E(xY(!+ z) i? t-

1)

=

Xt

-

1) for all l (E T and z y 0.

(8.29)

That is, the best predictor of xY(r+ z), given the information 9't-. 1, is A'(l 1) for any z >0. Intuitively, a martingale can be viewed as a fair game'. Defining .Y(1)to be the money held by a gambler after the rth trial in a casino game (say, black jack) and Vk to be the history' of the game up to time 1, then the because the gambler condition (iii)above suggests that the game is before trial t expects to have the same amount of money at the end of the trial as the amount held before the bet was placed. It will take a very foolish gambler to play a game for which -

%fair'

(iiil'

Eqxltj/.@t

1) .GX(l 1). defines what is called a supermartingale

(8.30)

-

This last condition (tsuper'for the casino?). The importance of martingales stems from the fact that they are general enough to include most forms of stochastic processes of interest in econometric modelling as special cases, and restrictive enough so as to theorems' (seeChapter 9) needed for their statistical allow the various analysis to go through, thus making probability models based on operational. In order to appreciate their generality let martingales consider examples of martingales. extreme two us tlimit

ilargely'

Example 3 Let tZ(r), l c T) be a sequence of independent r.v.'s such that F(Z(r))

=

0 for

146

Stochastic processes

al1 t e: T. If we define .:-(1) by f

z(k),

-Y41) =

k=1

then )X(r),

(8.3 1)

h, r e T) is a

martingale, with 9.t c(Z(M, Z(l - 1), c(A'(r), X(t 1), xY(1:. This is because conditions (i) and automatically satisfied and we can verify that =

.

-

.

.

.

,

.

.

,

z( 1))

=

(ii) are (8.32)

using the properties c-CE9 and 10 in Section 7.2.

Example 4

t 6 T) be an arbitrary' stocbastic process whose tz(!), F(lz(r)j) < for all t e:T. If we define ..Y(1)by that

only restriction is

Let

gz

l

A'(r)

=

E

EZ(k)

-

1)1,

Elzkl/.i

(8.33)

-

k=1

.Y(1:, then c(Z(k), Zk 1), Z( 1)) c(A-(k), .Y(k 1), tA'(r), ?,, t G T) is a martingale. Note that condition (iii)can be verified using the property c-CE8 (seeSection 7.2).

where 9k

=

=

-

,

.

.

,

-

.

.

.

,

The above two extreme examples illustrate the flexibility of martingales very well. As we can see, the main difference between them is that in example 3, aY(r)was defined as a linear function of independent r.v.'s and in example 4 as a linear function of dependent nv.'s centred at their conditional means,

i.e.

F(r)

.E'(Z(l)/Mt

.:-(/)

==

.-.

-

(8.34)

1),

lt can be easily verified that )F(l), r e: T defines martingale dfference process relative to tt because '(F(l),

j

In the case where F(F(l)F(k))

:)

-

=0,

for all

'(.E'(F(r)F(/())

= E gF(k)'(F(f)/f/, That is, (F(l), t e: T) is an orthogonal

rG T

q -

is known as a

(8.35)

f G T.

< x for all .E'(IZ(f)I2) =

what

we can

deduce that for r > k

yj

:)j

=0

sequence as well

(8.36) (seeChapter 7).

special stochastic

8.4 Some

147

processes

Dehnition 10

.4 stochastic process tF(l), t (E 7-) is said to be a martingale diffrence c c process relative to the increasing sequence of c-.l/l.s .f4

j.j Q.y (uu F41) is a nn, relative to (f) '(lF(r)!) < Jz'; and (if) f)F(r) . , - 1) 0, t c T. (fff) .

(uu

.

.

.

.

.

%;

=

Dehnition 11

-4 stochastic process )F41), t (EFT' ) is said to be an innovation process c(-(l), Xt 1), #- it is a martinqale dierence wfr/l respect to .f4

=

.

.

. (i) (f)

,

xY(0),where '(1

)

-:-(1)

=

1'(r)I
1 the process is neither asymptotically independent stationary. nor For the stochastic process (xY(!), t e: T), as generated by (45),to be stationary we need to change the index set T to a double infinite set T* (0, That is, assume that the process X(t) stretches back to the u!r1, + 2, infinite remote past (a convenient fictionl). The stochastic process (.,:71), t 6 T*) with GM (45)can be expressed in the form: =

.).

.

.

f(AX0)) 0, =

X

'(A-(l).Y(r+ z))

=

)

E i

=

X

liult

-

i)

0

j Vult + z j 0 =

X

= Hence, for

c2

< 1 the stochastic 1(z1 and asymptotically

stationary

i

j0

-jj

C&

aiaf +1

=

a2

c2a1 i

=

=

0

5

(8.54)

process )A'(r), r c T*) is both second-order independent with autocovariance function

2

?-?(z) =

-

(1

-

z

-

)

(8.55)

a',

az, and the process is not even On the other hand, for k: 1, (:/7.0 a2) f(tAr(r)t2) order since second is not bounded.

)(

-+

stocllastic processes

Some spial

8.4

The main purpose of the above discussion has been to bring out the importance of seemingly innocuous assumptions such as the initial conditions (Ar(0) 0, Ar(- F)-+O as T-+ :yz ), the choice of the index set (T or T*) and the parameter restrictions ()al 1), as well as the role of these assumptions in determining the properties of the stochastic process. As seen above, the choice of the index set and the initial conditon play a very important role in determining the stationarity of the process. In particular, for the stochastic process as defined by the GM (45)to be stationary it is necessary to use T* as the index set. This, however, although theoretically convenient, is an unrealistic presupposition which we will prefer to avoid if possible. Moreover, the initial condition .Y(0) 0 or .Y( T) 0 as T :yz is not as innocuous as it seems at first sight. -Y(0) 0 determines the mean of the process in no uncertain terms by attributing to the origin a very special status. The condition A'( T) 0 as T :x- ensures that A'Ir)can be < 1. expressed in the form (52)even without the condition modelling IV) it is interesting For the purposes of econometric (seePart 1, 2, in relation to to consider the case where the index set is T )0, above, in this case kfxYtfl, stationarity and asymptotic independence. As seen restriction #40) 0, under second-order T) stationary even the is not l iE asymptotically < 1. Under the same conditions, however, the process is independent. The non-stationarity stems from the fact the autocorrelation function t?(f, t +z) depends on t because for z 0, =

-..

--+

=

-

=

-+

-+

-

Ial .)

=

.

.

=

Ial

/1

+ ,t

1')

1

J2aT =

.

- tz

j

lt

c

y

-

(8.56)

,

This dependence on 1, however, decreases as t increases. This led Priestley (198 1) to introduce the concept of asymptotic stationarity which enables us to approximate the autocorrelation function by

t,tr,t +z)

::x:

c2(g

-#

1

(8.57)

,

1

At this stage, it is interesting to consider the question of whether, instead it by of postulating (45)as a generating mechanism, we can actually imposing certain restrictions on the structure of an arbitrary stochastic stochastic' process )A'fr), t c T) is: (i)normal, process. lf we assume that the (ii) Markov and (iii)stationary then lderive'

f)Ar(l)/f/

t- j

)

=

fJ(A'(r)/c(X(l

= aA'tr

-

1/)

-

(8.58) (8.59)

1),

.-40)). The first equality stems from c(A'(r - 1), Arlf - 2), where 1 the Markov property, the linearity of the conditional mean from the .%

=

-

.

.

.

,

Stochastic processes

normality and the time invariance of a from stationality. In order to ensure < 1 we need to assume that the process is also (iv)asymptoticall), that IaI independent. Defining the process u(r) by u(r) A-(r) Elxtlj.,Lit't =

-

1

),

N(0)

-Y40),

=

(8.60)

we can construct the GM A'(r)

yX(t - 1)+ u(r),

=

(8.61)

where lk(1) is now a martingale difference orthogonal process relative to This is because by construction Elutl/t 0 and for t > k, j - ) 1q) 0 c-CE8 of 7.2. by Moreover, Section the process .E'(FElg(r)lt(/()/f4 t(u(l), ! G 5-) ean be viewed as a wllilp-noisc process or an innosation process relative to the information set ? because c,#t.

'(u(r)l(k))

=

=

=

,

E'(u(r))

'(.E'(lf(l) f. t

=

'(l,/(f)2)FtF(I#l)2/.(#()

- 1

f(-Y/)

=

0-,

(8.62)

j))

=

=

)

-

-a/.E'(-YJ- trx2(1 -af) 1)

=

c2,

=

(8.63)

say. This way of defining a white-noise process is very illuminating because it brings out the role of the information set relative to which the white-noise process is not predictable. ln this case the process (l(1),t t5 T). as defincd in That is, ?.(r) contains (60) is white-noise (or non-predictable) relative to no systematic' information relative to tt. This, however, does not preclude the possibility of being able to predict u(r) using some other information set .@)= Lh with respect to which l1(f) is a random variable. To summarise the above argument, if :A'(r), t e:T) is assumed to be a normal, Markov, stationary and asymptotically independent process, then the GM (61) follows by The question which naturally arises at this point is that (61)seems identical to (45) what happens to the difficulties related to the time dependence of the autocorrelation function as shown in (50)75 The answer is arises stationarity of given that the difficulty never because the we can derive its autocorrelation function by ...'

,.

tdesign'.

kgiven

'(r)

tl(T)

=

EX(t)Xt

t

= E @.Y(r

-

z))

1) + !(r.))aY(r z) ) -

=

az

-

1),

since '(l/(r).Y(r z)) 0. This implies that -

t(z)

=

(f/0).

=

(8.65)

8.4 Some

spial

stochastic

processes

Moreover,

d0)

f'(A-(r)Ar(l)) Exxt

- 1).:-(8) + /)tI(1)-Y(1)) =ar(1) + c2 a2tj0) + c2.

=

=

=

Hence, G

d0)

2

=

(F

and

-

(1 a)

tz

2

=

'

'

-

(1

-

-aZ)

(8.67)

af.

tdesigned' This shows clearly that in the case where the GM is no need to change to asymptotic stationarity arises as in the case where (45) is postulated as a GM. What is more, the role of the various assumptions becomes much easier to understand when the probabilistic assumptions are made directly in tenns of the stochastic process tAr(r), t c: T) and not the

white-noise process. (5)

mth-order

Autoregressive,

(z1#(-))

The above discussion of the AR(1) process generalises directly to the AR(/# process where mb 1. Dejlnition 1 of order -4stochastic process t-Y(l), t G T) is said to be autoregressiYe lf satisjles the stochastic (v4R(m)) it equation dterence m Ar(r) alArtl =

1) + xzxt

-

- 2) +

'

'

.

+ xmzrt

1.n)

-

+ u(l),

(8.68)

where aj, a2, xmare constant and u(r) is a white-noise process. For the discussion of (68)viewed as a generating mechanism (GM) it is convenient to express it in the 1ag operator notation .

a(f)x(l)

=

.

,

.

(8.69)

/.k48,

-xzLl -am1r) with 1tX(t) Xt k),k> 1. The whereatLl 1 T* of equation 1, for 2, the difference + :I: can be solution (24) (0, writtenas =(

.-ajf-

.

.

.

=

-

.)

=

.71)

=

g(r) + x -

.

.

'(f-)l/(r).

(8.70)

g(1) c12l +c22@ + + c,,,2tmis the so-called general solution which is 2m) of the expressed as a linear combination of the roots (21,22, polynomial '

=

.

.

.

txm

-

ajx'l -

1

-

.

.

. -

aml

=

.

.

,

(8.71)

0

(assumed to be distinct for convenience) with the constants cj, c2,

.

.

.

,

cm

Stocbastic processes

being functions of the initial conditions particular solution is the second component

.:-40),

-Y(1), xtn1 1). The of (70)and takes the form: -

.

.

.

,

(8.72) where 7 t -F- 1 7 () (7

0

=

:

7,,,+

a j ),,,,

+

7,. vz

7,,,+z

l

+

j

-

+

1

-

.

'

+

'

'

'

myfl

+

'

(8 3) .7

.

0

=

0

m7,=

z

0, 1, 2,

=

.

.

.

ln the simple case m 1 considered above a(fa) (1 - xL), qt) The restriction (21 a), j= a/,.j 0, 1, 2, IaI< 1 in the AR( 1) case is now extended to al1 the roots of (71), i.e. =

=

=

=

I2f I


and the solution of the difference equation (68) can be Fritten as X

.Y4/.) =

-.j). )- o?.#l j 0 =

This form can be used to detennine the first two moments stochastic proccss )A'(r), l (E T*). In particular &.Y(r))

=

0,

'(Ar(r)A'(l+ z))

=

E j

=

l ),yl(l

-jj

0

pfutl + i

=

of the

)

': -

0

X'

= Z(, j

'jb-vz

'2.

-

tUz'

This is bounded when )?y?) < ts a condition which holds when the () of polynomial 1) within the the unit circle (seeMiller (1968/.ln a roots (7 lie conditions the sense (74)ensure that the stochastic process tA-(r), r (E T*) as generated by (68) is both second-order stationary and asymptotically independent because the condition (.l- (, -yJ?) < implies that t?('r) 0 as ,

'w.

T

-->

%;

-+

.

As in the case of an AR( 1) process when the index set T

=

)0,1, 2,

.)

.

.

is

Some spial

8.4

stochastic processes

used instead of T* )0,EE 1, +2, we run into problems with the secondT), EE of which stationarity order t we can partly overcome using the (.:-41), will not be pursued any further asymptotic stationarity. This of concept AR( argued in 1) when the the parametric model is not because, as case, making the necessary assumptions GM but by postulated as a T) tA-(l), of such problems arise. ln particular, if we in directly terms t e: no mth-order normal, Markov, (iii)stationary the process is (i) (ii) assume that independent asymptotically then and (iv) .)

=

.

.

idesigned'

E'(.Y(l) c(.X(l - 1),

.

.

.

,

.Y(0))) F(A-(r) c(Ar(l - 1), =

.

.

.

,

.Xtr

-

n'l)))

= i j 1 afzYtr-).

(8.78)

=

The first equality stems from the mth-order Markov property. The linearity of the conditional mean is due to the normality, the time invariance of the afs is due to the stationarity and asymptotic independence implies that the roots of the polynomial (71) 1ie inside the unit circle. lf we define the X(0)), t e T, and the increasing sequence of c-fields 9L tr(.Y(r), A-t 1), u(r), T) by t c process =

-

.

.

.

,

t

utrl-xtr)- Ex/'u

k-

(8.79)

1)-

fk, l 6 T) is a martingale difference, orthogonal (an we can deduce that )?,f(r), the AR(m) GM as in (68) innovation) process. This enables us to from rst principles with u(1) being a white-noise process relative to the information set %. The autocovariance function can be defined directly, as in the AR(1) case by multiplying (68)with X(t -z) and taking expectations to yield Sdesign'

tdesigned'

F(-Y(f)A'(r -z)) >

=

aj '(Ar(r- 1)Ar(l-z)) +

'

'

'

+ E(l(l)A-(t -z)),

(8.80)

(8.81) satisfy the same difference Hence, we can see that the autocovariances equation as the process itself. Similarly, the autocorrelation function takes the form

(8.82) The system of equations for z 1, 2, m are known as Flg/t?-lzl//k!r which play important role in the estimation of the coefticients equations an The relationship 198 1)). between the a. (see Priestley ( tz1 aa, =

,

.

.

.

,

.

.

.

,

Stochastic processes

autocorrelations and the asymptotic independence of )A'(r), t c T) is shown most clearly by the relationship z

=0,

1, 2,

.

.

.

(8.83)

,

viewed as a general solution of the difference equation (82).Under the 1, 2, < 1, i restrictions m (impliedby asymptotic independence) we can deduce that

I2fI

r(z) (6)

-+

=

.

.

.

,

0

(8.84) (4f-4)processes

Moving average Dejlnition 18

The stochastic process taY(r), t (5 T) is said to be a moving average process of order k (M,4(k)) 4- it can be expressed in the form

(8.85) where :1, bz, bk are constants and t e:T) is a white-noise the white-noise used to build the process process is process. Ttzr is, (aY(?), t c T), beinq a linear combination (#' the Iast k ?,k(r ils. Given that f .Y(f), t c T) is a linear combination of uncorrelated random variables we can deduce that .

.

tlt(r),

,

.

-

(8.86) 0 tKz %;k z> k

(8.87)

o/zGk (8.88) (!?() 1). These results show that, firstly, a MA(k) process is second-order stationary irrespective of the values taken by /?1 bz, bk, and, secondly, after k its autocovariance and autocorrelation functions have a periods. That is, a MA(k) process is both second-order stationary and kcorrelated (r(z) 0, z > k). ln the simple case of a MA(1), (85)takes the form =

,

.

.

.

,

tcut-off'

=

..(1)

=

Ik(1)

+

/71u(t

1),

-

(8.89)

with t7(0) =

r( 1)

=

(1+ hf)c2, 1)1 +hf)

(1

,

p(1)

r(z)

=

=

0,

#1c2,

r(z)

=

0,

(8.90) (8.9 1)

8.4 Some

special stochastic

processes

As we can see, a MA(k) process is severely restrictive in relation to timeheterogeneity and memory. It turns out, however, that any second-order independent stationary, asymptotically process )-Y(r), t e: T) can be as a MA(:ys ), i.e.

Kexpressed'

x(r)

)( bjut

=

-j),

j=

bjb)< (yo and t (E T). is an innovation process. This result where (N.)'o constitutes a form of the celebrated l4b/J decomposition theorem (see Priestley ( 198 1)) which provided the theoretical foundation for MA(k) and ARMNP, q) processes to be considered next. The MA(aa) in (92) can be constructed from tirst principles by restricting the time-heterogeneity and the memory of the process. If we assume that ).Y(r), l G T) is (i)second-order stationary, and (ii)asymptotically uncorrelated, then we can define the innovation process )u(r), r e:T) by .tu(!),

l(r)

A'(r)

=

EX(t)(%-

-

where fh c(A-(r), A'(1 1), c(u(l), u(l to deduce that 7.2) =

1),

-

.

.62,

=

.

.

(8.93)

.:70)). Asymptotic independence enables us u(0)) and thus by c-CE6 (seeStion - 1), ,

.

.

.

,

A-(r) S(aY(r)/f4).

(8.94)

=

tl4l),

Given that t G T) is an innovation process (martingaledifference, orthogonal process), it can be viewed as an orthogonal basis for %. This '(-Y(r) enables us to deduce that can be expressed as a linear combination of the l41 i.e. -4)

-.j)s,

.E'(-Y(1).@f )

=

j

bjut -j4,

(8.95)

j=O

from which (92)follows directly. ln a sense the process :u(1), t G T). provides blocks' for any second-order stationary process. This can be the seen as a direct extension of the result that any element of a linear space can be expressed in terms of an orthogonal basis uniquely, to the case of an infinite dimensional linear space, a Hilbert space (see Kreyszig ( 1978/. The MA(k) process can be viewed as a special case of (92) where the uncorrelatedness assumption of asymptotic is restricted to kcorrelatedness. ln such a case )Ar(r), l G -F) can be expressed as a linear u(? -k). function of the last k orthogonal elements l(l), l41 - 1), ibuilding

,

(7)

Autoregressive

.

.

,

moving avevage processes

As shown above, any second-order

stationary, asymptotically

uncorrelated

Stochastic processes

process can be expressed in a MA(az) form %)

.Y(r)

)

=

bjut

.-jj,

j=

where ()5t () %)< c/s and lg(r), r G T) is an innovation process. Such a representation, however, is of very little value in practice in view of its nonoperational nature. The ARMAIT,q) formulation provides a parsimonious, operational fonn for (96).

t

Dejlnition 19

,4 stochastic process .tvY(l), t e T) is said to be an autoregressive moving average process of order p, q (,4RMz4(p, q)) 4- it can be expressed frl the tlrm' .Y(f) + aI-Y(l

1) +

-

'xvxl.t '

'

+

.

-

p)

=

l/(r)

+

htutr

1) +

-

+ bqult q) -

1p, /71 bz, where a1, ac, white-noise process. .

.

.

,

,

.

.

.

,

bq,are

constants

and

.ju(l),

'

'

'

(8.97)

t g T) is a

In order to motivate the ARMAIP, qj formulation as an operational form of the MA(cc)) representation (96)let us express the latter in terms of the 1 + bj L + bzl? + infinite polynomial )*(f-) =

.

.

.

.:-(1) 47*(fa)u,.

(8.98)

=

Under certain mild regularity conditions b*L) can be approximated ratio of two finite polynomials (seeDhrymes ( 197 1:, b*L)

bq(L) =

=

xp(L)

(1+ lljfz + bzl? + (1 + a jfa + ac1a2+

.

.

.

-

+

bq) ,

.

.

.

+ uplup)

For large enough p and q, b*(L) can be approximated accuracy. Substituting (99)back into (98)we get

xpl-z-

=

k(1a)l/(r),

by the

(8.99) to any degree of

(8.100)

which is an ARMAIP, q) model. This is an operational form which is widely used in time-series modelling to provide a parsimonious approximation to second-order stationary processes. Time-series modelling based on ARMAIP, q) formulation was popularised by Box and Jenkins (1976). The ARMAIP, q) formulation (97)can be viewed as an extension of the part of the ARIrn)representation in so far as the non-homogeneous difference equation includes additional terms. This, however, makes no difference to the mathematical properties of (97)as a stochastic difference

8.4 Some

special stochastic processes

equation. ln particular, the asymptotic independence of the process depends only on the restriction that the roots of the polynomial x (2) (2#+ =

#

1

a12#-

+

'

'

'

+ aP) .

=

0

(8.101)

lie inside the unit circle. No restrictions are needed on the coefficients or the roots of bql. Such restrictions are needed in the case where an AR((x)) lie inside formulation of (97)is required. Assuming that the roots of bpli the unit circle enables us to express the ARMAIP, q) in the form =0

(8.102) This form, however, can be operational only when it can be approximated enough' m. The conditions on a/2) by an AR(m) representation for stability commonly known and those on k(2) as conditions as are invertibility conditions (seeBox and Jenkins ( 1976)). The popularity of ARMAIP, q) formulations in time-series modelling stems partly from the fact that the formulation can be extended to a stochastic processes; the so-called particular type of non-stationary homogeneous non-stationarity This is the case where only the mean is time dependent (the variance and covariance are time invariant) and the time change is local. ln such a case the stochastic process tZ(l), t G T) exhibiting such behaviour can be transformed into a stationary process by differencing, i.e. define 'large

-Y(f)

=

( 1 - f-)dz(l),

(8.103)

where J is some integer. For J=0, A'(l)=z(l); for J first difference and, for J 2,

=

ls Ar(r) Z(r) -Z(r =

-

1);

=

-:-(1) Z(l) - 2Z(l =

-

1)+Zt

-2).

(8.104)

Once the process is transformed into a stationary one the ARMAIP, q) fonnulation is used to model *-(8. ln terms of the original model, however, the formulation is ap(f,)(1 - L)dZ(t)

=

$(f)l(l),

(8.105)

which is called an ARIMA (p, J, /)-, autoregressive. integrated moving average, of order p, J, q (see Box and Jenkins ( 1976)). ln the context of econometric modelling the ARIMA formulation is of limited value because it is commonly preferable to model non-stationarity as part of the statistical model specification rather than transform the data at the outset.

Stochastic processes

8.5

Summarr

The purpose of this chapter has been to extend the concept of a random variable (r.v.) in order to enable us to model dynamic processes. The extension came in the form of a stochastic process (Ar(r), t 6 T) where .:71) is dened on S x T notjust on S as in the case of a r.v.; the index set T provides the time dimension needed. The concept of a stochastic process enables us x,,; 0), 0 6 0) to extend the notion of the probability model *= discussed so far to one with a distinct time dimension

t/xl,

%4,#! 6 Or' t 6 T1. * l'./'(x(l);

.

.

.

,

(8.106)

=

This, however, presents us with an obvious problem. The fact that the unknown parameter vector 0t, indexing the parametric family of densities, their values from depends on t will make our task of (commonly) a single sample realisation impossible. In order to make the theory build upon the concept of a stochastic process manageable we need to impose certain restrictions on the process itself. The notions of asymptotic independence and stationarity are employed with this purpose in mind. Asymptotic independence, by restricting the memory of the stochastic process, enables us to approximate such processes with parametric ones which reduces the number of unknown parameters to a finite set 0. Similarly, stationarity by imposing time-homogeneity on the stochastic process enables us to use timeindependent parameters to model a dynamic process in a is to reduce the equilibrium'. The effect of both sets of restrictions model 106) probability to ( testimating'

tstatistical

*=

(.J(x(l);0), 0 G (1),t e:T).

(8.107)

This form of a probability model is extensively used in Part IV as an important building block of statistical models of particular interest in econometrics. Impovtant concepts

Stochastic process, realisation of a process discrete stochastic processes, distribution of a stochastic process, symmetry and compatibility autoproduct, autocorrelation, restrictions, autocovariance, crossfunctions, normal stochastic process, covariance and cross-correlation vector stochastic process, lth-order process, time-homogeneity and memory of a process, stlict stationarity, second-order stationarity, nonasymptotically independent stationarity, homogeneous non-stationarity, m-dependent and uncorrelated process, strong mixing, processes,

8.5 Summary

163

Markov ergodicity, asymptotic independence, nth-order process, martingale, martingale difference, innovation process, Markov property, Brownian

motion process, white-noise, parametric and non-parametric processes, AR( 1), AR(m), initial conditions, stability and invertibility conditions, MA(m), ARMAIP, (3), ARIMAIP, J, q).

Questions What is the reason for extending the concept of a random valiable to that of a stochastic process? Define the concept of a stochastic process and explain its main

4. 5. 6.

components. 'Ar(.s,r) can be interpreted as a random valiable, a non-stochastic function (realisation) as well as a single numben' Discuss. Wild fluctuations of a realisation of a process have nothing to do with its randomness.' Discuss. How do we specify the structure of a stochastic process? Compare the joint distribution of a set of n normally distributed independent r.v.'s with that of a stochastic process (.:-(0, t E: T) for (r1, ;,,) in terms of the unknown parameters involved. l2, Let (Ar(r), l c T) be a stationary normal process. Define its joint < l,, and explain the effect on the unknown distribution for r < t, parameters involved by assuming (i)m-dependence or (ii)mth-order .

.

.

,

.

8.

9. 10.

14.

.

.

,

Markovness. (xY(r), l e: T) is a normal stationary process then: asymptotic independence and uncorrelatedness', as well as (i) strict and second-order stationarity, coincide.' (ii) Explain. Discuss and compare the notions of an m-dependent and an mthorder Markov process. Explain how restrictions on the time-heterogeneity and memory of a stochastic process can help us construct operational probability models for dynamic phenomena. restriction notions of asymptotic Compare the memory Fnth-order independence, asymptotic uncorrelatedness, Markovness, mixing and ergodicity. Explain the notion of homogeneous non-stationarity and its relation to .4RlMA(p, J, q) formulations. Explain the difference between a parametric AR(1) stochastic process tdesigned' non-parametric AR(1) model. and a Define the notion of a martingale and explain its attractiveness for tlf

modelling dynamic phenomena.

Stochastic

processes

15. Compare and contrast the concepts of a white-noise and an innovation process. The AR( 1) process is a Markov process but not a martingale unless we sacrifice asymptotic independence.' Discuss. 'T'he AR( 1) process defined over T tfo, 1, 2, is not a second-order stationary process even if < l.' Discuss. .)

=

tAny

19.

second-order

.

.

lal

stationary and asymptotically uncorrelated stochastic process can be expressed in MA(cfs) form.' Explain. Explain the role of the initial conditions in the context of an AR(1)

PI-OCCSS.

20.

Explain the role of the stability conditions

in the context of an AR(m)

PI-OCCSS.

22.

b'T'heARMAIP, q) formulation provides a parsimonious representation for second-order stationary stochastic processes.' Explain. Discuss the likely usefulness of ARIMAIP, q) formulations in econometric modelling. Additional references

Anderson (197 1); Chung ( 1974); Doob ( 1953.4;Feller ( 1970),' Fuller ( 1976); Gnedenko ( 1969); Granger and Newbold (1977); Granger and Watson ( 1984),' Hannan ( 1970); Lamperti ( 1977); Nerlove et (11. ( 1979); Rosenblatt ( 1974); Whittle ( 1970); Yaglom ( 1962).

C H A PT E R 9

Limit theorems

9.1

The early Iimit theorems

The term

tlimit

theorems' refers to several theorems in probability theory under the generic names, of large numbers' (LLN) and limit theorem' (CLT). These limit theorems constitute one of the most important and elegant chapters of probability theory and play a crucial role in statistical inference. The origins of these theorems go back to the seventeenth-century result proved by James Bernoulli. tlaw

Scentral

Bernoulli's theorem Let S', be the number q occurrences t#' an cpt?nr in n independent trials t? (1 random experiment rsi and p #4z1) is the probabilitq' 0)' iv each of rf? trials. F!n jr t/ny s > 0 occurrence .,4

=

.f

.4

lim Pr ->

,1

(c

S',

-p

--Fl

< z

=

1,

i.e. tbe Iimit t#' the probabilitq' q/' the event $((S,,,/n) J?)l 0 lim #r(1F;, 11->

cf)

-

FI
1) wr distribution functions n > 1) is said to conYerge in distribution to a r.p. F with distribution

9.1 The

early limit theorems

function F(y)

4J'

1im F,,(y) 11

--#

F(y)

=

C:l

D

at aII points oj' continuity, of F(y); denoted by F;,

--+

FL

lt should be emphasised that neither of the above types of convergence tells us anything about any convergence of the sequence ('F;,) to F in the sense used in mathematical analysis, such as for each s > 0 and s sS, there exists an N N(c, # such that =

IL(s) -

< F(s)I

for n > N.

;

(9.5)

Both convergence types refer only to convergence of probabilities or functions associated with probabilities. On the other hand, the definition of a r.v. has nothing to do with probabilities and the above convergence of F;, to F on S is convergence of real valued functions defined on S. The type of stochastic convergence which comes closer to the above mathematical sure' convergence. convergence is known as 'almost

Dehnition 3 .,4sequence of r.v.'s (Y;,,n > 1) con,erges to F almost surely or with probability one) ,)' lim Y;, F

Pr

=

lt

--#

or, equivalently',

lim Pr

u -/

(x)

=

1,' denoted b)' F;,

243

(f for any

U IYk

-

(f?r.1-,.or -+

F,

a constantj

(9.6)

) 2>0

Fl


0, ; =

=

=

I-S-

-+

,1

1

-.11. - ... N Fl

ljm pr x

(?Jindependent 1 - pi i

=

,

=

1, 2,

.

St?rnt?l/// r.!r.'y witb n, then, for t7/nl' .

.

,

''

jj i =

1

pi

< ;

(9.s)

1

=

.

The important breakthrough in relation to the WLLN was made by Chebyshev who realised that not only LT4 but LT2 was unnecessary for the Ar,,were Bernoulli r.v.'s was result to follow. That is, the fact that Xj, not contributing to the result in any essential way. What was crucially important was the fact that we considered the summation of n r.v.'s to form Sn 3Xi and comparing it with its mean. .

=

.

.

,

7-

(9.9)

Pl't?q/:. Since the A'ls are independent

since lim u

--+

.x

(, ll8

a

=

0, lim Pr ,,

--+

w.

j

. i 11

11 .=

Z =

j

1 Xi - 11

!1

i

Z Jtf > =

1

E

=

0,

9.2

The Iaw of Iarge numbers

1

1imPr

-/1

169

1

j( Xi -- 1 )2pi j

x) and x) ,? > 1, c-constant (i.e.all > f0r x > 0 xis A-). Tben are bounded /?y some r.p. Let

%c#(lA-I

z,-

y;.K Xi - Elxijfh j ), An equivalent way to state the WLLN is 1g

-N i

13

11

LXi =

(9.24)

1

-

E(Xi/%.

1)j

-

-->

0.

This result shows clearly how the assumption

of stationarity

of (A-,,, n y 1)

9.3

The central limit theorem

1), and );(-Y,, n > 1) (seeChapter 8) can strengthen the WLLN result to that of the SLLN. The above discussion suggests that the most important ingredients of the Bernoulli theorem are that: (i) we consider the probabilistic behaviour of centred nv.'s of the form ,f.6-

Zn S', - n;) j)- 1 l-Yj- F(Ar2)); Var(.,,) 0(,1)., and (f F,,. n y 1) is a martingale X,, Y;, for - f).Y,,), the sequence F1)) 0, n > 1. i.e. '(Y;, difference, c(L- 1, This suggests that martingales provide a very convenient framework for these limit theorems because by definition they are r.v.'s with respect to an increasing sequence of c-fields and under some general conditions they converge to some nv. as n tys. The latter being of great importance when r.v. is needed. Moreover, for any convergence to a non-degenerate 1) tXu, 9. u, n y: the martingale differences sequence martingale sequence orthogonal martingale sequence of r.v.'s which can help tL, n y 1) defines a us ensure (ii) above. =

=

=

=

.

.

.

=

,

-+

The SLLN is sometimes credited as providing a mathematical foundation for the frequency approach to probability. This is, however, erroneous because the definition is rendered circular given that we need a notion of probability to define the SLLN in the first place. Remarl:

9.3

The central limit theorem

As with the WLLN and SLLN, it was realised that LT2 was not contributing in any essential way to the De Moivre-Laplace theorem and the literature considered sequences of r.v.'s with restrictions on the first few jt j Xi, the CLT moments. Let tA-,,, 1177 1 be a sequence of r.v.'s and S,, of considers the limiting behaviour =

F,:

=

S,, - .E(S,,)

-7tvar(s,,); N' .

which is a normalised and SLLN.

,

version

Lindeberl-luevy,

of

F(%,), the subject matter of the WLLN

.%, -

theorem

j

#

lim F.,( y) ''

?,-.

cc.

)f'-

-

'

=

1im P(K. G r '

?,-.

w.

''

-

=

--,

'

-

.

x''(27:)

expt

-

1yu2)

d)?.

(9.28)

Limit theorems

Liapunov's theorem Let

kfX,,,

n > 1)

be a

VartA-zl

f

c2 i

.

< J >0. .E'(lAz'fI2+)

c/ < az,

=

r.v.'s wr/l 'aa,

1 2

11

)-

C= 1,

of independent

sequence

>

1

=

ten if

(9.30) theorem is rather restrictive because it requires the existence of higher than the second. A more satisfactory result providing both moments and sufficient conditions is the next theorem', Lindeberg in 1923 necessary established the if' part. part and Feller in 1935 the Liapunov's

'if'

'only

Lindeberq-Feller

theorem

Let (A',,, n 7: 1) be a sequence ofindependent )F,,(x),n > 1) such tat

r.p.'.s wf/'ll

distribution

jnctions

() Exi) ii) Vartxf)

=pi

Ten

=

(9.3 1)

c/ < vs,

the relations

(7) lim max l -+x un ?1

t2

11

Gi =

t??,

0,

where

tr,, = i

l =

c/

,'

1

(9.34) 11

jg (x i 1 l-pilxsc '

'

-pf)2

dFj(x)

-

=

0(c',2,) jr

alI

; >0.

(9.35)

The necessary and sufficient condition is known as the Lindeberg condition really gives rise to the result'. and provides an intuitive insight into twhat

The central Iimit theorem

9.3 Given that 1g

11

2 Cn i

F, =

y

1

:2

-/z)2

Ix

.-pikxircf

dFftx)

(x

max prtjAry - pfl > c(rf),

1% i ts n

this shows that the heart of the CLT is the condition that no one r.a dominates the sequence of sums, that is, each (Arf cf is small relative to the sum gS,,- f(S,,)q c,, as n increases. The Liapunov condition can be deduced from the Lindeberg condition and thus it achieves the same effect. Hence the CLT ref ers to ihe istributional bealliour of the summation of an increasing number of r.v.'s whfcll individuall), do not exert any signlkfcant c/-kcr on the behaviour of the sum. An analogy can be drawn from economic theory where under the assumptions of perfect competition (no individual agent dominates the aggregate) we can prove the existence of a general equilibrium. A more pertinent analogy can be drawn between the CLT and the theory of gas in physics. A particular viewpoint in physics considers a gas as consisting of an enormous number of individual particles in continuous but chaotic motion. One can say nothing about the behaviour of individual particles in relation to their position or velocity but we can determine (at least probabilistically) the behaviour of a large group of them. Fig. 9. 1 illustrates the CLT in the case where Ar,, are 1lD r.v.'s, 1, U( 1, 2, Xi l), uniformly distributed i.e. i n, and + Xz + + Xn. represents the density function of Y;, -/tf)

-:-1

,

,v

.

.

.

.

,

.

,(#)

=

-

.

,

.:-1

'

=

.

.

fl (p')

f (y) --#'

+.

Z#

y'

.* Z zz

N

-

--

-

a

(y1

*.

x

*

N

N

*

'

N

x.

0

y

Fig. 9. 1. lllustrating the CLT using the density function of are uniformly distributed for n 1, 2, 3.

where the

xfl.s

(y)

=

r

=

)'= Xi 1

Limit theorems

Returning to the Bernoulli and De Moivre- Laplace theorems, we can see that the important ingredients were the same in both cases and both families of limit theorems refer to the behaviour of the sum of a sequence of P

r.v.'s in a different probabilistic

sense. The WLLN referred n)Qp(land the CLT to

the SLLN to

to

r(,%,

' j--!S pj

jr 1.-,y.-?)

j-

r

--+

pj,

z xtt),1).

-..,

-

!),L/n)

,.

&

Let us consider the relationship between these limit theorems in the particular case of the binomial distribution. From the CLT we know that pv

...-

:u

. .--

,,'1),,p(

N/ '

b

S - np 1-p); ''

g
1) is a sequence oj' r.n.'s 1 0, n 1 sucb that: =

a martingale ,

,

.

.

.

,

(f)

(9.45) (9.46) 11

ctl, j

c/,

=

1

ci

=

EX,?),

j

then -.. f-n

?1

i

j

Xi 1

'w

N(0, 1). (9.47)

x = This theorem is a direct extension of the Lindeberg-Feller theorem. lt is important to note that (i)and (ii)ensure that the summations involved are of smaller order of magnitude than c,2, ....-

.

9.4*

Limit theorems for stochastic processes

The purpose of this section is to cosider briefly various extensions of the limit theorems discussed above to some interesting cases where )A',,, n > 1) is a stochastic process satisfying certain restrictions (seeChapter 8). The first step towards generalising the limit theorems to dependent r.v.'s has already been considered above for the case where kfX., n > 1) is a martingale relative to the increasing sequence of c-fields Jtf2pj, n 7: 1). Another interesting form of a stochastic process is the m-dependent process (seeChapter 8). For an m-dependent zero mean stochastic process (A',,, n > 1) with finite third moments, '(1A',,13) < K for all n > 1 and some

Limit theorems for stochastic processes

9.4*

constant K, it can be shown that 1:1

11

jg

V Xi --sn V

i

if c2

tT2),

ZxN(,

-+

--

1

)

1im 11 co

=

n

-+

11

oj-vi k

=

< cx',,

1

(9.48)

m-1

c/

=

j

2

Covtx''

+ Vart-f

Xi +.)

+j,

+.).

j=0

The importance of martingales and m-dependent processes lies with the and mixing stochastic processes behave fact that stationary-ergodic asymptotically like martingale differences and m-dependent processes, respectively. Hence, the limit theorems for martingales and m-dependent when certain processes can be extended to stationary and mixing processes restrictions related to their homogeneity and memory are imposed.

(t7) /(:n) O(r?1-'), =

(?) a(rn)

04?,n-1),

=

z>

(r (?--

(9.51)

1:,

1) is a mixing p?vpcas-s(44n1)or .:t4?A2)) then /./ny Borel lf zY,,is Xn k ) is also mixing. A-/rt?rct??.lpr, jllnction F;, f./,,(zY,,, O(n then F; is aIs() O(m z > 0. This result is of considerable interest in statistical inference where asymptotic results for functions of stochastic processes are at a premium. For stationary stoehastic processes which are also ergodic several limit theorems can be proved. =

.

.

.

,

-

-')

-')

1

SLLN

5

/br stationarq',

t.?rg(4Jfc'processes

1) be a stationary n > 1, then

.f..,:4f Xn,n > J;1.1,,(
1) dominates the behaviour of the sum (Z)'I A-), we can deduce that: .

jj

IA

1;

-n )'-=

i

-->

''

j

--+

11

i

1

a.s. z-i

=

J -

n

j

1

-rl i

Xi

-

11 i

j -

F(xYf) 1

(WLLNI;

11

) =

F(vYf) (sLLN); 1

Introduction

to asymptotic

theory

(CLT).

ln order to be able to extend these results to arbitrary Borel functions /7(X), notjust j)'- 1 Xi, we firstly need to extend the various modes of convergence (convergence in probability, almost sure convergence, convergence in distribution) to apply to any sequence of r.v.'s ).Y,,, ?1 y 1). The various modes of convergence related to the above limit theorems are considered in Section 10.2. The main purpose of this section is to relate notions of convergence to the probabilistic the various mathematical needed modes in asymptotic theory. One important mode of convergence encountered in the context of the limit theorems not convergence s 'convergence in the rth mean', which refers to convergence of moments. Section 10.3 discusses various concepts related to the convergence of moments such as asymptotic moments, limits of moments and probability limits in an attempt to distinguish between these concepts often confused in asymptotic theory. ln Chapter 9 it was stressed that an important ingredient which underlies the conditions giving rise to the various limit of magnitude'. For example, the theorems is the notion of the Markov condition needed for the WLLN, 'order

)

lim

c ?,-+x n

/1

k

j =

Vart-tk)

0,

=

1

is a restriction on the order of magnitude Var(&)

o(rl2).

=

In Section 10.4 the

tbig

of Var(&) of the form

( 10.6)

o' notation is considered in some detail as a O', prelude to Sections 10.5 and 10.6. The purpose of Section 10.5 is to consider the question of extending the limit theorems of Chapter 9 from (j)'- I Arf)to A',,) such as more general functions of (A-j Xz, ))'-1 XL,/-> 1. This is indeed the main aim of asymptotic theoo'. Asymptotic theory results are resorted to by necessity, when finite sample results are not available in a usable form. This is because asymptotic results provide only approximations. 'How good the approximations are' is commonly unknown because we could answer such a question only when the finite result is available. But in such a case the asymptotic result is not needed! There are, however, various error bounds which ean shed some light on the magnitude of the approximation error. Moreover, it is often possible to upon the asymptotic results using what we call Klittle

,

.

.

.

,

krough'

kimprove'

Modes of convergence

10.2

asymptotic expansions such as the Edgeworth expansion. The purpose of Section 10.6 is to introduce the reader to this important literature on error bounds and asymptotic expansions. The discussion is only introductory and much more intuitive than formal in an attempt to demystify this For a more literature whieh plays an important role in econometrics. complete and formal discussion see Phillips ( 1980), Rothenberg ( 1984), intelalia.

Modes of convergence

10.2

and play a very important role in of of only because the limit theorems discussed in probability theory, not they underlie Chapter 9 but also because some of the most fundamental concepts such as probability and distribution functionss density functions, mean, variance- as well as higher moments. This was not made explicit in Chapters 3-7 because of the mathematical subtleties involved. ln order to understand the various modes of convergence in probability theory 1et us begin by reminding ourselves of the notion of convergence in ftf.?,,, G mathematical analysis. n t is defined to be a function sequence from the natural numbers . # )1, 2, 3, ) to the real line R. 'convergence'

'limit'

The notions

'

,4

.

'

=

.

.

.

Dqpnition 1 f A smuence ( a n g # is said to converge to a Iimit a f/' jr t:'t,ta?-v Llrbitrar)' (1 l?un'l/?t?/- A'(;) such small nl/yl/afskls > 0 tbere ctp?-?-fpsptpnps ')

?;

,.

-

.

< t 1t.,, w't? denote tbis /1

l/-lar/'/?ta inequtllitq' n > N(;);

wlll

all rt?r,e'n-s ?,, t?/' /-':'IJs /?p lim,,-.. fk,, a. ./?-

.

-

the s:?t/?..',nce-

=

E'xtl/nkp/a1 ' .. ..

lim,,-, n (1,/'n)''-Ie .,x')'

=

1

j-

:.'

N

nj

=

=

.(log, 0, b > 0, b + 1,'1im,,-..I( 1 + nln) 1, for any n > 0'. 1im,,-+ + 2.7 1828-,1im,,- .,. g(n2 + n + 6)/(3n2 2n + 2)j -j;1im,,-+ . gx'''(n () fo r n g t ,

=

=

=

-

'.

5

,

whose This notion of convergence can be extended directly to kny --+ The necessarily but any subset of R, i.e. /?(.v):(J R. way domain is not this is done is to allow the variable x to define a sequence of numbel's 1x,,. h? G . t l converging to some limit x() and consider the sequence ) (x?,), --+ ?2i: . f as x,, .v() and denote it by limx .x(, /7(x) /. .lnction

.

'

','

=

.-.

Depn i rt?n 2 A function

said r(?converge to a Iimit I fxs (x) 1.%

x

--+

xo.

(J

'for

crt?rl?

Introduction

theory

to asymptotic

)> 0, bowever small, there exists

a number

J(#

0 such that

>

Example 2 For

(x)

=

ex, limx.-

/1(x) =

./1(x)

0 and for the polynomial

=

-

aox'' + t/lx'' -

1

+

.

.

+ a 11 j x + -

.

function

1imJl(x) a,,.

(I,t,

=

x-0

Ix

-x()l < J(c) excludes the point x xtl in the Note that the condition 0 < above definition and thus for /1(x) (x2 9),(x 3), limx-ps (x) 6, even though /7(x) is not defined at x 3. For =

=

=

-

=

(x)

=

t7X,

0,

a>

/l(x) (1 + x)1?-v, =

7

(x) /l(x)

=

=

lim /1(x) e, =

x-0

X

1+ X

=

lim /1(.x) 1, x-0

,

1im J?(x)

=

eJ,

x-0

gloget1 +

x) xj,

lim /?(x)

=

1,

x-0

Using the notion of convergence of a function to a limit we can detine the continuity of a function at a point. Djlnition

3

-4 function (x),dehned over some interval D) tf to be continuous at the point jr eacb (:) > 0 such tat .xo

l(x)

:

R, xa G D() is said 0, tere exists a

>

-(xo)I