Methods in Algorithmic Analysis

Methods in Algorithmic Analysis CHAPMAN & HALL/CRC COMPUTER and INFORMATION SCIENCE SERIES Series Editor: Sartaj Sahn...

Author: Vladimir A. Dobrushkin

398 downloads 2294 Views 5MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Methods in Algorithmic Analysis

CHAPMAN & HALL/CRC COMPUTER and INFORMATION SCIENCE SERIES Series Editor: Sartaj Sahni PUBLISHED TITLES ADVERSARIAL REASONING: COMPUTATIONAL APPROACHES TO READING THE OPPONENT’S MIND Alexander Kott and William M. McEneaney

HANDBOOK OF PARALLEL COMPUTING: MODELS, ALGORITHMS AND APPLICATIONS Sanguthevar Rajasekaran and John Reif

DISTRIBUTED SENSOR NETWORKS S. Sitharama Iyengar and Richard R. Brooks

HANDBOOK OF REAL-TIME AND EMBEDDED SYSTEMS Insup Lee, Joseph Y-T. Leung, and Sang H. Son

DISTRIBUTED SYSTEMS: AN ALGORITHMIC APPROACH Sukumar Ghosh

HANDBOOK OF SCHEDULING: ALGORITHMS, MODELS, AND PERFORMANCE ANALYSIS Joseph Y.-T. Leung

ENERGY EFFICIENT HARDWARE-SOFTWARE CO-SYNTHESIS USING RECONFIGURABLE HARDWARE Jingzhao Ou and Viktor K. Prasanna FUNDEMENTALS OF NATURAL COMPUTING: BASIC CONCEPTS, ALGORITHMS, AND APPLICATIONS Leandro Nunes de Castro HANDBOOK OF ALGORITHMS FOR WIRELESS NETWORKING AND MOBILE COMPUTING Azzedine Boukerche HANDBOOK OF APPROXIMATION ALGORITHMS AND METAHEURISTICS Teofilo F. Gonzalez HANDBOOK OF BIOINSPIRED ALGORITHMS AND APPLICATIONS Stephan Olariu and Albert Y. Zomaya HANDBOOK OF COMPUTATIONAL MOLECULAR BIOLOGY Srinivas Aluru HANDBOOK OF DATA STRUCTURES AND APPLICATIONS Dinesh P. Mehta and Sartaj Sahni HANDBOOK OF DYNAMIC SYSTEM MODELING Paul A. Fishwick

HIGH PERFORMANCE COMPUTING IN REMOTE SENSING Antonio J. Plaza and Chein-I Chang INTRODUCTION TO NETWORK SECURITY Douglas Jacobson METHODS IN ALGORITHMIC ANALYSIS Vladimir A. Dobrushkin PERFORMANCE ANALYSIS OF QUEUING AND COMPUTER NETWORKS G. R. Dattatreya THE PRACTICAL HANDBOOK OF INTERNET COMPUTING Munindar P. Singh SCALABLE AND SECURE INTERNET SERVICES AND ARCHITECTURE Cheng-Zhong Xu SPECULATIVE EXECUTION IN HIGH PERFORMANCE COMPUTER ARCHITECTURES David Kaeli and Pen-Chung Yew VEHICULAR NETWORKS: FROM THEORY TO PRACTICE Stephan Olariu and Michele C. Weigle


Vladimir a. dobrushkin brown uniVersity ProVidence, rhode island, u.s.a.

CRC Press Taylor & Francis Group 6000 Broken Sound Parkway NW, Suite 300 Boca Raton, FL 33487-2742 © 2009 by Taylor & Francis Group, LLC CRC Press is an imprint of Taylor & Francis Group, an Informa business No claim to original U.S. Government works Version Date: 20131120 International Standard Book Number-13: 978-1-4200-6830-6 (eBook - PDF) This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint. Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers. For permission to photocopy or use material electronically from this work, please access www.copyright.com (http:// www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com and the CRC Press Web site at http://www.crcpress.com

Contents Preface

xiii

List of Symbols

xvii

Abbreviations

xix

1 Preliminaries 1.1

1.2

1.3

Why Do We Analyze Algorithms? . . . . . . . . . . . . . . . . . . . . . . .

2

1.1.1

Cost of Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . .

2

1.1.2

One Problem – Several Solutions . . . . . . . . . . . . . . . . . . .

3

1.1.3

Matrix Multiplication . . . . . . . . . . . . . . . . . . . . . . . . . .

8

Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.2.1

Proof by Counting . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

1.2.2

Induction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Iteration and Recursion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

2 Combinatorics 2.1

25

Properties of Summation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.1.1

2.2

1

Index Transformations . . . . . . . . . . . . . . . . . . . . . . . . . 28

Multiple Sums . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 2.2.1

Changing Order of Summation . . . . . . . . . . . . . . . . . . . . . 34

2.2.2

Summations and Finite Differences . . . . . . . . . . . . . . . . . . 36

2.2.3

Summation by Parts . . . . . . . . . . . . . . . . . . . . . . . . . . 40

2.3

Principles of Counting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

2.4

Permutations and Combinations . . . . . . . . . . . . . . . . . . . . . . . . 44 2.4.1

2.5

Combinations and Lattice Paths . . . . . . . . . . . . . . . . . . . . 48

Binomial Coefficients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

2.6

2.5.1

Definitions and Properties . . . . . . . . . . . . . . . . . . . . . . . 56

2.5.2

Transformations and Basic Sums . . . . . . . . . . . . . . . . . . . . 58

2.5.3

Inverse Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

2.5.4

Vandermonde Convolution . . . . . . . . . . . . . . . . . . . . . . . 66

Binomial Coefficients and Hypergeometric Functions . . . . . . . . . . . . . 72 2.6.1

2.7

Abel’s Identity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

Stirling Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

3 Probability

79

3.1

Set Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

3.2

Sample Space and Random Variables . . . . . . . . . . . . . . . . . . . . . . 82

3.3

Calculating Probabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

3.4

Random Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 3.4.1

Probability Mass Function . . . . . . . . . . . . . . . . . . . . . . . 94

3.4.2

Expected Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

3.4.3

Variance and Moments . . . . . . . . . . . . . . . . . . . . . . . . . 112

3.4.4

Functions of Random Variables . . . . . . . . . . . . . . . . . . . . 119

3.5

Conditional Probabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

3.6

Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

3.7

Joint Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

3.8

Dependent Random Variables . . . . . . . . . . . . . . . . . . . . . . . . . . 130

4 More about Probability 4.1

135

Special Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 4.1.1

Bernoulli Variables and the Binomial Distribution . . . . . . . . . . . 136

4.1.2

The Multinomial Distribution . . . . . . . . . . . . . . . . . . . . . 140

4.1.3

The Geometric Distribution . . . . . . . . . . . . . . . . . . . . . . 141

4.1.4

The Negative-Binomial Distribution . . . . . . . . . . . . . . . . . . 148

4.1.5

The Hypergeometric Distribution . . . . . . . . . . . . . . . . . . . 151

4.1.6

The Poisson Distribution . . . . . . . . . . . . . . . . . . . . . . . . 154

4.1.7

The Normal Distribution . . . . . . . . . . . . . . . . . . . . . . . . 157

4.2

Types of Probabilistic Convergence

. . . . . . . . . . . . . . . . . . . . . . 158

4.3

The Theorem of Total Probability

4.4

Bayes’ Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170

4.5

Convolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174

. . . . . . . . . . . . . . . . . . . . . . . 162

4.6

Order Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183

4.7

Chebyshev Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188

4.8

Sundry Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192

5 Recurrences or Difference Equations

199

5.1

How Do Difference Equations Arise? . . . . . . . . . . . . . . . . . . . . . 200

5.2

Properties of Difference Equations . . . . . . . . . . . . . . . . . . . . . . . 208

5.3

First Order Linear Difference Equations . . . . . . . . . . . . . . . . . . . . 215

5.4

Recurrences with “Integer Functions” . . . . . . . . . . . . . . . . . . . . . 221

5.5

5.4.1

Divide-and-Conquer Recurrences . . . . . . . . . . . . . . . . . . . 221

5.4.2

Binary Search Recurrence . . . . . . . . . . . . . . . . . . . . . . . 224

5.4.3

Mergesort Recurrence . . . . . . . . . . . . . . . . . . . . . . . . . 227

Quicksort Recurrence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 5.5.1

Some Full-History Recurrences . . . . . . . . . . . . . . . . . . . . 233

5.6

Recurrences in Numerical Analysis . . . . . . . . . . . . . . . . . . . . . . . 236

5.7

Continued Fractions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242

5.8

Partial Difference Equations . . . . . . . . . . . . . . . . . . . . . . . . . . 251

5.9

5.8.1

The Relations with Continuous Calculations . . . . . . . . . . . . . . 254

5.8.2

Counting Arrangements—with and without Repetition . . . . . . . . 256

5.8.3

Stirling Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261

Some Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 5.9.1

Bounds from Recurrences . . . . . . . . . . . . . . . . . . . . . . . 263

5.9.2

Recurrences and Finite Differences . . . . . . . . . . . . . . . . . . 266

6 Introduction to Generating Functions 6.1

6.2

271

Generating Functions — Definitions . . . . . . . . . . . . . . . . . . . . . . 272 6.1.1

Ordinary Generating Functions . . . . . . . . . . . . . . . . . . . . . 272

6.1.2

Exponential Generating Functions . . . . . . . . . . . . . . . . . . . 277

6.1.3

Some Generalizations . . . . . . . . . . . . . . . . . . . . . . . . . 280

6.1.4

Multivariate Generating Functions . . . . . . . . . . . . . . . . . . . 283

6.1.5

Multisection of Series . . . . . . . . . . . . . . . . . . . . . . . . . 285

Extraction of Coefficients . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 6.2.1

Transformations between the Generating Functions . . . . . . . . . . 293

6.2.2

Multivariate Generating Functions . . . . . . . . . . . . . . . . . . . 294

6.2.3

Recurrences from Generating Functions . . . . . . . . . . . . . . . . 296

6.3

Counting Binary Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299

6.4

Solving Recurrences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 6.4.1

Ordinary Recurrence Relations . . . . . . . . . . . . . . . . . . . . . 305

6.4.2

Vector Recurrence Relations . . . . . . . . . . . . . . . . . . . . . . 315

6.4.3

Partial Difference Equations . . . . . . . . . . . . . . . . . . . . . . 319

6.4.4

Walks on the Integer Grid . . . . . . . . . . . . . . . . . . . . . . . 323

6.5

Snake Oil Summation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327

6.6

Applications in Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . 331

6.7

6.6.1

Definition of Generating Functions Used in Probability . . . . . . . . 331

6.6.2

Examples and Problems . . . . . . . . . . . . . . . . . . . . . . . . 335

6.6.3

Convolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337

6.6.4

Quicksort and Binary Search Analysis . . . . . . . . . . . . . . . . . 340

The Lagrange Inversion Theorem . . . . . . . . . . . . . . . . . . . . . . . . 347

7 Enumeration with Generating Functions

355

7.1

Definition of Enumerators . . . . . . . . . . . . . . . . . . . . . . . . . . . 355

7.2

Sum and Product Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359

7.3

7.4

7.2.1

The Sum Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359

7.2.2

The Product Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360

Counting Compositions of Integers . . . . . . . . . . . . . . . . . . . . . . . 364 7.3.1

Homogeneous Compositions . . . . . . . . . . . . . . . . . . . . . . 364

7.3.2

Inhomogeneous Compositions . . . . . . . . . . . . . . . . . . . . . 367

7.3.3

Compositions with Restrictions . . . . . . . . . . . . . . . . . . . . 369

7.3.4

Heterogeneous Components . . . . . . . . . . . . . . . . . . . . . . 372

7.3.5

Limited Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375

Further Set Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381 7.4.1

Substitution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383

7.4.2

Marking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 388

7.4.3

Power Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 390

7.4.4

Multiset Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . 391

7.5

Partitions of Integers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392

7.6

Exponential Enumerators . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403 7.6.1

The Sum and Product of Labeled Structures . . . . . . . . . . . . . . 404

7.6.2

Permutations and Cycles . . . . . . . . . . . . . . . . . . . . . . . . 408

7.6.3

Shuffle Product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413

7.6.4

The Birthday Problems . . . . . . . . . . . . . . . . . . . . . . . . . 416

8 Further Enumeration Methods 8.1

8.2

8.3

Enumeration of Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423 8.1.1

Unlabeled Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424

8.1.2

Labeled Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429

8.1.3

Counting Alternating Permutations . . . . . . . . . . . . . . . . . . 430

Occupancy Enumeration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434 8.2.1

Distribution of Identical Balls into Distinguishable Bins . . . . . . . 435

8.2.2

Distribution of Distinct Objects into Ordered Cells . . . . . . . . . . 439

8.2.3

Distribution of Identical Objects into Identical Cells . . . . . . . . . 445

8.2.4

Distribution of Distinct Objects into Identical Cells . . . . . . . . . . 445

The Principle of Inclusion and Exclusion (PIE) . . . . . . . . . . . . . . . . . 446 8.3.1

8.4

423

The PIE for Homogeneous Properties . . . . . . . . . . . . . . . . . 455

Extensions and Further Applications of the PIE . . . . . . . . . . . . . . . . 460 8.4.1

The PIE via the Symbolic Method . . . . . . . . . . . . . . . . . . . 464

8.5

Probabilistic Inclusion – Exclusion Principle . . . . . . . . . . . . . . . . . . 468

8.6

Runs in Permutations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479

8.7

8.6.1

Counting Permutations of [1..n] with k Ascents . . . . . . . . . . . . 480

8.6.2

Counting Permutations of [1..n] with Runs of Ascents of Length r . . 482

8.6.3

Counting Permutations of [1..n] with m Maximal Runs of Ascents . . 483

Special Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483

9 Combinatorics of Strings

489

9.1

Operations on Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490

9.2

Regular Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494

9.3

9.4

9.2.1

Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 495

9.2.2

Finite State Automata . . . . . . . . . . . . . . . . . . . . . . . . . 497

9.2.3

Finite State Automata and Regular Languages . . . . . . . . . . . . . 499

Counting Regular Languages . . . . . . . . . . . . . . . . . . . . . . . . . . 503 9.3.1

Word Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503

9.3.2

Counting Regular Languages . . . . . . . . . . . . . . . . . . . . . . 505

9.3.3

Admissibility Considerations . . . . . . . . . . . . . . . . . . . . . . 514

Waiting Time Probabilistic Problems . . . . . . . . . . . . . . . . . . . . . . 517

9.5

Algorithms and Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . 527

10 Introduction to Asymptotics

545

10.1 Asymptotic Notations and Applications . . . . . . . . . . . . . . . . . . . . 546 10.1.1 Properties of the Big Oh Notation . . . . . . . . . . . . . . . . . . . 547 10.1.2 Asymptotic Expansions

. . . . . . . . . . . . . . . . . . . . . . . . 550

10.1.3 Limits for Indeterminate Forms . . . . . . . . . . . . . . . . . . . . 559 10.2 The Critical Range Method . . . . . . . . . . . . . . . . . . . . . . . . . . . 563 10.3 Rice’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 569 10.4 The Euler Summation Formula . . . . . . . . . . . . . . . . . . . . . . . . . 579 10.4.1 Alternating Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . 589 10.5 Finding Primes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 591 10.6 Asymptotics from Recurrences . . . . . . . . . . . . . . . . . . . . . . . . . 598 10.6.1 Divide-and-Conquer Asymptotics . . . . . . . . . . . . . . . . . . . 598 10.6.2 Direct Asymptotics . . . . . . . . . . . . . . . . . . . . . . . . . . . 604 10.7 Limit Laws in Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . 609 10.7.1 Laws of Large Numbers . . . . . . . . . . . . . . . . . . . . . . . . 610 10.7.2 The Central Limit Theorem . . . . . . . . . . . . . . . . . . . . . . 616 10.7.3 Random Walks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623 11 Asymptotics and Generating Functions

629

11.1 Elementary Bounds from Generating Functions . . . . . . . . . . . . . . . . 629 11.1.1 The Lagrange Inversion Formula . . . . . . . . . . . . . . . . . . . . 632 11.2 Estimates from Singularities . . . . . . . . . . . . . . . . . . . . . . . . . . 634 11.2.1 Poles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634 11.2.2 Difference Equations: Asymptotics from GFs . . . . . . . . . . . . . 641 11.2.3 Removal of Singularities . . . . . . . . . . . . . . . . . . . . . . . . 644 11.2.4 Darboux Theorem and its Relation to the Binomial Theorem . . . . . 646 11.2.5 Logarithmic Singularities . . . . . . . . . . . . . . . . . . . . . . . . 648 11.3 Estimates from Entire Functions . . . . . . . . . . . . . . . . . . . . . . . . 649 11.3.1 Integration by Parts . . . . . . . . . . . . . . . . . . . . . . . . . . . 649 11.3.2 Asymptotics of Hypergeometric Series . . . . . . . . . . . . . . . . 653 11.3.3 The Laplace Method . . . . . . . . . . . . . . . . . . . . . . . . . . 656 11.4 Examples and Exercises

. . . . . . . . . . . . . . . . . . . . . . . . . . . . 658

12 Review of Analytic Techniques

661

12.1 Complex Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 661 12.2 Review of Power Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 667 12.2.1 The Taylor Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . 667 12.2.2 Operations on Power Series . . . . . . . . . . . . . . . . . . . . . . 673 12.3 Functions of a Complex Variable: Basic Concepts . . . . . . . . . . . . . . . 677 12.3.1 Diagonalization of Series . . . . . . . . . . . . . . . . . . . . . . . . 683 12.4 Differential Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 686 12.5 Partial Fraction Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . 692 12.6 Some Special Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 694 12.7 Stieltjes Integrals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 696 Appendices

699

Appendix A: Binomial Coefficients . . . . . . . . . . . . . . . . . . . . . . . . . . 699 Appendix B: The Bernoulli Numbers . . . . . . . . . . . . . . . . . . . . . . . . . 706 Appendix C: Stirling and Euler/Eulerian Numbers . . . . . . . . . . . . . . . . . . 708 Appendix D: Fibonacci Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . 715 Appendix E: Harmonic Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . 719 Appendix F: Miscellaneous Formulas . . . . . . . . . . . . . . . . . . . . . . . . 721 Appendix G: The Gamma Function . . . . . . . . . . . . . . . . . . . . . . . . . . 725 Appendix H: Random Variables and Distributions . . . . . . . . . . . . . . . . . . 728 Appendix I: Combinatorics of Permutations . . . . . . . . . . . . . . . . . . . . . 730 Appendix J: Continued Fractions . . . . . . . . . . . . . . . . . . . . . . . . . . . 732 Appendix K: Occupancy Enumeration . . . . . . . . . . . . . . . . . . . . . . . . 738 Appendix L: Generating Functions . . . . . . . . . . . . . . . . . . . . . . . . . . 740 Appendix M: Recurrences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 749 Answers/Hints to Selected Problems

753

Bibliography

772

Index

783

Preface This book started its life as a collection of lecture notes for the graduate course, “Analysis of Computations and Systems,” taught at the computer science department of the Worcester Polytechnic Institute (WPI). While we think of this as primarily a course about the analysis of algorithms, the title has been selected to distinguish it from the courses usually given this analysis in algorithms title. The reason for the distinction is that our course emphasizes the analysis—its aims and techniques—rather than the algorithms and their design. Surely many algorithms get mentioned, discussed, and analyzed, but the motivation for our choosing an algorithm is based on its analysis that contributes to the students’ understanding of the methodology we develop. Therefore, the goal of Methods in Algorithmic Analysis is to expose students to mathematical techniques and methods that are usable and effective in theoretical aspects of computer science. Analysis of algorithms is arguably a branch of applied mathematics, and, as such, should have any needed mathematical tools available for it. Thus, one of the functions of this text is to provide information about useful tools when they occur naturally in analysis. As a result the reader will find tutorials about diverse topics such as continued fractions, functions of a complex variable, numerical calculations, and Stieltjes integration, placed throughout the book.

Prerequisite We assume the following four areas as basic prerequisites for users of the text. ALGORITHMS The reader is assumed to have been exposed to an algorithms course at the undergraduate level, or equivalent experience. At WPI, we have courses covering algorithms at both the undergraduate and graduate levels. Any of these would be a good preparation for our course on the algorithmic portion of the book. Familiarity with common data structures is required for numerous topics. MATHEMATICS We assume a standard undergraduate exposure of mathematics that typical computer science students study. Moreover, realizing that many take the undergraduate course on discrete mathematics early in their study, and often succeed in completing their degree with very little reference to it, the first few chapters of the book are devoted to a review, with some extensions, of the combinatorics. PROBABILITY Computer science students typically follow a traditional curriculum with one or two courses in probability and statistics. Although these prerequisite courses do introduce some basic material, students experience difficulties in the application of probability

xiii

to the analysis of algorithms. In present day applications there are many probabilistic algorithms. Since a thorough understanding of these algorithms and the ability to analyze them are of great importance, the text provides (in two chapters) a ground for probability with theory, examples, and lots of exercises. The reader will find many applications of probability throughout the text. PROGRAMMING The reader is assumed to have had experience with computer programming, or at least have had some intuitive understanding of it. The text requires some Cprogramming background. Many algorithms are presented in C-code fragments and only a few in pseudocodes (just to avoid variable declarations). Such C-based style became a standard algorithm presentation for computer science audiences and should be easily accessible to our readers. A reader may get one more benefit of using C-codes in actual computations.

All presented codes could also be rewritten in other languages, including computer algebra systems. To support such an approach, we also provide some MAPLE applications. Also, we encourage the reader to use a computer algebra system, which simplifies tedious calculations. The code samples serve as pedagogical examples of algorithms and are not, as a rule, the most efficient versions. The reader may view them as an invitation to find improvements while analyzing the given imlementation of the algorithms. This is a usual pathway to perform the analysis on some version of the algorithm to determine what parts of it can be improved or modified.

Targeted audience The book is primary designed for use in a graduate course on algorithms. Therefore, the goal of Methods in Algorithmic Analysis is to expose students to mathematical techniques and methods that are usable and relevant in theoretical aspects of computer science. It is also our hope that the book will help students in pursuing Ph.D. degrees in computer science. We expect readers will find that the text goes beyond their course needs, and that it can serve as a reference for their subsequent careers. Some features of the text lend themselves to this aim. There is a substantial Appendix containing many useful formulas. The book can be used to support self-study by graduate students and computer professionals who pursue profound knowledge. The text has many exercises (there are almost thousand of them) from different areas, so an instructor may want to use the book as a supplementary resource for teaching. A solution manual will be made available for instructors containing answers to all the problems included. It could also be interesting for researchers because many problems or their solutions are original and hard to find in other sources.

How the book can be used This book is very flexible and it can be used in lectures or in class working through the problems that accompany the material being covered. Since examples and exercises are integrated into the text, we suggest taking advantage of their presence and embrace students into interactive discussion of the material. Our current schedule calls for the students to spend one half of the total time they devote to the course (outside of class) to preparatory reading. For every class, a few sections of the book are designated to be read, with the understanding on our part that some issues may be less than entirely clear. While we made a conscious effort to make

the book suitable for such modus operandi, as this material refracts through human nature, it is inevitable that the required points of view take time to develop. Subsequently, in class, we spend nearly all the time on a succession of exercises from the text that the instructor asks the students to solve, and after they work, usually in pairs, for 10–15 minutes, the problem is solved on the board, with—at this stage—active participation of the class. Most homework assignments are based on the text, with occasional extensions and variations. These take care of the other half of the time spent on this course by the students. The difficulty of the exercises in the book vary. We indicate the difficulty of the exercises by marking them with [1], [2], [3], [4], or [5], where exercises marked with [1] are the easiest and ones marked with [5] are the most challenging. Solutions to the problems marked with [5+] are not known to the author. The sources of problems and exercises are given when known. Almost all the exercises in the text call for algebraic or analytic calculations (as opposed to numerical ones). In certain situations, we found numerical work to be essential in gaining intuition and confidence in our work. When the analysis of an algorithm seems technically sound and yet, the result is surprising, just simulate the algorithm performance. It is a process you will find illuminating. While we could not avoid theorems and their proofs entirely (and explain in the text why understanding the proof of a theorem is needed to use the theorem intelligently), the real substance and basis of the course are its large selection of examples and exercises that are almost all scattered throughout the text, placed close to the point where the topic they deal with is developed. Exercises that are not closely related to the material covered are presented at the end of the sections. The text uses only standard notations and abbreviations [et al. (et alii from Latin) means “and others,” or “and co-workers;” i.e. from Latin “id est” meaning that is, that is to say, or in other words; e.g. stands for the Latin phrase “exempli gratia,” which means for example; and etc. (et cetera) means “and the others,” “and other things,” “and the rest”]. However, we find it convenient to type at the end of proofs or at the end of topic presented; the symbol ⊳ indicates the end of definitions; also, we use at the end of exercises and examples (unless a new one serves as a delimiter). We hope that the reader understands the difference between = (equal) and ≡ (equivalence relation). Sometimes we put the question mark (?) above = def to indicate that this claim should be either verified or dismissed. Also = is used for short to signal that the expression follows by definition. For convenience, we present the list of symbols and abbreviations at the beginning of the text. It should be noted that some symbols in mathematics serve different meanings depending of context. For example, the symbol Σ is used for summation, however, it is customarily used as the set of events (in probability) and as an alphabet (in language theory). Also, the symbol ∼ serves asymptotic relations and it is used to indicate that a random variable has a particular distribution. In the text, figures and tables are labeled by the number of the page where they are presented. We encourage our students to avail themselves of any computational resource open to them, especially computer algebra packages such as MAPLE, MAXIMA, or MATHEMATICA.

Acknowledgments This book would have not been created if students did not complain about the past texts used. In addition, however, I have gained much from their comments and suggestions about various components of the book, and for this I would like to thank the students of CS504 in WPI over the last seven years. Many of them have provided beneficial insights, questions, and feedback on the material in the text. Another valuable push to write the textbook was given by Professor Micha Hofri, who made contributions to the project in various ways: by advising on topics to be included, encouraging my interest in the subject, patiently monitoring my progress of writing, carefully reading and checking presented material, and helping with exercises and their solutions. I am also thankful to Professors Ray Beauregard, Norman Finizio, Richard Goldstein, Edmund Lamagna, Anany Levitin, Carl Libis, Stanley Selkow, and Arkady Yerukhimovich who generously contributed their time to provide detailed and thoughtful reviews of the manuscript; their helpful suggestions led to numerous improvements. Dr. and Mrs. Jarvis made a lot of suggestions to the final version of the text and the exposition of the material. Many thanks to both of them. Additional impetus and help has been provided by the professional staff of our publisher, Taylor & Francis Group, particularly Robert B. Stern, Stephanie J. Morkert, Michele A. Dimont, Shashi Kumar, and Kevin Craig. Finally, I thank my family for putting up with me while I was engaged in the writing of this book. Vladimir Dobrushkin Brown University, Providence, RI WPI, Worcester, MA

List of Symbols

n! (2n)!! (2n + 1)!! Z Z

+

N Ik [a..b] [1..n] ha, b, . . . , ci

A∩B

A∪B . ∪ |A| |n| A′

factorial,

n! = 1 · 2 · 3 · . . . · n.

double factorial, (2n)!! = 2 · 4 · 6 · · · (2n − 2)(2n) = 2n n!

double factorial, (2n + 1)!! = 1 · 3 · 5 · · · (2n − 1)(2n + 1) = (2n + 1)!/2n n! the set of all integers.

(or Z+ or I1 ) the set of natural numbers, { 1,2,3,4 . . .}. the set of nonnegative integers, { 0,1,2,3,4 . . .}. the set of integers > k, {k, k + 1, . . .}.

the set of the b − a + 1 integers, {a, a + 1, . . . , b} if a 6 b and ∅ otherwise. the set of the first n natural numbers, {1, 2, . . . , n}.

or (a, b, . . . , c) the array—ordered subset of a given set. ≡ AB intersection of two sets.

union of two sets.

the union of disjoint sets. cardinality (number of elements) of the set A. absolute value of the number n. or A or AC complement to the set A.

Pr[A]

probability of the event A.

E[X ]

≡µ

V [X ] N(µ , σ 2 )

≡σ

expected value or mean. 2

variance.

normal distribution, see §4.1.7.

⌊X ⌋

floor of X , the greatest integer less than or equal to X.

{X }

fractional part of X : {X } = X − ⌊X ⌋.

⌈X ⌉

π (n) nk nk

ceiling of X , the least integer greater than or equal to X. the number of primes not exceeding n.

falling factorial, n(n − 1) . . . , (n − k + 1), rising factorial, n(n + 1) . . . (n + k − 1), xvii

Eq. (2.34). Eq. (2.38).

n k ln x lg x

nk , or the number of combinations. k! natural logarithm, that is, the logarithm with base e: ln x = loge x. binomial coefficient,

logarithm with base 2, that is, lg x = log2 x.

Fn

Fibonacci numbers F0 = 0, F1 = 1,

Bn

Bernoulli numbers,

n k n k En (x) a1 , . . . x F b1 , . . . Hn

(s)

Hn

B1 = 1,

Fn = Fn−1 + Fn−2, n > 2. −1 n−1 n + 1 Bn = ∑ j B j. n + 1 j=0

signless Stirling numbers of the first kind, see Eq. (C.2). Stirling numbers of second kind, see Eq. (C.20). Euler polynomial, Eq. (C.54) on page 714. ≡

m Fn (a1 , . . . , am ; b1 , . . . , bn ; x)

=

ak , . . . , ak xk , k! k>0 n

∑ b1k , . . . , bmk

n 1 1 1 1 nth harmonic number Hn = 1 + + + · · · + = ∑ 2 3 n k=1 k

n

≡ ζ (n; s) =

∑ k=1

(n > 0).

1 s-harmonic number or the incomplete zeta function. ks n

en (x)

see §4.1.5.

1

incomplete exponential function :

en (x) =

xk

∑ k! .

k=0

∆ ∇

φ p(n) pk (n) pd (n) qk (n) f ∼g |=

δm,n

[C ] Σ∗ Σ+

forward finite difference: ∆xn = xn+1 − xn .

backward finite difference: ∇xn = xn − xn−1 . √ √ 1+ 5 1− 5 π π ˆ = 2 cos , φ = = −2 cos . golden ratio: φ = 2 5 2 10 number of partitions of n, n ∈ N.

number of partitions of n with k parts, n, k ∈ N.

number of partitions of n with distinct parts, n ∈ N.

number of partitions of n with parts up to k, n, k ∈ N. f (n) → 1 as n → ∞. asymptotically equal: g(n) correspondence by bijection.

Kronecker delta is 0 if n 6= m and 1 if n = m. ( 1, if the condition C is satisfied, the Iverson bracket. = 0, otherwise; = {An : n = 0, 1, 2, . . .}, def

the closure of the alphabet Σ, page 492;

= AA∗ = {An : n = 1, 2, . . .},

Definition 9.14 on page 492.

Abbreviations a.s. (a.s.)

→

Bn

almost sure convergence, Definition 4.68 on page 158, almost sure convergence, Definition 4.68 on page 158, Bell number, Eq. (C.4) on page 708.

BST

binary search tree, page 235,

CAS

Computer Algebra System,

CLT

Central Limit Theorem,

d

→

convergence in distribution (or in law), Definition 4.70 on page 159,

EGF

exponential generating function, Definition 6.10 on page 277,

ESF

Euler Summation Formula, §10.4,

En (u)

Eulerian polynomial, Eq. (C.51) on page 713,

gcd

greatest common divisor,

GF

generating function, §6.1,

Lp

→

convergence in L p , Definition 4.68 on page 158,

f [−1]

inverse function to f , for instance, arctan = tan[−1] ,

l.i.m.

limit in the mean, Definition 4.73 on page 160,

LIT

Lagrange Inversion Theorem, §6.7

MGF

moment generating function, Definition 6.146 on page 333,

OGF

ordinary generating function, Definition 6.1 on page 272,

PFD

partial fraction decomposition, §12.5,

PGF

probability generating function, Definition 6.141 on page 331,

PMF

probability mass function, Definition 3.36 on page 95,

PIE

Principle of Inclusion and Exclusion, §8.3,

Pr

probability or probability measure, Definition 3.13,

Pr

convergence in probability, Definition 4.67 on page 158,

→

RBS

recursive binary search, page 224.

xix

To my mother and to my wife

Chapter 1

Preliminaries In this chapter, we discuss a few issues that we need to clarify before moving into the technical material involved in the analysis of computations. We assume the reader is familiar with the concepts algorithm, program, computation, and execution. Let us get precise about some aspects of these. • By a computation we mean an execution of an algorithm. We use this alternative term to make a point: the analysis is of an actual computation, not an abstraction. Even when we consider a standard algorithm and write it down in the text using some high-level programming language, the analysis is of a specific variant and implies a particular implementation. • We assume the computation is done on a conventional computer. We make no particular assumptions about its architecture or operations, unless a special case requires this. • Most of the time we are concerned with the tools and methods of analysis, rather than the algorithms themselves. However, we shall sometimes digress on the insight analysis sheds on ways to improve algorithms. • We examine algorithms expressed in pseudo-code or a high-level language; since we do not assume an architecture, the machine instructions are not considered. • We obtain our results by calculation, rather than by measurement or simulation (or guesswork). That is why a background in discrete mathematics is useful, although we review all concepts in use. The opening section serves as a motivation for the analysis of algorithms—the topic has wide impact on many disciplines within and beyond computer science. Since the ultimate goal of the text is to present methods, techniques, and tools used in the analysis, the chapter gives an introduction to the basic mathematical methods needed to understand the material. The reader is assumed to be familiar with standard terminology used in data structure and discrete mathematics as well as having some programming experience. Therefore some topics and terms (arrays, binary search, hashing, sorting algorithms, expected value, and others) are used without thorough explanations, although they are defined in later chapters.

1

2

CHAPTER 1. PRELIMINARIES

1.1 Why Do We Analyze Algorithms? Roughly speaking, algorithms can be analyzed for their correctness (so they produce the proper output on every input they are expected to work on) and for their performance. Here we focus on the latter aspect. It is true that performance analysis will sometimes reveal logical flaws in an algorithm, or provide clues for its improvement, but that would be a (nice) byproduct only. Though let us add one caveat: to analyze an algorithm we need to understand it very intimately. Often, the most certain route to understanding an algorithm is to prove its correctness. Except for a few cases, this text does not follow this approach.

1.1.1 Cost of Algorithms Analyzing performance means finding the cost of executing, or running, an algorithm. This cost consists of running time and work space, sometimes called space usage. We shall see cases where they are related, in the sense that one can reduce the need for one by spending more of the other. A typical result of such analysis is a formula for the cost in terms of parameters of the problem—mostly its size, but there are often additional issues that affect the cost. What is our cost model for running time? How do we measure it? In seconds? In microseconds? In this context time units are not very useful. We prefer a cost unit that is independent of the machine on which we run our computations, especially its clock rate. We would also want to be able to disregard most details of machine architecture, such as pipelines, parallel processing, caches, storage hierarchy, and, to the extent possible, even important issues such as programming language and compiler. The reason is that all these factors are hardly relevant to comparisons between competing algorithms, which is the real issue in performance analysis. They affect certain coefficients that appear in the formula we mentioned, but not its structure: the way it depends on the problem parameters. And it is this structure we want to find when we analyze. We achieve this independence by counting certain “events” instead of measuring the elapsed time during the execution of an algorithm. We do not use events that are as “low” as machine instructions, for the reasons above. Sometimes we count all high-level instructions, but usually we single out some operations that we feel characterize that algorithm well. In the first example below we count comparisons of keys of records. In other cases we may count multiplications, page-faults, input/output accesses, or function calls. Note: This calculated cost used to be called the complexity of the algorithm. Not any longer. The term “complexity” in modern usage does not refer to an algorithm, but to a problem. The complexity of a problem is determined by the cost of the best algorithm for the problem (an algorithm that may not be known explicitly!). Furthermore, in this context, the “best” algorithm is the one that minimizes the cost of the worst-case instance of the problem, an instance that may be very odd, and very different from the usual or average situation. It is a min-max scenario. Almost always, the only parameter that determines the complexity of a problem is its size, whereas we can also consider additional factors, such as probability distributions. We view our approach as engineering-oriented. Example 1.1 The Linear Programming optimization problem is now known to require run-

1.1. WHY DO WE ANALYZE ALGORITHMS?

3

ning time that, even in the worst case, is polynomial in the number of column variables1 . The algorithm most commonly used to solve it—the simplex, invented by George Danzig in 1946—is exponential (in this parameter) in the worst case, but it is linear on the average, and with small coefficients! This is the reason it is so popular. While we shall occasionally refer to worst case cost, our main interest from now on is in techniques for evaluating the expected (or average) cost of algorithms and the likely variations of the cost relative to this average.

1.1.2 One Problem – Several Solutions The following is adapted from an exposition by J. L. Bentley [16]. We present a common data processing problem and discuss various algorithms that solve it. The problem is known as “Subset testing.” Let A and B be two sets of numbers with known sizes, |A| = n and |B| = m 6 n. We consider algorithms that answer the following question: is A ⊇ B? Only a yes/no answer is desired. Exercise 1.2 [1] This is not a very informative answer. What other, more detailed questions can be asked? Suggest at least three possibilities. We look at a few algorithms for this problem and use as our cost measure the number of element comparisons. Further assumptions are: (i) The elements of each set are in an array (and we call the two arrays, naturally, A and B). (ii) The elements in each set are distinct. In each case we shall provide actual code, written in the C programming language or a similar style, that returns 1 for “yes” and 0 for “no.” For reasons of brevity, and also to keep our attention on the main issues, only code for the critical part of the algorithm is provided. Method I – Brute Force Take each element b ∈ B, in order, and search for it in A. Stop when all the elements of B were found, or when a first b ∈ B is not in A. Here is a procedure that performs this test: 1: 2: 3: 4: 5: 6:

int BF (int ∗A, int n, int ∗B, int m) {int i, j, b; for (i = 0; i < m; i + +) {b = B[i]; for ( j = 0; j < n; j + +) if (b == A[ j]) break; // found this term return (0); // did not find this b } return (1); }

We only count the key comparisons, in line 4. The cost depends on the relation between A and B. We distinguish between two situations only: either B is in A or it is not. We deal with extreme situations first, and consider averages later. Consider first the case B ⊆ A. The algorithm will then find all the elements of B in A. The minimum cost is when it finds them in the first m positions of A, and then the number of required comparisons is k for the element found in position k, for a total of 1 + 2 + · · · + m = m(m + 1)/2. The worst case then occurs when we find them all in the rear of A, in the last m positions; that 1 Chapter

31 of [10] is a brief algorithmic and historical survey of this problem and discusses its complexity.


4

is, in positions n, n − 1, . . . , n − m + 1. Again we need to add m numbers, but now the sum is m(2n − m + 1)/2. More about this expression can be found in Example 1.24 and §2.1.1.

Consider now what happens when B * A. The situation is in principle more complicated, since the cost depends how B fails to be all in A, but the extreme cases are easy to determine. The minimum cost is n; this occurs when the first element of B we test is not in A, and then line 5 is executed.

The maximum cost occurs now when almost all of B is in A; in fact, when m − 1 elements are there. The cost has then two parts: (1) the first m − 1 elements of B are found in the last m − 1 positions in the array A, (2) the last element of B is not in A and is compared with all its n elements. Algorithmic notes: (a) The cost is invariant under the order in which the first m − 1 elements of B are found in A (this is also true for the cost we calculate above for the case B ⊆ A; do you see why?). (b) The final, missing element is tested in n comparisons, since the program does not keep track of the elements of A that have already been “used up,” or found; the cost of keeping track of this is much higher than the saving of the “unneeded” tests (why? how would you justify this?). The number of comparisons is then n

n+(n−1)+(n−2)+· · · +(n−(m−1)+1)+n =

∑i

i=n−m+2

+n =

(2n − m + 3)m − 2 . (1.1) 2

Note how close this is to the worst cost when B ⊆ A: it is larger by m − 1. Do you see a simple intuitive explanation? However, both expressions are cumbersome; to obtain insight from such expressions, we use two simplifications. First, we look for the most significant, or largest, terms; in the last expression these terms are mn − m2 /2. Second, when we have more than one parameter—as in the present case, where we have two, m and n—we reduce the number, by making various assumptions (such as: m is small, or m ≈ n, where the symbol ≈ means “roughly of the same size,” or “approximately equal to”). Here, the latter case is the more “interesting” of the two, and using it in the significant terms, we find that the worst possible cost then ≈ n2 /2.

Average costs. Note: Some of the derivations and exercises below require elementary concepts from probability theory. If you are not comfortable with them, you may want to revisit this part after we cover some probability theory in Chapters 3 and 4. In the average case, if B ⊆ A, each element of B is found, when averaged on all searches, midway in A. The total expected cost is then slightly over 12 mn term comparisons. Otherwise, when B * A, it is hard to say anything definite, since it depends on how the relation B ⊆ A fails. Exercise 1.3 [3] If you already know elementary probability theory, justify the claim made about the expected cost in the last paragraph, under the assumption that the elements of B are uniformly distributed in A.

Exercise 1.4 [2] Show that the minimal cost when the relation B ⊆ A holds is 1 + 2 + · · · + ≈ m2 /2. m = m(m+1) 2 Example 1.5 Here we consider the cost of this method in a specific failure mode of the re-


5

lation B ⊆ A. Let |A| = n = 2m, where |B| = m is even, and exactly half the elements of B are in A. What are the minimum, maximum, and expected values of the cost of running this algorithm?

Solution: The extreme values require no new observations: the minimal cost is when the first element of B is not in A, requiring n comparisons. We are hit with the maximum as above: when we first test all the elements of B which are in A, and we get the same expression, except that we have m/2 instead of m − 1, and therefore the maximal cost is now (assuming m is even), m(7m + 18)/8. Let C be the number of comparisons needed to verify that B ⊆ A using BF-algorithm. The expected cost, denoted by E[C], is the average number of comparisons and it is defined in §3.4.2. We observe that there are two sources of variability: the way the elements of B that can be found in A are arranged in B (the calculation above showed that this was the main difference between the maximal and minimal cost), and where each is located in A. If we wanted to calculate the probabilistic distribution of the cost, we would need to consider all the possible configurations of these factors (the problems involved in counting configurations belong to the branch of combinatorics and considered in the next chapter). So long as we only need the expected value, we can achieve it with far less effort. Let the number of elements of B that we search and find in A be called N (hence the total we check is N + 1), and the number of comparisons made with each such element from A be denoted by D. Using properties of the expectation (discussed in §3.4.2), we find that E[C] = E[N]E[D] + n. Then it is possible to show (Exercise 3.61 on page 106 provides the details) that E[N] is nearly 1, and equals m/(m + 2), and E[D] = (n + 1)/2 = (2m + 1)/2; hence, E[C] =

2m + 3 m 2m + 1 + 2m = 3m ∼ 3m, m+2 2 2(m + 2)

where the last symbol (∼) stands for an approximate value when m grows large.

You may find it disappointing that finding whether the m elements of one set are among the n terms of another may require an effort proportional to the product mn. . . . So we try and explore other ways to attack the problem. Method II – Sorting A Clearly linear (also called sequential) searching is an expensive activity. The binary search is much better, but it has an associated cost: the algorithm requires that the searched array, A here, be in sorted order. The binary search finds an element in an ordered array by locating the middle term of the list. If this term is not what it looks for, the array is split into two smaller sublists of approximately the same size and the process is repeated by applying the binary search to one of the sublists that contains the required element. This algorithm is used to look up a word in a dictionary or a name in a telephone directory. Let us assume that the array A is sorted, and we use binary search for each b ∈ B. In §5.4.2 we analyze this search method, provide code to implement it, and show that this search requires an almost deterministic number of comparisons, which is very close2 to lg n. Therefore, if B ⊂ A, the best—and because of the near determinacy also the worst or average—cost is 2 The

notation lg means logarithm to base 2. In [64, §8.1.2] it is shown that the only possible values are ⌊lg n⌋ and ⌈lg n⌉.


6

m lg n. Otherwise, when B is not contained in A, the lowest possible cost is lg n, when the first search fails. However, we must add the cost of sorting A, paying attention that both costs are expressed in the same cost units; here we use term comparisons. If we use the MergeSort algorithm (see §5.4.3 for code and analysis), the cost of sorting A is n lg n comparisons, and the total cost is m lg n + n lg n = (m + n) lg n. Notice that since n > m, most of the cost derives from the sorting, not the actual test! Since the total cost is far lower than the cost of Method I, you may feel the need to ask: if sorting A provides such a savings, shouldn’t we sort B as well?! Remembering that sorting A accounts for most of the cost of the last solution, and therefore we should be wary of making another “heavy” investment, and expect that it will improve the overall cost. Still, we should never disregard the potential of an entirely different approach, when the assumptions about the available data change. Method III – Total Sorting And a different method is indeed available! The same method that we use to merge two sorted arrays can be used to check that each b ∈ B is also in A. This is exactly the method used in the efficient merge sort mentioned above. A C-like program fragment for it (which assumes that both arrays are in increasing order, and that it is legal to refer to A[n] though it may contain an arbitrary value) follows: 1.

int TS(int ∗A, int n, int ∗B, int m) i = j = 0; //initialize running indices in A and B

2.

while ( j < m)

//until we tested all elements of B

3.

{while ((A[i] < B[ j])&&(i < n))i + +;

4.

if (A[i] == B[ j]) {i + +; j + +; }

5. 6. 7.

//discard small A elements //found a match

else return (”no”); } return(”yes”);

Now the minimal cost can be as small as two comparisons! (when A[0] > B[0]). If the relation B ⊆ A holds, the minimal cost is charged when all the elements of B are in the first m position of A. The cost is 2m comparisons. The maximal cost occurs when the last element of B is the last element in A as well, which increases the cost to n + m comparisons. The average cost requires somewhat stronger assumptions. If we assume that the elements from B are uniformly dispersed in the sorted array A, which is typical, then the cost is quite close to the maximum! This is very unusual: we shall see that expected costs are normally much closer to the lowest possible value than to the largest one. Exercise 1.6 [4] For two sorted arrays A and B (of sizes n and m, respectively, m 6 n), justify the pessimal claim that the average cost in the algorithm TS is close to the maximum. Assuming that the elements of B are all in A and dispersed uniformly along the array A, show n+1 comparisons to verify the that the merge-like procedure TS requires on average m 1 + m+1 inclusion B ⊆ A.


7

With the result of this exercise, and the value m lg m + n lg n for the expected cost of sorting both arrays, we find a total expected cost of m(1 + lg m) + n(m/(m + 1) + lg n) (plus a term that is close to 1, and hence small enough to neglect here; it equals m/(m + 1)). This cost is close to the cost of method II. Taking for comparison’s sake m ≈ n, we find the same leading term, 2n lg n, with an additional term of 2n here. Hence we have found that the extra sorting work, even when used in this way, does not pay. Method IV – Hashing If search time is excessive, and additional work-space is available, using hashing makes sense. A hash table is an effective dynamic structure that supports the dictionary operations—insert, delete, and search. We assume you are familiar with hashing with repeated probing, for example, and use standard results of its performance. The basic idea of hashing is to compute the slot in the table instead of using array index. There are known many approaches how to accomplish such evaluation, see, for instance, [32]; random hashing is discussed later in §4.1.3. We hash the set A (of size n) into a table T of size greater than n, say α n, where α > 1. Each entry requires on the average 1 + β hash calculations and comparisons. Both the increase of T over A, and the additional cost β represent the effects of collisions during the hashing calculations. Unless α is too small (a value of 1.5 is always adequate), β is a function of n and α , and is normally quite small—typically less than one—but we do not compute it here; see [32, Chapter 11] for a review of hashing tables. If n is large, even α = 1.05 is adequate. The total cost is nearly deterministic: The cost of hashing A equals n(1 + β ). The cost of finding (or missing) any element of B is 1 + β . Total cost when the relation B ⊆ A holds is then (m + n)(1+ β ). Unlike the previous methods, this is linear in the array sizes! The price we had to pay for this is the extra space (and the actual hashing time, which we did not include above, and which is essentially linear in n). Let us add a twist to the foregoing discussion. We observe that subset-checking operation is usually done within a larger calculation, and it may well be the case that either (or both) arrays were already sorted (or hashed) before the subset checking, or that subsequent steps in the calculation can benefit from either sorting the arrays or hashing. In any case, the cost of the sorting or hashing need not all be charged to the subset-checking operation. All of this, of course, does not change the above analyses, but changes the cost-accounting we need to do. Exercise 1.7 [4] In this exercise, you are asked to consider other data processing problems, of similar nature, that could benefit from sorting or hashing. In each case, design one or more algorithms, with different assumptions about the pre-processing, discuss its costs, at the level of detail we do above. In particular, find the extreme values of the cost, as above. Here are several possible problems, the first two use a single array, others use more. Do those and try to find additional problems. Here we no longer assume that the elements are distinct. (a) Test whether the elements in an array A of size n are all distinct. (b) This is similar to part (a), but needs more work: find how many distinct elements are in such an array. (c) With arrays A and B of sizes n and m, respectively, test whether they are identical (i.e.,


8

m = n, Ai = Bi , 1 6 i 6 n). (d) As in part (c), but you need to test whether they contain exactly the same elements (in any order, and in any number of repetitions). The arrays B = ha, bi and A = hb, a, b, bi would satisfy the test you are asked to perform here. (e) For arrays as in part (c), test that they are distinct: no element appears in both A and B. Exercise 1.8 [2] This problem asks you to count (numerically) the points with integer coordinates (x, y) that are in or on the ellipse (x/a)2 + (y/b)2 6 1. Assuming a and b, which are not necessarily integers, are relatively large, we expect the number of such points to be close to the area of the ellipse, π ab. How would you do the counting efficiently? In particular, how would you verify that numerical errors do not lead you to miscount points that are on, or very close to the ellipse itself? Hint: The ellipse encloses a convex area in the plane.

1.1.3 Matrix Multiplication Consider the product of two 2 × 2 matrices A and B to form a third, C = A B: ! ! ! c11 c12 b11 b12 a11 a12 . , C= , B= A= c21 c22 b21 b22 a21 a22 The usual definition of matrix multiplication (for n × n matrices) is n

ci j =

∑ aik bk j ,

1 6 i, j 6 n.

(1.2)

k=1

Therefore, for n = 2 we have c11 = a11 b11 + a12 b21 ,

c12 = a11 b12 + a12 b22 ,

c21 = a21 b11 + a22 b21 ,

c22 = a21 b12 + a22 b22 .

(1.3)

We count operations, and it is natural here to consider the arithmetic operations. Let M stand for one Multiplication and A for Addition, then the total number of operations needed to compute C is 8M + 4A. In 1969, Volker Strassen [127] invented the following algorithm (which is not easy to implement because it lacks numerical stability). He defined the following 7 quantities in terms of the elements of A and B:

m1 = (a21 + a22 − a11 )(b22 − b12 + b11 ), m2 = a11 b11 ,

m3 = a12 b21 , m4 = (a11 − a21 )(b22 − b12 ),

m5 = (a21 + a22 )(b12 − b11 ),

m6 = (a12 − a21 + a11 − a22 )b22 ,

(1.4)

m7 = a22 (b11 + b22 − b12 − b21 ).

Using these quantities, the above matrix C can be written as C = AB =

m2 + m3 m1 + m2 + m5 + m6 m1 + m2 + m4 − m7 m1 + m2 + m4 + m5

!

.

(1.5)


9

How many operations are needed here? The adjoined MAPLE program, besides providing a proof that the assignments in (1.5) are correct—since the last four lines there evaluate to zero—also shows how to compute (1.5) so that the total number of operations is 7M + 15A. # Maple demonstrates correctness of the Strassen algorithm t1 := a21 − a11; t2 := b22 − b12; t3 := t1 + a22; t4 := t2 + b11; m1 := t3 ∗ t4; m2 := a11 ∗ b11; m3 := a12 ∗ b21; m4 := −t1 ∗ t2; m5 := (a21 + a22) ∗ (b12 − b11); m6 := (a12 − t3) ∗ b22; m7 := a22 ∗ (t4 − b21); t5 := m1 + m2; t6 := t5 + m4; c11 := m3 + m2; c12 := t5 + m5 + m6; c21 := t6 − m7; c22 := t6 + m5; simplify(expand(c11 − a11 ∗ b11 − a12 ∗ b21)); simplify(expand(c12 − a11 ∗ b12 − a12 ∗ b22)); simplify(expand(c21 − a21 ∗ b11 − a22 ∗ b21)); simplify(expand(c22 − a21 ∗ b12 − a22 ∗ b22)); This result may appear anticlimactic: the Strassen Algorithm reduces the number of multiplications from 8 to 7, and nearly quadruples the number of additions! Remembering that in floating-point calculations (with ordinary precision) additions tend to run longer than multiplications only adds insult to injury. However, let us look what can be done when n > 2, especially for large matrices. Let us consider multiplication of two n × n matrices. We partition each matrix into four n/2 × n/2 blocks, which can be manipulated just as in Eq. (1.3), but we can also use them in Eqs. (1.4–1.5). Let T (n) be the number of operations (on numbers, not matrices) when multiplying two n × n matrices using Strassen’s algorithm. We lump A and M into “operation.” Multiplying two n/2 × n/2 blocks requires T (n/2) operations, but adding them only n2 /4. So we get the recurrence (assuming that once the algorithm gets down to a 2 × 2 matrix it does the sensible thing) T (n) = 7 T

n 2

+ 15

n2 4

T (2) = 12.

(1.6)

Note that using Eq. (1.2) can also be organized in a similar way, leading to the relation C(n) = 8C (n/2) + 4(n2 /4);

C(2) = 12.

(1.7)

In Chapter 5, we consider the solution of such recurrence equations. Here we merely state that the leading term in the results turns out to be C(n) ≈ n3 , and T (n) ≈ nlg 7 = n2.807... . The leading term is determined by the number of multiplications (as lg 8 = 3)! Surely, more additions, but careful programming produced implementations where already for n = 40 it is worthwhile to use the Strassen algorithm. For instance, if n = 64, then the difference between the two algorithms, counted in number of operations given by the leading term only is n3 − nlg 7 = 262, 144 − 117, 649 ≈ 642 × 35.28. This more than compensates for the increased number of additions. Exercise 1.9 [3] This exercise asks you to compare the cost of the Strassen algorithm (T (n) operations for matrices of size n), with the costs of multiplying matrices using Eq. (1.2):


10

count the A and M operations and add them together. Iterate Eq. (1.6) for n which are powers of 2, starting with the value we saw in Eq. (1.2): T (2) = 8M + 4A = 12. Where do the costs “cross” (you will need to estimate, due to the sparsity of the set for which you can compute the T (n) values)? The algorithm having asymptotic complexity of O(n2.376 ) was presented by Don Coppersmith and Shmuel Winograd [31] in 1990. However, the multiple of the leading term in the asymptotic expansion is so large that the algorithm becomes beneficial only for huge matrices. Winograd [140] showed that by reorganizing the computation, the number of multiplications can be reduced approximately by half with the number of addition operations slightly increased. Such a technique could be computationally advantageous when matrix multiplication is performed on parallel processors. His idea is based on vector-list representation of the product of two matrices that can be thought of as a dot product of a column-list vectors and a row-list vectors. As an example, consider two vectors of length 4: u = hu1 , u1 , u3 , u4 i and v = hv1 , v1 , v3 , v4 i. Their dot product, u · v = u1 v1 + u2 v2 + u3 v3 + u4 v4 , can be factored into the following expression: u · v = (u1 + v2 ) (u2 + v1 ) + (u3 + v4 ) (u4 + v3 ) − u1 u2 − u3 u4 − v1 v2 − v3 v4 . In general, Winograd’s algorithm defines the multiplication, C = A B, of two n × n matrices as follows: n/2

ci, j =

∑ (ai,2k+1 + b2k, j ) (ai,2k + b2k+1, j ) −V j −Ui,

(1.8)

k=1

where n/2

Ui =

∑ (ai,2k × ai,2k+1 ),

k=1

n/2

Vj =

∑ (b2k, j × b2k+1, j ).

k=1

What might not be obvious is that the values Ui and V j (i, j = 1, 2, . . . , n) can be preprocessed and stored for each row of the matrix A and for each column of the matrix B. Such formulation requires 0.5 n3 + n2 multiplications and n3 + 1.5 n2 − 2n additions.

1.2 Proofs We expect you know the importance of proofs: they tell you why theorems “work,” or under which conditions they do not! The way we formulate our proofs depends not only on the immediate need (to prove a claim), but is also shaped by tradition. Different areas of mathematics have developed their proof traditions. Algorithmic engineers like to think of themselves as practical-minded and application-oriented. This can color one’s personal taste in proofs, as evidenced by the opening sentence of this paragraph. We briefly mention a few kinds of proofs and dwell in some detail on two that are very common in our work, both for proving correctness and for our quantitative needs. We rarely need proofs of the kind you have encountered in geometry: working from a set of axioms towards an ever increasing hierarchy of theorems. Our proofs tend to provide quantitative by-products. We prove a solution of an equation by substituting the solution in

1.2. PROOFS

11

the equation, and showing that it is satisfied. We prove claims about growth rates of functions by computing asymptotic representations. We have several techniques to prove claims about sets of strings: one way is to construct a grammar that generates the set and create a proof using properties of the grammar. Sometimes you need a separate proof to show that the grammar is correct: that it generates the set. Another way to reason about strings is to design a finite-state automaton that accepts the set of strings and to argue from known properties of such automaton (a combinatorial way of doing this is shown in Chapter 9). Here is an example of a feasibility proof: we prove that some action can be carried out on all elements of an infinite set, by showing it can be done on a well-selected finite subset. This is a variety of ‘proof by inspection of cases.’ Example 1.10 Show that every natural number larger that 43 can be formed by a linear combination of the integers 6, 9, and 20, with coefficients that are non-negative integers. What happens if we remove the requirement of non-negativity? Let us first show the original claim. While there are many ways to combine the three given numbers to obtain many of the results (36 = 4 × 9 = 6 × 6), once we have shown how to obtain a sequence of six successive integers we know we can reach any number beyond the lowest number in the sequence. And while there seems to be no way to produce 43 in this way, we have the following: 44 = 20+ 4·6;

45 = 5·9;

46 = 2·20+ 6;

47 = 20+ 3·9;

48 = 8·6;

49 = 2·20+ 9.

Allowing the coefficients to be negative allows us, naturally, to obtain every integer, since now we can step back from the above sequence by subtracting the needed number of multiples of 6; in particular, 43 = 2 · 20 + 9 − 6. Let us introduce some logic notation. Our knowledge of a scenario and its properties is denoted by the symbol S; we want to prove a claim C about this scenario. The negations of these statements are denoted by ¬S and ¬C (some authors use the notation S and C).

Some claims can be proved directly: S ⇒ C. What if we cannot find such a proof? A type of proof that often saves the day is proof by contradiction. It is based on the logical equivalence of S ⇒ C and ¬C ⇒ ¬S. If we cannot prove the first, we may try to show the second: we begin by assuming ¬C: that is, C is false. If we can then show that this assumption leads to ¬S, to a conclusion which is incompatible with the given S, the above equivalence means that we achieved the desired proof. Here is a classic example. Example 1.11 Prove that the square root of 2 is not a rational number.

Solution: We know that t is a rational number if and √ only if t = a/b, where a and b are two integers. Now we assume the claim is false, that 2 is rational. Hence two integers can be found as above. √ We find them, cancel all common factors, call what we are left with p and q, and they satisfy 2 = p/q, with p and q having no more common factors. Squaring this relation leads to 2q2 = p2 . Therefore p2 is even, which implies that p is even because if an integer has an even square, that integer must be even, since the square of an odd integer is odd. So there is some integer n such that p = 2n, and therefore now 2q2 = 4n2 , or q2 = 2n2 , and the same argument leads us to see that q is also even! Hence both p and q are even and have a common factor 2, contradicting the claim that √ all common factors were canceled. Therefore we cannot find such a pair of integers, and 2 is irrational.


12

1.2.1 Proof by Counting It is the nature of our analyses that they very often amount to counting. How many elements are in a structure? How many edges are in a graph? Many of the tools we discuss throughout the course, primarily generating functions (introduced in Chapter 6), are designed to answer these questions. “How to count” is the basic need we answer. Here we want to mention counting by comparison. That is: studying a finite set X , we may wish to compare its unknown size, |X |, with that of a known one, |A|, where size is the number of elements in a set. This may be an appropriate time to introduce notation for the sets of integers we will be dealing with throughout the text: Z

The set of all integers.

+

The set of natural numbers, {1, 2, 3, 4, . . .}.

Z

N Ik [1..n]

The set of nonnegative integers, {0, 1, 2, 3, 4, . . .}.

The set of natural numbers > k, {k, k + 1, k + 2, . . .}.

The set of the first n natural numbers, {1, 2, . . . , n}.

Definition 1.12: (Set size equality) Two sets A and B are said to have the same size, |A| = |B|, if there exists a bijection (which is one-to-one and onto mapping) between the elements of the two sets. ⊳ This definition sits well with our intuitive notions of equal size for finite sets. On the other hand, it leads to scenarios that may offend your intuition (and sense of order) with infinite sets: e.g., the size of the set of all nonnegative integers N is equal to the size of the set of all nonnegative even integers, since the mapping i ↔ 2i performs the required task. It is common to reserve the term size to finite sets and refer to the equivalent concept for all sets as cardinality. Later in Chapters 6 and 7 we will use generating functions as enumerators. Then we will give an equivalent, in fact richer, formulation: two sets have the same size structure if their enumerators coincide. The definition also points the way to prove inequality of sizes. The key claim is known as the Pigeonhole Principle: you cannot house more pigeons (that are particular about privacy) than you have bins in your dovecote: Theorem 1.13 (The Pigeonhole Principle) Let A and B be two finite sets such that the cardinality of A is bigger than the cardinality of B (|A| > |B|), then there is no one-to-one mapping between the two sets. Let f : A −→ B be a mapping from A to B, then the inverse image of some element of B under f contains at least |A|/|B| elements. Exercise 1.33 asks you to prove this principle. Applications of this principle crop up often; the following claim is a classical example, but we first recall some definitions. A graph G = (V, E) is a collection of vertices, V , or nodes and edges, E, that connect pairs of vertices. A graph may be undirected, meaning that there is no distinction between the two vertices associated with each edge, or its edges may be directed from one vertex to another. A path of length n from vertex u to vertex v in a graph G = (V, E) is a sequence hx0 , x1 , . . . , xn i of nodes such that x0 = u, xn = v, and (x j−1 , x j ) ∈ E for every j = 1, 2, . . . , n. A closed (simple)

1.2. PROOFS

13

path, with no other repeated nodes than the starting and ending vertices is commonly called a simple cycle. Claim 1.14 Let G be a directed graph. The size of G, denoted by |G|, is the number of nodes in the graph, and it is assumed to be finite. If there exists a path of arbitrarily large length, then the graph has a cycle. P ROOF : Since we have paths of any size, let us pick one path hv1 , v2 . . . , vn i of size n > |G|. Now consider a mapping induced by this path: f : [1..n] −→ G such that f (i) = vi . By Theorem 1.13 this mapping is not one-to-one, so there are some i and j such that i < j and vi = v j , hence this node is on a cycle. You may remember from your studies of regular, or context-free languages, the related Pumping Lemmas [65]; they are direct results of this claim. Another famous result gives the following statement. Claim 1.15 (Erdös and Szekeres) For some n ∈ Z+ , let S be a set of n + 1 different numbers, selected from {1, 2, 3, . . . , 2n}. Then two elements can be found in S such that one divides the other. P ROOF : Any positive integer is equal to some 2i q with i > 0, where q is a positive odd integer. We express each element of S in this form, to obtain the set {q1 , q2 , . . . , qn , qn+1 }. This set contains n + 1 positive odd numbers less than 2n. Since there are only n such distinct numbers, the Pigeonhole Principle promises us a pair of indices i 6= j such that qi = q j = q. Thus there are two distinct integers in S of the form a = 2r q and b = 2s q, say. The one with the smaller power of two divides the other. We present two somewhat stronger forms of the Pigeonhole Principle, this time using real birds. Theorem 1.16 (a) If m pigeons are placed in n pigeonholes, when m > n, then some pigeon hole contains at least mn pigeons. (b) If m = m1 + m2 + m3 + · · · + mn − n + 1 pigeons are placed in n pigeonholes, where mk ∈ Z+ for 1 6 k 6 n, then there is an i, such that pigeonhole #i contains at least mi pigeons.

Exercise 1.17 [3] Prove both parts of Theorem 1.16. Suggestion: Use the method of proof by contradiction, twice.

The following exercises are all direct results of Theorem 1.16. Most require proof by contradiction. Exercise 1.18 [2] (a) Given a group of people that consists of n married couples, how many of them do we need to assemble, to be certain we have at least one of the couples? (b) Show that for any set of 20 people, there is at least one day of the week in which at least three of them were born. What is the smallest group size for which this claim can be made? (c) There are 43 different time periods in a week during which classes at a university can be scheduled. There are 2117 different class-meetings in a week. What is the minimum number of classrooms needed? (d) A box has 10 white balls, 12 black and 14 red balls. How many balls do we need to draw (without choosing) to have at least 4 white or 5 black or 6 red balls? Suggestion: Let colors play the role of the pigeonholes.


14

Exercise 1.19 [4] (Erdös and Szekeres) Every sequence A = ha1 , . . . , an2 +1 i of n2 + 1 distinct integers has a strictly increasing, or strictly decreasing, subsequence of size at least n + 1. Note: The subsequence need not be consecutive. For example, the sequence h3, −2, 7, 1, 5i has the increasing subsequence h−2, 1, 5i. The expected length of the longest monotonic √ subsequence in a random permutation of [1..n] has been shown to be close to 2 n. Hint: Create the n2 + 1 values si > 1, for 1 6 i 6 n2 + 1, where si is the length of the longest increasing subsequence you can find in A with ai as its first element. For the above sequence, the si are 2, 3, 1, 2, 1. Exercise 1.20 [2+] Let k1 , k2 , . . . , kn be positive numbers (sign and repeated values do not matter). Prove that there necessarily exist two indices i and j (1 6 i < j 6 n) such that the sum ki + ki+1 + . . . + k j is divisible by n. Exercise 1.21 [2] (a) A computer network consists of ten computers. Each computer is directly connected to at least one of the other computers. Show that there are at least two computers in the network that have equal connectivity (= are directly connected to the same number of other computers). (b) A computer network consists of ten computers. Each computer is directly connected to zero or more of the other computers. Show that there are at least two computers in the network that have equal connectivity. (c) What is the significance of the number 10 in the previous parts? Exercise 1.22 [3] Show that the decimal expansion of any ratio of two integers consists ultimately of the repetition of a block of digits. Examples: if the ratio is an integer, the repetition is of a 0; also 6/5=1.2(0), 1/3 equals 0.(3), and 3/130=0.0(230769). Hint: The maximal length of the repeated block is bounded by the divisor. Exercise 1.23 [5] Given an integer k > 2, define rk as the smallest number of people we need to assemble to be certain that among them there is a set of k people where either everyone knows all the others, or a set of k people none of which knows any of the others. Clearly r2 = 2; what is r3 ? (Finding r4 or higher is much harder). Remark: The assumption is that the relation “knowing a person” is symmetric: if Tom knows Dick, then Dick knows Tom. However, it is not transitive. With this question, we are touching on a domain in combinatorics called Ramsey3 theory. The value of rk is only known for very few (small) values of k.

1.2.2 Induction This is the most useful proof method in our work. We use it here mostly to prove formulas, such as sums and bounds, but it also has other applications, such as proving properties of structures and correctness of algorithms. The method of mathematical induction allows us to prove that a claim P(n) holds for all integers n > n0 by performing two steps: (a) [Basis] Show P(n0 ) is true; it may be necessary to show that P(·) is satisfied for additional values, and 3 Frank P.

Ramsey (1903 – 1930) was a British mathematician who has stimulated an enormous study in graph theory. His 1930 celebrated paper established a growing branch of combinatorics, now called Ramsey theory.

1.2. PROOFS

15

(b) [Inductive step] Show that the truth of P( j) for n0 6 j < n implies P(n) is true. Part (a) is sometimes called the initial step. The condition in part (b) is called the induction hypothesis, and the implication there is the induction step. The qualification in part (a) refers to the size of the basis, which is the number of entries for which we need to prove P(·) directly. This number depends on the induction proof. More precisely: it equals the number of terms explicitly involved in that proof (see Example 1.29 below)4 . This number is almost always one or two, but there are exceptions. We demonstrate the method by a few proofs. Example 1.24 Show the basic formula for summing an arithmetic series: n

P(n) :

Sn = ∑ i = i=1

n(n + 1) . 2

(1.9)

A single basis suffices here (as is usually the case in showing summations): S1 = ∑1i=1 i = 1(1+1) = 1. 2 The induction hypothesis is that Eq. (1.9) holds for all k, 1 6 k < n. To carry out the induction step we need to prove P(n) as given in Eq. (1.9): n−1

Sn =

∑i

+

n

separate the last term,

i=1

(n − 1)n +n 2 n(n + 1) = 2 =

use the induction hypothesis for k = n − 1, bring to common denominator and combine,

which is the value we wanted to show for Sn . We are done, except one parting shot: when proving by induction, it is useful to test your induction step proof on the values which immediately follow the basis you proved. In our case, this implies proving the claim for n = 2: mimicking the general proof, we take S1 , which we know to be 1, add 2, find 3, and substitute = 3. Now we are done. Why do we need to second-guess our proof? in Eq. (1.9), 2(2+1) 2 There are two reasons. It is a check that we have provided a sufficient basis; and the second reason is—we shall see examples below—that low values of the induction variable may introduce special conditions (such as avoiding division by zero) that do not afflict the general step. Exercise 1.25 [1] Prove by mathematical induction the formula for the sum of the first n odd numbers: 1 + 3 + 5 + · · · + (2n − 1) = n2 for any positive integer n. For our next proof we introduce a very common sequence of integers, ubiquitous in many areas of mathematics, and computer science is no exception: the Fibonacci numbers. In Appendix D we detail some of their properties. A great source of information about the Fibonacci sequence and related numbers is the Fibonacci Quarterly published by the Fibonacci Association and the book [81] by Thomas Koshy. Here they only appear as grist to our inductive proof mill. 4 Some authors make a fine distinction between an induction step which uses only P(n − 1) to prove P(n)— calling this mathematical induction—and the condition as given above, where all P( j), n0 6 j < n are needed to prove P(n), calling this the strong mathematical induction.


16

The Fibonacci numbers are denoted by Fn , n > 0, and are defined by the following relation: Fn = Fn−1 + Fn−2 , n > 2.

F0 = 0,

F1 = 1.

(1.10)

The recurrence (1.10) generates an infinite sequence, which starts with F0 = 0, F1 = 1, and other values are calculated from Eq. (1.10): F2 = 1, F3 = 2, F4 = 3, F5 = 5, F6 = 8, F7 = 13, F8 = 21, F9 = 34, F10 = 55, · · · . We extend the definition in Exercise 5.19. These numbers satisfy a bewildering variety of properties and identities, two of which we show here. Example 1.26 The Cassini Identity Fn−1 Fn+1 − Fn2 = (−1)n ,

n > 1.

(1.11)

P ROOF : Use n = 1, 2 as a double basis: F0 F2 − F12 = −1, and F1 F3 − F22 = 1 · 2 − 12 = 1, as needs be. The hypothesis is that (1.11) holds up to some n > 1 and the induction step for n + 1 requires that we consider ?

2 = (−1)n+1 . Fn Fn+2 − Fn+1

The left-hand side equals, using the definition (1.10): 2 2 LHS = Fn (Fn + Fn+1 ) − Fn+1 = Fn2 + Fn Fn+1 − Fn+1 .

(*)

Since Fn2 = Fn−1 Fn+1 − (−1)n (where we have used the hypothesis), we find LHS = Fn+1 (Fn−1 + Fn − Fn+1 ) − (−1)n = (−1)n+1 ,

(**)

which is the desired value. If you examine the proof you will observe the need for a double basis since the relations marked by (*) and (**) use the induction hypothesis at two different values of the index. Exercise 1.27 [4] Prove by induction Fn+k Fn−k − Fn2 = (−1)n+k+1 Fk2 (n > k).

Exercise 1.28 [2] For the sequence of the Fibonacci numbers, Fn (n = 0, 1, 2, . . .), defined in Eq. (1.10), prove by induction that (I + E)n Fj = Fj+2n , where E and I are the shift and identity operators, defined by E f ( j) = f ( j + 1) and I f ( j) = f ( j), for arbitrary function f (). Example 1.29 Let us use induction to prove another identity satisfied by the Fibonacci numbers: Fn+m = Fm Fn+1 + Fm−1 Fn m > 1, n > 0. (1.12) There are two integer parameters in this relation. Frequently it is possible to proceed in such situations by doing an induction proof on one parameter only, for arbitrary values of the other. This is how we work here: induction on n (for arbitrary m). In addition, as the step proof below shows, we need a double basis, that is, we need to show (1.12) directly for two successive values of the parameter n. P ROOF : We use the given values of F0 , F1 and F2 = 0 + 1 = 1: n = 0 : Fm = Fm F1 + Fm−1 F0 = Fm , n = 1 : Fm+1 = Fm F2 + Fm−1 F1 = Fm+1 ,

1.2. PROOFS

17

where Eq. (1.10) has been used in the bottom line. Hypothesis: the claim (1.12) holds up to some n > 1 (for any m). ? Inductive Step: we need to prove that Fn+1+m = Fm Fn+2 + Fm−1 Fn+1 . The left-hand term is expanded from the defining recurrence as Fm+(n−1) + Fn+m . Each of these two numbers is expanded, according to Eq. (1.12), using the hypothesis (this is why a double basis is needed here; do you see that we use it?) and we find that they equal (Fm Fn + Fm−1 Fn−1 ) + (Fm Fn+1 + Fm−1 Fn ). Now we separate this into two parts, the terms that accompany Fm and those with Fm−1 . We use the recurrence (1.10) for each of these groups to combine them and get Fm Fn+2 + Fm−1 Fn+1 , the desired right-hand side above. Exercise 1.30 [3] Show that the same proof of the last induction step can be used to show that Fn+m = Fm Fn+1 + Fm+1 Fn m > 0, n > 0, Error! Error! (1.13) a relation that cannot be consistent with the recurrence (1.12), which we have just proved. Exercise 1.31 [3] Use mathematical induction to show the following claims: (a) Prove the paired statements: (i) (b) (c) (d) (e)

2 F2n+1 = Fn2 + Fn+1 ,

F2n = 2Fn Fn+1 − Fn2 ,

(ii)

n > 0.

2 −F F n+1 , n > 1. Fn2 − Fn−1 n n−1 = (−1) 2 2 F2n = Fn+1 − Fn−1 , n > 1; the Hoggatt identity. F2n−1 = Fn Fn+1 − Fn−1 Fn−2 , n > 2. Show that Fn is even if and only if n is a multiple of 3.

Exercise 1.32 [3] Show the following sums by induction: n

(a)

∑ k=1 n

(c)

∑ F2k = F2n+1 − 1,

k=1 n

(e)

n

Fk = Fn+2 − 1,

∑ k=1

Fn+2 Fk−1 = 1− n , k 2 2

n > 0.

∑ F2k−1 = F2n,

(b)

n > 0.

k=1 n

n > 1.

(d)

∑ Fk2 = Fn Fn+1,

n > 0.

k=1 n

n > 0.

(f)

∑ k Fk = (n − 2)Fn+1 + (n − 1)Fn + 2.

k=1

Exercise 1.33 [2] Most proofs of the pigeonhole principle proceed by contradiction, but we ask you to show Theorem 1.13 (the Pigeonhole Principle) using induction. (Choose the size of the smaller set as your induction variable.) Exercise 1.34 [3] Show that for any set A and any partial order R on it, a total order T exists that agrees with R. (You may be familiar with topological sort, used to compute such T , but here you are only required to prove that it exists, by mathematical induction.) Hint: A simple proof is possible if you first show that every partial order R has an R-minimal element. An element r is R-minimal if there is no element a 6= r in A such that a R r. Show this claim twice: once by contradiction, and once by mathematical induction. Exercise 1.35 [3] For a binary tree with f leaves, which are in depths d1 through d f , prove that ∑ fj=1 2−d j 6 1. The simplest way seems to be by induction on the height of the tree. Reminder: The depth of a node in a tree is the number of arcs to it from the root; hence the depth of the root is zero. The height of the tree is the maximal depth of a node in it.


18

Exercise 1.36 [3] Consider free labeled trees of n nodes {vi }ni=1 , where the node vi has degree—the number of edges incident on it—equal to di . (a) Show that ∑ni=1 di = 2n − 2. (n−2)! . (b) Show that the number of such different trees over the n nodes is (d1 −1)!···(d n −1)! Hint: Do both parts by induction. For part (b), experiment with trees of 3, 4 and 5 nodes, to see how trees may differ. Note, for example, that for n = 5 and degree sequence 1, 2, 1, 3, 1 (the order of these integers does not matter because any node can be a root), the only possible shape is and the labels 2 and 4 are necessarily fixed on the nodes with degrees 2 and 3; the labels of the leaves may vary, but once the one connected to node 2 is named, the rest are determined. For the induction step, use the fact that there must be a node with degree 1, and “remove” it. The concept of mean, or average, is central to many calculations we do. Fittingly, over time, several different definitions for the mean have evolved. Let s and l be two values. Their arithmetic mean is (s + l)/2. It is intermediate between the two, or strikes a balance between them. In general, the arithmetic mean of any set of numbers is their sum divided by the number of elements. The ancient Greeks, who preferred to balance quantities based √on their ratios (rather than differences), defined the geometric mean of the above s and l as sl. This equals the side of a square with area equal to the rectangle with sides l and s. In general, the geometric mean of a set of n numbers is the nth root of their product. The Hill-Huntington rounding procedure rounds p a number q to ⌊q⌋ or ⌈q⌉, based on the geometric mean of those two integers, q∗ = ⌊q⌋⌈q⌉: it chooses the floor if q < q∗ and the ceiling otherwise. This formula has been used to apportion seats in the House of Representatives since 1940. Eq. (1.14) suggests a possible explanation for the change. The geometric mean has been long used in the government calculations. Since 1999, the Bureau of Labor Statistics has been using the geometric mean to determine averages of prices needed to calculate the Consumer Price Index (CPI). Exercise 1.37 [2] Prove the following claim by induction on p, for arbitrary q. Every fraction p/q ∈ (0, 1), with p and q natural numbers can be written as the sum qp = n11 + n12 + · · · + n1m where the ni are natural numbers and ni < ni+1 , for 1 6 i < m. This representation derives from an Egyptian document, the Rhind Papyrus, approximately 4000 years old, and is therefore known as the normal Egyptian fraction representation. Example 1.38 The following unusual proof by induction was apparently developed by Edwin Beckenbach and Richard Bellman. Our presentation is adapted from [56]. We want to prove n the geometric mean is less than or that for any finite sequence of positive numbers {xk }k=1 equal to its arithmetic mean, (x1 x2 · · · xn )1/n 6

x1 + x2 + · · · xn . n

(1.14)

This is a special case of a more complete treatment given in Exercise 10.30 (page 558). P ROOF : To prove this statement we twist around the mathematical induction method: we prove first an infinite basis (using mathematical induction, naturally); then we take care of the rest. The claim is empty for n = 1. For n = 2 we have to prove ?

(x1 x2 )1/2 6

x1 + x2 . 2

1.2. PROOFS

19

Raising to the second power and developing we obtain ?

x1 x2 6

(x1 + x2 )2 4

?

or 4 x1 x2 6 (x1 + x2 )2

=⇒

?

0 6 x21 + x22 − 2 x1 x2 = (x1 − x2 )2 ,

which is manifestly positive for any real numbers x1 , x2 (and since they are all positive we know no sign was missed or mishandled). The standard procedure requires us to assume that the statement is true up to n and prove it for n + 1. We apply an attractive—and here, more effective—method because the traditional method leads to a morass of calculations. Therefore, we begin by constructing an “infinite basis” by showing first that it holds for n = 1, 2; now assuming the statement true for n, we prove it for 2n, which shows that the claim is true for all integers that are powers of 2. We have to establish the inequality ?

(x1 x2 · · · x2n )1/2n 6

x1 + x2 + · · · + x2n . 2n

This relation can be rewritten as ?

(x1 x2 · · · xn )1/2n · (xn+1 xn+2 · · · x2n )1/2n 6

x1 + x2 + · · · + xn xn+1 + xn+2 + · · · + x2n + . 2n 2n

According to the induction assumption we have (x1 x2 · · · xn )1/n 6

x1 + x2 + · · · + xn ≡ A1 , n

likewise (xn+1 xn+2 · · · x2n )1/n 6

xn+1 + xn+2 + · · · + x2n ≡ A2 , n

where we denote by A1 and A2 the arithmetic means of {x1 , x2 , . . . , xn } and {xn+1 , xn+2 , . . . , x2n }, respectively. Multiplying both relations and using the fact that for any positive numbers a, b, c, d such that a 6 b and c 6 d, it follows that ac 6 bd, we get (x1 x2 · · · x2n )1/n 6

x1 + x2 + · · · + xn xn+1 + xn+2 + · · · + x2n · = A1 A2 . n n

(1.15)

Raising (1.15) to the power n yields x1 x2 · · · x2n 6 (A1 A2 )n . Since our statement has been proved for n = 2, we have A1 A2 6 to the previous relation we find x1 x2 · · · x2n 6

A1 + A2 2

2n

(A1 + A2 )2 . Applying this 4

.

The induction step is done, and we proved the statement (1.14) for n being any power of 2.


20

Now we proceed to cover the “holes.” Assuming that (1.14) holds for5 n, we show that the statement is valid for n − 1. Hence we assume that the inequality x1 + x2 + · · · + xn n x1 · x2 · · · · · xn 6 n is true for any n-sized sequence of positive numbers x1 , x2 , . . . , xn ; our specific needs are met when we set xn to be the arithmetic mean of the first n − 1 terms (do you see why choosing a specific xn does not jeopardize the argument?). That is, let xn = and then

x1 + x2 + · · · + xn−1 , n−1

(1.16)

! +···+xn−1 n x1 + x2 + · · · + xn−1 + x1 +x2n−1 n !#n 1 1 + n−1 = (x1 + x2 + · · · + xn−1 ) · n n x1 + x2 + · · · + xn−1 n 1 = . = (x1 + x2 + · · · + xn−1 )n n−1 n−1

x1 + x2 + · · · + xn−1 6 x1 · x2 · · · · · xn−1 · n−1 "

Reminding ourselves of the left-hand side, x1 + x2 + · · · + xn−1 6 x1 x2 · · · xn−1 · n−1 we find

x1 + x2 + · · · + xn−1 n−1

n

,

x1 + x2 + · · · + xn−1 n−1 . x1 x2 · · · xn−1 6 n−1 Exercise 1.39 [3] Using the claim of Example 1.38, prove that the harmonic mean is less or equal to the geometric mean, that is, 1 1 −1 1 6 (x1 x2 · · · xn )1/n , xi > 0. + + ···+ n x1 x2 xn

def

Hint: rewrite the required inequality in terms of Vi =

x1 x2 ···xn . xi

Exercise 1.40 [3] Induction is used to prove non-algebraic claims too: A type of graph called tournament is obtained when each edge of a complete graph is endowed with a direction (each arc can be seen as representing a match between the two “players” at the nodes). A Hamiltonian path of a directed graph is a list of all the nodes, once, in some order such that there is an arc from each node in the list to the one following it. (a) Show that each tournament graph has a Hamiltonian path. Note: Exercise 3.82 on page 112 considers the possibility that a tournament has a Hamiltonian cycle, which happens when the path which you are asked to prove always exists, has an edge from its last node to the first. (b) A dominant node in a tournament is one from which each node is reachable by a path of length at most two. Prove that every tournament has a dominant node. 5 It

does not matter if n is a power of 2 or not.

1.2. PROOFS

21

Exercise 1.41 [3] This exercise deals with the set of natural numbers Z+ and primes. Reminder: a prime is a natural number p > 1 that is divisible by no positive integer other than 1 and itself. (a) Prove by induction that every non-prime natural number > 1 equals a product of prime numbers. (b) Prove by induction that every non-prime natural number can be obtain by multiplying a unique set of prime numbers. For this you need part (a) and the fact that if a prime divides a product of positive integers, it divides one of the integers. Exercise 1.42 [3] We have two piles of cards, one on the left (L) and one on the right, (R), which has fewer cards than L. On each card there is one or more distinct names. Names are not repeated on a card, but the same name can be carried on any number of cards. The operation switch is defined for a name, and consists in moving each card that carries the name to the other pile. Prove that it is possible to make pile R larger than pile L by picking a list of names, and performing successively switch operations on the names in the list. Exercise 1.43 [2] Prove by induction that for all natural numbers n > 5 the inequality 2n > n2 holds. Example 1.44 This example asks about proving an inequality by induction, but reveals an interesting property of the method. Let {xn }n>0 be the sequence determined by the recurrence xn+1 = 2x2n − 2, for n > 0, with x0 = 2 initiating the sequence. There is no simple closed expression for the elements of this sequence. When we do not know how to express cost functions simply, we try to settle for bounds. Observing the first few values of the sequence {xn }n>0 , as seen in the table, suggests that we n compare it with the values ln = 22 . These appear lower. Can we prove ln 6 xn for all n > 0? Clearly we have a basis: l0 = 2 = x0 . Assume the relation holds for all indices below n. In n

?

order to show the induction step we need to prove ln = 22 6 xn . Then, by assumption n n−1 2 n n n xn = 2x2n−1 − 2 > 2 22 − 2 = 2 · 22 − 2 = 22 + 22 − 2 > 22 = ln ,

and hence the required bound holds for all n > 1. l x a

1 2 3 4 5 6 7 8 4 16 256 6.554 × 104 4.295 × 109 1.845 × 1019 3.403 × 1038 1.158 × 1077 6 70 9798 1.920 × 108 7.373 × 1016 1.087 × 1034 2.364 × 1068 1.118 × 10137 8 128 32768 2.147 × 109 9.223 × 1018 1.701 × 1038 5.790 × 1076 6.704 × 10153

So we found a lower bound. Can we bound xn from above? This is frequently the more acute question. So we are looking for another simple sequence of numbers, {un }, such that xn 6 un for all n > 0? Looking again at the numbers suggests that squaring the lower bound n+1 is more than adequate; so we pick un = ln2 = 22 as a likely candidate. Can we prove it? We use the same approach: induction. For n = 0 we find x0 = 2 < u0 = 4, and assuming n 2 n+1 the hypothesis holds for all 0 6 k < n, we need to show that 2 22 − 2 is less than 22 . However, developing the first expression shows immediately that this is not the case at all. What happened? Let us look further. Consider the similarly defined sequence {an }n>0 , which is determined by the recurrence


22

n+1

an+1 = 2a2n , for n > 0, with a0 = 2. It is a simple matter to see that an = 22 −1 satisfies this recurrence. Furthermore, comparing the two recurrences, it becomes obvious that we must have xn 6 an , since the an elements are not reduced by 2 at each step. And yet this an is just one half of un . So surely the conjecture xn 6 un is true, and still, mathematical induction failed us. But now we see something interesting, possibly surprising: if we try to prove by induction the inequality xn 6 an , using exactly the same induction approach as before, the proof is immediate—in one line— similar to the proof for ln . So the tighter bound is easier to prove!? The conclusion is that not all true facts have direct induction proofs. Why did the proof “work” for an and not for un ? A simple explanation is that the bound un is so crude, that the hypothesis xn−1 < un−1 was too weak to enable the induction step. Whereas the tighter bound given by an , which obeys a similar recurrence, was a sufficiently strong support. So we have a second conclusion, for proofs by mathematical induction that involve inequalities: counterintuitively, tight bounds tend to be easier to prove than poor bounds, because the induction hypothesis is better (or stronger). Exercise 1.45 [3] The table in Example 1.44 shows that the upper bound, while not particularly tight, is still far better than the lower bound, which is very loose. Can you suggest and prove a tighter lower bound, of a comparable quality to an ? Exercise 1.46 [2] Let n points be selected at random along a circle and labeled with 0 or 1. Prove that there are at most ⌊(3n − 4)/2⌋ chords which join differently labeled points and do not intersect inside the circle. Exercise 1.47 [2] Here is a proof by mathematical induction; what do you think of it? Claim: Let a be a nonzero constant, then an = 1 for all n > 0. Basis: For n = 0 we have a0 = 1, which holds for all a 6= 0. Induction Hypothesis: an = 1 for all n 6 m. Now we need to prove for n = m + 1 Induction Step Proof: Write am+1 as product and ratio of numbers which—by the induction am am hypothesis—are all equal to 1: am+1 = m−1 = 1. a

1.3 Iteration and Recursion Iteration and recursion are the two methods we use when programming to process a set, an array, and sometimes an implicit collection of elements, as we explain below. Here we are interested in the ways they differ from the perspective of analyzing their performance. Sometimes they are very similar. Consider examining an array A containing n integers. You want to verify that it has at least one odd number. Here is a procedure that does it, using C syntax, returning 1 if such a number exists, and 0 otherwise: int findoddI(A, n) int ∗A, n; {int i; for (i=0; i < n; i++) if ( (A[i] % 2) == 1) return (1); return(0); }

We are clearly iterating over a position in the array. Now consider

1.3. ITERATION AND RECURSION

23

int findoddR(A, n) int ∗A, n; { if (n==0) return 0; if ( (A[0] % 2) == 1) return (1); return( findoddR(&A[1], n − 1) ); }

Similarly, this style is sometimes called tail-recursion (the program text terminates with a single call to itself). Such a program can always be changed to an iterative one, which is usually faster. To show this vague claim, we look at the costs of these procedures and, to keep our calculations simple, we restrict our evaluation to the worst, most costly case, when the entire array needs to be scanned. In the iterative procedure, we see several one-time costs: entering the procedure, the final return, and the loop set-up. They amount together to a quantity we call p. Evaluating the loop condition cost, c, and it is done—in the worst case—(n + 1) times. The last component is the parity evaluation, which is done n times, where each has a cost d. We get a total of Citer = p + c + n(c + d). In the recursive case we again have the cost of entering the procedure; there is no loop to construct, but there is the evaluation of the termination condition, and setting up the recursive call. We denote the cost of all these by p′ , which is probably close to p, but it is repeated n times. As before we have n evaluations of the parity, d each, and a more expensive return, since it includes a function call, which we call e. The total cost is Crecur = n(p′ + d + e). Its structure differs from that of Citer , since it is entirely linear in n, without a fixed part; in addition, considering the actual quantities there, it is almost always more costly, since p′ is typically significantly larger than c or d. Does it mean that recursion is bad? No, but it must be handled with care. Here is another function evaluation, that implements Eq. (1.10); the recursion seems inherent, and leads to an expensive calculation: int FibonacciR(n) int n; { if (n >= 1; } } } return y; Exercise 1.51 [4] The following procedure, pn, prints a decimal number, using recursively the only printing service provided by the environment, pd, which prints a single digit: void pn(n) int n;

{if (n>=10) pn(n/10);

pd(n%10); }

(a) Show by induction that pn works correctly. (b) Show how to perform this operation, using the services of pd, in an iterative way.

Chapter 2

Combinatorics Life presents many situations where we find ourselves counting objects or their arrangements. You sit down to an exam and find that you need to do any five problems from the eight given. How many different choices can you make? You have three pairs of jeans, three shirts, and two pairs of shoes you like; how many different outfits can you make? Does it matter if you are color blind? Suppose 30 hockey teams from the National Hockey League compete for the Stanley Cup: in how many different ways can the tournament be arranged? In how many different ways can the teams rank in the final standing (with no ties allowed)? How many different lineups are possible for six local jazz bands at a benefit concert? The answers to questions like these require methods known as combinatorics, a topic of mathematics devoted to the art of counting. In particular, the typical questions that arise in the course of analyzing an algorithm, or designing a program, tend to require counting. Another practical example provides one of the most efficient error-detection method, called codabar. This method is used by all major credit card companies, as well as other enterprises. When a bank issues a credit card with the identification number a1 a2 a3 a4 -b5 b6 b7 b8 -c9 c10 c11 c12 d13 d14 d15 d16 , the last digit, d16 , is the check digit to control the accuracy, which is chosen as follows. Double the sum of all digits in odd positions, then add those digits that exceed 4 and then add all the remaining digits in even positions. The check digit is chosen to make the total ends with zero. How many such numbers exist? This is more complicated than the previous examples, but they all are solved using the same approach. Obviously, it is impossible to include all combinatorial methods in one chapter—there are many books devoted to this subject (see, for instance, [26, 30, 93, 115]). This chapter gives an introduction to combinatorics and will be used as a reference later on. In this chapter, three elementary counting methods are considered: the Fundamental Principle of Counting, permutations, and combinations. We also discuss the formal properties of summation and the most common set of combinatorial numbers: the binomial coefficients. In addition, we present some basic properties of canonical power series—hypergeometric function. At the end, a famous asymptotic formula—the Stirling approximation to a factorial—is considered.

25

CHAPTER 2. COMBINATORICS

26

2.1 Properties of Summation Sums are among the most frequently used operations on expressions. Recall some facts about them. For example, there are several ways to denote a summation: n

x1 + x2 + · · · + xn =

∑ x j,

∑

or

x j,

or

16 j6n

j=1

∑

x j,

or even

j∈[1..n]

∑ x j, p( j)

where p( j) is some property of j, where j is an integer, and 1 6 j 6 n. If n < 1, this is a summation over the null set, which is customarily defined to be zero1 . By convention, a sum over an empty set of indices equals zero. There may be more complicated cases, as examples show. Other possibilities for p( j) could be that j is a perfect square, or that j is a prime number greater than 13, and so on. n 1 . This example uses the ability to split a sum into Example 2.1 Find the sum ∑ k=1 k(k + 1) sums of its components without changing its value. Solution. Using partial fraction decomposition2 (PFD), we have n n 1 1 1 = ∑ − ∑ k+1 k=1 k k=1 k(k + 1) n n 1 1 always allowed for a finite summation = ∑ −∑ k=1 k + 1 k=1 k 1 1 1 1 1 1 1 + + ··· + + − = 1+ + + ···+ 2 3 n 2 3 n n+1 n 1 = . = 1− n+1 n+1 This is a telescopic sum because of the “cross-cancellations” between the two separated sums in the parentheses above. When n approaches infinity we get a compact result, ∞

∑ k=1

n 1 1 = 1. = lim = lim n→∞ n + 1 n→∞ 1 + 1 k(k + 1) n

We recall some basic properties of infinite sums. Let T be an infinite index set; we are only allowed to write ∑ (ak + bk ) = ∑ ak + ∑ bk if it is true

k∈T

k∈T

k∈T

that3

∑ |ak | < ∞

∑ |bk | < ∞,

and

k∈T

k∈T

a condition which is called absolute convergence. To see that the condition is not necessary, consider the last example: the two harmonic series

∑ k>1 1 Some

1 k

and

∑ k>1

−1 k+1

authors use the definition ∑m6 j6n x j = − ∑n6 j6m x j , where m > n. We do not follow this notation. topic is discussed in more detail in section 12.5. 3 This is a sufficient condition. 2 This

2.1. PROPERTIES OF SUMMATION diverge—as we presently prove—but their sum ∑k>1 to 1.

27 1 k

1 − k+1 was just shown to converge

Note: The sum of the reciprocals of the first n natural numbers is common enough to have def earned a special abbreviation: Hn = ∑nk=1 1k . It is called the nth harmonic number4 . Note that the sum for H0 is empty, hence we say H0 = 0. Observe that each term in the sum Hn , after the first, is the harmonic mean of its two nearest neighbors. Recall that the harmonic mean5 of two numbers, a and b, is 2ab/(a + b). The harmonic number Hn is never an integer for n > 1. As n increases Hn → ∞, because the asymptotic behavior of Hn is known to be logarithmic: 1 1 1 n → ∞, (2.1) Hn = 1 + + + · · · + ≈ ln n + γ + · · · , 2 3 n

where γ = 0.577215665 . . . is called the Euler constant, named after the famous mathematician Leonhard Euler.6 1 1 1 Alternatively, the harmonic series 1 + + + + · · · diverges logarithmically. Let us look 2 3 4 at successive prefixes of the series of Hn . Specifically, we consider prefixes of sizes 2m for increasing m. 1 m=0: 1> , 2 1 1 1 2 m=1: 1+ > + = , 2 2 2 2 1 1 1 1 3 1 + > 1+2 = 1+ = , m=2: 1+ + 2 3 4 4 2 2 1 1 1 1 1 1 3 1 4 1 + + + + + > + = . m=3: 1+ + 2 3 4 5 6 7 8 2 2 2 We see that the sum of the first 2m terms is greater than m+1 2 . By taking enough terms, we can find harmonic numbers that are as large as we please. The harmonic series diverges, but very slowly; for example, it takes more than 1.5 × 1043 terms for its partial sums to reach 100. Also H1,000,000 ≈ 14.39. It is interesting to note that because of the very slow increase of the harmonic numbers, if you set to calculate Hn with fixed-point precision, numerically for increasingly large values of n, you may be led to the conclusion that the series converges. In practice, calculations of Hn for big n are made with formulas other than the sum (2.1). 4 The notation ζ (n, 1) or ζ (1) is also used; see Definition 2.43 on page 41. It is hard to trace the origin of n the word “harmonic” here; this word seems to come from music, where the vibration of strings or pipes is used to produce sounds. It has been observed that not every sound is attractive to the human ear, except those that are in “harmony” with the fundamental mode, which means that their frequencies are small integral (1 : 2 : 3, etc.) multiples of the fundamental. It is believed that Pythagoras or one of his disciples introduced first the theory of music, based on numerical intervals that can be expressed in terms of ratios of small whole numbers. 5 The origin of the harmonic mean can be traced back to the ancient Pythagoreans (6th century BC) who used three means—arithmetic, geometric, and harmonic (called that time as subcontrary). For example, in music the fifth (3:2) is the harmonic mean (12:8) of the octave (2:1 = 12:6). 6 Leonhard Euler (1707, Basel, Switzerland – 1783, St. Petersburg, Russia) was one of the most influential and inventive mathematicians of all ages. Nearly all the mathematics in this book would have been known to him, more than two hundred years ago.


28

Definition 2.2 The floor of a real number x (written ⌊x⌋) is the greatest integer that is less than or equal to x. The ceiling of a real number x (written ⌈x⌉) is the least integer that is greater than or equal to x. ⊳ Obviously ⌊integer + fraction⌋ = integer and ⌈integer + fraction⌉ = integer + 1. For example, the floor of π = 3.1415926 . . . is ⌊π ⌋ = ⌊3.1415926 . . .⌋ = 3, as is ⌊3.0⌋ = 3. The ceiling of π is ⌈π ⌉ = 4, same as ⌈4.0⌉. For negative π , we have ⌊−π ⌋ = −4 and ⌈−π ⌉ = −3.

Similarly, for e = 2.718281828 . . . we have ⌊e⌋ = 2 and ⌈e⌉ = 3. In the C function library and in MAPLE the corresponding functions are floor and ceil.

Exercise 2.3 [2+] Let x ∈ R, n ∈ Z, and a, b ∈ Z+ . For what input values are the following identities valid? l m j k 1 x n ⌊n/a⌋ ; (c) ; ⌈x⌉ = = (a) ⌈x⌉ = −⌊−x⌋; (b) b ab n n (d) ⌈3x⌉ + ⌈x⌉ = 2⌈2x⌉ . x def + Exercise 2.4 [2+] (R. J. Simpson) For any positive number x > 0, let A(x) = √ ⌈ x⌉ √ √ x def + x . Express B(x) in terms of A(x). x and B(x) = √ ⌊ x⌋ lnm n ; (b) ∑nk=1 (−1)k k2 = (−1)n n(n+1) Exercise 2.5 [2] Prove (a) ∑ (−1)k k = (−1)n 2 . 2 k=1

2.1.1 Index Transformations Perhaps the most useful fact we need to remember about manipulating summations is that the summation index is called dummy index for a reason: it serves no inherent purpose except to trace the set covered by the summation. Many “difficult sums” become manageable once the index is manipulated. There are only two rules to observe when manipulating dummy indices. One is that we maintain a relation as follows: (2.2) ∑ a j = ∑ a f (i) , j∈P( j)

i∈Q(i)

where the function f (·) satisfies { j P( j)} = { f (i) Q(i)}.

(2.3)

The last equality is between sets. The one on the left-hand side is the set { j} for which the property P( j) holds, and on the right we have the set of indices { f (i)} for all i such that Q(i) holds. The other rule is that we do not reuse the name of an active index; we show below what this can lead to. Example 2.6 Let P( j) specify the set of all integers in [0..1000] that have remainder 2 when divided by 3. In this case, set j = 3i + 2 for some integer i ∈ [0..332]; this provides the needed range, since when 0 6 3i + 2 6 1000, or −2 6 3i 6 998, we find −2/3 6 i 6 332 + 2/3; rounding to the nearest integer—noticing the sense of the inequalities—we obtain i ∈ [0..332].

2.1. PROPERTIES OF SUMMATION

29

This change of index can be written as

∑

∑

ak =

k∈[0..1000] k ≡ 2 mod 3

a3i+2 .

i∈[0..332]

Example 2.7 Here is another illustration of the fungibility of dummy indices. n

∑

j=1

1 aj

n

n

∑ ak = ∑ k=1

k=1

n

n

1 ak

n

∑ ai = ∑ ∑

j=1 k=1

i=1

1 ak . aj

However, we cannot change dummy indices arbitrarily. Suppose we change k into j in the rightmost expression above, then we would get (since ∑nj=1 1 = n) n

1 ∑ ∑ a j ak = j=1 k=1

n

n

n

∑∑

j=1 j=1

1 a j = n2 aj

wrong!

This was wrong since the index j was already used in the same expression. Let us define the indicator function, which is usually called7 the Iverson bracket: ( 1, if the property P is true, [P] = 0, otherwise.

(2.4)

The closely related Kronecker’s delta8

δi j = [ i = j ] =

(

1, if i = j, 0, otherwise.

(2.5)

is a particular case of Iverson’s notation for the property of index i being equal to j. Example 2.8 This is a simple—and surprisingly effective—use of relation (2.2):

∑

xk =

06k6n

∑

xn− j .

06 j6n

It is the basis of many calculations. We now demonstrate its strength in a few examples. Define Φ1 (n) as the sum of the first n positive integers (called natural numbers). Then def

Φ1 (n) =

∑

16k6n

k =

∑

06k6n

k =

∑

(n − j) =

06 j6n

∑

06 j6n

n−

∑

06 j6n

j = n(n + 1) − Φ1 (n).

Hence, we have 2 Φ1 (n) = n(n + 1) 7 This

=⇒

Φ1 (n) =

n(n + 1) 1 2 1 = n + n. 2 2 2

(2.6)

notation was introduced by Kenneth E. Iverson (1920 – 2004) in his programming language APL. Kronecker (1823 – 1891) was a German (of Jewish descent) mathematician and logician, a student of E. Kummer. 8 Leopold


30

Sometimes sums can be found with the aid of the method of undetermined coefficients when a good guess about the result is available. Say, for the sum Φ1 (n), we may have a reasonable guess that Φ1 (n) is a polynomial of the second degree: n

∑ k = an2 + bn + c. k=0

Setting n = 0, n = 1, and n = 2, we get the system of algebraic equations 0 = c,

1 = a + b,

3 = 4a + 2b.

Hence, a = b = 1/2. Knowing Φ1 (n), we can generalize somewhat: S1 (n) ≡

∑

(a + kb) =

∑

a+

06k6n

06k6n

= (n + 1)a + b

∑

kb = (n + 1)a + b

∑

k

06k6n

06k6n

a + (a + nb) n(n + 1) n + 1 = [2a + nb] = (n + 1) , 2 2 2

where a is the first term and a + nb is the last term in the series. Next, we define def

Φ2 (n) =

∑

k2 =

∑

(n − j)2 =

06 j6n

06k6n

∑

(n2 − 2n j + j2 ).

06 j6n

Thus, Φ2 (n) = n2 (n + 1) − 2n Φ1 (n) + Φ2 (n).

The term Φ2 (n) cancels on both sides of the equation, leaving n2 (n + 1) − 2n Φ1 (n) = 0, which reiterates a result we know: Φ1 (n) = n(n + 1)/2. We need a stronger medicine to get Φ2 (n). So we try this method again, but one degree higher, def

∑

Φ3 (n) =

k3 =

06k6n 3

= n

∑

06 j6n

(n − j)3 =

∑

06 j6n

∑

(n3 − 3n2 j + 3n j2 − j3 )

06 j6n

2

1 − 3n Φ1 (n) + 3n Φ2 (n) − Φ3 (n).

Hence, 2Φ3 (n) = n3 (n + 1) − 3n2

n(n + 1) + 3nΦ2 (n) 2

or 2Φ3 (n) = 3nΦ2 (n) − (1/2) n3 (n + 1).

(2.7)

So, if we knew this pesky Φ2 (n), we would also know Φ3 (n) from this equation—but clearly Φ2 (n) cannot be captured by this method. To find it, we use the idea called perturbation of sums. The general idea is as follows. Let Sn = ∑nk=0 ak , then we can write Sn+1 in two forms: n+1

Sn+1 ≡

n

∑ ak = Sn + an+1 = a0 + ∑ ak+1 . k=0

k=0

(2.8)


31

If the rightmost sum can be expressed in terms of Sn , and we can solve the underlined equation, we usually get a useful result. Here we find

∑

Φ3 (n + 1) =

k3 = 0 +

∑

( j + 1)3 ,

06 j6n

06k6n+1

which we develop as Φ3 (n + 1) =

∑

( j3 + 3 j2 + 3 j + 1) = Φ3 (n) + 3Φ2 (n) + 3Φ1 (n) + (n + 1).

06 j6n

Since Φ3 (n + 1) = Φ3 (n) + (n + 1)3 as well, we obtain (n + 1)3 = 3Φ2 (n) + 3Φ1 (n) + n + 1. Aha! Since Φ1 (n) is known from Eq. (2.6), the sum of squares becomes 3 3Φ2 (n) = (n + 1)3 − (n + 1) − n(n + 1) 2

Φ2 (n) =

=⇒

n(n + 1)(2n + 1) . 6

(2.9)

Now we return to Eq. (2.7), which yields n2 (n + 1)(2n + 1) 1 3 1 − n (n + 1), 2Φ3 (n) = 3nΦ2 (n) − n3 (n + 1) = 2 2 2 and rearrangement produces n(n + 1) 2 n(n + 1)(2n + 1) 1 3 1 = (Φ1 (n))2 . − n (n + 1) = Φ3 (n) = /2 6 2 2

(2.10)

Note that since every function Φm (n), m = 1, 2, . . ., is a polynomial of degree m + 1, its exact expression can be determined by the method of undetermined coefficients. However, for large m, application of the method becomes arduous to implement. The sum Φ1 (n) = ∑nk=1 k can be generalized in a different way, so we iterate it. Let n

(2)

Φ1 (n) = Φ1 (1) + Φ1 (2) + · · · + Φ1 (n) =

∑ Φ1 (k) = k=1

n(n + 1)(n + 2) , 6

which can be proved by induction. Substituting instead of Φ1 (n) its expression (2.6), we obtain n 2 1 n 1 k 1 n(n + 1)(n + 2) 1 n k . ∑ 2 + 2 = 2 ∑ k2 + 2 ∑ k = 2 Φ2(n) + 2 Φ1 (n) = 6 k=1 k=1 k=1 Solving for Φ2 (n) gives (2)

Φ2 (n) = 2Φ1 (n) − Φ1 (n) =

n(n + 1)(n + 2) n(n + 1)(2n + 1) n(n + 1)(2n + 1) − = , 3 2 6

as expected. We may try next iteration (3)

Φ1 (n) =

n

(2)

∑ Φ1 k=1

(k) =

n(n + 1)(n + 2)(n + 3) 24


32

and, in general, (m)

n

Φ1 (n) =

(m−1)

∑ Φ1

(k) =

k=1

n(n + 1)(n + 2)(n + 3) . . . (n + m) . (m + 1)!

(2.11)

While this summation looks formidable, its proof can be easily done by mathematical induc (m) tion. Later, in §2.5, we learn that Φ1 (n) is the binomial coefficient n+m m+1 and Eq. (2.11) is a particular case of more general relation (A.22) in Appendix A. Exercise 2.9 [3−] Find each sum: (a)

∑

06k6n k is odd

k2 .

∑

(b)

k,

06k6n k ≡ 2 mod 3

Exercise 2.10 [2] Enumerate the number of (a) squares, (b) rectangles on an n×n “chess” board. Answer the same questions for a rectangular m × n board. Another series we can manage by the perturbation method is the geometric series, a series in which the ratio of successive terms is a constant: a + a x + a x2 + a x3 + · · · =

∞

∞

n=0

n=0

∑ a xn = a ∑ xn .

For the basic geometric series Gn (x) ≡

xk ,

∑

(2.12)

06k6n

the perturbation method produces the following: Gn+1 (x) = Gn (x) + xn+1 =

xk = x0 +

∑

= 1+

∑

x

= 1+x

∑

xk

16k6n+1

06k6n+1 j+1

∑

j

x = 1 + x Gn (x).

06 j6n

06 j6n

Equating the leftmost expression to the rightmost one, we obtain Gn (x) + xn+1 = 1 + x Gn (x). Hence 1 − xn+1 . (2.13) Gn (x) × (1 − x) = 1 − xn+1 =⇒ Gn (x) = 1−x For the somewhat more complicated series (1)

Gn (x) ≡

∑

k xk ,

06k6n

the form of the terms suggests that we use differentiation and its commutativity with summation. Thus d k (1) x Gn (x) = ∑ k xk = x ∑ k xk−1 = x ∑ dx 06k6n 06k6n 06k6n d d xn+1 − 1 d k x = x G (x) = x (2.14) n ∑ dx 06k6n dx dx x − 1 n+1 x x n n+1 n (n + 1)x (x − 1) − (x − 1) = nx − x (n + 1) + 1 . (x − 1)2 (x − 1)2

= x =

The careful reader would like to verify this for a low value of n, such as 2.


33

Example 2.11 Suppose you win a lottery with a jackpot of one million dollars. The lottery regulations say you can take your winnings now—but you only get 2/3 of the jackpot, or you can get it all, spread equally over 20 years9 , with equal payments every year of $50, 000. Inflation is an ineluctable fact of our life, and this is the only issue we consider now. Let us say it is constant, running at 5% annually. While the $50, 000 you get now has a present value of—$50, 000, the $50, 000 you get next year will have a present value of $50, 000/1.05 = $47, 619 only. The payment two years from now is worth only $45, 371. The total present value of 20 payments of $50, 000 over 20 years equals, with r = 1.05−1 , 1 − r20 = $654, 270. $50, 000 1 + r + r2 + r3 + · · · + r19 = $50, 000 1−r

So you are better off taking $666,666.67 right now.

Now consider another payment plan. In this one you are paid a thousand dollars this year, two thousand next year, three thousand the next year, and so on – indefinitely. The payment increases by a thousand dollars every year, but subject to inflation. We assume the same inflation rate as above, and ask about the present value of this plan; is it even finite? Let us calculate: 1000 + 2000r + 3000r2 + · · · = 1000

∞

∑ n rn−1 .

n=1

(1)

We have seen this series previously! According to Eq. (2.14) this is just (1/r)G∞ (r) = (1 − r)−2 . So it is finite. The total value is $1, 000 (1 − r)−2 = $441, 000 only. A simple calculation shows the value of the 150th payment is under $100, and the 357th is the first to be less than a penny. It is all down hill from there. Exercise 2.12 [3] Evaluate the following sums: n

(a)

∑ (3k − 2)2 ,

k=1

n

n

n

∑ k4 ,

(b)

(c)

k=1

2 ∑ k(k + 2) , k=1

(d)

k−1 ∑ k! , k=1

(e)

xk . ∑ k>0 k + 2

Exercise 2.13 [4] Evaluate the following sums. Whenever the range of summation can be extended to infinity, find the limit value. n

n

(a) (d)

(−1)k ∑ k ; k=1 n k+2 ∑ k2 − 1 ; k=2

(b) (e)

2k + 4 ; k2 − 1 k=2 n k+2 ∑ k(k2 − 1) ; k=2

∑ (−1)k

n

(c)

∑ (−1)k k3; k=1 ∞

(f)

∑ k=1

ak . k!(k + 2)

Exercise 2.14 [3] Show that (2)

Gn (x) =

n

x

∑ k2xk = (1 − x)3

k=1

1 + x − xn (n + 1)2 + xn+1 (2n2 + 2n − 1) − n2 xn+2 .

(2)

What is the value of limn→∞ Gn (x)? Hint: How does the limiting value depend on the size of x? 9 Let

us not consider tax issues and alternative lucrative investment opportunities here.


34

Exercise 2.15 [2] Show 1 1 = − ln(1 − q), for |q| < 1. (a) ∑ qk = ln 1−q k>1 k (−1)

(b) Gn

def

n

(q) =

1 ∑ k qk = k=1

Z q 1 − xn x=0

1−x

dx.

Exercise 2.16 [3] (a) Use the method of perturbation to compute the sum of the first n Fibonacci numbers, as defined on page 16. (b) Show how to obtain the sum over odd indices, ∑nj=1 F2 j−1 , by reducing the sum to part (a).

2.2 Multiple Sums Sometimes summations get nested. This nesting can go to any depth, but we only look at two-nested sums since all the principles are revealed with such sums. The ranges of summation are independent, as with the following double sum: 2

2

∑

16i, j62

ai j = a11 + a12 + a21 + a22 = ∑

2

∑ ai j = ∑ Ai,

i=1 j=1

(2.15)

i=1

where Ai = ∑2j=1 ai j .

2.2.1 Changing Order of Summation When the ranges are not independent, things get more interesting. Different orders of the summation call for different operations, and we shall see that this is often the key to deriving a closed-form result. The rules of changing the order of summation are reminiscent of the corresponding rules for multiple integrals. The key relation has the form:

∑ ∑

16i6n j∈K(i)

ai j =

∑ ∑

ai j .

(2.16)

16 j6m i∈T ( j)

There are no general rules of how to do this change of order beyond the obvious one: that both summations cover the same sets: { i, j | 1 6 i 6 n, j ∈ K(i) } = { i, j | 1 6 j 6 m, i ∈ T ( j) }.

(2.17)

The alert reader may question our assumption that in both orders the outer index runs on a single contiguous set of values. It turns out to be the case in all the applications we have in mind here. We shall see a few useful examples. The simplest, and surprisingly common and effective, one is the following, also shown in Figure 35. Make sure you see how both sides sum over the same set of grid-points in the plane. n

i

n

n

∑ ∑ ai j = ∑ ∑ ai j .

i=0 j=0

j=0 i= j

(2.18)

2.2. MULTIPLE SUMS

35

Example 2.17 We consider a double series, which we shall do in both orders of summation. Let S=

n

r

r=0

j=0

∑ ur ∑

j.

First, in the given order, we use the known value of the sum Φ1 (r) = ∑rj=0 j = then we extend the method used to derive Eq. (2.14), S=

1 2

n

n

u

∑ ur r(r + 1) = 2 ∑ r(r + 1)ur−1

r=0

=

r=0

n

u 2

∑

r=0

d 2 r+1 u d 2 u = du2 2 du2

r(r+1) 2 ,

and

n

∑ ur+1.

r=0

j 6

Th el in ei

=

j

n

j

- i

0 0

n

i Figure 35: Grid points for two-way summation.

Taking one power of u outside the sum, we can use Eq. (2.13): S=

u d 2 un+1 − 1 u d 2 un+2 − u u d (n + 1)un+2 − (n + 2)un+1 + 1 u = = , 2 du2 u−1 2 du2 u − 1 2 du (u − 1)2

and the second differentiation yields S=u

n(n + 1)un+2 − 2n(n + 2)un+1 + (n + 1)(n + 2)un − 2 . 2(u − 1)3

Consider reversing the order of summation, using relation (2.18) or the accompanying diagram, to try to get a nicer result: S =

∑

06r6n

=

1 1−u

ur

∑

j=

06 j6r

∑

06 j6n

∑

06 j6n

j

∑

j6r6n

ur =

∑

06 j6n

juj

1 − un− j+1 1−u

n+1 Φ (n) G(1) n (u) − u 1 j u j − un+1 = . 1−u


36

This result is already in terms of quantities obtained in equations (2.6) and (2.14). The sum(2) mation in the initial order could also be so handled; but it requires the function Gn (x), calculated in Exercise 2.14, which is more complicated to use. Exercise 2.18 [2] Change the order of summations. n+1

n−2

n−1

(a)

∑ ∑ ani ;

(b)

∑ ∑ ani ;

(d)

n>1 i=1

n>3 i=1

n>2 i=1

∑ ∑ ani ;

(c)

∑

∑

16 j6n 06k6n− j

ak j .

Exercise 2.19 [3] Evaluate the double sums. (a)

n

n

i=1

j=i

∑ ai ∑ b j ;

(b)

∑

ai bi− j .

16i6 j6n n

Exercise 2.20 [2] Evaluate the product Pn = ∏

n

∏ ai bi− j .

j=0 i= j

[2] The Ramanujan10

Exercise 2.21 Q-function Q(n) = ∑i>1 ni /ni , where ni = n(n−1) · · · (n− i + 1), has been generalized by Knuth to Q0 (m, n) = ∑i>0 ni /mi , which appears in many algorithms and data structures. This exercise only asks you to manipulate its definition. Show that n emt nt = . (2.19) Q (m, n)m ∑ 0 n! 1 − t n>0 Exercise 2.22 For p > 0, prove the following double sums: (a) [2] (c) [3]

1 1 = , m (p + n) p + 1 n>2 m>2 π 1 ∑ ∑ (4m − 2)2n = 8 , n>1 m>1

∑∑

(b) [2] (d) [3]

1 = ln 2, p > 0. m (2n) n>1 m>2 1 π 1 ∑ ∑ (4m − 1)2n+1 = 8 − 2 ln 2. n>1 m>1

∑∑

Hint: Sum first on the index that is in the exponent. Check that the geometric series converges. To continue, you have two alternatives. One is to use the Maclaurin power series for the logarithm function at 1 + x and the inverse tangent function at x. Another is to use the digamma function Ψ, described in Appendix B (in particular, Eqs. (G.29)–(G.31)).

2.2.2 Summations and Finite Differences Series and integrals show similarities in several ways. Summation is the leading idea behind the development of the concept of the integral—but this is not the road we take here. Integration and summation are particular cases of a more general concept that we define in the following: Definition 2.23 An operator is a transformation that transforms a function into another function. A linear operator, L, is an operator such that L[a f + bg] = aL[ f ] + bL[g] for functions f , g and constants a, b. ⊳ When summation can be expressed as ∑m6k 1). This is the promised relation to differentiation. For example, ∆n! = (n + 1)! − n! = n · n!. Hence ∑nk=1 k k! =

2.2. MULTIPLE SUMS

39

(n + 1)! − 1. The final “−1” plays the role of the “integration constant” to take care of the lower summation bound. On the other hand, if ∆ f (n) = g(n), then f (n) = ∑n−1 k=s g(k) for any s < n − 1. And similarly, if ∇ f (n) = g(n), then f (n) = ∑nk=s g(k) for any s < n. Hence we define the inverse difference operators ∆−1 and ∇−1 as ∆−1 g(n) =

n−1

∑ g(k) +C (s < n − 1),

∇−1 g(n) =

k=s

n

∑ g(k) +C (s < n),

(2.26)

k=s

where C is any function whose first difference is zero. For example, C may be a constant or a function having the period 1 (i.e., sin(2π n)). So the anti-difference is unique up to the addition of the function C (which is usually a constant in applications). More information about finite difference operators and their inverses can be found in Cohn’s book [29]. We consider the application of the anti-difference operator to a quadratic polynomial in the following example. The general case is treated in §5.9.2. Example 2.30 Let us find ∆−1 xk (k = 0, 1, 2). We need to find a function gk (x) such that ∆gk (x) = xk . Since the difference operator (∆ or ∇) reduces the order of a polynomial by 1, we make a guess for required function gk (x) (k = 0, 1, 2) to be a polynomial of the degree k + 1. Calculations show that ∆x3 = (x + 1)3 − x3 = 3x2 + 3x + 1,

∆x2 = 2x + 1,

∆x = 1.

Therefore ∆−1 (3x2 + 3x + 1) = x3 +C,

∆−1 (2x + 1) = x2 +C,

∆−1 1 = x +C,

where C is a function such that ∆C = 0. From the above equations, we derive ∆−1 x2 =

x (x − 1)(2x − 1) +C, 6

∆−1 x =

x (x − 1) +C, 6

∆−1 1 = x +C.

(2.27)

n

Now suppose that we want to evaluate the sum Φ2 (n) =

∑ k2. Since ∆Φ2(n) = (n + 1)2 , we k=0

apply Eq. (2.27) to obtain Φ2 (n) = ∆−1 (n + 1)2 =

n+1 n (2n + 1). 6

Exercise 2.31 [1] Prove that ∆( f × g) = f × ∆g + ∆ f × g + ∆ f × ∆g f × ∇g + ∇ f × g − ∇ f × ∇g.

and

∇( f × g) =

Exercise 2.32 [1] Show that the difference operators, ∆ and ∇, commute, that is, ∆ ∇an = ∇ ∆an . Exercise 2.33 [2] Derive the product rule and the quotient rule for the backward difference operator ∇, which are similar to Eq. (2.25). Exercise 2.34 [2] Calculate the forward and backward differences of the following two functions over the integers. x−1 . (a) ∆((x + 3)(4x − 1)); (b) ∆ 3x+2 Exercise 2.35 [2] Apply ∇−1 to a quadratic polynomial.


40

2.2.3 Summation by Parts The next step in using the forward difference operator for summation is the following rule, called “summation by parts:” n

n

∑ g(k) ∆ f (k) = f (n + 1)g(n + 1) − f (0)g(0) − ∑

Theorem 2.36

k=0

f (k + 1)∆g(k).

k=0

P ROOF : Observe the following calculation, where the range of summation remains fixed, 0 6 k 6 n. In the second line, we add and subtract a term, and then change the order of the terms:

∑

∆ f (k) g(k) =

06k6n

= = = = =

∑[ f (k + 1) − f (k)] g(k) ∑ [ f (k + 1) − f (k)] [g(k + 1) − g(k + 1) + g(k)] ∑ [ f (k + 1) − f (k)] g(k + 1) − ∑ [ f (k + 1) − f (k)] [g(k + 1) − g(k)] f (k + 1)g(k + 1) − ∑ f (k + 1)[g(k + 1) − g(k)] − ∑ f (k)g(k) ∑ ∑ f (k + 1)g(k + 1) − ∑ f (k)g(k) − ∑ f (k + 1)[g(k + 1) − g(k)] f (n + 1)g(n + 1) − f (0)g(0) − ∑ f (k + 1)∆g(k). 06k6n

The result follows because we have a telescopic sum: ∑ (ak+1 − ak ), with ak ≡ f (k)g(k), k = 0, 1, . . . , n. (1)

Example 2.37 Let us find again the sum Gn (q) = ∑nk=1 kqk using summation by parts. Since ∆qk = qk+1 − qk = qk (q− 1), we can express qk as a finite difference of itself times a constant, (1) that is, qk = ∆qk /(q − 1). Let ϕ (q) = (q − 1)Gn (q) = (q − 1) ∑nk=1 kqk . Then ϕ (q) = n k ∑k=1 k∆q and we can apply summation by parts, which gives n

n

∑ k∆qk = (n + 1) qn+1 − ∑ qk+1 = (n + 1) qn+1 − q

k=0

k=0

1 − qn+1 . 1−q

Dividing both sides by 1 − q, we obtain (1)

Gn (q) =

n

q

∑ kqk = (1 − q)2

k=1

1 − (n + 1)qn + nqn+1 ,

as expected. Exercise 2.38 [2] Prove another, similar summation-by-parts formula that uses the backward difference operator ∇:

∑

16k6n

g(k) ∇ f (k) = f (n)g(n) − f (0)g(0) −

Example 2.39 Let us consider the sum n

S(n) =

2

∑ k(k + 2) .

k=2

∑

16k6n

f (k − 1)∇g(k).

(2.28)

2.2. MULTIPLE SUMS

41

Using partial fraction decomposition (see §12.5), we break the given ratio as follows 1 1 2 = − . k(k + 2) k k + 2 Hence # " n+2 n n 1 1 1 1 1 1 = ∑ −1− ∑ −1− − S(n) = ∑ − ∑ 2 3 j=1 j k=2 k + 2 k=1 k k=2 k 1 1 5 5 1 1 − . = Hn − Hn+2 + = − = Hn − 1 − Hn+2 − 1 − − 2 3 6 6 n+1 n+2 n

n

Now we consider another sum

∑ (2k) 3k . Since ∆3k = 3k+1 − 3k = 3k (3 − 1) = 2 · 3k , we can

k=0

evaluate the sum using summation by parts as follows. n

n

n

∑ (2k) 3k

∑ k ∆3k = (n + 1)3n+1 − ∑ 3k+1 ∆k

=

k=0

k=0

k=0

n n+1

= (n + 1)3

k

n+1

− 3 ∑ 3 = (n + 1)3 k=0

1 3n+1 − 1 3 = + n− 3n+1 . −3 3−1 2 2

Exercise 2.40 [2] Using summation by parts or integration find the following sums. (a) ∑nk=0 (2k + 1)2k ;

(b) ∑nk=2 (9k − 9)4k+1 ;

(c) ∑k>2

1 k(k+b) ,

b ∈ R+ .

Exercise 2.41 [1] We frequently find it expedient to represent several summations using a single ∑ symbol, as in Eq. (2.15). For most manipulations, however, the use of one ∑ symbol per dummy variable is more convenient. Transform the following one-symbol sums to multiple-symbol sums:

∑

(a)

akm ,

(b)

∑

akm ,

∑

(c)

akm .

16k6m6n

16k<m6n

16k 2.

Exercise 2.46 [4] Obtain a closed form for the sum ∑nk=0 ak sin(kb), by using: (a) Eq. (2.13) and representing the sine function by the exponential function (use Eq. (12.6)) for final simplification); (b) summation by parts. Exercise 2.47 [4] Consider the double sum ∑06 j6i6n ai a j , where the range designation is equivalent to specifying the set on the left-hand side of relation (2.17). Use the symmetry of the summand to show:   !2 1 ∑ ai + ∑ a2i  . ∑ ai a j = 2 06i6n 06i6n 06 j6i6n Exercise 2.48 [4] (a) Using mathematical induction prove the Lagrange identity !2 ! !

∑

16k<m6n

(ak bm − am bk )2 =

∑

a2k

∑

16m6n

16k6n

b2m −

∑

ak bk

.

(2.30)

16k6n

(b) How would you prove it directly (non-inductively)? (c) Consider the more general sum

∑

(ak bm − am bk )(Ak Bm − Am Bk ). Find for it a result

16k<m6n

like the right-hand side of the Lagrange identity and prove it.

2.3 Principles of Counting The role of enumeration, or counting, is fundamental to analysis of algorithms. In order to know the cost of an algorithm, either in time or space, we need to count the demands it puts on those resources. By definition, counting means finding the cardinality of a specified set. The rules that define the set are the basis for our counting it. Before doing any counting, we should state what is the best labor-saving device known in the counting trade: it is counting by correspondence. Theorem 2.49 Two sets have the same size if there exists a one-to-one correspondence between them. Such a correspondence is also called a bijection. For example, if we can show a bijection between some set S and the set of first n positive integers, [1..n], we know the set S has n elements and its size is n. We have counted it indirectly. We now consider procedural methods of counting, and start with two elementary but farreaching principles: the Rule of Sums and the Fundamental Principle of Counting (known also as the rule of multiplication). The first one is formulated as follows.

2.3. PRINCIPLES OF COUNTING

43

The Sum Rule of Counting. If there are m objects of one set (or category) and n objects of another set (or category), with all objects distinct, then the number of ways to select an object from any of the two categories is m + n. ⊳ Example 2.50 The college library has 8 books in computer science written by Donald Knuth and 5 books in computer science written by Robert Sedgewick. How many different books in computer science written either by Knuth or by Sedgewick available for students? The Rule of Sums gives the answer: 8 + 5 = 13 books. The following rule is also known as the Fundamental Principle of Counting. The multiplication Rule of Counting. The number of ways to choose n elements, one from each of n > 2 sets, is the product of the sizes of the sets. ⊳ This definition assumes all the elements involved in the selection are distinct. Let A = {b, r, u} and B = {r,t}; having to make a choice from either of the sets, we cannot use the sum rule to count the choices since A ∩ B = {r}. If we are concerned with counting the sets of chosen items, rather then with the number of ways to choose them, then the multiplication rule does not apply unless all items, combined from all sets are distinct. Typically, the context clarifies whether we are counting ways to make a choice, or possible distinct choices. Combinatorial calculations often involve products of the type 3 × 2 × 1 = 6, as seen in the previous example. This type of product is called a factorial, and the product 3 × 2 × 1 = 6 is written as 3!. In a similar manner, 4! = 4 × 3 × 2 × 1 = 4 × (3!) = 24, and 5! = 5 × 4 × 3 × 2 × 1 = 5 × (4!) = 5 × 4 × (3!) = 120. In general, n factorial, denoted by n!, is the product of all positive integers less than or equal to n: n! = n × (n − 1) × (n − 2) × · · · × 2 × 1.

(2.31)

As a special case, it is convenient to define 0! = 1 and 1! = 1. Similarly, we use the symbol “!!” to denote the double (or alternating) factorial: (2k)!! = 2 × 4 × 6 × · · · × (2k − 2) × 2k;

(2k + 1)!! = 1 × 3 × 5 × · · · × (2k − 1) × (2k + 1).

(2.32)

Exercise 2.51 [1] A modest menu in a restaurant offers 2 kinds of soup, 6 entrees, 7 desserts, and 4 drinks. Can you dine for a full year in this restaurant each evening, having a different three-course meal with a drink? Exercise 2.52 [1] A shelf carries fifteen different books. Four books are in French, five are in Russian, and six books are in English. In how many ways can you pick a pair of books that are not in the same language? Exercise 2.53 [2] (a) How many functions from a set S of size m to a set T of size n exist? (b) How many one-to-one functions are there from a set S into a set T as above? Exercise 2.54 [3] Let A = [ai, j ] be an n × n matrix, in which we place the positive integers from the set [1..n2 ], so that ai, j contains the number (i − 1)n + j, for 1 6 i, j 6 n. We now create a sum S of n terms as follows: the first term is selected at random from the matrix. Then the row and column in which it resides are removed. The second term is selected at random from the remaining (n − 1)2 numbers, and following the selection that row and column are erased as well. This continues till the nth term is selected at the last step (it is the only one remaining).

44


(a) How many different summations can we create? (Two summations are said to differ if not all their terms are the same; the order of the terms does not matter.) (b) What are the possible values of S? Exercise 2.55 [4] Let S be a set of size n. We say a sequence of k subsets of S, denoted by {Ai }k1 , is successively including when the subsets satisfy A1 ⊆ A2 ⊆ . . . ⊆ Ak . Show that the number of such sequences of size k is (k + 1)n . Exercise 2.56 Let S be a set with n elements. A binary operation, denoted by ◦, is said to be closed on S if it assigns to every ordered pair (x, y) of elements from S a third element z, denoted x◦y = z, for x, y, z ∈ S. We count now the number of possible closed binary operations under certain symmetry and regularity constraints. (a) [1] How many distinct operations are there, under no constraint beyond closure ( = the result of each binary operation on S is an element from S)? (b) [2] How many such operations exist that have a unit? (A unit, e, is a special element that satisfies a ◦ e = e ◦ a = a.) (c) [2] How many operations exist, with and without unit that are commutative? (The operation is commutative if for every choice of two S elements we have a ◦ b = b ◦ a.)

Exercise 2.57 [5+] (a) Derive an algorithm to count the number of closed associative binary operations, with and without unit. (A binary operation is associative if for every choice of three elements we have (a ◦ b) ◦ c = a ◦ (b ◦ c).) For arbitrary n, the number of elements in S, no formula is known, but you may check your algorithm for known values with small n = 2: there are 8 associative operations and 4 with a unit. For n = 3, there are 113 and 33 such operations, respectively. (b) Finally, derive an algorithm to count how many operations exist, with and without unit that are both associative and commutative. For n = 2, there are 6 such operations without unit and 4 with it. For n = 3, there are 63 and 27 such operations, respectively.

2.4 Permutations and Combinations Often we count ways of selecting more than one item from a set. We need to distinguish between selections made with replacement and selections made without replacement. Selecting items with replacement means that the same item can be selected again, and again since after it is selected it is put back in the set. Selecting without replacement means that a selected item is removed, and is no longer available. Every book published in the world has its unique International Standard Book Number (ISBN). It consists of 10-digit (or 13digit) integers, some of them may be equal. When items are selected with replacement, one can use the Fundamental Principle of Counting to determine the total number of possible outcomes; there are 10 × 10 × . . . × 10 = 1010 possible ten-digit ISBNs. Actually, there are many fewer possible ISBNs because the last digit serves as check digit, for error control, and cannot be selected arbitrarily, and the first digit indicates the country’s language where the book was published. For instance, zero is used for English speaking countries. In many situations, items cannot be selected more than once. When for example, selecting a

2.4. PERMUTATIONS AND COMBINATIONS

45

committee of six people from a group of thirty, the same person cannot be selected more than once. There are two attributes of the selection process that need to be specified: selecting items with or without replacement, and the order of selection is taken into account or not. When items are selected without replacement from a set and the order of selection is important, the various possible outcomes are called permutations. In other words, a permutation of a set X is nothing but a rearrangement of the elements of X . So a permutation on a set X is a bijection π : X → X , that is, both one-to-one and onto. The customary object of permutations is a prefix of the set of natural numbers, [1..n]. Such a permutation can be written as an array ! 1 2 3 ... n = hπ1 , π2 , π3 , . . . , πn i π= π1 π2 π3 . . . πn or as a sequence (word) of numbers (distinct elements) π = π1 π2 π3 . . . πn , where πk = π (k), k = 1, 2, . . . , n. A permutation is called the derangement if and only if πi 6= i, i = 1, 2, . . . , n. The inverse of the permutation hπ1 , π2 , π3 , . . . , πn i is the permutation hq1 , q2 , q3 , . . . , qn i obtained by interchanging its rows and then sorting the columns into increasing order of the new top row: ! ! 1 2 3 ... n π1 π2 π3 . . . πn = . 1 2 3 ... n q1 q2 q3 . . . qn In other words, q(πk ) = k. For example, the inverse of h 3, 5, 1, 4, 2 i is h 3, 5, 1, 4, 2 i, the same permutation (called involution), but the inverse of h 4, 5, 1, 3, 2 i is h 3, 5, 4, 1, 2 i.

Definition 2.58 Let α = ha1 , a2 , . . . , an i be a permutation of the set [1..n] = {1, 2, . . . , n}. An inversion of the given permutation α is a pair (ai , a j ) such that ai > a j for i < j. The inversion table ( b1 , b2 , . . . , bn ) of the permutation α is the array with entries b j being the number of elements to the left of a j which are greater than a j . ⊳ In the inversion table of the permutation ha1 , a2 , . . . , an i, the value b j is the number of inversions whose second component is j. For example, the inversion table for the permutation h 2, 5, 6, 7, 1, 4, 3 i is ( 0, 0, 0, 0, 4, 3, 4 ) because 2, 5, 6, 7 are to the left of 1; there is no element to the left of 2; 5, 6, 7, 4 are to the left of 3 that are bigger than 3; three numbers (5, 6, 7) are bigger than 4 to its left; no bigger element to other numbers at their left. There are eleven inversions in all: (2,1), (5,1), (5,4), (5,3), (6,1), (6,4), (6,3), (7,1), (7,4), (7,3), (4,3). Inversion tables provide a useful tool to investigate the properties of permutations. Since the element k ∈ [1..n] can have at most n − k larger elements precede it in the permutation, the table entries satisfy the following inequalities: b1 = 0,

0 6 b2 6 1,

0 6 b3 6 2,

...

, 0 6 bn 6 n − 1.

(2.33)

The above inequalities identify the inversion table uniquely. Moreover, there are exactly n! different n-tuples of numbers that satisfy these inequalities (2.33). This means that there is a one-to-one correspondence between permutations and inversion tables. Actually, there is a simple rule that allows us to compute the unique permutation having B = ( b1 , b2 , . . . , bn ) as its inversion table. For a given inversion table B, we restore the permutation ha1 , a2 , . . . , an i as follows: start from the right end, and for i = n, n − 1, . . . , 1 write down the (bi + 1)-st largest of the numbers not yet used. For instance, the permutation corresponding to the table ( 0, 0, 2, 1, 1 )


46

is h 2, 5, 1, 3, 4 i since 4 is the second largest of h 1, 2, 3, 4, 5 i, 3 is the second largest of h 1, 2, 3, 5 i, 1 is third largest of h 1, 2, 5 i, 5 is the largest of h 2, 5 i, and 2 is the largest of h 2 i. Note: There is known another definition of the inversion table, see, for instance, Knuth’s book [79]. However, Definition 2.58 seems to be more natural for sorting algorithms (see Sedgewick’s analysis in [120]). Definition 2.59 Let A = {a1 , a2 , . . . , an } be a finite set of elements. An ordered k-tuple h a1 , a2 , . . . , ak i, where each ai ∈ A, i = 1, 2, . . . k, is called k-permutation of the set A. For simplicity, we also call such set a k-permutation of n. ⊳ The number of permutations—ordered arrangements of k items selected without replacement from a pool of n items (k 6 n)—is denoted by nPk or Pkn or P(n, k) or nk . The last form is not a name, but an expression, called the k-falling factorial of n: k factors z }| { nk = n(n − 1)(n − 2) · · · (n − k + 1) .

(2.34)

The permutation formula P(n, k) = n!/(n − k)! is only valid for nonnegative integers n and k; however, the falling factorial α k can be defined for arbitrary real or complex values of α . Since the falling factorial operation is important in our manipulations, let us digress for a brief discussion of it.

Following the pattern of Eq. (2.34), we write:

α k = α (α − 1)(α − 2) · · · (α − k + 1),

α 0 = 1,

k > 1,

(2.35)

If k > 1, then the falling factorial is a polynomial of degree k (with leading coefficient 1) whose roots are 0, 1, . . . , k − 1. We have also a “distributive rule” for the falling factorials. Let α ∈ C and m > k, k, m ∈ N, then it is obvious by substitution from Eq. (2.35) that

α m = α (α − 1) · · · (α − k + 1)(α − k) · · · (α − m + 1) = α k × (α − k)m−k .

(2.36)

We can use this property to extend the definition of falling factorials for negative integers k by requiring the distributive rule to hold for all integers k ∈ Z. It would require, for example, by shifting the numbers in Eq. (2.36), that 1 = (α + k)0 = (α + k)k × α −k . We need then to define 1 1 . (2.37) α −k = = k (α + k) (α + k)(α + k − 1) · · · (α + 1)

Observe that this identity provides 0−k = 1/k!.

Exercise 2.60 [2] Explain why it makes sense (and is consistent) to say that x0 = 1 for any x. A related useful expression is α k , the k-rising factorial (called the Pochhammer symbol (α )k in combinatorial literature):

α 0 = 1.

α k ≡ (α )k = α (α + 1)(α + 2) · · · (α + k − 1),

(2.38)

These two factorials, falling and rising, are related via

α k = (−1)k (−α )k = (α − k + 1)k =

1 (α + 1)−k

.

(2.39)


47

Similarly,

α k = (−1)k (−α )k = (α + k − 1)k =

1 . (α − 1)−k

(2.40)

Example 2.61 The usual three prizes are to be awarded at the Soccer World Cup competition among 24 teams. How many different ways can the prizes be awarded?

Solution. We must select 3 out of 24 teams without replacement, and the order in which they are selected matters to the fans. The number of permutations of 3 items from a set of 24 is then: ≡ P(24, 3) ≡ P324 =

24P3

24! 24! = = 243 = 24 × 23 × 22 = 12, 144. (24 − 3)! 21!

There are then 243 = 12, 144 different ways to award three prizes in a field of 24 teams. 1 1 . Example 2.62 Consider the function f (x) = (1+x)−m = m = x x(x − 1)(x − 2) . . . (x − m + 1) Then ∆ f (x) = ∆(1 + x)−m = f (x + 1) − f (x) = (2 + x)−m − (x + 1)−m =

1 1 − m m (x + 1) x

1 1 − (x + 1)x(x − 1) . . . (x − m + 2) x(x − 1) . . . (x − m + 2)(x − m + 1) 1 1 m 1 − =− = −m(x + 2)−m−1 . = x(x − 1) . . . (x − m + 2) x + 1 x − m + 1 (x + 1)m+1 =

Exercise 2.63 [1] Using the forward difference operator ∆ and the backward difference operator ∇, compute the following differences: (c) ∇xm ; (e) ∆(x + a)m . (b) ∆xm ; (d) ∇xm ; (a) ∆xm ; Exercise 2.64 [2] Representing the summand as a falling factorial, compute the following sums. n 1 1 , (b) ∑ . (a) ∑ k>2 k(k + 2) k=1 (k + 1)(k + 2) Exercise 2.65 [2] (a) Representing the summands as falling factorials, use the results in Exercise 2.66 to compute ∑nk=1 k2 and ∑nk=1 k3 , and recapture the results given in Eqs. (2.9) and (2.10). (b) Repeat part (a) using rising factorials. Exercise 2.66 [3] Use parts (a) and (b) from Exercise 2.63 and Theorem 2.36 to show for m ∈ N the following immediate results of summations: n

(a) ∑ km = k=0 n

1 m+1 ; m+1 (n + 1)

(b) ∑ k−1 = Hn+1 ; k=0 n

(c) ∑

k=1

km

=

1 m+1 ; m+1 n

1 (n + m + 1)m+1 ; m + 1 k=0 h i n 1 1 (n + 1)−m+1 − (m−1)! ; (e) ∑ k−m = −m+1 n

(d) ∑ (m + k)m = k=0 n

(f) ∑ (1 − m + k)m = k=1

1 m+1 . m+1 (n + 1 − m)

It is assumed in parts (a), (c), (d), and (f) that m 6= −1; in part (e), it is assumed that m 6= 1, and Hn in part (b) is the nth harmonic number.


48

2.4.1 Combinations and Lattice Paths When more than one item is selected without replacement from a group and the order of selection is not important, the various possible outcomes are called combinations according to the following definition. Definition 2.67 Let A = {a1 , a2 , . . . , an } be a finite set of elements. An unordered collection k elements { a1 , a2 , . . . , ak }, where each ai ∈ A, i = 1, 2, . . . k, is called k-combination of the set A. For simplicity, we also call such set a k-combination of n. ⊳ The number of distinct combinations of k items selected without replacement from a pool of n items (k 6 n) is obtained by dividing the number of corresponding permutations, P(n, k), by the number of ways to arrange the set, k!. This number is variously denoted by Ckn , C(n, k), nCk , n or k , said “n choose k” and equals

n! nk n P(n, k) = = . (2.41) = C(n, k) = = k! k!(n − k)! k! k Our symbol of choice is nk . An alternative wording of the above description is to say that n k is the number of ways to split a set of size n into two subsets of sizes k and n − k. Ckn

Example 2.68 A four-person, gender-balanced committee is to be selected from a group consisting of ten women and eleven men. How many different committees are possible?

Solution. Because the order of selecting the members of a committee is not important, we use the formula for combinations and the fundamental principle of counting to obtain 11! 10 × 9 11 × 10 10 11 10! × = × = 45 × 55 = 2475. = 2! 8! 2! 9! 2 2 2 2 There are 2475 different well-paired committees of four people possible here. The numbers nk occur in many applications. For example, the Binomial Theorem states that n k n (1 + x) = ∑ x, |x| < 1, n ∈ N. (2.42) k>0 k Here is how the formula arises; look at: (1 + x)n = (1 + x)(1 + x)(1 + x) · · · · (1 + x) when {z } | n factors n is a positive integer. Multiplying the factors can be seen as repeated passes over them, selecting one term from each factor—a 1 or an x—and the coefficient nk shows the number of passes in which we picked k terms with x from the pool of n factors, (1 + x)n . Setting x = −1 in Eq. (2.42), we obtain a surprising result n (2.43) ∑ k (−1)k = δn,0 , k

where δn,o is Kronecker’s delta (see Eq. (2.5) on page 29). In the sum, as in many other sums involving binomial coefficients, the range of summation is not indicated explicitly because n the binomial coefficient, k , is zero when k exceeds n or k < 0. Therefore, in such sums, the summation goes over all possible values of the index k.


49

Because of the role they play in the Binomial Theorem, the numbers nk are called binomial coefficients. They are ubiquitous, and later in this chapter we’ll discuss further some of their properties. The binomial coefficients can be generalized to trinomial (and higher order) coefficients. Again, just as the binomial coefficients can be viewed as playing a double role—one in the binomial theorem, and one in counting ways of selecting subsets, the same duality holds for the generalization. Thus we have the trinomial theorem, (x + y + z)n =

∑

06a,b,c6n a+b+c=n

(a + b + c)! a b c x y z, a!b!c!

a, b, c ∈ N,

(2.44)

which gives rise to the trinomial coefficients, a+b+c (a + b + c)! . = a!b!c! a, b, c

(2.45)

Analogously we have the multinomial theorem, (x1 + x2 + · · · + xn )N =

∑

06a1 ,...,an 6N a1 +···+an =N

(a1 + a2 + · · · + an )! a1 a2 x1 x2 · · · xann , a1 ! a2 ! · · · an !

with the natural definition of the multinomial coefficients, a1 + a2 + · · · + an (a1 + a2 + · · · + an )! . = a1 ! a2 ! · · · an ! a1 , a2 , . . . , an

(2.46)

(2.47)

Let us look at the combinatorial meaning of the multinomial coefficients. The direct generalization of the interpretation of the binomial coefficient is that this multinomial coefficient counts the number of ways to divide N = a1 + a2 + · · · + an distinct objects into n disjoint sets, with a j of them in the set # j, for 1 6 j 6 n, and with the additional proviso that only membership in the sets counts, and the order of selection—as well as the order within each subset—is immaterial. It turns out that there is a second interpretation of the count given by the multinomial coeffi2 +···+an cient a1a+a . It is the number of permutations (linear arrangements) of N items, given 1 ,a2 ,...,an that they are of n types, with ai items of type i, for 1 6 i 6 n, and that items of the same type are indistinguishable. To see this, observe that N = a1 + a2 + · · · + an items can be arranged in N! permutations. Now, the a1 items of type 1 can be rearranged in a1 ! different ways; but because the items are the same, keeping the other items fixed, these a1 ! permutations are identical. To count distinct permutations, we need to divide N! by this factor. A similar argument holds for each of the other types. Exercise 2.69 [2−] A box contains five chartreuse balls, four white balls, and three black balls (balls of the same color are indistinguishable). Ten balls are pulled out, one at a time. In how many ways can this be done?

50


Example 2.70 While the letters {u, v, w} can be ordered in 6 different ways, the set {u, v, v} has merely 3!/1!2! = 3 arrangements, uvv, vuv, vvu. In general, the number of distinguishable permutations of n items in which a1 items are identical, a2 items are identical, . . . , at items n n! = . are identical, with n = a1 + a2 + · · · + at , is a1 a2 · · · at a1 ! a2 ! · · · at ! Exercise 2.71 [2] An anagram is a reordering of the letters of a word, which need not be meaningful. How many distinct anagrams are there of the word ANAGRAM? Of the word INSUBSTANTIAL? Example 2.72 Give a combinatorial proof that the ratio (2n)!/2n is an integer. Solution: The number of ways in which the 2n symbols s1 , s1 , s2 , s2 , . . . , sn , sn can be arranged is (2n)! 2n (2n)! = n , = 2! 2! · · · 2! 2 2, 2, 2, . . . , 2 and surely “the number of ways” is an integer. (Another natural argument is that (2n)! is a product of 2n successive integers, exactly n of which are even.) Exercise 2.73 [3] Show that the multinomial coefficient can be written as a product of binomial coefficients. In particular, the trinomial coefficient from Eq. (2.45) is given by a+b+c a+b+c b+c = . (2.48) a, b, c a b Note: There are several ways of proving this, and you may want to look at least into the following: (a) the combinatorial interpretation(s) of this identity; (b) the binomial theorem, via writing (u + v + w)n as ((u + v) + w)n ; (c) direct proof, via the evaluation of the coefficients for integer upper argument by factorials. Exercise 2.74 [3+] Note that the number of free labeled trees, as given in Exercise 1.36, n−2 can be expressed by the multinomial coefficient Tn (d) = d1 −1,d2 −1,...,dn −1 , where the sets d = (d1 , d2 , . . . , dn ) are called admissible when they satisfy di > 1, and ∑ di = 2n − 2. Show that the sum ∑d Tn (d) = nn−2 , with the summation on all admissible sets d. In 1852, E. E. Kummer11 [82] proved a remarkable theorem, which has various generalizations and applications. Our purpose here is not to prove it (which could be found in [66]), but to show some applications in computer science. Theorem 2.75 (Kummer) For any nonnegative integers a1 , a2 , . . . , an and any prime p, the n exponent of the highest power of p that divides the multinomial coefficient a1a+···+a equals 1 ,...,an the number of carries that occur when the addition a1 + a2 + · · · + an is performed in the p-ary number system. For a nonnegative integer m, there is the integer-valued function ord p that is the highest power a1 + a2 + · · · + an α α +1 α for which p divides the integer m, but p does not. So ord p = a1 , a2 , . . . , an γ , where γ is the number of carries in the p-ary addition a1 + a2 + · · · + an . According to Eq. (2.47), this requires evaluation of ord(m!) to be equal to (m − ∑∞ i=0 mi )/(p − 1), where i is the p-ary representation of the nonnegative integer m. m p m = ∑∞ i=0 i 11 Ernst

Eduard Kummer (1810 – 1893) was a German mathematician.


51

Example 2.76 The binary representation of the following 5 integers e1 = 14, e2 = 26, e3 = 43, e4 = 53, e5 = 77 are as follows: e1 = 14 = 8 + 4 + 2 = 23 + 22 + 21 + 0 · 20 = (0, 0, 1, 1, 1, 0)2 , e2 = 26 = (0, 1, 1, 0, 1, 0)2 , e3 = 43 = (1, 0, 1, 0, 1, 1)2 , e4 = 53 = (0, 1, 1, 0, 1, 0, 1)2 , and e5 = 77 = (1, 0, 0, 1, 1, 0, 1)2 . Then to find the sum, e1 +e2 +e3 +e4 +e5 = 213 = (1, 1, 0, 1, 0, 1, 0, 1)2 , it requires 13 carries, 213 13 which is in accordance with Kummer’s theorem since 2 divides 14,26,43,53,77 , but 214 does not.

Exercise 2.77 [3] (E. Ruderman) Prove that the number of odd coefficients in the expansion (a1 + a2 + · · · + an )m is nd , where d is the sum of the digits in the binary representation of m.

Example 2.78 (Northeastern lattice path)

A lattice point in the Cartesian plane is a point (x, y) with integer coordinates x and y. Our view of binomial coefficients as counting ways to select subsets answers the following question: How many paths are there to go from a point with coordinates (i, j) in the plane, as in Figure 51, to a point with coordinates (k, l), while only moving along the integer-grid lines, one step up or one step to the right. Such a walk is called a northeastern lattice path and can be written as a sequence of two letters, say, E and N. That is, a lattice path can be written as a binary word, and, therefore, it serves as a model for coin tossing problems.

(6,7)

(2,2)

Figure 51: Plane grid for Example 2.78.

To answer the question note that a total of N = (k − i) + (l − j) steps need to be made, k − i to the right, and l − j going up, in any order, and therefore the number of paths is the same as the number of distinct permutations of N elements, made up of two subsets of these sizes: N k−i . The diagram shows two of the 126 paths from (2,2) to (6,7). The same reasoning produces a trinomial coefficient as the number of paths in a (three-dimensional) cube, from j+n−k the point (i, j, k) to the grid point (l, m, n), as l−i+m− l−i,m− j,n−k . For higher dimensions, we can use the same reasoning also.

Exercise 2.79 [2] What is the number of shortest paths between two points in an n-dimensional lattice?

In many applications, we need to count the number of grid paths subject to some constraints. As an important example of such restriction, we consider northeastern lattice paths that start at the point (i, j) and go to the point (k, l), k 6 i and j 6 l, in such a way that they touch or intersect the straight line y = x + m, with m being an integer. The calculation of the number of these paths can be reduced to the calculation of the northeastern lattice paths from the point that lies symmetrically to the point (i, j) with respect to the line y = x + m to the point (k, l) without any constraint. Since we know the number of

52


northeastern lattice paths without restriction from Example 2.78, we are left to establish a one-to-one correspondence between these two kind of paths, which is achieved by applying the reflection principle12 : Lemma 2.80 Let A(i, j, k, l; m) be the set of northeastern lattice paths from the point (i, j) to the point (k, l), with j < i + m 6 k < l + m, that touch or intersect the straight line y = x + m, with m being an integer. Further, let S( j − m, i + m; k, l) be the set of northeastern lattice paths from the point ( j − m, i + m) to the point (k, l), without any constraint. Then there exists an one-to-one correspondence between these two sets A and S and the number of such paths is k− j+l−i k− j+l−i = . (2.49) k− j+m l −i−m P ROOF : Each path from the set A(i, j, k, l; m) meets the line y = x + m for the first time at a point (s − m, s), s = i + m, . . . , l. This point separates the lattice path into two parts, one that goes from (i, j) to (s − m, s) without touching the line y = x + m, and the another one that goes from (s − m, s) to the ( j − m, i + m) point (k, l) without any restriction. Reflect(s − m, s) ing the first part (from (i, j) to (s − m, s)) with respect to the line y = x + m, we obtain the path from the point ( j − m, i+ m) to (i, j) (s − m, s). Combining it with the part from (s − m, s) to the point (k, l), we obtain the (0, m) path from ( j − m, i + m) to (k, l), which belongs to the set S( j − m, i + m; k, l). Therefore a path from S corresponds to each path from A; and the diagram exhibits an example of such correspondence.

(k, l)

To finish the proof, we need to show that for every path from S, there exists a unique path from A. To achieve this, it is enough to show that every path from S goes through or touch the line y = x + m on its way to the point (k, l). Reflecting the part of this path from ( j − m, i + m) to (s − m, s) with respect to the line y = x + m, we obtain the path from set A. However, it is obvious that any northeastern lattice path from the point ( j − m, i + m) to the point (s − m, s) crosses at least once the line y = x + m because this straight line separates the plane into two disjoint sets and the starting point ( j − m, i+ m) is above this line, while the final point (k, l) is below the line. Therefore any line connecting these points crosses y = x + m. The following diagram exhibits an example of such correspondence. For example, the number of northeastern lattice paths from the point (0, 0) to (m, n), with m > n that touch or intersect the straight line y = x + 1 is m+n m+n = ; (2.50) m+1 n−1 12 This

principle is attributed to the French mathematician Désiré André (1840 – 1918).


53

and the number of such paths that avoid the point (a, b) is m+n a+b m+n−a−b − . n a n−a

Example 2.81 What is the number of nondecreasing sequences of m natural numbers that are less than or equal to n? The problem is to determine the cardinality of the set Ω(n, m) = {(k1 , k2 , . . . , km ) ∈ [1..n]m : 1 6 k1 6 k2 6 . . . 6 km 6 n}.

The direct calculation yields the sum n

|Ω| =

n

∑ ∑

k1 =1 k2 =k1

n

...

∑

ki =ki−1

n

...

∑

1.

km =km−1

The summation can be accomplished using known techniques but it is not immediate. Instead we establish the one-to-one correspondence (k1 , k2 , . . . , km )

→

(k1 , k2 + 1, . . . , ki + i − 1, . . . , km + m − 1)

between the sequences in Ω(n, m) and the m-subsets of [1..n + m − 1] = {1, 2, . . . , n + m − 1}, which we denote by N(n, m). And clearly the number of m-combinations from n + m − 1] is given by n+m−1 |Ω(n, m)| = |N(n, m)| = . m Exercise 2.82 [2] When two (combinatorial) sets have the same size, Definition 1.12 (page 12) promises the existence of a one-to-one mapping between their elements (such a mapping is also called a bijection). Here we have two such sets: (a1 , a2 , . . . , an )-combinations of N disdef tinct elements, where N = ∑k ak , and permutations of N items from n distinct subsets of sizes (a1 , a2 , . . . , an ). For example, if we pick n = 3 and let (a1 , a2 , a3 ) = (4, 2, 4), and denote each item by the number of subset it comes from, we would consider the permutations of items from the sets (1,1,1,1), (2,2), (3,3,3,3). One such permutation is 1232313113, and the formula says that we have a total of 10!/(4!2!4!) = 3150 ways to arrange those 10 digits. Find the combinatorially interesting mapping between these two sets. The result known as Fermat’s Little Theorem has a standard proof, based on the pigeonhole principle. Here you are asked for an alternative approach. Exercise 2.83 [3] (Fermat’s Little Theorem) Prove that n p−1 − 1 is divisible by p whenever p is prime and n is an integer not divisible by p. Hint: How many terms in the multinomial coefficient are equal to one? Exercise 2.84 [2+] [79, §5.3.1] Show that the following two sums describe the same set of partitions of [1..m + n] into subsets of sizes m and n, and hence are equal: q−1 m m− p+n−k p−1+k n−q+m− j q−1+ j ; S2 = ∑ S1 = ∑ , m− p p−1 n−q q−1 j=p k=0 for m, n ∈ N, 1 6 p 6 m and 1 6 q 6 n. Alternatively, p−1 q−1 m− p+n−k p−1+k n−q+m− j q−1+ j n+m +∑ = . ∑ m− p p−1 n−q q−1 m j=0 k=0 n Hint; Consider the ordered subsets A = {ai }m i=1 and B = {bi }i=1 such that a p < bq .


54

Exercise 2.85 [4] (a) Show that n!(n−1)! divides (n!)!. What is the combinatorial interpretation of the claimed divisibility? Note: There is a simple arithmetic proof of this claim based on the fact that the product of any n successive integers is divisible by n!, and the fact follows from ∏ni=1 (a + i) = (a+n)! a! = a+n n! n . (b) The following claim is harder to show: let n > m be positive integers, then the value of n gcd(m,n) is an integer, where gcd(m, n) is the greatest common divisor of m and n. n m Hint: gcd(m, n) = gcd(m − n, n). Example 2.86 A die is rolled twice. How many different outcomes are there?

Solution: This innocent question brings up a crucial issue: when are outcomes considered different? Sometimes usage gives the needed information. If we rolled two identical dice, we would not consider as different the outcomes 1,2 and 2,1. But here, when we roll one die twice, there is a natural and well defined order between the two results, and we would consider these outcomes as different. The answer is then 6 × 6 = 36. The same answer is obtained when two dice differ by colors. In both cases, we could override the distinction between dice and say that only the two values matter, no color nor time nor order matter! In that case, the number of possible outcomes is 21 because there are 6 pairs (i, i) and 62 = 15 “heterogeneous” pairs.

Exercise 2.87 [1] A man has n friends whom he wants to invite to dinner. His wife points out that their place is too small to have more than three of the friends at a time. They compute however, that by entertaining every night of the 365-day year (except their anniversary night) they can have every threesome of their friends exactly once. How large is n?

Exercise 2.88 [2+] How many five-card poker hands containing two distinct pairs are possible? How many five-card poker hands containing exactly one pair are possible? Exercise 2.89 [2] Show the simple identities: n2 = n2 + n+1 , and n3 = n3 + 4 n+1 + 2 3 n+2 3 . In general, we have Worpitzky’s identity: s−1

s

n =

∑ as j

j=0

s−1 n+ j s n+ j =∑ , s s j=0 j

(2.51)

where the coefficients as j are the Eulerian numbers; they have a standard notation: as j =

s j s+1 s i j ≡ ∑i=0 i ( j + 1 − i) (−1) . Some of their properties are displayed in Appendix C.

Exercise 2.90 [2] (a) How many two-digit decimal numbers exist? How many consist of even digits only? Of odd digits only? (b) How many three-digit decimal numbers exist? How many consist of even digits only? Of odd digits only? (c) How many n-digit decimal numbers exist? How many of them are with even (or odd)only digits? Recall that a k-permutation of n is an ordered k-tuple ha1 , a2 , . . . , ak i of distinct elements selected from a set of cardinality n. There are n! permutations of n elements. In next few exercises we consider circular permutations. An example of such a permutation is an arrangement of n persons sitting around a circular table with no marked seats. To a circular permutation h j1 , j2 , . . . , jn i, correspond n lin-


55

ear permutations h j1 , j2 , . . . , jn i, h j2 , j3 , . . . , jn , j1 i, . . . , h jn , j1 , j2 , . . . , jn−1 i. The claim they correspond to the same circular permutation means that if we used each of these linear arrangements to seat the people we would get essentially indistinguishable arrangements at the table. Therefore the number of circular permutations of the set [1..n] = {1, 2, . . . , n} is equal to n!/n = (n − 1)!. A more general problem of counting the number of ways to seat n (distinct) people around k identical circular tables (where no tables are allowed to be empty) is considered in §5.8.3 and in §8.4.1. Exercise 2.91 [2] We look at the Seven Dwarfs seated around a round table. In how many ways can they be seated if Doc refuses to sit next to Grumpy? Exercise 2.92 [2] We look at a V-6 car engine. Let the cylinders be numbered 1 through 6. A firing order of the engine is a permutation of these numbers that determines the order in which the cylinders are sparked cyclically. (a) How many different firing orders are possible on the engine? (b) Not all firing orders are equally good. The engine runs more smoothly if successive firings are in cylinders that are on different arms of the ‘V.’ How many different orders that satisfy this restriction are possible? Exercise 2.93 [1] The crew of the space shuttle consists of two pilots and five astronauts. A crew is to be formed, and NASA has ready 9 pilots and 20 astronauts. How many different shuttle crews can be formed, (a) If the pilots have identical functions? (b) If the pilots are designated as pilot and co-pilot? Exercise 2.94 [2] How many ways are there to pair off 8 women at a dance with 8 out of 16 available men? Exercise 2.95 [2+] A group of five women and six men needs to create two four-person teams. How many ways are there to do this so that there are two women (and two men) on each team? Exercise 2.96 [2] (a) License plates in a certain state display six of the digits, 0 to 9. All combinations are legitimate, but every pair of plates must differ in at least two positions. How many plates can that state use? (b) Generalize to n long sequences over an alphabet of t symbols. Exercise 2.97 [2+] Consider identification numbers that consist of nine decimal digits which have the structure a1 a2 a3 -b4 b5 -c6 c7 c8 c9 . The digits are not independent. For a variety of security and accounting reasons, the possible numbers are limited. Suppose they satisfy the following—entirely apocryphal—constraints: the sum of the three a digits must be odd, the two b digits must be unequal, the digits c6 , c7 and c8 must form a strictly increasing sequence, and c9 is used as a check digit (it is a deterministic function of the other eight digits, a mechanism that can identify most common mistakes in writing the identification number). How many such numbers exist? Exercise 2.98 [3+] Consider the set of the first n positive integers, [1..n]. We select subsets from [1..n]. Find Ln , defined as the smallest number such that in any subset of size Ln (from [1..n]) it is possible to find five equally spaced pairs of elements: denoting those numbers by (ai , bi ), with ai < bi , 1 6 i 6 5, there should exist a positive integer c such that bi − ai = c for 1 6 i 6 5. What is the smallest n for which selecting just 1% of the set suffices?

56


Exercise 2.99 [4] In how many ways can you place 9 rings on the 4 fingers of a hand, if (a) The rings are all identical? (b) The rings are all distinct, and their order on a finger does not matter? (c) The rings are all distinct, and their order on a finger does matter?

Further enumerations, many of which tend to use binomial coefficients, will keep appearing in the course. We need to discuss their properties, and for this we have to prepare the ground.

2.5 Binomial Coefficients Binomial coefficients are without a doubt the most popular array of numbers in discrete mathematics—except the natural numbers themselves. They have been introduced in their combinatorial capacity in §2.4. Here we go beyond that. We look on some of their computational characteristics, and since they turn up in summands very frequently, we also develop summation formulas that involve binomial coefficients in various forms. Further results could be found in numerous books [37, 53, 72, 77, 115].

2.5.1 Definitions and Properties Many of the properties of the binomial coefficients can be shown from their role in the Binomial Theorem, which we have seen in Eq. (2.42). Using this relation, a more general binomial formula would look like this: n n k n−k a k a n n n n =b ∑ = ∑ a b . (2.52) (a + b) = b 1 + b b 06k6n k 06k6n k We extend the definition of the binomial coefficient nk to upper parameters that are not natural numbers, but we keep the requirement for k to be not-negative integer, needed for the falling factorial that we defined in Eq. (2.34). Definition 2.100 The binomial coefficient of an arbitrary number x and a nonnegative integer k is defined as x def xk (2.53) = . k! k The symbol ab always vanishes when b ∈ / N. ⊳ For example, if x = 1/2 and k = 3 we have 1 1 3 1 1/2 2 × −2 × −2 = . = 3! 16 3

With this in hand, we define (1 + z)α for arbitrary number α as the power series (also called the Newton series13 ) ∞ α k α def z. (2.54) (1 + z) = ∑ k=0 k 13 Most people associate Eq. (2.54) with the name of Isaac Newton (1643 – 1727). As a matter of fact, he did not invent this formula; however, he surely used it in a few cases. The first traces of the Binomial Theorem are found in Euclid, Omar Khayyam (around 1100), Chu-Shih-Chieh (1300), and James Gregory (1670).

2.5. BINOMIAL COEFFICIENTS An important case is the square root, 1/2

(1 + z)

57 √

1 + x = (1 + x)1/2 , and we have

1 1 1/2 1/2 2 1/2 3 1 = 1+ z+ z + z + · · · = 1 + z − z2 + z3 ∓ · · · . 2 8 16 1 2 3

Exercise 2.101 [2] What are the values of the following binomial coefficients? (a)

−1 , −1

(b)

3/2 , 3

−2 (c) , 3

(d)

3.2 , 0

Example 2.102 Let us consider the binomial coefficient

−1/2 n

−2 , 3/4

(f)

−1/2 which we transform n

64 . 32

as follows

1 − 21 − 1 · · · − 21 − n + 1 × 3 × 5 · · · 2n−1 2 = (−1)n 2 2 2 = n! n! n (2n − 1)!! n 1 × 3 × 5 · · · (2n − 1) × 2 n! = (−1) = (−1)n 2n n! 2n 2n n!n! (2n)! 2n (2n)! = = (−4)−n . = (−1)n n 2 n 2 (2 × n!) (−4) × (n!) n − 12

(e)

A generalized formula is presented in Appendix A (see Eq. (A.12)). Exercise 2.103 [3] For the following binomial coefficients, find expressions that involve factorials of positive integers only. 1/2 2n − 1 −n (a) , (b) , (c) . n n n Example 2.104 (Square roots via the Binomial Theorem) The binomial formula (2.52) can be used to find a rational approximation to the square root of a positive real number r ∈ R+ and to estimate the accuracy. Let m be the closest integer to √ √ √ the root, then m = ⌊ r⌋ or m = ⌈ r ⌉, whichever is closer. Since | r − m| 6 1/2 we expect √ √ def that η = ( r − m)n → 0 as n → ∞ (since |( r − m)n | 6 ( 21 )n ). Using the formula (2.52), we get n √ n k/2 n n j n n−k n−2 j √ r (−m) = ∑ r j (−m)n−2 j−1 . r (−m) + r∑ ( r −m) = ∑ k 2 j + 1 2 j j>0 j>0 k=0 The rational approximation follows when we disregard the small η :

√

r≈

n j n−2 j 2j r m n ∑ j>0 2 j+1 r j mn−2 j−1

∑ j>0

.

(2.55)

What is the accuracy? If η were not deleted, it would end up in the numerator of Eq. (2.55). Hence the result is accurate to within η divided by the denominator. Suppose we want to find the square root of 5, and choose n = 10 in Eq. (2.55). We find that m = 2 and get √ √ we 930249 . The accuracy is ( 5 − 2)10 /416020 ≈ 1.3 × 10−12 the rational approximation 5 ≈ 416020 √ 930249 = 2.236067977501 . . .). (indeed, 5 = 2.236067977500 . . . and 416020


58

2.5.2 Transformations and Basic Sums The binomial coefficients satisfy a large number of identities that have frustrated many attempts to classify them. A classical treatment is given by Riordan in [115]; a modern, algorithmic approach is the base of Petkovsek’s et al. book [107], titled A = B. Here we only consider a few identities that are of major importance in manipulating the expressions encountered in analysis of algorithms. It will be convenient to name these identities. The simplest identity, called symmetry transformation, changes the binomial coefficients in the same way as dummy indices are transformed: n n (I–1) = , with integers 0 6 k 6 n. (2.56) k n−k For integers, the binomial coefficients can be written in terms of factorials and then the equality is manifested: n n! n n! and . = = k!(n − k)! n−k (n − k)!k! k The following identity involves two binomial coefficients that share a parameter: r m r r−k (I–2) = , m, k ∈ N. (2.57) m k k m−k P ROOF : If m < k both sides are zero and the claim holds. Otherwise, write the desired identity in terms of falling factorials: rm mk ? rk (r − k)m−k = × × . m! k! k! (m − k)! Using Eq. (2.36), and the similar relation m! = mk × (m − k)!, the relation holds.

Setting in Eq. (2.36) m = 0, r = k + α , we obtain 1 = (k + α )k α −k . This formula is used in Eqs. (2.37), (2.40), page 46, that define the falling/rising factorial for negative k. One more “computational” identity: x x x−1 , = (I–3) k k−1 k P ROOF : x (x − 1)k−1 x xk = = = k! k (k − 1)! k

k 6= 0.

(2.58)

x x−1 . k k−1 Exercise 2.105 [2] Show relations, similar to (2.58), between xk and binomial coefficients that only differ from it by 1 in one argument: x−1 x x x−k+1 x x , x 6= k; , k 6= 0. (2.59) = = x−k k k k−1 k k

The simple summation of all the binomial coefficients used in the binomial theorem, Eq. (2.42), is obvious with the substitution x = 1: n n (2.60) ∑ k = 2n . k=0

2.5. BINOMIAL COEFFICIENTS

59

Using the transformations we present here, it appears that there is an infinite number of other sums involving binomial coefficients that can be evaluated to a closed form. But not all! For n a very simple example, the partial sum ∑m k=0 k has no closed form, unless m is n, or n/2 or n n − n − 1). = 2 is very close to these values (e.g., ∑n−2 k=0 k n k Suppose we need to find thesum way to accomplish this task is to use ∑k k k a . The natural n k n − 1 k−1 the identity (2.58): ∑ k a = na ∑ a × 1(n−1)−(k−1) = na(1 + a)n−1 . So k k − 1 k k we have proved the identity n (2.61) ∑ k k ak = na(1 + a)n−1 . k d p a = p a p−1 . Another way to prove Eq. (2.61) is to use well-known formula from calculus: dx k d k Hence k ak = ak ak−1 = a da da = a da a and n d n k d ∑ k k a = a da ∑ k ak = a da (1 + a)n = an(1 + a)n−1 . k k

The following important recurrence is known as the addition formula: n n−1 n−1 (I–4) = + 0 6 k 6 n. k k k−1 Note: this recurrence generates the Pascal triangle, which is shown on page 60.

(2.62)

P ROOF : We can prove the identity by induction, but we give instead two other proofs. First via the Binomial Theorem: n n−1 n k n−1 k n n−1 x = (1 + x) = (1 + x)(1 + x) = (1 + x) ∑ x ∑ k k=0 k k=0 n−1 n − 1 n−1 n−1 n − 1 k+1 n−1 k x + xk+1 = ∑ xk + ∑ x =∑ k k k k=0 k=0 k=0 n−1 n n−1 k n−1 j =∑ x +∑ x with change of index j = k + 1 k j=1 j − 1 k=0 n n−1 n−1 =∑ + xk with change of index k = j. k k − 1 k=0 We added two vanishing terms, n−1 and n−1 n −1 , to the sum in the last stage. This is an equality between two polynomials. Since two power series (of which polynomials are a special case) are equal if and only if all coefficients of like powers are equal, the claim holds. Note that nothing in the above required the upper parameter to be a natural number. While it is true that the notion of a well-founded recurrence is only meaningful for that case, the identity (2.62) still holds when n ∈ / N ≡ {0, 1, 2, . . .}.

The next computational proof makes this more apparent. We begin with the right-hand side of the identity: x−1 x−1 (x − 1)k (x − 1)k−1 (x − 1)k−1 x − k + = +1 . + = k! (k − 1)! (k − 1)! k k k−1


60

Using the identity (2.58), we get the result. Example 2.106 (The ballot problem) Example 2.78 (page 51) showed how to count the number of distinct walks along the integer grid (also called northeastern lattice paths), on the plane between two points. The only possible steps are one to the right or one up (see Figure 51). Such walks are important in many applications and current example presents one of them. Consider an election with two candidates, X and Y , who by the end of the voting received n and m votes, respectively. If n > m, then candidate X is the winner. Suppose that in the process of counting the first k votes, the number of votes received by X and Y are xk and yk , respectively. The sequence of ordered pairs (xk , yk ), k = 1, 2, . . . , n + m, can be seen as a grid walk. Let n > m, and denote by ψn,m the number of ways of counting the votes so that, at any step, the vote count for Y is not larger than or equal to the vote count for X . The ψn,m are known as ballot numbers.

The previous discussion also tells us that ψn,m is the number of northeastern lattice paths from the origin (0, 0) to the point (n, m) such that xk > yk , k = 1, 2, . . . , n + m, and pictorially this means that if the x numbers are on the horizontal axis, the walk stays under the diagonal (with possible touching it) all the time. Such northeastern paths that never rise above the diagonal are usually referred to as ballot paths because of the role they play in solving the Ballot Problem. This problem was originally solved by Joseph Bertrand (1822 – 1900) and it asks about proportion of ballot paths among all paths from the origin to (m, n), n > m. In order to calculate the ballot numbers, we subtract from the total number of lattice paths from the origin to (n, m), equals to n+m m , the number of walks from (0, 0) to (n, m) that touch or intersect the straight line y = x + 1 to obtain using the reflection principle: n+m n+m n−m+1 n+m+1 n−m+1 m+n − = = . (2.63) ψn,m = n+m+1 n+1 m m m m−1 1 2n In the particular case n = m, we have ψn,n = n+1 n = Cn , the Catalan number, named after Eugene Charles Catalan (1814 – 1894). L. Euler first describes them in the 18th century when he investigated the number of triangulations of convex polygons. E. Catalan used them one hundred years later to count the number of binary parenthesizations of a string. Numerous applications of the Catalan numbers are given in [60, 125]. The addition formula gives rise to a curious pattern of the binomial coefficients, that can be arranged in following triangle: 0 row 1st row 2nd row 3rd row 4th row 5th row

1 1 1 1 1 1

2 3

4 5

1 1 3 6

10

1 4

10

1 5

1

Combinations 1 k 2 k 3 k 4 k 5 k

This triangle is known as Pascal’s Triangle14 , and it was apparently known to Omar Khayyam 14 In

honor of the French mathematician, scientist, and philosopher Blaise Pascal (1623 – 1662). Pascal’s


61

(c.1100). Except for the 1’s at the ends of every row, each number in this arrangement is the sum of the two closest numbers above it as dictated by Eq. (2.62). The final identity is known as negating the upper argument: n k−n−1 k (−1) = . (I–5) k k

(2.64)

The proof is immediate by using the falling factorial representation: (−n)(−n + 1) · · · (−n + k − 1) n n(n − 1) · · · (n − k + 1) (−1)k = . (−1)k = k! k! k If we reverse the order of the terms in the numerator, we see that we have k−n−1 (−n + k − 1)k = . k! k We present several examples of summation that involve binomial coefficients. We start with t j t (2.65) ∑ j r = r 2t−r , t, r ∈ N. 06 j6t P ROOF : Using the identity (2.57), we obtain t j t t −r = ∑ ∑ r j−r 06 j6t j 06 j6t r t t −r t t −r = = ∑ r 06∑j6t j − r r 06k6t−r k t t −r k t t−r = 1 = 2 , ∑ r 06k6t−r k r where the last sum is given by Eq. (2.42), with x = 1. Exercise 2.107 [4] Inthis exercise, we consider the cost of computing the value of the binomial coefficient mn for integer entries, say n = 100 and m = 40. Specifically: we only look at the range of operands where all the results can be expressed in terms of the basic machine arithmetic, and there is no need to use extensions such as multi-precision calculations. How many arithmetic operations are needed to find this value using each of the following approaches? (1) The formulas: n n! , (a) = m!(n − m)! m (2) The iterative calculation

(b)

n nm . = m m!

min(m,n−m) n n−k+1 . = ∏ k m k=1

(3) We calculate the values using the lower triangular (from the Pascal triangle) array (ai, j )n×n recursively, where (i, j) is the location in which the value ai, j = ij is stored. From the addi i−1 tion formula, it follows that the values ij satisfy the recurrence ij = i−1 (0 < j−1 + j , triangle was first published in Europe by Apian in 1529.


62

j < i); 0i = ii = 1. Your calculation should assume that only the values actually needed to reach mn . (a) Use the recursive algorithm (divide-and-conquer approach or a top-down approach).

(b) Use the iterative algorithm (dynamic programming or a bottom-up approach). (4) Use the recurrence as in part (3) but assume that the binomial coefficients ai,1 = 1i = i ai,i−1 = i−1 = i are known. Again, use two approaches: divide-and-conquer and dynamic programming. Note: there could be many ways to accomplish these tasks. Exercise 2.108 [2+] (Continuation of Exercise 2.107) Which of the approaches in Exercise 2.107 is the preferred method if we need: (a) A single binomial coefficient? (b) Most of the values mn with a fixed n, but in an unknown order?

(c) Which method should be preferred when the important feature of the calculation is not efficiency but precision? n o n+1 The sequence of numbers { 0, 1, 3, 6, 10, . . .} = , arises naturally from the 2 n>0

following diagram and Eq. (2.6). And just as naturally, they are called triangular numbers.

n=0

n=1

n=2

n=3

n=4

Imagine those triangles placed one on top of the other. You will get a tetrahedron (a regular triangle-based pyramid). For example, placing the four drawn triangles would provide a tetrahedron with edges of length four. The number of elements in a tetrahedron with side n is called the nth tetrahedral number and from Exercise 2.129, we find: i+1 (i − 1) + 2 k+2 n+2 n+2 = ∑ = = . ∑ 2 = ∑ 2 2 n−1 3 16i6n 16i6n 06k6n−1 One of the oddest facts about triangular numbers was conjectured by P. Fermat and proved by C.F. Gauss: every positive integer can be written as the sum of three triangular numbers. The annotation in Gauss’ famous 1796-07-10 diary was: ⋆ ⋆ EϒPHKA! num = + + . Exercise 2.109 [1] Let α = ha1 , a2 , . . . , an i be a permutation of the set [1..n] = {1, 2, . . . , n}. Show that if the permutation α has k inversions, then the reverse permutation han , an−1 , . . . , a1 i has n2 − k inversions. j−1 Exercise 2.110 [3] Use identity (2.58) to find the sum ∑mj=n j n−1 , m > n.

Exercise 2.111 [3] (a) Using summation by parts, prove so called the “vertical” recurrence relation n k n+1 = , m, n ∈ N. (2.66) ∑ m+1 k=m m


63

(b) Using the result of part (a), show that (Hn is the nth harmonic number) n k n+1 1 . H = H − k n+1 ∑ m+1 m+1 k=1 m

(2.67)

Hint: Use double summation. There is also interest in the proof by induction. Exercise 2.112 [2] Show that n 2n 2 n(n − 1) 2n n 2n −n 2n ∆ =4 = n ; ; (a) 2∆ n 4 n 3 4n n 4 n n n n 2n + 1 2n + 1 n(2n + 1) 2n + 1 k 2k 1 2k = 2n+1 ; . (b) ∑ k ∑ k k = 3 · 22n+1 2 k n n k=1 4 k=1 4 n Exercise 2.113 [2] Show the easy (and not very good) bounds 2n < 2n n < 4 , n > 2. Exercise 2.114 [2] Show that the sequence of binomial coefficients nk is strictly increasing for k = 0, 1, . . . , m − 1, where m = ⌈n/2⌉, and strictly decreasing for k = m, m + 1, . . . , n. Exercise 2.115 [2] Use the equality k3 = 6 3k + 6 2k + 1k and Exercise 2.111(a) to find the def value of the sum Φ3 (n) = ∑nk=1 k3 . Exercise 2.116 [3] Compute the sum ∑06 j6t (−1) j tj rj , t, r ∈ N. Exercise [2] 2.117 Letn > 3 be an odd positive number; show that the sequence n n n , , . . . , n−1 contains an odd number of odd numbers. 1 2 2 Exercise 2.118 [3] Prove the following formulas: 1 (m + n)! x m+n = , −2 < x < 1. Hint: Eq. (2.64). (a) ∑ ∑ n!m! 2 1 − x n>0 m>0 (b)

1 (m + n + l)! x l+m+n = , n!m!l! 3 1−x n>0 l>0 m>0

∑∑∑

−3 < x < 1. Hint: Eqs. (2.44), (2.45).

Exercise 2.119 [3+] We define the following sequence of numbers: n n def Sm (n) = ∑ (−1)k (n − k)m . k k=0

(2.68)

Show that (a) Sm (n) = 0 for m < n. (b) Sn (n) = n!. Use induction on m, for arbitrary n, for part (a). Note: This sum Sm (n) has a combinatorial interpretation: it counts the number of ways to distribute m different objects into n distinct cells so that each cell is occupied. Hence it is not surprising the sum (2.68) vanishes for m < n, but you need to prove it using the sum without recourse to the combinatorial interpretation. Make sure you understand that the induction does not carry beyond m = n − 1; this may not be obvious. This sum is closely related to an important sequence of numbers, the Stirling numbers of the second kind, mn , or the subset numbers. We revisit this problem in Exercises 5.145 and 8.45.

Exercise 2.120 [4] Using the binomial theorem and Eq. (A.12), show the following sums (x 6= 0)

64

2k k 1 . (b) (a) ∑ x =√ k 1 − 4x k>0 √ k 1 − 1 − 4x 2k x = . (c) ∑ 2x k k+1 k>0

CHAPTER 2. COMBINATORICS 2k 2x . ∑ k kxk = √ 3 1 − 4x k>0 √ k 2k x 1 − 1 − 4x √ (d) ∑ = ln . k k x[1 + 1 − 4x] k>0

Hint: For the next two parts, use Eq. (2.59). √ 2k 1 − 2x − 1 − 4x 2k − 1 k 1/2 k √ . (f) ∑ . x = (e) ∑ x =√ k 1 − 4x 2x 1 − 4x k>0 k − 1 k>0

2.5.3 Inverse Relations The dual relationship between two sequences of numbers { f (n)} and {g(n)}: n n k g(n) = ∑ (−1) f (k) ⇐⇒ f (n) = ∑ (−1)k g(k) k k k k

(2.69)

is called an inversion formula or inverse relation. Its proof is straightforward: n n k n k k k j ∑ k (−1) g(k) = ∑ k (−1) ∑ j (−1) f ( j) = ∑ f ( j) ∑ k j (−1)k+ j j j k k k n n− j = ∑ f ( j) ∑ (−1)k+ j Eq. (2.57) j k − j j k n n k n− j δn− j,0 = f (n), = ∑ f ( j) (−1) = ∑ f ( j) ∑ j k k j j j

because of Eq. (2.43). The dual relation (2.69) is associated with the orthogonal identity, k k+ j n , (2.70) δn, j = ∑(−1) k j k where δn, j is the Kronecker delta, Eq. (2.5). In Appendix A, we present a variety of identities similar to (2.70) since they occur frequently in combinatorial analysis. As will appear later, these inverse relations provide a guide line through the forest of combinatorial identities. The dual relationship (2.69) can be rewritten in matrix form. Other inverse relations for binomial coefficients are known (see, for example, Eqs. (A.61), (A.62) in Appendix A). The matrix approach is demonstrated in deriving another inversion formula: n−1 n−1 g(n) = ∑ f ( j) ⇐⇒ f (n) = ∑ (−1)n− j g( j). (2.71) j − 1 j − 1 j j We define two matrices (of unspecified size, for the moment) A = [ai, j ] and B = [bi, j ], where i−1 i− j i − 1 ai, j = and bi, j = (−1) . j−1 j−1 Note that the matrices A and B are lower triangular matrices. For any positive integer n > 0, we extract square n-by-n matrices An = [ai, j ]ni, j=1 and Bn = [bi, j ]ni, j=1 . Then An Bn = Bn An =


65

In , where In = [δi, j ] (δi, j is the Kronecker delta) is the identity n × n matrix (having all zero entries except diagonal terms to be ones). Let C = AB = [ci, j ] be the product of these matrices. Then i−1 j−1 ci,k = ∑ ai, j b j,k = ∑ (−1) j−k j − 1 k − 1 j j i−1 i−k i−1 i−k j−k i−k = (−1) = (−1) ∑ (−1) p = δi−k,0 = δi,k . k−1 ∑ i − j k − 1 p p j The inversion formulas (2.69) and (2.71) can be extended for multiple cases. For example, a−1 b−1 c−1 g(a, b, c) = ∑ ∑ ∑ f (i, j, k), i−1 j−1 k−1 i j k (2.72) a−1 b−1 c−1 a+b+c−i− j−k f (a, b, c) = ∑ ∑ ∑ (−1) g(i, j, k). i−1 j−1 k−1 i j k Exercise 2.121 [1] Prove the inverse relation Eq. (A.59), which claims that g(n) = ∑k implies f (n) = ∑k nk (−1)n−k g(k), as a simple consequence of the Eq. (2.69).

n k f (k)

Example 2.122 We consider in part a classic problem called the hat-check problem (also known as le problem des rencontres) that was proposed by the French mathematician Pierre de Montmort in 1708. Its formulation is described in detail in Exercise 8.66, page 469, but here we consider only its counting part: how many derangements can be made out of the set of n first positive integers?

Let us denote by Dn the number if derangements of [1..n] (that is, the number of permutations in which no integer is in its natural position). We represent the set of all permutations as the disjoint union of n + 1 subsets A0 , A1 , . . . , An , where Ai is the set of all permutations having exactly i integers in their natural positions. The cardinality of each such set is |An−i | = n n n−i Dn−i = i Dn−i . Since the total number of permutations is n!, we get

n n n n n! = D0 + D1 + · · · + Dn−1 + Dn . 0 1 n−1 n Setting bn = n!, we have bn = ∑ni=0 ni Di . Using the inverse relation you are asked to show in Exercise 2.121, we obtain n n n n 1 n! n−k n n−k n = n! ∑ (−1)k . Dn = ∑ (−1) bk = ∑ (−1) k! = ∑ (−1)n−k (n − k)! k! k k k=0 k=0 k=0 k=0 There is no closed form for this sum, but it gets very promptly close to 1/e. Exercise 2.123 [3] Let the triple-sequence {L(a, b, c)} be given by the equation a+b+c a−1 b−1 c−1 (a + b + c)! ∑ ∑ ∑ i − 1 j − 1 k − 1 L(i, j, k) = a! b! c! = a, b, c . i j k Find L(a, b, c). Note: the context in which this formula appears is given in Example 8.14, page 437.


66

2.5.4 Vandermonde Convolution We now introduce the term convolution. It is a linear operation on sequences that appears innumerable times in applied mathematics, including the analysis of algorithms. Definition 2.124 Let a = {ak } and b = {bk } be two sequences of real (or complex) numbers. Their convolution is denoted by a ⋆ b; it is the sequence c = {cn } of numbers defined by cn = ∑ ak bn−k = ∑ an−k bk ,

(2.73)

k

k

where the range of summation is extended over all values of k for which both ak and bn−k are defined. From the change of index in Eq. (2.73), we see that a convolution is a commutative operation: a ⋆ b = b ⋆ a. ⊳ Example 2.125 In our first example, we consider a convolution of harmonic numbers with reciprocals of the integers. Call those two sequences H = {Hk } and R = {1/k}. We calculate the sequence H ⋆R = {dn }, where the limits of summation are determined by both sequences: 1 H j = 0 for j < 1 and 1/k being defined only for k > 1. We have (R ⋆ H)n = dn = ∑n−1 k=1 k Hn−k , 1 which we can write as dn = ∑n−1 k=1 n−k Hk . Since the reciprocals are the building blocks of the harmonic numbers, this sum has an intuitively appealing diagrammatic or pictorial form. For n = 7, we get k 1 2 3 4 5 6 1 6× 1 5× 1 4× 1 3× 1 2× 1 1×

1 1 1 1 1 1

1 2 1 2 1 2 1 2 1 2

1 3 1 3 1 3 1 3

1 4 1 4 1 4

1 5 1 5

1 6

We can use this diagram as an example of transforming summation in a way not yet mentioned in §2.2.1. We let the leading index of summation count neither rows nor columns, but diagonals, beginning at the lower left corner. The first diagonal is 11 × 1, the second is 1 1 1 2 × 1 + 1 × 2 etc. The i-th diagonal is i 1 1 i 1 1 1 def × = Si = ∑ ∑ j + i− j+1 , j i + 1 j=1 j=1 i − j + 1 where the equality is achieved by a partial fraction decomposition. Hence we get: Si =

1 i+1

i

1 1 + j i + 1 j=1

∑

The total sum is (R ⋆ H)n =

i

1

2

i

1 2Hi = . j i +1 j=1

∑ i− j+1 = i+1 ∑ j=1

n−1

n−1

i=1

i=1

2Hi

(2)

∑ Si = ∑ i + 1 = Hn2 − Hn

.

by Exercise 2.44(h). Exercise 2.126 [2+] Find the convolution of the sequence of harmonic numbers with itself.


67

Example 2.127 This example looks like a convolution of binomial coefficients ak = nk , and Fibonacci numbers Fk (see Eq. (1.10) or Appendix D). However, because of the second argument n of the binomial coefficient, it is not quite a convolution. Based on the result (I + E)n F0 = F2n from Exercise 1.28, where I is the identity operator and E is the shift operator, we obtain n n ∑ k Fn−k = ∑ k E n−k F0 = (I + E)nF0 = F2n. k k We consider two sequences of binomial coefficients an = np and bn = dn , n = 0, 1, 2, . . .. If p and d are positive integers, the sequences a = {an } and b = {bn } terminate when n > p and n > d, respectively. The convolution of the two sequences, p d def {cn }n>0 = c = a ⋆ b, where cn = ∑ , k n−k k has a nice expression cn =

∑ i

p+d n

and is known as the famous Vandermonde15 convolution:

p d p+d = , i+m n−i m+n

m, n ∈ N.

(2.74)

Note: Leaving the index i without range specification means it ranges over all possible integer values that contribute to the summation. Normally much information about the other parameters is required to be able to specify this summation completely. For example, if p > m > 0 (notice that m and n must be integers or it all vanishes), then −m 6 i 6 min(n, p − m).

P ROOF : We provide four different proofs that only establish Eq. (2.74) for m = 0. (1) The first is based on the binomial theorem p+d j p k d i p+d p d (1 + x) = (1 + x) (1 + x) = ∑ x =∑ x ×∑ x. j j>0 i>0 i k>0 k We set j = k + i, so that i = j − k, and (1 + x) p+d

" # p k d p d =∑ x ×∑ x j−k = ∑ ∑ x j. k j − k k j − k j k>0 j>k k

We have an equality between two polynomials in x; hence the coefficients of the same powers of x on both sides are equal. (2) The second proof uses induction on p for arbitrary d and n. We are proving for p ∈ N only. We use p = 0 for basis. Then d 0 d ? ∑ i n−i = n i 15 Alexander Vandermonde (1735 – 1796) from France. Eq. (2.74) was first proved by L. Euler and to many mathematicians this convolution is known as the Euler summation formula. Since we shall encounter another important result commonly called by this name, in Chapter 10, we do not follow this usage.

68


holds since only i = 0 contributes to the series. Assume that the claim holds for all values of the induction variable up to p − 1: p−1 d p+d −1 = . ∑ i n−i n i We use the addition formula to do calculations: p d p−1 p−1 d + ∑ i n−i = ∑ i i−1 n−i i i p−1 d p−1 d =∑ +∑ i n−i i−1 n−i i i p+d −1 p−1 d = +∑ n k n − (k + 1) k p+d −1 p+d −1 p+d = + = . n n−1 n (3) The third proof is a combinatorial one. This proof takes a more “intuitive” approach by using the combinatorial interpretation of binomial coefficients: n is the number of ways to pick k items out of a set of n distinct items. k Suppose we have p pennies and d dimes (and they are all well-worn, hence—distinguishable). In how many ways can we pick n coins out of these pennies and dimes? The above interpretation says that the answer is p+d , and now let us obtain it in another n way: First count the number of ways using no pennies: 0p dn . d ; We then add the number of ways using 1 penny: 1p n−1 .. . d We add the number of ways using i pennies, pi n−i ; .. . p+d p d up to n. Thus, =∑ . n i n−i i (4) Our last proof is based on the observation that the numberof northeastern paths (see m+n Example 2.78 on page 51) from the origin to point (m, n) is m+n = . n m

Consider the number of paths from the origin to point (p + d − n, n) (0 6 n 6 p 6 p + d − n); there are p+d such northeastern paths. Draw a straight line with slope −1 so that it is n somewhere between the origin and the point (p + d − n, n). More precisely, we consider the line that goes through the point (p, 0). Then each northeastern path crosses this line at a particular point, say (p − j, j). There is a binomial coefficient pj that says how many such d paths are possible. Another binomial coefficient n− j says how many paths there are from the point (p − j, j) to its final destination (p + d − n, n). Since each path from the origin to


69

(p + d − n, n) goes through some point (p − j, j), the total number of paths is the product of the two binomial coefficients. The total number of northeastern paths from the origin is just the sum of such products with the sum including all the points on the diagonal line where the paths can cross. Thus, we have the formula (2.74) for m = 0. Finally we need to show from this last result that the original convolution (2.74) holds. In the p d p d p+d . Changing we replace n by m + n, so that = result p+d = ∑ ∑ i i m+n−i i i n−i m+n n the index with the substitution j = i − m, we recover the required identity.

Note the remarkable simplification that resulted from letting the dummy index range over all integers! Variants of the Vandermonde convolution are given in Exercises 2.135 through 2.138. The following exercises establish other useful summation formulas.

Exercise 2.128 [3] Let x and y be real numbers and n be a positive integer. Prove the Vandermonde formula: n n k n−k n x y . (2.75) (x + y) = ∑ k=0 k x+k x+m+1 Exercise 2.129 [3] (a) Prove = m ∈ N, x ∈ C. ∑ k m 06k6m m−k (b) Prove = Fm+1 , the Fibonacci number, by showing that this sum satis∑ k 06k6m fies the appropriate initial values and recurrence relation as in Eq. (1.10) on page 16. n k m n−1 Exercise 2.130 [3] Prove ∑ (−1) = (−1) , n, m ∈ N. m 06k6m k

Hint: Use identity (I–5), that is, Eq. (2.64) and Exercise 2.129. Pay attention to what happens when m > n. Note: this identity sometimes is referred to as the “horizontal” summation. Exercise 2.131 [2] Using the result from Exercise 2.130, find the following sum, which involves the harmonic numbers: n n ∑ k (−1)k Hk . k=1 Exercise 2.132 [3] Using the binomial theorem and identity (I–5), Eq. (2.64), prove the summation formula (L.20) of Appendix L. m+k k xn−m ∑ n x = (1 − x)n+1 , n > m ∈ N. What do we get when m > n? k>0

Exercise 2.133 [4] Show the five “inverse relations” given in Appendix A, Eqs. (A.58)– (A.62). They can all be shown using the relations we have discovered so far, but an elegant approach to showing them (and a large number of other relations) is via the methods we develop in Chapter 6. Example 2.134 We are going to show the following convolution-like formula: p d p+d p ∈ N, m, n ∈ Z. ∑ m+i n+i = p−m+n , i From the identity (I–1),

p p = and the Vandermonde convolution, it folm+i p−m−i


70

lows:

∑ i

p d p d p+d =∑ = . m+i n+i p−m−i n+i n+ p−m i

Exercise 2.135 [3] For positive integers p and d, show the following four binomial sums, which are derived by tweaking the Vandermonde convolution identity. p+i m−i p+m+1 (a) ∑ = , p ∈ N, m, n ∈ Z. i n−i n i p p−i d +i p+d +1 (b) ∑ = , n > d, m, p ∈ N. m n m+n+1 i=0 p d +i d −m (c) ∑ (−1)i = (−1)m+p , m, n ∈ Z, d > m, p ∈ N. m+i n n− p i p−i d i m+p d − m − 1 (d) ∑ (−1) = (−1) , n ∈ Z, d, p, m ∈ N. m i−n p−n−m i6p Note: all four identities are valid for arbitrary integers p and d (not necessarily positive), but their derivation require more efforts (see [52]). Exercise 2.136 [2] Show that m

n

i+ j+m+n

∑ ∑ (−1)

i=1 j=1

   0, n−1 m−1 i+ j = 1,  j−1 i−1 i  2,

if |m − n| > 1, if |m − n| = 1, if m = n.

Hint: Use part (c) of Exercise 2.135. Exercise 2.137 Prove the following identities. 2n − 2 j j n k 2n (a) [3] ∑(−4) = (−1) . j k− j k j n r−sj (b) [4] ∑(−1) j = sn , n ∈ N, r, s ∈ R. j n j Hint: Use the binomial expansion for the product of two power functions in part (a) and induction on n in part (b). Exercise 2.138 [4] (a) Show a direct consequence of the Vandermonde convolution formula. a b c a+b+c , for integers k, l, m, n ∈ N and arbitrary numbers a, b, c. ∑ k l m = n k,l,m>0 k+l+m=n

Can you provide a combinatorial interpretation? (b) Part (a) can be generalized to a multinomial Vandermonde convolution. Show that p n− p n ∑ j k−j = k , j where n and p are integers in N, k = (k1 , k2 , . . . , km ) is a nonnegative integer-valued vector of m components (m > 1) which have the sum n = k1 + k2 + · · · + km , and the summation is carried over all vectors j = ( j1 , j2 , . . . , jm ) whose components sum to p = j1 + j2 + · · · + jm .


71

Example 2.139 (Using falling factorials) We start with the surprising evaluation of the following sum, which uses the relations (2.24), page 38, and Eq. (2.37), page 46: n n n 1 n (−1)k 2 n n (2.76) ∑ k x + k = 0 − 1 E + 2 E − · · · + (−1) n E n x k=0 1 = (1 − E)n = (−1)n ∆n (x − 1)−1 = (−1)n (−1)n n! (x − 1)−n−1 x 1 n! , E is the shift operator. = = x(x + 1) · · · (x + n) x x+n n n

x+n n ,

This last expression, 1/x

is known as the Leibniz number. −1 n k x+k Exercise 2.140 [1] Using Eq. (2.76) compute the sum ∑ (−1) . k k k

Example 2.141 (Summation by expansion) A summation technique that appears counterintuitive proceeds as follows: when a complicated sum is not reducible to a familiar form, try replacing some factor by an additional inner summation. Admittedly, n−m+p it requires some experience and proficiency. For example, to m−n+q we “symmetrize” the sum by sum ∑r n+r p−r , which appears rather obscure, p+q r r n n+r replacing p+q with the Vandermonde convolution ∑k q−k p+k and rearrange to obtain

∑ k

n p+k

∑ r

m−n+q r n−m+ p . r q−k p−r

Using Eq. (2.57), we get

m−n+q m−n+k n−m+ p ∑ q−k r−q+k p−r r k n m−n+q m−n+k n−m+ p =∑ . ∑ r−q+k p+k q−k p−r r k

=∑

n p+k

The summation on r can be carried out since it is a Vandermonde convolution. The result is p+k p+k = p+k−q q . Inserting the latter expression, and reusing Eq. (2.57), where the repeated term is p + k, we have

∑ k

m−n+q n n−q n m = , q−k q k+ p−q q p

where the summation over k again was a Vandermonde convolution. Exercise 2.142 [4] Prove the following formulas that are quite similar to the example just shown: k q n n n p q n+ j n n (a) ∑ = . (b) ∑ = . q k − p k q p j j p + q q p j k


72

2.6 Binomial Coefficients and Hypergeometric Functions This section brings a few examples of additional approaches that are useful in evaluating sums that contain binomial coefficients, a task we find anywhere we turn when analyzing algorithms.

When are two series the same? One of the impressions you have probably received from the previous discussions is that binomial coefficients can be transformed in various ways to look very different. This makes the question “How can we determine when two expressions are identical?” hard to answer. When applied to sums, the question tends to be even daunting. For example, Exercise 2.120(b) says k j+1/2 2 j+1 j+1 that the left-hand side, ∑k>0 2k x . We know this is true, k kx , also equals ∑ j>0 j 2 but it is hard to see. The general approach when analysts are confronted by such a confusion is to look for a canonical form. In order to determine whether two sums are equal, we transform both to a canonical form; if both sums are transformed to the same series, they are equal. What is the canonical form for such sums? The answer is: a hypergeometric series (or function). This kind of series appears in various areas of mathematics. We need a little of the standard theory, but should mention that a hypergeometric function can provide very powerful tools for investigations beyond what we provide here. A more detailed exposition is in Graham’s et al. book [56, Chapter 5], and especially in Petkovsek’s et al. book [107]. We use the notation of [56]. Definition 2.143 A hypergeometric series (or function) is a power series (in a variable, say, x) with m upper and n lower parameters16 : ak1 , . . . , akm xk a1 , . . . , am , (2.77) x = F ∑ k k b1 , . . . , bn k>0 b , . . . , bn k! 1

where lower parameters may not be zero or negative integers and the upper parameters are ⊳ nonzero numbers. The rising factorial of a of order k is denoted ak (see Eq. (2.38)). Looking at the definition you may notice that since k! = 1k , we could simply define bn+1 = 1, to replace the factorial. Probably for a historical convention, the factorial in the denominator is kept separately. Once the conventional notation has been adopted, you will be allowed to use the vast amount of literature about various properties of these functions. Note that while some of the a’s may be equal and the same holds for the b’s, we do not keep an a that equals a b. They cancel out, for instance, y xk = ex . (2.78) x =∑ F ( | x) = F k! y k>0

The first function above, F ( | x), is not in conventional form, which insists that both m and n y need to be at least 1. Thus, x , convention dictates to add a use for some reason F if we y,1 ‘1’ in each row and use F 1 x , which is the only time you shall see an a equal to a b. 16 Sometimes

other notations are also used: m Fn (a1 , . . . , am ; b1 , . . . , bn ; x) or just F(a1 , . . . , am ; b1 , . . . , bn ; x).

2.6. BINOMIAL COEFFICIENTS AND HYPERGEOMETRIC FUNCTIONS

73

Can we get from the hypergeometric series a plain geometric series? Yes. 1, 1 1 . x = ∑ xk = F 1−x 1 k>0

Two special cases of such series that have numerous applications are the confluent hypergeometric series and the Gaussian hypergeometric series, given by a+k−1 b+k−1 a a, b ak bk zk def k =∑ zk , (2.79) F z =∑ F k z = M(a, b, z), c+k−1 k b c k>0 k>0 c k! k

respectively.

Exercise 2.144 [2] Identify the standard functions given by the following hypergeometric series: ! 1 1 , 1 z2 −a, 1 ; (c) F 2 3 2 z2 ; −z ; (b) F (a) F 3 − 4 1, 2 1 2 We claim that the hypergeometric series can serve as canonical form for many summations involving binomial coefficients. Given a series U = ∑k>0 uk , how do we find the parameters of the corresponding hypergeometric series?

We need to do two things. The first is simple: find the first nonzero term in the given series and factor it out; we need this, since the first term in a hypergeometric series is always 1. Secondly, we form the signature of the series: uk+1 /uk . To see the reason, let us compute the signature of the hypergeometric series given in Eq. (2.77), which we rewrite as ∑k>0 uk , with uk =

ak1 ,...,akm zk . bk1 ,...,bkn k!

Since ak+1 /ak = k + a, we get uk+1 (k + a1 ) · · · (k + am ) z = . uk (k + b1 ) · · · (k + bn ) k + 1

(2.80)

This is a rational function in k; each parameter appears once, with k; the denominator always has the factor k + 1, and there is a constant, z, that absorbs all else. For example, if we find for the signature of the given sum k3 + 6k2 − k − 30 (k − 2)(k + 3)(k + 5) uk+1 =− =− , 2 uk k + 4k + 4 (k + 2)2 we need to add a factor k + 1 in thedenominator, and then add it also in the numerator, and we −2, 5, 3, 1 find that this is the signature of F − 1 , and hence, U = u0 F −2,2,5,23, 1 − 1 . 2, 2

Therefore, two series with the same u0 and the same signature are equal. While not all series will oblige you by being equivalent to a hypergeometric series—such as ∑ j2.5 —the series j>0

we find in analysis of algorithms usually do. Once the canonical form of an unknown sum has been found, we may try to find out whether it corresponds to any known sum of hypergeometric series. Most of the sums we have shown so far in this chapter can be transformed into sums of hypergeometric series. This looks straightforward, but we need to address a few points about this approach that need attention,

74


as we show in the following example. However, saying that the signature is a rational function in k, the ratio of two fixed polynomials in k, implies that even simple sums, such as ∑nk=1 Hk , have no such canonical form (but Hk itself, seen as a series, has one). Example 2.145 (a) Translate the sum ∑k nk xk = (1 + x)n into the hypergeometric series. n Here the signature is x k+1 / nk , and Exercise 2.105 provides the answer x(n − k)/(k + 1) = k−n k+1 (−x). Hence the parameters are −n above, there are none below. We add 1 to both positions, and the variable is (−x). The first term is 1, and we have found F −n,1 1 −x = (1 + x)n , which fits with the result of Exercise 2.144(a). k+1

(b) The logarithmic function has the Maclaurin expansion ln(1 + x) = ∑k>1 (−1)k xk and −kx = k+1 . This produces a parameter of zero value. Since the corresponding signature is uuk+1 k parameters of a hypergeometric function cannot vanish, we slide this ratio forward by one a factor k + 1 in the denominator. We add and get the signature −(k+1)x k+2 . The signature lacks that factor to both sides and find F 1,2 1 −x = 1x ln(1 + x). d p p+d (c) Finally, consider the Vandermonde convolution: ∑i i+m n−i = m+n . The signature is (i + m − p)(i − n)(i + 1) (p − m − i)(n − i) ui+1 = . = ui (i + 1 + m)(d − n + i + 1) (i + m + 1)(i + d − n + 1)(i + 1)

m−p, 1, −n This suggests the hypergeometric series F m+1,d−n+1 1 ; however, m may not serve as a lower parameter because it can be a negative integer. We resolve this by making the substip d tution j = i + m in the convolution, which then becomes ∑ j j n+m− j . The signature for u j+1 (p− j)(n+m− j) ( j−p)( j−n−m) d . Hence this sum is u j = ( j+1)(d−n−m+ j+1) = ( j+d−n−m+1)( j+1) . The first term is u0 = n+m p+d ( ) −p,−n−m . 1 = n+m F d−n−m+1 d ) (n+m In Exercise 2.135 you were asked to compute a few convolution-like sums based on the Vandermonde convolution. While finding the given sums is a matter of trial and error, we use canonical forms to eliminate guesswork. Let us look at the last part of Exercise 2.135. We c def i need to change the index, i, of summation in the sum S = ∑i6a a−i b i−t (−1) (i is used instead of j to avoid a clash with the notation above) so that it will run over inte positive c (−1)a+k . gers. The change a − i = k does this, and we have the sum S = ∑k>0 bk a−t−k This will produce a lower parameter of 1 − b. Since b > 0, we use a similar shift, k − c a+b+ j . This sum has the signature u j+1 = b = j, to obtain S = ∑ j>0 j+b uj a−t−b− j (−1) b ( j+b+1)( j+t+b−a) c a+b . With the term at j = 0 given by a−t−b (−1) , we find a total of ( j+b+c+t−a+1)( j+1) c b+1,t+b−a a+b F (−1) a−b−t c+t+b−a+1 1 . Comparing this to the result for the convolution—by solving for the values p, m, d, n in terms of a, b, c, t—we find the result offered in the exercise. Exercise 2.146 [3] Find the canonical form for the following series. n−1 rn − k k n+k 2k (−1)k = , m, n ∈ I1 . (b) ∑ r . (a) ∑ m−1 k k+1 k>n n − k k>0 m + 2k m + 1 n m n n n 1 (d) ∑ −k = . (c) Hn = ∑ . 2 2 m+1 k=0 k k=1 k

2.6. BINOMIAL COEFFICIENTS AND HYPERGEOMETRIC FUNCTIONS

75

Example 2.145 has given us two identities for hypergeometric series. One is equivalent to the binomial theorem (2.54), and another is equivalent to the Vandermonde convolution (2.74): p+d (p + d)n −p, − n −n, 1 n n . (2.81) −x = (1 + x) , F F 1 = d = dn d −n+1 1 n

respectively. The latter one is a Gaussian hypergeometric series, as presented in Eq. (2.79), at z = 1. While this relation was only derived for an upper parameter −n, a negative integer, the result holds when n is any complex number, subject to17 ℜ(d + p + 1) > 0, as shown by Gauss. In that case, we would not like to use a binomial coefficient with n as a lower argument. Instead, we utilize the gamma function for natural numbers, r! = Γ(r +1) (r ∈ Z+ ), and we write: a, b Γ(c − a − b)Γ(c) , ℜ(c − a − b) > 0, (2.82) 1 = F Γ(c − a)Γ(c − b) c ¨ formula: which is the Gauss identity. We list one more similar identity, called the Saalschutz a−c b−c n n a, b, − n n n 1 = (a − c) (b − c) = (2.83) F , n ∈ N. −c a+b−c (−c)n (a + b − c)n c, a + b − c − n + 1 n n The parameters here may appear confusing. However, we note that the formula has three upper parameters, one of which needs to be a negative integer, and the sum of the two lower parameters need to exceed the upper ones by 1.

Exercise 2.147 [4] Show that we have already proved the Saalschütz identity by performing the summation in Example 2.141. Note: The standard route to proving this formula passes through a differential equation satisfied by the hypergeometric series, and Euler’s first identity, given in [62, §1.5], a, b c − a − b, 1 c − a, c − b F x =F x F x . c 1 c Exercise 2.148 Prove the following relations. 2k 2(n − k) (a) [2] ∑ = 4n ; k n − k k x+y+k y x x+m y+n (b) [4] ∑ = . k m−k n−k n m k

Hint: part (b) can be done either using the approach of Example 2.141, or using Eq. (2.83).

2.6.1 Abel’s Identity J. Riordan discusses in [115] a surprising generalization of the binomial theorem, called Abel’s identity, after its discoverer N. Abel18 : n (2.84) ∑ k x(x + k)k−1 (y + n − k)n−k = (x + y + n)n. k 17 Here

ℜ stands for real part of a complex number. See section 12.1. is considered the most significant Norwegian mathematician, and an all-time

18 Niels Henryk Abel (1802–1829)

great.


76

It turns out that there is a slightly more general framework in which this identity can be embedded, leading to a natural proof of Eq. (2.84) and several other relations. Numerous consequences of the identity are developed in [115]. The following exercise explores a few of them. n def Exercise 2.149 [2] Define An (x, y; p, q) = ∑ (x + k)k+p (y + n − k)n−k+q . (R1) k k Clearly, we recover the sum above (up to a factor of x) as An (x, y; −1, 0). (a) Show the following relationships that the function satisfies: An (x, y; p, q) = An (y, x; q, p),

(R2)

An (x, y; p, q) = An−1 (x, y + 1; p, q + 1) + An−1 (x + 1, y; p + 1, q),

(R3)

where the former one follows from the addition formula on page 59. (b) By rewriting An (x, y; p, q) as ∑k nk (x + k)(x + k)k+p−1 (y + n − k)n−k+q , show that An (x, y; p, q) = xAn (x, y; p − 1, q) + nAn−1 (x + 1, y; p, q)

= yAn (x, y; p, q − 1) + nAn−1 (x, y + 1; p, q),

(R4) (R5)

where relation (R5) follows from a similar splitting of (y + n − k)n−k+q . Show that Eq. (R5) can also be obtained from Eq. (R4) with the aid of Eq. (R2). (c) Substituting (R3) in (R4), show An (x, y; p, q) = xAn−1 (x, y + 1; p − 1, q + 1) + (x + n)An−1 (x + 1, y; p, q) = (x + n)An (x, y; p − 1, q) − nAn−1 (x, y + 1; p − 1, q + 1).

(R6) (R7)

(d) You have the ingredients needed for a proof of Abel’s identity, Eq. (2.84). Write (R6) def with p = 0, q = −1, and in the left-hand side use (R2). Let tn (x, y) = An (x, y; −1, 0), then we obtain tn (x, y) = ytn−1 (y, x + 1) + (y + n)tn−1 (x, y + 1) (R8) By (R1) you have t0 (x, y) = 1/x and t1 (x, y) = (x + y + 1)/x. Then use (R8) to prove by induction on n that tn (x, y) = (x + y + n)n /x, from which the desired proof follows. (e) Use the result for tn (x, y), together with Eqs. (R3, R2), to show another common form of Abel’s identity: n 1 1 k−1 n−k−1 + (x + y + n)n−1 . (2.85) An (x, y; −1, −1) = ∑ (x + k) (y + n − k) = x y k k (f) Relation (R4) can be iterated by expanding the A into terms of lower order. Show that you get n An (x, y; p.q) = ∑ k!(x + k)An−k (x + k, y; p − 1, q), A0 () ≡ 0 (R9) k k and, in particular, writing (R9) for p = q = 0, substitute the solution for tn (x, y) from part (d), and obtain a formula called Cauchy’s binomial formula: n n n−k An (x, y; 0, 0) = ∑ k!(x + y + n) =∑ (x + k)k (y + n − k)n−k . (2.86) k k k k

2.7. STIRLING APPROXIMATION

77

n 1 k!(x + y + k)(x + y + (g) Use Eq. (2.85) in Eq. (R9) to find An (x, y; 0, −1) = ∑ k y k n)n−k−1 . Using Eq. (R2) for the left-hand side, show that it equals (x + y + n)n /y, and obtain the following two identities n ∑ k! k knn−1−k = nn , k

k−1

∑(−1) k

n knn−1−k = (n − 1)n−1 . k

This concludes our treatment of the binomial coefficients. In the following chapters, especially Chapters 5 and 6, we shall develop tools that enable us to compute additional formulas involving these marvelous numbers.

2.7 Stirling Approximation The following famous result we bring at this time without proof (derived in §10.4). It is an approximation of the factorial, named after James Stirling19 , who discovered it in the 18th century: √ 1 1 139 571 −n n + − − + ··· . (2.87) 1+ n! = n 2π n e 12n 288n2 51840 n3 2488320 n4 An alternative to the expression in Eq. (2.87) is the “bracket inequality:” 1 1 n! 1 n n √ < n exp −n + − < , n exp −n + 12n 360 n3 12n 2π n

(2.88)

which can be written more succinctly as n! = (2π n)1/2 nn exp {−n + θ /(12n)} ,

(2.89)

where θ = θ (n) satisfies 0 < θ < 1. The sum of decreasing terms in the brackets of Eq. (2.87) is a “multiplicative correction factor.” Further details about this formula, and additional terms are given in Appendix G, through the relation between the factorial and the Gamma function. With this formula, we can find an approximation for binomial coefficients: √ nn 2π ne−n n n! p · · · · · · (· · · · · ·) = √ = −k n−k −(n−k) k k! (n − k)! k 2π ke (n − k) k 2π (n − k)e r n−k n k n 1 1 1 1 n − − 1+ + ··· (2.90) = 2π k(n − k) k n−k 12 n k n − k r k 1 1 1 1 n exp nH − − 1+ + ··· , (2.91) = 2π k(n − k) n 12 n k n − k 19 James Stirling (1692 – 1770) was the greatest Scottish mathematician. The formula (2.87) was apparently found by Abraham de Moivre. See an interesting brief review of the historical facts surrounding the discovery of Stirling’s formula in [132].


78

where H is the entropy function: H(x) = −x ln x − (1 − x) ln(1 − x), for 0 6 x 6 1 . For consistency, H(x) is defined to equal 0 at the endpoints. Exercise 2.150 [3] Certain values of binomial coefficients keep recurring in calculations. Show the following special cases. 4n 2n 2n 1/2 2n √ 2 = ≈ 2π n2 n nπ

1/2 n 2 or ∼ 2n as n → ∞. (2.92) (a) nπ ⌊n/2⌋ 4n (b) Find a similar first-order approximation for . n Exercise 2.151 [3] Prove inequalities: n k n 2n nn 4n 4n , (b) < < k , 0 < k < n. 0, let Vn (r) denote the n-dimensional volume of {(x1 , x2 , . . . , xn ) | x21 + x22 + · · · + x2n 6 r2 }, the n-dimensional ball of radius r in Rn . Then rn π n/2 . For V0 (r) = 1, V1 (r) = 2r, V2 (r) = π r2 , V3 (r) = 4π r3 /3, and in general Vn (r) = Γ(1 + n/2) what values of r does the series ∑∞ n=0 Vn (r) converge? Answer the same question for the n-area σn (r) = (n + 1)Vn+1 (r)/r of the n-dimensional sphere x21 + x22 + · · · + x2n = r2 . Exercise 2.154 [2] Show that nk 6 n (n − 1)k−1 /k!. How does this bound compare with the right-hand side bound in Exercise 2.151(b)?

Chapter 3

Probability Projections, estimates, and predictions have become a common part of our life. A weather forecast says that there is a 60% chance of rain tomorrow. A TV announcer tells us that gross sales of American products will increase by 3% this year. A Gallup poll claims that 45% of Americans are afraid to go out at night because of fear of crime. A health-news reporter states that a smoker has a 72.33% higher chance of getting cancer than a non-smoker or a tuba player. A college student asks an instructor about his chances of getting an A, though he did not do well on the midterm exam. A system administrator has determined that there is 7% chance of at least one of his operating systems crashing every month. An Internet provider estimates that 0.031% of transferred data is corrupted . . . . The 20th century has seen large improvements in productivity and the quality of manufacturing in industry in the Western world. Much of this success has been attributed to the use of probabilistic and statistical methods. Probability theory is the field of mathematics that measures the likelihood of events. Anyone playing with dice or cards quickly develops some intuitive grasp of probability. Such notions of probability and randomness are as old as civilization itself. Tomb inscriptions and other archaeological evidence reveal that as early as 3500 B.C. Egyptians were using astragalus (four-sided ankle bones of animals) as a primitive prototype of playing dice. The earliest known six-sided dice are from ancient Mesopotamia, now northern Iraq, and date to approximately 2750 B.C. An awakening of interest in the mathematics of gambling occurred during the mid-seventeenth century. In the nineteenth century, probability was recognized as a useful tool in various applications in astronomy, physics, actuarial mathematics, genetics, and biology. At the International Congress of Mathematicians held in Paris in 1900, David Hilbert presented a famous list of problems. He claimed that their solutions would be essential for the further development of mathematics. Since that time, finding a solution of a Hilbert problem is considered by many as important in mathematics as winning a Nobel prize in the natural sciences1 . In 1933, Andrei N. Kolmogorov2 from Moscow University solved Hilbert’s sixth problem: he 1 No

Nobel prize is awarded in mathematics. Nikolaevich Kolmogorov (1903 – 1987) was one of the most famous Russian mathematicians of the 20th century. He tragically lost both of his parents at a young age and he worked for a while as a conductor on the railway after school graduation. A. Kolmogorov made a major contribution to the sixth Hilbert problem, and he completely solved Hilbert’s Thirteenth Problem in 1957, when he showed that Hilbert was wrong in asking 2 Andrei

79

80

CHAPTER 3. PROBABILITY

presented a universally accepted axiomatization of probability theory, which serves like Euclid’s axioms for geometry.

3.1 Set Operations A set is a basic concept in mathematics and has no direct definition. Its meaning is usually conveyed via examples: it is a collection of objects, or elements. We could talk about the set of students taking a graduate course in computer science; or the set of fish in the Atlantic Ocean; or the set of hockey players in the National Hockey League; or the set of state capitals in the United States, and so on. At this level, the only requirement is that a set be well defined, namely, its definition should allow us to provide a yes/no answer to the question: does this “thing” belong (synonymous with “is a member of” or “is an element of”) to this set? In this book, we mostly use finite sets, which consist of a finite number of elements. Such sets are of particular importance in computer science because computers are essentially finite— they operate on finite arrays of characters. Still, it is necessary (and possibly surprisingly easier) sometimes to use infinite sets, and, in particular, a few numerical ones, that we defined on page 12, Z, N, and Ik (in particular I0 , which is identical with N). The symbol ∈ means is an element of and ∈ / means the opposite claim, namely, is not an element of. For example, let W be the set of letters in the word worcester, then we write r ∈ W and u ∈ / W . The set W consists of seven letters: W = { w, o, r, c, e, s, t }. The cardinal number (or cardinality) of a set A is the number of elements in the set and is denoted by |A| or n(A). Thus |W | = 7.

Two sets are equal if they contain exactly the same elements. The order in which the elements are listed does not matter. The set with no elements is called the empty set and is denoted by ∅, so |∅| = 0.

When dealing with any problem, the universal set arises naturally. It is the set of all existing elements used in the problem and is denoted in the probability theory usually by Ω. For example, if we study the performance of some students at a particular university, we will take all the students at the university as our universal set. And if we want to include nationwide comparisons, a suitable universal set is the set of all students in the country: the universal set depends on how a given problem is treated. We say that B is a subset of A, denoted by B ⊆ A, if every element from B belongs also to the set A. As a special case, the empty set ∅ ⊆ A for any set A. The set B is said to be a proper subset of A, a fact denoted by B ⊂ A, if B is a subset of A and they are not equal, that is, A contains an element which is not in B. In Chapter 1, we introduced analysis of algorithms through considerations of procedures to test the subset relation. To visualize the relationship between sets and their elements, it is convenient to use Venn diagrams3 . In a Venn diagram, the universal set Ω is usually denoted as a rectangle. Other sets are drawn as closed curves inside this rectangle. Elements of a set are drawn as dots inside the sets (see Fig. 81). for a proof that there exist continuous functions of three variables that could not be represented by continuous functions of two variables. 3 John Venn (1834–1923) is considered by many to be one of the originators of modern symbolic logic. He was not the first person to use the diagrams that now bear his name, but he developed a consistent, systematic approach to using geometric figures in logic.

3.1. SET OPERATIONS

81

The intersection of two sets A and B (the symbol ∩ may be dropped) is A ∩ B = AB = { x | x ∈ A

and

x ∈ B }.

A B

A

Ω

B

Ω

a – With intersecting sets

b – With disjoint sets

Figure 81: Venn diagrams with two sets A and B.

The union of two sets A and B is A ∪ B = {x | x ∈ A

or x ∈ B }.

The complement of a set A, denoted by A′ (read “A prime”) or AC (read “A complement”) or A, is / A }. A′ = AC = A = { x | x ∈ Ω and x ∈ For any finite set A, we have the cardinality formula: |A| + |A′ | = |Ω|. For any two sets A and B, the number of elements in their union is |A ∪ B| = |A| + |B| − |A ∩ B|.

(3.1)

Two sets A and B are called disjoint or mutually exclusive if A ∩ B = ∅, or |A ∩ B| = 0. Thus, for two disjoint sets A and B, we have |A ∪ B| = |A| + |B|,

the sets A and B are disjoint, that is, AB = ∅.

Note that this is a restatement of the Rule of Sums of §2.3. The relationships (A ∪ B)′ = A′ ∩ B′

and

(A ∩ B)′ = A′ ∪ B′

(3.2)

are known as De Morgan Laws4 . We summarize the properties of intersection and union operations: 4 Augustin De Morgan (1806–1871)

was born blind in one eye in Madras, India. In 1828 he became a professor of mathematics at the University of London (later renamed University College) and after a while resigned his professorship as a protest against religious bias. De Morgan’s works include books and papers on symbolic logic, algebra, calculus, and probability. In 1845, he suggested using a slanted line to represent a fraction, such as 1/2 or 5/7.


82

• A ∪ B = B ∪ A,

AB = BA

• A ∪ (B ∪C) = (A ∪ B) ∪C,

(commutativity), A(BC) = (AB)C

(associativity),

• A(B ∪C) = AB ∪ AC,

A ∪ (BC) = (A ∪ B)(A ∪C)

• A ∪ A = A,

(idempotency).

AA = A

(distributivity),

3.2 Sample Space and Random Variables We begin with examples of probability models to illustrate the basic concepts of the theory. The main ingredients of probability theory are described by the terms sample space, outcome, event, experiment, probability of event, and random variable. An experiment or trial is any procedure or observation or measurement that can be repeated any number of times and yields a result. An experiment may consist of tossing a coin or checking whether a switch is turned on or off. A quality-control inspector tests each golf ball on the production line to determine whether it is good or defective. An experiment may consist of a very complicated process of counting function points in a program or determining the density of a material. The result obtained from an experiment is called the outcome of the experiment. The set of all possible outcomes of an experiment is the sample space associated with the trial. Such an outcome is called an element or member of the sample space, or a simple event. Definition 3.1 An experiment (or trial) is a procedure that results in exactly one of possibly many observations. One observation is called an outcome of the experiment. The union of all possible outcomes of an experiment is the sample space (of the experiment), usually denoted by Ω. ⊳ Example 3.2 The sample space of inspecting a golf ball is Ω = {good, defective}. The possible outcomes of tossing a coin are heads or tails. The sample space is then Ω = {H, T}, where H and T stand for heads and tails, respectively. Tossing a coin twice leads to the sample space: Ω = { HH, HT, TH, TT }. Example 3.3 Another example of such a binary sample space is determined by the birth of a baby: Ω = { Boy, Girl }. Example 3.4 Consider the experiment of rolling a die and observing the number that shows on the top face. The sample space is Ω = {1, 2, 3, 4, 5, 6} . If we only observe whether the number is even or odd, the sample space becomes Ω = {even, odd} .

If the sample space has a finite number of elements, we may list the members separately by commas and enclosed in braces. In some experiments, it is helpful to list the elements of the sample space by means of a tree diagram or Venn diagram.

3.2. SAMPLE SPACE AND RANDOM VARIABLES

83

T H

T

TTT

TTH

THT

THH

HTT

HTH

HHT

HHH

T

T H

H

T

T

H

H T

H

H

Figure 83a: Tree and Venn diagrams for Example 3.5. Example 3.5 Consider the experiment of tossing a fair coin three times and counting the number of heads (H) and tails (T). There are eight possible outcomes: Ω = { HHH, HHT, HTH, THH, THT, TTH, HTT, TTT }. Example 3.6 Consider the unusual experiment of flipping a coin such that if the coin shows tails, we flip it second time. If head occurs, we roll a die instead. The sample space, Ω = { TT, TH, H1, H2, H3, H4, H5, H6 } can also be shown via a tree diagram below.

H

1 H1

2 H2

3

4

H3

H4

T

5 H5

6 H6

H TH

T TT

Figure 83b: Tree diagram for Example 3.6. In this book, we consider mostly finite sample spaces with finite cardinalities. Sometimes it is useful to consider a countable sample space. This kind of set allows for its elements to have a one-to-one correspondence with the set N (see §12.1). For example, the sample space

84


of flipping a coin until a tails appears consists of infinitely many elements: Ω = { T, HT, HHT, HHHT, . . .}. Even larger sample spaces are possible; the typical example is a sample space that is a segment of the real line—a continuous interval. We do not frequently need such uncountable sample spaces in analysis of algorithms, but exceptions exist (in bin packing problems, optimization and others). Such uncountable sample spaces are proved to be useful in asymptotic analysis (see §10.7). In previous Examples 3.2 – 3.6, we considered experiments that result in repeated outcomes. In this case we say that we produce sampling with replacement. For instance, by tossing a coin more than two times we observe multiple occurrence of tails and heads. If we keep track on outcomes, the experiment of flipping a coin n times can be presented as a sequence of H’s and T’s of length n. For example, the sequence HHTH indicates that first two tosses resulted in heads followed by tails and then again by heads. So order in which we observed these outcomes matters. In general, we obtain an ordered sample, which is convenient to present in the form: ha1 , a2 , . . . , an i, where ai is the outcome on the ith experiment. There are mn such outcomes. We will use the notation {a1 , . . . , an } for unordered samples with replacement. The number of unordered samples is calculated in §7.3.1. For ordered samples without replacement (called permutations), the sample space Ω = {ha1 , a2 , . . . , an i : ai 6= a j }

space for unconsists of all n-permutations of the set of size m: mn ; whereas the sample m ordered sampling without replacement (called combinations) contains n outcomes. The common treatment of sampling procedure is the occupancy placement—the topic covered in §8.2. Definition 3.7 A subset of a sample space is called an event. An event that includes a single outcome of an experiment is called a simple event. A compound event is a collection of one or more outcomes for an experiment. The set of possible events, usually denoted by Σ, needs to satisfy the following three postulates: Σ1 Both the sample space, Ω, and the empty set, ∅, are in Σ. Σ2 If a finite or countable number of events ω1 , ω2 , . . . belong to Σ, then their union and their intersection belongs to Σ.

Σ3 If the event ω belongs to Σ, then its complement, ω ′ , belongs to Σ.

⊳

Note: A set that satisfies the three requirements above is called an algebra. Therefore, Σ is sometimes called the event algebra. This set needs not be the power set of the sample space, but can be “coarser.” From this definition, it follows that a not arbitrary collection of subsets from a sample space, Ω, can be considered as a set of events, but only those that satisfy postulates Σ1—Σ3. If a sample space is a discrete sample space (finite or countable), any subset of it may be considered an event. If a sample space is not countable, not every subset of Ω can be an event. In any case, the whole sample space Ω is an event, sometimes called the sure or certain event. In probability theory, the empty set ∅ is called the impossible event.

3.2. SAMPLE SPACE AND RANDOM VARIABLES

85

Example 3.8 A two-person committee is selected from a group of students. We observe whether the person selected each time is a man or a woman. M Let A be the compound event that at least one woman is selected. Hence the sample space and the event A are Ω = { MM, MW, W M, WW } W and A = { MW,W M, WW } (shaded), respectively.

M

W

MM

WW

WM

WW

Example 3.9 For the experiment of rolling two fair but different dice, the outcome set consists of 36 possible pairs of numbers from [1..6]: (1, 1) (2, 1) (3, 1) (4, 1) (5, 1) (6, 1)

(1, 2) (2, 2) (3, 2) (4, 2) (5, 2) (6, 2)

(1, 3) (2, 3) (3, 3) (4, 3) (5, 3) (6, 3)

(1, 4) (2, 4) (3, 4) (4, 4) (5, 4) (6, 4)

(1, 5) (2, 5) (3, 5) (4, 5) (5, 5) (6, 5)

(1, 6) (2, 6) (3, 6) (4, 6) (5, 6) (6, 6)

Let the event ω consist of the rolls with a total of 6 points. Then

ω = { (1, 5), (2, 4), (3, 3), (4, 2), (5, 1) } and its cardinal number is |ω | = 5.

We now come to the key concept of applied probability: Definition 3.10 A random variable is a numerical-valued5 function defined on a set of events. A random variable that has finite or at most a countable number of possible outcomes is called a discrete random variable. ⊳ In many problems, discrete random variables represent count data (as the following example shows). There is no mathematical restriction that discrete random variables only have outcomes at integers, but in practice this is usually what makes sense. Some experiments produce outcomes that are numerical values, such as rolling a die or measuring a temperature. The random variable that takes on these values is called the natural random variable for this sample space. Example 3.11 For the experiment of tossing a fair coin three times, let X be the number of heads seen. The following table defines the values of the random variable X : Outcomes Values of X

HHH HHT HTH THH TTH THT HTT TTT 3

2

2

2

1

1

1

0

Exercise 3.12 [2] The following list gives examples of possible discrete random variables: 5 In

some areas of probability applications, a random variable is considered to be a function from the algebra of events into a space other than the set of real numbers.


86

1. The number of vehicles owned by a family. 2. The number of accidents that occurred on a certain road last Tuesday. 3. The number of customers visiting a bank during one hour. 4. The number of messages arriving at an Internet provider during the noon hour. 5. The number of cars sold at a dealership during last July. 6. The number of tails obtained in four tosses of a coin. 7. The number of complaints received at the ombudsman’s office. 8. The number of students in this class. For each entry in the list, complete the definition of a random variable by giving the sample space and the function.

3.3 Calculating Probabilities We have seen two ingredients used to define a probability space: the sample space Ω and a set of events Σ defined over Ω. Now we are in position to add the third and last ingredient. Definition 3.13 A probability or probability measure is a real-valued function Pr : Σ → [0, 1] over some event algebra Σ that satisfies the following properties. Property 1: The probability of an event ω ∈ Σ is a real nonnegative number: Pr(ω ) > 0. Property 2: Pr(Ω) = 1. Property 3: If ω1 , ω2 , ω3 , . . . is a list of disjoint events from Σ, then Pr[ω1 ∪ ω2 ∪ ω3 ∪ · · · ] = Pr[ω1 ] + Pr[ω2 ] + Pr[ω3 ] + · · · . This holds for finite or countable unions.

(3.3) ⊳

Definition 3.14 A probability space is a triplet (Ω, Σ, Pr) that consists of Ω, a sample space, Σ, the set of all events in Ω, and a probability measure, Pr, assigning the probabilities to the events. ⊳ As a convention, the probability measure (or simply probability) of any event ω ∈ Σ is normally written as Pr[ω ]. The letter P is used for other purposes, but not for the probability measure. This letter could be used to denote the probability of a particular event, for example, the equation P = Pr[ω ] means that P is the numerical value of the probability of the event ω. For any event, the probability is between 0 and 1 inclusive, that is, 0 6 Pr[ω ] 6 1. The probability of the sample space Ω is 1, Pr[Ω] = 1, and the probability of the null set (empty set) is 0, Pr[∅] = 0. But a probability of zero does not mean impossibility. For example, in the experiment “choose an arbitrary integer,” picking any specified integer has probability zero.

3.3. CALCULATING PROBABILITIES

87

While the definition appears pedantic, we shall see when analyzing algorithms that ω the various parts map naturally to aspects ω 1 of the algorithm and the context in which 3 5 the algorithm operates. In formulating a probabilistic model for a specific sit6 uation, the construction of the sample 2 4 ω space and the algebra of events is usuΩ ally straightforward. If the number of outcomes is finite or countable, one norFigure 87: Venn diagram for Example 3.15. mally takes the algebra of events to be the set of all subsets of a sample space. Difficulties may arise in assigning probabilities to the sample points so that the properties of the probability measure are satisfied. In many actual cases, it is possible to find relations or symmetry between outcomes, which leads to unique definition of the probabilities of events. For the time being, we illustrate these concepts using immediately familiar examples. 2

1

3

Example 3.15 If a die is rolled, the experiment is reading the value on the top face. The sample space is Ω = { 1, 2, 3, 4, 5, 6 }. Let us consider three events:

ω1 = { 1 } “ one comes up,” ω2 = { 1, 3, 5 } “an odd number comes up,” ω3 = { 2, 3, 4, 5 } “a number between 2 and 5 inclusive comes up.” If the die is fair, the outcomes are equally likely. There are six possible outcomes, so the number 1 should come up about once out of every six rolls. The probability of an event is a measure of the likelihood that the event will occur. That is, the probability of event ω1 is 1/6. In this fraction, the 1 is the number of elements in the event ω1 = { 1 }. The 6 in the denominator is the number of elements in Ω = { 1, 2, 3, 4, 5, 6 }. Similarly, the probability of the event ω2 = { 2, 4, 6 } is Pr[ω2 ] =

1 1 1 3 1 + + = = . 6 6 6 6 2

Again, we see the number of elements in event ω2 given by the numerator 3. For event ω3 = { 2, 3, 4, 5 }, we have 4 2 Pr[ω3 ] = = . 6 3 The example demonstrates the simplest way to assign probabilities based on the Principle of Indifference. This principle says that if all outcomes of an experiment with finite sample space are equally likely, then the probability of an event ω is the number of outcomes in the event divided by the number of outcomes in the sample space,

Pr[ω ] =

| ω | number of favorable outcomes = , |Ω| total number of outcomes

(3.4)

where |ω | is the cardinality of the event ω and |Ω| is the cardinality of the sample space.


88

Consider the experiment of tossing a fair coin three times. We might want to find the probability of the event that the first and third tosses agree. This occurs if and only if we get one of the four outcomes: HHH, HTH, TTT, and THT. So its probability becomes 1 1 1 1 1 + + + = . 8 8 8 8 2 Exercise 3.16 [2+] (Galileo problem) Galileo Galilei is said to have been presented around 1613 with the following question. Both 10 and 12 have six partitions into numbers not exceeding 6. Gamblers have noticed that when three fair dice are rolled, getting a sum of 10 is likelier than getting a sum of 12. Why? There are many situations in which the possible outcomes cannot be considered equally likely. For example, the process of tossing a thumbtack instead of a coin results in two asymmetrical outcomes. A thumbtack can lad either on its flat end (head) or it can land with the edge of the flat end and the needle touching the ground. In many complicated cases, we do not know how to determine the probabilities a priori, or from theoretical considerations. However, we can always use the empirical approach and give a relative frequency interpretation. According to this, the probability of an event is the proportion of a number of times the event occurs in a long run of experiments. This relationship between probabilities and relative frequencies is the intuition behind the mentioned axiomatization of A. Kolmogorov and is embedded in the following result6 : Theorem 3.17 (Law of Large Numbers) If an experiment is repeated a large number of times, the relative frequency of an event converges (in some sense) to the probability of that event. ⊳ No systematic method is known for assigning probabilities to outcomes that are not assumed to be equally likely. In practical applications of the relative frequency approach, we actually don’t know the exact value of the probabilities. So we use the ≈ symbol to assign the probability of an event because we cannot be certain that the relative frequency is exactly equal to the probability regardless of how many trials are preformed. If, for instance, one observed ten thousand data transmissions through a particular channel and detected 27 failures then the relative frequency interpretation suggests that Pr[failure] ≈

27 = 0.0027 . 10, 000

Let us consider an example based on the work of Gregor Mendel7 . In his studies of flower colors, he pollinated peas until he produced pure-red plants (plants that would only produce red-flowering offspring) and pure-white plants. Mendel then cross-hybridized this first generation of pure-reds and pure-whites and obtained a second generation that showed only red flowers. Finally, he hybridized this second generation plants with itself. The result was a crop of approximately three-fourths red flowers, and one-fourth that were white. 6 There are actually several theorems to this effect, all called laws of large numbers. They are treated in a uniform manner in §10.7.1. 7 Gregor Johann Mendel (1822 – 1884) was a monk and a prominent Austrian botanist.


89

To explain these results, Mendel performed a remarkable leap of imagination, and claimed the following: • There exists a “gene”8 responsible for flower color, and it comes in two types: red and white. • Each pea plant carries two color genes—one from each parent—the two may be of the same type or mixed. • If a plant has either two red genes or two white genes the plant shows only red flowers or only white flowers, respectively. • The two types have another difference: the red-flowering type is “dominant” and the whiteflowering gene is “recessive” (explained below). • He also made a probabilistic assumption: which of its two genes a parent passes on to its offspring is a matter of chance, and each is equally likely. Geneticists visualize such information with a Punnett square (named after British geneticist Reginald C. Punnett). We use R to stand for the red type and w to stand for white (capitalization denotes dominance, and lower-case letters are used for recessive genes). The outcomes of cross-fertilization can then be presented as the following table. The sample space has four outcomes: R w Ω = {(R, R), (R, w), (w, R), (w, w)}. Only one of the four possible outcomes, ω1 = (w, w), results in a white-flowering plant. R (R,R) (R,w) Thus, assuming that without interference the types are uniformly w (w,R) (w,w) distributed, |ω1 | 1 = . Pr[ω1 ] = |Ω| 4 The other three outcomes constitute the event ω2 = { (R, R) (R, w), (w, R) }, which results in a red-flowering plant because red dominates white. Its probability is |ω2 | 3 = . |Ω| 4 Therefore, we should expect the actual frequency of red-flowering plants to be close to 3/4 and the actual frequency of white-flowering plants to be close to 1/4. We come across a more complicated case by considering chromosomes of human beings, who normally receive 23 chromosomes from each parent. For simplicity, a portion of each chromosome is a pair that we call a gene. The material in the two locations that make up a gene on the pair of chromosomes comes in forms called alleles. Each distinct combination of alleles (one on each chromosome) is called a genotype. For example, the gene for human blood type consists of a pair of alleles chosen from the three alleles commonly called O, A, and B (order does not matter). Pr[ω2 ] =

Hair color is determined by two pairs of genes, one pair determines placement on a blondbrown-black spectrum and one pair that determines the presence (R+ ) or absence (R− ) of red pigment. These two pairs of genes are independent. Melanin is a brown pigment that affects the color of hair (as well as that of eyes and skin). This gene, which controls the brown hair colors, has three forms, each traditionally labeled with an M (for melanin): M Bd , or blond (a light melanin deposit); M Bw , or brown (a medium melanin deposit); and M Bk , or black (a heavy melanin deposit). Everyone has two of these genes, whose combination determines the brown aspect of hair color. 8 A term that has little in common with the modern meaning of the word and should be read to mean “a carrier of hereditary properties.”


90

Exercise 3.18 [2] A couple is going to have a child. The woman has chestnut hair (M Bd , M Bk and R+ , R− ), and the man has shiny dark brown hair (M Bw , M Bk and R+ , R− ). Find the probability of each possible hair color for their child. Let us look more closely at how the probability of a composite event arises. For an event ω , let ω ′ denote the complementary event, that is, Ω = ω ∪ ω ′ . The events ω and ω ′ are mutually exclusive: ω ∩ ω ′ = ∅. The cardinality formula |Ω| = |ω | + |ω ′ | leads to 1 = Pr[Ω] = Pr[ω ] + Pr[ω ′ ] and

Pr[ω ′ ] = 1 − Pr[ω ].

(3.5)

This simple relation is the basis of many calculations! From the formula (3.5), we find Pr[ω1 ∪ ω2 ] = Pr[ω1 ] + Pr[ω2 ] − Pr[ω1 ∩ ω2 ].

(3.6)

Similarly, for three events ω1 , ω2 , and ω3 we have Pr[ω1 ∪ ω2 ∪ ω3 ] = Pr[ω1 ] + Pr[ω2 ] + Pr[ω3 ] − Pr[ω1 ∩ ω2 ]

− Pr[ω1 ∩ ω3 ] − Pr[ω2 ∩ ω3 ] + Pr[ω1 ∩ ω2 ∩ ω3 ].

(3.7)

It is not hard to extend relation (3.7) and derive the general formula, for n events. The corresponding theorem, called the Principle of Inclusion and Exclusion, is presented in §8.4. Example 3.19 (The birthday problem or birthday paradox) What is the probability that among n people at least two share a birthday? This problem has many other interpretations, some of them are discussed in Exercise 3.79 and §7.6.4, §8.2.

Solution. For simplicity, assume that the group was not pre-selected for their birth dates, that there are 365 possible birthdays (ignoring the existence of leap years with February 29), and that all birthdays are equally likely. The sample space is the set of all possible lists of n birthdays: {hb1 , b2 , . . . , bn i | bi ∈ [1..365]}. (In §7.6.4 we view this sample space as consisting of n-letter words from an alphabet of 365 letters.) This is an example of sampling with replacement when order matters—all people in the group are different and therefore can be ordered. There are 365n elements in this sample space! Clearly direct counting is infeasible, and we need the tools developed in Chapter 2. Let the event A be the set of lists of n birthdays in which at least two of those birthdays are the same. Then the complementary event A′ = {hb1 , b2 , . . . , bn i | bi 6= b j for all i 6= j} is the set of all lists of n birthdays in which no two of those birthdays are the same. Life is such that we frequently find ourselves trying to compute the probability of an event, and it is the complementary event which is much easier to tackle. To calculate the number of outcomes in the event A′ we note that there are 365 choices for the first person’s birthday to be selected, 364 choices for the second person, 363 for the third person, and so on. Thus, the cardinality of the complement A′ (the number of elements in it) is 365·364·363·· · · (365− n+ 1) = 365n . Then the probability of A becomes Pr[A] = 1 − Pr[A′ ] = 1 −

365n . 365n

The following table gives the probability of A for some values of n:

3.3. CALCULATING PROBABILITIES n Pr[A]

91

10

20

22

23

30

40

50

60

0.1170

0.4114

0.4757

0.5073

0.7063

0.8912

0.9704

0.9941

Most people find it surprising that the probability of a birthday repetition among 23 dates is larger than 0.5, and the problem is often called “the birthday paradox.” Consider the event B that a particular person (who may be you) shares the same birthday with somebody from a group of n people, then the probability of the complementary event B′ is (1 − 1/365)n = (364/365)n . For instance, in the group of n = 40 people, the probability of B′ is (364/365)40 ≈ 0.896 that none of the 40 persons shares your birthday. However, in a group of 40 people there is a good chance (with probability 0.8912) that at least two persons have the same birthday. The reason the birthday paradox works is that we are not just looking for matches between one fixed birthday, such as yours, and the others. We are looking for matches between any two birthdays in the group. You might be wondering, with respect to the birthday problem, why we constructed a sample space that involved the number of people, n. It seems that the natural sample space for this problem is just the set of 365 possible birth dates, and each person provides one trial that yields an outcome from this space. This simpler-looking approach would lead to a more involved discussion than we needed. Exercise 3.20 The birthday problem can be reformulated to refer not to a day in the year, but to the month in which the birthday occurs (or the prevailing sign of the zodiac, for the so-inclined). How large a group of people (with independent personal histories) do you need assemble, for the likelihood of at least two matching birth-months to exceed 0.5, assuming each month is equally likely, at 1/12? Exercise 3.21 [2+] To hire two new faculty members at a university, interviews are conducted one by one until two consecutive applicants with the same birthday are found. (a) What is the probability, pn , that the search process requires n or more interviews? (b) What is the value of n for which pn equals at most 1/2? Example 3.22 (A lottery problem) Consider a lottery that issued m tickets, numbered 1, . . . , m, but only n of them are winning tickets (naturally, n < m). Suppose you bought k tickets, what is the probability of winning at least one prize?

Solution. Since the order in which the tickets were bought plays no role, the sample space has the form: Ω = ω = {t1 ,t2 , . . . ,tk } : ti 6= t j for i 6= j, ti ∈ [1..m] . This space has mk elements. Now let A bethe event that there is no prize in the set of tickets purchased by you. Its cardinality is m−n because sampling was taken from the set of all k “unlucky” tickets—there are m − n of them. Therefore m−n . m (m − n)k , Pr[A] = = mk k k and consequently

n n (m − n)k n 1− ··· 1− = 1− . Pr[A] = 1 − Pr[A] = 1 − 1 − m m−1 m−k+1 mk

92


For example, if m = 105 tickets are raffled, carrying 500 prizes, and you buy two of them, your likelihood of winning is quite modest, at 0.0099750 . . . . Example 3.23 Again we consider successive tosses of a true coin. What is the probability we see the second heads before we see the third tails?

Solution: You should realize, after some thought, that the event we are considering is determined within the first four tosses. For the sequence of tosses to satisfy the condition (second H before third T), we must observe exactly one of the following prefixes: HH, THH, HTH, TTHH, THTH, HTTH. No other initial sequence is right. The probability of starting with any particular k-long sequence is (1/2)k . The above set of sequences is prefix-free (no sequence in the set is a prefix of another one in the set), hence 1 11 = 16 = 0.6875. the probability of the event we desire is 41 + 2 × 81 + 3 × 16

Exercise 3.24 [3] There are two ways to generalize the last example, and both are important in practice. (a) What is the probability of seeing the nth tails (T) before we see the mth heads (H)? (b) The same question as in part (a), but now the coin is no longer assumed to be true: Pr[H] = p; Pr[T] = q = 1 − p, for some 0 < p < 1.

Exercise 3.25 [4] In repeated tosses of a biased coin (Pr[heads] = p), what is the probability that n consecutive heads appear before m consecutive tails? The problem is considered later in Exercise 9.81, page 526. Example 3.26 (Comparing polynomials) p Given two polynomials, say P(x) = ∑k=0 ak xk , and Q(x) = ∑qj=0 b j x j , we are asked whether they are identical. The answer is immediate: Yes, if the {ak } are equal to the {b j } for all equal indices. The situation gets somewhat more complicated (and algorithmically more interesting) if they are presented in different forms. For example, suppose that one polynomial is given in the Horner form, Q(x) = c0 + x (c1 + x (c2 + · · · + xcq ) · · ·), but the other is presented either in a p p (x − tr ). Clearly, any ak xk or in a factored-out form, P(x) = ∏r=1 canonical form P(x) = ∑k=0 polynomial representation can be transformed into another one and then the criterion of equal coefficients answers the question. But can the equality of polynomials be determined without such transformation? It is known that finding the factored form can actually be quite hard! Here is a different, probabilistic approach: pick a numerical value for x, call it z, and calculate D(z) = P(z) − Q(z). If the result is zero, declare the polynomials equal, if it is nonzero, declare them unequal. How good is this algorithm? If D(z) 6= 0, the polynomials are surely unequal. Hence the result unequal is always correct. But we might be so unlucky as to have picked a value z which is a root of the polynomial D(z), and then the result equal would be in error. What can we say about the probability of making such a mistake? In general, not much. But there are numerous situations where we can. Suppose we know that all the roots of the polynomials here are integers (this would be the case, for example, when all the coefficients tr in the factored-out representation are integers). Further, suppose we can assume, from knowing the context of the problem, that all the roots must be in the range [−n, n], and the degree of the difference polynomial D(z) is known to be at most max(p, q).


93

Denote the degree of the polynomial D(z) by d. Then D(z) has at most d different roots, and if we choose z to be an integer in the specified range, the probability of mistake is at most d/(2n + 1). We can reduce this probability to zero in this case since, if we know that D(z) may have only integer roots, picking z ∈ / Z guarantees a correct result (up to some unlucky numerical instability). If additional information about the polynomial D(x), except that it has degree d, is not available, we may choose the value z in the range [1..100d], the “chance” of returning a wrong answer is no more than 1%. Calling this random verification 10 times reduces the probability of a wrong answer to less than 10−20 < 2−64 . Let us describe a modern deck of playing cards, endlessly useful for depriving people of their funds in casinos, and for writers of books on probability. The standard deck consists of 52 distinct cards. Each card belongs to one of four suites: spade (♠), heart (♥), diamond (♦), and club (♣). Also, each card has one of the 13 possible ranks: two, three, four, five, six, seven, eight, nine, ten, jack (J), queen (Q), king (K) and ace (A). The two and three are also called deuce and trey. Face cards are the jack, queen, and king, the ranks of jack to ace are picture cards, and the others are pip value cards9 . The card with one pip is called an ace, while cards from two to 10 pips are named according to the number of pips. Most popular games are played with a single deck or cards, but many variations exist. A set of cards held by a player is called a hand. Exercise 3.27 [4] Consider a card game where five-card hands are dealt from a well-shuffled deck of 52 cards. (a) What is the probability that a five-card hand has at least one card of each suit? (b) What is the probability that a five-card hand has exactly 3 kings? (c) What is the probability that a six-card hand has at least one card of each suit? (d) Explain why the answers in parts (a) and (c) do not depend on the number of hands dealt? Exercise 3.28 [3] (Adapted from [143]). Two hunters, A and B, set out to hunt ducks. Both are average shots: they hit as often as they miss when shooting at ducks. On that day A shoots at 50 ducks, and B shoots at 51. Find the probability that B bags more ducks than A? Exercise 3.29 [1+] Ten people are separated into two teams of five each randomly. What is the probability that two friends will be in the same team? Exercise 3.30 [3] What is the probability that a random permutation of a, b, c, d, e has (a) a occurring somewhere before b? (b) a and b side-by-side? (c) Now we add another a to the mix and consider all distinguishable permutations of the six letters equally likely. How many such permutations are there? What is now the probability of finding a and b side-by-side? Exercise 3.31 [2] Four balls are drawn in sequence from an urn that contains 8 red balls and 5 white balls. What is the probability that the fourth ball is white when (a) the drawn balls are replaced? (b) the drawn balls are removed? Exercise 3.32 [2] Consider a box in which are n marbles, a of them are red and the rest 9A

pip is another word for a small dot or seed. On playing cards, pips are the dots printed on the center front of each card. Same for the dots on the die.

94


green. We pick r of the marbles, at random, without replacement. Given that r > a, what is the probability that we picked all the red marbles? Exercise 3.33 [3] (Tennis tactics) Peter wants a new car, and his dad, who is a tennis fiend, wants Peter to improve his tennis play. He has a challenge for Peter. If he wins two games in a row in a sequence of three games, his opponents alternating between his younger brother Jack and sister Anna, he gets a new car. Should Peter choose to play Jack or Anna first? It is known that Peter’s probabilities of beating Anna (who developed a mean serve) and Jack are 0.1 and 0.8, respectively, and the games only continue until the result is known. Exercise 3.34 [4] A square 3× 3 matrix is filled randomly with the first nine positive integers. What is the probability that its determinant is an even number?

3.4 Random Variables In this section, we introduce discrete and continuous random variables and then discuss their main properties. We are interested in practical applications of this concept, but not in its general form. Therefore our definition of random variables is a simplified version of the general mathematical term used in probability theory [41, 124] because it serves our needs. Otherwise we will be forced to use measurable and Borel functions—the topic that is far away from our needs. Also we pay much of our attention to discrete random variables since they are mostly called for in analysis of algorithms.

3.4.1 Probability Mass Function Recall from Definition 3.10 on page 85 that a random variable is a function from a set of events, Σ, to R, the set of all real numbers. The random variable X associates the real number X (ω ) with any event ω from the algebra Σ. Example 3.35 We consider the roll of a die. We have seen the associated probability space (Ω, Σ, Pr) before. Let us single out two events from Σ: odd = {1, 3, 5}, and even = {2, 4, 6} (note that these two sets, together with Ω and ∅, form an algebra). Now we define a random variable X with assigned values for those two events only: X (odd) = −1 and X (even) = 1. Clearly this simply states the parity of the result of the exposed face of the die. (Why is this called parity? Think of the value of −1 raised to an integer power!)

The same effect could be obtained by using the natural random variable10 , denoted here by Y , which achieves the values 1 through 6, and asking whether Y is odd or even. Finally, we can define a third random variable Z, which has a value assigned for each simple event: Z(1) = Z(3) = Z(5) = −1, Z(2) = Z(4) = Z(6) = 1. Now, with each experiment, X and Z assume the same value, even though X has no value for any simple event (we defined it on the events odd and even). 10 It

is natural since the result of the experiment is expressed in numerical terms; if we tossed a coin instead, there is no such natural random variable.

3.4. RANDOM VARIABLES

95

Definition 3.36 The probability distribution of a discrete random variable X is an assignment of probabilities to all the outcomes that the random variable can assume. The function that keep truck of all these assignments is called the probability mass function (PMF) of the discrete random variable X and denoted by pX (k) = Pr[X = k] or simply p(k) when the random variable is understood. The connection to the underlying discrete probability space is given via the equality pX (k) =

∑

Pr[ω ]

⊳

(3.8)

ω ∈Ω X(ω )=k

Note that this assignment is not arbitrary: it is determined by • the values of the random variable over the sample space; and • the probability measure, Pr.

Actually, a probability mass function for a discrete random variable is a sequence {pX (k)} (which could be finite when the set of outcomes is finite) of not-negative numbers that satisfy the following conditions: 1. 0 6 pX (k) 6 1,

2.

∑ pX (k) = 1,

3. pX (k) = Pr[X = k].

(3.9)

k

Thus, in Example 3.35 above, Pr[Z = 1] = 1/2, and Pr[Z = 1.6] = 0 since no event was assigned this value. Had we assigned Z(3) = 1.6, for example, the situation would change, to maintain consistency we could have Pr[Z = −1] = 1/3,

Pr[Z = 1] = 1/2,

Pr[Z = 1.6] = 1/6.

Sometimes the natural random variable is the most meaningful random variable to be defined. Example 3.37 The following table lists the probability distribution of the number of detected break-in attempts per week into our computer network, based on past data. Evidently, using the natural random variable X = number Attempts per week 0 1 2 3 4 of attempts, and with this taProbability 0.1 0.15 0.25 0.35 0.15 ble we can find, for example, the probability that our system staff will notice at most 1 break-in attempt during a given week. This probability is given by the sum of the probabilities of 0 and 1 detections: Pr[X 6 1] = Pr[X = 0] + Pr[X = 1] = 0.1 + 0.15 = 0.25.

Definition 3.38 The right-continuous function

x ∈ R,

FX (x) = Pr[ω : X (ω ) 6 x],

is called the distribution function (sometimes called the cumulative distribution function). For a discrete random variable X , the distribution function is defined for every real number x as follows: F(x) = FX (x) = Pr[X 6 x] = ∑ Pr[X = k]. k6x def

The complementary distribution function is F(x) = 1 − F(x).

⊳


96

Definition 3.39 A random variable X is called continuous if its distribution function FX (x) is continuous for all real values x ∈ R. A random variable X is said to be absolutely continuous if there exists a nonnegative function f (x) = fX (x) such that its cumulative distribution function FX (x) can be expressed as F(x) = FX (x) =

Z x

−∞

fX (t) dt,

x ∈ R.

Such function fX (x) is called the probability density function.

⊳

If X is an absolutely continuous random variable, then its density function f (x) has the following properties: Z ∞

−∞

f (x) dx = 1

and

Pr[a < X < b] =

Z b

f (x) dx.

a

When a random variable X takes values in Z, the set of all integers, then pX (k) = Pr[X = k] = ∇F(k) = F(k) − F(k − 1).

(3.10)

For an absolutely continuous random variable, its density function f (x) is the derivative of the distribution function: f (x) = F ′ (x). In most practical problems, continuous random variables measured data over intervals, not single points. That is, the area under the curve between two distinct points defines the probability of a random variable to be within that interval. This means that the height of the probability function can in fact be greater than one. So in opposite to discrete case, the probability of a specific outcome is zero. The probability mass function p (or pX ) is just the assignment of probabilities to values the random variable achieves. There is a distinction between Pr and p: the measure Pr applies to all the events in Σ (simple or compound). Using this measure, we can define numerous random variables; each with its own probability mass function, which depends on Pr. The definitions of the cumulative distribution function and probability mass function for a discrete random variable are based on the assumption that the range of values is discrete and countable. It means that we can map the set of values of a discrete random variable into the set of integers, N. Thus, we can enumerate the values {xk }k>1 of a discrete random variable X and define an equivalent integer-valued random variable Y so that Pr[Y = k] = Pr[X = xk ]. Then in many problems its probability mass function pY (k) = Pr[Y = k] = Pr[X = xk ] can be used instead of pX (xk ) = Pr[X = xk ]. In this case, the subscript Y can be dropped and the function p(k) = Pr[X = xk ] can be considered as the probability mass function associated with the random variable X . The definition of the probability mass function p(k) depends on enumeration of the values of the random variable X and it is not unique. However, nonuniqueness usually does not cause any difficulty in applications and therefore most of the time we do look at integer-valued random variables. While the probability mass function is connected to a probability space, it is also true that any nonnegative function p over N (or its subset) whose values add to 1 is a proper probability


97

mass function of some random variable; that is, it should satisfy conditions (3.9). We can use this probability mass function (PMF) to define a probability space that will give rise to such a variable. The following PMF is used most often. Definition 3.40 Let U be a random variable that can assume the values in some finite set S ⊂ R. If the probabilities Pr[U = s] are the same, for all s ∈ S, then U is said to have a uniform distribution on S, or a uniform probability mass function. In particular, we have then Pr[U = s] = 1/|S|. A continuous uniform random variable on an interval (a, b), with a < b, has the constant probability density function: f (x) = 1/(b − a). ⊳ Note: It is customary to say, for example, “the value of T is chosen at random,” when the intent is that T is distributed uniformly. A random variable can have any distribution, but this way of using the word “random” expresses uniformity in everyday speech. For instance, choosing at random a card from the full deck of well shuffled playing cards has the probability 1/52; however it is not a uniform random variable because its value is a card but not a number. For an example of a uniform distribution, we look at the value shown on the upper face of a fair die: p(k) = 1/6 for k = 1, 2, . . . , 6 and 0 otherwise. Its distribution function    0, F(x) = k/6,   1,

satisfies the inequality F(x) 6 x/6

for x < 1, for k 6 x < (k + 1), for x > 6,

k = 1, 2, . . . , 5,

(x > 0).

Let us conclude the subsection with a final example, which takes the approach suggested at the end of our discussion of the birthday problem (Example 3.19). Example 3.41 Suppose we have a true coin. This means that flipping it results in getting heads with the same probability as getting tails (and that each probability is equal to 1/2—a

true coin never lands on its edge!). Let us flip the coin 10 times. A possible outcome is some string like TTHTHHTHTT, made up of 10 tails and/or heads, that is, a “10-letter word” from the alphabet { T, H }. Recall that in a word, the order of the letters in it is significant. Strictly speaking, we are missing here the notion of independent trials, which is discussed in §3.6; it would justify some of the calculations here. We have seen in Chapter 2 that the number of kletter words drawn from an n-letter alphabet is nk . So there are 210 = 1024 of these outcomes, and they are equally likely. Thus, the probability of each is 2−10 = 1/1024. For example, the probability of getting tails ten times in a row (which is the same as that of getting the above string TTHTHHTHTT) is 10 1 1 = = 0.0009765625. Pr[ TTTTTTTTTT ] = 2 1024 Define a random variable X to be the number of tails in the sequence of ten flips. The event “X = k” consists of all the 10-letter words of T’s and H’s that have k tails and 10 − k heads. 10 It is known from §2.4 that the number of such words is k because the order of tails does not matter. Therefore the probability mass function of X is . 10 pX (k) = Pr[X = k] = 210 . k


98

Suppose we want to determine the probability of getting exactly 2 heads. This event corresponds to X = 8 and 10 45 number of strings with 2 heads = 2 = ≈ 0.043945. Pr[X = 8] = 1024 1024 1024 Similarly, the event A = { getting at least 2 heads } is the complementary event to A′ = { getting zero or one heads } = { X = 10 or X = 9 } = { X = 10 } ∪ { X = 9 }. Then

Therefore

10 1 10 11 1 + 9 = + = ≈ 0.01074. Pr[A′ ] = 1024 1024 1024 1024 1024

11 ≈ 0.989. 1024 Exercise 3.42 [3] Using the probability space of Example 3.41 (with each element of the sample space a sequence of 10 coin flips), it is possible to define a few more random variables, such as the number of heads-to-tails changes in the sequence (there are three in the string TTHTHHTHTT), or the reverse changes (T-to-H), or the longest run with no changes, or the number of changes separated by 2 flips etc. . . . Can you define more? Pick three random variables, either of your own or from the above suggestions and find their PMF. Guidance: Analytic derivation of the probability mass function (PMF) for most of these random variables requires tools developed in Chapters 4 and 6. You are asked to take advantage of the relatively small size of the sample space (1024 equiprobable sequences) and calculate the PMFs by counting the suitable sequences. We recommend MAPLE for some of the tasks, but you can, of course, use any tool you find suitable. Pr(A) = 1 − Pr(A′ ) = 1 −

Exercise 3.43 [1] What is the probability distribution of the random variable Z defined for a biased die where the probabilities of rolling the result i is pi = i/21 (i = 1, 2, . . . , 6)? Exercise 3.44 [2] Let X be a random variable over n values x1 , x2 , . . . , xn such that pX (n) 6 1 pX (n − 1) 6 . . . 6 pX (1), where pX (k) = Pr[X = xk ]. Show that pX (k) 6 . k Exercise 3.45 [1] How many times do you need to flip a true coin so that the probability of seeing at least one heads would be greater than 0.9? Exercise 3.46 [4] (Banach matchbox problem11 ) We have two boxes, each initially containing n marbles. You repeatedly choose a box, with each box being equally likely to be chosen, and remove a marble from that box. After a certain number of repetitions, one of the boxes is empty. Let R be the number of marbles at that moment in the other box. (a) Consider the possible sequences of choices that produce the desirable result, and show 2n−r−1 −(2n−r−1) that the PMF of R is given by pr = n−1 2 , 1 6 r 6 n. (b) Show that the PMF of part (a) sums to 1. (Its proof requires a result given in Exercise 6.137 on page 331, or an induction on the initial capacity.) 11 Stefan Banach (born in Krakow, Austria-Hungary, 1892, died in Lvov, Russia, 1945) was a Polish mathematician who made many contributions in functional analysis. He was known as a serious smoker, who usually kept several match boxes in his pockets.


99

(c) Redo parts (a) and (b) for the same scenario, but with a difference in the time at which the number of remaining marbles is recorded: we consider the number of marbles R′ in a box when the other box is found empty (which will happen when an attempt is made to draw a marble from it). The answers are close, but not the same! Exercise 3.47 [4] Two real numbers x and y are chosen at random in the interval (0, 1) with respect to the uniform distribution. What is the probability that the closest integer to x/y is even? Hint: The direct solution uses the standard expansion of a trigonometric function. Exercise 3.48 [2] A lazy grader decides to mark only five questions, chosen at random (uniformly) from among the ten questions on an exam. A student, unaware of this marking strategy, answers only seven questions, but he knows that he answered them correctly. What is the probability distribution of his mark, assuming all questions were of equal value? Will he do better or worse on the average than if the grader marked all the questions?

3.4.2 Expected Values Historically, the concept of expected value of a random variable, also called mathematical expectation, arose in connection with gambling. Example 3.49 Consider a game that consists of flipping three fair coins. The player is paid $3 if all show heads or all tails. Otherwise, the player is paid nothing. Playing this game costs $1. We also assume the coins are fair. While this is not the same experiment as in Example 3.5 (there we flipped the same coin three times), the sample space for possible outcomes is the same: Ω = { HHH, HHT, HTH, THH, HTT, THT, TTH, TTT }. Each of these outcomes is equally likely and occurs with probability 1/8. The sample space is the union of two mutually exclusive events: Ω = ω1 ∪ ω2 , ω1 ∩ ω2 = ∅, where ω1 is the winning event, ω1 = { HHH, TTT }, and the losing event is ω2 = ω 1 = Ω \ ω1 , that is, ω2 = { HHT, HTH, THH, HTT, THT, TTH }. So the game probabilities are Pr[win] = Pr[ω1 ] = 82 = 41 , and the probability of no pay-off is Pr[ω2 ] = 86 = 43 . The value of the game, the expected gain in one round is,

3 1 1 − $1 = −$ = −$0.25. $3 · + $0 · 4 4 4

Hence, in this game the gambler is expected to lose a quarter each time the game is played. Example 3.50 Another popular game is American roulette. Its wheel has 38 bins around its circumference. Thirty-six bins are numbered from 1 to 36. One half of them are colored red and the other half are colored black. The remaining two bins are numbered 0 and 00 and are colored green. These two green spots belong to the house. The dealer spins a ball and the winning number is the number of the bin in which the ball comes to rest. In a well-tuned roulette table the likelihood of each bin getting the ball is equal. In a single-number bet, the house pays to a winner $36 ($35 is the prize and $1 is the reimbursement) for a dollar bet; otherwise, the player loses this $1. The probability of winning a


100

single-number bet is 1/38 . The expected gain would be $36 ·

1 37 2 1 − $1 = $35 · − $1 · = −$ ≈ −$0.05263 38 38 38 38

in a one-dollar game. The value of the game is negative: a knowledgeable gambler should expect to lose over a nickel of every dollar invested in the game. There is also a possibility of betting on color, either black or red. If the ball stops on a bin with selected color, the player is paid double his/her bet. Otherwise the player loses this $1. Exercise 3.51 [1] Referring to the last example, we observe that in opposite to American roulette, European roulette have a single green bin (that the house always takes), and a total of 37 bins. The other 36 bins are the same for all roulette. Evaluate a game of $1 bet on black (a winner is rewarded with $1) in the following games. (a) European roulette, (b) American roulette. Definition 3.52 Let X be a discrete random variable with the probability mass function p(k) = Pr[X = xk ]. The mean, or expected value of X is the number (if it exists), denoted by µX , or E[X ], and defined by

µX = E[X ] = ∑ xk p(k) = ∑ xk Pr[X = xk ]

(3.11)

k

k

provided that, if this sum is an infinite series, it converges absolutely. The summation index runs on all possible values of X . The E in Eq. (3.11) is the expectation operator. ⊳ The requirement of absolute convergence means that the series ∑k |xk | p(k) converges. If X assumes finitely many values only, the issue of convergence does not arise. Note: While the above definition of the expectation in terms of the probability mass function (PMF) is usually adequate, it is sometimes preferable to resurrect the infrastructure underlying the PMF as in Eq. (3.8). Then we can redefine—or compute—the expected value in terms of the measure over the probability space. In that case we have the definition

µX = E[X ] =

∑

X (ω ) Pr[ω ].

(3.12)

ω ∈Ω

We list the main properties of the expectation operator. They can be all seen as direct results of the initial definition, Eq. (3.11) though some of the proofs are much simplified (and become more general) by using the definition, Eq. (3.12). Property 1: The expected value of a constant is the constant, namely, E[a] = a. Property 2: The expected value of a sum of random variables (which have expectations) equals the sum of their expected values, that is, E[X +Y ] = E[X ] + E[Y].

(3.13)

The relation extends, by induction, to the sum of any number of random variables. The important thing to note here is that we do not require the variables to be independent for this relation to hold (we shall get to this issue in §3.8). Property 3: For any constant a, E[aX ] = aE[X ].


101

We can combine the last two properties as E[∑r ar Xr ] = ∑r ar E[Xr ], for any collections of constants {ar } and variables {Xr }. The equality even extends to the case when one or more expectations do not exist, or the sums do not converge. Properties 2 and 3 explain why E is called a linear operator. Property 4: If X (ω ) > Y (ω ) for every event ω then E[X ] > E[Y ]. In particular, if X is a positive random variable, then E[X ] > 0. Property 5: (E[|XY |])2 6 E[X 2 ] · E[Y 2 ] (Bunyakovskii inequality12 or B.C.S. inequality). Property 6: Let X be a random variable with the PMF pX (x) and let Y = h(X ), where the function h(·) is finite at any value X can assume. Then E[Y ] = E[h(X )] = ∑ h(x) pX (x) = ∑ h(x) Pr[X = x]. x

(3.14)

x

Exercise 3.53 [3] Prove all six properties of the expectation operator using the definition, Eq. (3.11). Note: The proof of property 2 depends on §3.8 for the concept of dependent random variables and their joint distribution, and on Theorem 4.78, page 162 (the theorem of total probability), for the device of randomization. For property 5 consider the nonenegative polynomial E[(tX +Y )2 ]. There are two other parameters that describe the behavior of a given random variable. Definition 3.54 Let F(x) be the (cumulative) distribution function of a random variable X. Its median (possibly not unique) is the smallest value of k such that F(k) > 1/2. Hence, Pr[X 6 m] > 12 and Pr[X > m] 6 21 . The value of k at which the PMF is maximal is called the mode13 of the random variable. For 0 < s < 1, the s-quantile of F(x) is the smallest number x such that F(x) > s. ⊳ In other words, the median of a discrete random variable X (i.e., having finite or countably infinite set of values) is a value m such that the variable has the same probability of falling on either side of m: 1 (3.15) ∑ Pr[X = xk ] 6 2 6 ∑ Pr[X = xk ] . xk <m xk 6m Among quantiles, there are some of particular interest: the median x1/2 , the lower and upper quartiles x1/4 and x3/4 , the lower and upper octile x1/8 and x7/8 . In Example 3.19, we discussed probabilities of sharing birthdays in a group of n people. There is no exact formula for such event and we found numerically that the median is approximately n = 23 if the number of days in a year is m = 365. Suppose we wish to know the least n for which the probability that in a group of n individuals, at least one of them has the same birthday as yours is greater than or equal to 1/2. Using properties of logarithm, we see 12 The Bunyakovskii inequality is also known as the Schwarz inequality; however, V. Ya. Bunyakovskii published his study as early as 1859, whereas in H. A. Schwarz’ work this inequality appeared as late as 1884 without any reference to the work of Bunyakovskii. A particular case was proved by A. L. Cauchy in 1821. A very readable excursion around this formula is the topic of [126]. 13 The mode may not exist, and when it does it is not necessarily unique. If the same maximum max p (x), x X where pX (x) = ∑X (ω )=x Pr[ω ], is attained at more than one value of x, then all such values of x are called modes of the distribution pX (x).


102

1. m−1 m ln = ln 2/ ln . When than the least value for which this holds is n > ln 2 m m−1 m = 365, we get n = 253. Definition 3.55 A random variable X is symmetric about a constant C if and only if X − C and C − X have the same distribution, and E[X ] = C if it exists. When C = 0 we call X simply a symmetric random variable. ⊳ Let U be a random variable having a uniform distribution over a set S with n = |S| elements. Its expected value is 1 1 µ = E[U ] = ∑ s Pr[U = s] = ∑ s = ∑ s. n s∈S s∈S n s∈S For example, if S = [1..n] is the set of first integer numbers, then E[U ] = n1 ∑nk=1 k = n+1 2 . Obviously, every uniform distribution over a regularly spaced set is symmetric about its mean. If U is a continuous uniform random variable on an interval (a, b), with a < b, then U is symmetric about its mean, (a + b)/2. Example 3.56 (Inequalities among expectations) Although expectations of random variables have been defined in terms of distributions, see Eq. (3.11), some arguments flow much more naturally if we fall back to the underlying probability space via Eq. (3.12). Consider, for example, two random variables: X and Y ; then the following inequalities hold: a

b

c

E[min(X ,Y )] 6 min(E[X ], E[Y ]) 6 max(E[X ], E[Y ]) 6 E[max(X ,Y )]. The expressions are only meaningful if the variables are defined on the same probability space. Inequality (a) follows from the fact that E[min(X ,Y )] can be shown to be no larger than either of E[X ] or E[Y ], and therefore also from the smaller of the two. From Eq. (3.12), E[X ] − E[min(X ,Y )] =

∑ Pr(ω )[X (ω ) − min(X (ω ),Y (ω ))] > 0

ω ∈Ω

Observe that the bracketed value is non-negative. It is zero at points ω where X (ω ) is the smaller of the two, and strictly positive elsewhere. Hence the entire sum is at least nonnegative. Similarly, E[Y ] is no smaller than E[min(X ,Y )], so the smaller between these two satisfies the inequality (a). Inequality (b) is between the smaller and larger of two numbers, so it is obvious. Inequality (c) follows from a calculation very similar to the one used for (a): it shows that E[max(X ,Y )] is larger than either of E[X ] or E[Y ], and therefore also from the larger of those two. Is it possible for all inequalities to be in fact equalities? Yes, but only if X (ω ) = Y (ω ) for any ω for which Pr(ω ) > 0. Is it possible for all three inequalities to be strict? Yes, you could even say that this is what we would usually expect. Note that it is quite possible for (a) and (c) to be strict, when (b) is an equality, due to E[X ] = E[Y ]. Example 3.57 (Insertion sort) Insertion sorting is a simplest algorithm that is commonly used by card players when putting


103

their hands into order. As a computer sorting method, it is only effective for small arrays, or when the numbers are nearly sorted. The idea of insertion algorithm is based on considering every element in an array (ordered set of real numbers), one at a time, and comparing it with preceding elements that are already sorted. So the algorithm finds the correct position in the prefix and inserts considered element into this position to make subarray sorted. The following C-code is one of its implementations. insort(float ∗a, int n) { int i, j; float t; if (n = 0; i − −) if (t < a[i]) a[i + 1] = a[i]; else break; a[i + 1] = t; } }

1. 2. 3. 4. 5. 6. 7.

To analyze the performance of this procedure, we need to know the number of times each statement is executed. First we make some assumptions about the input set. Consider an array a[0..n − 1] of size n, n > 2. For simplicity, we assume that all entries of a are distinct. Then we can adopt without loss of generality that the array a consists of integers 1, 2, . . . , n because the program makes its decisions based only on the relative order of the entries, never on their actual values. The performance of insertion sort depends on statistical properties of the input list a. Here we suppose that the distribution of numbers in the array is uniform. This means that each of the n! permutations of [1..n] is equally likely. Note that uniform distribution of entries in the input array is only a model and the program runs correctly for any array of numbers, not necessary integers. Also, its actual execution does not depend on whether entries are different or some of them are equal—the assumption about distinct entries was made for simplicity of its analysis. The running time of the program depends on three unknown quantities: C I M

— — —

the number of comparisons, the number of t-assignments, the number of entries moved during insertion,

t < a[i], line 6, t = a[ j], a[i + 1] = t, lines 4, 7, a[i+1] = a[i], line 6.

The first for loop in line 4 with respect to j is executed n − 1 times deterministically; it places the current jth entry from the input array a into a temporary location t. The number of times the body of the second for loop (on i in lines 5, 6, 7) is executed depends on dispersion of entries within the array. The jth value in the array, denoted by t, is compared with the prefix values a[0.. j − 1] until the program finds an entry that is less than t or it reaches the end of the prefix. Therefore, for each j ∈ [1..n − 1], the number of comparisons of the element t = a[j], denoted by c j , is within the range 1 6 c j 6 j. When the loop over i starts with t = a[ j], the subarray a[0.. j − 1] of size j is already sorted in nondecreasing order. The new number t = a[j] that is about to be inserted has probability 1/( j + 1) of being the largest number so far, and the program will make only one comparison, and no moves. It has the same probability of being the smallest one, or being the second smallest and so on. Hence the for loop with respect to i has the probability 1/( j + 1) of being


104

any length. The number of moves is one less than the number of comparisons, for each j, except when t moves all the way to the first position of the array. Denoting by m j the number of moves done in phase j, we have ( c j − 1, 1 6 c j < j, (3.16) mj = c j, c j = j. A convenient approach to evaluating M, the number of moves, is via the inversion table of the input, see Definition 2.58 on page 45, since each move settles a single inversion. The ith entry in the inversion table, denoted by bi , is the number of elements to the left of a[i] in the initial array that are larger than it. When its turn comes to find its place, these bi elements are now immediately to its left since that subarray is sorted. Each of them would need to be moved one place to the right. Hence m j = b j , and M is given by sum of entries in the inversion table: M = b0 + b1 + b2 + · · · + bn−1 (b0 = 0). (3.17) This sum is also called the number of inversions of the permutation. Its minimum value is 0 and the maximum value is n2 because this is the number of ways to choose a pair from [1..n]. Since each inversion table determines a unique permutation, and we assume that the input permutations are equally likely, each bi takes each of its possible values with equal probability as well, independently of the values of all other table entries. Therefore, we assume that the entries in the inversions table are independent, and uniformly distributed. Let en,k be the probability that a permutation of [1..n] has exactly k inversions. We observe that if a permutation ha 1 , a2 , . . . , an i has k inversions then the backward permutation han , an−1 , . . . , a1 i has k′ = n2 − k inversions (see Exercise 2.109 on page 62). The expected number of inversions is given by n n n ′ E[M] = ∑ k en,k = ∑ − k en,k′ = ∑ − k en,k = ∑ en,k − ∑ k en,k . 2 2 2 k k k k k′ Adding the sum ∑k k en,k to both sides, we obtain n n n n(n − 1) 2 E[M] = ∑ en,k = ∑ en,k = 2 = 2 2 2 k k because ∑k en,k = 1. Therefore on average the insertion sort makes n(n − 1)/4 moves.

From Eq. (3.16), the difference between M and C is n − 1 less the number of times all the comparisons resulted in a move, until the range of i was exhausted. So we introduce another random variable to express this difference: C = M + n − 1 − Y , where Y is the number of times the value from t was placed in position 0. This happens when all the entries to the left of a[ j] are larger than it, and the probability this occurs is 1/( j + 1), and it is independent for distinct entries. Hence Y is the sum of indicator random variables Y = ∑n−1 j=1 h j , where ( 1, if b j = j − 1, 1 . and Pr[h j = 1] = hj = j + 1 0, if b j < j − 1; Therefore n−1

E[Y ] =

1 ∑ i + 1 = Hn − 1; i=1

=⇒

E[C] =

n2 + 3n − Hn , 4


105

where Hn is the nth harmonic number. The number of assignments (a[i+1] = t) in line 7 is also related to the inversion table. It is the number of entries in the input array that have at least one greater element to the left. This is exactly the number of non-zero entries in the inversion table. Its minimum value is 0 and the maximum value is n − 1. The number of t-assignments in line 4 is deterministic—it is executed n − 1 times. Example 3.58 (Non-transitive dice) Let X be a random variable corresponding to the roll of a fair die (all six possible outcomes, 1 through 6, have the same probability, 1/6). Then the expected value of X is E[X ] = ∑6k=1 k · 6·7 1 7 1 6 = 2 · 6 = 2 = 3.5.

The objective of this example is to issue a warning that our intuition is not very reliable when handling probabilistic reasoning. Instead of a fair die, we consider four Efron dice, designed by Bradley Efron. He labeled them with letters A, B, C, and D. Six sides of the die A contain two 0’s and four 4’s. The die B consists of all 3’s. Dice C (2, 2, 2, 2, 6, 6) and D (1, 1, 1, 5, 5, 5) are similar to die A, since there are at most only two digits on each of them. Let XA , XB , XC , and XD be the corresponding random variables. Their expected values are 8 E[XA] = , 3

E[XB] = 3,

E[XC ] =

10 , 3

E[XD] = 3.

It is interesting to note that die A is twice as likely to beat die B, as die B beats die C, which is twice as likely to beat die D, which in turn beats die A: 2 Pr[XA > XB ] = , 3

2 Pr[XB > XC ] = , 3

2 Pr[XC > XD ] = , 3

2 Pr[XD > XA ] = . 3

Hence the surprising conclusion that unlike the relation “larger than,” which is transitive, the relation “larger than in probability” is not transitive (it is of course still anti-symmetric); see also Exercise 9.113, page 542. Let us consider another example of three unusual dice, colored red, green, and blue for purpose of identification: Let R, G, and B be random variables equal to the value shown on the top face, when rolling the red, green, and blue dice, rered: 2,2,6,6,7,7 spectively. Suppose that we want to calculate the probability green: 3,3,4,4,8,8 Pr[R > G] when rolling these two dice together. There are nine blue: 1,1,5,5,9,9 possible outcomes for (R, G): (2,3), (2,4), (2,8), (6,3), (6,4), (6,8), (7,3), (7,4), (7,8). Only four of them, (6, 3), (6, 4), (7, 3), (7, 4) are favorable for R. Therefore Pr[R > G] = 4/9. Similarly, Pr[G > B] = 4/9 and Pr[B > R] = 4/9. Thus, the green die “beats” the red one with odds 5:4, blue die beats green with the same score, and the red die defeats the blue die with the same odds, 5:4 again. These three dice give another example of not transitivity for the relation “larger in probability.” Non-transitive dice fall into the general category of non-transitivity paradoxes, about which there has been considerable study since 1785 by Marquis de Condorcet14 , who was the first to propose the determination of a winner between different alternatives (in our case, dice play the role of candidates) by a head-to-head comparison. 14 Marie

Jean Antoine Nicolas, the Marquis de Condorcet (1743 – 1794).


106

It is possible to arrange the numbers from 1 to 18 on each face of each cube using each number once and only once such that when the dice are rolled, the probability that R beats G is greater than 1/2, the probability that G beats B is greater than 1/2. Yet, the probability that B beats R is also greater than 1/2. For example, choosing the following red, green, and blue dice R(5, 6, 7, 8, 9, 18), G(2, 3, 4, 15, 16, 17), and B(1, 10, 11, 12, 13, 14), it is easily to check that Pr[R > G] = 21/36 = 7/12, Pr[G > B] = 7/12, and Pr[B > R] = 25/36. The problem can be generalized for arbitrary n-sided dice [133]. Exercise 3.59 [3] (C. Rump) Consider a game where each of two players rolls two different Efron dice, with the goal of rolling the highest total. Which two out of four Efron dice are you going to choose in order to minimize the probability of loss? Exercise 3.60 [3] The following formula provides sometimes a convenient alternative to the definition of the expected value given in Eq. (3.11). We state it for non-negative integer random variables (and assume the expectation exists): E[X ] =

∑ k p(k) = ∑ Pr[X > k] = ∑ (1 − Pr[X 6 k]) = ∑ (1 − F(k))

k>0

k>0

(3.18)

k>0

k>0

since it provides a useful way of computing E[X ] in terms of tail probabilities. Prove this equality using summation by parts (Theorem 2.36). (a) Prove this equality using summation by parts (Theorem 2.36). Here F(x) = ∑k6x Pr[X = k] = Pr[X 6 x] is the cumulative distribution function of the random variable X . (b) The sum in Eq. (3.18), in both formulas for the expectation, is over the entire support of the random variable. Let the support (the set of events where X 6= 0) be infinite. A numerical calculation of the expectation must be stopped, truncated at some point, say n. Show that whatever n we pick, using the same stopping point in both formulas, leads to a smaller error using the cumulative distribution. Would this advantage be always maintained if we sum over the same number of terms in both formulas? Exercise 3.61 [4] Complete the calculation hinted at in Example 1.5 on page 4. Specifically, calculate the probability mass functions (PMFs) of the random variables denoted there by N and D, and use the PMFs to find the expected values of these variables. Definition 3.62 Let A be an event. We define a random variable χA by ( 1, if ω is in A, E[χA ] = Pr[χA = 1] = Pr[A]. χA (ω ) = 0, if ω is not in A. Such a random variable is called the indicator random variable of the event A.

(3.19) ⊳

The random variable (3.19) can be written via the Iverson bracket: χA (ω ) = [ω ∈ A]. Since the sample space, Ω, can be broken up as the union of two disjoint events, Ω = A ∪ (Ω \ A), we have E[χA ] = Pr[A]. The indicator random variables are often very useful in evaluating expected values since Eq. (3.12) can then be written as

µX = E[X ] =

∑

ω ∈Ω

X (ω ) E[χ (ω )].

(3.20)

Example 3.63 Consider the following algorithm to return the minimum element of an array A[1..n]:

3.4. RANDOM VARIABLES MIN

107

low ← A[1] for j ← 1 to n do if ( A[ j] < low ) then low ← A[ j] return low

The array A is assumed to be a permutation of [1..n] = {1, 2, . . . , n}, and that each permutation is equally likely. We will establish a probability model for the random variable X , the number of times low ← A[ j] is executed. This random variable is constructed from the sequence of indicator random variables X j , 2 6 j 6 n, ( 1, if low ← A[ j] is executed, Xj = 0, otherwise. The sample space is the set of all n! permutations of [1..n]. The random variable X is the sum of the indicator random variables, X = ∑26 j6n X j . Then # " 1 = Hn − 1. E[X ] = E ∑ X j = ∑ E[X j ] = ∑ Pr[X j = 1] = ∑ 26 j6n j 26 j6n 26 j6n 26 j6n The expected value of the indicator random variable is E[X j ] = 1/ j because in order for X j to be 1, it is necessary for A[ j] to be the smallest entry in the j-long prefix of the array. We are using here the fact that every subsequence of a random permutation has its values randomly permuted. (This algorithm is further considered in Example 4.159, page 190.) Example 3.64 We present one more example to demonstrate the beauty and versatility of indicator random variables. Let n distinct pairs of socks be put into the laundry. We assume that each of the 2n socks has exactly one mate. We also assume that an unusual event happens: when the laundry is returned, we actually got back 2n socks, all of them! Consider the process of taking out the socks one at a time, trying to match each sock with its mate if that mate has already been drawn. Let E[Xk ] be the expected number of pairs formed after k socks have been drawn and let Xi j (1 6 i < j 6 k) be the indicator random variable that the ith and the jth sock, in the order of drawing, form a matching pair: Xi j = 1 if they match, or else Xi j = 0. Since the mate of the ith drawn sock can be in any of theother 2n − 1 positions in the drawing sequence, E[Xi j ] = Pr[Xi j = 1] = 1/(2n − 1). With 2k such pairs possible, # " k k(k − 1) 2 E[Xk ] = E ∑ Xi j = ∑ E[Xi j ] = 2n − 1 = 2(2n − 1) . 16i< j6k We may use another indicator random variable to determine E[Xk ]. Let Yi = 1 if both socks of pair i, 1 6 i 6 n, are among the first k socks; otherwise Yi = 0. Saying Yi = 1 determines 2 2n−2 of the k selected socks, and we are free to create k−2 other selections regardless of which i it is. Hence 2n−2 n 2n − 2 . 2n k(k − 1) . E[Y ] = n =⇒ E[X ] = = Pr[Yi = 1] = k−2 k ∑ i 2n 2(2n − 1) k − 2 k i=1 k

Example 3.151 (page 133) continues discussion of this problem.

108


Exercise 3.65 [3] There are n pairs of socks in a drawer, all different. One sock is drawn out at a time until a matching pair has been found. (a) If n is known, what is the most likely value of the number of socks drawn out? (b) If the number of socks drawn, say r, is known, what value of n gives the greatest chance of this happening? Exercise 3.66 [2] Suppose a coin is flipped n times producing a sequence of H’s and T’s. Let Xn denote the number of singles in a sequence, that is, the number of runs of length 1 (a toss that is opposite to its adjacent tosses). Find average number of singles—E[Xn ]. An asymptotic property of Xn is considered in Exercise 10.143 on page 622. Example 3.67 (The birthday problem, continuation of Example 3.19) We assume that there are m = 365 days in a year and that birthdays are uniformly distributed with leap years being ignored. Let Yn denote the number of different birthdays among n people (n 6 m), regardless of how many people celebrate each. What is E[Yn ]? It is more convenient to consider first the random variable Yn′ = m − Yn , the number of days on which nobody celebrates a birthday. We define the indicator random variable χk as 1 if day k is the birthday of no one, and 0 otherwise, for k ∈ [1..m]. The probability of a person not being born on a particular day is 1 − 1/m. Hence, day k is birthday-free if this event holds for all n people: E[χk ] = Pr[χk = 1] = (m − 1)n /mn . The total number of birthday-free days is the sum ∑16k6m χk . So the expected value of Yn′ is m ((m − 1)/m)n . Subtracting E[Yn′ ] from m yields the expected value of Yn : n n m−1 n =∑ (−1)k m1−k . E[Yn ] = m − m m k k=1 Finally, consider the difference n −Yn ; subtracting the number of distinct birthdays from the number of people will give us the number of days which are celebrated by at least two of the n. For m = 365 and n = 27, the expected value of this difference is approximately 0.9400. The difference increases to 1.0114 if we add one person, for n = 28. Thus, we need to collect at least 28 people, before we can expect to find a day on which at least two people share their birthday (compare with the result of Example 3.19. Is there a contradiction between the results?). The probability that a pair shares their birthdays is 1/m; since the number of pairs is n2 , the expected number of pairs with common birthdays is n(n − 1)/2m, and substituting the above values, (m, n) = (365, 28), gives the value 1.0356. Why is this number larger than n − E[Yn ]? Because it is possible that pairs of “twins” will be revealed as torn off from a triplet (or even higher-order group), hence fewer days are needed to accommodate these shared birthdays. Exercise 3.68 [3+] How many people should be invited to a party so that the expected number of triples that have the same birthday to be at least one? Hint: Use the indicator random variables Xi, j,k , 1 6 i < j < k 6 n, for each triple (i, j, k) of n individuals sharing the same birthday. Example 3.69 (Waiting time in the birthday problem) We consider another variant of the birthday problem that involves calculation of the waiting time until the match of birthdays occurs. Again, suppose we have a group of n people (n < m, where m = 365 is the number of possible birthdays) and we choose at random one of them in a sequence. If at least one of the remaining people in the group has the same birthday as the


109

selected person, then the experiment terminates. Otherwise that person is eliminated along with his or her birthday and the process is repeated. Let p(n, r) denote the probability that the above procedure terminates after the rth person in a group of n people is questioned and F(n, r) = ∑ri=1 p(n, i) be the corresponding cumulative function. Then " n−r # m − r + 1 n−r+1 m−r m − 1 n−1 m − 2 n−2 ··· 1− p(n, r) = m m−1 m−r+2 m−r+1 r m (3.21) = n (m − r + 1)n−r − (m − r)n−r , m where 2 6 r 6 n− 1 < m. The formula (3.21) is obtained by sequential application of the Fundamental Principle of Counting 2.3 (page 43): we multiply the probabilities that no matches are found after the first, second, third, . . . (r − 1)th persons are questioned and at least one is found after the rth.

The calculation of the cumulative function is an example of a telescoping sum since mi (m − i)n−i = mi+1 (m − i + 1)n−i−1 : r mr F(n, r) = ∑ p(n, i) = 1 − n (m − r)n−r , m i=1 where 1 6 r 6 n − 1 < m. Substituting this result into Eq. (3.18), we get the expected number of persons should be asked to see one match of the birthdays in the group of n people to be n−r n−1 m − 1 n−1 m − 2 n−2 m−r 1 n−1 ··· = n ∑ mr (m − r)n−r . µ (n, m) = ∑ m m−1 m−r+1 m r=1 r=1 No closed expression is forthcoming for this formula, but it can be used for numerical calculations. The following examples show situations where the expected value of a random variable does not exist. Example 3.70 (The St. Petersburg paradox)15 Consider the following game between two gamblers, say Alice and Bob. A fair coin is tossed repeatedly until heads appears. If heads shows on the first flip, Alice pays $2; if the first heads appears on the second toss, Alice pays $4, and so on: if heads occurs first on the kth toss, Alice pays $2k . At the start Bob pays a fixed amount $D to Alice. How much should D be for this to be a fair game? (The game is fair if D equals the expected value Alice pays in a game.)

Solution. The first step is to identify the probability space—finite sequences of tosses of a true coin. The second step is to look for a suitable random variable. Let us choose X , the amount Alice pays Bob. This variable assumes the values 21 , 22 , 23 , . . ., i.e., all the positive powers of 2. The payment is 2r if the first r − 1 tosses produce tails, and the rth is heads. The corresponding PMF is pX (k) = Pr[X = 2k ] = 15 Daniel

1 , 2k

k = 1, 2, 3, . . . .

Bernoulli first published about this problem in the journal of the St. Petersburg Academy of Science (Russia) in 1738.


110

Another way to reason here is that the possible number of outcome sequences from tossing a coin k times is 2k , and only one of these outcomes belongs to the event {X = k}. With this PMF, 1 D = E[X ] = ∑ 2k pX (k) = ∑ 2k · k = ∑ 1 = 1 + 1 + · · · = ∞. 2 k>1 k>1 all k Why do we say there is a paradox here? This is a game that has a small expected length in terms of the moves: the mean is only 2. The payoff function is finite at each move and simply related to the overall length. Yet, the payoff has an infinite expectation! We come back to this game in §4.1.3. Example 3.71 (Discrete Cauchy distribution.) Let X be a random variable on the set of all positive integers, {1, 2, 3, . . .}, having the probability mass function (PMF) a j > 1. Pr[X = j] = 2 , j Here a is a normalization factor: we choose a such that 1=

∑

j>1

a =a j2

∑

j>1

1 π2 = a j2 6

since ∑ j>1 j−2 = ζ (2) = π 2 /6 (see Eq. (12.55) on page 695). Therefore we set a = 6/π 2 ≈ 0.68. The expected value is then

E[X ] =

∑

j>1

j·

a =a j2

∑

j>1

1 , j

and this sum diverges! Hence, X has no moments (since its PMF decreases too slowly). However, this is a well-defined random variable with a valid probability mass function that can be used to find the probability that X < 7, or larger that 29, or has an even value, etc. Exercise 3.72 (Pólya urn model) (a) [3] A box contains one white ball and one black ball. We play the following game by drawing a ball at random. If the drawn ball is black, the game stops. If it is white, the drawn ball is replaced and an additional white ball is placed in the box. We continue to play this game until the black ball is finally drawn. Let X be the number of drawings required to obtain a black ball. What is expected value of X ? (b) [4] Consider the same game except that we start with more than a single ball of each color: we start with w white balls and b black ones. Repeat the calculation. Example 3.73 What is the expected number of children a couple will have until they have at least one boy and at least one girl, assuming each has the same birth probability?

Solution. Let X be the desired number; clearly, X > 2. A simple way to look at it is that once a child is born, they enter a wait for a child of the another sex—a sequence of trials that behaves just like the game Alice and Bob played described in Example 3.70. That is, X = 1 +Y, where

Pr[Y = k] =

1 . 2k

(1)

The expected value of Y is evaluated by the formula for G∞ (1/2), as given in Eq. (2.14): E[Y ] =

1/2

∑ k · 2−k = (1 − 1/2)2 = 2.

k>1


111

And since X is larger than Y by one (the first-born), the expected number of children the couple will raise is three. Exercise 3.74 [1] Prove that if there exists a constant a such that Pr[X > a] = 1, then E[X ] > a. Conversely, if there exists a constant b such that Pr[X 6 b] = 1, then E[X ] 6 b. Exercise 3.75 [2] From the set [1..n] = {1, . . . , n} we choose equiprobably a number, one at a time with replacement. Let X be the number of times when a chosen number is greater than any number drawn previously. What is expected value of X ? Exercise 3.76 [2] Below are the first four lines of Lord Byron’s verse One struggle more, and I am free: One struggle more, and I am free From pangs that rend this heart in twain; One last long sigh to Love and thee, Then back to busy life again. Suppose a word is selected at random—uniformly—from those lines. What is its expected length? Exercise 3.77 [2] The Acme Insurance Company entered the business of insuring data centers against random failures. They estimate that a given center has a 1% chance of failing in a year, and the likelihood of more than one failure in a year is thought to be so small that it may be disregarded. The damage caused by a failure is a random variable with the following PMF: Damage ($K) Probability

10

20

40

50

60

0.12069

0.2069

0.44828

0.17241

0.05172

What should Acme’s annual premium for failure insurance be on this center? Assume that there is no deductible, and that its policy is to pay, on average, 40 cents out of a dollar premium. Exercise 3.78 [2] An urn contains nine balls, which are numbered as follows: 0,1,1,2,3,3,5,5, and 9. Three of the balls are randomly selected without replacement. What is the expected value of the sum of the numbers on the selected balls? Note: While it is surely possible to tally all possible choices, either by hand or using a machine (for example, using MAPLE, the function choose generates all the required selections and you can sum them up), a closer look at the properties of the expectation operator will show you a way to avoid nearly all calculations. Exercise 3.79 [2] From the set of the first n positive integers, [1..n], a number is chosen at random, one at a time, until a repetition occurs. Find the expected number of samples. Exercise 3.80 [2] Suppose we want to insert a new entry in a table of size m containing n entries, and our way of doing it is by selecting positions at random, uniformly, until we find a position which does not have an entry. That is where we insert the new one. Let us define R > 1 as the number of positions we look at, until the entry is inserted. (a) Compute the PMF of R and its expected value under the assumption that each selection is done uniformly over the m positions in the table. (Sampling with replacement.) In §4.1.3 we shall see that R is a geometrical random variable. (b) Compute the PMF of R and its expected value under the assumption that each selection

112


is done uniformly over the positions in the table which have not been tried before. Now R has a finite support: since we only fail by hitting an entry already in the table, R 6 n + 1. (Sampling without replacement.) Exercise 3.81 [2] In a random permutation of [1..n], let Sn be the random variable that equals the number of entries that are in their proper places. What is the expected value, E[Sn ]? The variable Sn is further discussed in Exercise 5.133. Hint: Use an indicator random variable. Exercise 3.82 A tournament can be considered as a complete directed graph, with each arc representing a match between the two “players” at the nodes; it was defined in Exercise 1.40, page 20. We examine here labeled tournaments, in which each node carries a distinct label. (Counting unlabeled structures is usually much harder, and this is no exception.) A Hamiltonian cycle of a directed graph is a list of all the nodes, in some order, v1 , say, to vn , such that there is an arc directed from vi to vi+1 , for 1 6 i < n, and also from vn to v1 . (a) [1] What is Tn , the number of different labeled tournaments of size n (size is the number of nodes)? (b) [2] Show that a tournament of 4 nodes can have at most a single Hamiltonian cycle. (c) [2] Given a complete, undirected graph, we create a tournament by making each edge an arc: giving it a direction. Assume that the orientation of each arc is determined at random with probability one-half for each direction, and all choices are independent. What is the probability that a particular permutation of the n nodes provides a Hamiltonian cycle? (d) [2] Compute the expected number of Hamiltonian cycles in a tournament of size n, constructed as in part (c). (Suggestion: define, for a given permutation π , an indicator random variable Xπ , which is one when the permutation is a Hamiltonian cycle, and zero otherwise.) Compute numerical values for this expected value, for n at some points in the range [3, 60]. Any comments? (e) [4] What is the probability, P(n), that a tournament of size n, constructed as in part (c), has a Hamiltonian cycle? Note: We do not know an explicit answer to this part, but it is possible to write a recurrence for this value, and J. Moon shows in [101, p. 2] how to get from it a (very) good approximation. Exercise 3.83 [3] An urn contains 3 white balls and 2n − 3 black balls, with n > 2. Balls are drawn from the urn two at a time, at random and without replacement. At each stage, the two balls drawn are inspected. If they are of the same color, they are set aside and two more balls are drawn. This process continues until a drawn pair consists of two balls of different color. Find the expected number of balls drawn before the process is stopped. Exercise 3.84 [3] Compute the expected values for the random variables R and R′ defined in Exercise 3.46 (Banach matchbox problem).

3.4.3 Variance and Moments We have introduced the expected value, or mean, of a random variable. The expected value provides only limited information about the probability mass function, and, in particular, none on the shape of the distribution. Complete disclosure of the probability distribution is possible if we have an explicit expression for it, but it turns out that this is equivalent to knowing all the moments of the random variable. Moments are defined as follows: Definition 3.85 The nth moment about the origin (if it exists) of a random variable X ,


113

denoted by µn′ , is the expected value of X n ,

µn′ = E[X n] = ∑ Pr[X = x] xn ,

n > 1.

(3.22)

x

def

A common notation for the first moment is µ = µ1′ .

⊳

Definition 3.86 The nth moment about the mean (also called the central moment) of a random variable X , denoted by µn , is the expected value of (X − µ )n ,

µn = E[(X − µ )n ] = ∑ Pr[X = x] (x − µ )n ,

n > 1.

(3.23)

⊳

x

This is probably the most commonly used type of moments. When the word moment is used in this text, without any qualifier, the moment about the mean is understood. Exercise 3.87 [2] Let X be a discrete random variable and n be a positive integer. (a) Show that if E[X 2 ] exists then E[X ] exists. (b) Generalize the previous argument to show that if E[X n ] exists then so does E[X j ] for j = 0, 1, . . . , n − 1. (c) Show that if the nth moment about the origin exists then so does the nth moment about the mean. You may think that two types of moments are enough, especially since you can express all the moments of one type by the other ones (as you are asked to show in Exercise 3.92). However, statisticians found that other combinations turn out to be useful. Definition 3.88 The nth factorial moment of a random variable X , denoted by µ[n] , is the expected value of X (X − 1) · · · (X − n + 1) = X n ,

µ[n] = ∑ Pr[X = x] x(x − 1)(x − 2) · · · (x − n + 1) = ∑ Pr[X = x] xn .

⊳

(3.24)

x

x

While the random variables we use commonly have moments of all orders, we find that the first two provide most of the information needed in practice. In particular, it is the second central moment that we use in addition to the mean, µ :

Definition 3.89 The second moment about the mean is called the variance of a random variable X , or simply the variance16 of X , and is denoted by σX2 or V [X ]. The subscript is dropped when no confusion is likely. The positive square root of the variance, σ , is called the standard deviation: q p σ = E[(X − µ )2 ] = V [X ], provided the expected values exist.

⊳

Exercise 3.90 [3] For a non-negative integer random variable X , show that !2 V [X ] ≡ µ =

∑ (2k − 1) Pr[X > k] − ∑ Pr[X > k]

k>1

k>0 2

= 2 ∑ k Pr[X > k] − E[X ] − (E[X ]) . k>1

16 Sometimes,

the variance is also called the dispersion.

(3.25)


114

Also

E[X 2 ] =

∑ k2 p(k) = ∑ (2k − 1)Pr[X > k].

(3.26)

k>1

k>0

Exercise 3.91 [2] Show that E[(X − t)2 ] is minimized in t when t = E[X ].

Exercise 3.92 [3] (a) Use the definition of the central moments to write an explicit expression for µn in terms of all µk′ , 0 6 k 6 n. Then reverse the expression to get µn′ in terms of all µk , 0 6 k 6 n. (b) Write an expression for the moments (about the origin) in terms of the factorial moments; also in the reverse direction. Hint: Use Stirling numbers; see Appendix C. The variance characterizes the variability, or dispersion, of a random variable about its mean. We usually prefer to compare the standard deviation with the mean, rather than the variance, since σ and µ have the same dimensions. When σ is small, compared to µ , the random variable is concentrated near its mean. As σ increases, using the mean provides less and less information about likely values of the variable. The mean and standard deviation are usually called location and scale parameters, respectively. They do not determine a shape of a distribution, and other parameters are used to characterize them at least partially. Moments of order higher than 2 are relevant in many classical statistical tests when one assumes conditions of normality. These assumptions may be checked by examining third and fourth moments. A measure of the asymmetry (or lack of symmetry) of the distribution about its mean is the skewness, defined as the normalized third central moment. The fourth moment is associated with the “peakedness” or “flatness” of a probability distribution. Definition 3.93 Let X be a random variable whose fourth moment exists. The dimensionless ratios E[(X − µ )3 ] E[(X − µ )4 ] and −3 = β1 = β 2 σ3 σ4 are called the coefficient of skewness and the coefficient of kurtosis, respectively.

⊳

The skewness of a distribution symmetrical about the mean is zero and, in general, it describes the symmetry of the tails. When it has a longer tail to the right of the mean, β1 is positive, and the distribution is said to be skewed positively, or to the right. The kurtosis, also known as the second shape parameter, gives a refinement of the information provided by the variance, and is more sensitive to the dispersion of the distribution. It describes the peakedness of a distribution. The “−3” is chosen such that β2 vanishes for the standard Normal distribution N(0, 1), defined in §4.1.7. Example 3.94 Let X be the number that comes up while rolling a fair die. Then 6

µ ≡ E[X ] = ∑ i · i=1

1 6·7 1 7 = · = , 6 2 6 2

6

and

E[X 2 ] = ∑ i2 · i=1

1 6 · 7 · 13 1 91 = · = . 6 6 6 6

The variance of a roll of such a die is:

σ 2 ≡ V [X ] =

91 49 35 − = ≈ 2.917 ≈ 1.7082 . 6 4 12


115

p Hence the square root of the variance, the standard deviation, has the value σ = 35/12 ≈ 1.708. The third and fourth moments of the random variable X have the following values:

µ3′ = E[X 3] = µ4′ = E[X 4] =

1 6 1 6

6

1

147 = 73.5 2

(see Eq. (2.10)),

1

2275 ≈ 379.1666 . . . 6

(see Eq. (F.10)),

∑ i3 = 6 Φ3(6) =

i=1 6

∑ i4 = 6 Φ4(6) =

i=1

µ3 = E[(X − µ )3 ] = E[X 3 ] − 3µ E[X 2 ] + 3µ 2 E[X ] − µ 3 3 3 7 7 7 91 147 −3· · +3 − = 0, naturally, by symmetry, = 2 2 6 2 2

µ4 = E[(X − µ )4 ] = E[X 4 ] − 4µ E[X 3 ] + 6µ 2 E[X 2] − 4µ 3 E[X ] + µ 4 2 4 4 7 7 7 7 147 91 707 2275 −4· · +6 ≈ 14.729 . . . , · −4 + = = 6 2 2 2 6 2 2 48 91 7 35 − = ≈ 11.6666 . . . , 6 2 3 147 91 = E[X (X − 1)(X − 2)] = E[X 3] − 3E[X 2 ] + 2E[X ] = − + 7 = 35, 2 2 4 3 2 = E[X ] − 6E[X ] + 11E[X ] − 6E[X ] = 84.

µ[2] = E[X (X − 1)] = E[X 2] − E[X ] = µ[3] µ[4]

The coefficient of skewness is zero, so the random variable X has a symmetric distribution (what would you expect from a uniform distribution?). The coefficient of kurtosis is

β2 = The three quartiles are

707 · 12 µ4 −3 = − 3 ≈ −1.2686 . . . . 4 σ 4 · 352

x1/4 = 2,

x1/2 = m = 3,

x3/4 = 5.

The mode does not exist for a uniformly distributed random variable (as in our case) because each member of the sample space has the same probability. The median is 3. Example 3.95 (Uniform distribution) Consider a continuous random variable having uniform distribution on the interval (a, b), a < b. Its nth non-central moment is

µk′ =

1 b−a

Z b

xn dx =

a

bn+1 − an+1 , (b − a) (n + 1)

n = 1, 2, . . . .

(3.27)

From this equation, we obtain the values of mean, µ , and the variance, σ 2 :

µ=

a+b , 2

σ2 =

(b − a)2 . 12

So the coefficient of skewness is

√ a3 + 2a2 b + 2ab2 + b3 8 3 (a3 + 2a2 b + 2ab2 + b3 ) = , β1 = 3σ3 (b − a)3

(3.28)


116

which becomes zero when a = −b, and the coefficient of kurtosis is 6 β2 = − = −1.2 . 5 When we looked at the properties of the expected value operator, Property 2 on page 100 says that E[X +Y ] = E[X ] + E[Y ], regardless of the possible dependence between the random variables. We show that the situation with variances is different. Using two variables, we have V [X +Y ] = E[(X +Y − µX − µY )2 ] = E[((X − µX ) + (Y − µY ))2 ]

= E[(X − µX )2 + (Y − µY )2 + 2(X − µX )(Y − µY )],

the linearity of the expectation operator E gives again: = V [X ] +V [Y ] + 2E[XY − X µY −Y µX + µX µY ] = V [X ] +V [Y ] + 2E[XY − µX µY ]. This result contains the expression E[XY ]. When E[XY ] = E[X ] × E[Y ] = µX µY , the variance of the sum becomes V [X + Y ] = V [X ] + V [X ]. Generally speaking this is not true. In §3.8, we discuss the conditions on random variables to make it happen (see property C4 in Theorem 3.143, page 130). Theorem 3.96 Let X be a random variable and E[X 2] < ∞. Then the following properties hold: V1: V [X ] = E[X 2 ] − (E[X ])2 . V2: V [X ] = 0 if and only if there exists a constant C such that Pr[X = C] = 1. V3: V [aX + b] = a2V [X ] for any constants a and b. Exercise 3.97 [2+] Prove Theorem 3.96. Example 3.98 Consider a random variable X that takes integer values in the range [0..n], where n is a positive integer, with probabilities pi = Pr[X = i] = a ni . The normalization constant a is needed to satisfy the normalization condition: ∑i pi = 1: n 1 = ∑ pi = a ∑ = a2n =⇒ a = 2−n . i i i Let us compute the moments of X . Using Eq. (2.58), we find n n−1 n −n −n = 2 ·n·∑ = 2−n · n · 2n−1 = , µ ≡ E[X ] = ∑ ipi = 2 ∑ i 2 i i−1 i i i as expected, by dint of symmetry. Similarly, we calculate: n−1 n−1 2 −n 2 n −n −n E[X ] = 2 ∑ i = 2 ∑i · n· = n·2 ∑ (i − 1 + 1) i i−1 i−1 i i i " # n − 1 n − 1 = n · 2−n ∑ (i − 1) + ∑ i−1 i−1 i i


117

1 n n2 n n 1 = n · 2−n (n − 1)2n−2 + 2n−1 = n(n − 1) + = − + = n(n + 1). 4 2 4 4 2 4

− E[X ]]2 = E[X 2 ] − E 2[X ] = n(n + 1)/4 − n2 /4 = n/4. Now the variance: σ 2 ≡ V [X ] = E[Xp √ Then the standard deviation is σ = V [X ] = n/2. This is the “expected deviation” from E[X ], and its salient feature here is that when n is large, the deviation is very small compared with the mean E[X ]. Another way of saying the same is that the PMF gets concentrated, the entire weight lies near the mean value. From Exercise 2.114, it follows that the random variable X has one mode, 2n , if n is even and n+1 there are two modes, n−1 2 and 2 , if n is odd. The median is, by symmetry, ⌈n/2⌉. The higher moments are evaluated similarly. n n−1 n−1 E[X 3 ] = 2−n ∑ i3 = 2−n ∑ i2 · n · = n · 2−n ∑( j + 1)2 i i−1 j i i j n−1 n−1 n2 −n 2 n−1 −n −n = n·2 ∑ j + 2n · 2 ∑ j +n·2 ∑ = (n + 3), 8 j j j j j j n 3 (n + 6n2 + 3n − 2). E[X 4 ] = 16 Therefore, E[(X − µ )3 ] = 0 =⇒ β1 = 0, 2 n (3n − 2) =⇒ β2 = − . E[(X − µ )4 ] = 16 n Since the coefficient of skewness, β1 , is zero, the distribution of the random variable X is symmetric (which also follows from Eq. (2.56) on page 58).

Example 3.99 Consider the set of all graphs that are unlabeled and undirected with 3 nodes (this set is denoted by U G3 ):

Let us introduce a uniform probability distribution over all graphs from the set U G3 . It is clear that |U G3 | = 4. Now define a random variable D to be the number of edges in a graph. We have one graph with each value of this variable from 0 to 3. Since it inherits the uniform 1 1 1 1 probability assignment, pD (·) = 4 , 4 , 4 , 4 .

We evaluate the mean and find

1 1 1 6 1 E[D] = 0 · + 1 · + 2 · + 3 · = = 1.5. 4 4 4 4 4 This is the expected number of edges. The second moment becomes 1 1 14 1 = 3.5 E[D2 ] = 1 · + 4 · + 9 · = 4 4 4 4

=⇒

V [D] = 3.5 − 2.25 = 1.25.


118

Now let us look at LG3 , the set of all labeled, undirected 3 node-graphs. Using the previous diagram to visualize the new sample space, we see that we have one graph with no edges, 3 graphs with one edge (since the “free” node can get three different labels, and the other two are symmetrically used), 3 graphs with two edges, and one graph with three edges. Assuming again they are all equally likely, we get a different probability mass function for the same random variable: 1 3 3 1 , , , , pD (·) = 8 8 8 8 and

3 3 1 12 1 = 1.5. E[D] = 0 · + 1 · + 2 · + 3 · = 8 8 8 8 8 This is the same mean value as before due to symmetry! However, the variance 3 1 3 E[D2 ] = 1 · + 4 · + 9 · = 3 8 8 8

=⇒

V [D] = 0.75,

is much smaller now in a way that corresponds to the non-uniformity of the PMF. To complete this comparison of probability mass functions (PMFs), suppose we turn the above PMF around and denote by Y the random variable with the PMF: pY (·) = 83 , 81 , 81 , 83 . A similar calculation provides E[Y ] = 1.5, as required by symmetry, but now E[Y 2 ] = 4 and V [Y ] = 1.75, which is larger than any of the values above. This corresponds to the probability mass function being concentrated at points away from the mean. Exercise 3.100 [2] (E. Deutsch) Let π = π1 π2 · · · πn be a random permutation of [1..n] and let X (π ) = max{i : π j > π1 for all j 6 i}. Find the expected value and variance of X .

Exercise 3.101 [2] A permutation π of n (n > 1) first positive integers is said to have a local maximum at k if π (1) > π (2) for k = 1, π (n) > π (n − 1) for k = n and π (k − 1) < π (k) < π (k + 1) for 1 < k < n. Find the expected number of local maxima in the set of all permutations of [1..n]. Exercise 3.102 [3] Given the information that the random variable X satisfies E[(X − 1)2 ] = a > 0 and E[(X − 2)2 ] = b > 0. (a) Compute the mean and variance of X in terms of a and b. (b) How are the values of a and b limited? Can they be a = 1, b = 5? How about a = 10, b = 5? (c) Given E[(X − α )2 ] = a and E[(X − β )2 ] = b, find the mean and variance of X . Are there any constraints on numbers a, b, α , and β ? Exercise 3.103 [2] Consider the integer-valued random variable X having distribution Pr[X = k] = 2−k ], k = 1, 2, . . .. Find its mean, variance, median, mode, the coefficient of skewness, the coefficient of kurtosis, and the lower and upper octile x1/8 and x7/8 . Exercise 3.104 [2] Suppose we have a large program, consisting of 105 lines of code. It is produced in a process that is known to leave Y errors per 1000 lines of code where Y is a random variable having the PMF pY ( j) = (0.01, 0.25, 0.4, 0.3, 0.04), for j = 0, 1, 2, 3, 4. Assuming that successive blocks of code have independent numbers of lines containing an error, what are the mean and variance of the number of lines of code with at least one error in the program?

3.5. CONDITIONAL PROBABILITIES

119

3.4.4 Functions of Random Variables In many cases, the random variables being used in an experiment are not the researcher’s ultimate objective. Sometimes the random variable of interest is too complicated to analyze. Thus, it makes sense to represent such random variable as a function of known random variables. For example, let X be the velocity of a particle of mass m. We might be interested in analyzing the particle’s kinetic energy Y = 12 mX 2 . Other situations may require a change in scale of a random variable. A life of an electronic system may depend on the sum of two or more components lives. Such a very important and practical situation is treated separately in §4.5. Another problem of transforming a uniformly distributed random variable appears frequently in system simulation. A pseudorandom generator is one of the most useful tools. However, it supposedly generates uniformly distributed numbers in the unit interval. Denote the corresponding random variable by U . A researcher might need to find a function f such that the random variable Y = f (U ) has the required properties. In general, let X be a discrete random variable with the probability mass function pX (k), and let Y = f (X ) be its transformation. Finding pY ( j), the probability mass function for the random variable Y , is not an easy task and it depends on the complexity of the transformation f . If, for example, f is a linear function, f (x) = ax + b (a 6= 0 and b are constants) and Y = aX + b, then j−b . pY ( j) = pX a

In a similar way, we can find the probability mass function for Y = f (X ) when the function f has a simple form, such as f (x) = x2 or f (x) = 1/x. The general case depends on the form of the inverse function. Nevertheless, it is always possible to find the basic characteristic of Y . For instance, (3.29) E[ f (X )] = ∑ f (k)pX (k) = ∑ f (k) Pr[X = k]. k

k

When confronted with a function that seems to have a complicated form, we approximate it def through a Taylor series about the mean µX = E[X ]:

1 Y = f (X ) ≈ f (µX ) + (X − µX ) f ′ (µX ) + (X − µX )2 f ′′ (µX ). 2 Taking expectation of the right-hand side, we obtain 1 E[Y ] = E[ f (X )] ≈ f (µX ) + σX2 f ′′ (µX ), 2 provided that the variance, σX2 = V [X ], exists and is known. How good such approximation is depends on how accurate the quadratic formula is in a neighborhood of µX and on the value of σX .

3.5 Conditional Probabilities The conditional probability of an event normally differs from unconditional probability, because it uses information we already know about the experiment. Once we learn such in-


120

formation, we update the sample space, keeping only the portions that are still possible. We recalculate the probabilities of the events of the new, reduced sample space. Formally, we construct a new probability space with a sample space that is limited to the outcomes (elementary events) that satisfy the given condition. Definition 3.105 The conditional probability Pr[A | B] (also denoted by PrB [A]) is the probability of an event A when it is known that the event B occurred. This probability can be computed as follows: Pr[A | B] ≡ PrB [A] =

Pr[A ∩ B] . Pr[B]

(3.30)

⊳

Notes: (1) There are versions of the definition that add the restriction Pr[B] > 0; this is certainly not needed when dealing with discrete spaces (if Pr[B] = 0, the conditional event could not happen). (2) The explanation or definition may suggest that we refer to event A occurring after B occurred. This is a grammatical artifact; time relates here only to the “passage” of information: we already have information about B, but not about A, and both refer to the outcome of the same experiment, as the following examples show. The following properties of conditional probability follow immediately from Definition 3.105: • Pr[A | A] = 1,

Pr[∅ | A] = 0,

• Pr[A | B] = 1

if B ⊆ A,

• Pr[A1 ∪ A2 | B] = Pr[A1 | B] + Pr[A2 | B]. The ratio Pr[A | B]/ Pr[A] gives a measure of how strongly associated A and B are. The event A is less likely to occur given B if Pr[A | B] < Pr[A]. Two events A1 and A2 are said to be more strongly associated conditionally given event B than event C if PrC [A1 A2 ] PrB [A1 A2 ] > . PrB [A1 ] PrB [A2 ] PrC [A1 ] PrC [A2 ]

(3.31)

The events A1 and A2 are equally associated conditionally under B and C if equality holds in Eq. (3.31). Example 3.106 (Rolling a die) We roll a fair die. Let A be the event of getting either a 4 or a 6, and B denote the event of rolling a 5 or better (that is, 6). Find the probability of A. Compute the conditional probability of A given that the event B has already occurred. The sample space and the related events are shown in the figure on the side.

1

3

5 B

2

4

A

6

Solution: To compute the probability of A, we notice that |A| = 2, |Ω| = 6, and hence Pr[A] = 2/6 = 1/3. For the conditional probability, the cardinality of A ∩ B is |A ∩ B| = 1. From

3.5. CONDITIONAL PROBABILITIES

121

Eq. (3.30), using the fairness of the die (equivalent to saying that all possible outcomes are equally likely), we have Pr[A ∩ B] |A ∩ B|/|Ω| |A ∩ B| 1 = = = . Pr[B] |B|/|Ω| |B| 2 Example 3.107 Consider the genders of the children in two-child families, with b and g corresponding to a boy and a girl respectively. The relevant sample space is Ω = { (b,b), (b,g), (g, b), (g,g) }, where the first letter refers to the older child and the second letter refers to the younger one. For example, the event (b,g) means that the first child is a boy whose sister is younger. What is the probability that both children are boys given that at least one is a boy? Solution. Denote A = { (b, b) } and let B be the event “at least one child is a boy.” Then B = {(b,b), (b,g), (g,b)}, and Eq. (3.30) leads to Pr[A | B] =

Pr[A | B] =

Pr[A ∩ B] |A ∩ B| 1 = = . Pr[B] |B| 3

Example 3.108 You are given two coins, one is fair and the other coin has heads on both sides. Suppose you choose a coin at random where each has equal probability to be chosen. Then we flip it, what is the probability to get a fair coin given that you saw heads? The general case is considered in Exercise 4.110, page 172. Solution. Define the sample space to consist of the ordered pairs: hcoin, outcomei:

{(a, H), (a, T), (b, H)}, where a denotes a fair coin and b—two heads coin. Let E1 =“choose coin A” and E2 = “ outcome is heads.” Then Pr[E1 ∩ E2 ] 1/4 1 = = . Pr[E1 | E2 ] = Pr[E2 ] 3/4 3 Example 3.109 (A lottery problem, continuation of Example 3.22, page 91) Suppose you bought two tickets of total m, and you knew that there were n lucky tickets with a prize. What is the probability that the second ticket is lucky assuming that the result of the first one is unknown? Solution. Let B be the event that the first ticket is lucky, and A the event that the second one is also a winning ticket. Then n−1 Pr[AB] n(n − 1)/m(m − 1) = = , Pr[B] n/m m−1 n Pr[AB] n(m − n)/m(m − 1) = = , Pr[A | B] = (m − n)/m m−1 Pr[B]

Pr[A | B] =

and

Pr[A] = Pr[A | B] Pr[B] + Pr[A | B] Pr[B] n m−n n n−1 n · + ··· = . = m−1 m m−1 m m Hence, when the nature of the first ticket is unknown, it does not affect the probability that the second ticket is lucky.


122

Exercise 3.110 [1] A card is drawn at random from a full deck of 52 cards. What is the probability that the card is club given that it is a queen? Exercise 3.111 [1+] Consider throwing two fair dice with events A1 = {(x, y) : y = 2, 3, 4}, A2 = {(x, y) : x = 5, 6}, B = {(x, y) : x + y = 7}, and C = {(x, y) : x + y = 9}. Show that the events A1 and A2 are equally associated conditionally under events B and B′ , but not C and C′ . Sometimes a tree diagram helps to understand problems involving conditional probabilities. Example 3.112 Two cards are dealt from a full deck. Find the probability that both are clubs.

Analysis. We first observe that there is no need to consider conditional probabilities, as this is a special caseof the hypergeometric distribution considered in §4.1.5, and the desired 52 1 probability is 13 / = 17 . But we wish to use this problem to illustrate a point. Let A be 2 2 the event that the first card is a club, and B be the event that the second card is a club. Then since 13 of the 52 cards are clubs, Pr[A] =

13 1 = . 52 4

To find the desired probability, Pr[A ∩ B], we use Eq. (3.30); it calls for Pr[B | A], which is the probability of getting one of the 12 clubs out of the 51 remaining cards in the pack: Pr[A ∩ B] = Pr[B | A] · Pr[A] =

1 12 13 · = ≈ 0.0588. 51 52 17

This can be described pictorially with a tree diagram (Figure 123). The first column lists the possible outcomes of the first stage of the experiment, drawing the first card, which are “club” and “not a club.” The second column gives the possible outcomes of drawing the second card. A branch leading from a first-stage outcome to a second-stage outcome carries the conditional probability. The alert reader will observe that there is really no need to go through the process of evaluating conditional probabilities here. We can imagine the two cards removed “at once,” and 52 ways to pick two clubs out of the ways of drawing two out of the entire there are 13 2 2 deck. The desired probability is the ratio of these numbers, which is the same 1/17, of course. But life is not always that accommodating. Exercise 3.113 [3] (a) A discrete random variable X has following probability distribution: 8 1 k pX (k) = 15 2 , for k ∈ [0..3]. What is the probability Pr[X > 2 | X > 1]? (b) Let n > m be two positive integers, and let X be a random variable with support on [1..n], with probabilities pX (k) = Pr[X = k] that are proportional to k. (This distribution is called the linear PMF). What is Pr[X = 1 | 1 6 X 6 m]?

Exercise 3.114 [1] Given that Pr[A] = a and Pr[B] = b, show that Pr[A | B] > a+b−1 b . Hint: Denote the joint probability Pr[AB] ≡ Pr[A ∩ B] by c, and use the fact that no event has a probability that exceeds 1. We extend Definition 3.105 to any finite number of events. Suppose that A1 , A2 , . . ., An and B are events such that Pr[B] > 0 and Pr[A1 ∩ A2 ∩ · · · ∩ An ] > 0. Then Pr[A1 A2 · · · An ] = Pr[A1 ]Pr[A2 | A1 ]Pr[A3 | A1 ∩ A2 ] · · · Pr[An | A1 ∩ A2 ∩ · · · ∩ An−1 ]. (3.32)

3.6. INDEPENDENCE

123 First card

Second card

Probability of entire limb

12/51 club

club

13 12 52 51

not a club

13 39 52 51

club

39 13 52 51

not a club

39 38 52 51

39/51

13/52

39/52

13/51 not a club

38/51

Figure 123: Tree diagram for Example 3.112. Example 3.115 Three balls are selected at random, without replacement, from a box containing b black balls and w white balls. An outcome is a 3-letter word of the two letters B and W . Let B j denote the event of obtaining a black ball on j-th draw and W j denote the event of getting a white ball on j-th draw. Then from Eq. (3.32), we obtain Pr[W1W2 B3 ] =

w−1 b w · · . b+w b+w−1 b+w−3

Exercise 3.116 [1] A box contains b black balls and w white balls. One ball is picked up at random and its color is observed. The ball is returned with additional k balls of the same color. Then another ball is chosen at random, its color is observed, and it is returned to the box together with k additional balls of the same color. The process is repeated in similar manner. If three balls are selected, what is the probability that the first two balls will be white and the third ball will be black? Exercise 3.117 [2] A fair coin is tossed independently n times and let k 6 n be a positive integer. Determine the probability of obtaining exactly n − 1 tails given (a) that at least n − k tails are obtained; (b) that tails only are obtained on the first n − k tosses.

3.6 Independence It may happen that the occurrence of an event A does not affect the likelihood of the event B. Suppose you flip a coin (the event A) and roll a die (the event B). Does the result of one provide information on the other? The answer is no. Definition 3.118 The events A and B are independent if Pr[A | B] = Pr[A];

(3.33)

otherwise they are said to be dependent. Eq. (3.33) implies that PrA [B] ≡ Pr[B | A] = Pr[B], which corresponds to the intuitive understanding of independence as a symmetric relation.


124

Two events A1 and A2 are said to be conditionally independent with respect to B if PrB [A2 | A1 ] = PrB [A2 ]. The conditional independence becomes mutual independence when B is the sample space. ⊳ Example 3.119 For the experiment of rolling a fair die, consider the following events: A1 = {2, 4}, A2 = {3, 4}, and B = {1, 2, 3, 4}. Then the events A1 and A2 are dependent because Pr[A1 ] = Pr[A2 ] = 31 , but Pr[A1 ∩ A2 ] = 61 . On the other hand, Pr[A1 ∩ A2 | B] = 41 and Pr[A1 | B] = Pr[A2 | B] = 12 , so Pr[A1 ∩ A1 | B] = Pr[A2 | B] Pr[A2 | B] and the events A1 and A2 are conditionally independent with respect to B. From the same sample space, Ω = {1, 2, 3, 4, 5, 6}, we specify a few new events out of 26 subsets of Ω, in terms of the roll values (the number of pips on the up-turned face of the die): (even) E = {2, 4, 6},

(prime) P = {2, 3, 5},

(large) L = {5, 6}

What is the probability we roll an even value? Clearly Pr[E] = 1/2. But if we ask about the probability of rolling an even value, given that we obtain a prime value, we find, using Eq. (3.30), that Pr[E ∩ P] Pr[{2}] 1/6 1 1 = = = 6= . Pr[E | P] = Pr[P] 1/2 1/2 3 2 Hence the events E and P are dependent. Now an entirely analogous calculation provides Pr[E | L] = L are independent.

1 2

= Pr[E]. So, the events E and

Do you find saying so uncomfortable? It is a claim which satisfies the definition! and yet, many people, the author among them, would rather say the events are uncorrelated17, rather than independent, answering to the feeling that the above equality is a numerical fluke rather than a reflection of the structure of the probability space.

Note that events can only be independent if they are not disjoint; otherwise the probability of their intersection is zero! Combining Eqs. (3.30) and (3.33), we see a decision criterion for the independence of events: Theorem 3.120 Two events A and B are independent if and only if Pr[A ∩ B] = Pr[A] · Pr[B].

(3.34)

This theorem makes it apparent that independence (as well as dependence) is a symmetric property, which corresponds to our intuition: if A is not dependent on B, the latter should not depend on A. And yet, there have been formulations, based on the notion that dependence reflects ‘flow of information,’ which violate this symmetry, see [99, §6.7]. Definition 3.118 may sometimes lead us to non-intuitive claims. For instance, consider again Example 3.106 on page 120, and let C be the event there of rolling an even number. Then it is easy to see that Pr[C] = 1/2, and also that the probability of getting an even number given that this number is either 5 or 6 is PrB [C] ≡ Pr[C | B] = Pr[B ∩C]/ Pr[B] = (1/6)/(1/3) = 1/2. While C and B are surely “coupled,” the event probabilities are such that the occurrence of B does not change the probability of C, even though some sample points in C are excluded! An 17 This

is a term that arrives from properties of random variables, see Definition 3.142; it is used here as a term which is weaker than ‘independent.’

3.6. INDEPENDENCE

125

alternative statement that this example highlights is that dependence between events results not only from the “geometry” of the sample space and the events, but is also due to the specific assignment of probabilities. Let D be another event that rolling a die produces a number less than or equal to 4. Since Pr[D] = 46 = 32 and Pr[D ∩C] = 62 = 31 , we have 2 = Pr[D]. 3 The events D and C are independent and the knowledge that you rolled a number 6 4 does not help in deciding whether it is odd or even. Pr[D | C] = Pr[D ∩C]/ Pr[C] =

Dependence (or independence) may even be contingent on a continuous parameter: Let us toss a biased coin, with Pr[heads] = p, three times, and consider the two events A = { at most one toss was a tails }, and B = { all three tosses gave identical results}. These events correspond to the sequences A = {HHH, HHT, HTH, THH}, and B = {HHH, TTT}. Hence Pr[A] = p3 + 3p2 (1 − p),

Pr[B] = p3 + (1 − p)3 ,

Pr[AB] = p3

When can we have Pr[AB] = Pr[A] Pr[B]? It poses the equation p3 = [(p3 + (1 − p)3 ][p3 + 3p2 (1 − p)], which is equivalent to p2 (2p3 − 5p2 + 4p − 1) = 0. This equation has the solutions p = {0, 1/2, 1}. For these values, the events are independent, but not otherwise. Exercise 3.121 [1] (a) Show that the empty event (the empty set, ∅) is independent of any event. Show that the same claim holds for its complement, the sample space. (b) Show a claim that appears counter-intuitive at first: two disjoint events (with nonzero probabilities) are always dependent. Exercise 3.122 [2] Prove: if events A and B are independent, then so are (a) A′ and B; (b) A′ and B′ . ′ ′ Note: A and B are the events complementary to A and B, respectively.

Now we define the independence of several events. Definition 3.123 The n given events A1 , A2 , . . ., An are called independent if, for every subset Ai1 , Ai2 , . . ., Aik of k of these events (k = 2, 3, . . . n), Pr[Ai1 ∩ Ai2 ∩ . . . ∩ Aik ] = Pr[Ai1 ] Pr[Ai2 ] . . . Pr[Aik ].

⊳

In particular, three events A, B, and C are independent if and only if the following four relations must be satisfied: Pr[A ∩ B] = Pr[A] Pr[B], Pr[A ∩C] = Pr[A] Pr[C],

Pr[C ∩ B] = Pr[C] Pr[B], Pr[A ∩ B ∩C] = Pr[A] Pr[B] Pr[C].

(3.35)

Note that disjoint events of positive probabilities are always not jointly independent; however, they could be pairwise independent as Exercise 3.127 shows. Definition 3.124 A sequence X1 , X2 , . . . , Xn of random variables is said to be independent (mutually independent) if its cumulative distribution function is the product of the marginal cumulative distribution functions: Pr[X1 6 x1 , X2 6 x2 , . . . , Xn 6 xn ] = Pr[X1 6 x1 ] Pr[X2 6 x2 ] · · · Pr[Xn 6 xn ] for all real n-tuples, (x1 , x2 , . . . , xn ).

⊳


126

Example 3.125 Consider an experiment of rolling a fair die in which the sample space contains six outcomes. Let us consider the three events defined as follows: A = {1, 2, 3},

B = {1, 4},

C = {1, 5}.

Then dropping the intersection sign for simplicity, we have AB = AC = BC = ABC = {1} and their probabilities are the same, 1/6. On the other hand, Pr[A] = 1/2, Pr[B] = Pr[C] = 1/3. These three events A, B, and C are not independent, but events A and B are pairwise independent, as are A and C. Can it be that the three events A, B,C in some arbitrary space Ω satisfy the first three equalities in relation (3.35) but not the fourth one? In this drawing, Pr[A] = n + m + s + r, Pr[B] = q + m + t + r, Pr[C] = u + s + t + r, Pr[AB] = m + r, A B Pr[AC] = s + r, Pr[BC] = t + r, Pr[ABC] = r. We then q m n require three out of four equations (3.35) to be satr t isfied, and would like to see an inequality instead of the fourth equation. Surely with seven parameters, s constrained to be positive (and sum to be less than 1) u Ω all probabilities should be possible! C Indeed, here is one possibility: n = q = u = 0.05; r = 0.03; m = s = t = 0.1995643924 ≈ 0.2 (the last value was provided by MAPLE, by solving the three equations). Also, with this solution, Pr[A] Pr[B] Pr[C] is very close to 0.11, much larger than r. Hence these events are pairwise independent, but not independent. Exercise 3.126 [2] Selecting a number at random from the set [1..n] = {1, 2, . . . , n}, let A be the event that this integer is divisible by 5, and let B be the event that this integer is divisible by 7. Are events A and B independent? Exercise 3.127 [1] Consider an urn containing four balls, numbered 110, 101, 011, and 000, from which one ball is drawn at random. For i = 1, 2, 3 let Ai (correspondingly Bi ) be the event of drawing a ball with 1 (correspondingly 0) in the i-th position. Show that these events are pairwise independent, but not jointly independent. Exercise 3.128 [2] In Example 3.57 on page 102, we considered two random variables: M, the number of moves made by insertion sort, and Y , the number of times the value from t was placed in position 0. Show that the variances of these random variables are V [M] =

n(n − 1)(2n + 5) 72

and

(2)

V [Y ] = Hn − Hn ,

(2)

where Hn and Hn are harmonic numbers (see Eq. (E.1), page 719). Hint: Represent Y as the sum of indicator random variables, introduced in Example 3.57. Exercise 3.129 [3+] This exercise considers the appearance of special subgraphs in a random graph. Let n be a positive integer. A random graph with n nodes is generated by connecting a pair of vertices with an edge in probability p. In such a graph, each of n2 possible edges could be chosen independently from others. For example, the graph would be full in n probability p(2) . For each of the two types of random graphs below, calculate the number of nodes required for the expected number of occurrences of such subgraphs to equal 1, for a

3.7. JOINT DISTRIBUTIONS

127

given fixed p. Then compute, for that critical number of nodes, the variance of the number of occurrences. (a) A 5-clique: this is a 5-node subgraph which is full, has all its 10 edges. (b) The complete 6-node bipartite graph K3,3 : Exercise 3.130 [3] (Rey, J.) A set consisting of n men and n women is partitioned at random into n disjoint pairs of people. What are the expected value and variance of the number of male-female couples that result?

3.7 Joint Distributions In many practical cases we desire to take more than one measurement on a random observation. For example, consider a system containing two computers, both are used, but one primarily for backup. Suppose the only way the system will malfunction is when both computers fail to operate. The distribution of the system’s life depends jointly on time-to-failure distributions of both computers. More generally, we may need to deal with the joint behavior of n random variables. For example, biomedical researchers face the problem of determining several factors of effect— serum cholesterol, blood pressure, and so on. Definition 3.131 Let X and Y be two discrete random variables defined on the same sample space Ω. The joint probability mass function is the function f (x, y) (or fX,Y (x, y)), where f (x, y) = Pr[X = x

and Y = y].

⊳

Note: The requirement that both random variables live in the same sample space is made here for convenience only, and can be waived, if needed. This definition can be similarly extended for several random variables. This joint probability mass function, f (x, y), has the following properties: • 0 6 f (x, y) 6 1; • ∑(x,y)∈Ω×Ω f (x, y) = 1; • Pr[(X ,Y ) ∈ A ⊆ Ω × Ω] = ∑(x,y)∈A f (x, y). Example 3.132 (Lottery, adapted from N. Henze [63]) From an amazing variety of lotteries played all over the world, we consider the most popular type—the 6/49 lottery. In this game, ping-pong balls numbered 1 through 49 are placed in a clear plastic drum and mixed thoroughly. Six balls are then selected randomly; if the numbers on your ticket agree with those on the selected balls (in any order), then you win the jackpot. The set of all lottery tickets is defined as follows T = {t = (t1 ,t2 , . . . ,t6 ) | 1 6 t1 < t2 < . . . < t6 6 49}. The purpose of this example is to introduce the “spacing” random variables X j (t) = t j+1 − t j ,

1 6 j 6 5.


128

What is the PMF of X j , evaluated over all possible tickets? We begin with Pr[X j > k], and note that requiring X j to be at least k prevents any of the k − 1 numbers following t j from being selected as t j+1 . Hence, assuming a uniform distribution over all 49 6 tickets, we find 49 − (k − 1) . 49 50 − k . 49 Pr[X j > k] = = , 1 6 k 6 44. (3.36) 6 6 6 6 Eq. (3.36) allows us to obtain the probability mass function (PMF) of X j as the difference 49 − k . 49 Pr[X j = k] = Pr[X j > k] − Pr[X j > k + 1] = . (3.37) 5 6 From Eq. (3.18), page 106, and the “vertical” recurrence (2.66), page 62, it follows that −1 44 50 − k 49 50 ≈ 7.1428 . . . . = E[X j ] = ∑ Pr[X j > k] = ∑ 7 6 6 k=1 k>1 Similarly, from Eq. (3.25) on page 113, we find the variance V [X j ] = 2 ∑ k Pr[X j > k] − E[X j ] − (E[X j ])2 = k>1

3225 ≈ 32.9081 . . . . 98

Consider the joint probability mass function of two spacings Xi and X j , where 1 6 i < j 6 5. It is shown in the following exercise that 51 − k − n . 49 Pr[Xi > k, X j > n] = , 1 6 k, n 6 48, (3.38) 6 6 By inspecting regions in the (Xi , X j ) plane and substituting from Eq. (3.38), we find: Pr[Xi = k, X j = n] = Pr[Xi > k, X j > n] + Pr[Xi > k + 1, X j > n + 1]

(3.39)

− Pr[Xi > k, X j > n + 1] − Pr[Xi > k + 1, X j > n] 49 − k − n . 49 = , 4 6 where we have used the addition formula of the binomial coefficients three times (see part (c) of Exercise 3.133 on page 128). Exercise 3.133 [3] We define a generalized r/n lottery as a game in which the winning “ticket” is an r-element subset of [1..n] = {1, 2, . . . , n}. Consider a ticket t = (t1 ,t2 , . . . ,tr ) with 1 6 t1 < t2 < . . . < tr 6 n, and let X j (t) = t j+1 − t j , j = 1, 2, . . . , r − 1, denote a single spacing. (a) Show that . n n − ∑m i=1 (ki − 1) , 1 6 m 6 r − 1. Pr[X j1 > k1 , X j2 > k2 , . . . , X jm > km ] = r r (b) Find the expected value, E[X ], and the variance, V [X ], where X is any spacing random variable. (c) Determine the joint probability mass function of two such random variables Xi and X j .

3.7. JOINT DISTRIBUTIONS

129

Exercise 3.134 [2+] An m-element set A from [1..n] = {1, 2, . . . , n} is called an m-set. In this exercise, we assume that elements of A are ordered (increasingly). If A contains at least two (respectively no) consecutive integers, then A is called a consecutive (respectively, nonconsecutive) m-set. In an r/n lottery, let K(n, r) and N(n, r) denote the number of a consecutive and nonconsecutive r-sets, respectively. Show that

n−r+1 N(n, r) = r

and

n n−r+1 K(n, r) = − . r r

Definition 3.135 Let X and Y have the joint probability mass function p(x, y) with the same sample space Ω. The probability mass function of X alone, which is called the marginal probability mass function of X , is defined by pX (x) = ∑ p(x, y) = Pr[X = x], y

where the summation is taken over all possible y values for each given x. Similarly, the marginal probability mass function of Y , is defined by pY (y) = ∑ p(x, y) = Pr[Y = y].

⊳

x

A succinct way to describe the independence of random variables is the following statement. Theorem 3.136 A finite number of random variables is independent if and only if their joint probability mass function (PMF) is the product of their marginal PMFs. In particular, two random variables X and Y are independent if and only if for all x and y, pX,Y (x, y) = pX (x) · pY (y).

(3.40)

P ROOF : See in Feller’s book [41]. Definition 3.137 Let X and Y be two random variables. The conditional distribution of the random variable Y , given X = x, is f (y | x) = Pr[Y = y | X = x] =

fX,Y (x, y) Pr[Y = y and X = x] = , Pr[X = x] fX (x)

fX (x) > 0.

(3.41)

When two random variables are independent, the combination of Eqs. (3.40) and (3.41) is fX|Y (x | y) =

fX (x) fY (y) fX,Y (x, y) = = fX (x), fY (y) fY (y)

independent variables!

(3.42)

Exercise 3.138 [2] Rolling a pair of unbiased dice, let X denote the smaller and Y the larger outcome on the dice. Find its joint probability mass functions and derive the relevant conditional distribution.


130

3.8 Dependent Random Variables Sometimes we need to measure the relation between two dependent random variables. Definition 3.139 Let X and Y be random variables with means µX and µY , respectively. The covariance of X and Y , which is denoted by Cov(X ,Y ), is defined as follows Cov(X ,Y ) = E[(X − µX )(Y − µY )]. when the needed expectation exists.

(3.43) ⊳

The covariance between two random variables X and Y is a measurement of their dependence. When positive, it suggests they grow or decline at the same time (which means here: in the same parts of the sample space). If the covariance is negative it indicates a measure of their tendency to change in opposite directions. When X and Y are independent, their covariance is zero. But the converse is not true as the following example shows. Example 3.140 Let the random variable X have two values, −1 and 1, with probability 1/2 for each. Let Y = X 2 . The random variables X and Y are obviously dependent but not correlated since E[XY ] = E[X 3 ] = E[X ] = 0

and

E[X ]E[Y ] = 0.

A non-zero correlation announces dependence, but uncorrelated variables may still be dependent. It is easy to construct a pair of uncorrelated depended random variables based on the following algorithm. We choose a symmetric random variable, X , as a uniform random variable over a symmetric interval. Let f (x) be an odd and g(x) be an even real-valued function. Then the random variables Y = f (X ) and Z = g(X ) are uncorrelated, provided that Y and Z are non-generate and all the first and second moments of Y and Z exist. Exercise 3.141 [1+] Let χA be the indicator of event A and χB be the indicator of event B. Show that Cov(χA , χB ) = Pr[A ∩ B] − Pr[A] Pr[B]. To obtain a measure of dependence between random variables that is not influenced by changes of scale, we introduce the correlation, a dimensionless number. Definition 3.142 Let X and Y be two random variables with finite variances σX2 and σY2 . The correlation of X and Y is denoted by ρ (X ,Y ), and is defined as a ratio

ρ (X ,Y ) =

Cov(X ,Y ) . σX σY

(3.44)

If ρ (X ,Y ) > 0 then we say that X and Y are positively correlated, and if ρ (X ,Y ) < 0, we say that they are negatively correlated. The random variables are uncorrelated if ρ (X ,Y ) = 0. ⊳ Note that Cov(X ,Y ) and ρ (X ,Y ) have the same sign. We summarize the properties of the covariance and correlation below. Theorem 3.143 Let X and Y be random variables with finite means µX and µY , and variances σX2 and σY2 , respectively.

3.8. DEPENDENT RANDOM VARIABLES

131

C1: Cov(X ,Y ) = E[XY ] − E[X ]E[Y ].

(3.45)

C2: Let Y = aX + b for some constants a and b. Then ρ (X ,Y ) = 1 if a > 0 and ρ (X ,Y ) = −1 if a < 0. C3: For arbitrary constants a, b, and c, we have V [aX + bY + c] = a2V [X ] + b2V [Y ] + 2ab Cov(X ,Y ).

(3.46)

C4: If X1 , X2 , . . ., Xn are uncorrelated random variables, and c1 , c2 , . . ., cn are arbitrary constants, then V [c1 X1 + c2 X2 + . . . + cn Xn ] = c21V [X1 ] + c22V [X2 ] + . . . + c2nV [Xn ].

(3.47)

C5:

ρ (aX + b, cY + d) = sign(ac) × ρ (X ,Y ). C6: For arbitrary constants a, b, c, and d, we have Cov(aX1 + bX2 , cY1 + dY2 ) to be ac Cov(X1 ,Y1 ) + ad Cov(X1 ,Y2 ) + bc Cov(X2 ,Y1 ) + bd Cov(X2 ,Y2 ). Exercise 3.144 [3] Prove the six properties in Theorem 3.143. Corollary 3.145 Let X1 , X2 , . . . , Xn be n random variables, and c1 , c2 , . . ., cn are arbitrary n

constants. Then

V [c1 X1 + c2 X2 + · · · + cn Xn ] =

∑ c2k V [Xk ] + 2 ∑ ci c j Cov(Xi, X j ). i< j

k=1

Now we consider some implications of the new concepts that have been introduced. Specifically, we present the ingredients of statistical estimation. Suppose you need to determine the “true value” of some measurement—it could be a cost of computations or probability of failure of a reliable system, or something else. A typical approach in this case is to repeat the experiment over and over again under the same conditions and estimate the “true value.” Let it result in the n independent observations: X1 , X2 , . . . , Xn , called sampling (with replacement). Our principal interest is in estimating the expected value, µ ≡ E[X ], since we do not know the distribution of X . We assume that X is a random variable, with some unknown distribution; that is, every sample entry is independent of the others and derived from the same distribution.

A mathematical model for this situation is the sequence of measurements (experiments) described by X1 , X2 , . . . , Xn , which are independent random variables, where each Xi has the same distribution. We say that such a sample is independent and identically distributed (i.i.d.). Then each Xi has the same expected value, µ , representing the “true value.” The average of many observations gives usually better estimation of the “true value” than a single measurement. Thus, for n observations, the sample sum is simply Sn = ∑ni=1 Xi, and the sample mean is the average of the measured values, Xn ≡ X =

1 1 Sn = n n

n

∑ Xi.

i=1


132

Since each of the Xi has the same distribution as X , the expected value of X becomes E[ X ] =

1 n

n

∑ E[Xi] =

i=1

1 n

n

∑ E[X ] = µ .

i=1

The intuitive way to find the first moment of X , by computing the sample average, is correct; technically, it provides an unbiased estimate of µ . How good is this estimate? Clearly it is possible to get a sample of unusual values, and the sample mean will not be too close to µ . We also need to estimate in addition to µ the likely error of this estimate. Let us calculate the variance of the sample mean. Evidently, V [ X ] = E[(X − µ )2 ], is a measure of such an error. The following calculation relies on the independence of the observations Xi and on Eq. (3.47): 1 n V [X ] 1 n . (3.48) V [ X ] = 2 ∑ V [Xi ] = 2 ∑ V [X ] = n i=1 n i=1 n The result is reasonable: as V [X ] grows, it becomes more likely that some of the observed values will be far from µ . This will lead to an error in its estimator. It also tells us that as the sample size increases, the variance of X decreases, which corresponds to its PMF getting progressively tighter around its mean value. Hence, all the likely values of X get closer and closer to µ . In addition to X being an unbiased estimator of µX , its error decreases towards zero as the sample size increases. (Statisticians express this property by saying that this estimator is consistent.)

From a practical point of view, the rate of decrease of this expected error is important. We look at the square root of the variance, at σX¯ , which has the same dimension as X . This √ value decreases at a rate proportional to n, which you may find disappointing: to halve the expected error in estimating µ you need to quadruple the sample size! Let us consider an important example when X1 , X2 , . . . , Xn is a sequence of n independent and identically distributed uniform random variables defined over the interval [0, 1]. Then the probability density function of X n = n1 ∑nk=1 Xk takes the form fX n (x) =

nn (n − 1)!

n−1 j , x − ∑ n + j=0 n

0 6 x 6 1,

where u+ = 0 if u < 0, and u+ = u if u > 0. Integration yields the relation between the probability and Eulerian numbers (see Appendix C and Example 8.102 on page 487): 1 n 1 k+1 k+1 k j n+1 n < Xn < = (3.49) Pr ∑ (−1) j (k + 1 − j) = n! k . n n n! j=0 Exercise 3.146 [1] Suppose you are given two independent observations, X and Y , such that E[X ] = E[Y ] = µ , but their standard deviations, σX and σY are not equal. For a scalar α (0 6 α 6 1), consider the weighted (Seidel) average Z = α X + (1 − α )Y . For what value of scalar α the variance V [Z] attains its minimum?

Exercise 3.147 [2] Let X1 , X2 , . . . , Xn be a sequence of independent random variables having the same mean, µ and standard deviation, σ . Find the covariance and the correlation of the random variables R = ∑k k Xk and T = ∑k k2 Xk .

3.8. DEPENDENT RANDOM VARIABLES

133

Exercise 3.148 [3] For r/n lottery (see Example 3.132 and Exercise 3.133), let X j (t) = t j+1 − t j , j = 1, 2, . . . , r − 1, be the space of a ticket t = (t1 ,t2 , . . . ,tr ). Find the covariance and the correlation of random variables Xi and X j . Definition 3.137 carries over to that of conditional expectation directly: E[X | Y ] = ∑ kpX|Y (k | Y ).

(3.50)

k

Note that E[X | Y ] is a random variable and subject to the distribution of Y . If a random variable X has a finite mean then EY {EX [X | Y ]} = E[X ]. (The subscripts that state with respect to which distribution the expectation is taken were put in here for clarity. Usually this is assumed to be self-evident, and they are not used.) Instead using of the random variable E[X | Y ] it is sometimes more convenient to work with its PMF: Definition 3.149 Let X and Y be random variables and Pr[Y = y] > 0. The conditional expected value of X given Y = y, denoted E[X | Y = y], is defined by the following equation E[X | Y = y] =

∑

ω ∈{Y =y}

X (ω )

Pr[ω ] = xk Pr[X = xk | Y = y], Pr[Y = y] ∑ k

(3.51)

where the summation is over all sample points ω from the event {Y = y}. The conditional expected value exists if and only if the sums in Eq. (3.51) are absolutely convergent. ⊳

Definition 3.150 Let X and Y be random variables. The conditional variance of X given {Y = y} is defined as follows V [X | Y = y] = E (X − E[X | Y = y])2 | Y = y = E[X 2 | Y = y] − (E[X | Y = y])2 . ⊳

Example 3.151 (Continuation of Example 3.64, page 107) Consider the process of taking out socks, one at a time, from the laundry and matching each sock with the previous drawn, if there is its mate. We want to find the expected number, E(k), of pairs formed after k socks have been drawn, if there were initially 2n socks with n distinct pairs. Let Xk be the number of pairs contained in the first k socks. Let Yk+1 = 1 if the (k + 1)-st sock matches a previously drawn sock or else Yk+1 = 0. The conditional probability is Pr[Yk+1 = 1 | Xk ] = (k − 2Xk )/(2n − k). This leads to E[Xk+1 | Xk ] = E[Xk +Yk+1 | Xk ] = Xk + (k − Xk )/(2n − k), which implies E(k + 1) = E(k) + (k − 2E(k))/(2n − k). This first order recurrence, with initial condition E(1) = 0, is easily solved (use the general formula (5.13) on page 216) to obtain E(k) = k(k − 1)/2(2n − 1), as expected from Example 3.64, page 107. Exercise 3.152 [3] For random variables X and Y , prove the formulas E[X ] = E [E[X | Y ]] ,

V [X ] = E [V [X | Y ]] +V [E[X | Y ]] .

Exercise 3.153 [2−] Prove or disprove the equation EX [XY | Y ] = Y EX [X | Y ].

Exercise 3.154 [2+] Suppose we choose a graph (of n nodes) which is a 1. forest with more than one tree with probability 0.3; 2. single non-binary tree with probability 0.2;

(3.52)


134

3. binary tree with probability 0.4; 4. cyclic graph with probability 0.1. Let ti be the processing time (in msec) per node in case i:

i

1

2

3

4

ti 1.1 0.7 0.6 1.5

Compute the processing time T500 for a 500-node graph: (a) E[T500 | the input is not a cyclic]; (b) E[T500 ]. (c) What is the probability that T500 equals 350? that it equals 450?

Let us consider the expected value of the product of two random variables E[XY ] = ∑x ∑y xy pXY (x, y). If the two variables are independent, then from Eq. (3.42) we see that pX|Y (x | y) = pX (x), and the joint PMF ‘splits’ to a product of the two PMFs, pX,Y (x, y) = pX (x)pY (y). As a result we find that E[XY ] = ∑ ∑ xy pXY (x, y) = ∑ ∑ xy pX|Y (x | y) pY (y) x

y

x

independent

y

= ∑ ∑ xy pX (x)pY (y) = ∑ x pX (x) ∑ y pY (y) = µX µY . variables! x

y

x

y

Inserting this in the last value for the variance we find that for independent random variables V [X +Y ] = V [X ] +V [Y ], as stated in Theorem 3.96. Note: Unlike the independence condition, this is not a necessary condition. It is not difficult to design PMFs such that the last equality holds for dependent random variables, just as we saw that dependent random variables can be uncorrelated. Exercise 3.155 [3−] Prove that for all random variables the correlation coefficient, if it exists, is bounded in the interval [−1, 1]. What would it tell us about the random variables if the correlation is any of the extreme values? There are standard ways of generating new variables out of given random variables. We look at one of the operations, called convolution, in §4.5. The distribution of the maximum and minimum values of a random sample are discussed in section 4.6. Various other ways to form new variables from previously defined ones are possible. The following exercises give some examples. Exercise 3.156 [2−] Let X and Y be discrete random variables with joint probability mass function f (x, y) = (x + 1)(y + 2)/54 for x, y = 0, 1, 2, and zero otherwise. (a) Find the expected value of g(X ,Y ) = X 2Y . (b) Find E[X ] and E[Y ]. Exercise 3.157 [2] A balanced die is rolled three times and produces the three values Xi (i = 1, 2, 3). Find Pr[X1 < X2 < X3 ] (note: strict inequalities). Exercise 3.158 [3] A fair coin is tossed three times. Let X be the number of heads in the first two throws and let Y equal the number of heads in the last two throws. Show the following results: (a) E[X ] = E[Y ] = 1. (b) Cov(X ,Y ) = 1/4. (c) E[1/(1 +Y )] = 7/12. (d) E[X /(1 +Y )] = 1/2.

Chapter 4

More about Probability In this chapter, we continue introducing the theory of probability needed in the analysis of algorithms. Our presentation is not exactly the theorem-proof style. However, all statements are proven in the text (some of them are left as exercises) to involve the reader in precise mathematical arguments. The emphasis is on establishing the dialog between frequent theorems and their applications through plentiful examples and exercises. We first survey the most common distributions of discrete random variables and one continuous distribution—the normal distribution, as they are needed in analysis of algorithms. The limiting behavior of a sequence of random variables requires several types of convergence— four of them are introduced in the second section. The Central Limit Theorem relates discrete and continuous distributions with the normal distribution and it lays down the ground for asymptotic behavior, which is the topic of §10.7. Then we present the Theorem of Total Probability—which is the basis of randomization. It is customary to use the symbol ∼ to indicate the particular distribution of a random variable (e.g., to say that X is normally distributed we write X ∼ N). Working directly with discrete distributions is a subtle way of developing intuition of probability theory. Many examples and exercises in the text that involve discrete random variables will sharpen a reader’s capabilities for mathematical reasoning. Two powerful techniques, Bayes’ theorem and the convolution of random variables, attract our attention in this chapter. Order statistics, which is intriguing and very useful in many algorithms, cannot be avoided in the text. Finally we present bounds on probabilities, in particular tail-bounds—the Chebyshev and Markov inequalities. The chapter is concluded with sundry examples.

4.1 Special Distributions This section presents examples of four famous discrete random variables as well as the continuous one of great importance—the normal distribution. Plenty others could be found in [138]. 135

CHAPTER 4. MORE ABOUT PROBABILITY

136

4.1.1 Bernoulli Variables and the Binomial Distribution The Bernoulli random variable, named after Jakob Bernoulli (1654 – 1705), who described it in his book “Ars Conjectandi” (The Art of Guessing) in 1713, enjoys being the simplest possible (non-degenerate1 ) random variable. It is also one of the most useful random variable for modeling needs. It is the natural random variable for a probability space where the experiment—traditionally called a Bernoulli trial—has only two possible outcomes. They may be labeled success or failure. For instance, the testing of gates when they checked in a completed chip may indicate a defective or a functioning gate. The Bernoulli random variable is assigned the values 0 and 1 for the two possible outcomes. To fix notation, let the variable be called B, and the probability of success, assigned the value 1, be p. Then B has the following probability mass function (PMF):

pB (i) = Pr[B = i] =

(

q ≡ 1 − p, if i = 0, p, if i = 1.

(4.1)

Its mean and variance are easy to calculate: E[B] = p

and V [B] = E[B2 ] − (E[B])2 = p − p2 = p(1 − p) = pq.

There is not much more we need to say at this stage about this random variable and its distribution, except that they can be compounded in several ways to create other, more complex, and very useful distributions.

Binomial Distribution The binomial probability distribution is one of the best known discrete distributions in probability theory. Its source is the binomial experiment, which consists of a sequence of Bernoulli trials that satisfies the following conditions. 1. There is some fixed number, denoted by n, of identical Bernoulli trials. 2. The Bernoulli parameter p remains constant during trials. 3. All experiments are independent; that is, the outcome of one trial does not affect the outcome of any other trial. The number of successes in such an n-long sequence is called a binomial random variable and denoted by B(n, p). The probability distribution of this discrete random variable is called the binomial probability distribution and its values are given by2 n Pr[B(n, p) = j] ≡ b( j; n, p) = p j qn− j , q = 1 − p, (4.2) j where 1A

n

the total number of trials,

random variable is said to be a degenerate variable if it has only one possible value in probability 1. expression in the right-hand side traditionally is called the Bernstein polynomial and denoted by B j,n (t) = nj t j (1 − t)n− j . 2 The

4.1. SPECIAL DISTRIBUTIONS

137

the number of successes in n trials,

j n− j

the number of failures in n trials,

p

probability of success,

q = 1− p

probability of failure,

n j

=

n! j! (n− j)!

a binomial coefficient as introduced in §2.4.

The number of trials, n, and the probability of success, p, are called the binomial parameters. Example 4.1 One percent of all VCRs manufactured by Acme are defective. You and your friend bought two such VCRs. What is the probability that exactly one of them is defective?

Solution. The total number of trials is n = 2 and the number of successes (here, curiously, the number of failing VCRs) we ask about is j = 1. So n − j = 1 as well. The Bernoulli parameter here is p = 0.01, and thus q = 1 − p = 0.99. According to Eq. (4.2), the desired probability is given by 2 b( j; 2, 0.01) = 0.99 × 0.01 = 2 × 0.99 × 0.01 = 0.0198. 1 Exercise 4.2 [1] If a fair coin is tossed 6 times, what is the probability of getting 4 heads or more? Let us find the expected value of the binomial distribution. Substituting the probability (4.2) into Eq. (3.12) on page 100, we get the expected value to be n n n p j (1 − p)n− j . E[B] = ∑ j · b( j; n, p) = ∑ j j j=0 j=0 This is identical to the calculations we did in deriving Eq. (2.61) on page 59; hence we find that the expected value of the binomial distribution is E[B(n, p)] ≡ µ = np.

(4.3)

Its variance is σB2 ≡ V [B(n, p)] = E[B2 (n, p)] − (E[B(n, p)])2 = npq because n j n− j d 2 2 2 n j n− j pq E[B (n, p)] = ∑ j pq = p ∑ dp j j j j d 2 (p + q)n = p[n(p + q)n−1 + pn(n − 1)(p + q)n−2 ] = p dp = p[n + pn(n − 1)] = np(1 − p) + p2 n2 = npq + p2 n2 ,

p + q = 1.

Exercise 4.3 [4] Suppose B(n, p) is binomially distributed with parameters n and p. (a) Find the skewness and kurtosis coefficients of B(n, p). Is B symmetric? (This is common shorthand for saying that the PMF of B is a symmetric function about its mean, which then equals its median.) (b) Show that B(n, p) is symmetrically distributed about c if and only if p = 1/2 and c = n/2 = E[B]. (c) Determine its n-th moment and n-th factorial moment, n = 2, 3, . . ..


138

Example 4.4 You are flipping a true coin. What is the probability you get heads half of the time? To get one out of two, we compute 21 21 21 = 21 , and two out of four we get in 2 1 2 3 1 = 8 < 2 , which may, at first sight, appear surprising since there are probability 42 12 2 more “paths” or ways to get two out of four than 1 out of 2. In general, let Sn be the number of heads in the n tosses of a true coin. The probability to see heads half of the time in 2n tosses is 2n (2n)!2−2n number of favorable outcomes = n2n = . Pr[S2n = n] = total number of outcomes 2 n!n! From the Stirling approximation formula (see Eq. (2.92) on page 78), it follows that 1 Pr[S2n = n] ∼ √ nπ

as n → ∞.

Therefore, the probability of getting equal numbers of tails and heads in 2n tosses approaches zero as the number of flips grows. Its expected value is the sum 2n 2n −2n 2n − 1 −2n E[S2n ] = ∑ k Pr[S2n = k] = ∑ k 2 4=2 ∑ 2n k − 1 = 2−2n 2n × 22n−1 = n. k k=0 k=0 k=0 2n

2n

To show that Pr[S2n = n] > Pr[S2n = n ± k], 1 < k 6 n, we calculate the ratio Pr[S2n = n]/ Pr[S2n = n ± k] =

2n n 2n n±k

=

(n + 1)k > 1. nk

A particularly informative way of viewing the distribution is to define the difference, Wn , of the number of heads and the number of tails in n flips. Then Wn = Sn − (n − Sn ) = 2Sn − n shows the progress of the differences throughout the tosses. It can be interpreted as a position of a random walker on the line at step n going right by one unit if flipping the coin leads to heads, and going opposite direction with tails (see Example 4.127 on page 178). Since Sn has the binomial distribution, the random variable Wn has the following mean and variance: 1 n−n = 0 E[Wn ] = 2 2

and

σW2 n

2 1 n = n. = V [Wn ] = 4 2

Suppose we want to find the probability that in 2n tosses the number of heads does not exceed the number of tails by 2, that is, Pr[|W2n | 6 2]. This event happens whenever S2n ∈ {n − 2, n − 1, n, n + 1, n + 2}. Hence 2

Pr[|W2n | 6 2] =

∑ k=−2

Pr[S2n = n − k] =

1 4n

2

∑ k=−2

2n n−k

n(n − 1) n 1 2n +2 1+2 = n 4 n+1 (n + 1)(n + 2) n 2 5 1 2n 5n + 5n + 2 ∼√ as n → ∞. = n 4 n n2 + 3n + 2 nπ


139

Exercise 4.5 [2+] Consider the following two random variables: X is the number of times of getting a 4 out of four rolls of a fair die; Y is the number of times of getting two 4’s out of 24 rolls of two fair dice. Show that E[X ] = E[Y ]. Exercise 4.6 [3] For a sequence of independent Bernoulli trials, which outcome is likelier: m successes in n trials, or 2m successes in 2n trials? Plot the ratio of these probabilities as a function of p = m/n. Exercise 4.7 [2] Two dice are thrown n times. Show that the count of throws in which the number on the first die exceeds the number on the second die is binomially distributed with parameters n and p = 5/12. Exercise 4.8 [2] A manufacturer guarantees a refund or a replacement for any disk that malfunctions within 2 years from the date of purchase. It is known from past data that 3% of the disks manufactured by the company malfunction within a two-year period. A store sold 12 (randomly selected) disks one day. Find the probability that at most 2 of these 12 disks malfunction. Exercise 4.9 [3] Frequent flyers know that flying is a gamble. Nonetheless, they would still like to limit the risk. Let us adopt the unrealistic scenario that all flights are subject to an accident rate with the same probability, which the industry estimates to be at 4 × 10−5 . Let N be an annual number of flights that a frequent flyer may have. To limit N, what is its upper bound in order to produce a probability of at least 99.9% of having an accident-free year? Exercise 4.10 [2] Appending three extra bits to a 4-bit word in a particular way (to create a Hamming code) allows detection and correction of up to one error in any of the bits. If each bit has probability p of being changed during communication, and the bits are changed independently of each other, what is the probability that the word is correctly received? How does this probability (when, for example, p = 0.05) compare to the probability that the word with no check bits is transmitted correctly? Exercise 4.11 [3] We consider the benefit of using an error-correcting code that requires adding c bits to each transmission word of w bits, and can correct one bit error. Representative values for c and w are 12 and 128. Let the probability that a bit is corrupted be denoted by p, so q = 1 − p. We assume this probability applies to all transmitted bits, code, and content alike, independently. For which values of p is this a worthwhile scheme? Note: The scheme is worthwhile if the probability that we get at most one error in w + c bits is larger than the probability of no errors in w bits. Exercise 4.12 [2] Let B(n, p) be a binomial random variable based on n trials and success probability p. Show that

Pr[B > 1 | B > 1] =

1 − qn − npqn−1 , 1 − qn

q = 1 − p.

Exercise 4.13 [3+] (L. Räde) Suppose we have n identical coins for each of which heads occurs with probability p. Suppose we first toss all the coins, then toss those that show tails after the first toss, then toss those that show tails after second toss, and so on until all the coins show heads. Let X be the number of coins involved in the last toss. Find Pr[X = k] for k = 1, 2, . . . , n and E[X ].


140

4.1.2 The Multinomial Distribution The multinomial distribution generalizes the binomial one by looking at independent trials that have a number of possible outcomes that is larger than two and counting those outcomes. It is naturally viewed as a joint distribution of several discrete random variables. We saw that B(n, p) arises as the number of successes in a sequence of n Bernoulli trials, B(p). When each trial has more than two possible outcomes and the probabilities of the respective outcomes are the same for each trial, we obtain the multinomial distribution. Consider n independent trials, each having m mutually exclusive outcomes A1 , A2 , . . ., Am with probabilities p1 , p2 , . . ., pm , respectively. Clearly we must have p1 + p2 + . . . + pm = 1. It is then natural to define a vector random variable (X1 , X2 , . . . , Xm ), where Xi = number of times Ai occurs,

i = 1, 2, . . . , m.

Note that these Xi are not independent since ∑m i=1 Xi = n. The marginal random variable Xi , i ∈ [1..m], has the binomial distribution B(n, pi ); think of the outcome Ai as a success and any of the others as a failure. The joint distribution of the Xi ’s in this situation is what is called the multinomial distribution. Definition 4.14 The random variables X1 , X2 , . . ., Xn have a multinomial distribution if and only if their joint probability mass distribution is given by n M(x, p, n) = Pr[Xi = xi , i ∈ [1..m]] = px1 px2 · · · pxmm , x1 , x2 , . . . , xm 1 2 def

m

m

i=1

i=1

where M(x, p, n) = M(x1 , . . . , xm ; p1 , . . . pm ; n) with xi ∈ [0..n], ∑ xi = n, and ∑ pi = 1.

⊳

Note that, as in the case of the binomial distribution, we lump all possible orders in which these outcome counts occurred. Example 4.15 If we toss three fair coins, there are four possible outcomes: no tails, with probability 1/8; one tail, with probability 3/8; two tails, with probability 3/8; three tails, with probability 1/8. Now suppose that we conduct this experiment 16 times, what is the probability that we get three tails 5 times, one tail 2 times, and two tails 5 times? Using the multinomial distribution the answer is: 5 2 5 4 1 3 3 1 16! ≈ 0.00023519. 5! 2! 5! 4! 8 8 8 8 Exercise 4.16 [2] Suppose that (X1 , X2 , X3 , X4 ) is multinomially distributed with parameters n, p1 , p2 , p3 , p4 . What is the distribution of (X1 + X2 , X3 , X3 + X4 )? Exercise 4.17 [3] Suppose (X1 , X2 , . . . , Xm ) have the multinomial distribution with parameters n, p1 , . . ., pm . Show that the covariance of Xi and X j (i 6= j) is Cov(Xi , X j ) = −npi p j . What is their correlation? Hint: Use the indicator random variable. Exercise 4.18 [2+] A die is loaded so that pi = Pr[ face i appears ] = ki, i = 1, 2, . . . , 6. It is rolled 24 times. What is the probability that each face will appear exactly four times? In the next two subsections, we describe two additional types of discrete distributions related to the Bernoulli distribution. When the number of experiments, n, in the Bernoulli formula


141

(4.2), is big, calculations of probabilities become problematic. In this case, the binomial distribution (4.2) is usually approximated by another one (see §4.1.7), which is more convenient for calculations.

4.1.3 The Geometric Distribution A geometric distribution is the simplest waiting-time distribution for Bernoulli trials, which were denoted by B(p) in §4.1.1. Such a Bernoulli process generates 1 (or S, success) with probability p, and a 0 (or F, failure) with probability q = 1 − p. Successive trials are assumed to be independent. Let X be a random variable, defined as the number of trials we observe until the first “1” occurs. What values can X take? Clearly, any positive integer, k > 1. What is the probability that X takes any particular value? X = 1 means the first trial produced a 1 (or S). This will occur with probability p; X = 2 is the event that corresponds to the sequence 01, and this will happen with probability qp. Similarly, X = 3 means the trials produced 001, with probability q2 p; .. . X = k means the first 1 appeared on the kth trial, and this will occur with probability qk−1 p. We denote this distribution by G(p) and say that it is geometrically distributed. The corresponding probability mass function then has the entries pk = Pr[G(p) = k] = qk−1 p, k > 1

G(p) :

(q = 1 − p).

(4.4)

Similar scenarios turn out to be common. For instance, the probability that there are k failures between two successive successes is qk p. The reason for the name of the distribution is evident—these probability values have constant ratio, just as the terms of a geometric series. Expected value and variance of a geometric distribution can be found either directly (see Exercise 4.20 below) or using generating functions (see Example 6.156 on page 335). Example 4.19 A reliable manufacturing machine produces parts at a rate of 1000 per hour. The probability of a defective part is fairly small, 10−4 . What is the probability that the machine operates an 8-hour shift without a malfunction? Solution: We assume that producing each part is a Bernoulli experiment B(10−4 ) and that successive parts are independently likely to be good or defective. In 8 hours, the machine manufactures 8000 parts. If X is the number of parts manufactured till the first failure, we are asked to calculate the probability Pr[X > 8000]. In general, if X ∼ G(p), then Pr[X > d] =

∑ q j−1 p = p ∑ q j = p j>d

j>d

qd = qd . 1−q

(4.5)

In terms of the definition above p = 10−4 , so the answer is Pr[X > 8000] = 0.99998000 ≈ 0.4493. Hence even this reliable machine will malfunction on most shifts! Exercise 4.20 [2+] Let a random variable X have the geometric distribution G(p). (a) Show that the pk = Pr[X = k], k > 1, sum to 1, that is, ∑k pk = 1. (b) For n = 1, 2, . . ., prove the following formulas:

µ = E[X ] =

1 , p

σ 2 = E[X 2 ] − (E[X ])2 =

q , p2

µ[n] = E[X n] = n!

qn−1 . pn

(4.6)

142


(c) For p = 0.2, also find the median and the mode of X . Example 4.21 (Hashing) Hashing or scatter storage technique is the most widely used family of algorithms that is designed for information storage, retrieval, and insertion. An efficient method to store entries in a table so they can be accessed with low overhead is to build it as a hash table. To create such a table, we map the keys to the table positions (also called slots or bins). The function that maps key values to bin indices is called a hash function. A good hash function3 assigns indices equiprobably. When presented with a key, the hash function can calculate the slot that contains the record, which allows us to retrieve just that slot, instead of sequentially searching through all bins in the structure. In the typical case, a hash function maps a key from a large set to an index in a table with a relatively small number of slots. For example, we may have 10,000 (alphabetical) 8-letter names—which are from a set of size 268 ≈ 2.088 × 1011 —and map them to a table of 11,000 positions. The hash function may return the same address for several keys; when this happens, we say that the keys collide at the computed address. As Example 3.19 asserts, the insertion of 40 keys randomly into a table of size 365 produces a collision with probability 0.8912. One of the approaches to resolve a collision is to use second hashing. This scheme uses a sequence of independent hash functions f1 , f2 , . . . , each of which attempts to spread the keys uniformly across the table sampling with replacement. If a record insertion requires the services of j hash functions ( j = 1, 2, . . .) to successfully find space for a new record, we say that the insertion requires j probes. Suppose that the fraction α < 1 (also called the load factor) of the table is filled when we try to store a new record. Then α m addresses are occupied and the probability (of success) of finding an empty basket is 1 − α = (m − α m)/m. Normally, we don’t want the load factor α to be far from 1 in order to achieve efficient use of space. So we have a sequence of Bernoulli random variables, B j (p), p = 1 − α , associated with each jth hash function to find an empty slot, that is, B j = 1 (success) if f j finds an empty basket to insert the new record, j = 1, 2, . . .. Let X = min{ j | B j = 1} be the number of probes required for an insertion of a new record. Then X is a geometrically distributed random variable with parameter 1 − α . Suppose we have n keys that we want to hash into m slots. One can ask various questions about the processes that this activity creates, and we shall ask here two. 1. For the first question, we assume that n is rather small, and we would like to choose m in such a way that the probability of having any collision would be less than 0.5. How large should m be? This question has the same mathematical structure as the birthday problem, presented in Example 3.19 on page 90. The number of ways to map n keys to m positions is mn . The number of ways to place each of the n keys in its own place with no collision is the number of ways to select n places out of m with no repetition, which is the number of mpermutations from [1..n]: Pnm = mn = m!/(n − m)!. Hence the probability that n keys do not collide in a table with m bins is mn /mn , and when n is small compared to m, the ratio is very n(n−1)/2 (see Example 10.19 on page 554 well approximated, to first order in m1 , by 1 − m1 for details). We want to equate this to 1/2 and solve the equation for m in terms of n. There 3 From

theoretical point of view, it is impossible to build random data from a non-random data, however, there are known many practical ways to produce random looking data.


143

exists an asymptotic way of solving such a nonlinear equation. Recall that 1 um 1 r −1 =e =⇒ 1− ≈ e−u . lim 1 − r→∞ r m This is 1/2 when u = ln 2. We shall equate um with n(n − 1)/2 ≈ n2 /2, and find that the table must be of size m ≈ n2 /(2 ln 2) at least. For example, this says we can accommodate 22 keys in a table of size 365 (the calculation gives 22.494...); quite close to the result of Example 3.19 on page 90. 2. Normally we cannot afford such large tables—storing a 1,000 keys would require a table of more than 721,000 positions! Instead, one finds an efficient way to deal with the results of collisions. Let us look at a particular approach, called for some reason open addressing or closed hashing, which requires us to resample a position once we have a collision, until a new free location is found. Now we ask: how large does m need to be, for a given n, so that the average number of mapping attempts when the next key is inserted is no more than 2. Note that now we ask about collisions of one key, not the entire collection. Earlier keys had a smaller likelihood of collision, and we are asking about the cost at the end of the insertion process. Once there are n keys in m locations, the probability that a random mapping produces a collision is n/m; this ratio is called the loading factor of the table, and is denoted by α . Each insertion attempt is a Bernoulli experiment, with success probability 1− α , and the number of collisions until a free place is found is a geometric random variable, G(p), with p = 1 − n/m. The expected value of G(p) is 1/p; hence we have the equation 2 = 1/(1 − n/m), which produces m = 2n. If you fill more than half the table, you should be willing to encounter longer insertion sequences, on average.

Exercise 4.22 [2] Imagine you are using a hashing function to store n = 105 records into a table. You must use a hashing approach since the keys are large and hard to handle. You are very worried about collisions as this is a critical application. You allocate a table of ten times the number of entries and let table size be a million. During test runs you count the number of collisions and are mortified to see that your profligate use of space did not help much: there were 4628 collisions! Should you consider looking for a better hash function? Example 4.23 There are many interesting practical situations involving sampling with replacement. One of them was discussed in Example 3.67 on page 108 when a group of n people was asked about their birthdays. A similar situation occurs in verification of a random number generating computer program. For instance, a generation of n-digit random number is actually a sampling of n items from the population [0..9]. The number of distinct elements in a sample of size n chosen with replacement from a population of size m is a random variable, which we denote by X (n, m). Unfortunately, direct derivation of its probability distribution is frustratingly complicated, but it becomes relatively easy when using appropriate tools (see §7.6.4 and §9.4). However, its mean and variance can be found as follows with less efforts. Let X = m − Y , where Y = Y (n, m) is the number of elements never chosen during n trials. The random variable Y can be considered as a sum, Y = ∑m i=1 Bi , where Bi is a Bernoulli (or indicator) random variable that is 1 only if the i-th object in the population of n items is never sampled. Then p = Pr[Bi = 1] = (1 − 1/m)n and q = Pr[Bi = 0] = 1 − p. It follows


144

immediately that m

E[X (n, m)] = E[m −Y ] = m − ∑ E[Bi] = m − mp = mq. i=1

From Theorem 3.96 on page 116, it follows m

V [X (n, m)] = V [n −Y ] = V [Y ] = ∑ V [Bi ] + ∑ Cov(Bi , B j ). i=1

i6= j

The variance of Bi is known to be V [Bi ] = pq. The indicator random variables Bi and B j (i 6= j) are not independent: if the i-th item is never chosen, it increases the chance of j-th object to be sampled. We calculate the covariance to get Cov(Bi , B j ) = E[(Bi − pq)(B j − pq)] = E[Bi B j ] − p2

(i 6= j).

Since E[Bi B j ] = Pr[Bi B j ] = Pr[ neither the i-th nor the j-th object is sampled ] = n 2n − m−1 < 0. Hence find Cov(Bi , B j ) = m−2 m m

m−2 n , m

we

m−2 n 2 −p V [X (n, m)] = mpq − m(m − 1) m (m − 1)n (m − 1)(m − 2)n (m − 1)2n + − . = mn−1 mn−1 m2n−2

Definition 4.24 A shifted geometric distribution is a distribution where there is a fixed ratio, q, between successive pk , as in Eq. (4.4), but the smallest k for which pk > 0 is not 1. We could call the distribution in Eq. (4.4) the natural geometric distribution. Let Gr (p) have the probabilities pk = pqk−r , for k > r, where r > 0 is the smallest possible value. The notation G0 (p) is particularly common when r = 0. ⊳ Note that G0 (p) is the distribution of the number of failures before the first success in a Bernoulli process with success probability p. It is also the distribution of the number of failures before the next success after any given trial, and it is the distribution of the number of failures between any success and next success. Exercise 4.25 [2] For a shifted geometric distribution, show that the sum of all probabilities, pk , must be equal to 1. Explain why the mean and variance of this (shifted) random variable are available from Eq. (4.6) with no additional calculations. Exercise 4.26 [3] An urn contains one white ball and one black ball and the game consists of choosing a ball at random. If it is white, the game is over; if it is black, then the ball and another black one are put back into the urn. After that a new drawing at random is made and the same rules for ending or continuing the game are followed. This game continues until a white ball is selected. (a) Find the expected number of drawings necessary to get a white ball. Hint: Show first that the probability that exactly k drawings are needed is 1/[k(k + 1)]. (b) Repeat the calculation for the case where, following the drawing of a black ball, we put back in the urn the black ball and a white ball. Also when we put back r black balls and s white ones.


145

Exercise 4.27 [2−] Let a “toss” consists of flipping n fair coins together. What is the distribution of the number of tosses required until all the coins show tails at the same time? Exercise 4.28 [3] The geometric random variable is memoryless. (a) Let X be geometrically distributed with parameter p (and q = 1 − p). Show, that for every positive integer n, Pr[X = n + k | X > n] = qk−1 p = Pr[X = k],

k = 1, 2, . . . .

(b) Now argue in reverse. Let a random variable X be defined on N, and Pr[X = 1] > 0. For every positive integer n, given that the random variable has the following property: Pr[X = n + k | X > n] = Pr[X = k],

k = 1, 2, . . . ,

show that X must be geometrically distributed with parameter p = Pr[X = 1]. Hint: Pr[X = 2 | X > 1] = Pr[X = 2]/ Pr[X > 1] = Pr[X = 1], hence Pr[X = 2] = pq.

Exercise 4.29 [3] Computer simulation frequently needs a generation of samples with specified distributions. Over the years, several quite reliable pseudo-random number generators have been invented, that produce numbers satisfying strong statistical tests of independence and uniform distribution over the interval [0, 1]. Other distributions are generated by operations over the output of such generators. We denote by X below the output of these generators, and assume that X has a true uniform distribution over the interval [0, 1]. (a) Let Y be defined as 1 if X < p and 0 otherwise for some p from the open interval (0, 1). Show that Y is a Bernoulli random variable, B(p). (b) A text in probability contains the claim that the way to generate a geometrically distributed variable over N with parameter p = 1/2 is to calculate Y = h(X ), where h(·) is the following piecewise constant function   0, if 0 6 x < 1/2,    1, if 1/2 6 x < 1/22 , h(x) = 2, if 1/22 6 x < 1/23 ,    . . . . . . . . . .

Now this is clearly wrong. Why? In addition, you have the following tasks. (1) Fix it; find a somewhat similar function g(·) such that Y = g(X ) will indeed have a geometric distribution over N with parameter 1/2. (2) Find yet another (similar) function f (·) such that Z = f (X ) is geometrically distributed over N with parameter q ∈ (0, 1). Hint: Use the memorylessness of the geometric distribution, shown in Exercise 4.28.

Exercise 4.30 [2] (Continuation of part (b) of Exercise 4.29) Explain why your function g(·) is not likely to do a good job if you are interested in values of Y that are in the “remote tail values,” say, on the order of 100. That is, if you need to calculate g(x) when x < 2−k for large values of k (which results in simulation of geometrically distributed random variable for the cases when success occurs on kth trial). Although the probabilities of rare events are not frequently required, care must be exercised in generating such values. How would you improve your function g(·) to handle such needs? Exercise 4.31 [2+] The name of the following sorting procedure is self explanatory: Possibly Slowest Sorting Method 1. void pssm(int n, int ∗v)

146

2. 3. 4. 5. 6. 7.

CHAPTER 4. MORE ABOUT PROBABILITY {void perm(int, int *); int k, done = 0; while( done == 0) {perm(n, v); for(k = 1; k < n; k ++) if (v[k] < v[k − 1]) break; if (k == n) done = 1; }}

The procedure perm produces a random permutation of its input, in place. According to rumors, the reason this procedure is the only sorting method used in the products of the local Acme company is that their CEO wrote about it in his senior thesis, where he was asked to prove a sorting method of his choice. (a) Let the random variable Xn be equal the number of calls to perm when pssm is entered with an array of size n, containing distinct values. What can you say about its distribution? (b) Let Cn be the random variable counting the number of comparisons between elements of v[ ] that are made in line 5 of the code, until pssm terminates. What are its minimal, maximal and average values? Exercise 4.32 [2+] (Continuation of Exercise 4.31) The following sorting procedure is a vast improvement on the pssm() method, even if it is not a desirable one yet. The processes it generates are suitable for analysis here. The function perm() performs as in the previous exercise. 1. 2. 3. 4. 5. 6. 7.

void psort(int n, int ∗a) { int init = 0, j; while (init < n − 1) { perm(n − init, &a[init]); for ( j = init + 1; j < n; j ++) if (a[init] > a[ j]) break; if ( j >= n) init ++; }}

(a) How many times is the function perm() called? Minimum? Maximum? Average? What are the probabilities of the extreme cases? (b) What is the expected number of comparisons made by this procedure, between elements of the array a, in line 5? Exercise 4.33 [3] When data is read from a magnetic medium, there is a probability of error due to a variety of physical causes. The device driver detects an error that occurs and repeats the read operation until it is successful. However, there is an upper limit K on the number of trials before the driver gives up and declares the data corrupt. The value of K is determined so that, if the error is transient, there would be high likelihood that a correct read of the data will be achieved. With old deteriorating tapes and disks, the probability of error is not negligible, and then K may have to be larger. However, we do not want to make K too large since this will tie up the system trying too many times to read data that is hopelessly corrupt. Suppose the usual read command, for a sector, needs the successive reading of r blocks, and each is checked separately for correctness of data. We may assume these reads are a sequence of independent Bernoulli trials, and denote the probability of an error in reading each block by q. Let the limit K be given in terms of the number of times the driver attempts to read individual blocks, and we set K large enough so that the entire sector will be read correctly in probability of at least Q. How large do we make K to achieve this goal? Use a


147

numerical search to compute suitable values for K for the following possible values of the parameters (do it for all 27 combinations) Q = (98%, 99.5%, 99.99%); r = (4, 10, 48); q = (10−4 , 0.01, 0.1). Example 4.34 In a tennis competition, suppose that a player A wins a point on serve against a player B with probability p (and loses with probability q = 1 − p). The winner of a game is the first person to score 4 points, unless the game reaches a 3-to-3 tie. Then it continues until someone goes ahead by 2 points. (The four points are called 15, 30, 40, and game.) In tennis, the scores after a 3-to-3 tie are frequently referred to as deuce. 4 k−4 The probability that player A wins the game against B serving first is k−1 , k = 4, 5, 6. k−4 p q If neither player has won by the 6th point, then the winner isdetermined by a player who first advanced two points. The probability of a 3-to-3 tie is 63 p3 q3 = 20p3 q3 . Next deuce may occur only if a player gets a loss and a win in two sequential serves with probability 2 pq = 2pq. In general, the probability of winning the (2n + 6)th point is the probability of 1 a 3-to-3 tie, 20p3 q3 , splitting each pair of the next 2n − 2 points, (2pq)n−1 , and then winning the last 2 points, p2 . So we get the geometrically distributed random variable. The expected number of serves, N, that player A will make is 4 5 6 3 3 2 4 4 4 4 4 2 2 4 N = 4(p + q ) + 5 [p q + pq ] + 6 [p q + p q ] + 8 p q (p + q2 ) + · · · 1 2 3 = 4(p4 + q4 ) + 20pq(p3 + q3 ) + 60p2 q2 (p2 + q2 ) + 20p3 q3 (p2 + q2 ) ∑ (8 + 2k) (2pq)k k>0

= 4(p4 + q4 ) + 20pq(p3 + q3 ) + 60p2 q2 (p2 + q2 ) + 20p3 q3 (p2 + q2 ) If p = q = 1/2, the expected length of a game is 9 78 serves.

8 − 12pq . (1 − 2pq)2

This tennis problem has a natural generalization when two players (or teams) play against each other until one of them first wins n games. It is assumed that the probability to win remains constant for each of the players and that there are no draws. The expected length of the games can be calculated similarly to the case considered in the example; its explicit formula is known to contain the Catalan numbers [123]. Exercise 4.35 [2] Suppose that a tennis player A wins a point on serve against a player B with probability p (and loses with probability q = 1 − p). What is the expected number of additional serves in a game of tennis that has reached deuce? Exercise 4.36 [2] In volleyball, as opposed to tennis, the winner of any point during a game gets to serve for the next point. Volleyball has deuce games and the winner is a team winning two sequential points first. Let A and B be two volleyball teams, and suppose they win their serve with probability p and r, respectively. Let M be the number of additional serves from deuce in a volleyball game when A begins serving. Find the expected value of M. Example 4.37 Suppose that you need an unbiased bit generator, producing 0 and 1 in probability 1/2, but at your disposal there is a procedure B IASED -R ANDOM, that outputs either 0 with some probability p and 1 with probability q = 1 − p, where 0 < p < 1, but you do not know what p is. (Another physical interpretation: you are given a biased coin that comes up heads with some probability possibly different from one-half and you need to use it to simulate an unbiased coin toss.) This problem has many applications when more secure randomness is needed and regular


148

pseudo-random generator is not sufficient. Software could obtain randomness from the computer system, including such behaviors as disk movement, user keystrokes, mouse clicks, and others. While these physical phenomena can produce random events, the task is to mimic uniform distribution of bits. A simple solution, attributed to J. von Neumann, makes use of symmetry: let the procedure x = f lip() produce such a possibly-biased heads or tails; if x = f lip() and y = f lip(), then Pr[x = heads ∧ y = tails] = Pr[x = tails ∧ y = heads]. This is translated into algorithm: repeat forever x ← f lip() y ← f lip() if x 6= y then return x

The probability of “succeeding” (returning heads or tails) is 2p(1 − p), and the probability of “failing” (getting 2 heads or 2 tails, and having to continue) is p2 + (1 − p)2 . To simplify the notation, let p∗ = 2p(1 − p) denote the probability that a pair of trials suffices, and q∗ = p2 + (1 − p)2 denote the probability that a pair of trials doesn’t suffice and we must continue. Hence, this is a geometric distribution and the expected number of trials is 1/p∗ = 1/2pq. Note that a trial consists of 2 flips, so the expected number of flips to see tails and heads (with equal probability) is 2/2pq = 1/pq. For example, when p = 2/3, it requires on average 9/2 flips. Exercise 4.38 [2] Suppose a biased coin comes up heads with probability p = 2/3. Can you generate a bit more efficiently than by the von Neumann method? Exercise 4.39 [3] Show that the rth moment about the origin of the shifted geometric random variable with parameter (1 − 1/n) can be written in terms of lower-rank moments, using the notation µr′ = (1 − 1/n)Sn (r), where ∞

Sn (r) =

r −k

∑k n k=1

n r r = ∑ k (−1)k−1 Sn (r − k). n − 1 k=1

4.1.4 The Negative-Binomial Distribution The negative-binomial distribution is a direct generalization of the geometric distribution. AS opposed to the binomial random variable, which counts the number of successes obtained in n trials, the negative-binomial random variable counts the number of trials required to achieve r successes (for the first time) in a series of independent trials, where each experiment has a probability of p of ending in success. Let X be the number of trials required to see r successes. Then the distribution of X is called the negative-binomial distribution with parameters r and p, or the G(r, p) distribution for short. It derives its name from the form of its generating function, which we discuss in §6.6. Let X have this distribution that we abbreviate as X ∼ G(r, p). What is the value of Pr[X = j]?

It is not hard to compute: X is j when we have a j-long sequence of independent Bernoulli trials, ending with a success; it is the rth one. Hence the previous j − 1 trials had exactly r − 1


149

successes, and from Eq. (4.2) we get that result in probability G(r, p) :

j−1 r−1 j−r q , r−1 p

j − 1 r j−r Pr[G(r, p) = j] = p q , j>r r−1

and

(q = 1 − p).

(4.7)

Note: The negative-binomial distribution, as given in Eq. (4.7), sometimes is called the Pascal distribution (after Blaise Pascal). The survey [69] by Norman L. Johnson et al. provides historical and background information about the distribution, including several variants useful in modeling. Similar to Definition 4.24, for any integer r > 1 and a positive number p (0 < p < 1), we define a shifted negative-binomial random variable that has the following distribution: r+ j−1 r j r+ j−1 r j G0 (r, p) : Pr[G0 (r, p) = j] = pq = pq, j > 0. (4.8) j r−1 A shifted negative-binomial random variable counts the number of failures before the rth success in a sequence of Bernoulli trials with success probability p. Formula (4.8) gives the probability of j failures before the rth success on the trial number r + j. Example 4.40 Find the probability that a person flipping a fair coin gets the third heads on the sixth flip. Redo this problem in the case of unbalanced coin where the probability of getting heads is p.

Solution: Let X be the random variable of the number of flips required for three heads. Then X has a negative-binomial distribution G(3, p), with p = 1/2 for a fair coin. Therefore, 5 3 6−3 Pr[X = 6] = p q = 10 (pq)3 , 2 where q = 1 − p. For a fair coin we have Pr[X = 6] = 10 × 2−6 =

5 32

= 0.15625.

Exercise 4.41 [2] Show that the expected value of a random variable having the G(r, p) distribution equals r/p. In view of previously obtained results, explain why this value is natural. Use MAPLE (or another software) to plot the distribution for various r and p. Try to guess beforehand how its shape will change as r increases. Can you account for the shape changes as r gets larger? Exercise 4.42 [2+] Find first four moments of a negative-binomial random variable. Hint: Use formula (L.20) proved in Exercise 2.132 on page 69. Exercise 4.43 [4] (Coin-toss waiting-time problems) (a) Find the probability mass function of the random variable defined as the number of tosses of a fair coin until we observe two successive identical outcomes. (b) Find the probability mass function of the random variable defined as the number of tosses of a fair coin until we observe the sequence HT (heads follows by tails). Note: In both cases you need to count the number of strings using the letters T and H so that the first time the desired string appears is in positions k − 1, k. This uses the tools shown in Chapter 2. Example 4.44 (Random search) Suppose one needs to find a value x in an unsorted array A[1..n] consisting of n distinct

150


elements. Here is an inefficient probabilistic algorithm for the purpose, called R ANDOM S EARCH. It chooses a random index i, uniformly from [1..n] = {1, 2, . . . , n} and compares A[i] with x. If A[i] = x, it terminates; otherwise it continues the search by choosing a new random index from [1..n]. The procedure continues picking indices into A until it finds an index j such that A[ j] = x or until it checked every element of A. Note that since successive choices of i are independent, values can be repeated (equivalent to sampling with replacement). Assuming that every permutation of the set [1..n] is equally likely, we want to determine the average number of probes needed to find x if this element presents in the array A, or, if x in not in the array, before the procedure terminates. Let X be the number of required probes. We distinguish the cases of the searched value being or not being in the array A. If the x is in the array, then X is a geometrically distributed random variable with parameter p = 1/n. The expected number of probes is 1/p = n. When x is not present in the array A, the problem is equivalent to one seen elsewhere; for example, in the urn-and-ball setting, it is equivalent to finding the expected number of balls thrown until all urns are occupied. This is also equivalent to the classical coupon collector problem described below in Example 4.46. To investigate the behavior of the random variable X that tracks the number of indices checked in the array A, it is convenient to view X as the sum of random variables Xk , k > 1. Each Xk is a geometrically distributed random variable equals to the number of probes till the kth ”success.” A success, in this scenario (where the searched value is not in the array), means checking a new location for the first time. So Xk is the number of probes till the kth new position is tested, after k − 1 are already accounted for. The distribution of Xk is determined by the fact that k − 1 cells have already been checked: there are then n − k + 1 unchecked cells. def The probability of success becomes pk = (n − k + 1)/n, and Xk ∼ G(pk ). Those successive waiting times for success are all mutually independent, but since the parameter depends on k, they are not identically distributed. For the computation of E[X ] the independence is of no concern, but the exercise immediately following requires that property. The expected value is # " n n n n 1 n = n ∑ = n Hn , E[X ] = E ∑ Xk = ∑ E [Xk ] = ∑ j=1 j k=1 k=1 n − k + 1 k=1 where Hn is the nth harmonic number. This is larger by a factor of approximately ln n than the time required for a sequential search to establish that x is not in the array. There is also a performance penalty to implement the check that all positions were tested. It is needed to maintain an array of n entries (can even be bits), initialized to zero, and flipped to 1 when first tested. A counter of those flips terminates the search when it reaches n. Exercise 4.45 [2] Find the variance of the random variable X from Example 4.44. Example 4.46 (Coupon-collector problem) A candy manufacturer promotes sales by enclosing a coupon with each candy bar. There are n types of coupons. Once a collector has assembled a complete collection she is given a handsome reward. How many candy bars need be purchased to complete the collection? Clearly it is a random variable, and we denote it by Tn . We are here interested in the expected value of Tn , when each candy bar is equally likely to come with any of the coupons. The questions of higher moments, and non-uniform coupon


151

populations need more elaborate tools, and are deferred to Example 8.84, page 477. The situation is simplest if the coupon enclosed by the candy-manufacturer is drawn with uniform probability from all types, and there is no dependence between different candy bars. With uniform distribution, if the collector has already i types, her likelihood to get a new one def with purchase is pi = (n − i)/n, and therefore the number of purchases until the next new, (i + 1)st coupon appears, is a geometric random variable, G( n−i n ). Using Eq. (4.6) we see that the following total is the desired expectation, n−1

E[Tn ] =

∑

1

n−i i=0 n

n−1

=n∑

i=0

1 = nHn ≈ n(ln n + γ ), n−i

γ ≈ 0.5772156649015328.

4.1.5 The Hypergeometric Distribution The hypergeometric distribution may be seen as a variation on the binomial distribution in the following sense. Recall that a variable with the binomial distribution is the number of “successful draws” from a population of two types of items, where success is defined as drawing one type, while drawing the other is a failure. The probability of a successful draw is fixed, p. Then q = 1 − p is the probability of a failure. The reason p is fixed is that we consider the population to be infinite (or alternatively, the population is finite, but we replace the drawn item each time, so that each drawing is from the same population and hence with the same probabilities of success or failure). What if we draw without replacement? Each drawing is then from a different, slightly changed population. This is how the hypergeometric distribution arises. Suppose we have a finite population consisting of N items of two types labeled S (success) and F (failure). Let the number of S elements be m, hence the number of F’s is N − m. We take a random sample of n elements from the population without replacement, and let X be the number of items in the sample that belong to category S. This is precisely the situation considered in Example 2.68 on page 48, and, generalizing the result there, we find that the number of ways toperform the choice of n elements such that s are from S and the rest are from F is ms N−m n−s . Hence the probability of having x successes in a sample of size n is provided by the so-called hypergeometric probability mass function: h(x, n; N, m) =

m x

N−m n−x N n

,

max(0, n + m − N) 6 x 6 min(n, m).

(4.9)

The denominator is of course simply the total number of such choices. The reason for using the name “hypergeometric” is that the probabilities are the successive coefficients F (−n, −m; N − n − m + 1 | 1), where the in the power series expansion of the function (N−n)!(N−m)! N!(N−m−n)! 2 1 last factor is a particular case of a well-known function, the Gaussian hypergeometric function: F

ak bk zk a, b , z ≡ F (a, b; c| z) = 2 1 ∑ k k! c k>0 c

where ak is rising factorial, Eq. (2.38).

(4.10)


152

Example 4.47 A committee of 5 people must be selected from 8 electrical engineers and 4 mathematicians. Find the probability that a randomly chosen committee will contain 3 electrical engineers and 2 mathematicians.

Solution. Let X be the number of electrical engineers in the committee. The probability is given by 8 4 8! 4! 8! 4! 5! 7! 14 3! 5! × 2! 2! 3 2 = = ≈ 0.42424242, = Pr[X = 3] = 12 12! 3! 5! 2! 2! 12! 33 5! 7! 5

where we have cast the mathematicians in the role of failures.

The notion of sampling without replacement is natural when we think of algorithms consuming a pool of data, but it is easier to use it to encapsulate a scenario in other fields, as the following two exercises suggest. Exercise 4.48 [2] A vote is to be taken next week in town on initiative A. The town paper takes a sample of 100 adults and announces that the initiative is going to win in a landslide, since 57% of the sample would vote for it, and only 43% objected. In fact, the 5,000 voters in town are evenly divided on the initiative. Calculate the probability of the reported sample in a population equally divided on the issue, assuming population sizes of 500 or 5,000. What is the probability of a sample of 100 in an infinite population giving such a result? Exercise 4.49 [2] A flower seed producer keeps close control on the expected color of his product: he prepares his merchandise in batches of two thousand seeds, 1,900 of which are red—the most popular color. The rest is a random mix of other colors. Suppose the producer packages this batch in packets of 100 seeds and that the packing process is random. Customers are guaranteed that at least 90% of their seeds produce red flowers. What is the probability a given packet violates the warranty? With what probability a given packet violates a warranty of 95% red flowering? Let us evaluate the expected value of a random variable that has the hypergeometric PMF with the parameters of Eq. (4.9). Let the dummy index be unspecified since we want the contributions of all nonzero terms: m N−m m m−1 N −m x n−x = N ∑ µ =∑x , N x−1 n−x x x n n which is equivalent, using Eq. (2.58) on page 58, to m N −1 nm , µ = N = n−1 N n

(4.11)

where the Vandermonde convolution, Eq. (2.74) on page 67, was applied. It is interesting to note that this expected value coincides with the expected value for the binomial distribution (compare with Eq. (4.3)), which means that the mean does not depend on whether the sampling is with or without replacement. Exercise 4.50 [2] Show that for a hypergeometric random variable X , E[X (X − 1)] =

m(m − 1)n(n − 1) , N(N − 1)

σ 2 ≡ V [X ] = µ

(N − m)(N − n) mn(N − m)(N − n) = . N(N − 1) N 2 (N − 1)


153

Exercise 4.51 [3+] For a hypergeometric random variable X , find its mode, median, and the coefficient of skewness. Note: Some of the questions do not have answers in closed or even explicit form. In such cases, calculate numerical results for the particular case N = 600, n = 200, m = 40. Exercise 4.52 [4] Show that the expression for the PMF h(·) given in Eq. (4.9) admits what may be called dual or symmetric forms: n N−n dual x m−x , max(0, n + m − N) 6 x 6 min(n, m). (4.12) Pr[X = x] = h(x, n; N, m) = N m

Show this in two ways: one by a combinatorial interpretation, and the second one by an algebraic equivalence.

Another view of the sampling process that gives rise to the hypergeometric distribution is to see the population of size N as subject to two kinds of classification at the same time, independently, yielding the subsets I and J (with sizes i and j) of elements that are successes and elements that are sampled, respectively. The previously used random variable X is then the size of the intersection I ∩ J. Draw the Venn diagram corresponding to these sets to show that the population is divided into four sets, of sizes x, i − x, j − x,, and N − x − (i − x) − ( j − x) = N − i − j + x. The number of ways to get this partition is given by the multinomial N coefficient x,i−x, j−x,N−i− j+x . Hence, . N N N Pr[X = x] = . (4.13) x, i − x, j − x, N − i − j + x i j Expressing this in terms of factorials, it is immediately seen to equal the right-hand side of Eq. (4.9). (Alternatively, you can use Exercise 2.73 to write the probability in Eq. (4.13) as a product of binomial coefficients and then transform them, using the rules we developed in Chapter 2, into the form given in that equation.)

Example 4.53 Suppose that a jury was composed of 9 men and 7 women and was hung at 11 to 5 in the trial. Let U be the group of jurors, let I be the subgroup of men, and let J be the subgroup of the majority. The probability that exactly 7 jurors among the majority are men is . 16 16 16 15 ≈ 0.28846 . . . . Pr[X = 7] = = 52 7, 2, 4, 3 9 11

Exercise 4.54 [2] A hockey team of 20 players includes 7 defenders and 14 players who shoot right-handed. Assuming independence of these traits, what is the probability that exactly 5 players (chosen at random) are right-handed-throwing defenders? Definition 4.55 We say that random variables X1 , X2 , . . ., Xn have a multivariate hypergeometric distribution if their joint probability distribution is given by M1 M2 Mm x1 x2 · . . . · xm , h(x1 , . . . , xm ; n; M1 , . . . Mm ) = N n

m where xi = 0, 1, . . . n for each i and xi 6 Mi , ∑m i=1 xi = n, ∑i=1 Mi = N.

⊳

Exercise 4.56 [2+] Let random variables X1 , X2 , . . ., Xn have the multivariate hypergeometric distribution. Find the covariance of two of them.


154

4.1.6 The Poisson Distribution The Poisson probability distribution, named after the French mathematician Simon D. Poisson (1781–1840), is a frequently occurring distribution of a discrete random variable. It can be used in a number of contexts; the Poisson probability distribution models well the number of occurrences of relatively rare phenomena. If we look at events occurring over time, then the Poisson distribution arises when the following conditions hold: • The probability of an occurrence in a short time interval ∆ is proportional to its length, λ ∆. The constant of proportionality is called the rate of occurrence (and pronounced lambda). • The probability of no occurrence during ∆ is 1 − λ ∆; the probability of more than one occurrence is negligible (hence the reference to rare events). • The events in disjoint intervals are independent. This is the basis of the common claim that the Poisson process has no memory. The number of events can be described well by this distribution also when it is not small, but occurs as the sum of a large number of independent event streams each of which is relatively rare. In addition, this distribution can be shown (as we do in Example 10.21 on page 556) to be the limit, in a certain sense, of the binomial distribution B(n, p), as n increases, and p decreases while the mean np remains stationary. Example 4.57 The following list shows several examples of both types of situations that lead to the Poisson distribution. • Breakdowns of a highly reliable system. • Customers arriving at a bank during the lunch hour. • Number of automobile accidents occurring in Providence, RI. • Calls for an elevator at an office building. • The number of channel interrupts a file server receives in a millisecond. • The number of defective items manufactured by a machine per day. On the other hand, if we consider the number of arrivals of commercial airplanes at an airport, we find a different scenario: these arrivals are not independent, because planes are scheduled to arrive with some minimal separation. A somewhat similar situation is seen when we observe arrivals of patients at a physician’s office: the arrival times are not independent, since patients are typically asked to come at fairly regular intervals. Exercise 4.58 [1] Show that the incomplete exponential function en (t) = ∑06k6n t k /k! can def be related to the incomplete gamma function, which is defined via the integral Γ(a,t) = R −s a−1 ds, so that we have s>t e s et et en (t) = Γ(n + 1,t) = n! n!

Z ∞ t

e−s sn ds.

This relationship is useful in calculations of the Poisson distribution.

(4.14)


155

We consider now the probability mass function. Let the random variable X denote the number of occurrences generated by a rate-λ Poisson process during a time interval s. The distribution of X is denoted by Poisson(λ s). The probability that X has the value k equals def

Poisson(λ , k) = Pr[X = k] =

(λ s)k e−λ s , k!

k > 0.

(4.15)

Its cumulative distribution function is expressed via en (t) = ∑06k6n t k /k!, the incomplete exponential function, as follows F(x) =

(λ s)k = e−λ s ex (λ s). k! k6x

∑ Pr[X = k] = e−λ s ∑ k6x

When we look at a single time unit, s = 1, we get the standard Poisson distribution. The rate λ is also called the Poisson parameter, and we show below that it equals the mean number of events in a time unit interval; e = 2.71828 . . . is the familiar base of the natural logarithm. The times between successive Poisson events have the exponential distribution with parameter λ . The probability that an event occurs within an interval of size s is one minus that of the complementary possibility of no occurrence, or e−sλ . This leads to the following result: the probability that we need to wait for the first event at least a, but no more than b, is given by (4.16) e−λ a 1 − e−λ (b−a) = e−λ a − e−λ b ,

where the independence of the events in the two intervals allowed us to multiply the two probabilities. We can even say more: the words “first event” are not really needed. We would have the same result if we picked an arbitrary instant, and started waiting for the next event. The mean of the Poisson distribution is

µ = E[X ] =

∑ k>0

k

λ k e−λ = k!

∑ k>1

λk e−λ = (k − 1)!

∑

j>0

λ j+1 −λ e = λ e−λ j!

∑

j>0

λj = λ. j!

def

As a result, the mean time between two successive Poisson events is β = λ1 . We can use this special interval in Eq. (4.16), and find that the probability of at least one event during a mean inter-event interval is given by 1 − e−λ β = 1 − e−1 ≈ 0.6321. The Poisson distributions are right-skewed distributions over the values k = 0, 1, 2, . . ., whose probability distributions have their peaks at λ as the following exercise shows.

Exercise 4.59 [3+] Calculate the variance, mode, and the coefficients of skewness and kurtosis for Poisson distributed random variable. Find the median numerically for λ = 0.5 and λ = 7.5. Example 4.60 On average, 8.5 rooms stay vacant each day at a large hotel. Assuming the number of empty rooms follows the Poisson distribution, what is the probability that on a given day exactly four rooms will be vacant? That at most four rooms are vacant?

Solution. Let X be the number of rooms that stay vacant per day. Given that the mean is 8.5, this is the rate λ = 8.5, and we apply Eq. (4.15) to obtain Pr[X = 4] =

(8.5)4 e−8.5 (5220.06)(0.0002035) = ≈ 0.044255. 4! 24

156


Thus, the probability is just over 0.044, that exactly 4 rooms will be vacant on a given day. For the second part, we need to sum the five values Pr[X = i] for 0 6 i 6 4. There is no closed formula for this summation, and doing the calculation we find 0.074364, suggesting most of the probability mass is quite close to the mean value. Exercise 4.61 [3] Let X1 , X2 , . . . , Xm be mutually independent Poisson distributed random variables with parameters λ1 , λ2 , . . . , λm , and write λ = ∑m i=1 λi . Show that the joint conditional distribution of X1 , X2 , . . . Xm given that X1 + X2 + . . . + Xm = n, is multinomial with parameters n, λ1 /λ , . . . , λm /λ (n > 0). Exercise 4.62 [3] A manager of an IT division buys PCs from time to time. The number of times the manager orders PCs in a year is Poisson distributed with mean λ . Every time the division buys a PC, it is p% faster and q% cheaper. Given that in the first purchase a PC of speed A was priced at $B, what is the average speed and price over a year? What is the expected cost of the computers purchased in the year? Example 4.63 Packets arrive at a router according to a Poisson process with a rate of λ = 2.4 packets per µ sec (microsecond). What is the probability that the first packet arrives within 0.15 µ sec after the router is initialized? What is the mean time between arrivals of packets? What is the probability that at least five packets arrive during such a period?

Solution. The Poisson parameter is λ = 2.4 and the waiting time is the interval [0, 0.15) (use the same time units!). We apply Eq. (4.16) to obtain the required probability e−2.4×0 − e−2.4×0.15 = 1 − e−0.36 ≈ 0.3023. If packets arrive at a rate of 2.4 per µ sec, the mean time between packets is 1000/2.4 = 417 nanoseconds. (There are a 1000 nanoseconds in one µ sec.) The probability that at least 5 packets arrive during 417 nanoseconds is 1 − ∑06 j64 e−1 / j! ≈ 0.00366.

Exercise 4.64 [2] In Exercise 3.77 on page 111, we said that the probability of one failure of a system is 1% per year, and that we may disregard the likelihood of more failures. Now let us examine this claim under the (reasonable) assumption that the number of failures per time unit (a year, here) of a reliable system follows the Poisson distribution. (a) Let 0.01 be the rate of a Poisson process, what is the probability that the associated random variable, X , has the value 1? What is Pr[X > 1]? (b) Suppose that Pr[X = 1] = 0.01 for a Poisson random variable X . What is the value of the Poisson parameter? What is Pr[X > 1]? Note that there are two solutions to the implied equation for the parameter. Why should we expect this to be the case? Note: To the reader who has seen an exposition of continuous random variables previously, it must have been clear that the way we presented the Poisson probability mass function is merely the discrete facet of a dual structure. The other facet is the time between successive occurrences of the events, which are accounted for by the Poisson probability mass function, and these durations follow the exponential distribution, as said before Eq. (4.16). Exercise 4.65 [2] Show that E[X n ] = λ n , n = 1, 2, . . ., for the Poisson random variable X with parameter λ . Exercise 4.66 [3] (J. Lamperti) Let N balls be placed one by one either in urn A (with probability p, 0 < p < 1) or in urn B (with probability q = 1 − p), resulting in NA balls in urn A


157

and NB = N − NA in urn B. Show that the random variables NA and NB are independent if and only if N is a random variable having a Poisson distribution.

4.1.7 The Normal Distribution The only continuous distribution we find necessary to outline is the normal distribution, which has the following probability density function

φ (x; µ , σ ) = √

1 2πσ 2

e−(x−µ )

2 /2σ 2

for x ∈ R ,

(4.17)

with parameters µ and σ , or the N(µ , σ 2 ) distribution for short. The standard normal distribution corresponds to N(0, 1). This distribution is also called the Gaussian distribution after Carl Friedrich Gauss, who proposed it as a model for errors of measurement. It is customary to use the symbol ∼ to indicate the distribution of a random variable; so we can abbreviate the phrase “X follows a normal distribution with parameters µ and σ ” by writing just “X ∼ N(µ , σ 2 ).” The probability of X being in an interval (a, b) (since normal distributions are absolutely continuous, this is also the probability of the other intervals with the same endpoints) can be found as Pr[a < X < b] = Pr[a 6 X < b] = Pr[a < X 6 b] = Pr[a 6 X 6 b] =

Z b

φ (x; µ , σ ) dx .

a

In particular, Pr[µ − 2σ < X < µ + 2σ ] ≈ 0.954,

Pr[µ − 3σ < X < µ + 3σ ] ≈ 0.997.

(4.18)

The expected value and variance of the normal distribution X ∼ N(µ , σ 2 ) with parameters µ and σ are V [X ] = σ 2 . E[X ] = µ , These parameters µ and σ identify the normal distribution uniquely. The positive square root of σ 2 , which obviously denoted by σ , is the standard deviation of X . Note that a random variable being normally distributed has all moments, but not only first two; however, these first two moments specify precisely the normal distribution. A common symbol for a random variable that follows standard normal distribution N(0, 1) is Z and its cumulative density function is 1 def Pr[Z 6 z] = Φ(z) = √ 2π

Z z

−∞

2 /2

e−x

dx =

Z z

−∞

φ (x) dx ,

(4.19)

2

where φ (x) = √12π e−x /2 . The normal distribution plays a central role in probability and statistics. Therefore, we present its basic properties without proofs (which can be found, for example, in classical Feller’s textbook [41, principally Chapter X]). If X ∼ N(µ , σ 2 ) and Y = α X + β ,

then Y ∼ N(α µ + β , α 2 σ 2 ) .

(4.20)

X −µ , σ

(4.21)

In particular, if X ∼ N(µ , σ 2 ) and

Z=

then

Z ∼ N(0, 1) .


158

Eq. (4.21) says that any normal random variable becomes standard normal when we express it in this way. Therefore the transformation Z = (X − µ )/σ is called the standardization of a normal random variable X . If X1 and X2 are independent and X1 ∼ N(µ1 , σ12 ) and X2 ∼ N(µ2 , σ22 ), then X1 + X2 ∼ N(µ1 + µ2 , σ12 + σ22 ) .

(4.22)

As a corollary, it can be shown (see [41]) that if X1 , X2 , . . . , Xn are independent random variables, each having the N(µ , σ 2 ) distribution, then the sample sum, Sn = X1 + X2 + · · · + Xn , has the N(nµ , nσ 2 ) distribution. In particular, the sample average X n = Sn /n has the N(µ , σ 2 /n) distribution.

4.2 Types of Probabilistic Convergence In various applications, we need fine details of asymptotic behavior of random variables. It behooves us to recall that there exist several types of stochastic convergence. Since, as a rule, the probability measures we use are well behaved, we can afford a rather casual consideration. This topic is more fully explored in [124, 129]. We consider the following four types of convergence. Definition 4.67 Let {Xn }n>1 be a sequence of random variables. We say that the sequence Pr converges to a random variable X in probability, denoted Xn → X as n → ∞, if for any ε > 0 lim Pr[|Xn − X | < ε ] = 1.

n→∞

⊳

This convergence does not claim that the difference between Xn and X becomes very small everywhere on the sample space, it only assures that the probability of that difference exceeding an arbitrary small value becomes small, and vanishes in the limit. It is possible to find an example that Xn and X differ by a significant amount, but their total probability vanishes in the limit. This is the type of convergence underlying the law of large numbers (see §10.7). Another, more demanding mode of convergence is called strong convergence, as well as almost-sure or almost-everywhere convergence. It asserts the following: Definition 4.68 A sequence of random variables {Xn }n>1 converges to a random variable X (a.s.)

almost surely, denoted Xn → X as n → ∞, if for any ε > 0 lim Pr sup |Xn − X | < ε = 1. N→∞

⊳

n>N

Recall that sup denotes supremum of a set S, the least upper bound, defined as a quantity M such that no member of the set exceeds M. The almost sure convergence, also called the convergence with probability 1, assures that the set of sample points for which Xn does not converges to X has probability zero. Since this convergence is used in the strong law of large numbers (§10.7.1), it is sometimes referred to as the strong convergence. A possibly more illustrative formulation of this mode is Pr[ω : Xn 6→ X ] = 0; the measure of the set over

4.2. TYPES OF PROBABILISTIC CONVERGENCE

159

which the sequence {Xn } fails to converge to X is zero. So beyond that (random) points the difference |Xn − X | is always less than ε . A simple sufficient condition for almost-sure convergence gives the following

Theorem 4.69 (Borel-Cantelli) If for every positive ε the sum ∑n>1 Pr[|Xn − X | > ε ] < ∞, (a.s.)

then Xn → X . P ROOF : It follows directly from the following chain of inequalities: Pr sup |Xn − X | > ε = Pr [∪n>N (|Xn − X | > ε )] 6 ∑ Pr [|Xn − X | > ε ] → 0. n>N

n>N

The last inequality follows from Boole’s inequality that the probability of the union of events is smaller than the sum of the probabilities of the events. Finally, if a series converges, then its “tail” tends to zero. Definition 4.70 A sequence of real-valued random variables {Xn } is said to converge in disd

tribution (or in law) to the random variable X , denoted Xn → X as n → ∞, if their distributions, Fn , converge to the distribution function, F, of X at every point where F is continuous: lim Fn (x) = F(x).

n→∞

In this case, we also say that a sequence of cumulative distribution functions {Fn (x)}n>0 converges to the limit distribution function, F(x), in distribution. ⊳ It is shown in [124] that this definition is equivalent to lim E[ f (Xn )] = E[ f (X )]

n→∞

(4.23)

for all bounded continuous functions f . This is the type of convergence that leads to the central limit theorem 10.136. Convergence in distribution says that F(x) may be used in place of Fn (x) in approximate calculations. This is useful in the common situations when Fn (x) is complicated and F(x) is a comparatively simple function. Example 4.71 In this example, we show that neither continuity nor boundedness can be eliminated to hold Eq. (4.23). Consider a sequence {Un } of uniformly distributed random varid

ables on the interval [1, n]. Observe that Un → U∞ , where U∞ is identically zero. Let f (x) be a bounded, but not continuous function: ( 1, if x > 0, f (x) = 0, if x 6 0.

Then we have the following expectations for each finite n: E[ f (Un )] = f (1/n) = 1, which do not converge to the expectation of that same function of the limiting distribution: E[ f (U∞ )] = f (0) = 0.


160

Define the sequence of random variables Xn that take only two values 0, with probability 1− n1 d

and n, with probability 1n . Observe that Xn → X∞ , where X∞ is identically zero. Choosing the function f (x) = x, we have 1 1 +n = 1, E[ f (Xn )] = E[Xn ] = 0 1 − n n which do not converge to E[ f (X∞ )] = E[X∞] = E[ f (0)] = 0. Example 4.72 Let the sequence of cumulative distribution functions be r Z x 2 2 n e−nt /2σ dt. Fn (x) = 2πσ 2 −∞

Then at every x 6= 0, this sequence converges to the discontinuous distribution function: ( 0, if x < 0, Fn (x) −→ F(x) ≡ n→∞ 1, if x > 0. At x = 0, Fn (0) = 1/2 for all n, hence Fn (0) does not converge to F(0) = 1 (the cumulative distribution function must be continuous from the left, so F(0) = limε →0 F(ε ) for positive ε ). Nevertheless, the sequence Fn converges in distribution to F. However, if we consider another sequence of distribution functions ( 0, if x < n, Fn (x) = 1, if x > n, then Fn (x) → 0 for all x. But this sequence does not converge in distribution to F(x) ≡ 0 because zero cannot be a cumulative distribution function. The relation between these three modes of convergence is summarized in the following relation: (a.s.) Pr d Xn → X =⇒ Xn → X =⇒ Xn → X . (4.24) Definition 4.73 The sequence, { Xn }, of random variables converges in mean of order p, 0 < p < ∞, to the random variable X if E[|Xn − X | p] → 0,

n → ∞.

(4.25) Lp

In functional analysis this convergence is known as convergence in L p , and denoted by Xn → X . When p = 2, we have what’s called the mean square convergence, which is denoted by X = l.i.m. Xn , for “limit in the mean.” ⊳ The convergence in probability follows from convergence in the mean of order p > 0: Lp

Xn → X

=⇒

Pr

Xn → X ,

but its proof requires application of Chebyshev’s inequalities (see §4.7).

(4.26)

4.2. TYPES OF PROBABILISTIC CONVERGENCE

161

Example 4.74 Let χ(a,b) be the indicator function of the interval (a, b), that is, ( 1, if a < ω < b, χ(a,b) (ω ) = 0, otherwise. Define the sequence of random variables Xn (ω ) = n χ(0,1/n) (ω ). Then, clearly, Xn converges almost surely to 0. However, E[Xn ] =

Z 1/n

n d ω = 1.

0

(a.s.)

Actually we show that Xn converges to 1 in mean of order p, provided 0 < p < 1, but Xn → 0.

Instead we choose fn (x) = n χ(0,1/n) (x) to be a density function of the random variable Xn′ . Then we have fn (x) → 0 as n → ∞ for all real x. However, these densities do not converge to zero in L1 because Z Z ∞

−∞

1/n

| fn (x) − 0| dx =

n dx = 1,

0

which does not tend to zero as n → ∞.

(a.s.)

Now we consider another sequence of random variables Yn = 2n χ(0,1/n) . The clearly Yn → 0, Pr

and so Yn → 0. However,

p

E[|Yn − 0| =

E[Ynp ]

=

Z 1/n

(2n ) p d ω =

0

2np →∞ n

as n → ∞,

for every p > 0. Therefore Yn does not converge to zero in L p . The third sequence Zn = n1/2 χ(0,1/n) (x) converges to zero almost surely, in probability, in distribution, and in L1 as n → ∞. However,

E[Zn2 ] =

Z 1/n √ 2

n dx = 1.

0

i h Hence limn→∞ V [Zn ] = limn→∞ E[Zn2] − (E[Zn ])2 = limn→∞ [1 − 1/n] = 1 6= 0 = V [0].

Exercise 4.75 [2] Give an example of a sequence that converges both in probability and in mean, but does not converge almost surely. Exercise 4.76 [1] Consider the following two sequences, and determine which types of convergence they exhibit (and which they do not): ( ( 0 in probability 1 − 1/n, 0 in probability 1 − 10−n , Yn = Xn = −n 1 in probability 1/n. 1 in probability 10 ; Exercise 4.77 [2] For given sequences of cumulative distribution functions (a) Fn (x) = x −

2nπ x , 2nπ

0 6 x 6 1;

(b) Fn (x) = 1 − (1 − x)n ,

0 6 x 6 1;

n sin nx x ; , x > 0; (d) Fn (x) = (c) Fn (x) = 1+x πx determine which of them converges in distribution; if your answer is positive, find the limit cumulative distribution function.


162

A4 A1

B

A3

A5 A2

5 with an intersecting set B. Figure 162: A partition {Ai }i=1

4.3 The Theorem of Total Probability The main tool in this section is the following statement, usually called the theorem of total probability: Theorem 4.78 Let the events4 A1 , A2 , . . . , An be a partition of the sample space Ω. That is, n A , • Ω = ∪k=1 k • the {Ak } are disjoint (mutually exclusive: they satisfy Ak ∩ A j = ∅, for k 6= j), and • Pr[Ak ] > 0 for k = 1, 2, . . . , n.

Then for any event B,

n

Pr[B] =

n

∑ Pr[B ∩ Ak] = ∑ Pr[B | Ak ] Pr[Ak ].

k=1

(4.27)

k=1

P ROOF : The event B is the union of the mutually exclusive events B ∩ A1, B ∩ A2, . . . , B ∩ An and hence B = (B ∩ A1 ) ∪ (B ∩ A2 ) ∪ · · · ∪ (B ∩ An ). Then from property 3 (page 86), it follows that n

Pr[B] = Pr[B ∩ A1] + Pr[B ∩ A2 ] + · · · + Pr[B ∩ An ] =

∑ k=1

n

Pr[B ∩ Ak ] =

∑ Pr[B | Ak ] Pr[Ak ].

k=1

Corollary 4.79 Let conditions of Theorem 4.78 hold. Then we have the law of total expectation: E[B] = ∑ E[B | Ak ]Pr[Ak ]. k

The same derivation holds for conditional probabilities as well, and we get 4 Their

number may be either finite or countably infinite.

4.3. THE THEOREM OF TOTAL PROBABILITY

163

Corollary 4.80 Let A1 , A2 , . . . , An be a partition of the sample space Ω, and let B and C be events. Then Pr[B | C] = ∑ Pr[B | Ak ∩C] Pr[Ak | C], (4.28) k

since the same claim holds for the reduced sample space as well. This method of computing the probability of an event by constructing it from its conditional probabilities over a partition is called randomization. The same approach holds for computing the PMF of a random variable, as the following examples show. Example 4.81 Three urns contain both red and white balls. Urn 1 contains two reds and three whites, urn 2 contains three reds and four whites, and urn 3 contains 40 reds and 50 whites. An urn is chosen at random (with probability 1/3 for each) and a ball is chosen at random from it (all balls in an urn are equally likely). What is the probability that the ball is red?

Solution. Let Ak be the event that urn k is chosen (k = 1, 2, 3) and let B denote the event that the ball is red. Then Pr[Ak ] = 1/3 (k = 1, 2, 3). The values of conditional probabilities B and Ak follow from the problem statement: 2 Pr[B | A1 ] = , 5

3 Pr[B | A2 ] = , 7

Pr[B | A3 ] =

40 4 = . 90 9

The events A1 , A2 , and A3 constitute a partition of the sample space because they are disjoint and their union includes all possible outcomes. From the theorem of total probability, it follows that 3 1 401 1 2 3 4 + + ≈ 0.424. = × Pr[B] = ∑ Pr[B | Ak ] Pr[Ak ] = 3 5 7 9 3 315 k=1 Example 4.82 As an application of the corollary, let us consider the following scenario. Again, as in Example 4.81, we have three urns, but with different contents: each contains two coins. Urn 1 contains two gold coins; urn 2 contains a gold coin and a silver coin; and urn 3 contains two silver coins. We choose one of the urns at random, with all three urns equally likely. Then we take a coin at random from it and put it away. What is the probability that we get a gold coin on the second draw from the same urn given that the first coin was gold?

G

G

G

Urn 1

S

Urn 2

S

S

Urn 3

The events Ak (k = 1, 2, 3) are defined as before; they are obviously a partition of the sample space, and we continue to assume Pr[Ak ] = 1/3, k = 1, 2, 3. Let G1 and G2 be events of taking a gold coin on the first and the second drawing respectively. We are asked to determine the conditional probability Pr[G2 | G1 ]. In order to find its value, we apply Corollary 4.80 to obtain 3

Pr[G2 | G1 ] =

∑ Pr[G2 | Ak ∩ G1] Pr[Ak | G1].

k=1


164

Calculations according to Definition 3.105 show that 1 1 ×1 = ; 3 3 1 1 1 Pr[A2 ∩ G1 ] = Pr[A2 ] Pr[G1 | A2 ] = × = ; 3 2 6 1 Pr[A3 ∩ G1 ] = Pr[A3 ] Pr[G1 | A3 ] = × 0 = 0. 3

Pr[A1 ∩ G1 ] = Pr[A1 ] Pr[G1 | A1 ] =

Adding these values we find, unsurprisingly, as the symmetry between gold and silver in the layout suggests, that the probability of first drawing a gold coin is 1/2. Now we can calculate the needed conditional probabilities, Pr[A1 ∩ G1 ] 1/3 2 = = ; Pr[G1 ] 1/2 3 Pr[A2 ∩ G1 ] 1/6 1 = = ; Pr[A2 | G1 ] = Pr[G1 ] 1/2 3 Pr[A3 | G1 ] = 0, since there is no gold coin in the third urn,

Pr[A1 | G1 ] =

Similarly, we can find the following conditional probabilities: Pr[A1 ] 1/3 Pr[G2 ∩ G1 ∩ A1 ] = = = 1; Pr[A1 ∩ G1 ] Pr[A1 ∩ G1 ] 1/3 Pr[G2 | A2 ∩ G1 ] = 0; Pr[G2 | A3 ∩ G1 ] = 0; Pr[G2 | A1 ∩ G1 ] =

because you can withdraw a second gold coin only from urn # 1 (with probability 1). Now we obtain the required probability to be 1 2 2 Pr[G2 | G1 ] = 1 × + 0 × + 0 = . 3 3 3 Thus, the probability is 2/3 to pick up the gold coin from an urn chosen at random, given that the first draw was a gold coin. Note on terminology: A common usage in describing the action in Example 4.82 is to say that the probability distribution leading to the choice of an urn is a mixer of the distributions of drawing gold coins from the urns. The distributions Pr[G1 | Ak ] are mixed by the choice of Ak . Other common usages for the mixer are randomizing, or conditioning distributions. We also use this terminology to describe the mixing of distributions of random variables. Exercise 4.83 [2] (Mixing of distributions) Three boxes are numbered 1, 2, and 3; box number k contains k gold coins and 5 − k silver coins, k = 1, 2, 3. In a two-step experiment, a box is selected at random, with the probability of box k being selected proportional to k. (a) Two coins are drawn from the selected box (without replacement). What is the probability that the two coins are different? (b) In a second experiment with the same equipment, once a box is selected, coins are withdrawn from it repeatedly with replacement until the first time a gold coin is retrieved. What is the distribution of the number of withdrawals, R?


165

Example 4.84 (Monty Hall problem5 ) Consider a TV show in which you are the contestant. You are asked to choose among three doors, behind one of which there is a prize. Following your choice, but before the door is opened, the host opens with a flourish one of the other doors, and shows there is nothing behind it. He then challenges you to change your choice. Should you?

Solution: We need to make assumptions and define notation. We first assume the host always opens a door (in other words: his opening it does not depend on the choice you made). Let us denote the prize door by P, the door you choose by Y and the door Monty opens by M. The set of the three doors is denoted by D = {a, b, c}.

Next we make distributional assumptions: (i) P is uniformly distributed over D; (ii) Y is uniformly distributed over D, and is independent of P ( = nobody gave you a hint you believe); (iii) if P = Y , then M is distributed uniformly over D \ P, that is: each of the other two doors is equally likely to be opened; but if P 6= Y then M is deterministically the only remaining door in D \ {P,Y }.

From (i), (ii) and Eq. (3.34), it follows that Pr[P = Y ] = 1/3. This is not a happy situation, but if you have no further information, there is no point in switching. Fortunately the impetuous Monty usually helps. To convince you that this is the case, let us compute the probability you win (which we denote by W ) if you switch (denoted by S), using the theorem of total probability: Pr[W | S] = Pr[W | S, P = Y ] Pr[P = Y ] + Pr[W | S, P 6= Y ] Pr[P 6= Y ] = 0 ×

2 2 1 +1× = . 3 3 3

The 0 and 1 above reflect the determinism of loss if you switch once you guessed right (Y = P) or gain when Y 6= P, because of the determinism of Monty’s action when you initially guessed wrong. And if you do not switch, denoted by Sc , we need to calculate Pr[W | Sc ]: 2 1 1 Pr[W | Sc ] = Pr[W | Sc , P = Y ] Pr[P = Y ] + Pr[W | Sc , P 6= Y ] Pr[P 6= Y ] = 1 × + 0 × = . 3 3 3 You double your chances if you switch! This is then the way to go (never tell this to the guy who did the math and switched from a winning choice!). Exercise 4.85 [3] What is your probability to win when Monty opens a non-winning door (in the case you pointed first on the prize door) with probability q? Example 4.86 In the USA, approximately 6% of the females and 9% of the males are lefthanded; 48% of the population are males. What is the probability that a randomly chosen person is left-handed?

Solution. Let B be the event that a person is left-handed and let A1 (A2 ) be the event that the person chosen is female (male). Then we are given that Pr[B | A1 ] = 0.06

and

Pr[B | A2 ] = 0.09.

5 This problem is a simulation of a popular TV show “Let’s Make Deal” hosted by Monty Hall. The name to the problem was given by Marilyn vos Savant who posed the problem in her column in 1990. A good review of literature on Marilyn’s game and relevant problems is given in the article by Herb Bailey [13].


166

Let us pretend that A1 and A2 constitute a partition of the sample space, with Pr[A1 ] = 0.52 and Pr[A2 ] = 0.48. We can apply Theorem 4.78 to obtain Pr[B] = Pr[B | A1 ] Pr[A1 ] + Pr[B | A2 ] Pr[A2 ] = 0.06 × 0.52 + 0.09 × 0.48 ≈ 0.074.

Now we can restate the theorem of total probability for random variables replacing the conditions Ak in Eqs. (4.27)–(4.28) by the events Y = y for various values of a random variable Y . The event B in these equations can be described in terms of another random variable X . Then, for example, the law of total expectation becomes E[X ] = ∑ E[X | Y = y] Pr[Y = y] = ∑ E[X | Y = y] pY (y). y

y

Let φ (y) = E[X | Y = y] and we rewrite the last equation as E[X ] = ∑ φ (y)pY (y), y

which is just the expected value of the function φ (Y ). Now we can make it even more succinct. If φ (y) denotes E[X | Y = y], then we denote φ (Y ) by E[X | Y ], that is, E[X | Y ] is a random variable being a function of Y . Now we recalculate E[X ] = ∑ E[X | Y = yr ] Pr[Y = yr ] = ∑ ∑ xk Pr[X = xk | Y = yr ] Pr[Y = yr ] r

r

k

= ∑ ∑ xk Pr[X = xk ,Y = yr ] = ∑ xk Pr[X = xk ], r

(4.29)

k

k

as we compute the marginal distribution of X . Example 4.87 (Computing conditional expectations) A number is picked up at random from the set [1..n] = {1, 2, . . . , n}, which we denote by Y . Then a second number is chosen from the set [1..Y ], which we denote by X . Since Pr[X = i | Y = k] = 1/k, the conditional expected value of X given Y = k is k

k

k+1 i = , 2 i=1 k

E[X | Y = k] = ∑ i Pr[X = i | Y = k] = ∑ i=1

and E[X | Y ] =

Y +1 2 .

Similarly, k

k

i2 (k + 1)(2k + 1) = . 6 i=1 k

E[X 2 | Y = k] = ∑ i2 Pr[X = i | Y = k] = ∑ i=1

The mean values of the random variables X and Y have the following values: n

E[Y ] =

k=1 n

E[X ] =

n

k

∑ k Pr[Y = k] = ∑ n = k=1

n+1 ; 2 n

k+1 1 n+3 · = . n 4 k=1 2

∑ E[X | Y = k] Pr[Y = k] = ∑

k=1


167

In order to calculate the covariance, we first need to find n

E[XY ] =

n

n

y

1

1

n

1

∑ ∑ xy Pr[X = x,Y = y] = ∑ y n ∑ x y = 2n ∑ y(y + 1) = x=1

y=1

y=1 x=1

y=1

(n + 1)(n + 2) . 6

Using Eq. (3.45) on page 131, we obtain Cov(X ,Y ) = E[XY ] − E[X ]E[Y] =

(n − 1)(n + 1) . 24

For the correlation we need the second moments: E[Y 2 ] = E[X 2] =

n

n

k2 (n + 1)(2n + 1) = ; 6 k=1 n

∑ k2 Pr[Y = k] = ∑ k=1 n

∑ E[X 2 | Y = k] Pr[Y = k] =

k=1

(k + 1)(2k + 1) 1 4n2 + 15n + 17 · = . ∑ 6 n 36 k=1 n

Hence, the variances are

σY2 =

(n − 1)(n + 1) , 12

σX2 =

7n2 + 6n − 13 . 144

Thus, for n > 1,

Cov(X ,Y ) = ρ (X ,Y ) = σX σY

s

n2 − 1 = 7n2 + 6n − 13

r

n+1 . 7n + 13

Also k + 1 2 k2 − 1 i2 = . V [X | Y = k] = E[X | Y = k] − (E[X | Y = k]) = ∑ − 2 12 i=1 k 2

2

k

Exercise 4.88 [3] This exercise relies on the assumptions of Mendelian genetics (page 88). Suppose that one of the genes in our body exhibits two alleles, a dominant R, and a recessive w. If the probabilities of the genotypes RR, Rw, and ww are currently p, q, and r respectively, for both males and females, what is the probability that their child will have ww? Redo the question assuming that the two genders have different probabilities of having RR, Rw, and ww say pm , qm , rm and p f , q f , r f . Exercise 4.89 [2] An urn contains 2 red balls and 3 white ones; a second urn has 3 and 4 of these colors, respectively. A ball is chosen at random from the first urn, and tossed (unseen) into the second urn—from which we now draw a ball at random. What is the probability that this ball is red? The problem is continued in Exercise 5.61 on page 220. Exercise 4.90 [2] The following balls-in-a-bin game starts with two balls, one black and one white. Repeatedly, a random ball is drawn and then replaced and another one of the same color added. We continue until the bin holds n balls. Show that the number of white balls is then uniformly distributed between 1 and n − 1.

168


Exercise 4.91 [2+] Consider the movement of an arm carrying the reading heads of a disk. We model the disk as an array of tracks (or cylinders) and consider the movement of the arm between two successive operations. In a computer that has multiple concurrent processes active, it is reasonable to assume that both requests are in uniformly and independently distributed tracks. Let the tracks for the two requests be I and J and then the arm has to travel a “distance” of D = |I − J| tracks. (a) What is the PMF of D? (b) Calculate the expectation of D and its variance. Exercise 4.92 [3] Let X1 be a number chosen at random from the set [1..n] = {1, 2, . . . , n}. Then X2 is picked up at random from the set [1..X1 ], and additional random variables X3 , X4 , . . ., are generated in the same way. Find the expected value E[Xk ] and the variance V [Xk ]. Hint: Consult §5.3 if you decide to use a recurrence. Exercise 4.93 [2] A gambling game is called fair if your expected win (or loss) is zero. For example, a game in which you can win or lose the same amount in probability 1/2 each is a fair game. A game in which you win w with probability p and lose l with probability q = 1 − p is also fair when wp = lq. We look at a fair game in which you can win $1,000 or lose $N on each round. Here is the description of a round of the game: start with an empty urn and a number of white balls and one red ball. Begin flipping a true coin. When heads comes up, put a white ball into the urn. When tails shows up, put the red ball into the urn and stop tossing. Then mix the balls in the urn and pick one at random. You win if you get a white ball and you lose if you get the red ball. What is the value of N given that the game is fair? Exercise 4.94 [4] The files sent to a central printing spool in our system have surprising statistical properties: their lengths, measured in pages printed, have a uniform distribution from 1 to k pages. One morning the printer is found to have jammed hours ago, and the queue, which includes one file you sent for printing, has n print jobs. It was found a quite surprising fact that no two jobs had the same length. (It also means, of course, that n 6 k.) The spooler prints jobs according to size—smallest first. Let the length of the your file be denoted by L, and its position in the queue by R. (a) What is the probability of this “surprising development,” of distinct page counts? (b) You find that your job is in the rth position in the (length-sorted) queue (r ∈ [1 .. n]). Using the given information, what is the conditional distribution of L? That is, compute Pr[L = j | R = r]. (c) You remember the job you sent was j pages long. Again, using the above information, what is the probability mass function of R? That is, compute Pr[R = r | L = j]. (d) Calculate Pr[R = r | L = j] when there is no information about the possibly repeated values of job lengths. Note: There is no obvious closed form for the result of this part. Exercise 4.95 [3] A box contains 10 white balls, 20 red balls, and 40 yellow balls, from which 10 balls are selected at random, one at a time. Determine the probability that at least one color will be missing from 10 selected balls when the selection is done (a) with replacement, (b) without replacement. Exercise 4.96 [2] Two players take turns rolling a pair of true dice. The winner is the first player who obtains the sum 7 on a roll of the two dice. Find the probability that the person who rolled first wins.


169

Exercise 4.97 [2] Let k numbers be chosen at random (equiprobably) from the set [1..n] with replacement. Let X be their sum. Show that E[X ] = k(n + 1)/2. Exercise 4.98 [3] (M. Hofri) Let A and B be two randomly selected integers. We define the ratio q = A/B = α /β , where α and β are integers that have no common factors. What is the probability that the denominator, β , is even? Exercise 4.99 [3] Suppose you play a game based on probability p of success (by flipping a biased coin). Each time a failure occurs, you put a ball into an urn. Each time a success occurs, the urn empties. What is the expected waiting time until the urn contains n balls for the first time starting from an empty urn? Exercise 4.100 [3+] Suppose you are given two arrays, A and B, of n numbers each. Let the random variable K count the number of entries in A that are strictly larger than the corresponding B entries: K =| {i, 1 6 i 6 n | Ai > Bi } |

Compute the mean of K and its variance based on the following distributions of entries in these arrays: (a) A and B contain two random, independent permutations of [1..n]. (b) A and B contain random samples from the real interval [0, 1]. You can assume that with probability 1 all 2n numbers are distinct. (c) A contains a sample as in part (b), but from the interval [0, 2]; B is just as in part (b). (d) The arrays A and B together contain a random permutation of [1..2n], uniformly and independently distributed over the arrays. Exercise 4.101 [3] (adapted from [99, §1.4]) The minimum cut-set problem (min-cut problem, for short) is a standard graph-theoretic problem; given a simple, connected graph, the goal is to find a minimum cardinality cut-set, defined as a set of edges whose removal breaks the graph into two (connected) components. This problem arises in many contests, including network reliability. The following algorithm is a simple randomized version to solve the min-cut problem for a simple connected graph with n vertices: 1. Repeat n − 2 times: (a) Pick an edge uniformly at random among all that exist, and let its endpoints be (u, v). (b) Merge u and v, eliminating all edges connecting them, and retaining edges to other nodes, (it may lead to a nonsimple graph). 2. Output the set of edges connecting the two remaining nodes. (a) Show a simple example of a graph where a lucky choice of edges to merge leads to the final result being a minimal cut-set, and an unlucky one leads to a result of a larger cut set. Argue that the algorithm can never produce a cut-set smaller than that of the original graph. 2 . (b) Prove that the algorithm outputs a minimal cut-set with probability > n(n−1) (c) The probability bound you obtained in part (b) is rather discouraging. How many times would you need to repeat the experiment, assuming independent repeats can be made, to get a legitimate minimal cut-set in probability exceeding 1 − ε ? Find an upper bound on the number of required repeats, in terms of n and ε . Exercise 4.102 [2+] A tennis club invites 32 players of equal ability to compete in an elimination tournament. What is the probability that two given players will compete against each


170

other?

4.4 Bayes’ Theorem The theorem derives its name from the Reverend Thomas Bayes (1702 – 1761), whose 1763 paper (published posthumously) contained the result. The first explicit statement of Bayes’ theorem was made by P. Laplace in 1812. We present it in modern terminology: Theorem 4.103 If a finite or countable set of events A1 , A2 , . . . form a partition of a sample space Ω, then for any event B and an element of the partition, Ak , we have Pr[Ak | B] =

Pr[B | Ak ] Pr[Ak ] Pr[B | Ak ] Pr[Ak ] = . Pr[B] ∑ j Pr[B | A j ] Pr[A j ]

(4.30)

P ROOF : By the theorem of total probability the denominator equals Pr[B], and then using Eq. (3.30) twice—once to see that the numerator is the probability of the event B ∩ Ak , and then to evaluate the ratio—completes the proof. Note: This odd-looking theorem is a marvelous tool to reason about probabilities. This reasoning takes the form of updating our estimates of probabilities of certain events, based on observations. Read the example and exercises below to understand the meaning of this claim. While we do not want to get too far afield, we note that over the last century, this notion of evolving probabilities has led to a view of statistical inference that is distinct from the “frequentist” approach and leads to different interpretations of data. It has also produced remarkable controversies. Most of the books that discuss the distinction seem to shed more heat than light on the topic, but several direct, excellent treatments of Bayesian statistics are available, notably the classical reference written by D. V. Lindley [91] and the more recent exposition by Donald A. Berry [17]. For two events A and B, Eq. (4.30) gives us so called Bayes’ little theorem: Pr[A | B] =

Pr[A] Pr[B | A] Pr[A] Pr[B | A] = , Pr[B] Pr[A] Pr[B | A] + Pr[A′ ] Pr[B | A′ ]

(4.31)

where A′ is the complementary event. From the inequality (1 + x−1 )−1 < x for x > 0, it follows Pr[A] Pr[B | A] . Pr[A | B] < Pr[A′ ] Pr[B | A′ ]

One of the possible practical applications of Bayes’ theorem we find in jurisprudence when DNA blood-matching technique is used to establish guilt of a person. Every human carries DNA in each blood cell containing information that is so distinctive that the probability of a random match is between 10−8 and 10−10 . In practice, not all information from a site can be determined, which depends on the amount of DNA or degree of contamination. Example 4.104 Computer virus diagnostic tests—just like medical ones—are not entirely reliable. A test might indicate the presence of a virus when the software is in fact clean (but contains a trigger string—this is called a false positive), or it might indicate the absence of a virus when the software in fact has some virus (of a kind unknown to the testing program—a

4.4. BAYES’ THEOREM

171 0.98

+

0.05 x 0.98 = 0.049

0.02

--

0.05 x 0.02 = 0.001

+

0.95 x 0.03 = 0.0285

-

0.95 x 0.97 = 0.9215

VIRUS 0.05

0.03

0.95 FREE

0.97

Figure 171: Decision tree for Example 4.104. false negative). For the sake of argument, suppose that approximately6 95% of available software products are virus-free. Suppose a company released a new version of virus diagnostic software tool that can correctly diagnose the presence of a virus in 98% of the cases and can correctly diagnose absence of viruses in 97% of the cases. Find the probability that a software for which the virus test result is positive is actually infected.

Solution. First, we introduce notations. Let V be the event that the software has a virus; the complementary event V ′ means that the software is clean. We use “+” to denote the event that the software received a positive diagnosis (claiming a virus is present) and “−” to denote the event that the software received a clean verdict. Then we have the following probabilities: Pr[V ] = 0.05

and

Pr[V ′ ] = 0.95;

Pr[+ | V ] = 0.98

and

Pr[− | V ′ ] = 0.97.

Hence Pr[− | V ] = 0.02

and

Pr[+ | V ′ ] = 0.03,

but the conditional probability, Pr[V |+], is to be determined. We draw the tree diagram and get the following four probabilities at the right of the tree: Pr[V ∩ +] = 0.049,

Pr[V ∩ −] = 0.001,

Pr[V ′ ∩ +] = 0.0285,

Pr[V ′ ∩ −] = 0.9215.

Therefore, we use the Bayes formula (4.30) to obtain the remarkable result: Pr[V | +] =

0.049 98 Pr[V ∩ +] = = ≈ 0.632. Pr[V ∩ +] + Pr[V ′ ∩ +] 0.049 + 0.0285 155

An inspection of this terrible accuracy indicates that this is the case when the error-level of the test is too close to the likelihood of the event it is testing. The following exercises strengthen this observation. Exercise 4.105 [2] A lie detector is accurate 4/5 of the time; that is, if a person is telling the truth, the lie detector indicates he or she is telling the truth with probability 4/5, while if the person is lying, the lie detector indicates that he or she is lying with probability 4/5. Assume that a person taking the lie detector test is unable to influence its results, and also assume that 6 The

particular numbers we use here are apocryphal, and, we hope, pessimistically biased as well.

172


90% of the people taking the test tell the truth. (a) What is the probability that a person is lying if the lie detector indicates that he or she is lying? (b) What is the probability that a person is telling the truth if the lie detector indicates that he or she is telling the truth? Exercise 4.106 [2] In so called jailer’s paradox, three prisoners, A, B, and C, know that two of them will be freed, but only jailer knows which. Now A wants to send a letter to his girlfriend, and since he knows that with certainty either B or C is going free, he asks the jailer to tell him the name of one of the other two who will be freed and thus would be able to deliver the letter. The jailer rejected to reveal the name, explaining that as of now A has 1/3 probability of staying in prison but if he were to tell the name, then A’s probability of being jailed would increase to 1/2. Is jailer correct? Note: This problem sometimes is formulated as the prisoner paradox. Three prisoners learn that two of them have been chosen at random to be executed the following morning and the jailer knows who will survive, but will not tell the prisoners. Prisoner A realizes that his chance of survival is 1/3. If the jailer will provide to him the name of one of the other two who will be executed, would the probability rise the chance of A’s survive to 1/2? Exercise 4.107 [2] In 1996, the Center for Disease Control estimated that 1,000,000 of the 261,000,000 residents of the United States are HIV-positive. (HIV is the virus that is believed to cause AIDS.) The SUDS diagnostic test correctly diagnoses the presence of AIDS/HIV 99.9% of the time and correctly diagnoses its absence 99.6% of the time. Find the probability that a person whose test results are negative is free of HIV, and find the probability that a positive test result implies the patient is sick with HIV. Exercise 4.108 [2] In a large class, four teaching assistants grade the homework. Since they have different levels of experience, they are allocated different amounts of work and have differing likelihoods of making grading errors. Alice grades 10% and is extremely meticulous— her error rate is 2%. The same data for Bob are 20% and 4%, for Chris, 30% and 3%, and for Doris, 40% and 2.5%. When a student comes to complain about a grade, what is the probability (assuming her complaint is justified) that Alice is at fault? Exercise 4.109 [2+] Let X be a number chosen randomly and uniformly from the set [1..n] = {1, 2, . . . , n}. We continue to sample the set [1..n] in the same way (with replacement) until we get the number Y > X for the first time. This happens at the Rth attempt. Find the distribution of R and the probability of the event Y = n. Exercise 4.110 [2+] Given a box containing n fair coins and m coins with tails on both sides, we select a coin at random and flip it. Given that tails is observed, what is the probability that we flipped a fair coin? How do repeated flips, which all show tails, modify this probability? Exercise 4.111 [2] The three machines M1 , M2 , and M3 are used to produce a large volume of some product. Suppose that 30 percent of the product is produced by machine M1 ; 20% by machine M2 ; and 50% by machine M3 . Assume it is known that 3% of the product produced by machine M1 is defective, 2% of the product produced by machine M2 is defective, and 1% by machine M3 is defective. Finally, suppose that one item is selected at random from the entire batch and it is found to be defective. What is the probability that this item was produced by machine M1 ? Example 4.112 (Continuation of Example 4.104)

4.4. BAYES’ THEOREM

173

Suppose that the computer virus diagnostic test is based on a randomized routine. This means that application of the diagnostic test twice to the same software may result in two different conclusions—false or true. This is very similar to the medical or engineering tests because screening procedures are not 100% accurate. As Example 4.104 shows, even a very reliable computer diagnostic test does not assure us that a particular software, which passes the test, has the indicated property (a virus). One of the “obvious” remedies is to apply the test again. Having observed the posterior probabilities (after application of the computer virus diagnostic test) 57 98 , Pr[V ′ | +] = , Pr[V | +] = 155 155 we revise the probabilities for the particular software to contain a virus or to be clean as follows Pr[V ] = 98/155 and Pr[V ′ ] = 57/155. After that we test the software a second time, and suppose that the test reveals the same verdict—the program contains a virus. In light of the second test result, using 98/155 and 57/155 as our new prior probabilities, we obtain Pr[V ∩ {+ and +}] Pr[V ∩ {+ and +} + Pr[V ′ ∩ {+ and +} 9604 98 × 98 = ≈ 0.9825. = 98 × 98 + 57 × 3 9775

Pr[V | {+ and +}] =

Our confidence in the hypothesis that the software contains a virus has been strengthened considerably after double tests. This indicates clearly that application of the diagnostic test twice virtually eliminates the possibility of a virus to slip through. The problem discussed in the previous example can be generalized to have a practical application. Suppose that a random experiment has been performed n times, and an event E has been observed to occur k times, what is the probability that E will happen to be one more time if the experiment is repeated? The answer is due to P. Laplace, and is referred to as the Law of Succession: Theorem 4.113 Suppose that an event E occurred k times in n independent trials. The probability that it will be observed one more time at the (n + 1)-th trial is (k + 1)/(n + 2) if • the probability for the occurrence of E remains the same from trial to trial; • trials are independent; • the unknown probability of the event E, treated as a random variable, has a uniform (continuous) distribution over the interval (0, 1). P ROOF : Let p denote the unknown probability of the event E in any single trial. The number of times the event E occurs in n independent trials is a binomially distributed random variable, denoted by N. Its conditional density given p is n k def f (k | p) = Pr[N = k | p] = p (1 − p)n−k , for k = 0, 1, . . . , n, 0 < p < 1. k Let B be the event that E occurs k times in n trials (that is, N takes the value k), and let A denote the event of observing E at the (n + 1)-th trial. Since p is uniformly distributed, we


174

have Pr[B] =

Z 1 0

Z 1 n n k! (n − k)! k n−k f (k | p) d p = p (1 − p) d p = k 0 k (n + 1)!

due the value of the standard beta integral (see [1]): Z 1 0

xa (1 − x)b dx =

a! b! . (a + b + 1)!

The first assumption of the theorem guarantees that trials are independent, so we get Z 1 Z 1 n (k + 1)! (n − k)! n k+1 n−k . p (1 − p) d p = p f (k | p) d p = Pr[A ∩ B] = (n + 2)! k k 0 0 From Eq. (3.30), it follows Pr[A | B] =

Pr[A ∩ B] k + 1 = . Pr[B] n+2

Exercise 4.114 [1] Suppose that you have observed two automobile accidents in seven days. What is the probability that another accident will be observed on the eighth day? Can you apply the law of succession to find the probability? Exercise 4.115 [2] For a finite set of events (or hypotheses), A1 , A2 , . . . , Am , that form a partition of a sample space, let pi = Pr[Ai ], i = 1, 2, . . . , m, be their (prior) probabilities. Suppose that the conditional probabilities, ei = Pr[B | Ai ], i = 1, 2, . . . , m, are known for the given event B, then the Bayes theorem allows us to determine the posterior probabilities (when the event B has been observed) pi ei (1) , i = 1, 2, . . . , m. pi = Pr[Ai | B] = ∑ j p je j Use these revised probabilities as new prior probabilities of hypotheses A1 , A2 , . . . , Am , to (2) (n) find pi = Pr[Ai | B occurs twice ]. By repeating this process, determine pi = Pr[Ai | B (n) (n) occurred n times ]. Obtain the limiting behavior of pi as n → ∞ from the formula for pi .

4.5 Convolution In this section, we show that the calculation of the distribution of a sum of independent, nonnegative, integer-valued random variables leads us to an operation we visited before— convolution formula. When random variables are not independent, it is still possible to obtain an explicit formula for their sums, but only when a joint probability mass function is available. Suppose we have two independent discrete random variables X and Y that take values on the set of non-negative integers, N. Let W = X + Y be their sum. Then the set of values taken by W is N as well (because N is closed under addition; however, W does not take all values from N). We ask about Pr[W = k] and use the theorem of total probability. Randomization on X provides: Pr[W = k] =

∑ Pr[W = k | X = i] Pr[X = i] = ∑ Pr[X +Y = k | X = i] Pr[X = i]

i∈N

i

4.5. CONVOLUTION =

175

∑ Pr[Y = k − i | X = i] Pr[X = i]. i

Using Eq. (3.33) on page 123 and the claim of independence of X and Y , we find Pr[Y = k − i | X = i] = Pr[Y = k − i]

=⇒

Pr[W = k] = ∑ Pr[X = i] Pr[Y = k − i]. i

Let pX and pY be probability mass functions of the discrete random variables X and Y , respectively. Then the above sum can be written as pW (k) = Pr[W = k] = Pr[X +Y = k] = ∑ pX (i) pY (k − i) = pX ⋆ pY ,

(4.32)

i

where pX ⋆ pY is the convolution of two sequences of numbers defined by Eq. (2.73), page 66. Now we consider two independent continuous random variables, X and Y , having densities fX (x) and fY (y). For any real number t, whether it is a possible value of X + Y or not, the cumulative distribution function of W = X +Y equals FW (t) = Pr[X +Y < t]. This probability is evaluated as the integral of the joint density of X and Y over the region x + y 6 t: FW (t) =

Z ∞

−∞

dx

Z t−x −∞

dy fX (x) fY (y) =

Z ∞

−∞

fX (x) dx

Z t−x −∞

fY (y) dy.

We can also express FW (t) via continuous cumulative distributions functions: FX (x) = Pr[X 6 x] =

Z x

−∞

fX (x) dx,

FY (y) = Pr[Y 6 y] =

Z y

−∞

fY (y) dy.

Then the distribution function for their sum becomes FW (t) = Pr[X +Y 6 t] =

Z ∞

−∞

fX (x) FY (t − x) dx =

Z ∞

−∞

FY (t − x) dFX (x),

(4.33)

where dFX (x) = fX (x) dx. The integral in the right-hand side is denoted by FX ⋆ FY = FY ⋆ FX and is called the convolution of the two distributions. Therefore, the cumulative distribution function FW of the sum of two independent random variables X and Y is the convolution of their distribution functions FX and FY . Differentiating both sides of Eq. (4.33) with respect to t (which is valid under certain conditions that are assumed to be satisfied; it is sufficient to assume that fX and fY are continuous), we get the density function: fW (t) = FW′ (t) =

Z ∞

−∞

fX (x) FY (t − x) dx = fX ⋆ fY .

(4.34)

The integral in Eq. (4.34) is called the convolution of two density functions fX and fY . The following definition generalizes this observation to n random variables.


176

Definition 4.116 Let X1 , X2 , . . . , Xn be independent random variables (for some fixed n). The distribution of their sum Wn = X1 + X2 + · · · + Xn , is called n-fold convolution of the distributions of X1 , X2 , . . . , Xn . ⊳ Example 4.117 (Convolution of fair dice) Let an experiment consist of rolling two fair dice and denote by X and Y the numbers that appear on the upturned faces. The fairness assumption means that when a die is rolled, each element of the sample space { 1, 2, 3, 4, 5, 6 } occurs with probability 1/6. Therefore, we have a uniform distributions with the PMF, pX (x) = pY (y) = 1/6,

x, y ∈ [1..6],

Let us calculate the PMF of their convolution W = X + Y . Using independence of X and Y and the theorem of total probability, we have 6

Pr[W = k] = ∑ Pr[X = i] Pr[Y = k − i] = i=1

Obviously, Pr[Y = k − i] =

(

1 6,

0,

1 6

6

∑ Pr[Y = k − i].

i=1

when 1 6 k − i 6 6, otherwise.

Solving the inequalities 1 6 k − i 6 6, we obtain k − 6 6 i 6 k − 1. Hence Pr[W = k] =

1 6

6 i=1

For 2 6 k 6 7 we have max{1, k − 6} = 1 and Pr[W = k] =

1 1 × 6 6

k−1

∑1=

i=1

1 36

6

∑

1=

i=k−6

1 . 6

∑ i=max{1,k−6}

min{6, k − 1} = k − 1. Therefore k−1 , 36

For 8 6 k 6 12 we have max{1, k − 6} = k − 6 and Pr[W = k] =

min{6,k−1}

1

∑ Pr[Y = k − i] = 6

2 6 k 6 7.

min{6, k − 1} = 6. Then

1 13 − k (6 − k + 6 + 1) = , 36 36

8 6 k 6 12.

Thus, the probability mass function of the convolution W = X +Y has a “triangular” shape: k Pr[W = k]

2

3

4

5

6

7

8

9

10

11

12

1 36

2 36

3 36

4 36

5 36

6 36

5 36

4 36

3 36

2 36

1 36

Exercise 4.118 [3] Generalize the last example to dice with n equiprobable faces, and compute E[W ] as well. Exercise 4.119 [3] In Example 4.117 we saw that the sum rolled with two dice is not uniformly distributed. Show that for any biased two dice, the three probabilities of the sum being 2, 7, or 12, cannot be equal (with any probability distribution of their faces). The general case will be considered in Example 6.164 (page 338).

4.5. CONVOLUTION

177

Example 4.120 (Convolution of uniform distributions) Suppose you have access to a pseudo-random generator that, when invoked, produces a real number from the interval [0, 1] equiprobably. In n calls you get the numbers X1 , X2 , . . . , Xn . Let W = X1 + X2 + · · · + Xn be their sum. Of course, X1 , X2 , . . . , Xn are independent and identically distributed uniform random variables. We want to find the cumulative distribution function FW (t) ≡ Pr[W < t] = Pr[X1 + X2 + · · · + Xn < t]. Recall from multivariate calculus that Pr[W < t] is just the hyper-volume in n-dimensional space bounded by hyperplane x1 + · · · + xn = t. So when t 6 1, we have Pr[W < t] =

Z t 0

dx1

Z t−x1 0

dx2 · · ·

Z t−x1 −...−xn−1 0

dxn =

tn . n!

(4.35)

This equation can be proved by induction. Note that this is only the leftmost tail of the convolution. When t > 1, the exact formula for the cumulative distribution function becomes tricky. Each Xk , k = 1, 2, . . . , n, is uniformly distributed on the interval [0, 1], hence its distribution ( t, if 0 6 t 6 1, function is F(t) = Pr[Xk 6 t] = 1, if t > 1. For n = 2 and 1 < t 6 2 we have W = X1 + X2 and its distribution becomes ( Z 1 Z 1 1, if t − x > 1, dx dx F(t − x) = FW (t) = Pr[X1 + X2 6 t] = t − x, if 0 6 t − x 6 1. −∞ 0 Performing integration, we get FW (t) = 12 4t − 2 − t 2 = 21 ∑1j=0 (−1) j 2j (t − j)2 when 1 < t 6 2. The general case is considered in Example 8.102, page 487. Exercise 4.121 [3] Let us consider the random sum S = ∑Ni=1 Xi , where Xi are independently and identically distributed uniform random variables on the interval [0, 1] ⊂ R. The value of the random variable N is the first number of terms such that S > k, where k is any positive integer. What is the average number of terms required to first achieve S > k, that is, what is E[N]? Note that a similar problem is discussed in Exercise 5.86, page 233. Exercise 4.122 [1] Let X and Y be two independent, identically distributed geometric random variables on the positive integers: Pr[X = i] = Pr[Y = i] = pqi−1 , i > 1. Calculate the PMF and expected value of their sum. Exercise 4.123 [3] Again we look at the sum of two independent, identically distributed random variables, X and Y . Each has a distribution determined by flipping a fair coin until heads appears, or the coin is flipped ten times. The random variables get the value zero if heads does not appear, or the number of the flip in which heads was achieved. Show that Pr[X = i] = (1/2)i for 1 6 i 6 10, and Pr[X = 0] = a. Use normalization to determine the value of a. Compute the probability mass function and expectation of the sum W = X +Y . Exercise 4.124 [2] What is the distribution of the sum of two independent random variables that have the negative-binomial distribution, with the same parameter p, but their rparameters need not be the same: denote the random variables by ra and rb . Exercise 4.125 [2−] We have been mostly looking at convolutions of two random variables. As the definition of convolution suggests, there is no difficulty in going beyond two, and we


178

could consider the convolution of a larger number of random variables. As a rule, the results look complicated and are not informative. However, most rules have exceptions, and this rule is not uncommon. Show that for independent random variables having Poisson distributions (not necessarily with the same parameters), the sum of any number of such random variables is a random variable with a distribution of the same type. As Exercise 4.124 shows, negative-binomial distributions with the same parameter p have this property as well. Exercise 4.126 [2] Find the convolution of two independent, identically distributed Engset random variables, X and Y , having the following probability distributions: −m m −n n k ρ , Pr[Y = k] = (1 + ρ ) ρ k , k = 0, 1, . . . , m or n. Pr[X = k] = (1 + ρ ) k k Quite often we need to find the convolution of many random variables, not just two. Here is an example. Example 4.127 (Random walk) Let X1 , X2 , . . ., Xn be a sequence of independent, identically distributed random variables that take the values 1 and −1 with constant probabilities p and q = 1 − p, respectively. We use them to construct the path of a walker moving randomly on the integers, starting at zero, using one variable per time unit, to decide the next step. At every integer time n, the walker moves one step to the right (if Xn = 1) with probability p or one step to the left (if Xn = −1) with probability q = 1 − p. The position of the walker at time n is denoted by Wn : Wn = X1 + X2 + · · · + Xn ,

W0 = 0.

This describes what is usually called a random walk, an example of a stochastic process (see §9.5). The one-dimensional random walk can be visualized as a northeastern lattice path (see Example 2.78 on page 51) by drawing a horizontal unit segment for each step to the right and a vertical unit segment for each step to the left. Where does the walker expect to be after n steps? To answer this question we calculate the probability Pn,k = Pr[Wn = k] that the walker is at position k after n steps. Since the walker always moves from an even integer to an odd integer or from an odd integer to an even integer, we can conclude that Pn,k = 0 if n and k have different parities. We begin with the mean and the variance of the random variable Wn . Observe that E[Xi ] = 1 × p − 1 × q = p − q, E[Xi2 ] = 1 × p + 1 × q = 1, and V [Xi ] = 1 − (p − q)2 = 4pq. Hence E[Wn ] = n(p − q), and the variance is additive as well for the independent components, so V [Wn ] = 4npq. Alternatively, E[Wn2 ] = E[(X1 + X2 + · · · + Xn)2 ] =

n

∑ E[X j2] + ∑ E[XiX j ] = nE[X12] + n(n − 1) (E[X1])2

j=1

i6= j

2

= n + n(n − 1)(p − q) ,

V [Wn ] = n + n(n − 1)(p − q)2 − n2 (p − q)2 = n(1 − (p − q)2 ) = 4npq. We use the way of constructing the binomial random variable in §4.1.1 to determine the probability: n+k n−k n (4.36) Pn,k (p, q) = Pr[Wn = k] = n+k p 2 q 2 . 2

4.5. CONVOLUTION

179

This equation follows from the observation that for the walker to be in position k after n steps, it must have done (n + k)/2 steps to the right, and (n − k)/2 steps to the left. (The parity argument of the binomial coefficient comes back to tell us that we need not worry about the fractions, (n + k)/2, not becoming integers.) In particular, 2n n n P2n,0 (p, q) = Pr[W2n = 0] = p q . n For p = q = 1/2, P2n,0 12 , 21 = 41n 2n n → 0 as n → ∞ (see Exercise 2.150 on page 78). It is possible to give other interpretations to such walks. Suppose you toss a fair coin repeatedly and at the end you want to know whether the number of heads is the same as the number of tails. On Cartesian plane along lattice points with integer values, this game can be represented graphically as a zigzag path: each northeast move with step (1, 1), upstep ր, denotes a heads, each south-east move with step (1, −1), downstep ց, denotes a tails. We can encode each rise by the letter U (for up, i.e., northeast) and each fall by the letter D (for down, i.e., southeast). Then each such walk is a binary string or letters U and D. In particular, a Dyck n-path is a lattice path of n upsteps (1, 1) and n downsteps (1, −1) that starts at the origin and never goes below the abscissa (which means that it ends at (2n, 0)). The number n is referred to as the semi-length of the path. Exercise 4.128 [1] Show that (Wn + n)/2 is binomially distributed with parameters n and p, where Wn is a random walk on the line as defined in Eq. (4.36). Exercise 4.129 [2] Using Stirling formula (2.90) on page 77, find approximation of the probability (4.36) as n → ∞. Consider two cases when p = q = 1/2 and when p 6= q. To simplify calculations, pick even integers: n = 2r and k = 2 j. Exercise 4.130 [3] On a windy night, a man begins walking from the origin of a twodimensional coordinate system. Each step is 1 unit in length and is random in the following way. With probability 1/6 he takes a step east, with probability 1/4 he takes a step north, with probability 1/4 he takes a step south, and with probability 1/3 he takes a step west. (a) What is the probability that after 12 steps he will be no more than 5 units away (using the Manhattan metric: the distance between two points is simply the coordinate difference) from the origin? Suggestion: Write a program to generate and count the possible sequences of steps for this problem. (b) Find the probability distribution of his position after n steps in the general case: let the probabilities to go horizontally be p (going right) and q (going left) and probabilities to go vertically be r (going up) and s (going down). Exercise 4.131 [1] Let Y1 , Y2 , . . . , Yn be independent random variables on a given probability space taking only the values 0 and 1 according to the same Bernoulli law: Pr[Yi = 1] = p and Pr[Yi = 0] = q = 1 − p, i = 1, 2, . . . , n. Let Sn = Y1 + · · · +Yn be the associated (n-step) random walk. How are the expected value, E[Sn ], and its variance, V [Sn ], related to those found in Example 4.127? Example 4.132 (Hashing) In Example 4.21 on page 142, we showed that the number of probes needed to insert a record into a hash table is a geometric random variable with parameter 1 − α (0 < α < 1). Suppose that each probe takes 10−6 seconds and the load factor α = 3/4. We also assume that the table is so large that adding a small number of records does not change α significantly.


180

Suppose we need to insert 20 new records and let T be the time required to do this job. Then T is 10−6 N, where N is the number of probes. Since each probe requires calculation of a hash function (which take a constant time, say 1) in addition to the attempt for key insertion, the random variable N is 1 plus a geometric random variable, that is G(p). Then 1 we get N = G {z· · · + G} = G(20, p), where p = 1 − α = /4 and G(20, p) is the negative| +G+ 20

binomial random variable. From Exercise 4.41, we know that the mean of G(20, p) is 80/3. Then E[N] = 80/3 and E[T ] = 83 × 10−5 ≈ 2.666 × 10−5 sec.

Example 4.133 (adapted from D. Darling [33]) Let X be a random variable uniformly distributed in the interval (0, 1), so X = X1 is a real number from 0 to 1. Let L1 = (0, X1 ) be the interval of length X1 . Let X2 be a random number chosen uniformly from the interval L1 and let L2 = (0, X2 ). Repeating this, we form the sequence of intervals Lk = (0, Xk ), where Xk is a uniform random variable from 0 to Xk−1 . Define the random sum W = ∑∞ k=1 Xk . What is the probability distribution of W ? To answer this question, we transform the required sum W as follows: let Yi be the sequence of independent, identically uniformly distributed in the interval (0, 1) random variables. Then W = Y1 +Y1Y2 +Y1Y2Y3 + · · · = Y1 (1 +Y2 +Y2Y3 + · · · ) = Y1 (1 +W ′ ), where W ′ has the same distribution as W , and W ′ and Y1 are independent. The cumulative distribution function of the random variable W is F(x) = Pr[W 6 x] = Pr[Y1 (1 +W ′ ) 6 x] = E Pr[Y1 (1 +W ′ ) 6 x] | Y1 Z 1 Z min{x,1} x ′ = Pr[y(1 +W ) 6 x]dy = Pr W 6 − 1 dy y 0 0 Z ∞ Z min{x,1} F(t) dt x − 1 dy = x . F = y (1 + t)2 max{0,x−1} 0 This shows that F(x) = Cx is a linear function on the closed interval [0, 1], where C = F(1) = R ∞ F(t) dt 0 (1+t)2 . The integral equation F(x) = x

Z ∞

F(t) dt 2 max{0,x−1} (1 + t)

allows us to determine the function F(x) successively in intervals [1, 2], [2, 3], and so forth. For example, if x ∈ [1, 2], then the integral in the right-hand side can be broken as follows: Z ∞ F(t) dt x−1

(1 + t)2

=

Z 1

x−1

+

Z ∞ 1

=

Z 1

x−1

+

Z ∞ 0

−

Z 1 0

=

Z ∞ 0

−

Z x−1 F(t) dt 0

(1 + t)2

= F(1)−

Z x−1 Ct dt 0

(1 + t)2

Since F(1) = C and the antiderivative of t(1 + t)−2 is (1 + t)−1 + ln(1 + t), we get 1 F(x) = xC 1 − − ln x + 1 = C(2x − 1 − x ln x), if 1 6 x 6 2. x R x−1 t lnt In a similar way we determine F(x) = C 3x − 3 + x ln 2 − 2x ln x + x 1 (1+t)2 dt for x ∈ [2, 3]. The analytic expression of the cumulative distribution function can be obtained recursively on each interval of unit length with integer endpoints.

.

4.5. CONVOLUTION

181

We compute the value of F(1) by applying the Laplace transform to the function F(x):

φ (λ ) =

Z ∞ 0

h

i h i h ′ e−λ t dF(t) = E e−λ W = E e−λ Y1 (1+W ) −λ Y1

= E e

−λ Y1W

E e

i

| Y1 =

Z 1

−λ y

e

0

1 φ (λ y) dy = λ

Z λ 0

e−t φ (t) dt.

Hence φ (λ ) is the solution of the differential equation ddλ [λ φ (λ )] = e−λ φ (λ ), subject to the initial condition φ (0) = 1. Since this equation can be rewritten as φ ′ (λ )/φ (λ ) = (e−λ − 1 /λ , it is a logarithmic derivative, and its solution is given by Zλ 1 − e−t dt . φ (λ ) = exp − t 0 Exercise 11.74 on page 660 shows that the integral converges to γ = 0.5772156649 . . ., the Euler constant. Since ln(1 + λ ) = ln λ + o(1) as λ → ∞, it follows that λ φ (λ ) → e−γ as λ → ∞. Then we obtain the value F(1) = e−γ , which follows from the Tauberian theorem 11.55 (see also [136]): If F(t) is a nondecreasing function and its Laplace transform, R ∞ −λ t dF(t) → A/λ k as λ → ∞, then F(t) → At k /Γ(k + 1) as t → 0. 0 e

Example 4.134 (Euclidean distance in the plane) Consider a square of side N in which n points are placed at random. Specifically, the 2n coordinates of the points are independent and identically distributed according to a continuous, uniform distribution. √ We calculate E[D], the expected number of point-pairs closer than d, for some d ∈ [0, N 2], using the Euclidean distance metric. Let the points be denoted by ai , 1 6 i 6 n. The coordinates of ai are Xi and Yi . Define Xi j to be 1 or 0 if the points ai and a j are closer than d or not. Then D = ∑i< j Xi j . While the Xi j are dependent, their expectations can be written to provide us with the needed answer: E[D] = ∑ E[Xi j ] = ∑ Pr[Xi j = 1] = i< j

i< j

n(n − 1) Pr[|a1 − a2 | < d], 2

(Expectation)

since all distances have the same marginal distribution. Now q Pr[X12 = 1] = Pr[ (X1 − X2 )2 + (Y1 −Y2 )2 < d] = Pr[(X1 − X2)2 + (Y1 −Y2 )2 < d 2 ]. def

Let V = (X1 − X2 )2 , and the (squared) distance is a convolution of V with itself. We first derive the distribution of V and then convolve it. We condition on X1 , which has the uniform distribution with density 1/N: Pr[V 6 x] =

1 N

Z N

u=0

Pr[(u − X2 )2 < x] du =

1 N

Z N

u=0

Pr[|u − X2 |
N two things change. We need to replace not only the upper limit of the integral, but also the lower limit. As long as Vx < d 2 − N 2 it is guaranteed that the inequality for the sum of Vx + Vy is satisfied. The probability that Vx < d 2 − N 2 is given by Eq. (4.37) i h √ 2 2 . In addition, we have to integrate the same integrand over the as N22 N d 2 − N 2 − d −N 2 interval u ∈ (d 2 − N 2 , N); it can be done manually, but MAPLE is quicker to produce the integral value ! ! √ N 2 2 N3 p 2 1 N 4 d4 d2 − N 2 N 2d 2 Nd − − . arcsin − arcsin − + d − N2 − 2 d d 2 3 6 6 8

p √ Since we have the general relation arcsin x − arcsin y = arcsin x 1 − y2 − y 1 − x2 , the last result simplifies slightly to 2 2N − d 2 2 2 N3 p 2 1 N 4 d4 N 2d 2 Nd − − (d > N). − arcsin + d − N2 − 2 2 d 2 3 6 6 8 Exercise 4.135 [3] Repeat the calculation done in Example 4.134 for the Manhattan metric: the distance between two points is simply the coordinate difference: a1 = (X1 ,Y1 ),

a2 = (X2 ,Y2 )

=⇒

|a1 − a2 | = |X1 − X2 | + |Y1 −Y2 |.

4.6. ORDER STATISTICS

183

4.6 Order Statistics This section is concerned with the values seen when we order a random collection—usually called a sample—of independent and identically distributed random variables. The order statistics are the elements of the random sample sorted from smallest to the largest. Let X1 , X2 , . . . , Xn be a random sample from some distribution. Let Y1 ,Y2 , . . . ,Yn denote these values when sorted. A common notation is Y j = X( j) . Till now we have used almost exclusively discrete random variables and hardly mentioned continuous ones. We find it more natural to discuss order statistics in the context of continuous random variables. As we shall see, we need even then to use discrete distributions! Let the components of the sample X1 , . . . , Xn be independent, all drawn from the same distribution, having the probability density function f (x) and distribution function F(x). The joint probability density function of X1 , . . . , Xn is the product: fn (x1 , x2 , . . . , xn ) = f (x1 ) f (x2 ) · · · f (xn ). Notice that random variables Y1 , Y2 , . . . , Yn cannot be independent because Y1 6 Y2 6 Y3 6 . . . 6 Yn

by construction.

Let Xmax denote the maximum of the Xi and Xmin be the minimum. Then Xmax = Yn , the n-th order statistic, and Xmin = Y1 , the first order statistic. When n = 2m + 1 is odd, Ym+1 is the median of the sample. The distributions of the random variables Xmax and Xmin have simple expressions. For the maximum to be less than some v, all the Xi have to be less than v. Since they are all independent, we find Fmax (v) = Pr[Xmax 6 v] = Pr[X1 6 v] Pr[X2 6 v] . . . Pr[Xn 6 v] = F n (v). As usual, the density function is obtained by differentiating the distribution; so we get fmax (v) = n f (v)F n−1 (v).

(4.38)

This relation has a useful combinatorial interpretation: the differential fmax (v)dv represents the probability of finding the maximum in an infinitesimal neighborhood dv at v. The probability for this is that one of the Xi is in the neighborhood, f (v)dv, times the probability that all the others are less than v. Finally, there are n ways to choose the maximal sample element. Similarly, Xmin > u if and only if Xi > u for all i. Hence, its cumulative distribution function is Fmin (u) = 1 − Pr[X1 > u, X2 > u, . . . , Xn > u] = 1 − [1 − F(u)]n . The density function of Xmin is then fmin (u) = n f (u)[1 − F(u)]n−1 .

(4.39)

Let us now consider the case n = 2, and compute the joint probability density function for Y1 and Y2 . It is typically easier than to tackle the joint distribution. Let us show why. The event


184

{Y1 6 y1 , Y2 6 y2 } means that both X1 and X2 are less than y2 , and at least one of them is also smaller than y1 . Since there is a symmetry between the two, we find {Y1 6 y1 , Y2 6 y2 } = {X1 6 y1 , X2 6 y2 } ∪ {X1 6 y2 , X2 6 y1 }. However, even though X1 and X2 are independent random variables, these joint events are not independent because of possible intersections. This appears unclear, so let us observe again the density. We shall find the random variables (Y1 ,Y2 ) at (y1 , y2 ) for y1 6 y2 , if the Xi are there at any order. Hence, fY1 ,Y2 (y1 , y2 ) =

(

2 f (y1 ) f (y2 ), if y1 6 y2 , 0, elsewhere.

Here the limitation to y1 6 y2 is natural by construction of the order statistics. It is possibly interesting to try to use this density to compute the joint distribution. Proceeding by definition, we write Pr[Y1 6 y1 ,Y2 6 y2 ] =

Z

u6y1

Z

v6y2

fY1 ,Y2 (u, v) =

Z

u6y1

Z y2

f (u) f (v)dudv,

v>u

where the integral with upper argument that is smaller than its lower one vanishes. Let us assume for simplicity that y1 6 y2 . Then we find Rthat the inner integral can be written as f (u)R (F(y2 ) − F(u)). The final answer is u6y1 f (u) (F(y2 ) − F(u)) du = F(y1 )F(y2 ) − u6y1 f (u)F(u). The lack of closed form is directly related to the difficulty mentioned previously: we cannot express the joint distribution by simple univariate probabilities, even though the Xi are independent.

The same method can be used to find the joint probability density function of any subset of the n order statistics Y1 , . . . , Yn , or all of them: ( n! f (y1 ) f (y2 ) . . . f (yn ), if y1 6 y2 6 . . . 6 yn , fY1 ,...,Yn (y1 , . . . , yn ) = (4.40) 0, elsewhere. While joint distributions are not then easy to get, the marginal ones are easy to write for all the order statistics, not just the extreme values. Exercise 4.136 [2+] (a) Consider a sample of size n drawn independently according to the continuous distribution F(·) (with density f (·)). Use the same reasoning that led to the distributions of the sample minimum and maximum to derive the distribution of the kth order statistic: n n F j (y)(1 − F(y))n− j . (4.41) FYk (y) = ∑ j j=k (b) Differentiate the distribution you obtained in part (a) to find the density n−1 fYk (y) = n f (y)F k−1 (y)(1 − F(y))n−k . k−1

(4.42)

Note that in the process you derive a telescoping series; provide a combinatorial interpretation for Eq. (4.42).


185

Example 4.137 Consider an array with n distinct sorted values. You choose three entries at random. What is the probability that the median of the 3 elements is the kth element of the array? The probability of choosing the kth element first is 1/n because we assume that the elements are chosen uniformly. The probability of choosing the next element to the left of position k is (k − 1)/(n − 1), and the probability of choosing the third to the right of the kth is (n − k)/(n − 2). The required probability is the product of the above probabilities times 3! = 6 since all the permutations of these choices contribute equally: 6

(k − 1)(n − k) . n(n − 1)(n − 2)

Example 4.138 (Order statistics of die rolls) Suppose that we roll a fair die n times and let X1 , . . . , Xn be the values on each toss. From Eq. (4.40), it follows that the joint mass function of their order statistics Y1 , . . . , Yn is ( n!/6n , if y1 6 y2 6 . . . 6 yn , fY1 ,...,Yn (y1 , . . . , yn ) = 0, elsewhere. Let Xmin = Y1 and Xmax = Yn . If X is the value on the face of a rolling die, then Pr[X > y] =

7−y , 6

y Pr[X 6 y] = , 6

y = 1, 2, . . . , 6.

Since X1 , X2 , . . . , Xn are independent and identically distributed random variables, the probability mass functions of Xmin and Xmax have the following values: 7−y n . Pr[Xmin > y] = Pr[y 6 X1 , y 6 X2 , . . . , y 6 Xn ] = Pr[y 6 X1 ] . . . Pr[y 6 Xn ] = 6 y n . Pr[Xmax 6 y] = Pr[y > X1 , y > X2 , . . . , y > Xn ] = 6 Example 4.139 We consider n independent, geometrically distributed random variables with the same probability of success, p. We dress this case by considering n biased identical coins that are tossed simultaneously until heads comes up. Let X denote the number of tosses of a single coin necessary to obtain a head, then Pr[X = k] = pqk−1 , where q = 1 − p is the probability of a tail. Then the distribution function for Xmax = max{X1 , X2 , . . . , Xn } is !n y

Pr[Xmax 6 y] = (Pr[X 6 y])n =

∑ pqk−1

k=1

= (1 − qy )n .

Thus, the probability mass function, g(y), for the maximum tosses required to eliminate all tails is given by

g(y) = Pr[Xmax = y] = Pr[Y 6 y] − Pr[Y 6 y − 1] = (1 − qy )n − (1 − qy−1 )n . If there are two coins, for example, then g(y) = (1 − qy )2 − (1 − qy−1 )2 = q2y 1 − q−2 + 2q−y (q−1 − 1) .

(4.43)


186

Now suppose that after each toss of a group of coins we remove those that come up heads and the remaining coins, which show tails, are tossed again. The process is continued until each of the coins comes up heads. Let Y represent the number of tosses needed to eliminate all tails. How many (group) tosses will be required, on average, to accomplish this task? To answer this question, we use the derived formula (4.43) for the probability mass function and Eq. (3.18) to obtain n E[Y ] = ∑ Pr[Y > k] = ∑ (1 − Pr[Y 6 k]) = ∑ 1 − (1 − qk ) k>1

k>1

k>1

n n n qs n n ks s+1 ks s+1 . = ∑∑ (−1)s+1 (−1) q = ∑ (−1) ∑ q = ∑ 1 − qs s=1 s s=1 s k>1 s=1 s k>1 n

Exercise 4.140 [1+] Let us roll a fair die n times, and let Y1 , Y2 , . . . , Yn be the sample order statistics. What is the smallest n for which Pr[Y1 6 2] > 0.9? Exercise 4.141 [2+] Let Xi , i = 1, 2, 3, be sampled at random from the set [1..n] = {1, 2, . . . , n}. Find Pr[X1 < X2 < X3 ] assuming that sampling is done (a) with replacement (b) without replacement. Hint: See Exercise 3.157. Exercise 4.142 [3+] Let X ∼ G(p) and Y ∼ G(s) be independent, geometrically distributed random variables over the set of nonnegative integers with parameters p and s, respectively. Let U = min(X ,Y ), V = max(X ,Y ) and q = 1 − p, t = 1 − s. (a) Show that ( ps(1 − p)k (1 − s)k , if r = 0, Pr[(U = k, V = k + r)] = k k+r k+r k ps(1 − s) (1 − p) + ps(1 − s) (1 − p) , if r > 0. (b) Using the result of part (a), show that U is geometrically distributed with parameter a = 1 − qt. ( s if r = 0, 1−tq , (c) Show that Pr[Y −U = r] = sp r if r > 0. 1−tq t , This appears similar to a geometric probability mass function, but it is not quite since it places relatively more probability at r = 0. (d) Explain why you would expect U and V − U to be independent. Show it formally from the above results. Exercise 4.143 [3] Three numbers are chosen at random equiprobably without replacement from the set [1..n] = {1, 2, . . . , n}. Let X be their median. For example, X (8, 2, 4) = X (4, 3, 8) = 4. Its probability mass function was found in Example 4.137. (a) Show that E[X ] = (n + 1)/2. (b) Answer the same questions in the general case when x numbers are chosen at random, assuming x to be an odd positive number and x 6 n. Exercise 4.144 [3+] Suppose m numbers are chosen at random equiprobably from the set [1..n] = {1, 2, . . . , n}. Let Xmax be the largest number obtained and Xmin be the smallest. (a) If the sampling is done with replacement, show that Pr[Xmax = k] =

km − (k − 1)m , nm

k = 1, 2, . . . , n.


187

(b) Still assume that the sampling is with replacement, show that n

km . m k=0 n

E[Xmax ] = n − ∑

(c) Now consider sampling without replacement (then m 6 n). Show that . k n k−1 . n Pr[Xmax 6 k] = , Pr[Xmax = k] = . m m m−1 m

(d) Show that

m(n + 1) . m+1 (e) Repeat questions (a) – (d) for the random variable Xmin , the minimum number obtained. E[Xmax ] =

Exercise 4.145 [4] (Continuation of Exercise 4.144) Now you are asked to look at the joint distribution of the maximum and the minimum, Xmax and Xmin , respectively—twice, for sampling m > 2 values with, and without, replacement. (a) Find the joint probability mass function (PMF) of Xmin and Xmax . (b) Calculate Cov(Xmin , Xmax ), using the PMF of part (a). (c) Denote the kth order statistic of the sample by Yk , 1 6 k 6 m. Show that its probability distribution is 1 m m j−1 m − r (k − 1)s (n − k)m−s−r with replacement, Pr[Y j = k] = ∑ r ∑ s nm r=1 s= j−r −1 n k−1 n−k = for sampling without replacement. m j−1 m− j (d) Find the joint distribution for two order statistics Yi and Y j .

Exercise 4.146 [1] Consider the lottery 6/49 (see Example 3.132 on page 127) and let X j = t j+1 − t j ( j = 1, 2, . . . 5) be the individual spacing random variable on the set of all lottery tickets t = (t1 ,t2 , . . . ,t6 ). Let Xmin = min{X1 , X2 , X3 , X4 , X5 } be the minimum spacing. Show that 54 − 5k . 49 Pr[Xmin > k] = , 1 6 k 6 9, 6 6 and derive the expected values, variances, and modes for the random variable Xmin . Note that the case Xmin = 1 corresponds to at least two sequential values in a lottery ticket; its probability is close to one half, Pr[Xmin = 1] = 22483/45402 ≈ 0.495198.

Exercise 4.147 [4] A random number generator produces a sequence of numbers x1 , x2 , . . . from the interval (0, 1) ⊂ R. For any positive integer m, let xN be the first number that is less than exactly m of all its predecessors x1 , x2 , . . . , xN−1 . Let Y1 , Y2 , . . . , YN be the corresponding order statistics Y1 < Y2 < . . . < YN ; then xN = YN−m . Determine the distribution of xN . Hint: Use Eq. (L.20) from Appendix L. Exercise 4.148 [2] Let X1,2 , . . . X1,n , X2,1 , . . . , Xm,n be independent and identically distributed random variables, having the same cumulative distribution function, where 0 6 Xi, j 6 1 for i = 1, . . . , m and j = 1, . . . , n. For δ ∈ (0, 1), define ( 1, if max {Xi,1 , . . . , Xi,n } 6 δ , Yi = 0, otherwise.


188

What is the distribution of Yi ? What is the distribution of the sum, T = ∑m i=1 Yi ?

4.7 Chebyshev Inequalities The inequality named after Pafnuty Chebyshev7 gives an upper bound on the probability of deviations of a random variable from its expected value. Theorem 4.149 (Chebyshev) Let X be a random variable with a finite variance. Then for any positive number a, V [X ] (4.44) Pr [|X − E[X ]| > a] 6 2 . a P ROOF : For simplicity, we prove this theorem only for discrete random variable with mean µ = 0. Calculations show that

V [X ] = E[X 2 ] =

∑ x2i Pr[X = xi ] > ∑ i

>

∑

|xi |>a

x2i Pr[X = xi ]

|xi |>a

a2 Pr[X = xi ] = a2 Pr[|X | > a].

Therefore,

E[X 2 ] . a2 When we need to distinguish between this inequality and a host of other similar ones, we shall call it the Chebyshev-Bienaymè inequality. Pr[|X | > a] 6

In the Bernoulli case, let p be the probability of the occurrence of success in a single trial and let m be the number of occurrences of success in n independent trials. Then the Chebyshev inequality asserts that i h m pq where a > 0 and q = 1 − p. (4.45) Pr − p > a 6 2 , n na Exercise 4.150 [1] Complete the proof of Chebyshev’s theorem for the case where µX 6= 0. Indeed, explain why Eq. (4.44) is invariant under changes of E[X ].

Exercise 4.151 [3+] Let X be any random variable with mean µ and standard deviation n

σ . Show that Pr[|X − µ | > 2σ ] + Pr[|X − µ | > 3σ ] 6 1/4. In general,

∑ Pr[|X | > αk ] 6

k=1

max {k/αk2 }.

16k6n

Theorem 4.152 (One-sided Chebyshev’s Inequality) Let X be a random variable with mean µ and variance σ 2 = V [X ]. Let ε > 0 be a constant. Then Pr[X > µ + ε ] 6 7 Pafnuty

V [X ] , V [X ] + ε 2

Pr[X 6 µ − ε ] 6

V [X ] . V [X ] + ε 2

(4.46)

Lvovich Chebyshev (1821 – 1894) was an influential Russian mathematician in the university of St. Petersburg. The French mathematician Irénée-Jules Bienaymé (1796 – 1878) was the first to circulate this result, but he was publicly acknowledged by P. Chebyshev prior to the publication. Actually there is no unique spelling of his name either in Russian (known two) nor in languages other than Russian. In his papers, he used six different spellings and ‘Tschebyscheff’ was one of most frequently mentioned spellings in Roman letters.

4.7. CHEBYSHEV INEQUALITIES

189

This result is also called the Chebyshev-Cantelli Inequality, and you are asked to prove it in Exercise 4.157. Corollary 4.153 The probability that any random variable X , with an expected value, µ , and a finite variance, σ 2 , will have a value within k standard deviations of the mean is at least 1 − 1/k2 , namely, 1 (4.47) Pr[|X − µ | < kσ ] > 1 − 2 . k Note that, just as equation (4.44) produces a useless bound when a2 6 V [X ], the value of k in this corollary needs to be at least 1! This corollary says, for example, that at least 3/4 of a data set is within two standard deviations of its mean. So the Chebyshev theorem promises that if the data is sampled from a distribution with a small variance (and standard deviation), it will be tightly packed about its mean. If we do not know the variance (or even that it exists), we are reduced to a weaker result, credited to Andrei Markov8 : Corollary 4.154 (Markov Inequality) Let X be a positive random variable with finite expected value, µ < ∞. Then for any a > 0,

Pr[X > a] 6

µ . a

(4.48)

Exercise 4.155 [2] Use the basic inequality (4.44) and its proof method to establish Corollary 4.153, and the Markov’s inequality (4.48), respectively. As we shall see, the bounds that inequality (4.44) produces often are very crude. This is a direct result of its simplicity, or, more precisely, its use of a very weak assumption: we only assume that the second moment exists. Given this limitation, the bound cannot be improved, as the following example shows. Example 4.156 Let X be a random variable that takes three values: a, −a, and 0, with probabilities p, p, and 1 − 2p, respectively, where p is some positive real number less than 1/2. Then E[X ] = ap + (−a)p + 0(1 − 2p) = 0, (of course, symmetry!), and V [X ] = E[X 2] − (E[X ])2 = E[X 2 ] = a2 p + (−a)2 p + 02 (1 − 2p) = 2pa2 . From the Chebyshev inequality (4.44), it follows that Pr[|X − EX | > a] = Pr[|X | > a] 6

V [X ] = 2p. a2

On the other hand, for this random variable, Pr[|X | > a] = Pr[X = a] + Pr[X = −a] = p + p = 2p. We have found a random variable for which the inequality has no slack; hence there is no way to improve it unless we have more information about the distribution. 8A

Russian mathematician Andrei Andreevich Markov (1856 – 1922), of Markov chains fame, was a student of P. Chebyshev.


190

The inequality is often beneficial, although there are many cases when it is entirely useless. This can happen when the distribution of X is flat. For example, in the case of throwing a 35 . If we use the Chebyshev inequality true die, we have found that E[X ] = 7/2, and V [X ] = 12 for Pr[|X − 7/2| > 3], which is of course 0, we get for the upper bound 35/(12 × 9) ≈ 0.324. Then, looking for the bound of Pr[X − E[X ] > 1], which covers the rolls 1,2,5,6, and has a probability of ≈ 0.667, the bound is 35/12 ≈ 2.917 (which is true, but useless). The Chebyshev inequality may be useful if we have very little additional information about X (beyond its two first moments). Much of its effective use is in providing “order of magnitude” results when dealing with large-scale structures. Exercise 4.157 [3+] Prove the Chebyshev-Cantelli Inequality, as given in Theorem 4.152. Guidance: Compared with the proof of Eq. (4.44), this one is indirect. Start by showing the claim that if the function u(x) > 0 for all x, and for some a > 0, also u(x) > a for all x in the interval (s,t), then Pr[s 6 X 6 t] 6 E[u(X )]/a. next pick u(x) = (x + c)2 , for c > 0. Show that if a > 0, then Pr[X − E[X ] > a] 6 E[(X + c)2 ]/ (E[X ] + a + c)2 . After that calculate the value of c that minimizes the last right-hand side, which yields Eq. (4.46). It helps if you start with a random variable that has zero expectation. Also MAPLE helps. Exercise 4.158 [3] Suppose you know that X has a symmetrical PMF about its mean. With this information, the Chebyshev-Bienaymè and Chebyshev-Cantelli Inequalities are equally usable. Which one is stronger? Example 4.159 (Search for a maximum element) Given an array a[0..n − 1] of n numbers of type type, we use the algorithm of Example 3.63 on page 106, rewritten in the form of a C-function to find the largest entry in the array: type max(type ∗a, int n) { type m; int i; m = a[0]; for (i = 1; i < n; i++) if (m < a[i]) m = a[i]; return m; }

Clearly, the calculation involves n − 1 comparisons, but how many assignments are made to m beyond the initial one? The range is 0 to n − 1. It is zero, when a[0] happens to be the maximal element in the array. It is n − 1, when the array is sorted in increasing order. What can be said of its expectation or variability? The expectation was computed in Example 3.63. Here we find its variance and use the Chebyshev inequality for an upper bound on the probability. For example, we may need to know the probability that more than 15 assignments are needed when n = 1000. As usual, we assume that all the n values in the array a are distinct and all their n! permutations are equally likely. Note:

The above assumptions are adequate for many realistic situations. But surely, not for all. It is possible that only r < n distinct values are in the array, and it is possible that due to previous processing, all the permutations are not equally likely. When such is the case, our ability to give precise answers is much reduced. This is due to the fact that now we must randomize both on population of the array and its order. Some answers survive: when r distinct values exist, the maximal number of re-assignments to m is

4.7. CHEBYSHEV INEQUALITIES

191

r − 1. Unlike the simple case, the probability of this event need not be 1/n or 1/r, but depends on the number of occurrences of that minimal value in the array, and possibly on other issues as well.

An assignment is possible for each of the values that i runs in the procedure, 1 to n − 1. Define the n − 1 indicator random variables Xi to be 1 when the corresponding assignment is made, and 0 otherwise. For an assignment to be made, a[i] needs to be the largest among the (i + 1)-long prefix of the array, and the equiprobability assumption implies that Pr[Xi = 1] = 1/(i + 1). Let S be defined as the number of additional assignments made in the loop. Then S = ∑n−1 i=1 Xi . Moreover, these indicator random variables are independent. Let i < j, the ordering of the subarray a[0..i] with the maximal element in the position i does not affect the element a[ j] of being larger than all previous entries in the array a[0.. j]. The expected value of S was n−1 n−1 1 = X calculated in Example 3.63: E[S] = E ∑n−1 ∑ i=1 E[Xi ] = ∑i=1 i+1 = Hn − 1. The i=1 i variance needs more work. Let us go for the second moment using the independence of indicator random variables: #2 " E[S2 ] = E

n−1

∑ Xi

i=1

n−1

=

∑ E[Xi2] + 2 ∑

E[XiX j ].

16i< j 15 − E[S]] = Pr[|S − E[S]| > 8.515]
1 and q < 1? Exercise 4.172 [2] Suppose that in the given array A[1..n] there are k (1 6 k 6 n) indices i such that A[i] = a. What is the expected number of indices into A that must be picked before the key a is found by linear search? Your answer should be a function of n and k. Exercise 4.173 [2] (Improved insertion sort?) The assignment t = a[ j] in line 4 of the insort procedure (Example 3.57 on page 102) is made to avoid one of the three pointer evaluations in line 6 there. However, if a[ j] is in its correct position at that time, no action is needed. To speed up the procedure, the following

4.8. SUNDRY EXAMPLES

195

improvement has been suggested: a[ j] should be first tested against a[ j − 1], and if it is the larger, j can be immediately advanced, also saving the assignment in line 7. If a[ j] needs to be compared further down the line, then the assignment to t is carried out, and all proceeds as before. Perform an evaluation of this improvement. Note: An even better solution is when a compiler is good enough to notice that j does not change during the loop over i, and would keep a[ j] in a register throughout. Exercise 4.174 [2] (Binary insertion sort) The inner loop (line 5) in the insertion sort algorithm presented in Example 3.57 on page 102 consists of finding the correct place to insert a[ j] in the sorted prefix in positions a[0.. j − 1]. This is done by linear search. Consider using instead binary search to locate the position. (a) Such a change cuts the cost of a common linear search, as shown in Example 4.170 from linear to logarithmic, in the length of the list. Why is this not possible here with common processor architectures? (b) Consider making the change described in part (a) when the sorting is performed on a processor that has a move-memory instruction, parameterized with three arguments (a, b, c) which shifts the content of b memory positions, beginning at a, by c positions. That is, positions a + c . . . a + b + c − 1 are rewritten with the content of positions a . . . a + b − 1. Since c would be 1 in this application, the move-memory instruction can be thought as being done in a single time unit. Reevaluate the binary insertion sort in such an architecture. Exercise 4.175 [4] We look at a medium file system, such as in an academic department, or of a software development project. Let N = 106 be the number of user files in the system. We assume that ps = 0.95 is the fraction of “small” files and pl = 0.05 is the fraction of “large” files. The issues of how small is “small” etc. do not matter now. The rates of change of the files in a day depend on their size. It has been estimated that the probability that a specified “small” file is changed in any given day is 10−3 , independently of what happens to all other files. For a “large” file, the rate of change is much smaller: it is 10−4 per day. Define K as the number of files that change in a day. This is clearly a random variable. You may find it convenient to use explicitly the obvious relation K = Ks + Kl , which are the numbers of files changed in the two size classes. (a) Compute the expected values of Ks , Kl , and K. (b) Compute the variances of Ks , Kl , and K. (c) Compute bounds, using the Chebyshev inequality for the following tail probabilities: Pr[K 6 750], Pr[K > 2000]. (d) Compute estimates for the same tail probabilities of part (c), using standard normal √ ∼ N(0, 1), where the symbol N(0, 1) denotes the approximation (4.21) on page 157: K−E[K] V [K]

standard normal random variable (4.19). (e) When a new system administrator was hired, he was worried about the backup procedures used in the system and went into a more detailed measurement of the statistics of the file system, especially the rate of change. He found that the rates hid a great dependence between changes in successive days. In particular, he found that the probability that a file is changed given that it was changed the previous day, is a surprising 0.75. This held for both size classes. Find, for a “small” file that was not changed yesterday, the probability it will be

196


changed today. Suggestion: Use the theorem of total probability, with the two-way partition (was changed yesterday, was not) as a basis. (f) Let the probability you derived in part (e) be called t. Define for a small file the random variable R, the number of days that pass between successive changes. (We only count days with no change; hence if it is changed on two successive days, we say the value of R for that interval was zero.) What is the probability mass function of R in terms of t? What are its first two moments? Using “balls-and-urn” language is surprisingly helpful in modeling very complicated physical processes, as the following example shows. Example 4.176 (Gaseous diffusion). In 1907, the Austrian physicist Paul Ehrenfest (1880 – 1933) and his wife, Russian mathematician and collaborator Tatyana (1876 – 1964), proposed the following model of gaseous diffusion. Imagine 2n balls numbered consecutively from 1 to 2n, distributed in two urns, A and B, so that at the beginning there are n + x, −n 6 x 6 n, balls in urn A. An integer between 1 and 2n is chosen at random (all numbers are assumed to be equiprobable) and corresponding ball is moved from the urn in which it is, to the other urn. This process is then repeated t times and we ask for the probability that after t drawing there should be n + y balls in urn A. This is a simple and convenient model of heat exchange between two isolated bodies at unequal temperatures. The temperatures are symbolized by distinct number of balls in the urns. There is another model of gas diffusion. Suppose two urns, A and B, contain an identical number, n, of white and black balls, respectively (the molecules of two different gases in two sealed chambers, initially separated). At time t = 1, a ball is selected at random from each urn and instantaneously placed in the other urn. This process, “select-and-transfer,” is repeated t times, so that each urn always contains n balls, but only at t = 0 we do know the colors of the balls. We assume that successive choices are independent of each other and of the color of the selected balls. The state of both urns (which is called the system) is completely determined by specifying the number of white balls in urn A. Let P(i, j) be the probability that the system in state i (the urn A contains i white balls) at time t changes to state j (there are j white balls in urn A) by one exchange. If j = i + 1, i < n, then the number of white balls in urn A increases by 1. This means that before the exchange the urn A had i white balls and (n − i) black balls, and urn B had (n − i) white balls. Then we choose a black ball with probability (n − i)/n from urn A and a white ball from urn B with probability (n − i)/n. Since these two events are independent, n−i 2 n−i n−i , if j = i + 1. = P(i, j) = n n n If j = i, then the number of white (and black) balls remains unchanged by the exchange. There are two independent events that preserve the situation. First, a white ball could be selected from each urn and simply swapped or a black ball would be selected from each urn

4.8. SUNDRY EXAMPLES

197

and swapped. Thus, P(i, j) =

2i(n − i) i n−i n−i i , + = n n n n n2

if i = j.

Similarly, 2 i , P(i, j) = n

if

j = i − 1.

Obviously, P(i, j) = 0, otherwise. Exercise 4.177 [3] Consider the following game in which you need to put n (n > 1) balls numbered by 1, 2, . . . , n, in this order, into n cells with the same labels. You throw the first ball randomly into one of the cells equiprobably. After that you put every next ball into appropriate cell with the same label if it is not occupied. If the corresponding cell is occupied, you throw a ball randomly and repeat the process. Show that the probability, pk , that the kth ball will occupy bin #k is n−k+1 n−k+2 . Exercise 4.178 [2] Let X1 , X2 , . . . , Xn be independent Bernoulli random variables with parameter p, taking the values 0 and 1 according to the law: Pr[Xi = 1] = p and Pr[Xi = 0] = q = 1 − p. Let Sn = X1 + · · · + Xn be the associated random walk and X4n= Sn /n be its “empirical” average. Find an upper bound for the fourth moment E (X n − p) . Example 4.179 (Pattern matching) Pattern matching problems arise in many applications. For instance, to maintain frame synchronization in digital communication system, a fixed pattern is inserted at regular intervals into transmitted sequence. Another common example gives retrieve/search for a pattern in data.

Suppose that finite (or semi-infinite) sequence of letters (or digits) is formed by selecting independently, with equal probability d −1 , from the alphabet of d letters. We shall call such sequence T = [t1 ,t2 ,t3 , . . .) a random data stream. A finite data string (or word) will be written as T[i, j) = [ti ,ti+1 , . . . ,t j−1 ] for 0 6 i < j. Furthermore, let P = [p1 , p2 , . . . , pm ] be a fixed pattern of m letters. For every data stream T, we define a random variable X to be the number of positions examined before the pattern is found, that is, X = i if and only if T[i, i + m) = P,

T[ j, j + m) 6= P

for 1 6 j < i.

Simple observation shows that the conditional probability of X given that m-digit pattern occurs in the first positions of the word T is E[X | T[0, m) = P] = d m because of the assumption that all letters are uniformly distributed in a word. So on average it will take d trials to check the first digit (or letter) in the word. Exercise 4.180 [3] (Pattern matching [32]) Suppose we are looking for a “pattern,” a string P of m letters in a “text” T of length n (n > m). We use the following na¨ıve algorithm: for s ← 0 to n − m

198

CHAPTER 4. MORE ABOUT PROBABILITY do if P[1..m] = T[s + 1..s + m] then print “Pattern occurs with shift s”

(a) To evaluate the cost of this algorithm, we assume that P is some fixed (arbitrary) string, and the letters in T are randomly chosen from a d-ary alphabet. Find the expected number of letter comparisons made in the implicit loop (second line of the algorithm). Show that this number equals 1 − d −m 6 2(n − m + 1). (n − m + 1) 1 − d −1 Assume that the algorithm stops comparing characters for a given shift once a mismatch is found or the entire pattern is matched. (b) Let us get closer to real life: we still assume T and P are from the same alphabet, and are created using the same mechanism, which differs from what was used in part (a). Languages are not entirely random strings of letters, We assume the language statistics is as follows (the simplest type of a Markov source): • the first letter in the text or the pattern is random; could be any, in probability 1/d; • the probability that the letter following a ∈ Σ is a as well is p. With probability 1 − p it will be a different letter, and then each letter, in the text or the pattern is again random and equally likely: any a′ 6= a appears with probability (1 − p)/(d − 1). This rule can be summarized as follows: let either of the strings P or T be [x0 x1 . . . xi xi+1 . . .], then  1  i = 1,  d Pr[xi = a | xi−1 ] = p i > 1, a = xi−1 ,   1−p d−1 i > 1, a 6= xi−1 . Recalculate the total expected number of letter comparisons.

Theorem 4.181 Let X be a random variable with values from N ≡ {0, 1, 2, . . .}, and let n be a positive integer. The following two conditions are equivalent: 1. the first n factorial moments of X are identically equal to 1; 2. the kth moment of X equals the kth Bell number, Eq. (C.4) on page 708, for every 1 6 k 6 n. Since the distribution (4.15) is uniquely determined by its moments, the Poisson distribution with parameter λ = 1 is the unique distribution whose nth moment equals the nth Bell number, Bn , for every n. Note: Exercise 8.89 on page 479 asks to prove the first property using the principle of inclusion and exclusion.

Chapter 5

Recurrences or Difference Equations Sequences, also known as functions of a discrete variable, appear in many applications of computer science, in numerical analysis, and, naturally, in discrete mathematics. Many of these applications are rooted in practical, engineering problems, where systems have a finite or denumerable state space. Other such scenarios arise when systems change their state at discrete instants of time only. A recurrence is an equation that describes the evolution or behavior of such a function. In this chapter, we present many such equations, and solve only a few special types. More solutions will be introduced and derived in subsequent chapters. A major application that relies on recurrences is the derivation of numerical approximation schemes for differential equations, obtained by a discretization of the continuous problem space; this scenario is not our objective here. However, a special section (§5.6) is dedicated to some numerical approximations. In general, a recurrence is used to find out the properties of a sequence, numeric or symbolic. In our applications, a sequence almost always is a function defined on a set of integers, typically, N = {0, 1, 2, . . .} with values in the set of real (or complex) numbers. Traditionally, terms of sequences are denoted by subscripted symbols such as an rather than a(n). There will 2 be exceptions, for example, when the subscripts become complex; thus we prefer A(⌈ n2 ⌉) to A⌈ n2 ⌉ . A sequence is usually denoted by {an }∞ n=0 or {an }n>0 or simply {an }, when the range 2 is obvious. If a sequence of interest is finite, it is assumed to be embedded into an infinite one typically padded with zeroes. Infinite sequences also exist outside analysis of algorithms. For example, they arise in communication whenever we consider a signal that is sampled at discrete times. A signal might be electrical, mechanical, or optical. Control systems for the space shuttle, modern aircrafts, or any complex system now use discrete or “digital” signals. The clear sounds from a compact disc player are produced from music sampled at the rate of 44,100 times per second. At each measurement, the amplitude of the music signal is recorded as a number, say, xk . This sequence {xk } contains enough information to reproduce the music with high fidelity.

199

CHAPTER 5. RECURRENCES OR DIFFERENCE EQUATIONS

200

5.1 How Do Difference Equations Arise? We answer this question by examples—seeing is believing. Example 5.1 The first one is rather far from computer science—it is the loan amortization problem, where a loan, both principal and interest, must be paid in a number of equal monthly installments. We need some notation. Let A m r pn

be the amount of the principal borrowed, be the amount of each monthly payment, a constant, be monthly interest rate, and be the outstanding loan amount at the end of month n.

Each installment is applied to the interest accrued during the past month on the debt, r pn , and the rest retires part of the principal. We have pn+1 = pn − (m − r pn ) = (1 + r)pn − m,

n > 0,

with the initial condition

p0 = A.

Together, they constitute an initial-value problem for the first-order constant coefficient linear difference equation or recurrence. Example 5.2 The next example comes closer to computer science: we consider sheep and wolves. Sheep, left alone, increase at an average annual rate1 of α . Let sn be their number at the beginning of year n, then a year later their expected number would be sn+1 = (1 + α )sn . However, a population, wn , of wolves leads to a reduction of β wn sn sheep, where β is the rate at which wolves eliminate sheep. Sheep-less wolves languish and dwindle at an annual rate of γ , but with sn sheep around to feed on there is also an expected increase of δ wn sn . As a result, we have the following double system of nonlinear recurrences, for the expected annual population evolution: sn+1 = sn + α sn − β sn wn ,

wn+1 = wn − γ wn + δ sn wn ,

with given s0 and w0 .

These models used to be popular in modeling ecological niches, trying to find which sets of parameters and initial populations lead to equilibrium (in steady state, or cyclically varying) populations. n n (−1)k+1 , for n ∈ Z+ . Is there a simple expresExample 5.3 Consider the sum an = ∑ k k k=1 sion for its value? Later, we show general methods that are effective for such sums, but none of those we have seen so far are directly effective. The following idea is sometimes useful. def

Every sum Sn = ∑nk=0 fk satisfies the obvious recurrence Sn = Sn−1 + fn . How can we use it here? Since we do not know an , we cannot write this recurrence directly, but we can use it indirectly: we apply the addition formula (2.62) to the binomial coefficient in the definition of an and find n n n − 1 (−1)k+1 n−1 n − 1 (−1)k+1 = an−1 + ∑ . an = ∑ + k k k k−1 k=1 k − 1 k=1 1 The

notion that a population of living organisms increases at a constant rate was the basis of Essay on the Principle of Population by T. R. Malthus, published initially in 1798, and destined to leave a great impression. This “Malthus law” only holds in the absence of any resource constraints and requires certain assumptions, such that aggression within the group is insignificant.

5.1. HOW DO DIFFERENCE EQUATIONS ARISE? The binomial coefficient

n−1 k−1

can be written as

k n n k ,

201 which gives us the relation

1 n n n (−1)k = an−1 − ∑ (−1)k . an = an−1 − ∑ n n k k k=1 k=1 n

Using the binomial formula (2.54) on page 56, we evaluate the sum: n n n n k ∑ k (−1) = ∑ k (−1)k − 1 = (1 − 1)n − 1 = −1. k=0 k=1 Hence an = an−1 + 1n . Comparing an with the recurrence for the sum Sn , we see that an = ∑nk=1 1k , which has been defined as the nth harmonic number. Hence we have found the summation formula n n (−1)k+1 = Hn n ∈ Z+ . (5.1) ∑ k k k=1 Example 5.4 Another example gives the recursive definition for the factorial function: n! = n(n − 1)!,

with the initial values 1! = 0! = 1.

Writing n as n = n − 1 + 1 and the factorial as (n − 1)! = (n − 1)(n − 2)!, we obtain another recurrence n! = n(n − 1)! = (n − 1)(n − 1)! + (n − 1)! = (n − 1) [(n − 1)! + (n − 2)!] . R

The Γ-function, defined as Γ(x) = 0∞ t x−1 e−t dt, is a generalization of the factorial function. It satisfies the same recurrence: Γ(x + 1) = (x − 1)[Γ(x) + Γ(x − 1)]. Remark: Recurrences may be viewed in two ways. According to the first view it is a functional equation. Solving it means deriving an explicit expression for the value of the function at each point. The solution is proved correct by substituting it into the equation and showing that it produces an identity. The other view is prescriptive, which sees it as a computational scheme: given the initial values, we can calculate directly, numerically or symbolically, any number of successive terms. Note that while any definition of a function in terms of itself is recursive, a valid computational scheme requires that a calculation as described can actually be carried out. For example, it is a simple matter to see that the recurrence f (n) = f (⌊n/2⌋) + 1, together with the initial value f (0) = 3 generates the sequence of values 3, 4, 5, 5, 6, 6, 6, 6, 7, . . ., but the recurrence f (n) = f (⌊n/2⌋ + 1) + 1, with the same initial value f (0) = 3 (or any other value) cannot be similarly used; in fact, it is inconsistent for n = 1. In computer science, difference equations arise in our programs when either iteration or recursion is used (we discussed the two approaches briefly in §1.3). Iteration, as in the previous examples—and the next one—leads to an equation expressing a sequence element in terms of one (or, in some cases, a few) previous elements. Example 5.5 (S. Selkow) The following code is the central segment of a procedure, written in C, designed to find the two largest elements in the array A of n distinct integers, and return them in the variables first and second:

202


if (A[0] < A[1]) { first= A[1]; second= A[0];} else { first= A[0]; second= A[1];} for ( i = 2; i < n; i++ ) {if (second < A[i]) if (first < A[i]) {second = first; first = A[i];} else second = A[i]; } We are looking at the term comparisons this segment performs. We distinguish comparisons between elements of the array, from comparisons of the loop control variable, i here. We want to count the number of term comparisons: this is our cost unit here. Since the action of this code fragment does not depend on the values of the entries in A, but only on their order, then for the purpose of reasoning about the code we can assume that the entries are integers, [1..n], and that all n! permutations are equally likely. So the variables first and second always return the values n − 1 and n, respectively. There are three term comparisons in the code: A[0] 2) of the elements in the array must be in one of the wk worst-case permutations of k elements. As the iteration continues, the variables first and second ‘find’ and keep the largest values on the way. When the index i reaches its last value, n-1, the condition second < A[n-1] must be satisfied to ensure the maximum value of D. Since second has the next-to-largest number found so far, it must be either n or n-1 (and then first is either n-1 or n). This fact, and the knowledge that A[0..n-2] is in one of the wn−1 worst arrangements, allow us to write the recurrence wn = 2wn−1 . This difference equation needs an initial value, and we notice that if n = 2, then D = 0, deterministically (this is the best, average and worst case, all at once). Hence we have w2 = 2. By inspection, wn = 2n−1 solves the recurrence, and Pr[D = n − 2] = 2n−1 /n!. For n = 3, the four “bad” permutations are 1,2,3; 1,3,2; 2,1,3; and 3,1,2. To analyze the random variable D beyond its extreme values, we adopt a common technique to represent it as the sum D = ∑n−1 k=2 Dk , where Dk is the indicator random variable of the event

5.1. HOW DO DIFFERENCE EQUATIONS ARISE?

203

{second < A[k]}. Now Dk = 1 in two cases: (1) when A[k] is the largest element in the array A[0..k], or (2) when A[k] is the second largest element so far. In both cases, second is the third-largest element, and each case has the probability 1/(k + 1). Therefore Pr[Dk = 1] = 2/(k + 1). We find the expected value of D to be E[D] = ∑k E[Dk ] = ∑n−1 k=2 2/(k + 1) = 2Hn − 3, where Hn is the nth harmonic number. In order to compute the variance of D, we need to consider the product Di D j (with i < j), and we want to find the probability Pr[Di D j = 1]. This is immediate since the two variables 4 . To see that the claim of independence are independent! Hence, Pr[Di D j = 1] = ( j+1)(i+1) holds, you only need to observe the way we argued previously for the recurrence of wn : whether D j = 1 or not only depends on the possible values of A[ j], and is independent of the permutation that resides to its left, including the fact whether Di = 1 or not. Exercise 5.6 [3] Use the discussion in Example 5.5 to compute the variance of the number of comparisons, which we denoted by n − 1 + D. Assume that all permutations are equally probable. Recursion is the technique used by an algorithm that calls on itself to complete its work. The corresponding recursive structure may be less tidy than in the iterative case, and its solution is usually not easy to obtain. For example, the analysis of a recursive sorting algorithm typically leads to a recurrence relation between Tn , the number of operations needed to sort a list of n elements, and the values of Tk for several, or even all k < n, as we see in §5.5. Example 5.7 Here is an example of a recursive calculation (of the divide-and-conquer type): the inefficient procedure RandP generates a random permutation of a given set S:

RandP(S) 1. if (S = ∅) return (∅); / / Simply return the empty set. 2. Remove a random element from S and assign it to y; 3. create S1 = ∅, S2 = ∅; / / Empty sequences. 4. for each x ∈ S 5. put x in tail of either S1 or S2 in probability 1/2; 6. return (RandP(S1 ), y, RandP(S2 )): As usual, we recommend you to test this procedure on a few small examples to see how it operates. Let N(s) denote the number of calls to RandP when entered to permute a set of size s. This notation carries the observation that N does not depend on the content of S but on its size only. Line 1 says N(0) = 1. For higher values we only look at the mean def n(s) = E[N(s)]. For i = 1, 2, what can we say about the values of the random variable si = |Si |? Each selection in lines 4 & 5 is a binomially distributed random variable (described in §4.1.1) with probability parameter 1/2, but they obviously are not independent since s1 + s2 = s − 1. It −(s−1) is clear that Pr[s1 = k] = Pr[s2 = s − 1 − k] = s−1 2 . k Now we prove by induction on the size s of the array S that the algorithm RandP generates each permutation of the given array S with the same probability, 1/s!. We shall denote the output of operating with RandP on S by R(S). Basis: for s = 1 the claim is obvious. Induction Hypothesis: Assume that the algorithm RandP performs as required for any array of size i less than s: every possible permutation has the same likelihood of being returned, 1/(i!), for 1 6 i 6 s − 1.


204

Induction Step: We consider an arbitrary array S of size s and we need to prove that the algorithm RandP creates each permutation with probability 1/s!. For arbitrary array S of size s, let Y be the random variable denoting the element we pick in line 2, with probability 1/s. When applied to an array S, the procedure RandP results in some array, which we denote by R(S). Let S′ be any permutation of S. What events lead to Pr[S′ = R(S)]? We look for the value of Y , which we denote by y, in S′ , such that S′ = S1′ yS2′ . If y is in the jth position, we need to have the initial call to RandP that creates the sets S1 and S2 with the correct sizes j − 1 and s − j (the size of S1 determines that of S2 ), and put in them the correct terms, the same as in S1′ and S2′ . Finally, we need S1′ = R(S1 ) and S2′ = R(S2 ). The size of each of these sets is smaller than s. Therefore the arrays Sk′ , (k = 1, 2), are produced from the sets Sk (k = 1, 2) by the calls RandP(S1) and RandP(S2 ) with the probabilities 1/( j − 1)! and 1/(s − j)!. The sets S1 , S2 form a partition of the array S \ {y}, and we get a specific allocation of the entries with the probability 2−s+1 since each term has the probability 1/2 of landing in each set. Combining it all together, we find that the probability of the event [S′ = R(S)] is given by Pr[R(S) = S′ ] =

s

∑ Pr[Y = y] Pr[S1 ] Pr[S2 | S1 ] Pr[R(S1 ) = S1′ ] Pr[R(S2 ) = S2′ ]

j=1 s

1 =∑ j=1 s

s−1 s−1 s 1 1 1 1 s−1 1 1 ×1× = = , ∑ 2 ( j − 1)! (s − j)! s(s − 1)! 2 s! j=1 j − 1

as expected. Note that once we know the set S1 , the set S2 is known as well, hence Pr[S2 | S1 ] = 1. Warning note: While this algorithm is correct, it is hugely inefficient, as the following exercise asks you to show. And to convince you to never even think of using this atrocious algorithm, in case you need to generate a random permutation, we give here a reasonable algorithm for the task: randp: assuming the set is in an array S[0..n − 1], then for 0 6 j 6 n − 2 select an element at random, uniformly, from the subarray S[ j..n − 1], and exchange it with S[ j].

Exercise 5.8 [3] For the algorithm RandP presented in Example 5.7, write (a) a recurrence for n(s), the expected value of N(s); (b) a recurrence for E[N 2 (s)]. (c) Change RandP so that the choice in line 5 is for x to go to S1 in probability p, which may be 6= 1/2. What can now be said about the likelihood of getting a particular permutation? Hint: Think about an extreme case: p = 0. Let us look at a few recurrences that are not directly related to programs but count standard combinatorial structures (which in turn determine the performance of certain algorithms). Example 5.9 (Composition count) The first recurrence is for the composition count cn,k , defined as the number of ways to write an arbitrary integer n ∈ N as a sum of k nonnegative ordered integers: n = ∑16i6k ai , where ai ∈ N as well. Each such sum is called a composition of n, and the k terms are called components. For example, c4,3 = 15, beginning with 0,0,4; 0,4,0; 4,0,0 and 12 more which do not have the component 4 (six have the components 0,1,3; three each have the components 0,2,2 and 1,1,2). One can approach the problem in various ways, and the most effective is


205

discussed in §7.3. However, here is one recurrence that relates cn,k to the counts c j,k−1 : n

cn,k = cn,k−1 + cn−1,k−1 + cn−2,k−1 + · · · + c0,k−1 = ∑ ci,k−1 .

(5.2)

i=0

To prove the recurrence (5.2), we break the set of all compositions, cn,k , of the number n with k components into n + 1 disjoint subsets. We visualize the components of a composition as numbers distributed into enumerated cells because the order matters. Choosing, for instance, the last cell, we build disjoint subsets depending on the number in this last cell. If the last component has 0, then all previous components constitute a composition of the number n with k − 1 components, and there are cn,k−1 of them. If the last cell contains 1, then all other numbers in the cells constitute a composition of n − 1 with k − 1 components, and there are cn−1,k−1 of them. An so forth, which leads to the recurrence (5.2). Example 5.10 This example considers partitions. While similar to compositions, they differ as follows: in compositions, the order of the components matters, their number is fixed, and zero components are allowed; thus 4 = 1 + 3 = 3 + 1 = 4 + 0 are three different 2component compositions of 4. However, all these compositions include only one legitimate 2-part partition of 4. Usually, when one considers partitions of an integer n, they have any number of parts, but from Z+ only (zero is not allowed), and their order does not count. Here are all the partitions of 4: 4, 1 + 3, 2 + 2, 1 + 1 + 2, 1 + 1 + 1 + 1. Several partition counting functions have been defined. Here we mention three of them. The first, qk (n), is defined as the number of partitions of n, with parts up to k allowed. For example, q2 (4), which counts partitions of 4 with any number of parts (at most four, naturally) up to 2 in size, equals 3, from the list (2+2, 1+1+2, 1+1+1+1). A closely related count is qˆk (n), the number of partitions where the largest allowed part, k, is used at least once (for example, from the list, qˆ2 (4) = 2, since the partition 1+1+1+1 does not participate). Finally, a different partition count, pr (n), where r specifies the number of parts, but we do not limit the value (size) of the parts. Thus p1 (4) = 1, and p2 (4) = 2. One can continue about the combinatorics of partitions indefinitely; the book [6] by George E. Andrews offers much related material of interest. Later, §7.5 discusses some properties of the partition counts. Exercise 5.11 [3] (a) Justify the following recurrence by considering the partitions of n separated according to the number of times the part k is used: qk (n) = qk−1 (n) + qn−k (k − 1) + qk−1 (n − 2k) + · · · .

(5.3)

What initial values do we need to specify? (b) Write a recurrence for qˆk (n), defined as the number of partitions where the largest allowed part k is used at least once. (c) Prove another recurrence, satisfied by the partition count pr (n) introduced in Example 5.10: pr (n) = pr−1 (n − 1) + pr (n − r). (5.4) Justify it by considering the role of partitions that contain the part 1. (d) Find relations between the counting functions we introduced, as many as you can. Use each to give an expression for p(n), the total number of unrestricted partitions of n.


206

k , n = 1, 2, . . ., satisfies the recurrence relaExercise 5.12 [1] Show that Sn = ∑k>0 n−k tion (1.10) on page 16 for the Fibonacci numbers.

Exercise 5.13 [1] Find recurrences in n satisfied by the truncated factorials. There are two def in this family: the n-falling factorial, xn = x(x − 1)(x − 2) · · · (x − n + 1), and the n-rising def factorial, xn = x(x + 1)(x + 2) · · · (x + n − 1), for an arbitrary number x. Note that (x + n − 1)n = xn . Example 5.14 Suppose we count sequences of flips of a true coin until the second heads appears. Let an be the number of such sequences ending on the n-th flip or sooner. We break this set into two disjoint subsets. One of them includes all the sequences that ended on the (n − 1)st flip or sooner, and there are an−1 such sequences. Then there are the sequences that end on the nth flip—the last flip is of course heads. There should be exactly one more heads in the previous n−1 flips, which can occur in n−1 different ways. Hence an = an−1 +(n−1), with a1 = 0; finally, an = ∑nj=2 ( j − 1) = n(n − 1)/2. Exercise 5.15 [3] A two-sided coin is flipped repeatedly. (a) Let an be the number of n-long flip sequences that have no two adjacent tails. Derive a recurrence for an and find its solution. Hint: Use the Fibonacci numbers. (b) Let bn denote the number of such n-sequences ending with the first occurrence of T T (two consecutive tails) in them. Derive a recurrence for bn . (c) Derive a recurrence for the sequence {cn } that counts n-long sequences of flips ending with m consecutive tails. Example 5.16 Recurrences arise naturally in probability. Let us use a pseudo random number generator to fill a d-long array with numbers from the set [1..n] = {1, 2, . . . , n}, each with probability 1/n to be chosen. We want to find the probability for a particular number, say 1, to be in this array (at least once). This probability is the ratio p=

number of d-arrays from the alphabet [1..n] having at least once 1 M(n, d) = . total number of distinguishable d-arrays nd

To find the number of arrays of length d with entries from [1..n], having 1 in some position— which is denoted by M(n, d)—we break the set of such arrays into the union of two mutually disjoint subsets. The first subset, denoted by A, contains all d-strings starting with 1, and another set, denoted by B, contains all d-words that do not have 1 in the first position. The cardinality of the set A is obviously nd−1 , the number of all strings of length d − 1 from the alphabet [1..n]. Every d-array from the set B has (n − 1) choices for the first position and contains 1 in some other position. Therefore, the cardinality of B is (n − 1)M(n, d − 1) and we obtain the recurrence: M(n, d) = (n − 1)M(n, d − 1) + nd−1 ,

d > 1,

N(1) = 1.

(5.5)

The way to solve such a recurrence is presented in Example 5.45 on page 216. Example 5.17 (Log piles, adapted from [137]) We consider now piles created from logs. For combinatorial purposes, all the logs have the same diameter, and they are stored in piles, as the one shown here. Each log is either in the bottom layer or touches two logs in the layer below it.


207

For this to be a single pile, the bottom layer must be uninterrupted, otherwise we have a (lumber) yard. Higher layers need not obey this rule. Piles in which all layers are continuous, unlike the one above, are block piles. This is the type we wish to count here, and log piles below refer to block log piles. There are several counting functions one can fashion for block piles: f (k) — the total number of piles with k in the bottom layer; f (k, n) — the number of n-log piles with k in the bottom layer; g(n, r) — the number of n-log piles with r layers; g(n) — the total number of n-log piles. Clearly, just specifying the number of layers does not limit the number of possible piles. However, the size of the bottom layer does, and f (k) satisfies a very natural recurrence: the layers above the bottom one in a pile are also piles. If this pile, which begins at the second level, has j logs in its bottom layer, it can be configured in f ( j) different ways, and placed in k − j different positions. The exception is when the bottom layer is the only layer, of which there is a single configuration. Hence we get the recurrence: k−1

f (k) = 1 + ∑ (k − j) f ( j). j=1

This is a full-history linear recurrence (see Definition 5.25) of a type we shall find quite easy to handle. The problem is taken up again in Exercise 6.99 on page 313. Exercise 5.18 [2] Consider n lines (of infinite length) in the plane; no two of which are parallel, and no three intersect in a point. Write recurrence relations, and prove them (usually by induction), for (a) an , the number of intersections of such n lines. (b) rn , the number of regions—both open and closed—of the plane defined by these lines. (c) Now let a fixed number, k, of these n lines be parallel to each other. How does this affect the recurrences you found in parts (a) and (b)? Exercise 5.19 [1] The definition of the Fibonacci numbers can be extended to negative indices when we assume that the recurrence (1.10) on page 16 holds for all indices n ∈ Z. Thus, F1 = F0 +F−1 , which gives the equation 1 = 0+F−1 . Hence F−1 = 1, and similarly, F−2 = −1, etc. Let us use the notation Gn = F−n , n ∈ Z+ . Find a recurrence for the sequence {Gn }n>0 .

Exercise 5.20 [2] (J. Brawner) Suppose that n married couples (2n people) are invited to a bridge party. Bridge partners (two persons) are chosen at random, without regard to gender. Show that the probability, pn , that no one will be paired with his or her spouse satisfies the second order difference equation: pn−2 2n − 2 pn−1 + , n > 2, p0 = 1, p1 = 0. pn = 2n − 1 2n − 3

208


Exercise 5.21 [3] Derive a recurrence relation, including appropriate initial values, for the number of n-digit ternary sequences (using the digits 0, 1, and 2) with each of the following properties: (a) no two consecutive digits are the same; (b) no two consecutive 0’s allowed; (c) the sum of all digits is even. Exercise 5.22 [2+] Let an denote the number of ways a 2 × n rectangle can be covered with 1 × 2 domino pieces. Note that there is an issue of deciding when two coverings are the same. For example, the covering of a rectangle of length 3, is considered the same as , if we permit rotations of the rectangle, but not otherwise. A similar issue exists with reflections. Here you are free to decide which symmetries you want to allow (but you need to state your choice). (a) Find an , 1 6 n 6 4. (b) Find a recurrence providing an in terms of an−1 and an−2 . Exercise 5.23 [2] Consider n independent tosses of a fair coin, and let An (m) be the number of sequences of length n in which the longest run of tails does not exceed m. Show that the elements of the sequence {An (m)} satisfy the following full-history recurrence: m

An (m) =

∑ An− j−1(m)

for

n > m,

j=0

subject to the initial conditions An (m) = 2n when n 6 m. Note that the sequence {An (1)} that contain no two consecutive tails is the (n + 2)nd Fibonacci number. A particular case for n = 10 was considered in Exercise 3.42 on page 98. The length of the longest tails run is approximately log2 n − 1 (its expected value ≈ log2 n − 2/3).

5.2 Properties of Difference Equations Definition 5.24 A recurrence or a difference equation is an equation that relates different elements of a sequence {ak }. A solution of a recurrence is a function of the recurrence index that satisfies the recurrence throughout its range. ⊳ The complete specification of a sequence usually includes a recurrence, a statement about the index range where the recurrence holds, and values for elements which are not determined by the recurrence. As a rule, we shall apply the term solution only to relations that provide the value of an element in terms of elements with smaller index value. Thus we would be reluctant to apply it to an equation of the form an = f (an−1 , an+1 ), when f is such that we cannot solve this equation for an+1 . Definition 5.25 The order of a recurrence is the difference (that cannot be made shorter) between the largest and smallest index values of the sequence elements that appear in the equation. The general form of a recurrence relation of order p is an = f (an−1 , an−2 , . . . , an−p ) for some function f . A recurrence of a finite order is often called a difference equation. A recurrence of the form an = f (an−1 , an−2 , . . . , a0 ) is called a full-history recurrence. ⊳

5.2. PROPERTIES OF DIFFERENCE EQUATIONS

209

The recurrence pn+1 − (1+ r)pn = −m, from Example 5.1 (loan amortization), is a difference equation of the first order. The Fibonacci numbers were defined as the sequence that satisfies the following recurrence of the second order Fn = Fn−1 + Fn−2

for all n > 2, and the initial conditions F0 = 0

and

F1 = 1.

In the definition of the full-history recurrence, an = f (an−1 , an−2 , . . . , a0 ), the function f may not contain some of the values, a0 , . . . , an−1 . For example, a linear recurrence an = ∑n−2 k=1 βk ak is a full-history recurrence, but it does not depend neither on an−1 nor on a0 . It is not hard to n n kn verify that the sum Mn = ∑k>0 k! satisfies the full-history recurrence Mn+1 = ∑ j=0 j M j , provided the initial value M0 = e (the term 00 /0! is taken to be 1). We will see many full-history recurrences in the text, for example, the Bell numbers are multiples of Mn (see Eq. (C.4)). While we shall nearly always treat the terms recurrence and difference equation as synonyms (especially since the first is desirable—being shorter than the second), they are not equivalent. Sometimes we get a recurrence that is not a difference equation, as the following example demonstrates! Example 5.26 — Euclid greatest common divisor algorithm The earliest algorithm apparently to be formulated was Euclid’s method to calculate the greatest common divisor (gcd) of two integers. We say d is the gcd of positive integers x and y when there are integers u and v such that x = ud, and y = vd—hence d is a divisor of both x and y—and the numbers u and v have no common factor that can be added to d. For example, the gcd of 96 and 56 is 8, and we have 96=12× 8 and 56=7× 8. Here is a C implementation of the algorithm; you may want to walk through the code for a few pairs of numbers to observe its beauty, and see how it can be slightly modified to produce, in addition to the gcd d, also the multipliers u and v: int gcd (int x, int y) { int rem; do { rem = x%y; x = y; y = rem; } while (y > 0); return x; }

// gcd calculation using Euclid’s method // on entry, x > y // rem gets the remainder of x divided by y

Let T (i, j) be the number of iterations that are performed when calculating the gcd of the two integers i and j. Reading directly from the code, we see the recurrence

T (i, j) = 1 + T ( j, i% j),

i, j ∈ Z+ .

(5.6)

While this is clearly a recurrence, it does not satisfy the normal form of a difference equation. Indeed, developing a solution for T (i, j) is not a trivial problem, although some of its properties are very easy to determine, and we urge you to find out as many as you can. A detailed analysis of the function T (i, j) is carried out in the second volume of Knuth’s book [78, §4.5.3]. Definition 5.27 If in the difference equation an = f (an−1 , an−2 , . . . , an−p ) of order p the function f is linear in all its arguments, then the equation is called linear. The first- and secondorder linear difference equations are of the following form:


210

αn an + βn an−1 = fn αn an + βn an−1 + γn an−2 = fn

first order linear equation, second order linear equation,

where { fn }, {αn }, {βn }, and {γn } are known sequences of coefficients. When all members of these sequences do not depend on n, the equation is said to have constant coefficients, otherwise, these are difference equations with variable coefficients. The sequence { fn } is called the non-homogeneous sequence (or forcing sequence) of the linear difference equation. If all elements of { fn } are zero, the difference equation is called a homogeneous equation (the accent is on the syllable “ge”). ⊳ Here are some examples of difference equations: an − a2n−1 = 0 (first order, non-linear, homogeneous) 2 an + an−1 − n an−2 = 0 (linear second order, variable coefficients, homogeneous) an − 2an−2 = n2 (linear second order, variable coefficients, non-homogeneous) In the last example, the corresponding homogeneous equation (an − 2an−2 = 0) has constant coefficients. A large amount of literature on the properties of difference equations has been produced over the last two centuries due to their prominence in many areas of applied mathematics and numerical analysis. We wish here to introduce just one result from this literature before examining particular cases. The general solution of a recurrence is a function that satisfies the recurrence and may contain arbitrary constants (which can be used to fit the solution to any compatible set of initial or boundary values). As a rule, the only condition on these constants is that their number does not exceed the order of the recurrence. If there are fewer initial values— or none—the solution will have in it “free,” undetermined constants. A solution with no arbitrary elements in it is called a particular solution. Theorem 5.28 The solution, {xn }, of the general linear difference equation of order p: p ∑ j=0 a j (n)xn− j = f (n), a0 (n) 6= 0, can be written as the sum of the general solution of the corresponding homogeneous equation, which satisfies ∑ pj=0 a j (n)xn− j = 0, and a particular solution of the non-homogeneous equation. What if a0 (n) = 0 for some n in the recurrence in Theorem 5.28? It means that xn is left undefined by the recurrence. We want to disallow such situations, and have only “proper” recurrences. Therefore we make a requirement: the function a0 (n), for any n > p, may not vanish! There seem to be no general tools to find a particular solution of non-homogeneous linear difference equations of order greater than 2, but only for some special cases of the function f (n). However, asymptotic methods are available to estimate the rate of growth of these solutions (see, for example, §10.6). On the other hand, the general solution of a homogeneous, constant coefficient, linear difference equation can be obtained via the following Theorem 5.29 Let xn = a1 xn−1 + a2 xn−2 + · · · + ak xn−p be a linear, homogeneous, constant coefficient difference equation of order p. Denote the


211

distinct roots of the auxiliary (characteristic) polynomial p

∑ a j λ j = a1 (λ − λ1)m (λ − λ2)m 1

2

j=1

. . . (λ − λr )mr = 0

by λ1 , λ2 , . . . λr and each root λi be of multiplicity mi (i = 1, 2, . . . , r), so p = ∑ri=1 mi . Then r mi −1

xn = ∑

∑ bi j n j λin− j

i=1 j=0

for some constants bi j . Both theorems (5.28 and 5.29) can be proved by substitution. Here is a particular case of the latter one. Corollary 5.30 The general solution of the second order homogeneous linear equation with constant coefficients, un+2 = pun+1 + qun (n > 0), (5.7) is explicitly expressed through the initial values (u0 and u1 ) as follows: un = Aλ1n + Bλ2n ,

A=

u0 λ2 − u1 , λ2 − λ1

B=

u1 − u0 λ1 , λ2 − λ1

(5.8)

where λ1 and λ2 are two distinct roots of the characteristic equation λ 2 = pλ + q. If p2 + 4q = 0, the characteristic equation λ 2 = pλ + q has a double root λ = p/2 and the difference equation (5.7) has the general solution un = u0 (1 − n)λ n + nu1 λ n = λ n [n(u1 − u0 ) + u0 ],

λ=

p , 2

p2 + 4q = 0.

(5.9)

Sometimes a difference equation of higher order can be reduced to a lower order equation. Exercise 5.31 [2−] Reduce the third-order difference equation T (n) = (n − 1)(n − 2)T (n − 3) + (n − 1)T (n − 1), T (0) = 1, T (1) = T (2) = 0, to a second-order equation.

Example 5.32 The following recurrence is not quite so general—it is only of order 1, and has constant coefficients, but it covers the essential points of Theorem 5.28. Consider the recurrence: xn+1 = 2 xn + 1, n > 0, x0 = 0.

The associated homogeneous equation is xn+1 = 2 xn , with the immediate solution2 xn = A2n , where A is an arbitrary number. Looking at the non-homogeneous equation, we see that xn = −1 solves it (in §6.4 you will learn how to find some particular solutions). The combined solution is then xn = A 2n − 1, and it only remains to find a value for A that satisfies the initial condition as well: this is A = 1 and hence the final answer is xn = 2n − 1. 2 The

reason we call such a solution immediate is that it can “be seen” from the equation: xn+1 = 2xn = 4xn−1 = 8xn−2 = · · · = 2r+1 xn−r , etc.

212


Example 5.33 Let xn+2 + 3xn+1 + 2xn = n2 , for n > 0, with the initial values x0 = 3 and x1 = 1. The corresponding homogeneous equation, xn+2 + 3xn+1 + 2xn = 0, has the general solution xn = (−1)n (u + 2n v), with arbitrary u and v. Now we look for a particular solution of the non-homogeneous equation based on the term, n2 , in its right-hand side. We guess a particular solution as a polynomial of the second order: xn = an2 + bn + c. Substituting in the equation, we find xn = n2 /6 − 5n/18 + 1/27. Adding the two provides the desired solution, still with the free constants u and v. They are now determined from the initial values, and we find u = 7 and v = −109/27. Exercise 5.34 [2−] Let {xk }k>0 be the solution of the linear first order difference equation xn = an xn−1 + b, n > 1, with x0 = c. Write in terms of the xk the solution of yn = an−m+1 yn−1 + b, n > m, ym−1 = c. Exercise 5.35 [2] Let xn+t = ∑t−1 i=0 a(i)xn+i , n > 0, be a linear homogeneous recurrence of order t. You need to show that it is linear in the initial values x0 , x1 , . . . , xt−1 . This can be ( j) done as follows. Let {xn }n>0 be the solution of the recurrence with the following initial values: all xk , k ∈ [0..t − 1] are zeroes, except x j = 1. Show that the solution for a given set ( j) of initial values xi = yi , i ∈ [0..t − 1], is given then by the sum xn = ∑t−1 j=0 y j xn . Exercise 5.36 [2] (Fibonacci recurrence revisited) Write expressions for the solutions of the following recurrences, in terms of the Fibonacci numbers, Fn , which satisfy the recurrence Fn+2 = Fn+1 + Fn ,

n > 0,

F0 = 0, F1 = 1.

(5.10)

You will find it easier to assume the conventional F−1 = 1. (a) xn+1 − xn − xn−1 = 0, n > 1, x0 = a, x1 = b. (b) xn+1 − xn − xn−1 = r, n > 1, x0 = a, x1 = b. Hint: See Exercise 5.35. Exercise 5.37 [2] (More with the Fibonacci recurrence) Let Fn be the nth Fibonacci number, Eq. (D.3). Show that wn = F2n±p satisfies the recurrence wn+2 = 3wn+1 − wn and find an explicit expression for wn , using Corollary 5.30. Example 5.38 (Waiting time of Bernoulli trials) This example looks at recurrences that arise in a probability problem involving sequences of experiments. Let a random number generator produces a sequence of n independent binary digits, equally likely to be zero or one. What is the probability that the sequence contains at least h consecutive 1’s?

Solution: Let An be the set of binary n-strings that do not contain h consecutive 1’s, and denote its size by An = |An |. The desired probability is 1 − An /2n . By appending a digit to all the elements of An , we obtain a set of 2An words, which is the union of two disjoint sets: An+1 and B. The latter set must have h 1’s, in the last positions, preceded by a 0 (otherwise this word, before the addition could not be in An ). The number of such words is given by An−h , that is, all words that do not have such a run preceding the 0 we just identified in them. Finally, since all the strings in An+1 can only arise from n-strings which are in An , we obtain the recurrence relation with appropriate initial conditions: 2An = An+1 + An−h ,

An = 2n for n < h and Ah = 2h − 1, h > 1, A0 = 1.

The case h = 2 leads to the second order difference equation An+1 = 2An − An−2 ,

A1 = 2, A2 = 3,

n = 2, 3, . . . ,


213

which is exactly the twice-shifted recurrence (5.10) for the Fibonacci numbers. Hence in that case An = Fn+2 , and the probability of a word of size n having two consecutive 1’s is 1 − Fn+2 /2n .

Example 5.39 (Continuation of Example 5.38) Consider now a biased generator, which produces 1’s with probability p and 0’s with probability q = 1 − p. What is the expected number of generated digits needed until h = 3 consecutive 1’s are obtained for the first time?

Solution: Although the situation looks similar, the fact that we ask about the 111 occurring for the first time makes a fundamental difference. As a result, we go in an indirect way. Let P(n) denote the probability that three 1’s occur consecutively for the first time in drawings (n − 2, n − 1, and n). Then P(1) = P(2) = 0 and P(h) = ph . If we need a longer sequence, it must be in one of the following three sets, characterized by their prefixes: {0 . . .}, {10 . . .} or {110 . . .}, and we get the recurrence P(n) = qP(n − 1) + pqP(n − 2) + qp2 P(n − 3),

for n > 3.

(5.11)

We do not plan to solve this third order recurrence here because we are trying to avoid finding the entire probability mass function; but it turns out that such a recurrence can provide quite a few benefits even when not solved. First, we show that the set {P(n)} is a proper PMF: these probabilities add to 1. Summing P(n) in the recurrence (dropping initial terms known to be zeroes), we get ∞

∞

n=1 3

n=4 2

def

P =

∞

∞

∞

k=3 3

k=3

k=3

∑ P(n) = p3 + ∑ P(n) = p3 + q ∑ P(k) + pq ∑ P(k) + p2q ∑ P(k) 3

= p + qP + pqP + p qP = p + (1 − p )P, which yields P = 1. Next we compute the expected number, denoted by µ , of generations until three consecutive 1’s appear: ∞

µ=

∑ n P(n) = 3p3 + ∑ n

n=1

n>4

qP(n − 1) + pqP(n − 2) + qp2 P(n − 3) .

Shifting the index of summation, we obtain

µ = 3p3 + q[µ + pµ + p2 µ ] + q[1 + 2p + 3p2 ] = (1 − p3 )µ + 1 + p + p2. Therefore, the expected number of generations to obtain 3 consecutive 1’s is

µ=

1 1 1 + 2+ 3. p p p

We revisit this problem in Example 9.71, page 517. Exercise 5.40 [3] We flip n times a biased coin, which shows heads with probability p. What is the probability to have at least one run of h heads?

214


Example 5.41 Tri-diagonal matrices turn up often in applications (especially in finite difference approximations of partial differential equations), which leads to computation of their determinants. Consider the n × n matrix M given byMi,i = 1 for 0 6 i 6n and Mi,i±1 = −t. 1 −t 0 0   −t 1 −t 0  .  Here is an example of such matrix for n = 4: M =    0 −t 1 −t  0 0 −t 1 It is convenient to let the symbol for the determinant, d = det M, carry the order explicitly, d4 for the determinant of the displayed matrix. We see that d1 = 1 and d2 = 1 −t 2 . To proceed to higher orders, we use the cofactor expansion along the first row. This yields only two terms. The first one is simply dn−1 . For the second term we find t multiplied by the determinant of a matrix obtained from M when the first row and the second column are deleted. The corresponding matrix  has the first column starting with −t; thus from the matrix for n = 4 we get  −t −t 0   1 −t . To evaluate its determinant, we expand the 3 × 3 matrix by the first column 0 0 −t 1 and find −td2 . In general, we obtain the recurrence dn = dn−1 − t 2 dn−2 for n > 2, with the initial values given above. This recurrence can be solved by a number of elementary methods (such as the method of undetermined coefficients or usage of Corollary 5.30), but we delay its solution to §6.4, where we can handle a larger variety of recurrences in a unified manner.

Exercise  5.42 [2] The entries of the (n + 1) × (n + 1) tri-diagonal matrix B are given by   − jz, j = i + 1, Bi, j = 1 − z, j = i, 0 6 i 6 n,   −z, j = i − 1. (a) Write the 4 × 4 matrix B4 . (b) Show that bn , the determinant of Bn , satisfies the recurrence bn = (1 − z)bn−1 − nz2 bn−2 . (c) Show by substitution that the determinant of Bn is given by the function n+1 √ , where Hn (x) is the nth order Hermite polynomial defined by Hn+1 z1−z bn (z) = √z2 2 the recurrence (M.7), page 750. Exercise 5.43 [3] (AVL binary tree) An AVL tree (AVL trees were invented in 1962 by two Russian scientists Adel’son-Vel’skii and Landis [2], after whom this data structure is named) is a binary tree with a balance condition: at each internal (that is, non-leaf) node, the heights of the left and right subtrees may differ by at most 1. (a) Draw all AVL trees of up to 5 nodes. (b) Let S(h) be the minimum number of nodes in an AVL tree of height h. Show that this count satisfies the following recurrence, S(h) = S(h − 1) + S(h − 2) + 1, h > 2, determine the initial values, and derive its solution. (c) What is H(n), the maximal height of an AVL tree of n nodes? What is the minimal height of such a tree? (d) Repeat part (c) for regular binary search trees, with no balance constraints. Note: You may have noticed what we failed to ask for the number of AVL trees with a given

5.3. FIRST ORDER LINEAR DIFFERENCE EQUATIONS

215

number of nodes (or of a given height). That analysis is much harder, and no simple answer is known. Exercise 5.44 [2] Show that the non-linear recurrence relation rn rn−1 − arn − brn−1 + c = 0,

n > 1,

can be reduced to a linear one by the substitution rn = un+1 /un + a. Using Corollary 5.30, find rn . Not all recurrences are so plain-looking as those we have seen so far. Indeed, with the exception of the recurrence introduced in Example 5.26, they have all been of degree at most two. Here are a few other recurrences that are harder to handle: √ • xn+1 = an xn + bn xn−1 , n > 1, x0 and x1 are given. • yn+2 = sin(yn+1 ) + cos(yn ), n > 0,  y0 , y1 are given.  if m = 0,  n + 1, • The Ackerman function: A(m, n) = A(m − 1, 1), if m > 0, n = 0,   A(m − 1, A(m, n − 1)), otherwise. • A nested recurrence: h(n) = 0 for n 6 0, and otherwise ( h(2 + h(2n)), if n 6 10, h(n) = n, if n > 10

(which results in the sequence 22, 24, 22, 16, 20, 24, 14, 16, 18, 20, 22, 11, 12, 13,. . . ).

5.3 First Order Linear Difference Equations Perhaps the simplest type of difference equation is the linear recurrence of the first order, namely, xn+1 = an xn + bn , n > m, (5.12) where both sequences {an }n>m , {bn }n>m , and the initial element xm are known.

The homogeneous recurrence (when all bn = 0) is very simple to handle: we do it by iteration. Since the relation xn+1 = an xn holds for all n > m, then we can iterate it: xm+1 = am xm ,

xm+2 = am+1 xm+1 = am+1 am xm ,

xm+3 = am+2 xm+2 = am+2 am+1 am xm · · · .

This suggests a guess solution as follows: xn = xm

∏

ak .

m6k 1,

x1 = d is known.

From Eq. (5.13), we obtain

∑ bj

j=1

∏

k= j+1

n

n

n

n

xn+1 =

a+d

∏a =

k=1

∑ b j an−( j+1)+1 + dan =

j=1

n

∑ b j an− j + dan

j=1

j b n 1 − (b/a)n b n n n n b 1− a n + da = a d + b + da = a , = a ∑ a 1 − ab a−b j=1 a n

if a 6= b. The case a = b is reduced to the difference equation xn+1 = (a + 1)xn that has a solution xn+1 = (a + 1)n d. In particular, for a = b − 1 and d = 1 we obtain xn = bn − (b − 1)n . Therefore the solution of the recurrence (5.5) on page 206 is M(n, d) = nd − (n − 1)d .

(5.14)

Example 5.46 The following simple recurrence highlights a point we saw before, in Example 5.3. Solve the recurrence xn+1 = xn + an , n > m. Substitution in the solution formula yields the unsurprising result: n

xn+1 =

∑ ak + xm . k=m

(5.15)


217

So every sum is a solution of a difference equation of the first order. A constant coefficient first order difference equation xn+1 = a xn + b, n > 0, has a rather simple solution: xn = an x0 + b(1 + a + a2 + · · · + an−1 ) = an x0 + b

1 − an . 1−a

(5.16)

Example 5.47 In this example, we show that some probabilistic problems lead to vector difference equations, which we discuss in §6.4.2 later. In particular, we consider a problem of calculating the number of heads-to-tails changes (HT for short) in a binary string of size n generated by flipping a coin. This problem arises in a test on randomness when binary strings are produced by some source. A particular case of the problem was considered in Exercise 3.42 on page 98. Let tn and hn be the number of H-to-T changes in all strings of length n from the alphabet { H, T } starting with tails and heads, respectively. Any such string of size n can be obtained by placing in front of each (n − 1)-string a letter, either H or T. If we add H in front of a string starting with H, we add nothing in the number of H-to-T changes in a string of length n − 1, which is hn−1 . If we add H in front of an (n − 1)-string starting with T, we add 2n−2 (the total number of strings of length n − 1 starting with T) HT changes to the previously existed HT. So we get the system of difference equations: hn = hn−1 + 2n−2 + tn−1 ,

tn = hn−1 + tn−1 ,

n = 2, 3, . . . ,

t2 = 0, h2 = 1.

The second recurrence, tn = hn−1 +tn−1 , follows from the observation that when T is placed in front of a string it does not affect the number of heads-to-tails changes. Then for xn = tn + hn , total number of such changes, we get the first order non-homogeneous difference equation xn = 2 xn−1 + 2n−2 ,

n = 3, 4, . . . ,

x2 = 1,

which has the solution xn = (n − 1) 2n−2 .

Exercise 5.48 [2] Consider a simple instance of a common situation: we define a combinatorial count, in this case un , as the number of bit sequences of size n that have an even number of 0 bits. We wish to derive a recurrence for un ; this turns out to be much simpler if we also define another related count: vn , the number of bit sequences of size n that have an odd number of 0 bits. Using the initial conditions u0 = 1, v0 = 0, find a pair of recurrence relations in n for un and vn , and solve it. Exercise 5.49 [2] In §1.1.3, we presented the Strassen and conventional matrix multiplication methods and observed that they differ in the amount of the additions (A, for short) and multiplications (M) they performed, and that the number of multiplications was the dominant one. Here you are asked to look at it more closely, write recurrences that do not lump the two operations but keep them separate. Thus, the relations for the two methods would be 2 2 T (n) = 7 T n2 + 15 n4 A and C(n) = 8C 2n + 4 n4 A, with C(2) = 8M + 4A. Note that for T (2) we have a choice of using the same value as for C(2), or we may be “stubborn” about this and use the value 7M + 15A. Write solutions for those functions at values of n which are powers of 2, and see where T gets lower than C under different assumptions about the relative cost of A and M and choice of T (2).

218


Exercise 5.50 [2] You have coins C1 , C2 , . . . , Cn . For each k, coin Ck is biased so that, when tossed, it has probability (2k + 1)−1 of coming up heads. If the n coins are tossed, what is the probability that the number of heads is odd? Exercise 5.51 [3] An urn contains r distinguishable red balls and b identical brown balls, which are drawn one at a time at random with replacement, until a brown ball is drawn. Some red balls may be drawn more than once before the first brown ball is drawn. Let Xn denote the number of distinct balls after n red balls have been drawn. Find the expected number of Xn by deriving a first-order recurrence. What is the minimal value of b for which the expected number of red-ball repetitions is less than 1? Exercise 5.52 [2] A cask of unit volume is full of wine. A man withdraws from the cask an amount a (0 < a < 1), and then adds amount a of water, which is assumed to mix perfectly with wine. Next time, he takes out amount a of mixture and adds amount a of wine. He continues this process by alternatively adding water and wine after each withdrawal. Derive and solve a recurrence for the amount of wine in the cask after n such double operations (that include adding water and then wine). Example 5.53 Consider the recurrence nxn = (n − 2)xn−1 + 2,

n > 2,

x1 = 1.

2 While it can be easily reduced (through dividing by n) to a standard form, xn = n−2 n xn−1 + n , for which the solution formula (5.13) is known, there is a neater way. Multiply both sides of the recurrence by (n − 1) to obtain

(n − 1)n xn = (n − 2)(n − 1) xn−1 + 2(n − 1),

n > 2,

x1 = 1.

Note that since we only use the equation for n > 2, no information is destroyed by this def multiplication. Using the notation: un = n(n − 1)xn , we get that un satisfies the first order constant coefficient non-homogeneous recurrence un = un−1 + 2(n − 1), n > 2, and u1 = 0. This is almost as in Example 5.46; to use it, shift the index, so that un+1 = un + 2n, n > 1, and then n un+1 = ∑ 2 j + 0 = n(n + 1) =⇒ un = (n − 1)n. j=1

Thus xn = 1, which may appear odd, but it satisfies all the requirements.

While the previous example may be considered neat, the following exercise is a caution about neatness. Notice that the fact that the representation we used “lost” the value of x1 is not an accident: it has no effect on the values of the succeeding terms! Exercise 5.54 [2] Solve the recurrence in Example 5.53 with the initial value x1 = 2. Example 5.55 Sometimes when we are trying to evaluate a complicated sum or product, and we do not see a clear way, the best method is to rearrange the expression to produce a solvable recurrence. Here is an example that goes beyond the direct approach of Example 5.3. n 2k n 2k n 1 =∑ , = δn,0 − ∑ (−1)k 1 − (−1)k Sn = ∑ (−1)k 2k + 1 2k + 1 2k +1 k k k k k k

the last where δn,0 is the Kronecker delta, defined in Eq. (2.5) on page 29. Using Eq. (2.58), k 2n . This (−1) expression in the right-hand side can be changed to yield δn,0 − ∑k n−1 2k+1 k−1


219

suggests the usage of the addition formula (2.62) in order to split the binomial coefficient in the sum for Sn , and we have n−1 1 n−1 n−1 1 = Sn−1 + ∑ . (−1)k Sn = ∑ + (−1)k k − 1 2k + 1 2k +1 k k − 1 k k Denoting the right-most sum by A, we have Sn = Sn−1 + A = δn,0 − 2nA. Eliminating A from the two equations, we get

(2n + 1)Sn = δn,0 − 2nSn−1

=⇒

Sn =

2n Sn−1 + δn,0 , 2n + 1

which is now straightforward to plug into Eq. (5.13), and we find −1 n−1 (2n)!! 4n 2n 2j+2 = = . Sn = ∏ (2n + 1)!! 2n + 1 n j=0 2 j + 3 Example 5.56 Let us consider our last plain vanilla—the first order difference equation xn+1 = (n + 1) xn + (n + 1)!,

n > 0,

x0 = 1.

Derivation of the solution just follows the protocol of Eq. (5.13). Setting an = (n + 1) and bn = (n + 1)!, we obtain n

xn+1 =

n

n

∑ ( j + 1)! ∏

j=0

k= j+1

(k + 1) + ∏ (k + 1). k=0

Since n

∏

(k + 1) = ( j + 2)( j + 3) · · · (n + 1) =

k= j+1

(n + 1)! , ( j + 1)!

n

∏(k + 1) = (n + 1)!,

k=0

we get n

xn+1 =

∑ (n + 1)! + (n + 1)! = (n + 2)! .

j=0

Exercise 5.57 [1+] Solve the recurrence xn+1 = (n + 1) xn + n!,

x0 = 0

Exercise 5.58 [1] Although the n-th harmonic number, Hn , is never an integer for n > 1, it can be expressed as a rational number Hn = pn /n!. Derive a recurrence for its numerator, pn . Exercise 5.59 Solve the following recurrences, which all hold for n > 0 and have x0 = 1: (a) [1] (c) [2] (e) [2]

xn+1 = xn + n. xn+1 = 12 xn + n(n + 1). xn+1 = 2 xn + 2n .

(b) [1] (d) [2] (f) [3]

xn+1 = xn + (n + 1)2 . xn+1 = 23 xn + n + 6. xn+1 = (n + 1) xn − 1/(n + 2).

Exercise 5.60 Solve the following recurrences, which do not depend on any initial value: n an + 2Hn , n > 1. (a) [2] an+1 = n an + 2n , n > 0. (b) [3] an+1 = n+1

220


Exercise 5.61 [4] (Continuation of Exercise 4.89 on page 167) (a) We consider a sequence of n urns. The kth urn has k + 1 red balls and k + 2 white ones, for 1 6 k 6 n. We perform an n-step process: at step k a ball is picked at random from urn #k, and moved, unobserved, to the next urn in the sequence. You only look at the final, nth ball. What is the probability that it is red? (There seems to be no closed form solution for the recurrence you need to develop for this probability.) Show numerical results, and find the limit, as n → ∞. (b) Consider now the case where the original populations of the kth urn had k balls of each color. The same process is performed. Show that the initial probability to draw a red ball, 1/2, remains so for any urn. Exercise 5.62 [3] The famous Moore’s law of VLSI has many versions; the most common one says that the density of gates in memory chips doubles every 18 months. Translate this claim to a recurrence, relating the gate density, dn , at the end of year n to the density at the end of the preceding year. The engineers in WoWvLsI Inc. have shown that they can do better: they can sustain the same exponential rate, and add 106 more gates per square inch each year. Given that d2000 = 107.3 gates per square inch, and assuming that the trend continues, when will the density reach one trillion (1012 ) gates per square inch according to Moore’s Law? What density will the ambitious guys at WoWvLsI achieve then? Exercise 5.63 [3] Consider a square with side 1. Its area is 1 and its perimeter is 4. The square is then transformed into a 20-sided polygon by removing the middle third of each side and attaching there a new square, “pointing outside.” Each side of this attached square has length 1/3. This polygon has area 13/9 and perimeter 20/3. Continuing this process we generate a sequence {Pn } of polygons; the original square is denoted by P1 . At each transition we delete one third of each side and attach a square on each deleted segment, transforming Pn to Pn+1 . We use below the following notation: sn denotes the number of sides of Pn , an denotes its area, and pn its perimeter. (a) Write recurrence relations for sn , an , and pn and solve them. (b) For what n does the area reach 1.5 first? When it reaches 1.99? (c) Repeat parts (a) and (b) for an alternative transformation process. Now the steps alternate between attaching an equilateral triangle on each deleted segment, and attaching a square, as before. Assume the first transformation uses triangles. Exercise 5.64 [2] Here is a silly procedure to sum the first n nonnegative integers, written in C: int sum (int n) { int i, j, temp = 0; for (i = 0; i < n; i++) for (j= 0; j 1 n−k k . Show that it satisfies the recur 2n−1 rence nun = 2(2n − 1)un−1 + n , and has the solution un = 21 2n n Hn , where Hn is the nth harmonic number.

5.4. RECURRENCES WITH “INTEGER FUNCTIONS” Exercise 5.66 [3+] Define the sum Sn = ∑k6n (a) Show that it satisfies the recurrence

n+r k n−k , k a b

221 n ∈ N.

n+r−1 n Sn = (a + b)Sn−1 + a . n (b) This equation for Sn has, of course, a solution, given by Eq. (5.13). While it has no closed form, show that it leads to the curious identity n + r k n−k r+k−1 k Sn = ∑ ab =∑ a (a + b)n−k . (5.17) k k k6n k6n There is an elegant and direct argument in the classical book [56] by Graham, Knuth, and Patashnik of the last identity. Explain, first, why it is true that when n > 0 and r is an integer in the range −n 6 r 6 0, each side in Eq. (5.17) equals (a + b)n+r /br . Then explain why this is enough to claim that the identity holds for all r (including noninteger values).

5.4 Recurrences with “Integer Functions” The reader is expected to know the integer functions of the section name: the floor and ceiling (see Definition 2.2 on page 28). Exercise 5.67 [2+] In the classical Towers of Hanoi problem3 the rings are moved one at a time. Consider a variation: whenever there is more than one ring on a peg, you are allowed to move two rings in a single step. (a) Prove by induction that it is always advantageous to move two rings when you can. (b) Develop a recursive relation for the number of steps required to perform the transfer of n rings. Solve the recurrence.

5.4.1 Divide-and-Conquer Recurrences Many useful algorithms lead to recurrences with “integer functions.” A particular class of these are divide-and-conquer algorithms. Such an algorithm typically breaks the problem into several subproblems that are similar to the original one, but have smaller sizes, and then solves the subproblems recursively. The divide-and-conquer approach is a top-down approach because the solution to a top level instance of a problem is obtained by going down and obtaining solutions to smaller instances. The running time of such an algorithm can often be described by a recurrence with integer functions. We saw this in Example 5.7 on page 203. Example 5.68 Here is a simpler equation, arising, for example, if we analyze an inefficient way to calculate the sum of digits of a binary number: an = a⌊n/2⌋ + α , 3 This

n > 1,

with the initial condition a0 = β .

(5.18)

famous puzzle, proposed in 1883 by a French mathematician, Edouard Lucas, deals with three poles (pegs) and n rings of different sizes. Initially the rings are stacked in order of increasing size on the leftmost pole (labeled pole 1) in such a way that the largest ring is at the bottom. The problem is to find the minimum number of steps required to move the n rings from pole 1 to the rightmost pole (labeled pole 3). A step consists of transferring a single ring from one peg to another, with one limitation: a ring may never be placed on top of a smaller ring.

222


In general, there is no algorithm to solve recurrences, but there are classes for which we have methods that work. With this class, at least in the relatively simple cases, we have two approaches. One method is simply to iterate the equation, compute a prefix of the sequence, try to guess a formula for the general form, and prove that it satisfies the equation. Let us consider this approach for Eq. (5.18). We find a1 = a0 + α = β + α , a2 = a1 + α = β + 2 α , a3 = a1 + α = β + 2 α , a4 = a2 + α = β + 3 α , a5 = a2 + α = β + 3 α , a6 = a3 + α = β + 3 α , a7 = a3 + α = β + 3 α , .. . This may not yet be enough to guess the solution, though with some experience it might! However, let us use another approach by considering values of n that are powers of 2, that is, def n = 2r , r ∈ N. Then we can use a domain transformation, defining br = an for n = 2r . This yields a simpler recurrence br = br−1 + α , with b0 = a1 = α + β , and its solution is br = r α + b0 = (r + 1) α + β . Since4 r = lg n, the corresponding formula for an is an = α lg 2n + β . Note that this agrees with the entries a1 , a2 , a4 , in the above list. However, it does not produce the integer multiples of α that the other values there show. The values in the list suggest the following formula for a solution of Eq. (5.18): ?

an = (1 + ⌊lg n⌋) α + β .

(5.19)

First, we check that it fits all entries calculated previously—it does—and then we prove this formula by substitution: h j j n kk i ? α + β + α. (1 + ⌊lg n⌋)α + β = 1 + lg 2

After canceling β , we get j j n kk ? α ⌊lg n⌋ = α 1 + lg 2

=⇒

j j n kk ? ⌊lg n⌋ = 1 + lg . 2

Every number n can be written as n = 2r + s, 0 6 s < 2r , with an integer r, then 2r 6 n < 2r+1 4 The

=⇒ r 6 lg n < r + 1 and ⌊lg n⌋ = r

notation lg n is used for the logarithm to base 2 of n, that is, lg n = log2 n.

∀n, r ∈ N.

(5.20)

5.4. RECURRENCES WITH “INTEGER FUNCTIONS” 1/ v 2 k

223

1/ v 2 k+1

............................................................................................................................ ....................... .................. .................. .............. .............. .............. .............. ........... . . . . . . . . . . ........... ....... . . . . . .......... . . . . . ......... ...... . . . . . . . . .......... . ...... . . . ......... . . . . . ........ ...... . . . . . . ........ . . ... ........

uk+1

1/ u 2 k

Archimedes’ diagram for π estimation. uk and vk are sides of inscribed and circumscribed regular polygons of n = 2k sides, as discussed in Exercise 5.69.

α /2

α

Similarly, 0 6 2s < 2r−1 =⇒ 0 6 2s < 2r−1 . Then lg 2n = lg 2r−1 + 2s . Notice that the right hand side is in the interval [r − 1, r), hence lg n2 = r − 1, which we substitute in Eq. (5.20). This leaves the true equality ⌊lg n⌋ = r; therefore the guess-solution is the right one. Note: This is the correct solution, not merely that it satisfies the equation, but because the solution of such an equation is unique. Exercise 5.69 [2+] Probably the first “divide-and-conquer” algorithm was proposed by Archimedes5 to calculate 2π , the length of the circumference of a circle of radius one. Let an and bn be the perimeter of the circumscribed and inscribed n-gon. These two sequences, {an } and {bn }, give upper and lower estimates of the number 2π , respectively. Show that they satisfy the following vector recurrence:

an =

2an/2 bn/2 , an/2 + bn/2

bn =

q

an bn/2 ,

a4 = 8,

√ b4 = 4 2,

n = 2k , k = 2, 3, . . . .

Also show that the convergence rate of this calculation (both a and b) reduces their relative error by a factor of 4 each iteration. Exercise 5.70 [3] (Karatsuba algorithm) Consider the problem6 of multiplying two large decimal n-digit integers. For simplicity, we assume that n = 2k . We split two given integers 5 Archimedes was born in 287 BC in Syracuse, Sicily, and was killed by a Roman soldier in 212 or 211 BC during the Siege of Syracuse. 6 The first general method and algorithm for fast multiplication of numbers was introduced by a famous Russian mathematician A. A. Karatsuba in 1960.


224

into two halves: N1 = 10n/2 x + y,

N2 = 10n/2 u + v,

where x, y, u, and v are 2n -digit integers. Then the product of N1 and N2 can be computed as N1 N2 = 10n/2 x + y 10n/2 u + v = 10n xu + 10n/2 (xv + yu) + yv.

Let us introduce three products: p = xu,

q = yv,

r = (x + y)(u + v) = p + q + (xv + yu).

Then xv + yu = r − p − q and therefore the product N1 N2 can be done in three multiplications of n2 -digit numbers: N1 N2 = 10n p + 10n/2 (r − p − q) + q, since multiplication by powers of 10 is just a shift. Clearly, so long as we still have long numbers we will continue recursively. Let T (n) be the number of arithmetic operations needed by this algorithm to compute N1 N2 . Derive (or estimate) a recursive formula for T (n). Show that T (n) ∈ Θ nlg 3 , where lg 3 = log2 3 ≈ 1.58496. Write a C-program to calculate the number of arithmetic operations. Do you see any advantages compared to classical multiplication? Exercise 5.71 [3] (E. Deutsch) An object moves along grid in the plane, at each step it goes one unit either up, down, to the right, or to the left. Find the number of such paths that stay in the first quadrant {(x, y) | x, y > 0}, and consist of a total of n steps, exactly k of which are vertical (up or down).

5.4.2 Binary Search Recurrence Recursive binary search (RBS) is one of the simplest and best-known divide-and-conquer algorithms. While it appears here for the interesting recurrence it generates, it is sufficiently important—and elegant—to merit discussion on its own. This algorithm solves the problem of finding the position of a given value in an array containing sorted numbers (which we shall assume to be in increasing order). The considered algorithm also assumes the searched value to be in the array. If it is not, then this algorithm, as implemented below, returns an incorrect result. Let a[0, . . . , n − 1] be an array sorted in increasing order, that is, a[i] < a[ j] whenever 0 6 i < j < n. Note that this implies no two values in a are equal, to avoid ambiguity. Here is a C-language implementation of the algorithm, where we assumed the elements are floats, simply to distinguish them from indices: 1. float RBS (float ∗A, int j, int r, float x) 2. {if ( j == r) return (r); 3. int m = ( j + r + 1)/2; 4. if (x < A[m]) 5. return RBS(A, j, m − 1, x); 6. else return RBS(A, m, r, x); 7. }

5.4. RECURRENCES WITH “INTEGER FUNCTIONS”

225

The initial call is RBS(a, 0, n − 1, x) with the extreme ends of the array, 0 and n − 1. At each recursive call, the median position is located and one comparison directs the continuation of the search. Exercise 5.72 [2+] Prove the correctness of the procedure RBS by induction on the array size. We want to calculate an , the expected number of key comparisons when RBS is looking up a value in an array of size n. We calculate it assuming that x is equally likely to be at any position in the array, its probability is 1/n. We choose the bisecting value to be m = ⌊ n2 ⌋. Then the probability of a given number to be in the left half of the array in the initial call is p=

m ⌊ 2n ⌋ = n n

(5.21)

and the probability of a given number to be in the right half is 1− p = 1−

⌊ n2 ⌋ n − ⌊ 2n ⌋ ⌈ 2n ⌉ = = . n n n

Since every recursive call we make needs one comparison, the recurrence for an becomes an = 1 +

⌈n⌉ ⌊ n2 ⌋ a⌊ 2n ⌋ + 2 a⌈ 2n ⌉ , n n

n > 1, a1 = 0.

(5.22)

To solve Eq. (5.22), we use the second method mentioned in Example 5.68: pick n that is a power of 2, such as n = 2r , r ∈ N. For this value of n, we find that ⌊ 2n ⌋ = ⌈ 2n ⌉ = 2r−1 , and the probability p defined in Eq. (5.21) is just one half, hence a2r = 1 + a2r−1 . Using the same domain transformation, br = a2r , we get br = 1 + br−1 ,

b1 = 1,

=⇒

br = r.

So a2r = r

=⇒

an = lg n, for n = 2r .

(5.23)

What happens with an arbitrary n? Clearly the values of an are rational numbers, and the logarithm of n, binary or not, is not rational for most values of n. We can evaluate a few values directly from Eq. (5.22) to obtain the following table: n

an

n

an

1 2 3 4 5 6

0 2/2 5/3 8/4 12/5 16/6

7 8 9 10 11 12

20/7 24/8 29/9 34/10 39/11 44/12


226

We have left a few numbers in the table in a form that could be simplified further (e.g., 8/4 def and 16/6) so that each an has a denominator equals to n. Let an = un /n, and consider the numerators un . The table is quite suggestive: looking at successive differences, we see a steady increase, and on inspection we find un+1 − un = 2 + ⌊lg n⌋, n > 1. This again is an equation of the form solved in Example 5.46 by summation: un = u1 + ∑n−1 j=1 (2 + ⌊lg j⌋). The value of u1 is zero, but the summation requires some work. First, we take out the ‘2’ in the summand; it is repeated n − 1 times, and contributes 2(n − 1). With the same approach used for Eq. (5.20), write n = 2r + s, with integers r and 0 6 s < 2r . Then we split the summation range 1 6 j 6 n − 1 into steps, at successive powers of two: {1 6 j 6 n − 1} = {0 6 k 6 r − 1 : 2k 6 j < 2 × 2k = 2k+1 }

[

{2r 6 j 6 n − 1}.

There is a reason for this particular partition: when 2k 6 j < 2k+1 , we have ⌊lg j⌋ = k. Now, n−1

∑ ⌊lg j⌋ =

j=1

r−1 2k+1 −1

∑ ∑

k=0

j=2k

2r +s−1

k+

∑

j=2r

r−1

r=

∑ k2k + sr = 2 k=1

1 − r2r−1 + (r − 1)2r + rs.

Since 2r = n − s, the result can be rearranged as 2 + rs + (r − 2)(n − s). Adding the 2(n − 1) to the sum, we are left with 2s + rn = 2(n − 2⌊lg n⌋ ) + n⌊lg n⌋, where r = ⌊lg n⌋. Hence an = ⌊lg n⌋ + 2(1 − 2⌊lg n⌋ /n) = ⌊lg n⌋ + 2s/n. Note that the last term is in the interval [0,1). Exercise 6.119 provides a value for the PMF of the number of key comparisons. Exercise 5.73 [1] While the solution just derived may be checked against the table, and shown to fit it all, its correctness is still needed to be proved. By substituting an into the equation (5.22), do so. Exercise 5.74 [2] Show that if n = 2r + 1, then Eq. (5.22), with the help of the solution Eq. (5.23), can be transformed into a linear first order recurrence. Use formula (5.13) to show that 2 1 r (2 + r + r2 ) = r+ . a2r +1 = r 2 +1 n Exercise 5.75 [3+] Here is an iterative version of the binary search, which also introduces an additional key comparison, for equality. int search (float a[], int n, float key) {int L, i = 0, j = n − 1; while (i key) j = L − 1; else i = L + 1; } return (−1); }

(a) Prove the correctness of the procedure search by induction on the size of the array. (b) Write a recurrence relation for {dn }, the expected number of “divide by 2” operations performed when the list has length n. Assume that the key is in the array, and it is equally likely to be any of the values. Solve the recurrence, which can be reduced by introducing an = ndn − (n − 1)dn−1 for n > 1.

5.4. RECURRENCES WITH “INTEGER FUNCTIONS”

227

(c) Obtain from the solution of part (b) the expected number of key comparisons this version does when the searched key is in the array. Comment on the effect of the extra keycomparison in this case (if the key is not in the array such a change leads to pure waste). Exercise 5.76 [3] Solve the recurrence E(n) = 1 +

n − ⌈n/2⌉ ⌈n/2⌉ − 1 E(⌈n/2⌉ − 1) + E(n − ⌈n/2⌉), n n

E(0) = 0,

E(1) = 1.

Hint: Setting W (n) = nE(n), you can reduce this recurrence to the recurrence (5.18) for an = W (n) −W (n − 1), where n > 1.

Exercise 5.77 [2] The following game is associated with the names Bill Gates and Steve Ballmer, the founders of Microsoft. A certain integer (call it “mystery”) is chosen at random from [1..n] = {1, 2, . . . , n} and you are asked to guess it. If you make a correct guess, you are rewarded with r dollars. Otherwise, you pay one dollar before making each next guess, stopping when the correct number is discovered. After each unsuccessful guess, you are provided the information if your choice was higher or lower the mystery number. For what values r and n this game is advantageous for you? A sensible strategy to win the game is an implicit binary search: make a guess in the middle of the current interval, so that the number of candidates is reduced by half with each guess. Let E(n) be the expected number of guesses needed under this strategy to find a uniformly distributed mystery integer in the set [1..n]. Show that E(n) satisfies the recurrence from Exercise 5.76 and that the game is fair if n = 2r+1 − r − 2.

5.4.3 Mergesort Recurrence Like the binary search, merge sort is conceptually one of the simplest sorting algorithms and is based on the divide-and-conquer approach. To sort a list in an array, it splits the array into two equal (or almost equal) sub-arrays, recursively sorts each part, and then merges them into a solution of the original problem. There are several versions of merge sort that differ in minor technical details. The following algorithm is adapted from [121]. The variables b,e, and m stand naturally for the beginning, end, and middle of the (current) array a[0..n − 1] of size n = |a|, and the procedure is initially called as mergesort (a, 0, |a|-1);. The array tem[] is provided from outside the procedure. This implementation is not the easiest to follow since it is clever in its use of the array tem in a way that saves index-boundary checking.

mergesort (int A[], int b, int e) { int i, j, k, m, t; if (e > b) {m = (b + e)/2; t = m + e + 1; mergesort(A, b, m); mergesort(A, m + 1, e); for (i = m; i >= b; i–) tem[i] = A[i]; i++; for( j = m + 1; j 1,

u0 = a.

Since this is of the form Eq. (5.12), the solution is immediate, from Eq. (5.13), and we find: ur = a2r + br2r

=⇒

v(n) = an + bn lg n,

n = 2r , r > 0.

(5.25)

Exercise 5.79 [3] Solve the merge sort recurrence (5.24). To do this, calculate enough entries of v(n) to be able to guess a solution and prove it by showing directly that it satisfies the equation. The next problem was inspired by the Jewish soldier and historian Josephus Flavius (37– 101AD) who took the Roman emperor Vespasian’s family name instead of his own (Joseph ben Matthias). In this problem, n numbers, 1 through n, are arranged in a circle with elimination of every second number. Let J(n) be the position of the last number that remains. A thorough exposition of the Josephus problem is made in the book by Graham et al. [56]. Exercise 5.80 [3] The following equations are known as Josephus (vector) recurrence: J(2n) = 2J(n) − 1,

J(2n + 1) = 2J(n) + 1,

n > 1, J(1) = 1.

While it does not involve explicitly the integer functions, it clearly is of the same type. Solve it by iterating the recurrence, guess a solution, and prove it by substitution. Check your answer with the Josephus solution J(41) = 19. Exercise 5.81 [3] This problem is a natural generalization of the previous one. Let J(n, q) be the position of the last number when every q-th term in a circle of n sequential integers is eliminated, and q > n is possible: the entire circle is scanned ⌊q/n⌋ times before the number in position q mod n is eliminated. Show that for n > 1, q > 1, the sequence of numbers {J(n, q)} satisfies the recurrence J(n + 1, q) = J(n, q) + q (mod n + 1).

5.5. QUICKSORT RECURRENCE

229

5.5 Quicksort Recurrence Quicksort, invented in 1962 by C. A. R. Hoare, is probably the fastest sorting algorithm in the average case. Mergesort, just considered in §5.4.3, is in fact almost as fast, and has the added merit of having a nearly deterministic running time. However, its competitiveness is hurt by the need for a large temporary storage—Mergesort requires an amount equal to twice the length of the list being sorted. The memory can be significantly reduced by extra moves of the data, but it then loses its performance edge. Quicksort (QS) is also of the divide-andconquer family, but unlike Mergesort, it uses the probabilistic partition of the given array into two sublists. As most good algorithms, Quicksort is based on an inherently simple idea. At every stage, Quicksort selects from the given list of numbers a value called the pivot. The array is then rearranged in such a way that the values less than the pivot are placed to its left and elements that are greater than or equal to the pivot are moved to its right. The values in each partition are not yet sorted at that stage: each sub-array is sorted independently by recursive calls to the algorithm. Here is a straightforward, recursive implementation of the Quicksort algorithm: 1. QS (A, i, j) 2. { if (i >= j) return; pivot (A, i, j, k); 3. QS (A, i, k − 1); 4. QS (A, k + 1, j); } 5.

where pivot is the procedure that partitions the array around a chosen element (that is why it is sometimes called a partition). This procedure is crucial for understanding the Quicksort algorithm because its performance depends significantly on the subroutine pivot. There are many known methods to implement the procedure pivot—some of them are analyzed later, in Exercise 5.85. Extensive exposition of Quicksort algorithms together with several variations of pivot subroutines is given by R. Sedgewick [120]. The procedure pivot(A, i, j, k) rearranges the array A[i.. j] so that two conditions are satisfied: • the kth element is in its final position of the sorted list; • all elements to the left of A[k] are less than or equal to it and all elements to the right of A[k] are greater than or equal to it. Therefore, when Quicksort calls for pivot, it reduces the original problem to sorting two subarrays of total size one less. If a list is of size 0 or 1, it is already sorted. Initially, this Quicksort algorithm is called upon the entire array, that is QS(A, 0, |A| − 1). The recursion cannot be replaced by iteration here, but some implementations maintain the recursion stack, instead of relying on the compiler.

We denote by T (n) the expected run-time of Quicksort on an array of size n. The cost of entering QS and performing pivot, when it partitions n elements, can be written as an + b, where b is the time needed for the comparison of i and j, and the overhead of the call to pivot. The cost in this procedure is essentially proportional to the array size, n. We assume that the


230

pivot value is chosen randomly because either the procedure chooses a random element or the initial data is distributed uniformly. The assumption of randomness of the pivot value leads to a uniformity of the position of k in [0..n − 1], with the probability 1/n. Hence, n−1

1 (T (k) + T (n − k − 1)) , n k=0

T (n) = an + b + ∑

n > 2.

(5.26)

We perform some obvious manipulations: 1 T (n) = an + b + n

n−1 k=0

k=0

1 = an + b + n

n−1

n−1

= an + b +

!

n−1

∑ T (k) + ∑ T (n − k − 1) !

∑ T (k) + ∑ T ( j) k=0

2 n−1 ∑ T (k) n k=0

j=0

( j = n − k − 1) n−1

=⇒

nT (n) = an2 + bn + 2 ∑ T (k). k=0

This is a recurrence of unbounded order (which is also called the full-history recurrence). However, it can be converted into a first-order recurrence by creating a copy with shifted index and subtracting the two relations: n

(n + 1)T (n + 1) = a(n + 1)2 + b(n + 1) + 2 ∑ T (k)

shift

k=0

(n + 1)T (n + 1) − nT (n) = a((n + 1)2 − n2 ) + b(n + 1 − n) + 2T (n) (n + 1)T (n + 1) = (n + 2)T (n) + a(2n + 1) + b a(2n + 1) + b n+2 T (n) + , T (n + 1) = n+1 n+1

subtract rearrange n>2

(5.27)

This is a recurrence we know how to solve because it is in the form (5.12) and its solution is given by Eq. (5.13): n−1 i+2 a(2 j + 1) + b n−1 i + 2 + T (2) ∏ ∏ j+1 i= j+1 i + 1 i=2 i + 1 j=2

n−1

T (n) =

∑

n−1

n+1 a(2 j + 1) + b + T (2) 3 j=2 ( j + 1)( j + 2)

T (n) = (n + 1) ∑ = (n + 1)

n−1 n−1 2j+1 1 T (2) +a ∑ +b ∑ 3 j=2 ( j + 1)( j + 2) j=2 ( j + 1)( j + 2)

!

.

Using partial fractions decomposition and the notation Hn = ∑nk=1 1k , we may write the first sum as: n−1

2j+1 ∑ ( j + 1)( j + 2) j=2

n−1

n−1 1 1 +3 ∑ j=2 j + 1 j=2 j + 2 11 3 3 . = 2Hn − 4 + = −Hn + + 3 Hn+1 − 2 6 n+1

= −∑


231

The second sum is telescopic: n n−1 1 1 1 n+1 1 1 1 − = ∑ j+1 j+2 ∑ i − ∑ i = 3 − n+1 i=4 i=3 j=2 Therefore 1 3 1 T (2) + a 2Hn + −4 +b − . T (n) = (n + 1) 3 n+1 3 n+1

(5.28)

Since Hn ∼ ln n+ γ , we find that T (n) ∼ 2an ln n is a first order approximation for T (n) (when b, which is included in T (2), is not “too” large).

Notes: 1. The Quicksort algorithm is considered to be so efficient because it can be implemented with a small a. 2. We take up the analysis of this algorithm again in §6.6.4 with the aid of generating functions. 3. Exercise 5.125 on page 254 asks about using the same partition function of Quicksort for the selection problem, which leads to a very different recurrence! Exercise 5.82 [3] The analysis of Quicksort gave us a detailed information about the dependence of the expected cost on various parameters. This scheme is not easily applied to find higher moments of the cost. Here is a somewhat different approach. Eq. (5.28) says that the main cost is proportional to a, which is the coefficient in the linear term in the cost of pivot. Most of the work of pivot is simply comparing terms with the pivot, hence we argue that just counting such comparisons captures the main cost component. We proceed to count these comparisons. Let a1 , · · · , an be the elements of the array in their sorted order and a pair of indices (i, j), 1 6 i < j 6 n, is picked up. Examining the code of the algorithm, we can make two observations: (1) the elements ai and a j are compared at most once, and this happens only when one of them is chosen as a pivot; (2) if any of the intervening elements ai+1 , · · · , a j−1 is chosen as pivot before either ai or a j , then that execution of pivot will separate the two, and they will never be compared. Let us introduce the indicator random variable Xi j (i < j) to be 1 in a run where ai and a j are def compared, and zero otherwise. Then X = ∑ni=1 ∑ j>i Xi j is the total number of comparisons. 2 . (a) Explain why the observations above imply Pr[Xi j = 1] = j−i+1 (b) Are the Xi j independent? How can you decide this easily? (c) What are the smallest and largest possible values of X ? Exercise 5.83 [4] Use the approach developed in Exercise 5.82 to show that the expected value of the total number the Quicksort algorithm makes is7 E[X ] = 2(n + 1)Hn − 4n. 7 When

the manuscript of this text was typeset (by LATEX2e), it contained n = 2218 index entries; when sorted, by an internal implemetation of Quicksort, 26879 comparisons were made. Using this value of n, we find 2(n + 1)Hn − 4n = 27883, approximately 3.7% higher. An illustration of how sensitive such counts can be occurred when we added the index term ‘Quicksort’ to this exercise: n went up to 2219, and the number of required comparisons to 29144. . . .

232


While this approach can be used to find the variance of this number as well, the calculations become surprisingly heavy. The method taken in §6.6.4 is much better adapted for this calculation. The description of pivot in Exercise 5.82 implies that the number of comparisons made in one phase, when the size of the array is n, equals n − 1: the pivot is compared once with any other value. In fact, while it is possible to organize pivot in this way (such a partition function is presented in [32, §7.1]), practically all efficient ways of doing it require a few more (typically two) comparisons. It is then possible to make relatively few data moves, far fewer than in the reference just given. Example 5.84 When the Quicksort algorithm was introduced on page 229, we only mentioned the role of the partition procedure pivot, but said nothing about its implementation details. These details only manifest themselves through the constants a and b of Eq. (5.26) and for the analysis so far nothing more specific was needed. Let us consider two common implementations of pivot that partition an input array A[s..r]. The algorithm Partition1 is taken from [122] and the next one is from [32]. Partition1(A, s, r) Partition2(A, s, r) 1. p ← A[r]; i ← s − 1; j ← r; 1. p ← A[r] 2. repeat 2. i ← s−1 3. repeat i ← i + 1 until A[i] > p; 3. for j ← s to r − 1 4. repeat j ← j − 1 until A[ j] 6 p; 4. do if A[ j] 6 p 5. t ← A[i]; A[i] ← A[ j]; A[ j] ← t; 5. then { i ← i + 1 6. until j 6 i; 6. exchange (A[i], A[ j]) } 7. A[ j] ← A[i]; A[i] ← A[r]; A[r] ← t; 7. exchange (A[i + 1], A[r]) 8. return i; 8. return i + 1 Both algorithms pick up the last element in the array as the pivot. Each maintains two pointers. The first pointer of Partition1 scans from the left while the second pointer scans from the right. On the other hand, both pointers in Partition2 scan from the left where the first pointer is exactly the same as the first pointer of Partition1. One of the criteria leading to the choice of these two partition procedures is that they both move data by exchanges. Other implementations alternate moves of terms that are larger or smaller than the pivot, and achieve the effect of an exchange with two assignments, instead of three. Their structure tends to be slightly more complicated. Exercise 5.85 [3] Compare the two implementations of pivot shown in Example 5.84 on arrays that hold distinct values. Why does partition1 require a sentinel outside the array, at A[0]? (a) For each algorithm, find Qn , the number of comparisons it performs when given an array of size n, and compute Cn , the expected total number of comparisons required for sorting the array completely. Explain why Qn is deterministic. (b) Describe for each implementation the number of exchanges it makes, denoted by Fn . Explain why Fn is not deterministic. What are its maximum and minimum, and what are


233

the probabilities of these extremes, given that all permutations of the given array are equally likely? Compute fn , the expected number of exchanges in a single partition, and En , the total expected number of exchanges used in the complete sort.

5.5.1 Some Full-History Recurrences We consider some examples that lead to a full-history recurrence solvable by the method that was successfully employed in Quicksort analysis. To start with, let numbers n1 , n2 , . . . be randomly chosen with replacement from the set [1..n] = {1, 2, . . . , n}, and let X be the smallest r for which n1 + n2 + · · · + nr > k, where k is some positive integer less than or equal to n. Clearly X 6 k + 1. Let φ (n1 , n2 , . . . , n j ) = (n1 , n1 +n2 , . . . , n1 +n2 +· · ·+n j ) be a strictly increasing function with j coordinates. For 1 6 j 6 k, consider Pr[X > j + 1]. If X > j, then n1 + n2 + . . . + n j 6 k and φ is a 1-to-1 mapping from the set {hn1 , n2 , . . . , n j i | ∑16i6 j ni 6 k} onto the set of strictly increasing j-tuples in [1..k], and since there exist exactly kj such strictly increasing j-tuples8 , there is the same number of points in the set of j-tuples with the sum less than or equal to k. So we get 1 k Pr[X > j + 1] = j n j j because there are n j-tuples hn1 , n2 , . . . , n j i altogether.

Exercise 5.86 [3] (a) If numbers are drawn randomly (using uniform distribution with replacements) from the set [1..n] = {1, 2, . . . , n} until their sum first exceeds k ∈ N, derive a full-history recurrence for the expected number of draws, and solve it. (b) Solve the same problem for numbers selected from [0..n − 1] (n > 2) until their sum exceeds k − 1. (c) Solve the previous problem when only two numbers are selected: 0 with probability q and 1 with probability p = 1 − q. Exercise 5.87 [4] The procedure cz, given below in a C language implementation, counts zeroes in an array of n integers. (It may be suitable for processors in which comparison operations are much more expensive than arithmetic operations.) The constant b is a threshold value. Let Dn be the number of entry comparisons the procedure does till it is completed (we do not count index comparisons). int cz( int ∗a, int n) { int i, k; if (n < b) // count zeroes { k = 0; for(i = 0; i < n; i++) if (a[i] == 0) k++; return(k); } k = a[0]; for(i = 1; i < n; i++) k ∗ = a[i]; if (k ! = 0) return(0); 8 This is also the number of j-component “weak compositions,” or the number of the ways to form the sum r with j components from the set [1..r], for all j 6 r 6 k. The claim is shown in Exercise 7.31, page 372.

234

CHAPTER 5. RECURRENCES OR DIFFERENCE EQUATIONS i = n/2; return( cz(a, i) + cz( &(a[i]), n − i)); }

(a) Assuming n to be a power of 2, what are the smallest and largest possible values of Dn ? (b) Repeat part (a) when n is not limited to powers of 2. A binary search tree is a binary tree, where each node is associated with a value (a key); all values are distinct. The binary search tree property requires that the values of the keys in the left subtree of the root be smaller than the key at the root, and that all the keys in the right subtree be larger than the key at the root. This holds recursively at each node of the tree. Exercise 5.88 [3] Here is a strange modification of the Binary Search: instead of the compared element being selected in the middle, it is selected at random, with each position equally likely. Let A = A[1..n] be a 1-origin array of size n containing sorted integers. The procedure BSr is used to search the position of a value x known to be in the array, with the initial call BSr(A, 1, n, x). Your task is to find cn , the expected number of comparisons performed in line (*) of BSr.

BSr (int ∗A, int j, int r, int x) { int m; if ( j == r) return (r); m = random ( j..r) // uniform selection (*) if (x > A[m]) return BSr (A, m + 1, r, x); else return BSr (A, j, m, x); } (a) Show the calculation leading to c2 = 2. (b) Write a recurrence for cn and solve it. The computation is a mix of the calculations used to analyze the binary search and the QS sorting algorithms. In the following exercise, we define a number of random variables over such trees. The distributional properties of the trees are based on the assumption that the random binary search tree with n nodes is created by inserting the numbers from [1..n] into an initially empty tree, with all n! permutations equally likely. Exercise 5.89 [4] (Binary search tree (BST) parameters) (a) Let L(n) be the number of leaves in a random binary search tree of n nodes. Show that this random variable satisfies the recurrence L(n) = L(U − 1) + L(n −U ), n > 2, where U is uniformly distributed random variable in the range [1..n]. Derive from it a recurrence for the def expectation l(n) = E[L(n)] and show that it equals (n + 1)/3. (b) Consider the variable F(n), the number of nodes in the tree that are full (have 2 children). Write a recurrence for the random variable, and solve for its expectation. Then show a relation between L(n) and F(n) that obviates the need for this calculation. (c) In a binary search tree, let D(n) denote the internal path length, defined as the sum of the depths of all nodes in a random tree Tn of n nodes. Show that the random variable D(n) satisfies the recurrence D(n) = D(K − 1) + D(n − K) + n − 1, n > 1, D(1) = 0, where K


235

is the key at the root (or K − 1 is the size of the left-hand subtree at the root). As in part (a), derive from this a recurrence for the expected value of D(n) and solve it. (d) The internal path length can be tackled in a different way. We observe that the depth of a node is the same as the number of key comparisons needed to locate it when inserted. Let C(n) be this number, then D(n) = ∑nk=1 C(k). Here you need to write a recurrence for cn , the expected number of these comparisons, solve it, and recover, via the summation, the result of part (c). All of the questions in Exercise 5.89 can be repeated for the parameters of a common binary tree. The set of possible binary trees is identical with the set of binary search trees. However, by convention, we associate with these two sets different probability distributions. A binary search tree, Tn , with n nodes, can be generated by insertion operation that picks up numbers, one at a time, from an array of size n (for simplicity, numbers are assumed to be taken from the set [1..n]) and place them as nodes into an initially empty tree. Hence, any permutation of [1..n] can be used to build a binary search tree. With all n! permutations equally likely, the probability of a given binary search tree is assumed to be equal to C(Tn )/n!, where C(Tn ) is the number of insertion sequences (permutations of [1..n]) that generate Tn . Thus, we are led to the term (binary) search tree statistics. For example, any permutation of [1..3] can be used to build the following binary search trees:

1 R

3

2 (3,1,2)

2

3

1 (3,2,1)

1 R

3

2 (1,3,2)

1 R

2 R

3

(1,2,3)

1

2 R

3

[(2,1,3), (2,3,1)]

So we see that two permutations, h2, 1, 3i and h2, 3, 1i, generate the same tree. Hence this binary tree has 2/6 = 1/3 chance to be chosen. For the set of binary trees, we assume all the trees of a given size (number of nodes) are equally likely. We count these trees in §6.3, finding 2n the number of binary trees of size n to be n /(n + 1), and then revisit in Exercise 6.176 on page 346 the distribution of these tree parameters under the so-called Catalan statistics. Exercise 5.90 [2] We have seen in Exercise 5.89 several equations, all with the same structure: n−1

n−1

an = bn + (1/n) ∑ (ak + an−k−1 ) = bn + (2/n) ∑ ak , k=1

for n > 1,

(5.29)

k=0

where bn and the initial value, a0 , are given. The sequence {bn }n>1 is called in this context the toll function. In this exercise, you are asked to develop the following general formula to handle all equations of this form: n−1

bj . j=1 ( j + 1)( j + 2)

an = (n + 1)a0 + bn + 2(n + 1) ∑

(5.30)

Evaluate this solution for the special case when bn is a polynomial of degree 2 in n, that is, bn = un2 + vn + w. Exercise 5.91 [3] The diameter of a graph is the longest path between two vertices of the

236


graph. If there is a weight function on the edges it can be used to define the diameter, otherwise it is the maximal count of edges on the shortest path between a pair of nodes (ends included; this is true for heights as well, hence the height of a tree consisting of a naked root is one). The following function calculates the diameter of a binary tree: 1. diameter(Tree ∗t) // return diameter of t 2. if (t == ε ) return 0; 3. lheight ← height(t → left); 4. rheight ← height(t → right); 5. ldiameter ← diameter(t → left); 6. rdiameter ← diameter(t → right); 7. return max(lheight + rheight + 1, max(ldiameter,rdiameter)); The following function is used for the height of a tree (the number of nodes on the longest root-to-leaf path): 8. height(Tree ∗t) // returns height of t 9. if (t == ε ) return 0; 10. else return 1 + max(height(t → left),height(t → right));

(a) Argue that diameter computes the correct diameter of the tree. (b) What is the expected number of function calls when diameter is called to compute on a random binary search tree of n nodes? Include in the answer the number of calls to the function height.

5.6 Recurrences in Numerical Analysis Recurrences arise naturally in numerical analysis, the discipline that deals with solutions of mathematical problems by means of arithmetic processes. For example, evaluations of all standard elementary functions first undergo some kind of range reduction—usually with iterative procedures—and then employ rational approximations (discussed in the next section). Our goal is not to present methods and algorithms used in the numerical analysis, but to demonstrate some approaches that lead to recurrences. We also pay limited attention to the convergence of the sequences involved in difference equations and to numerical stability of the corresponding algorithms. We start with the problem of solving equations of the form F(x) = 0,

(5.31)

where F(x) is a real–valued continuous function of a real or complex variable. A number r for which F(r) = 0 is called a root of the equation F(x) = 0 or a zero of F. Often in applications, it is convenient to transform the equation (5.31) into a fixed point problem by writing f (x) ≡ F(x) − x and looking for x such that x = f (x).

(5.32)

5.6. RECURRENCES IN NUMERICAL ANALYSIS

237

A number satisfying this equation is called a fixed point of f . If the function f in Eq. (5.32) is defined on some closed finite interval [a, b], and the values of f lie in the same interval, that is, f : [a, b] → [a, b], then we can generate the sequence {xn } recursively from the relation xn+1 = f (xn ),

n = 0, 1, 2, . . . ,

(5.33)

where x0 is an initial approximation of a fixed point of Eq. (5.32). If f satisfies the Lipschitz condition: | f (x) − f (y)| 6 L|x − y|, with a constant L < 1, then for any choice x0 ∈ [a, b] the sequence defined by procedure (5.33) converges to the unique solution of the equation x = f (x). For example, the sequence {xn }n>0 generated by the recurrence xn+1 = cos(xn ) converges to the unique root of the equation x = cos(x) for arbitrary x0 (which is a simple example of a global attractor). However, in general, the sequence generated by the recurrence (5.33) may converge to different points depending on the initial choice x0 (check the iterative procedure xn+1 = sin xn ). The sequence (5.33) may exhibit fascinating properties, as for example, the logistic function f (x) = rx(1 − x), 0 6 r 6 4. We recommend to experiment with the sequence (5.33) for the √ logistic function with different values of r and, in particular, try r = 1 + 2. Exercise 5.92 [2+] Solve the recurrence xn = (4 − xn−1 )−1 if n = 1, 2, . . ., and x0 = 0. Hint: Look for a fixed point.

For a real-valued continuous function, we seek a point x∗ such that F(x∗ ) = 0. In finiteprecision arithmetic, there may be no machine number x∗ such that F(x∗ ) is exactly zero. An alternative approach would be to seek a very short interval [a, b] in which F changes its sign. Below is pseudocode using the divide-and-conquer technique, called the bisection method, which begins with an initial bracket and successively reduces its length until the solution has been isolated as accurately as desired (or the arithmetic precision will permit). So the input is a function F, an interval [a, b] such that sign(F(a)) 6=sign(F(b)), and an error tolerance tol for the length of the final interval. procedure bisection ([a, b], F, tol) 1. while (b − a) > tol do 2. m = a + (b − a)/2 3. if sign(F(a)) = sign(F(m)) then 4. a=m 5. else b = m 6. end if 7. end The alert reader will notice the similarity of this procedure to the binary search discussed in §5.4.2. Unlike the discrete binary search, the bisection method converges rather slowly compared to other methods to be discussed later. One of them is known as the regula falsi or false position method, which is similar to the bisection procedure by bracketing the root: procedure RegulaFalsi ([a, b], F, tol) 1. while (b − a) > tol do


238

2. 3. 4. 5. 6. 7.

m = [F(b)a − F(a)b]/[F (b) − F(a)] if sign(F(a)) = sign(F(m)) then b=m else a = m end if end

Note: While binary search is commonly considered sufficiently fast, the discrete search method, similar to the false position method, called the “interpolation search,” is sometimes used. It can be shown that when interpolation search is used in a fairly uniform array of size n, the expected number of comparisons is lg(lg n) as shown in Knuth’s book [79]. √ Exercise 5.93 [1] Find approximations to 5 7 correct to within 10−6 using the bisection algorithm and the false position algorithm. Let a = 1 and b = 2, how many iterations are needed for each algorithm? Definition 5.94 We say that a sequence {xn } converges to α with (at least) order p > 1 if |xn − α | 6 εn with εn+1 lim p = c > 0. n→∞ εn

If p = 1, we call such convergence linear. For the bisection method, we have |xn − α | 6 Therefore

1 b−a (bn − an ) = n+1 , 2 2

εn+1 1 = εn 2

so

εn =

b−a . 2n+1

for all n,

which shows that the bisection method converges (at least) linearly. We present another method, the secant method, which is one of the oldest methods known for solutions of Eq. (5.31). It starts with two arbitrary initial approximations x0 , x1 , and continues with the recurrence of the second order: xn+1 = xn −

xn−1 F(xn ) − xn F(xn−1 ) xn − xn−1 F(xn ) = , F(xn ) − F(xn−1 ) F(xn ) − F(xn−1 )

n = 1, 2, 3, . . . .

If the initial approximations x0 , x1 are√sufficiently close to a root, then the secant method converges (locally) with the order (1 + 5)/2 = 1.61803 . . . (golden ratio). The secant method is an example of a two-point iteration procedure since it leads to difference equations of the second order. Let the function F be defined and twice continuously differentiable on the interval [a, b], and F ′ (x) 6= 0 on this interval. The Newton method or Newton-Raphson9 iteration procedure 9 In 1690, Joseph Raphson (1648 – 1715) from England published a tract in which he gave an application of Newton’s power series procedure for solving non-linear equations. We now call it Newton’s method.

5.6. RECURRENCES IN NUMERICAL ANALYSIS

239

can be thought of as a limit case of the secant method if we let xn−1 move into xn . Actually Newton’s method is based on a linear approximation of F(x) at a certain point, which leads to F(xn ) , n = 0, 1, 2, . . . , (5.34) xn+1 = xn − ′ F (xn ) where x0 is some appropriate initial approximation. If started sufficiently close to the root of the equation F(x) = 0, Newton’s method converges to the zero quadratically. This means that the number of correct digits in the answer is nearly doubled at each successive step. In general, this method can lead to chaotic domains of attractions, which is a nightmare for the numerical analyst. Another quadratically convergent method that does not require the knowledge of the derivative, F ′ (x), is Steffensen’s method: xn+1 = xn −

F 2 (xn ) , F(xn + F(xn )) − F(xn )

n = 0, 1, 2, . . . .

(5.35)

n = 0, 1, 2, . . . .

(5.36)

Also, there is the Chebyshev iteration10 scheme: xn+1 = xn −

F(xn ) F 2 (xn )F ′′ (xn ) , − F ′ (xn ) 2 [F ′ (xn )]3

Example 5.95 Suppose we want to find R1/n , a positive nth root of a real positive number R. Then we consider the function F(x) = xn − R. Application of the Newton-Raphson method √ √ 3 for finding R and R yields R 2 R 1 and yn+1 = yn + 2 , xn + xn+1 = 2 xn 3 3yn respectively. Newton’s formula √ for the square root goes back to the Babylonians: if xn is an R, then a better approximation is the average of xn and R/xn : approximation to the number √ R 1 2 xn + xn . It should be noted that numbers xn and R/xn bracket the root (if xn > R, then √ √ √ R/xn < R, and vice versa, if xn < R, then R/xn > R). Instead of the simple arithmetic mean in Newton’s formula, we may try the Farey mean (also known as the mediant) of two positive rationals α = a/b and β = c/d, defined as the ratio a+c , where a, b, c, d are positive integers, and these fractions for α and β are irreducible. b+d √ So if R = a/b is a rational number and xn = an /bn is its rational approximations to R, then the improved Farey estimate will be xn+1 ≡

an+1 an + a bn = , bn+1 bn + b an

n = 1, 2, . . . .

10 P. Chebyshev derived Eq. (5.36) in a paper (student project) written in 1838, but was not published until 1951. This paper took a silver medal in a competition in 1840–41. The formula (5.36) is also sometimes referred to as Newton-Raphson extended formula.


240

A remarkably simple way to calculate rational approximations to ladder11 : √

R≈

yn xn + (R − 1)xn−1 = , xn xn−1 + yn−1

where

√ R > 1 is known as Theon’s

xn = xn−1 + yn−1 , yn = xn + (R − 1)xn−1 .

Slow convergence of Theon’s ladder can be accelerated by jumping over two rungs of the ladder [xn , yn ]: √ y2n y2n + Rx2n 1 yn R xn = = + , x2n = 2xn yn , y2n = Rx2n + y2n , R≈ x2n 2xn yn 2 xn yn √ which leads to Newton’s formula (for zk = y2k /x2k )! Steffensen’s method for R gives xn+1 =

xn (x2n − R) + x2n + R , x2n − R + 2xn

n = 0, 1, 2, . . . .

The square root can also be approximated via the Chebyshev iteration scheme: (x2 − R)2 R 1 − n 2 , n = 0, 1, 2, . . . . xn + xn+1 = 2 xn 4xn As a particular example, letqus demonstrate the application of these methods to find rational √ approximations to 8.6 = 43 5 ≈ 2.932575658 . . .. As a first estimate we take x0 = 3. After

44 = 2.9333333 (Newone iteration, we obtain the following rational approximations: x1 = 15 47 23 ton’s method), x1 = 8 = 2.875 (the Farey estimate), x1 = 16 = 2.9375 (Steffensen’s method), and x1 = 659 225 = 2.92888 (Chebyshev method) all accurate to 2 decimal places. A further it376 3871 = 2.932575753 (Newton’s method), x2 = 125 = 2.983739837 (the eration yields x2 = 1320 64473822491 354587 = Farey estimate), x2 = 120912 = 2.932603877 (Steffensen’s method), and x2 = 21985475625 2.932564371 (Chebyshev method).

Let R be a positive rational number, for the square root of which we want √ to find a positive rational approximation. Let x√be one of such approximations with |x − R| < 1. Choosing n some n, we can expand (x − √ R) ≈ 0 using the binomial theorem (see Example 2.104 on page 57). Then solving for R, we obtain √ R ≈ fn (x);

for example, f4 (x) =

x4 + 6Rx2 + R2 4x3 + 4Rx

if n = 4 .

(5.37)

Then √ successive evaluations of fn (x), fn ( fn (x)), etc. give the values that always converge to R, which are usually referred to as Grant’s approximation. Exercise √ 5.96 [3] Let R be any real positive number and let x ∈ R√+ be any approximation to √ R, say R = x + ε for some ε ∈ R . Show that if ε > 0 and R < 3x, then 21 x + Rx is √ √ closer to R than x; if ε < 0, then 21 x + Rx is always closer to R than x. 11 Little

is known of the life of Theon of Smyrna (circa 140 AD).

5.6. RECURRENCES IN NUMERICAL ANALYSIS Exercise 5.97 [3] Prove that the sequence xn+1 = order.

1 2

241

xn + xRn

converges to

√

R with second

Exercise 5.98 [3] Starting with x0 = 1, perform iterations using Newton’s method, the bisection method, the false position algorithm, the secant method (choose x1 = 2), the Steffensen √ method, and the Chebyshev iteration scheme for computing 2. How many iterations are needed for each method in order to obtain 10−6 accuracy? How many arithmetic operations are involved in the iterations you have already used? Is there any advantage by choosing a better initial approximation x0 = 1.4? Exercise 5.99 [2+] The reciprocal of a number R can be computed without division by the iteration formula xn+1 = xn (2 − xn R), n = 0, 1, 2, . . . . Starting with x0 = 0.2, perform the iteration formula to find the reciprocal of 2. How many iterations do you need to achieve accuracy 10−6 ? Answer the same question by choosing another initial approximation, say x0 = 1. √ Exercise 5.100 [3] The equation x2 = R (for the square root R) can be written equivalently in the form x = f (x) in many different ways, for example: R 1 x+ , f (x) = 2 x

f (x) =

R , x

R f (x) = 2x − , x

f (x) = x2 + x − R,

f (x) =

x+R . x+1

(a) Using fixed point iteration xn+1 = f (xn ), derive a first-order recurrence for each of the above functions. √ (b) Suppose you want to apply these recurrences for computing 2 starting with x0 = 1. How many iterations are needed to obtain 10−6 accuracy? Can every function be used? (c) How many arithmetic operations are involved in the iterations used in the previous part? (d) Derive recurrences based on Newton’s method and redo parts (b) and (c). An additional, more powerful (but more demanding) method is Halley’s method12 (which converges cubically): xn+1 = xn −

F(xn ) n) F ′ (xn ) − 21 F ′′ (xn ) FF(x ′ (x ) n

,

n = 0, 1, 2, . . . .

(5.38)

Usually Halley’s formula makes the convergence of the process of iteration extremely rapid when the first approximation is fairly good. For example, application of Halley’s method to solving the equation (5.31) with the function F(x) = x2 − px − q leads to the recurrence xn+1 =

x3n + 3qxn − pq , 3x2n − 3pxn + q + p2

n = 0, 1, 2, . . . .

(5.39)

12 Edmund Halley discovered this method in 1694. He is well known for first computing the orbit of the comet that bears his name.

242


√ Exercise 5.101 [2] (a) Starting with x0 = 1, compute 2 using Halley’s method. How many iterations are needed to obtain 10−6 accuracy? (b) How many arithmetic operations are involved in the iterations used in the previous part? (c) Repeat the two previous parts for the initial approximation x0 = 1.5. In practical applications various hybrid methods have been developed. Both Newton’s method and the secant method fail to bracket a root. Therefore hybrid schemes involving the bisection method or modified False Position method may help to locate a root. A commonly used stopping criterion in iteration methods is |xn+1 − xn | < ε ,

(5.40)

where xn is the nth term of the sequence generated by the method, and ε > 0 is the tolerance. Some methods, like Newton’s method, fail to satisfy (5.40), while others, like the bisection method, support the criterion. There exist sequences, e.g., the partial sums of the harmonic series, for which (5.40) is true, but which nonetheless diverge. The following exercise shows that Newton’s method has a repelling fixed point at zero. √ 2 Exercise 5.102 [2+] The function f (x) = 3 x e−x has a single zero at x = 0. Show that iteration under Newton’s method can never converge to this zero unless we are lucky enough to choose the initial value x0 = 0. However, if the convergence criterion (5.40) is used, Newton’s method will appear to converge.

5.7 Continued Fractions Evaluating mathematical functions on a digital computer is a very important task in applications. There are many known effective approaches for this problem, for example, Chebyshev approximations (or minimax approximations). However, if a function has to be evaluated millions of times, then another objective of the approximation is to achieve maximum speed. Rational function approximations lead to smaller maximum error than polynomial approximations, and are usually faster. Let Rm,n (x) =

Pm (x) , Qn (x)

m

Pm (x) =

∑ k=0

pk xk ,

n

Qn (x) =

∑ qk xk ,

(5.41)

k=0

be a rational approximation to a function f (x), where Pm (x) and Qn (x) are polynomials of degree at most m and n respectively. The rational functions (5.41) are called the Padé13 approximants to the function f (x) if the following requirement holds f (x)Qn (x) − Pm (x) = O xn+m+1 as x → 0. (5.42)

The Padé approximants Rm,n (x) of the function f (x) form two-dimensional array, called the Padé table of f . 13 Henri

Eugene Padé (1863 – 1953) was a French mathematician, a student of Hermite.

5.7. CONTINUED FRACTIONS

243

Example 5.103 For the function f (x) = cos 2x + sin x, the coefficients of the Padé approximants R1,1 (x), R2,1 (x), R3,1 (x) are as follows: R1,1 (x) =

p0 + p1 x , q0 + q1 x

R2,1 (x) =

p0 + p1 x + p2 x2 , q0 + q1 x

R3,1 (x) =

p0 + p1 x + p2 x2 + p3 x3 . q0 + q1 x

Let us consider the approximant R1,1 (x). From the Maclaurin series for sine and cosine functions: x2k+1 (2x)2k , sin x = ∑ (−1)k cos 2x = ∑ (−1)k (2k)! (2k + 1)! k>0 k>0 and Eq. (5.42), it follows that 1 3 (2x)4 2 − · · · − (p0 + p1 x) = O x3 . (q0 + q1 x) 1 + x − 2x − x + | {z } | {z } 6 24 {z } | Q1 (x) P1 (x) f (x)=cos 2x+sin x

Equating coefficients of like powers of x, we obtain q0 − p0 = 0,

q0 + q1 − p1 = 0,

−2q0 + q1 = 0.

Therefore p0 = q0 and p1 = 3q0 . So R1,1 (x) = 1+3x 1+2x . For R2,1 (x) we have 1 3 (2x)4 2 − · · · − p0 + p1 x + p2 x2 = O x4 . (q0 + q1 x) 1 + x − 2x − x + 6 24 Similarly, we find R2,1 (x) =

12+11x−25x2 . 12−x

The third approximant is evaluated to be R3,1 (x) =

1+5x+2x2 −49x3 /6 . 1+4x

Examples (for instance, Padé approximations R3,1 (x) = R2,0 (x) = 1 − x2 /2 for cos x) show that polynomials Pm (x) and Qn (x) in Eq. (5.41) may have common factors. We assume without loss of generality that Qn (0) = 1. The number of coefficients at our disposal in Rm,n (x) is m + n + 1. These coefficients are uniquely determined by the requirement (5.42). A Padé table is called a normal Padé table if the degrees in numerator Pm (x) and denominator Qn (x) are exactly m and n respectively, and they cannot be reduced by cancellation. The sum N = m + n is called the index of Rm,n (x). For a particular function, all approximations with the same index require similar amount of computation and achieve similar accuracy. For example, the Padé rational approximation, R6,6 , for the sine is (for |x| 6 π /6) # " 34911 4 479249 29593 2 x + 7613320 x − 11511339840 x6 1 − 207636 . sin x ≈ x 1671 2 97 2623 1 + 69212 x + 351384 x4 + 1644477120 x6 The exponential function ex has the following Padé approximant Rm,n (x) = Pm (x)/Qn (x) with n m m (n + m − k)! k k m (n + m − k)! k x ; Qn (x) = ∑ (−1) x. Pm (x) = ∑ (n + k)! (n + k)! k k=0 k=0 k

244


Exercise 5.104 [3] Find the Padé approximants R6,6 (z) (or/and R7,6 (z)) for the following functions: (a) ez ; (b) cos z; (c) tan z; (d) tanh z; (e) z ln(1 + z); (f) (1 + x)3/2 . Fortunately, a normal Padé approximation for a given function can be obtained by the use of continued fractions that we describe below. Many of the special functions and recurrences that occur in applications of mathematics and computer science are defined by infinite processes, such as series, integrals, and iterations. The continued fraction14 is one of these processes and it can be defined as composition, Φn (0), of a sequence {φk (w)} of mappings: Φn (w) = φ1 ◦ φ2 ◦ · · · ◦ φn (w),

where φk (w) =

ak , bk + w

(5.43)

similar to the way in which sums and products are handled, for example. The partial sums a1 + a2 + · · · + an = φ1 ◦ φ2 ◦· · · ◦ φn (0) = Φn (0), with φk (w) = w + ak , and the partial products p1 · p2 · · · pn = Φn (1), where φk (w) = w · pk , are all compositions of a sequence of mappings Φn (w) = φ1 ◦ φ2 ◦ · · · ◦ φn (w). A continued fraction is often defined as the limit of “an expression of the form:” an . xn+1 = xn + an−1 xn−1 + an−2 xn−2 + .. . x3 + a2 x2 + a1 x1 + x0 Exercise 5.105 [2] Show that the above continued fraction is equivalent to the nonlinear recurrence xn xn+1 = x2n + an for n > 1, when the initial values x0 and x1 are known. In view of the unwieldiness of this notation, various authors have proposed other ways of writing continued fraction; it has become customary to write in a typographically convenient form as follows: a1 a1 a2 a3 a1 ak ∞ ak ∞ = b0 + or b0 ; ; . · · · or b0 ; b0 + b1 + b2 + b3 + bk k=1 b1 bk 2 a2 b1 + a3 b2 + .. . (5.44) A continued fraction (5.44) is called proper if b0 = 0, otherwise we call such a fraction improper. The proper continued fraction is usually denoted by K∞ k=1 (ak /bk ). The fraction ak /bk is called the kth partial quotient of the continued fraction (5.44). We call ak and bk the 14 The

word “fractio continua” (continued fraction in Latin) was used for the first time by L. Euler in 1737.


245

kth partial numerator and denominator respectively, or simply coefficients of the continued fraction. The terminating continued fraction b0 +

an An a1 a2 a3 ··· ≡ = Kn b1 + b2 + b3 + + bn Bn

(5.45)

is called the nth convergent (or approximant) of the continued fraction (5.44), and An and Bn are called nth canonical numerator and nth canonical denominator, respectively. Hence, the continued fraction is the limit limn→∞ An /Bn (if it exists). For example, A0 b0 = , B0 1

A1 b0 b1 + a1 = , B1 b1

A2 b0 b1 b2 + b0 a2 + a1 b2 b2 A1 + a2 A0 = = .... B2 b1 b2 + a2 b2 B1 + a2 B0

The coefficients in Eq. (5.45) satisfy the following recurrences An = bn An−1 + an An−2 ,

Bn = bn Bn−1 + an Bn−2 ,

n = 1, 2, 3, . . . ,

(5.46)

subject to the initial conditions (suggested by L. Euler) A−1 = 1, A0 = b0 ,

B−1 = 0, B0 = 1.

Canonical numerators and denominators are most easily displayed by the use of matrices # #" # " " An−1 an An−2 bn an+1 An an+1 An−1 = 1 0 Bn−1 an Bn−2 Bn an+1 Bn−1 # # " #" " bn an+1 b0 a1 b1 a2 . (5.47) ··· = 1 0 1 0 1 0 Setting an = bn = 1 in Eq. (5.46), we obtain a recurrence for the Fibonacci numbers. Therefore, the canonical numerators {An } and denominators {Bn } of the continued fraction K(1/1) = 1 1 1 1+ 1+ 1+ · · · are solutions of the Fibonacci recurrence Fn = Fn−1 +Fn−2. Then from Eq. (5.47), it follows #n+1 # " " 1 1 Fn+1 Fn . = 1 0 Fn Fn−1 Exercise 5.106 [4] Solve the difference equation an+2 = an+1 Fn+2 + an ,

n > 1,

a1 = α ,

a2 = β ,

where Fn is the nth Fibonacci number. Transform your solution into a continued fraction. Hint: Try to represent solution as a determinant. Let all the coefficients of a continued fraction, K, be positive, then A2k−1 A2k 6K6 B2k B2k−1

for any natural k.


246

Many functions and constants can be evaluated via continued fractions, for example, Lord William Brouncker discovered in approximately 1650 that ∞ 4 4 1 32 52 4 (2n − 1)2 (2n − 1)2 = ... ... = ; . π= 1+2+ 2 + 2 + + 2 + 1 2 1 n=1 1+ 9 2+ 25 2+ . 2+ .. There are known many others, for instance, ∞ 4 12 22 32 n2 . . . = 4; π= 1+ 3 + 5 + 7 + 2n + 1 n=1 since continued fraction representation is not unique. Continued fractions for many functions can be used to derive the associated Padé approximations. See Appendix J for other examples. Exercise 5.107 [2] (a) What is the recurrence satisfied by the sequence of ratios of sequential ? Fibonacci numbers Gn = FFn+1 n (b) What is the limit limn→∞ Gn ? √ Exercise 5.108 [2] Express the golden ratio, φ = (1 + 5)/2, and its reciprocal in the form of continued fraction. Exercise 5.109 [2] Use the Euclidean algorithm to derive a terminating regular continued fraction for the ratio of two natural numbers, a/b, a > b. Exercise 5.110 [3] Using Taylor’s series x x2 x3 x2 x3 x4 ln(1 + x) = x − + − + · · · = x 1 − + − + · · · , 2 3 4 2 3 4 (a) derive the continued fraction for ln(1 + x), x > −1; (b) find the Padé approximation R8,7 (x); (c) how many arithmetic operations are needed to perform 7 iterations in the continued fraction approximation of ln(1 + x)? (d) Suppose you want to find an approximate value of ln 2, which of the following procedures (1) continued fractions; (2) the Padé approximation; (3) classical Taylor’s series are you going to use? Explain your answer! We can multiply coefficients an , bn , and an+1 by an arbitrary number, not equal to zero, without altering the value of the continued fraction. With such transformations, we can reduce the continued fraction (5.44) to a form in which all partial numerators are equal to 1. So the expression (5.44) becomes K = α0 +

with

α2k−1 =

1 1 1 def · · · = [α0 ; α1 , α2 , . . .], α1 + α2 + α3 +

a2 a4 . . . a2k−2 b2k−1 , a1 a3 . . . a2k−1

α2k =

a1 a3 . . . a2k−1 b2k , a2 a4 . . . a2k

(5.48)

k > 1.


247

For example, 1

2 = [0; 1, 2] = 3

1

. 1 1 1+ 1+ 2 1+1 Of course, we can always reduce the continued fraction (5.44) to the form in which all partial denominators are equal to one: K=

= [0; 1, 1, 1] =

β1 β2 βn ··· ··· . 1 + 1 + + 1 +

(5.49)

From the definition of the continued fraction, it follows that the coefficients in Eqs. (5.48) and (5.49) are related via

β1 =

α1 , 1

βn =

1 , αn αn−1

n = 2, 3, . . . .

A continued fraction of the form (5.48) or (5.49) is called the ordinary continued fraction. These continued fractions may be written in compact form as arrays: [α0 ; α1 , α2 , . . .]. Ordinary continued fractions are mostly used in numerical computations. Continued fractions where all partial numerators are 1 and all partial denominators are positive integers are called regular continued fractions. For example, the history of the number π reveals two rational approximations: Archimedes’ estimate 22/7, which gives two-decimal accuracy, and the Chinese estimate 355/113, which gives six decimals. We say that two real numbers α and a agree with K-digit accuracy if |α − a|
1. The number y in turn can be represented as the sum ⌊y⌋ + {y} giving x = ⌊x⌋ +

1 1 = ⌊x⌋ + . y ⌊y⌋ + {y}

1 , a rational number, and the continued fraction terminates. But If {y} = 0, then x = ⌊x⌋ + ⌊y⌋ if {y} > 0, then {y} = 1/z for some z > 1, and

x = ⌊x⌋ +

1 1 . = ⌊x⌋ + 1 y ⌊y⌋ + ⌊z⌋+{z}

248


By repeating this process, we get a continued fraction representation of x in the form (5.48), where α0 = ⌊x⌋, α1 = ⌊y⌋, α2 = ⌊z⌋, and so on. If x is rational, the process terminates after a finite number of steps and the continued fraction is finite. But if x is irrational, the continued fraction is infinite. For example,

π = [3; 7, 15, 1, 292, 1, 1, 1, 2, 1, 3, 1, 14, 2, 1, 1, 2, 2, 2, . . .]. Truncating the continued fraction of an irrational number α at the integer αn produces a rational number An = [α0 ; α1 , α2 , α3 , . . . , αn ], Bn called the nth convergent to α . For example, the third convergent to π is [3; 7, 15, 1] = 355/113, a fraction closer to π than any other fraction with denominator less than 113. By induction we obtain the fundamental relation (which is a particular case of Eq. (5.47)): #" # " #" # # " " α0 1 α1 1 αn−1 1 αn 1 An An−1 ··· . (5.50) = 1 0 1 0 1 0 1 0 Bn Bn−1 Since the determinant of the matrix is the product of the determinants of its factors, we obtain the important relations: A A A 1 An An−1 (−1)n+1 n n−1 n n − = =⇒ α − < . = (−1) , Bn Bn−1 Bn Bn−1 Bn Bn−1 Bn Bn Bn+1

Theorem 5.111 Let An /Bn be the n-th convergent of a continued fraction for an irrational number α , and let K be the number of digits in the denominator Bn . Then the decimal representations of An /Bn and α agree with at least (2K − 2)-digit accuracy for every n.

P ROOF : is given in the article [8] by Tom M. Apostol & Mamikon A. Mnatsakanian. Exercise 5.112 [3] Using continued fractions, find 7-, 9-, and 10-digit accuracy approximations for the following numbers: √ √ (a) 26, (b) 2, (c) e, (d) ln 2. Theorem 5.113 Let f (z) = 1 + c1 z + c2 z2 + c3 z3 + · · · be a formal power series with the property that its Padé approximants R0,0 (z), R1,0 (z), R1,1 (z), R2,1 (z), R2,2 (z), R3,2 (z), . . . are all normal. Then f (z) has a corresponding regular continued fraction f (z) = 1 + K∞ n=1

an z = lim Kn (z), n→∞ 1

its approximants satisfy the equations K2m = Rm,m (z), K2m+1 = Rm+1,m (z), m = 0, 1, 2, . . ..


249

P ROOF : See [70]. 2 2 Suppose we need to p x = px + q, with p + 4q > 0. Its two psolve the quadratic equation solutions x1 = (p + p2 + 4q)/2 and x2 = (p − p2 + 4q)/2 can be expressed via continued fractions as follows. We choose some number a 6= p/2, and consider the vector recurrence

An = aAn−1 + qBn−1 ,

Bn = An−1 + (a − p)Bn−1 ,

n = 1, 2, . . . .

For the ratio yn = An /Bn we obtain the recurrence a ABn−1 +q aAn−1 + qBn−1 An = = An−1 n−1 Bn An−1 + (a − p)Bn−1 Bn−1 + a − p

or yn =

ayn−1 + q . yn−1 + a − p

(5.51)

Therefore the limit y = lim→∞ yn , if exists, satisfies the equation y=

ay + q . y+a− p

(5.52)

If a > p/2, then yn → x1 as n → ∞; if a < p/2, then yn → x2 as n → ∞.

Let us take, for simplicity, the initial approximation a = p. Then one of the roots of the quadratic equation x2 = px + q has the following continued fraction representation (if it converges): q (5.53) x = p+ q p+ q p+ . p+ .. with the successive convergents An /Bn , n = 0, 1, . . . (A0 = p, B0 = 1, A−1 = 1, and B−1 = 0). Using Eq. (5.47), we get #n+1 # " # " " p q An qBn An qAn−1 (5.54) = = 1 0 Bn An − pBn Bn qBn−1 since qBn−1 = An − pAn−1 = An − pBn and An = Bn−1 . Squaring this, we obtain #2n+2 #2 " # " " An qBn p q A2n qB2n , = = Bn An − pBn 1 0 B2n A2n − pB2n and expanding, we find A2n + qB2n A2n = . B2n Bn (2An − pBn )

(5.55)

On the other hand, Newton’s formula (5.34) for computing the roots by successive approximation is x2 + q x2 − pxk − q = k , k = 0, 1, 2, . . . . xk+1 = xk − k 2xk − p 2xk − p

250


Hence if we set xk = An /Bn , we get from Eq. (5.55) that xk+1 = A2n /B2n . Thus, proving the claim that each successive application of Newton’s method is equivalent to doubling the length of the continued fraction. Exercise 5.114 [2+] Let An /Bn be the n-th successive convergent of the continued fraction (5.53). If Halley’s method (5.39) is used to compute a root of quadratic equation x2 − px−q = 0, starting with x0 , and it is equal to one of the approximants, An /Bn of Eq. (5.53), then show that x1 = A3n /B3n , x2 = A6n /B6n , . . . . Prove that each successive application of Halley’s method is equivalent to tripling the length of the continued fraction approximants. y which corresponds to (5.52) with a = 1, Example 5.115 Consider the identity y = 1−y+y p = −1, q = 0 and therefore p2 − 4q = 1 > 0. We use it repeatedly to obtain the continued fraction y y y ··· ··· . K= 1−y + 1−y + + 1−y + Its nth approximant, yn , with a little algebraic effort, can be found explicitly to be

yn =

y (1 − (−y)n ) . 1 − (−y)n+1

Therefore, the continued fraction K converges to y in the domain |y| < 1 and to −1 in the domain |y| > 1. √ Exercise 5.116 [2+] We derive the recurrence for computing R based on continued fractions as follows: √ √ √ R−1 R−1 R−1 R−1 =⇒ .... R = 1 + ( R − 1) = 1 + √ R = 1+ 2 + 2 + 2 + R+1 √ Starting with x0 = 3/2, compute 2 based on the recurrence xn+1 = 1 +

1 , 1 + xn

n = 0, 1, 2, . . . ,

to obtain 10−6 accuracy. How many iterations do you need? Now consider Grant’s iteration (5.37) with f4 (x). If you choose x0 = 3/2 √ as the initial approximation, how many iterations do you need? Answer similar question for 5 with starting point x0 = 1. Exercise (5.51) with the initial condition y0 = 1 to find both √ 5.117 [2] Use the recurrence −6 roots ± 5 with an error < 10 . How many iterations do you need? Your answer depends on the choice of a in Eq. (5.51). Exercise 5.118 [4] Use the difference vector equation

 Pn = aPn−1 + α Qn−1 + α Rn−1 ,   n = 1, 2, . . . , Qn = Pn−1 + aQn−1 + α Rn−1 ,   Rn = Pn−1 + Qn−1 + α Rn−1 , √ √ 3 α 2 based on continued fraction to derive the approximation of the cubic roots 3 α and √ Hint: The ratio Qn /Rn converges to 3 α and the ratio Pn /Rn converges to representation. √ 3 2 α .

5.8. PARTIAL DIFFERENCE EQUATIONS

251

5.8 Partial Difference Equations In many applications, we can see recurrences with multiple indices, called partial difference equations. We present some examples in this section. When two indices are involved, the corresponding recurrence is referred to as a bivariate recurrence relation. We say that a bivariate recurrence is of the order (r, p) if it has the order r in one index and the order p in the other. We start with the following famous game. Example 5.119 (Russian roulette) There are many known games when one of the contestants is singled out and eliminated in a particular stage (in sports, they are usually called playoffs). We apply mathematical analysis to one of these games, called Russian roulette. It is played by two duelists using a revolver equipped with a rotary magazine of six shots. The revolver is loaded with one shot. The first duelist rotates the magazine at random, points the revolver at his head and presses the trigger. If, afterwards, he is still alive, he hands the revolver to the other duelist who acts in the same stupid way. The game ends when one of the duelists is eliminated. In order to avoid unpleasant associations, we will now replace the revolvers with biased coins, which turns up tails with probability p and heads with probability q = 1 − p. Suppose there are n participants that are listed in some order: A1 , A2 , . . . , An . When a player tosses the coin and obtains tails, he or she goes out of the game and doesn’t flip again. The remaining players continue to toss until all go out. The last person remaining on the list is the winner. We want to determine the probability pn,k , k = 1, 2, . . . , n, that Ak is the winner. We can determine the values of pn,k recursively, beginning with two players (n = 2). If A1 obtains tails at the first toss, he or she goes out and A2 is the winner. If A1 obtains heads, the roles of the players become interchanged and this leads to the recurrence p2,1 = p · 0 + q(1 − p2,1 ), therefore

1 q , p2,2 = . 1+q 1+q If p = 1/6, we have classical Russian roulette with probabilities p2,1 = 5/11 and p2,2 = 6/11.

p2,1 =

If there are three players (n = 3) in the order A1 , A2 , A3 , we have to consider the following two main cases (to which the game is always reduced): (1) The first toss results in tails then the player A1 goes out and the players A2 and A3 remain. So the game is reduced to the problem with two players considered above. (2) The first toss results in heads and all three players remain and we repeat the game with the list A2 , A3 , A1 . Applying these considerations three times, we obtain the system of equations: p3,1 = p × 0 + q p3,3 ,

p3,2 = p p2,1 + q p3,1 ,

p3,3 = p p2,2 + q p3,2 .


252

Since we already know the probabilities p2,1 and p2,2 , we can solve the above system to obtain p3,1 =

q3 pq + , 1 + q 1 + q + q2

p3,2 =

q , 1 + q + q2

p3,3 =

q2 p + . 1 + q 1 + q + q2

The pattern is obvious and we derive the recurrence: pn,k = p pn−1,k−1 + q pn,k−1 ,

k = 2, 3, . . . , n,

n > 1,

pn,1 = q pn,n ,

p1,1 = 1. (5.56)

The recurrence (5.56) is not convenient for computations because it requires the knowledge of all probabilities on previous steps. So we derive another recursion. Let us start with pn,1 and the player A1 . If at the first toss A1 gets tails, he is eliminated, if he obtains heads, he will appear at the end of the second round. Suppose that there are j + 1 people on the list after the first round. This may happen if j of the players A2 , . . . An get heads during the first round; according to the binomial distribution (see subsection 4.1.1) this happens with probability n − 1 j n−1− j q p . k On the other hand, when there are j + 1 people on the list and A1 is first in it, the probability that A1 wins the game is p j+1,1 . Summing over the binomial probabilities, we obtain the following full-history recurrence: n−1 n − 1 j n−1− j pn,1 = q ∑ q p p j+1,1 , n > 1, p1,1 = 1. (5.57) j j=0 Now we consider the player Ak . Suppose that, in the first round, j of the players A1 , . . . , Ak−1 j k−1− j . When A tosses his coin in the obtain tails. This happens with probability k−1 k j p q first round, he is first in a game with n − j people, hence he wins with the probability pn− j,1 . Summing over the binomial probabilities, we derive the recursion k−1 k − 1 j k−1− j pn,k = ∑ pq pn− j,1 , n > 1. (5.58) j j=0 The following exercise asks for solving these recurrences. Exercise 5.120 [3] Show that the recurrences (5.57) and (5.58) have the solution: pn,k = p ∑ (1 − q j+1 )k−1 (1 − q j )n−k q j ,

k = 1, 2, . . . , n,

(5.59)

j>0

where the summation runs from zero to ∞ for k = n. Exercise 5.121 [1+] Example 5.17, page 206, shows a recurrence for f (k), the number of block log piles that can be constructed with a bottom layer of k logs. A similar relation for the count p(n), total number of n-log piles, does not seem available. Your task here is to sidestep this difficulty by finding a two-variable recurrence for the counting function f (k, n),


253

the number of such n-log piles with a bottom layer of k logs. This function yields p(n) when summed on all k 6 n. Derive the following recurrence, and determine the appropriate boundary values: k−1

f (k, n) = δk,n + ∑ (k − j) f ( j, n − k),

1 6 k 6 n.

j=1

The solution of this recurrence is considered in Exercise 6.118 on page 323. Exercise 5.122 [2+] This problem is a generalization of Exercise 3.130, page 127. We consider a set of n men and m women (n + m is assumed to be even), which is partitioned at random into (n + m)/2 disjoint pairs. Let Xn,m be the number of male-female couples; define (2) 2 ] be its second moment. Note that µn,m = E[Xn,m ] as its expected value, and let µn,m = E[Xn,m X0,m = Xn,0 = 0 and Xn,1 = X1,m = 1. (2) (a) Show that the doubly-indexed sequences, µn,m and µn,m , need to satisfy the recurrences

µn,m =

1 [n(n − 1) µn−2,m + 2nm (µn−1,m−1 + 1) + m(m − 1) µn,m−2 ] (n+m)(n+m−1)

and (2)

µn,m =

i h 1 (2) (2) n(n − 1) µn−2,m + 2nm E[(Xn−1,m−1 + 1)2 ] + m(m − 1) µn,m−2 . (n+m)(n+m−1)

(b) Show by direct calculation that def

µn,m = E[Xn,m ] =

nm n+m−1

and

(2)

def

2 µn,m = E[Xn,m ]=

nm(nm − 2) (n + m − 1)(n + m − 3)

leading to 2 V [Xn,m ] = E[Xn,m ] − (E[Xn,m])2 =

2nm(n − 1)(m − 1) . (n + m − 1)2 (n + m − 3)

Finally, show that the expectations you derived satisfy the recurrences of part (a). Exercise 5.123 [2+] Consider a problem of infinite sum evaluation hn (x) = ∑k>0 kn xk . Clearly ∞ n+1 xk = k −1 ′ h0 = ∑∞ k=0 x = (1 − x) . Differentiating term-by-term, we see that xhn (x) = ∑k=0 k hn+1 (x). Applying the recurrence repeatedly, we get the rather complicated ratio: hn (x) =

∑nk=1 An,k xk , (1 − x)n+1

where the double sequence {An,k } satisfies the following recurrence relation: An+1,k = kAn,k + (n + 2 − k)An,k−1 ,

An,1 = An,n = 1,

Prove the recurrence (5.60) for the sequence {An,k }.

for 2 6 k 6 n.

(5.60)

254


Exercise 5.124 [3+] In judging whether a sequence of 0’s and 1’s is random, it is common to use many statistical tests. One of them is based on the distribution of singles. A single is a run of length 1, that is, it differs from its adjacent sequence elements. For instance, the sequence (0,0,1,0,0,1,1,0) has two singletons. Assigning the values 0 and 1 to heads and tails, one generates a binary sequence by flipping a coin. (a) Let S(n, k) denote the number of sequences of n tosses that contain exactly k singles. For n > 1, prove the recurrence S(n + 1, k) = S(n, k) + S(n, k − 1) + S(n − 1, k) − S(n − 1, k − 1),

S(n, 0) = 2 Fn−1 , (5.61)

where Fn is the nth Fibonacci number. (b) Show that ∑k k2 S(n, k) = (n + 1)(n + 8) 2n−4 for n > 3. (c) Let Sn denote the number of singles in a sequence of n tosses of a coin, where n > 2. For any sequence of tosses ht1 ,t2 , . . . ,tn i, let Xi = 1 if ti is a single, otherwise Xi = 0. Find the expected value and the variance of the random variable Sn = X1 + X2 + · · · + Xn .

Exercise 5.125 [4+] (Quickselect analysis) In the 1962 article where he introduced Quicksort, C. A. R. Hoare also showed how the same partition function can be used to select the kth smallest element in an array of n numbers. He called his algorithm FIND, but the term Quickselect is more common now. Following a call to partition the array A[1..n], which returns the position r for the pivot, if r = k, then the pivot is returned as the selected element. If r > k, the search for the kth element continues in the array A[1..r − 1]; if r < k the search continues for the (k − r)th element in A[r + 1..n]. This tail-recursion means that Quickselect is simpler than Quicksort, since it does not need a stack. Derive a recurrence for cn,k , the expected number of comparisons needed by Quickselect to find k in a random permutation of [1..n], and rn,k , the expected number of calls a recursive version would perform in the process, including the initial call. Assume that the partition function pivot makes n − 1 term comparisons15 on an array of size n. Here are the final results, for k ∈ [1..n]: rn,k = Hk + Hn+1−k − 1;

cn,k = 2 [(n + 1)Hn − (n + 3 − k)Hn−k+1 − (k + 2)Hk + n + 3],

where Hn is the nth harmonic number. Hint: The solution could be found similarly with the method used to solve Eq. (5.26) on page 230, but getting past the full-history feature of the recurrence is tricky. Another approach was suggested by D. Knuth [76]. Note: While this problem can be solved with the methods we have seen, it calls for wizardry. We return to it in Exercise 6.175, page 346, where more powerful tools are at our disposal.

5.8.1 The Relations with Continuous Calculations Recurrences appear naturally in numerical approximations to the solutions of ordinary or partial differential equations. As an example, we start with the initial value problem for the 15 See

the preamble to Example 5.84, page 232.


255

first order ordinary differential equation dy = f (t, y), dt

a 6 t 6 b,

y(a) = y0 ,

We assume that f (t, y) is a continuous function satisfying the Lipschitz condition in variable y. We select n mesh points {t0 ,t1 , . . . ,tn } that are equally distributed throughout the interval [a, b]: tk = a + kh, h = (b − a)/n, k = 0, 1, 2, . . . , n. The simplest (and oldest) approximation to the problem gives the Euler’s method which construct uk ≈ y(tk ) for each k = 1, 2, . . . n by solving the first order difference equation: uk+1 = uk + h f (tk , uk ),

k = 0, 1, 2, . . . , n − 1,

u0 = y0 .

The theory of partial difference equations, that is, difference equations in two or more variables, has been extensively used in modern numerical analysis and in its various applications in such fields as digital image processing, X -ray image enhancement, material science, partial differential equations, and many others. To illustrate, we consider the first order hyperbolic equation

∂ T (x,t) ∂ T (x,t) + + T (x,t) = 0 ∂t ∂x with initial and boundary conditions T (x, 0) = f (x) and T (0,t) = ϕ (t), which describes some thermal processes, for instance, in chemical reactors, heat exchanges, and pipe furnaces. Here T (x,t) is the temperature at x (space) and t (time). A difference equation associated with the given equation is obtained if we replace the continuous range of the arguments, (x,t), by a computational grid, (n∆x, k∆t), and instead of the function T (x,t), we consider the discrete function T (n, k) = T (n∆x, k∆t). Replacing the partial derivatives by the backward and forward difference quotients

∂ T (x,t) T (n, k + 1) − T (n, k) ≈ , ∂t ∆t

∂ T (x,t) T (n, k) − T (n − 1, k) ≈ , ∂x ∆x

we obtain the partial difference equation T (n, k + 1) = aT (n, k) + bT (n − 1, k) , where a = 1 − ∆t − ∆t/∆x and b = ∆t/∆x, with initial and boundary conditions T (n, 0) = f (n∆x), T (0, k) = ϕ (k∆t).

As another example, we consider the problem of numerical integration. Suppose that we Rb need to compute the integral a f (x) dx for some smooth function f (x). We present here a recursive trapezoid formula for 2n equal subintervals known as the Romberg algorithm,


256

which is based on Euler summation formula, discussed in §10.4. Setting h = (b − a)/2n , we define 2n −1 h def n > 1. R(n, 0) = h ∑ f (a + ih) + [ f (a) + f (b)], 2 i=1 Note that R(n, 0) is obtained easily from R(n − 1, 0): n−1

2 1 R(n, 0) = R(n − 1, 0) + h ∑ f [a + (2k − 1)h], 2 k=1

R(0, 0) =

b−a [ f (a) + f (b)]. 2

Both expressions, R(n, 0) and R(n − 1, 0), are approximations to the integral Z b

f (x) dx = R(n − 1, 0) + a2 h2 + a4 h4 + a6 h6 + · · · ,

(5.62)

4 6 2 h h h + a4 + a6 + ··· . f (x) dx = R(n, 0) + a2 2 2 2

(5.63)

a

Z b a

The discussion in §10.4 clarifies why only even powers appear in this expansion. Subtracting Eq. (5.62) from 4 times (5.63), we obtain another approximation Z b a

5 1 f (x) dx = R(n, 1) − a4 h4 − a6 h6 − · · · , 4 16

with R(n, 1) = R(n, 0) + [R(n, 0) − R(n − 1, 0)]/3; this is more accurate than either of R(n, 0) or R(n − 1, 0) because the expression for its error begins with an h4 term, whereas Eqs. (5.62) and (5.63) have h2 terms. The same procedure, called the Romberg integration scheme, can be repeated over and over again to “annihilate” higher terms in the error. Hence, we obtain a triangular array of numbers R(n, m), m 6 n, via the partial recurrence: R(n, m) = R(n, m − 1) +

1 [R(n, m − 1) − R(n − 1, m − 1)]. 4m − 1

(5.64)

If the integrand, f , satisfies the smoothness criterion, the limit of each column in the array equals the integral: lim R(n, m) =

n→∞

Z b

f (x) dx,

m > 0.

a

5.8.2 Counting Arrangements—with and without Repetition In this subsection, we show how partial difference equations can help us to count samples taken from [1..n], the set of the first n integers. To start, we consider a familiar sequence—the binomial coefficients—that count the number of unordered selections without replacement taken from a set of finite distinct items.


257

Example 5.126 (Recurrence for the binomial coefficients) We have already seen in §2.5 both algebraic and combinatorial properties of these coefficients. However, the binomial coefficients can be defined as a solution of the difference equation in two indices: an,k = an−1,k + an−1,k−1 for all 0 < k < n;

n = 1, 2, . . . ,

(5.65)

with boundary conditions a0,0 = an,n = 1. The bivariate recurrence (5.65) is of the order (1, 1). This recurrence is of course just the addition formula we had in Eq. (2.62). Let us use basic combinatorial considerations to rederive the recurrence (it may remind you of Exercise 2.82, page 53). Consider a bin that contains an adequate number of black and white balls (the balls are identical except the color). What is the number N(n, k) of ways of drawing n balls and ending with k black balls and n − k white ones? When counting sequences of drawings, we say that two sequences are different if their ordering of white/black drawings differ. For example, here are the six ways to draw four balls with two of each color (denoted by B and W for short): BBWW

BWWB

BWBW

WWBB

WBBW

WBWB

To write a recurrence for N(n, k), we observe that the sequence can end either in B or in W. In the former sequence, the first n − 1 drawings included k − 1 black balls, in N(n − 1, k − 1) possible different sequences, and for the sequences ending in W there are N(n − 1, k) possible orderings of the prefix. Hence the recurrence (5.65) follows. Exercise 5.127 [2+] There are known other two-dimensional arrays of numbers related to the binomial coefficients. Here is one of them. Let E(n, k) be the number of k-combinations from a set of size n, with repeated selections allowed. Then the E(n, k) satisfy the difference equation E(n, k) = E(n − 1, k) + E(n, k − 1),

n, k ∈ N,

E(n, 0) = 1,

E(n, 1) = n.

(5.66)

(a) Prove the recurrence (5.66), for k-samples with replacement. (b) Show that there is a 1-to-1 correspondence between k-combinations with repetitions from a set of size n, and k-combinations (without repetitions) from a set of size n + k − 1. n+k−1 This will establish that E(n, k) = is the solution of the difference equation (5.66). k (c) Show that the ballot numbers, ψn,k , defined in Eq. (2.63) on page 60, satisfy the recurrence (5.66) subject to the boundary conditions: ψn,0 = 1, ψn,n = ψn,n−1 . Exercise 5.128 [3] For n > m, let bn,m be the probability that the candidate who eventually won the election with n votes stayed ahead of his opponent throughout the voting. Derive a recurrence for these probabilities, based on the last vote cast, and show that bn,m = (n − m)/(n + m) satisfy it. Note that unlike the ballot numbers introduced in Example 2.106 on page 60, which allow equality m = n during the voting process, we now require the winner to be strictly ahead, all the time, from the first vote on; in terms of random walks bounded to be on one side of the line (0, 0)–(n, m), the walk may not touch the line.

258


Exercise 5.129 [3+] Show that rn + sn =

⌊n/2⌋

∑

bn,k pn−2k qk , where r and s are roots of the

k=0

quadratic equation x2 − px − q = 0 and n−k n , bn,k = 0 bn,k = n−k k

if n < k,

bn,0 = 1, b2,1 = 2,

Derive a recurrence for the double-index sequence {bn,k }. Hint: rn+1 + sn+1 = p(rn + sn ) + q(rn−1 + sn−1 ).

Exercise 5.130 [2+] Derive recurrences (and boundary conditions) satisfied by the following two number arrays: k n k n k−n , and G f (n, k) = ∑ i G p (n, k) = 2 ∑i i i i i Definition 5.131 Let hi1 , i2 , . . . , in i be a permutation of the set [1..n] = {1, 2, . . . , n}. The point ik is called a fixed point of this permutation if ik = k. A permutation is called a derangement if it does not have a fixed point, that is, if ik 6= k for all k = 1, 2, . . . , n. The number of permutations of the set [1..n] with k fixed points is usually referred to as a coincidence number and denoted by dn,k . ⊳ For example, there are six permutations of the set [1..4] with two fixed points: h1, 2, 4, 3i, h1, 4, 3, 2i, h1, 3, 2, 4i, h4, 2, 3, 1i, h3, 2, 1, 4i, h2, 1, 3, 4i and two derangements of the set [1..3]: h3, 1, 2i and h2, 3, 1i.

Example 5.132 (Permutations, fixed points, and derangement counts) Let Xn be the number of matches when the numbers from the set [1..n] = {1, 2, . . . , n} are placed randomly in n positions (it is also called the number of coincidences). Show that all its factorial moments are the same and equal to 1; that is, E[Xnk ] = 1, k = 1, 2, . . . , n. Solution. Since we are in a chapter dealing with recurrences, this is going to be our tool. Let pn,k = Pr[Xn = k] = dn,k /n!. Here dn,k is the number of permutations that keep exactly k entries in place. How do we find a relation for it? We first relate dn,k to dn,0 , the count of permutations where no element is left undisturbed. Such permutations are called derangements, and their number is conventionally denoted by Dn . We can choose k items in nk ways, and then require the rest to be all deranged, i.e., be in any but their proper place. Hence, n n def dn,k = dn−k,0 = Dn−k . k k

L. Euler discovered the recurrence for the derangement counts that we derive below. If a given permutation is a derangement, one of the numbers from [1..n − 1] occupies the last position. Let it be r. Now consider the positions of all other numbers that must be deranged. If n happens to be in position r, then the remaining elements can be deranged in Dn−2 possible ways. If we “forbid” position r to be n, we have n−1 numbers to arrange with each prohibited from occupying one position; this can be done in Dn−1 ways, and we have shown that Dn = (n − 1)[Dn−1 + Dn−2 ],

n > 2,

D0 = 1, D1 = 0.

(5.67)


259

Before we continue to prove the claim about the factorial moments, we show that this recurrence leads to another one: Dn = nDn−1 + (−1)n ,

n > 1,

D0 = 1.

(5.68)

The proof16 is by induction; verify it for n = 1, 2, then assume that the claim is true for all indices up to r −1. To prove the inductive hypothesis, we first write Dr = (r −1)[Dr−1 +Dr−2 ] and then observe, using relation (5.68) for n = r − 1, that Dr = (r − 1)Dr−1 + (r − 1)Dr−2 = (r − 1)Dr−1 + Dr−1 − (−1)r−1 = rDr−1 + (−1)r , as needed. We now derive an expression for the jth factorial moment of Xn in terms of the derangement counts using pn,k = dn,k /n!: n n n dn,k j n− j 1 n− j 1 Dn−k n j j k =∑ D = k = E[Xn ] = ∑ ∑ n − k n−k (n − j)! ∑ r Dr . (n − j)! k= k r j k= j n! k= j n! All we need to do is to show that this last sum is always 1. To prove this, denote n − j by m, and consider the m! permutations of m elements, splitting them according to the number of fixed-points they have. Consider permutations with exactly m − r fixed-points, the other r must be all displaced, and they can be arranged in Dr different ways. The number of ways to choose the m − r fixed-points (or the r out-of-position points) is mr , and when we sum m r Dr over all r we go over the entire collection of all m! permutations.

Exercise 5.133 [3] We reconsider the random variable Xn , defined in Example 5.132. In a random permutation of [1..n], Xn is the number of entries that are in their natural places. Assuming all permutations are equally likely, define pn,k = Pr[Xn = k]. Show the recurrence k pn,k = pn−1,k−1 , for n > 1, k > 0. Hint: Express pn,k in terms of the number of derangements in the permutation. Exercise 5.134 [2] Derive a coupled recurrence for the number of odd and the number of even derangements of [1..n], and then solve the difference equations by expressing their solutions via Dn = ∑nk=1 (−1)k /k!, the number of derangements. Exercise 5.135 [3+] (Dinner problem, adapted from [21]) Suppose that n − k people (k is a non-negative integer less than n) are invited to a dinner party. Seats are assigned and a name card made for each of n − k guests. In addition to invited people, k people attended the party, called party crashers, and no name cards exist for them. However, floral arrangements on the tables unexpectedly obscure the name cards. When the n guests arrive, they seat themselves randomly at the tables. Let pn,k be the probability that no invited guest sits in his or her assigned seat. Show that the double sequence {pn,k } satisfies the following recurrence: pn,k =

n−k k pn−1,k−1 + pn−1,k , n n

n

n > 1, 0 < k 6 n,

pn,0 =

(−1)k . k=0 k!

∑

16 A rather difficult combinatorial proof was given by J. B. Remmel, “A note on a recursion for the number of derangements,” European Journal of Combinatorics, 4, 1983, 371 – 374.


260

Exercise 5.136 [3+] (Dancing problem) Suppose that n − k married couples and 2k single people are invited to a party (so total is 2n people). Dance partners are chosen at random, without regard to gender. Let qn,k be the probability that nobody will be paired with his or her spouse. Show that qn,k =

2n − 2k 2k − 1 qn−1,k−1 + qn−1,k , 2n − 1 2n − 1

qn,0 =

2n − 2 qn−1,1 . 2n − 1

Definition 5.137 A permutation hi1 , i2 , . . . , in i of the set [1..n] is said to be of the rank m if im is the first element in its natural position, that is, im = m but ik 6= k for k = 1, 2, . . . , m − 1. A derangement is said to be of the rank n + 1. ⊳ Let Rn,m be the number of permutations of the set [1..n] of rank m, m = 1, 2, . . . , n; and let Dn be the number of derangements of the set [1..n]. The sequence Rn,m satisfies the recurrence relation: Rn,m = Rn,m−1 − Rn−1,m−1 , 2 6 m 6 n; (5.69)

subject to the boundary conditions Rn,1 = (n − 1)!, Rn,n = Dn−1 , Rn,n+1 = Dn . We set Rn,m = 0 if m > n + 1. The number Rn,m is usually referred to as the rank number.

The following three problems reveal an interesting property that the solution of double index recurrence is also a solution of other recurrences. Exercise 5.138 [2+] Show that the difference equation (5.69) is equivalent to the following recurrence relation Rn,m = (n − m)Rn−1,m + (m − 1)Rn−1,m−1,

m = 2, 3, . . . , n; n = 2, 3, . . . ;

(5.70)

since they both have the same solution: m−1 Rn,m = ∑ (−1) (n − k − 1)!. k k=0 m−1

k

(5.71)

Provide a combinatorial interpretation for this recurrence. Exercise 5.139 [3] Let E2 (n, k) be the number of k-combinations of [1..n] with repetition when each element is limited to appear at most twice. Show that the double sequence m n n− j k , k ∈ [1..2n], n ∈ I1 ≡ Z+ ; E2 (n, k) = ∑ , m = min n, 2 j k − 2 j j=0 satisfies each of the following recurrences subject to the same initial and boundary conditions (an,0 = 1, an,1 = n, an,k = 0 for k > 2n), and provide combinatorial interpretations for them: (a) an,k = an−1,k + an−1,k−1 + an−1,k−2 ,

k ∈ [2..2n],

n ∈ Z+ ;

(b) kan,k = nan−1,k−1 + 2nan−1,k−2 , k ∈ [2..2n], n ∈ Z+ ; k n (c) an,k = nan−1,k−2 + [an−2,k−2 + 4an−2,k−3 + 4an−2,k−4 ] k ∈ [4..2n], n ∈ I2 . 2 2


261

Exercise 5.140 [2] (adapted from C. Charalambides [26]) Show that two sequences −1 n n m n! 1 n−k and bm,n = = (m + 1) am,n = ∑ (−1) n+1 m−k+1 k (m + 1) n k=0 satisfy the same recurrences n xm−1,n−1 , n ∈ [0..m], m ∈ N, m+1 = xm−1,n−1 − xm,n−1 .

xm,n =

Are the two arrays identical? Note: The numbers bm,n are called the Leibniz numbers; we already saw them in Example 2.139 on page 71.

5.8.3

Stirling Numbers

The symbol17 mn denotes a Stirling number of the second kind. These numbers play an important role in combinatorics. Their standard combinatorial characterization is “the number of ways to partition a set of n distinguishable items into m non-empty subsets,” where partitions differ by the set sizes, but not their content. As a consequence, the mn are sometimes called the subset numbers. For example, there are three ways to split a three-element set into two parts: {1, 2, } ∪ {3}, {1} ∪ {2, 3}, {1, 3} ∪ {2}. Alternatively, the Stirling numbers can be defined via the following recurrence relation: n n−1 n−1 =m + , n > 1, (5.72) m m m−1 subject to the boundary conditions n n n n = 1, = 1, and = 0, for m > n. = δn,0 , 1 n m 0

(5.73)

Here δn,0 denotes Kronecker’s delta symbol (2.5), page 29. In Chapter 8, the Stirling numbers will be defined with the aid of generating functions through the relation (8.30). Further information about these numbers is given in Appendix C. The Stirling numbers of the first kind, denoted by sn,m , are solutions of the following recurrence relation sn,m = sn−1,m−1 − (n − 1)sn−1,m ,

sn,0 = δn,0 ,

sn,m = 0

for n < m.

(5.74)

The number (−1)n−m sn,m = |sn,m | is known as a signless Stirling number of the first kind or n Stirling cycle number and is usually denoted as m . These numbers also have a combinatorial 17 The

motivation of such notation is discussed by D. Knuth in [80].

262


meaning: they count the number of ways to arrange n objects into mcycles and are sometimes called the cycle numbers, pronounced “n cycle m.” Equivalently mn counts the number of ways for n distinct people to sit around m identical circular tables, where no tables are allowed to be empty. Clearly, Eq. (5.74) shows that these numbers satisfy the recurrence n n n−1 n−1 n = 1. (5.75) = (n − 1) + , = δn,0 , n m m m−1 0 An m-cycle (or simply cycle) is a cyclic arrangement of m objects that has no sub-cycles. For example, 3-cycles of three numbers {1, 2, 3} includes the following two sets of arrangements: h 1, 2, 3 i, h 2, 3, 1 i, h 3, 1, 2 i

and h 1, 3, 2 i, h3, 2, 1 i, h2, 1, 3 i. So there are two different 3-cycles and hence 31 = 2. On the other hand, there are three different 2-cycles of three numbers: (1,2), (1,3), (2,3). You can write the cycle (2,3) also as (3,2), but it is usually best to agree on a canonical representation of cycles, and the one most often chosen is to have the smallest element (usually called the leader) of the cycle be the first. Also the leaders of different cycles are assumed to be in decreasing order. In canonical form, the parentheses are redundant, since “)(” occurs just before each number which is smaller than all of its predecessors. Every permutation can be decomposed uniquely into its cycles. We have used the term “fixed point of a permutation” before, in Definition 5.131; in this context it is called a cycle of length one, or a singleton cycle. Thus, if you follow the moves of elements (from their original positions) in the permutation h 3, 9, 10, 2, 5, 4, 8, 7, 6, 1 i, you see that 3 moved to position 1, 1 moved to position 10 that is in place 3. Hence, we discovered the cycle h1, 10, 3 i, and the others here are h 2, 4, 6, 9 i, h 7, 8 i, and h 5 i, which is the fixed point. The canonical representation of this permutation will be h 7, 8, 5, 2, 4, 6, 9, 1, 10, 3 i. The topic of cycles in permutations is covered in §7.6.2 in more detail. Exercise 5.141 [3] Derive the recurrence relations (5.72) and (5.75) from the combinatorial definition of the subset and cycle numbers. Derive in this way also the boundary values in Eqs. (5.73), (5.75) and (C.11). (When needed, use the recurrences.) Unlike the binomial coefficients, which satisfy a similar recurrence, there are no explicit closed solutions for the Stirling numbers. We find an expression for these numbers in Exam m k m n ple 8.57, from which we learn that mn = (−1) m! ∑k (−1) k k . Exercise 5.142 [3−] Prove that for n > 0, (a) n+1 = n!, (b) n+1 = n! Hn , 1 2 n+1 n+1 n+1 (c) n = n = 2 . Theorem 5.143 Let x be a real number and n be a positive integer, then n n n n n n n k n−1 n x + x + ···+ x+ , x =∑ x = n n−1 1 0 k=0 k n n x x n x x xn = ∑ k! = n! + (n − 1)! + ··· + , k k n n − 1 n − 1 1 k=1

(5.76)

(5.77)

5.9. SOME APPLICATIONS

263

n n (−1)n−k xk = ∑ sn,k xk , (5.78) x =∑ k=0 k k=0 = (−1)n−k nk are Stirling numbers of the second kind and the first kind, n

n

where nk and sn,k respectively.

Exercise 5.144 [2] Prove Theorem 5.143. Hints: In expansions (5.76) and (5.78) show that the coefficients satisfy the recurrence relations (5.72) and (5.75) for Stirling numbers. Use the relations: x · xk = xk+1 + kxk and xk = (x − n) · xk−1 + xk−1. Exercise 5.145 [3−] In part (a) of Exercise 2.119, page 63, you were asked to prove that Sm (n) = ∑nk=0 nk (−1)k (n − k)m = 0 for m < n. Here we mostly deal with m > n. (a) Find a recurrence for Sn+r (n), n, r ∈ N. (b) Using Eq. (5.76), find a closed form of Sm (n). Check your answer with Eq. (5.88). def (c) Generalize the formula obtained in part (b) for the sum Sm,n (a, b) = ∑k nk (−1)k (a + bk)m .

Exercise 5.146 [3] Prove the inversion formulas (C.16) from Appendix C. Exercise 5.147 [3] Show ∆k 0d = k! dk where dk are Stirling numbers of the second kind, presented on page 261, and the odd notation ∆k 0d denotes the kth-order difference of the function f (n) = nd , evaluated at n = 0.

5.9 Some Applications Many practical applications lead to recurrences that are impossible to solve. As an alternative, finding bounds and estimates for their solutions can provide decisive information. We also present a classical method of solving constant coefficient linear difference equations based on some properties of finite differences.

5.9.1 Bounds from Recurrences Recurrences can be hard to solve. We have seen how to handle such first order linear equations. Another method is illustrated in the next chapter, and you should expect to see more later in the text. Still, some put up quite a resistance. A lesser goal than a complete solution may be easier to attain: instead of finding the complete solution, we may settle for a bound. This is often entirely adequate for our purpose of reasoning about the performance of the algorithm from which the recurrence arose. See Example 1.44 on page 21, where a bound for a first-order recurrence was shown as an exercise in proof by induction. Examples are the best way of showing this possibility. Example 5.148 Consider the recurrence: xn = xn−1 + 2xn−2 for n > 3, with the initial values x1 = 3/2, , x2 = 6. You have reason to believe that xn < (2.5)n , for all n. How would you prove it?


264

It is easy to see that it holds for the first two given initial values. And beyond that? The answer “mathematical induction” should have occurred to you. The agreement of the bound with the two initial values satisfies the basis for the claim; we assume that xk < (5/2)k for all k < n, and proceed to show for the value of xn . The induction hypothesis provides n−2 n−1 5 5 , and xn−2 < xn−1 < 2 2 n−2 n−1 5 5 +2 =⇒ xn = xn−1 + 2xn−2 < 2 2 n n n 5 18 5 5 4 2 = < , +2× = 5 25 2 25 2 2 as needed. Now we get more ambitious: a further examination of the recurrence suggests a tighter bound: xn < 2n . Is this true? Can this be shown? (The two questions need not have the same answer!) This new bound is satisfied by the x1 , but not by x2 . Is it wrong? Obviously yes, but it may not be too wrong: we change the hypothesis to xn < c 2n , for some constant c, and see that for the hypothesis to hold for both initial values we need c > 3/2. We make the indicated induction hypothesis, with this c, and forge ahead, as before: xn−1 < c2n−1 , and =⇒

xn−2 < c2n−2

xn = xn−1 + 2xn−2 < c 2n−1 + 2 × 2n−2 1 n 1 +2× 2 = c2n , = c 2 4

as required. Interestingly, we see that the constant did not take part in the induction step—we only needed it for the basis. Emboldened by this success, we now want to show xn < (3/2)n . However, we cannot do it: introducing a constant, as above, to have the bound xn < d(3/2)n , we can always make it ‘work,’ with a sufficiently large d, to cover the basis: here we only need d > 8/3 for this bound to hold for x1 and x2 , but trying the induction step, we come to a cropper. Do the computation in full and see! Indeed, using the method of the next subsection, you can show that the correct solution is xn = (5/4)2n + (−1)n . Exercise 5.149 [2+] Consider the recurrence (1.10), Fn = Fn−1 + Fn−2 , with F0 = 0, F1 = 1, and show that the growth rate of the Fibonacci numbers is smaller than 9/5 or 5/3, but not 8/5, just as in Example 5.148. In all the preceding examples, we actually had a simple method available to solve the recurrences exactly. We did not need the bounds, but when we leave the cozy world of linear equations, things get much more complicated in a hurry. The following example concerns the same recurrence presented in Example 1.44 on page 21 . There our interest was in the proofs of the bounds, and the light they shed on the properties of mathematical induction.


265

Example 5.150 Consider the recurrence xn+1 = 2x2n − 2, x0 = 2. The somewhat simpler equation, an+1 = 2a2n , a0 = 2, while nonlinear, is not hard. Just iterating it a few times, we n+1 see a pattern, which suggests that an = 22 −1 . Surely xn 6 an for all n > 1. The table given on page 21 suggests that the bound is very loose, and it’s simplicity is also apparent. Example 5.151 Suppose we want to test whether a given integer is a power of 2. Here is a fragment of C code for such a procedure, which tests its input, and returns ±1 accordingly; it also returns the highest power of 2 that divides the input. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12.

int Test( int n, int ∗m) { if (n%2 == 1) // this ‘if’ has no else {if (n == 1) return(1); else return(−1); } (∗m)++; return Test(n/2, m); } main() { int v, m, ∗c = &m; for (v = 1; v < 18; v++) // arbitrary values for example m = 0; printf (”n=%d, Test =%d\n”,v, Test (v, c) ∗ m); }

The basis case verification and execution in the odd case takes some constant time, say C. If n is even, the else branch takes time T (n/2) to find Test(n/2), plus some time D to compute b and m. Thus, T (n) 6 C + T (n/2) + D. (5.79) Suppose n = 2m · k with k odd and m > 0. Sequential application of inequality (5.79) yields T (n) = T (2m · k) 6 T 2m−1 · k +C + D 6 T 2m−2 · k +C + D +C + D 6 ...

6 T (k) + m(C + D) 6 C + m(C + D) 6 2m(C + D). Since n/k = 2m , we have m = log2

n k

=⇒

T (n) 6 2(C + D) log2 n.

Example 5.152 In many linear algebra texts, determinants of matrices are computed by means of cofactor expansion: det A = ai1 Ai1 + ai2 Ai2 + · · · + ain Ain ,

1 6 i 6 n,

where det A denotes the determinant of a square matrix A and Ai j are appropriate cofactors. Let f (n) be the number of multiplications necessary to evaluate the determinant of an arbitrary n × n matrix using cofactor expansion (see Exercise 10.115 on page 609 for the total

266


number of arithmetic operations). For n > 3, the number of multiplications needed to evaluate each Ai j is equal to f (n − 1) by definition. Therefore, to compute ai j Ai j , we need f (n − 1) + 1 multiplications. Because there are n such terms, we conclude that f (n) satisfies f (n) = n[ f (n − 1) + 1],

for n > 3,

f (2) = 2,

which is a first-order linear non-homogeneous difference equation with variable coefficients. Our goal is to obtain a simple lower bound for f (n), but not to solve it. Noting that f (n) = n f (n − 1) + n > n f (n − 1), we get f (n) > n f (n − 1) > n(n − 1) f (n − 2) > · · · .n(n − 1) · · · 3 f (2) = n! . This inequality shows that it will take more than 50! or approximately 0.3 × 1065 multiplications to evaluate the determinant of a 50 × 50 matrix.

5.9.2 Recurrences and Finite Differences The (forward) difference and the shift operators have already been introduced in §2.2.2, where we used them for summation by parts. They are defined as follows, for any sequence {an }n>0 : ∆{an } = {an+1 − an }; E{an } = {an+1 }. (5.80) Hence ∆ = E − I, where I is the identity operator: I{an } = {an }. With this in hand, we define the powers ∆r of the difference operator as r r r k r r r r−k ∆ an = (E − I) an = ∑ E (−I) an = ∑ (−1)r−k an+k . (5.81) k k k=0 k=0 An interesting and very useful application of the above approach yields r r r r ∆r xn = (E − I)r xn = ∑ (−1)k (x + r − k)n = ∑ (−1)n−k (x + k)n , k k k=0 k=0

(5.82)

where ∆ operates on x. Since ∆r xn = 0 for r > n, the sum (5.82) is zero when such inequality holds. In addition to the forward difference operator ∆, it has been found useful to define a related operator, the backward difference, ∇an = an − an−1 . Hence, ∆an = E∇an = ∇an+1 . Setting n = 0 in Eq. (5.81), we obtain an explicit formula for ∆r a0 in terms of values a0 , a1 , . . . , ar . We can easily invert (5.81) and express an in terms of the numbers ∆i a0 : n n k n n an = E a0 = (1 + ∆) a0 = ∑ ∆ a0 . (5.83) k=0 k Similarly, for the backward operator ∇ = I − E −1 we have −n n −n an = E a0 = (1 − ∇) a0 = ∑ (−1)s ∇s a0 , s s>0

(5.84)


267

since E = (I − ∇)−1 . Eqs. (5.76), (5.78), and (5.83) can be used to express the Stirling numbers via the forward difference operator and differential operator. Its derivation is based on the well-known formula: ∞ k k d x (Maclaurin series) f (x) f (x) = ∑ k x=0 k=0 k! dx and its finite difference analogy: ∞

f (x) =

xk h

∑ k!

k=0

i ∆k f (x)

∞ x=0

,

f (x) =

xk h

∑ k!

k=0

i ∇k f (x)

x=0

.

Note that Eqs. (5.85) follow from Eqs. (5.83) and (5.84), respectively, because k = n+k−1 = (n + k − 1)k /k! = nk /k! . and −n (−1) k k

(5.85) n k

= nk /k!

Application of Eq. (5.85) to f (x) = xn and f (x) = xn together with Eqs. (5.76), (5.78) yields n 1 dk 1 dk n n x (x + n − 1) , , (5.86) = sn,k = k! dxk k! dxk k x=0 x=0

and 1 n 1 h k ni ∆ x = = x=0 k! k! k

k k−i

∑ (−1)

i=0

k i n 1 E x x=0 = i k!

k k−i

∑ (−1)

i=0

k n i . i

(5.87)

The last equation reminds us of the equation previously studied in Exercise 2.119, page 63, and it can now be rendered more fully: n m k m n m n m (5.88) ∑ k (−1) (n − k) = ∆ x x=0 = ∆ 0 = n! n . k

The following result shows the effect of the operators ∆ and ∇ on a polynomial: Lemma 5.153 m

∆n =

m−1

∑

k=0

m k n. k

m

m−1

∇n = (−1)

m−1

∑

k=0

m (−n)k . k

(5.89)

P ROOF : The definition of ∆ and the binomial theorem leads to m−1 m m k m k m m m m n, n −n = ∑ ∆n = (n + 1) − n = ∑ k k k=0 k=0 where nm canceled with the k = m term of the sum. The formula for the backward operator can be proved in a similar way. Thus, the result of applying the difference operator ∆ to the power nm is a polynomial of degree m − 1, in other words, we lower the degree of the polynomial by one. We say that an operator annihilates a function if the result of applying it to the function is identically zero. The following theorem is an immediate extension of the lemma.


268

Theorem 5.154 The operators ∆m+1 and ∇m+1 annihilate any polynomial in n of degree up to m. In particular, ∆m+1 nk = 0;

∇m+1 nk = 0.

k = 0, 1, . . . , m.

Exercise 5.155 [3−] (a) What is the general solution to (E − I)k xn = 1, k ∈ N, where I is the identity operator, that is, Ixn = xn ? (b) What is the solution of the recurrence of part (a) with the initial value x0 = 1? The following theorem generalizes Theorem 5.154 to deal with right-hand sides that contain exponential terms: Theorem 5.156 (En − a)m+1 nk an = 0,

0 6 k 6 m,

n ∈ N.

(5.90)

where the shift operator En operates on the argument n. P ROOF : We observe that (En − a)nk an = En (nk an ) − an+1 nk = an+1 ∆n nk . Repeated applicak tions provide similar expressions: (En − a)m nk an = an+m ∆m n n . The result then follows from Theorem 5.154. These theorems lead to the general solutions of constant coefficient linear difference equations of the order p: an+p + b1 an+p−1 + b2 an+p−2 + · · · + b p an = 0,

b p 6= 0.

(5.91)

Using the shift operator, E, we rewrite this equation as (E p + b1 E p−1 + b2 E p−2 + · · · + b p )an = 0.

(5.92)

λ p + b1 λ p−1 + · · · + b p = 0,

(5.93)

The algebraic equation

corresponding to the difference equation (5.91) is called the characteristic equation. Solutions of Eq. (5.93) are called characteristic roots of the difference equation. As we saw in Theorem 5.29, these characteristic roots can be used to give an explicit formula for all solutions of Eq. (5.91). Looking for solutions in the form an = λ n , we obtain Eq. (5.93). Suppose that the characteristic equation has k different roots of multiplicity mi (i = 1, 2, . . . , k). Then we can factor Eq. (5.92) to obtain (E − λ1 )m1 (E − λ2 )m2 · · · (E − λk )mk an = 0 (m1 + m2 + · · · + mk = p). Application of Theorem 5.156 yields the general solution of the recurrence (5.91): an = Pm1 −1 (n)λ1n + Pm2−1 (n)λ2n + · · · + Pmk −1 (n)λkn , where Pmi −1 (n) are some polynomials of degree mi − 1 (i = 1, 2, . . . , k).

(5.94)


269

Exercise 5.157 [2] Let P(x) = pn xn + pn−1 xn−1 + · · · + p0 be a polynomial of degree n. A necessary and sufficient condition for P(k) to have integer values for all k ∈ Z is that ∆i P(0) ∈ Z, 0 6 i 6 n. Hint: Use Eq. (5.83). Exercise 5.158 [3] While the difference operators were introduced here as annihilators for the specific purpose of handling a particular type of recurrence, Theorem 5.154 and Eq. (5.81) can be applied elsewhere. Using ∆n P(x) = ∑k nk (−1)k+n P(x + k) = 0, for n > d, show the k+1 summation (A.30) on page 702: ∑k>1 nk (−1)k (x + ky)r = xr Hn + rxr−1 y, for r 6 n ∈ N, where Hn is the nth harmonic number. There is no useful closed form for r > n. Hint: Differentiate the sum with respect to y first. Sometimes we need to know ∆ak (or ∇ak ) when ak is a solution of a given difference equation. Exercise 5.159 [2−] For any real parameter t ∈ [0, 1], let {xn (t)} be the solution of the following recurrence relation: xn (t) − 2t xn−1 (t) + xn−2 (t) = 0,

n > 2,

x0 = 1, x1 = t.

Calculate ∇2 xn (t), ∇4 xn (t) and ∆2 xn (t), ∆4 xn (t). Note: Solutions of this difference equation are usually denoted by Tn (t), and are called the Chebyshev polynomials of degree n. They may be expressed as Tn (t) = cos(n cos[−1] t), where cos[−1] is the inverse cosine function (also called arccosine). Now we turn our attention to first order difference equations considered in §5.3. Its general solution (5.13), page 216, is rather difficult to use. Therefore we consider below a particular case—finite sums of polynomials and exponents Sn = ∑nk=0 km ak , m ∈ R and a ∈ C. To start with, we consider the case when a = 1. If m > 0, then the sum can be evaluated explicitly. Since ∆Sn = Sn+1 − Sn = (n + 1)m , we apply the anti-difference operator (2.26), page 39, to obtain (using Eq. (5.76) and the relation ∆x xm = m xm−1 ) m m m m −1 m 1 m (n + 1) j+1 . ∆n (n + 1) j = ∑ ∆−1 (n + 1)m = ∆−1 ∑ (n + 1) j = ∑ j + 1 j j j j=0 j=0 j=0 The case of negative m leads to summation of rational functions and it turns out to be more complicated since it requires application of a polygamma function (see Eqs. (G.33), (G.34) on page 727). There are several known algorithms18 (used in computer algebra systems) that allow us to evaluate sums with rational coefficients, but the topic is too expansive to cover in its entirety. Example 5.160 Let us find Φ4 (n) = ∑nk=0 k4 . Eq. (5.76) expresses k4 as a linear combination of falling factorials: k4 = k + 7k2 + 6k3 + k4 . Now the result of Exercise 2.66(a) on page 47 gives what we need:

∑ k2 + 6

n

n

∑ k+7

∑ k3 + ∑ k4

k=0

k=0

k=0

k=0

n

Φ4 (n) =

n

18 We mention only three well-known algorithms proposed by R. Gosper [51], M. Karr [71], and R. Moenck [100].


270

= =

n(n + 1) 7 6 1 + (n + 1)3 + (n + 1)4 + (n + 1)5 2 3 4 5 1 n(n + 1) n(n + 1)(2n + 1)(3n2 + 3n − 1). 6n3 + 9n2 + 21n − 16 = 30 30 n

Exercise 5.161 [3] Find Φ5 (n) =

∑ k5 . k=0 n

∑ km . We can represent this polynomial (in n) as

Exercise 5.162 [3+] Let Φm (n) =

k=1 m

Φm (n) =

∑ cr,m

r=1

m+1 m+1 n n+1 = ∑ am,r nr . = ∑ bm, j j r+1 r=1 j=1

Find the recurrence relation for each of the double sequences {am,r }, {bm,r }, and {cr,m }. Hint: The difference operator and the derivative operator commute.

Evaluation of sums ∑nk=0 km ak , a 6= 1, can be performed with the aid of summation by parts (Theorem 2.36 on page 40) and it is based on the following equations ∆x xm = m xm−1 ,

∆x ax = ax (a − x − 1),

∆x ax = ax (a − 1).

Example 5.163 We calculate the sum as follows: 2

n

∑ k2 3k = 2

n

∑

k=0

k=0

=

k2 + k 3k =

n

n 2 k k + k ∆3 ∑ ∑ ∆3k

k=0

k=0

k=n+1 k=n+1 + k 3k −2 k2 3k k=0

k=0

n

n

∑ k 3k+1 − ∑ 3k+1

k=0

k=0

n

= (n + 1)2 3n+1 + (n + 1) 3n+1 − 3 ∑ k ∆3k − 3 k=0

k=n+1 = (n + 1)n 3n+1 + (n + 1) 3n+1 − 3k 3k k=0 + 6

n

∑ 3k

k=0 n k

∑3 k=0

= 3n+1 [(n + 1)(n − 2) + 3] − 3 = 3n+1 n2 − n + 1 − 3.

Exercise 5.164 [3] Using summation by parts, find sums n

(a) 6

∑k k=0

2 3

k ,

n

(b)

∑k k=0

3 k

2,

k (c) ∑ k . 2 k=0 n

2

n+k−1 n+m Exercise 5.165 [2+] Use summation by parts to show ∑ k =n . k m−1 k=1 m

Chapter 6

Introduction to Generating Functions The time has come to introduce the most powerful tool we use in analysis of algorithms: generating functions (GFs). They provide a natural and elegant way to deal with sequences of numbers by associating a function (of a continuous variable) with a sequence. In this way GFs provide a bridge between discrete and continuous mathematics. Generating functions, like those we use below—and others with minor variations—support numerous applications in other disciplines, such as probability and statistics, quantum mechanics, control theory, signal processing, and on and on. There are several ways to associate a function with a sequence of (real or complex) numbers {an }n>0 . The most natural one is to consider a power series a(z) =

∑ an zn .

(6.1)

n>0

Such a function a(z) is also known as an ordinary generating function (OGF). We obtain the exponential generating function (EGF) by defining a(z) ˆ =

zn

∑ an n! .

(6.2)

n>0

Sometimes, it is convenient to introduce a generating function A(z) = ∑n>0 an gn (z) with respect to a given sequence of functions {gn (z)}. Of these, the Appell polynomials are more frequently used. The method of generating functions is naturally embraced by integral representations and formal Laurent series. This elegant technique1 was developed by G. P. Egorychev [37], but this topic is beyond the scope of our book. In Chapter 12 we present a brief survey of the definitions and properties of the mathematical infrastructure needed for generating functions. While it includes sufficient material for our uses, the reader who would like to delve more deeply into these topics needs additional sources. Happily, excellent texts exist. We recommend Henrici [62] and Egorychev [37] for a detailed modern coverage. 1 Some

of its applications are exposed in [58], [98].

271

272

CHAPTER 6. INTRODUCTION TO GENERATING FUNCTIONS

6.1 Generating Functions — Definitions We present the two main types of GFs we shall use, ordinary (OGF) and exponential (EGF) generating functions. They were invented by Abraham de Moivre (1667 – 1754) in the early eighteenth century and, like much of discrete mathematics, came to prominence through the work of Leonhard Euler (1707 – 1783) in the middle of that century. L. Euler used GFs to investigate partitions, which are considered in §7.5.

6.1.1 Ordinary Generating Functions Definition 6.1 For a given sequence of numbers {an }n>0 , the power series (6.1) is called the ordinary generating function (OGF) of the sequence {an } or z-transform of this sequence. ⊳ In almost all our applications, the terms of the sequence {an }n>0 are integers, but all the properties of the GFs we discuss hold for all numbers in the complex plane. Convergence of the power series (6.1) is not an issue in this chapter. There exists a theory of formal power series, of which an excellent account is given by Ivan Niven [102] that allows us to ignore convergence issues and also permits us to manipulate formal power series as we do polynomials (for example, multiply them term by term). As a rule, we start all our sequences at the zero index. Normally we don’t need to assume anything about elements with negative indices. If they arise, we shall need to say something definite about them (and what we often say is that they vanish—equal zero). We start with some basic examples—these may be viewed as a re-cap of some of the summation formulas used previously. Consider a sequence of identical values, namely, ak = C, k > 0. Then the corresponding OGF is a(z) =

C

∑ C zk = C ∑ zk = 1 − z .

(6.3)

k>0

k>0

Similarly, let ak = rk , k > 0, where r is a given number. Then a(z) =

1

∑ rk zk = 1 − rz .

k>0

The next example is a similar sequence having the corresponding generating function: ( 0, 0 6 k 6 2, r3 z3 k k . ak = r z = =⇒ a(z) = ∑ 1 − rz rk , k > 3, k>3 The issue of convergence raises a question that should puzzle you: If we say that convergence is not meaningful here, how can we use a formula, such as Eq. (6.3), which, as we saw in calculus, is only true when the series

(6.4)

6.1. GENERATING FUNCTIONS — DEFINITIONS

273

converge? It seems there is life outside of calculus, if we agree that two power series are equal when coefficients of like powers are equal. This principle is important, in the sense that it is fundamental to the applicability of GFs for our needs. We are talking about the uniqueness of generating functions (GFs). The topic is further elaborated in §12.2.2, in the context of power series, and GFs are power series. As a result, they enjoy the property given in Eq. (12.22) on page 675: generating functions are unique in the following sense: If two GFs, say a(z) and b(z), are equal (this means a(z) − b(z) = 0, identically, i.e., for all complex numbers z), then the sequences {an } and {bn } are identical, term by term. Suppose we take the sum in Eq. (6.4) ! and rewrite it as follows:

∑ rk zk

k>0

(1 − rz) =

∑ cn zn .

n>0

We multiply the two factors on the left-hand side and compare coefficients of like powers with the expression ( on the right and find 1, n = 0, cn = n n−1 r −r×r = 0, n > 1. k k −1 Hence ∑k>0 r z indeed equals (1 − rz) , as claimed. Consider now the binomial sequence ak = uk , where u is an arbitrary number. From Eqs. (6.1) and (2.42) on page 48, we have u k a(z) = ∑ z = (1 + z)u . (6.5) k k>0 If u is a natural number, the sequence is finite, and the OGF is a polynomial. Next, let ak = 1/k, k > 1, then Z z Z z zk xk−1 dx = =∑ a(z) = ∑ ∑ xk−1 dx k 0 0 k>1 k>1 k>1 =

Z z dx 0

1−x

= − ln(1 − x)|z0 = − ln(1 − z) = ln

1 . 1−z

(6.6)

Let n be a natural number, and consider the sequence ak = nk . From Eq. (6.1), it follows, using transformations of the binomial coefficients, that k k a(z) = ∑ z Eq. (6.1) k>0 n k = ∑ zk symmetry, Eq. (2.56)—note the range of k k − n k>n −n − 1 = zn ∑ (−1)k−n zk−n negating the upper index, Eq. (2.64), k − n k>n −n − 1 n = z ∑ (−z) j change of dummy index j j>0 zn binomial theorem, Eq. (2.42) on page 48. = (1 − z)n+1


274

(We could simplify matters by consulting Eq. (L.20) in Appendix L.) This may be used in scenarios that appear different. For example, when ak = k3 − k, we can write it as k(k2 − 1) = 6z2 (k + 1)k(k − 1) = 3! k+1 3 , and then the calculation shows that a(z) = (1−z)4 .

Consider the sequence ak = 1 + (−1)k , that is, a0 = 2, a1 = 0, a2 = 2, a3 = 0, . . .. Then we have

∑ [1 + (−1)k ] zk = ∑ zk + ∑ (−1)k zk

a(z) =

k>0

k>0

k>0

1 1+z+1−z 2 1 + = = . 2 1−z 1+z 1−z 1 − z2

=

Operations on Sequences The correspondence between sequences and GFs is so useful because, in addition to the uniqueness property, it turns out that many simple operations on sequences are reflected by simple operations on the corresponding generating functions. These operations, listed below, are so important that they have earned a name, and are called the admissible operations with respect to the OGF. A list of the basic OGF-admissible transformations of sequences is presented in Table 275. • LINEARITY cn = α an + β bn =⇒ c(z) = α a(z) + β b(z).

• RIGHT SHIFT cn = an+1 =⇒ c(z) =

a(z)−a0 . z

This result can be shown as follows:

c(z) = ∑ ci zi = ∑ ai+1 zi = i>0

i>0

1 a(z) − a0 1 ∑ ai+1 zi+1 = z ∑ ak zk = z . z i>0 k>1

(6.7)

There is a straightforward generalization. • MULTIPLE RIGHT SHIFT cn = an+k has the corresponding generating function: c(z) =

i a(z) − ∑k−1 i=0 ai z . zk

(6.8)

Let us look at what happens for k = 2: dn = an+2 = cn+1

c(z) − c0 = =⇒ d(z) = z

a(z)−a0 z

z

− a1

=

a(z) − a0 − a1 z . z2

• LEFT SHIFT cn = an−1 =⇒ c(z) = za(z) + c0 . This follows from reversing the relation for the right shift, naturally. Since c0 is not defined here, we leave it open—it is an element that may take any value. The explicit result follows from Eq. (6.7), with reversal of the roles of a(z) and c(z).


275

Sequence element cn

Ordinary generating function

α an ± β bn

α a(z) ± β b(z)

(1)

a(z) − a0 z a(z) − a0 − · · · ak−1 zk−1 zk za(z) + c0

an+1 an+k cn = an−1 (a ⋆ b)n = ∑ ak bn−k

#

(2) (3) (4) (5)

a(z)b(z)

k

nan

da(z) = zDz a(z) dz P[zDz ]a(z)

(6)

z

P(n)an , P is a polynomial an n

c0 +

(n > 1)

n

∑ aj

j=0

Z z a(t) − a0 0

t

(7)

dt

a(z) 1−z

(8) (9)

an (z)

(10)

(n + 1)an+1

Dz a(z)

(11)

nan−1

(z2 Dz + z)a(z)

(12)

∑

a j1 a j2 . . . a jk

j1 + j2 +···+ jk =n

Table 275: Transformations of the ordinary generating function. • CONVOLUTION cn = ∑ni=0 ai bn−i =⇒ c(z) = a(z) b(z). The convolution operation (usually denoted by ⋆ ) is one of the most important, but the proof is easy: k

c(z) =

∑ ck zk = k>0

∑ ∑ ai bk−i zk = ∑ ∑ ai bk−i zk = ∑ ai zi ∑ bk−i zk−i = ∑ ai zi ∑ b j z j .

k>0 i=0

i>0 k>i

i>0

k>i

i>0

j>0

• No standard name is given to the natural operation of multiplication by a non-constant function. First consider a simple case: cn = nan =⇒ c(z) = za′ (z). P ROOF : We have ! d a(z) d d(zk ) k k ak z = z =z . c(z) = ∑ k ak z = ∑ ak z ∑ dz dz k>0 dz k>0 k>0 • If we multiply the terms of the sequence by P(n), an arbitrary polynomial in n, we obtain in the same way: cn = P(n)an =⇒ c(z) = P(zD)a(z), where D = d/dz is the differentiation operator. For example: let cn = (n2 − n − 1)an , then its generating function is


276

c(z) = (zD)2 − zD − 1 a(z). Since

(zD)2 a(z) = zD(zD a(z)) = zD(za′ (z)) = z[a′ (z) + za′′ (z)] = za′ (z) + z2 a′′ (z),

we obtain c(z) = za′ (z) + z2 a′′ (z) − za′ (z) − a(z) = z2 a′′ (z) − a(z).

Exercise 6.2 [1] The last example shows that (zD)2 = z2 D2 − I, where I is the identity operator (Ia(z) = a(z)). In high school algebra you saw that (xy)2 = x2 y2 , but this property is not valid for operators of differentiation, D, and multiplication by z. There is an implicit condition on operators to commute. What is the condition? In what sense does it not work? What is the reason? Hint: See Example 12.75 on page 688. • Similarly, consider a less frequent operation: cn = a(z) − a(0) = zc′ (z)

=⇒

c′ (z) =

a(z) − a0 z

an n

(n > 1). Then

=⇒

c(z) =

Z z a(t) − a(0) 0

t

dt + c0 .

As an example, let us consider the constant sequence an = 1 (n > 0), hence a(z) = ∑n>0 zn = 1 1 1 zn 1 1−z . The sequence {c0 , 1, 2 , 3 , . . . , n , . . .} has the generating function c(z) = c0 + ∑n>1 n = 1 = 1z (a(z) − 1). c0 − ln(1 − z). Then c′ (z) = 1−z • PREFIX SUMMATION is a common special case of convolution: cn = ∑nj=0 a j implies that c(z) = a(z)/(1 − z), as Exercise 6.3 asks you to show.

Exercise 6.3 [2] Show that the sequence of partial sums, ∑nj=0 a j , has the ordinary generating function c(z) = a(z)/(1 − z), as given by rule (9) in Table 275. What is the sequence {an } convolved with? 1 1 ln , where Exercise 6.4 [2] (OGF for harmonic numbers) Show that ∑ Hn zn = 1−z 1−z n>1 Hn = ∑nk=1 k−1 is the nth harmonic number. Hint: You may want to use some of the rules of Table 275 and Eq. (6.6).

We can generalize the previous exercise by considering the sequence an = (an + b)−1 , n = 0, 1, 2, . . .. Using the Laplace integral, we obtain a(z) =

1 zn ∑ an + b = a n>0

∑ zn

n>0

Z ∞ 0

e−nx e−(bx/a) dx

=

Z Z 1 ∞ −(bx/a) 1 ∞ −(bx/a) e dx ∑ zn e−nx = e

=

Z 1 1 u(b/a)−1

Exercise 6.5 [1] Find

a a

0

0

n>0

1 − zu

(−1)n

∑ n + 1/2 .

n>1

du.

a

0

dx 1 − z e−x


277 n

Example 6.6 Let {Φm (n)}n>0 be the sequence of numbers Φm (n) = ∑ km that we met in k=0

§2.1.1. Its ordinary generating function is 1 am (z) = Φm (z) = ∑ Φm (n)z = 1−z 1−z n>0 n

since am (z) =

d z dz

m

m m k d k 1 d m 1 m k = , z k z = z ∑ k ∑ dz 1 − z k=0 dz 1−z k>0

1 1−z

see Eq. (5.76).

Exercise 6.7 [2] In terms of the ordinary generating function for the sequence {ak }, find the ordinary generating function for the sequence sn = ∑nk=0 nk bn−k ck ak , n = 0, 1, . . ., where b and c are real constants. Definition 6.8 A sequence of polynomials Pn (t) is said to be Newtonian if P0 (t) = 1 and n

Pn (x + y) =

∑ Pk(x)Pn−k (y) for n = 0, 1, 2, . . ..

⊳

k=0

From the convolution rule (see rule (5) in Table 275), it follows that the sequence {Pn (t)} is Newtonian if and only if its ordinary generating function, P(t; z) = ∑ Pn (t) zn , is muln>0

tiplicative, that is, P(x + y; z) = P(x; z)P(y; z). We have seen that the generating function t P(t; z) = (1 + z)t for the sequence of binomial coefficients, P (t) = is multiplicative and n n t hence the sequence of polynomials n is Newtonian.

Exercise 6.9 [2−] Show that the ordinary generating functions for the following sequences are multiplicative. t +n−1 tn (b) Pn (t) = . (a) Pn (t) = ; n! n

6.1.2 Exponential Generating Functions There are many ways to assign a function of a continuous variable to a sequence of numbers. The OGF is but one case; we can define a “weighted generating function” A(z) =

zn

∑ an wn ,

n>0

where {wn } is a sequence of weights. One of the most widely used weights is wn = n!. This case has a special name and notation: Definition 6.10 Any given sequence of numbers {an }n>0 , is associated with an exponential generating function (EGF) defined as Eq. (6.2). ⊳ We use the notation a(z) ˆ to distinguish exponential generating functions from the ordinary generating function a(z). The series (6.2) resembles the famous Maclaurin expansion of the


278

Sequence element cn

α an ± β bn

Generating function ˆ α a(z) ˆ ± β b(z)

(1)

an+1

aˆ′ (z)

(2) d k a(z) ˆ

#

an+k

Dk a(z) ˆ =

nk an

(zD)k a(z) ˆ

(4)

an−1

Z

(5)

a(t) ˆ dt

dzk

(3)

z a(z) ˆ

(6)

P(n)an , P is a polynomial

P(zD)a(z) ˆ

(7)

an+1 n +1 n ∑ k ak bn−k k n ∑ k ak k

a(z) ˆ − a(0) ˆ z

(8)

ˆ a(z) ˆ b(z)

(9)

ez a(z) ˆ

(10)

aˆk (z)

(11)

Dz zDz a(z) ˆ

(12)

nan+1

zD2z aˆk (z)

(13)

∆n a0

e−z a(z) ˆ

(14)

n an−1

∑

j1 + j2 +···+ jk =n

(n + 1)an+1

n a j1 a j2 · · · a jk j1 , j2 , . . . , jk

Table 278: Transformations of the exponential generating function function a(z); ˆ so if it converges, then the coefficients an are just the values of the derivatives d a(z) ˆ at the origin: an = dz . z=0

Here are a few examples. For a sequence of the 1’s, we have a(z) ˆ =

zn

∑ 1 n! = ez .

(6.9)

n>0

When the sequence is “geometric,” an = qn , we find a(z) ˆ =

∑

n>0

qn

(qz)n zn =∑ = eqz . n! n>0 n!

For the related case ak = 1 + (−1)k , we have a(z) ˆ =

∑ [1 + (−1)k ] k>0

zk zk zk =∑ + ∑ (−1)k = ez + e−z = 2 cosh z. k! k>0 k! k>0 k!

(6.10)


279

The sequence of falling factorials, {uk }k>0 , has the exponential generating function which coincides with the ordinary one for the sequence of binomial coefficients found in Eq. (6.5): k u k k z =∑ z = (1 + z)u . u(z) ˆ =∑u k! k k>0 k>0 We pay special attention to rule 9 in Table 278 called • BINOMIAL CONVOLUTION: cn = ∑i ni ai bn−i . n ai zi bn−i n−i n zn zn z , ai bn−i = ∑ ∑ c(z) ˆ = ∑ cn = ∑ ∑ n! n>0 i i n! n>0 i=0 i! (n − i)! n>0 where we reverse the order of summation, and bn−i n−i ˆ ai zi z = b(z)a(z). ˆ ∑ i>0 i! n>i (n − i)!

=∑

(6.11)

We summarize the basic properties of exponential generating functions in Table 278. Most are similar to those of the OGF, except the type of convolution it handles, and its transformation when a sequence is shifted. When a sequence is shifted to the right, then instead of losing the prefix, as in Table 275, we now see Dk ∑ a j j>0

zj = j!

z j−k a j( j − 1)( j − 2) . . . ( j − k + 1) = j ∑ | {z } j! j>k jk

∑ aj j>k

zi z j−k = ∑ ai+k . ( j − k)! i>0 i!

Exercise 6.11 [2] Find exponential generating functions for the following sequences (a) an = n!.

(b) bn = 2n + 1.

(c) cn = (n + a)2 .

(d) dn = sn .

(e) en = sn .

Exercise 6.12 [2] Find exponential generating functions for the following sequences and give your answer in terms of trigonometric functions. (a) sn = sin nt.

(b) cn = cos nt.

Exercise 6.13 [2+] Use the binomial convolution to prove a trigonometric form of the Pythagorean theorem: sin2 x + cos2 x = 1. Hint: Use Taylor’s series for sine and cosine.

Sometimes the generating functions are helpful in establishing the equality between two series. Exercise 6.14 [2+] Let Dn = n! ∑06 j6n (−1) j / j! denote the nth derangement number, the number of permutations on n letters without fixed points. For nonnegative integers n and k, min{n,k} k k k+n− j show that two sums ∑ Dk+n− j and ∑ Dk− j are equal. j k j=0 06 j6k j A sequence of functions { fn (x)} is said to be binomial (or to have the binomial property) if it uses binomial convolution in exactly the same way that a Newtonian sequence uses a simple

280


convolution (see page 277): n fn (x + y) = ∑ fk (x) fn−k (y). k=0 k n

Let ˆ z) = F(x;

∑

n>0

fn (x)

(6.12)

zn n!

be the exponential generating function of the sequence { fn (x)}. From the binomial convolution rule (see rule (9) in Table 278), it follows that the sequence { fn (x)} satisfies the ˆ + y; z) = F(x; ˆ z)Fˆ (y; z). For relation (6.12) if and only if its EGF is multiplicative, i.e., F(x n instance, Eqs. (2.52) and (2.75) in §2.5 show that the sequences {x } and {xn } are binomial, and their EGFs, exz and (1 + z)x , respectively, are manifestly multiplicative in x. n−1

Exercise 6.15 [3] Show that the sequence of polynomials { fn (x)}, where fn (x) =

∏ (ax + i=0

bi) = ax(ax + b)(ax + 2b) · · · (ax + bn − b), has the binomial property. Hint: Use the result from Exercise 6.11(e). " # n zn zn = ez a(z), ˆ where a(z) ˆ = ∑n>0 an n! . Exercise 6.16 [2] Show ∑ ∑ ak n! k n>0 k

6.1.3 Some Generalizations In this subsection, we give some generalizations of Definitions 6.1 and 6.10. Definition 6.17 Let {ak }k>0 be a sequence of numbers and gk (z) (k = 0, 1, 2, . . .) be a sequence of linearly independent functions2 . The sum G(z) =

∑ ak gk (z)

(6.13)

k>0

is called a generating function of the sequence {ak }k>0 with respect to the sequence gk (z). ⊳ So far we considered two particular cases: gk (z) = zk and gk (z) = zk /k!. Now we discuss some other generating functions. Probably the most natural generalization is the Newtonian generating function or the binomial generating function: z zn (6.14) = ∑ an . A(z) = ∑ an n! n n>0 n>0 Another useful sequence of numbers is falling factorials which leads to the factorial generating function: (6.15) B(z) = ∑ an zn . n>0

2 A sequence of functions is called linearly independent if any finite subset of them is linearly independent. In other words, any finite linear combination ∑nk=1 ck gk (z) with constants ck cannot vanish.


281

Note the similarity of this pair of functions to the OGF-EGF pair. Another important case deserves its own name—the Appell polynomials3 . For a given sequence of numbers, {an }n>0 , the series of Appell polynomials, {An (x)}, is defined by either n (6.16) An (x) = ∑ ak xn−k or A(z) ezx = ∑ An (x) zn , k n>0 k>0 where A(z) = ∑n>0 an zn is the OGF. The corresponding exponential generating function for zn , the exponential the sequence of polynomials (6.16) can be expressed via a(z) ˆ = ∑n>0 an n! generating function for the original sequence {an }: n n n (xz)n n zn def −k n−k z ak x = ∑ ak x ∑ F(x, z) = ∑ An (x) = ∑ ∑ n! n>0 k=0 k n! k>0 n! n>0 n>k k =

∑ ak k>0

(xz)n x−k x−k = ∑ ak (xz)k exz = exz a(z). ˆ ∑ k! n>k (n − k)! k>0 k!

Hence we get def

F(x, z) =

∑ An(x)

n>0

zn zn = exz a(z) ˆ ≡ exz ∑ an . n! n! n>0

(6.17)

If the exponential generating function, F(x, z), of the sequence of polynomials {An (x)} satisfies the relation (6.17), then such polynomials are referred to as the Appell polynomials. Since the derivative of the polynomial (6.16) is expressed as n n−1 ′ n−k−1 An (x) = ∑ ak (n − k) x = n∑ ak xn−1−k = n An−1 (x), (6.18) k k k k we get that its EGF is a solution of the following differential equation

∂ F(x, z) = z F(x, z), ∂x

(6.19)

which sometimes allows us to determine F(x, z) without summations. For instance, the Bernoulli polynomials, defined in Eq. (B.1), and the Euler polynomial, Eq. (C.54), form a sequence of Appell polynomials. Definition 6.18 For a given sequence of numbers {an }n>1 , we define the Dirichlet generating function (also called the Dirichlet series) by an def ℜs > α > 1, (6.20) D(s) = ∑ s , n>1 n where α is a real number for which the series (6.20) converges. The inverse transform is obtained by the line integral: n−1

n ∑ (n − k) ak = 2π i k=1

3 Named

Z a+i∞ D(s) ns a−i∞

s(s + 1)

ds,

´ after the French matehmatician Paul Emile Appell (1855 – 1930).


282

where a > max{1, α } and the integral in the right-hand side is the limit over symmetrical interval: Z a+iω Z a+i∞ f (s) ds, (6.21) (V.P.) f (s) ds = lim ω →∞ a−iω

a−i∞

called the principal value or the Cauchy principal value and is denoted by V P or V.P.

⊳

def

For example, the constant sequence {1}n>1 has the Dirichlet transform ζ (s) = ∑n>1 n1s , the Riemann zeta function. Its square gives the Dirichlet generating function for the sequence of divisors: 1 1 dn 1 1 ζ 2 (z) = ∑ z ∑ z = ∑ ∑ =∑ =∑ z ∑ z z i>1 j>1 (i j) n>1 j divides n n n>1 n i>1 i j>1 j where dn is the number of divisors of n. Exercise 6.19 [1+] Let k be a positive integer and bn,k be the number of integers whose binary representation ends with k sequential zeroes. Show that the Dirichlet generating function for the sequence {bn,k }n>1 is ζ (z)/2kz .

Exercise 6.20 [3+] Let {an }n>0 and {bn }n>0 be two sequences of numbers and let A(z) and B(z) be their Newtonian generating functions. Prove thatC(z) = A(z)B(z) is the Newtonian n generating function for the sequence cn = ∑ ai+ j b j+k . i+ j+k=n i, j, k Example 6.21 (Newton’s factorial series) Let f be an entire function (that has infinitely many derivatives at any finite point in the complex plane) and let x0 , x1 , . . ., xn , . . . be a sequence of points that we assume for simplicity to be arranged with equal spacing: xk = x0 + kh, k = 0, 1, 2, . . .. Let us denote by f (k) the def value of the function f at xk , that is, f (k) = f (xk ). Using the shift operator, E f (k) = f (k + 1), we obtain from Eq. (5.83), page 266, that n n k ∆k f (0) k n, f (n) = E f (0) = (1 + ∆) f (0) = ∑ ∆ f (0) = ∑ k! k=0 k k=0 n

n

n

(6.22)

where ∆k f (0) = ∆k f (x) x=0 denotes the forward difference of order k of the function f at the point x = 0. Eq. (6.22) is called Newton’s forward-difference formula, which we met on page 267. Similarly, using the backward difference operator, ∇, we can derive the Newton’s backward-difference formula: k −s f (s) = ∑ (−1) ∇k f (0). k k>0 Taking Eq. (6.22) off the integer grid leads to the Newton’s factorial series: f (x) =

∆k f (0) k ∑ k! x k>0

and

f (x) =

∇k f (0) k ∑ k! x . k>0

(6.23)


283

The former series is the factorial generating function for the sequence we obtain the binomial generating function x k f (x) = ∑ ∆ f (0) k>0 k

1

k!

∆k f (0) . Similarly, (6.24)

for the sequence {∆k f (0)}, k = 0, 1, . . .. For example, Eq. (5.88) shows that ∆k 0d = k! dk , d where k are Stirling numbers of the second kind. Exercise 6.22 [2] Using Eq. (6.23), derive Vandermonde’s formula from Exercise 2.128.

6.1.4 Multivariate Generating Functions We can easily extend the definition of the various generating functions to multi-index sequences. Definition 6.23 Let {an,k }, k, n = 0, 1, 2, . . ., be a sequence of numbers. The sum in two variables (6.25) a(u, z) = ∑ ∑ an,k un zk n>0 k>0

is called the bivariate (ordinary) generating function. Similarly we can define the exponential generating function (in one of the arguments) A(u, z) =

zk

∑ ∑ an,k un k!

or B(u, z) =

un

∑ ∑ an,k n! zk .

(6.26)

n>0 k>0

n>0 k>0

If the number of indices is more than 2, the corresponding generating function, also called the multivariate (ordinary or exponential) generating function, is defined in a similar way. ⊳ Note that we can assign a “double exponential” generating function to the double-index sequence {an,k } un zk , (6.27) a(u, ˆ z) = ∑ ∑ an,k n! k! n>0 k>0 but this function isn’t useful. All the above generating functions, a(u, v), a(u, ˆ v), A(u, v), and B(u, z) are particular cases of the following general form: F(u, z) =

∑ ∑ an,k fn (u)gk (z),

(6.28)

n>0 k>0

where { fn (u)} and {gk (z)} are two given sequences of functions. A double-index sequence {an,k } can be arranged in an infinite matrix, with an,k placed in row n and column k. A sum in one of the indices an (z) =

∑ an,k zn k>0

and

ak (u) =

∑ an,k uk

n>0


284

are sequences (depending on a parameter) of ordinary generating functions for {an,k }k>0 and {an,k }n>0 , respectively. Sometimes the functions an (z) and ak (u) are referred to as horizontal and vertical generating functions, respectively. Example 6.24 Let us consider the array of binomial coefficients: an,k = nk , n, k ∈ N. Since the binomial coefficients in this range vanish if the lower argument exceeds the upper one, we have n n n k 1 a(u, z) = ∑ ∑ u z = ∑ un (1 + z)n = 1 − u(1 + z) n>0 k=0 k n>0 because the horizontal generating function is ∑n>0 nk un = uk (1 − u)−k−1 and the vertical generating function is ∑k>0 nk zk = (1+ z)n . The corresponding exponential generating function becomes un un n n k z = ∑ (1 + z)n = eu(1+z) . B(u, z) = ∑ ∑ n>0 n! n>0 n! k=0 k Note that the function B(u, z) is the exponential generating function for the sequence bn,k = nk . Exercise 6.25 [2+] Obtain the various GFs for the closely related sequence of multinomial coefficients k1 ,k1N,...,kn , where k1 + k2 + · · · + kn = N. Use x j to mark the k j and z to mark N.

Exercise 6.26 [3] An explicit form for the sequence of coincidence numbers, defined in Definition 5.131, page 258), and discussed in Example 5.132, is given by 258) dn,k =

n! n−k (−1) j ∑ j! , k! j=0

k = 0, 1, 2, . . . , n, ∞

Compute its bivariate generating function D(z, u) =

n

n = 0, 1, . . . .

(6.29)

un

∑ ∑ dn,k zk n! .

n=0 k=0

Exercise 6.27 [3+] Let N(n, k) be the number of sequences that can be formed from n 1’s and k (−1)’s, such that all the partial sums are non-negative, that is, the number of arrangements of a1 , a2 , . . . , an+k so that a1 + a2 + · · · + ai > 0 for all

1 6 i 6 n + k.

Justify the recurrence N(n + 1, k) = N(n + 1, k − 1) + N(n, k) for 1 < k < n + 1,

n = 1, 2, . . . ,

and the boundary conditions N(n, 0) = 1 for n > 0,

N(n, 1) = n,

N(n + 1, n + 1) = N(n + 1, n) = Cn+1 , for n > 1,

1 2n where Cn = n+1 n is the Catalan number. Derive the bivariate generating function for the double sequence {N(n, k)}, n = 1, 2, . . .; k = 0, 1, . . . , n.


285

2 Exercise 6.28 [3] Find the bivariate generating function for the sequences an,k = nk , bn,k = 2 . Hint: Use the generating function (L.59) in Appendix L for the k nk , and cn,k = 2n+1 k Legendre polynomials. Exercise 6.29 [2] Find the exponential bivariate generating functions for the Stirling cycle numbers. Hint: See Example 2.145(a) on page 74.

6.1.5 Multisection of Series The variables of our generating functions (GFs) are referred to as indeterminate, or formal variables, but there is much to gain from viewing them as numbers, which may be complex numbers. With two examples, we hope to convince you that while GFs can live on a diet of real numbers, they really flourish on the complex plane. Chapter 12 carries this process much further. The first example in §12.3.1, Diagonalization of series, is developed in detail since it requires integration. Here we show another device that helps in the evaluation of a certain type of series. Given a sequence {ak }k>0 and its generating function a(z) = ∑ ak zk , we may be asked to k>0

compute the sum of every second term: ∑k>0 a2k . Similar situations seem to crop up often enough in analysis of algorithms to deserve a unified approach. The following computation gives what we need: # " 1 1 1 ak + ∑ ak (−1)k = ∑ ak 1 + (−1)k = ∑ a2k , (6.30) [a(1) + a(−1)] = ∑ 2 2 k>0 2 k>0 k>0 k>0 n ? The ordinary generating since only the even terms survive. For example, what is ∑k 2k n k n n . Following the function for the sequence ak = k is known to be a(z) = ∑k k z = (1 + z) n n idea above, we see that a(1) = 2 , and a(−1) = δ0,n , and therefore, ∑k 2k = (2n + δ0,n )/2, where δ0,n is the Kronecker delta symbol (2.5), page 29. Not a surprising result, really.

The same idea holds when we need the sum of every rth term, for any r ∈ I2 ≡ {2, 3, . . .}. The calculation now calls for the GF of the sequence evaluated at the unit roots of order r.

The unit roots of order r are the r complex numbers ωk,r = e2iπ k/r , k = 0, 1, . . . , r − 1, where i is the unit vector in vertical direction so that i2 = −1 (see§12.1). We call ω1,r the primitive r = e2iπ /r r = e2iπ = 1, and it is called the unit root of order r. It is a root of 1, since ω1,r primitive one since the kth power of ω1,r equals ωk,r ; hence its powers generate all the other r − 1 unit roots of order r. Note: You may think that this is less than a perfect reason for the term ‘primitive’ since it is possible to find other non-primitive roots that have the same property (although they generate the other roots in a different order!). For example, u = e2×2iπ /3 and its powers u2 and u3 are the three roots of order three. Very true, but notation and definitions aim at naturalness, not artificial uniqueness. def

Let us keep r fixed, and denote ω = ω1,r . Such roots of unity satisfy the following “circular”


286

(or orthogonal) relation: r−1

∑ (ω

k j

) =

j=0

(

r, when k is a multiple of r; 0, otherwise.

(6.31)

k j Eq. (6.31) can be rewritten as ∑r−1 j=0 (ω ) = δk,mr for some m ∈ N, where δ is Kronecker’s symbol. Indeed, let k be any positive integer, then r−1

∑ (ω k ) j = j=0

1 − ω kr 1 − (ω k )r = . 1 − ωk 1 − ωk

1 − ω kr

= 1 − (ω r )k = 1 − 1 = 0 for any integer k; unless k is a multiple The numerator is of r the denominator is nonzero, and the right-hand side above vanishes. However, when k = mr for some integer m, the right-hand side has the form 0/0, and we use l’Hôpital’s rule to evaluate it. Differentiating both numerator and denominator (with respect to ω ), we obtain −krω kr−1 = rω k(r−1) = rω mr(r−1) = r (ω r )m(r−1) = r. −kω k−1 We use this result in the following calculations: r−1 1 r−1 1 1 1 r−1 j jk k j a( a a ω ω ) = = k k ∑ (ω ) = ∑ ∑ ∑ ∑ ∑ ak × rδk,mr = ∑ amr . r j=0 r j=0 k>0 r k>0 j=0 r k>0 m>0

Such a sum is called multisection of order r. n . We pick the term ak = Example 6.30 Compute ∑ k 3k n k n ∑k k z = (1 + z) . We showed previously that n 1 ∑ 3k = 3 k

n k

(6.32)

that yields the OGF: a(z) =

2

∑ a(ω j ),

j=0

with ω here be the cubic root of one: √ √ 1 1 3 3 2π i/3 2 −2π i/3 , ω =e , ω =e = − +i = − −i 2 2 2 2

1 + ω + ω 2 = 0,

and substitution provides a less-than-obvious expression: 1 n 1 ∑ 3k = 3 2n + (1 + ω )n + (1 + ω 2)n = 3 k

n

2 +

√ !n 1 3 + +i 2 2

√ !n ! 1 3 . −i 2 2

√

However, since 12 ± i 23 = e±iπ /3 , we use Euler’s relation (12.3) and get 1 nπ n 1 n inπ /3 n −inπ /3 2 + e 2 + 2 cos . = + e = ∑ 3k 3 3 3 k Hence the multisection is essentially a third of the full sum, with a small oscillation.

6.2. EXTRACTION OF COEFFICIENTS Exercise 6.31 [2−] Find the sum

287

1

∑ (3k)! .

k>0

Exercise 6.32 [3] Find the sum

⌊n/2⌋

∑

n 1 . 2k + 1 2k

k=0 ∑n>0 y5n .

Surely, you can do this geometric sum directly. Show Exercise 6.33 [3] Calculate how to recover the same result when viewing this sum as a 5-way multisection. Exercise 6.34 [2] The following series are simple geometric series. Nevertheless, show how the multisection mechanism handles them. You can use the GF from Eq. (L.22), page 742. (a)

x2k , ∑ 2k+1 k>0 (1 − x)

(b) ⌊n/2⌋

Exercise 6.35 [2] Find the sum ∑k=0

x3k . ∑ 3k+1 k>0 (1 − x)

n k 2k (−1) .

Exercise 6.36 [3] Redo Exercise 5.50 on page 218 by applying the multisection technique to the corresponding generating function. √ 2n Exercise 6.37 [3] Show ∑ (−a) j = (1 + a)n cos[2n arctan a]. Use this result to evalj>0 2 j j uate the sum ∑ j 2n 2 j (−3) . Exercise 6.38 [2] Using the generating function for the Fibonacci numbers, Eq. (D.42) in Appendix D or Eq. (6.58), evaluate the sum ∑ F3n 2−3n . n>0

n m 2n+m ± a(m − n, r) , where a is 1 or 2 Exercise 6.39 [3] Prove that = ∑ 3 k j−k=r mod 3 j and the summation is over all k and j that satisfy the modular relation; and r ∈ {0, 1, 2}. Give a simple rule to determine the value of a.

6.2 Extraction of Coefficients In the previous section, we established a one-to-one correspondence between sequences of real (or complex) numbers and power series. In many applications we also need to know a particular coefficient of the obtained power series, and therefore we introduce a special operation (and corresponding notation), which we call the coefficient operator. However, strictly speaking, this operator should be called (and it is called by some authors) the functional because it assigns a number (coefficient) to every function written in power series form. Definition 6.40 (The coefficient operator or functional) Let f (z) = ∑n>0 an zn be a series in powers of z. Then the symbol [zn ] f (z) stands for the coefficient an of zn in the power series expansion of f (z). We call this operation extraction of dn ⊳ the coefficient. Whenever the differentiation is possible, [zn ] f (z) = n!1 dz n f (z)|z=0 .


288

How can we compute or evaluate the effect of this operator? Again, there is no algorithm, although there is a general formula, using differentiation, which is rarely helpful. For the most part, we rely on several familiar cases: the power series representation of standard functions. Combined with a few rules we list below, this approach carries the day in a surprising number of situations; but not always, in which case we turn to a variety of analytic or numerical methods that are discussed in subsequent chapters. One specialized resource needs to be mentioned: computer algebra systems, like MAPLE4 or MATHEMATICA5 , all have a procedure that computes the prefix (of any given length) of the series expansion. Here are some examples of coefficient extraction (either “obvious” or left as exercises): 1. a(z) = ez 2. a(z) = c etz 3. a(z) =

=⇒ [zn ] ez =

1 , n!

=⇒ [zn ] a(z) = c

c 1 − bz

n ∈ N. tn , when c is a constant. n!

=⇒ [zn ] a(z) = c bn , when c and b are constants. α

4. a(z) = (s + tz)

α n α −n t s =⇒ [z ] a(z) = . Part 3 is a special case of this one. n n

5. a(z) = ln[(q − z)−1 ] =⇒ [zn ] a(z) =

1 . n qn

See Exercise 6.41(b).

2n/2 sin n4π . 6. a(z) = e sin z =⇒ [z ] a(z) = n! nr + s qs+1 (z) n , =⇒ [z ]a(z) = , r ∈ I2 = {2, 3, . . .}, s ∈ R, where 7. a(z) = r − (r − 1)q(z) n q(z) is the root of the equation q = 1 + zqr . z

n

Appendix K contains a large number of power series expansions. Exercise 6.41 [1] (a) Using the binomial theorem, justify the claim made in part 4 above. (b) The statement in part 5 is not complete. What is missing? We will repeatedly use the following rules, when extracting coefficients (n ∈ N): Rule 1. (linearity)

[zn ] ( f (z) ± g(z)) = [zn ] f (z) ± [zn ] g(z),

Rule 2. (constant multiplication)

1 n [z ] f (z), [zn ] f (az) = an [zn ] f (z). c [zn ] zb f (z) = [zn−b ] f (z), b ∈ Z.

[c zn ] f (z) =

Rule 3. (power function multiplication)

[zn ] (c f (z)) = c [zn ] f (z).

4 In MAPLE : taylor(expression(z), z=0, n); 5 In MATHEMATICA : Series[expression[z], { z,

0, n}]

6.2. EXTRACTION OF COEFFICIENTS

289

n

[zn ] f (z)g(z) =

Rule 4. (convolution)

∑ [zk ] f (z) × [zn−k ] g(z).

k=0

Rule 5. (shifting)

k f (z) = g(z + a) =⇒ [(z + a) ] f (z) = gn ; [z ] f (z) = ∑ gk ak−n . n k n

Rule 6. (composition)

n

∑ [zn] f (z)gn (x) = f (g(x)). n

Rule 8. (inversion)

Z

1 f (z) dz = [zn−1 ] f (z) n n z k n−k k n . [z ] f (z) = [z ] n f (z)

Rule 7. (differentiation)

n

[z ]

or

1 [zn ]g(z) = [zn−1 ]g′ (z). n

In Rule 8, f (z) denotes the compositional inverse of f (z), that is, the formal power series such that f ( f (z)) = f ( f (z)) = f ( f (z)) = z. It is known that a formal power series f (z) has a compositional inverse if and only if [z0 ] f (z) = 0 and [z] f (z) 6= 0. Rule 8 is the Lagrange inversion theorem, which is presented in §6.7. Most of the rules are relevant with no other considerations whether the generating function we are looking at is an ordinary or exponential one. Since [zn ]a(z) = an , [zn ]a(z) ˆ = an /n!, the only change for exponential functions is required: remove the extra factorial to recover the coefficient an . As a last resort, when all fails, we have the following consequence of the Cauchy residue theorem (page 682), I a(x) 1 dx, (6.33) [(z − u)n ]a(z) = 2π i γ (x − u)n+1 where the function a(z) is analytic in and on γ . The contour γ is a closed, simple (= it does not intersect itself) curve in the complex plane (Definition 12.38). This relation is the basis of many asymptotic methods. In our applications, the point u is almost always the origin, and the contour is a circle centered there. A computer algebra system like MAPLE has a special command to extract the coefficient: coeftayl(function(z), z=0, n);

Example 6.42 From Rules 1 and 4 and Examples 3 and 4 on page 288, we see how to handle extraction of coefficients from a generating function that is the ratio of a polynomial and a binomial. For example, [zn ]

a + bz + cz2 + dz3 = avn + bvn−1 + cvn−2 + dvn−3 = av3 + bv2 + cv + d vn−3 . (6.34) 1 − vz

Clearly, the same rule is applied when the numerator is any function g(z) for which a power series expansion, g(z) = ∑ j>0 g j z j , is known. (We might need to go through an extraction process on its own to get these g j .) It becomes a specific case of Rule 5 above: [zn ]

n n g(z) = ∑ [zk ] g(z) × [zn−k ] (1 − vz)−1 = ∑ gk vn−k . 1 − vz k=0 k=0

(6.35)

290


In particular, when v = 1, we get from Eq. (6.35) the partial sum formula (also followed from Exercise 6.3) g(z) (6.36) = ∑ [zn ]g(z) = g0 + g1 + · · · + gn . [zn ] 1 − z 06k6n Exercise 6.43 [2] Show that [zn ] 2z(1 − 4z)−3/2 = n 2n n .

Exercise 6.44 [2] Write an equation analogous to Eq. (6.35) when the binomial in the denominator is not (1 − vz), but (u − vz), with u 6= 0.

Exercise 6.45 [1] Let h(z) = ∑k>0 ak zk be the ordinary generating function of the sequence {ak }; compute [zn ]h(z)(1 − z)2 . Exercise 6.46 [4] Show that [zn ](1 − z)k f (z) = ∇k [zn ] f (z).

Example 6.47 Since the trigonometric (sine and cosine) and hyperbolic (sinh, cosh) functions are sums and differences of exponentials, Rule 1 (page 288) makes extraction of coefficients immediate as well. For example, cos z = (eiz + e−iz )/2, hence [zn ] cos z is given by ( (−1)k n = 2k, in 1 n iz −iz n n (2k)! , = (1 + (−1) ) = [z ] cos z = [z ] e + e 2 2n! 0, n is odd.

Exercise 6.48 [2] Using Example 6.47 and the de Moivre formula (12.6), page 664, show that [zn ] ez sin z = n!1 2n/2 sin n4π .

z Exercise 6.49 [3−] Extract the coefficients of the function f (z) = 1−2z+2z 2. Hint: You will find a relationship with the previous exercise! e−az . We view h(z) as the product of Example 6.50 Let us extract coefficients from h(z) = z−b two functions, and plan to use rule 4. For preparation, we write

h(z) =

−1/b

1

∑ hnzn = e−az z − b = e−az 1 − z/b ,

n>0

and then n −1/b −1/b = ∑ [zi ]e−az [zn−i ] 1 − z/b i=0 1 − z/b 1 n (−ab)i 1 n (−a)i 1 n−i , = − n+1 ∑ = − ∑ b i=0 i! b b i! i=0 | {z }

hn = [zn ] e−az ×

≡en (−ab)

where en (x) is a notation for the incomplete exponential function: n

en (x) =

∑ k=0

xk . k!

(6.37) i

= e−ab . In our case, the complete exponential function would be: ∑i>0 (−ab) i! −ab We can consider en (−ab) ∼ e if ab < 1 or n is sufficiently large.


291

Exercise 6.51 [3] (R. Stanley) Find the coefficient of zn in each of the following functions: r z 2 1+z . ; (b) sin(t sin[−1] z); (c) cos(t sin[−1] z); (d) 2 sin[−1] (a) 1−z 2 Note: Not all parts have closed-form answers. The symbol sin[−1] z is the inverse of the sine function, also denoted by arcsin z. ln(1 − z) , r ∈ N. Example 6.52 Sometimes an indirect approach works best. Let a(z) = (1 − z)r+1 Again a(z) is a product of (1 − z)−(r+1) and ln(1 − z), and we use the convolution scheme. The individual components contribute as follows: −r − 1 r+k r+k k −(r+1) k [z ](1 − z) = (−1) = = , k > 0, k k r and [zk ] ln(1 − z) =

−1 , for k > 1. Therefore k n r + n − k −1 n =?? [z ]a(z) = ∑ k r k=1

(6.38)

It is not clear how to obtain a closed-form for Eq. (6.38), so we try another approach, using the following fortuitous observation: d ln(1 − z) d (1 − z)−x = e−x ln(1−z) = e−x ln(1−z) (− ln(1 − z)) = − . dx dx (1 − z)x Hence the function a(z) has a different representation: a(z) = −

d (1 − z)−x x=r+1 . dx

Next we use the commutativity of the operators [zn ] and

d dx :

   x + n − 1 d d d  [zn ]a(z) = −[zn ] (1 − z)−x x=r+1 = − [zn ](1 − z)−x x=r+1 = −   dx dx dx n x=r+1 n−1  x+n−1 1  d (x + n − 1)(x + n − 2) · · · x =− =−  ∑ dx n! n x=r+1 j=0 x + j x=r+1 m n + r n+r 1 1 n + r n−1 1 =− , and since ∑ = Hm − Hk−1 , =− ∑ ∑ n n i=r+1 i j=0 r + 1 + j j=k j n+r [zn ] a(z) = − (Hn+r − Hr ). n n+r r+n−k 1 n So, we have found a new summation formula: ∑k=1 k = n (Hn+r − Hr ). r

292


i n + m ln2 (1 − z) h (2) (2) 2 , for = (H − H ) − H − H m n+m m n+m n (1 − z)m+1 (2) m, n ∈ N, where Hn ≡ ζ (n, 2) = ∑nk=1 (1/k2 ) is the second order harmonic number.

Exercise 6.53 [4] Show [zn ]

Exercise 6.54 [1] Show the following identity, by extracting coefficients of each side: xk

x

∑ 2k − 1 = ∑ 2n − x .

k>1

(6.39)

n>1

Example 6.55 (Finite differences and the EGF) def We wish to calculate qn = ∑i ni (−1)n−i ai , where the {ai } are not given explicitly; only zn , as in Exercise 6.62. their EGF is provided, a(z) ˆ = exp(z + z2 /2) = ∑n>0 an n! There is no real difficulty in getting an expression for an , but it is not obvious that it helps much. Writing a(z) ˆ as a product of exp(z) and exp(z2 /2), we realize that an is a binomial convolution. The appearance of the z2 in the EGF makes the convolution a bisection6 , and we end up with i (2k)! ; ai = ∑ 2k 2k k! k inserting that into the sum ∑i ni (−1)n−i ai is not inviting. However, this sum was carefully selected, to make this example interesting. If you look at page 266, where we discuss the difference operators, and in particular, Eq. (5.81), you will find that qn = ∆n a0 , the n-order difference of the first element of the sequence {ai }. If you go back further, you will find in i (see also Eq. (A.66) in Appendix the binomial identities (2.59) and (2.62) that ∆i ij = j−1 A). Hence 0 (2k)! 0 (2k)! n n =∑ , qn = ∆ a0 = ∆ ∑ k 2k − n 2k k! 2k 2 k! k k 0 and since 2k−n = δn,2k , (δ is Kronecker’s symbol), the only contribution to the sum arises when 2k = n, and we find that qn = n!/(2n/2 (n/2)!), if n is even and zero otherwise.

The fact that the sequences {ai } and {qi } are related via the difference operator suggests that there exists an alternative route to associate their EGFs through the following transformation: q(z) ˆ =

∑ ∆na0

n>0

n zn zn (−1)n−i ai . =∑ ∑ n! n>0 n! i i

The expected reversal of the order of summation produces q(z) ˆ =∑ i

6A

2 zn−i ai i z ∑ (−1)n−i = e−z a(z) ˆ = ez /2 . i! n>i (n − i)!

special case of a multisection, discussed in §6.1.5. See Eq. (6.30) there.


293

Now all is clear again, and we get as before: qn =

(

(2m)! m!2m ,

n = 2m; n is odd.

0,

Hence we get the transformation cn = ∆n a0

=⇒

c(z) ˆ = e−z a(z) ˆ

(6.40)

6.2.1 Transformations between the Generating Functions We have seen two quite different functions associated with the same (infinite) sequence— ordinary and exponential generating functions. In future applications, you are likely to feel that they are even more different than the formal definitions suggest. Still, it is possible to transform one generating function analytically to the other type. By this we mean that the ordinary generating function is expressed though the exponential generating function although they seem to be quite different. The corresponding formula is known as the Laplace-BorelRtransform, and is based on the definition of the gamma function via an integral: Γ(n) = 0∞ e−t t n−1 dt, and Γ(n) = (n − 1)!, when n is a positive integer. Consider the following calculation Z ∞ 0

−t

a(zt)e ˆ dt =

∑

Z ∞

n>0 0

an

zn (zt)n −t e dt = ∑ an Γ(n + 1) = a(z) n! n! n>0

after the factorial and Γ(n + 1) cancel out. Hence a(z) =

Z ∞ 0

−t a(zt)e ˆ dt,

Laplace-Borel transform.

(6.41)

It turns out that expressing the exponential generating function through the ordinary counterpart is more complicated. Indeed, the only prescription we can give is to write the sequence terms of the EGF as ak = k! [zk ]a(z), ˆ and insert them into the definition of the OGF. As we shall see in our applications, the prevalence of rational OGFs suggests something slightly better. If we have the OGF a(z) = 1/(1−qz), then ai = qi , and the EGF becomes a(z) ˆ = ∑i (qz)i /i! = eqz . This suggests the transformation (1 − qz)−1

7→

eqz .

(6.42)

Things are rarely quite that simple, but when the OGF is an arbitrary rational function, partial fraction decomposition will produce terms of the type 1/(1 − qz) or its powers, even though we may need to accept rather complex forms for the constants q. When the ordinary generating function has a double or a triple pole, transformation to the corresponding EGF becomes more complicated. For instance, the OGFs a2 (z) = 1/(1 − qz)2 and a3 (z) = 1/(1 − qz)3 have the EGFs aˆ2 (z) = ∑i (qz)2i /(2i)! = cosh(qz) and aˆ3 (z) = ∑i (qz)3i /(3i)!, respectively. Example 6.56 Let us consider the sequence {n2 }n>0 ; its ordinary generating function is a(z) =

1

1+z

∑ n2 zn = (zDzD) ∑ zn = (zDzD) 1 − z = z (1 − z)3 ,

n>0

n>0


294

d is the operator of differentiation. Similarly, the exponential generating function where D = dz of the same sequence becomes

a(z) ˆ =

zn

zn

∑ n2 n! = (zDzD) ∑ n! = (zDzD) ez = (z + z2) ez . n>0

n>0

The Borel transform of the function a(z) ˆ can be calculated as follows Z ∞

zt −t

2 2

2

(z t + zt) e e dt = z

0

Z ∞

z−1)t 2

e

t dt + z

0

Z ∞

ez−1)t t dt =

0

R R because 0∞ t 2 e−at dt = 2/a3 and 0∞ t e−at dt = 2/a2 .

z 2z2 + = a(z) , 3 (1 − z) (1 − z)2

Consider another ordinary generating function

21 1 96 1 − 2z − 3z2 − + = 28 − 51z − 265z2 − 102z3 1441(7 + 3z) 655(−4 + 17z) 55(1 + 2z) 21 1 96 . + + = −3z 17z 55(1 + 2z) 10087(1 − 7 ) 2520(1 − 4 )

a(z) =

This yields the corresponding exponential generating function to be a(z) ˆ =

21 17z/4 1 −2z 96 −3z/7 e + e + e . 10087 2520 55

The partial fraction decomposition was done by MAPLE, and the rest is straightforward. ˆ = ∑n>0 aˆn zn and A(z) = ∑n>0 an zn be exponential and ordinary Exercise 6.57 [2] Let A(z) n! generating functions for the given sequences {an } and {aˆn }, respectively. Show that z 1 zn n −z ˆ (b) A = ∑ (∆n a0 ) zn . (a) e A(z) = ∑ (∆ aˆ0 ) . n! 1 + z 1 + z n>0 n>0 Exercise 6.58 [2] For two sequences {an }n>0 and sb = ∑nk=0 nk bn−k ck ak , let A(z) = ∑n>0 an zn , ˆ = ∑n>0 sn zn be their ordinary and exponential ˆ = ∑n>0 an zn , S(z) S(z) = ∑n>0 sn zn and A(z) n! n! generating functions. Prove that cz 1 ˆ = ebz A(z)(cz). ˆ A , (b) S(z) (a) S(z) = 1 − bz 1 − bz

6.2.2 Multivariate Generating Functions There is nothing to extracting coefficients from multivariate GFs beyond successive application of the operations we described previously. A relevant rule is that the operators of coefficient extraction commute. As an example, consider the GF computed in Example 6.24 (page 284), where we use rules 3 and 4 from page 288, in this order: n 1 1 k n k n k n = [v ] [u ] = [v ](1 + v) = . an,k = [v u ] 1 − u(1 + v) 1 − u(1 + v) k


295

We also get ! 1 1 1 1 n k n k = [u ] [v ] = [u ] [v ] an,k = [v u ] vu 1 − u(1 + v) 1 − u − uv) 1 − u 1 − 1−u ! k ! 1 n 1 uk u 1 n k n n = [u ] [v ] = [u ] = , = [u ] vu k 1−u 1 − 1−u 1−u 1−u (1 − u)k+1 k n

where we now used the same rule 3, followed by a form of the binomial theorem, as given in Eq. (L.18), page 742. Sometime we come across bivariate generating functions that are ordinary in one variable and exponential in another one. For instance, suppose we are given the function f (z, u) = zn k u and we would like to multiply it by ez(u−1) . Extracting coefficients, we ∑n>0 ∑k>0 fn,k n! obtain n− j n n k z(u−1) [z u ]e f (z, u) = ∑ ∑ k − i f j,i (−1)n− j+i−k . j i j Exercise 6.59 [2+] (Coefficients from multivariate GF) (a) Let a(x) = ∑k>0 ak xk and define x = x1 + x2 + · · · + xm , then show n1 + n2 + · · · + nm n1 n2 nm [x1 x2 · · · xm ]a(x) = an1 +n2 +···+nm n1 , n2 , · · · , nm (b) Show the following extraction, which comes up in the analysis of quicksort with duplicate keys: m m n1 + · · · + ni + i − 1 1 n1 n2 nm =∏ . [x1 x2 · · · xm ] ∏ ni i=1 i=1 1 − (xi + xi+1 + · · · + xm ) Example 6.60 Some generating functions may have closely related coefficients. For instance, consider the following two generating functions n n k zn 1−u def def = u , E(u, z) = e−z(u−1) e(u, z) = e−z(u−1) ∑ ∑ n! 1 − u e−z(u−1) n>0 k=0 k where E(u, z) is the generating function for Eulerian numbers and e(u, z) = ez(u−1) E(u, z). Of course, we can use the convolution rule to relate the coefficients of these two functions, E(u, z) and e(u, z): n n n−k {e(n, k)} = (−1) ⋆ , k k def

where e(n, k) = n![uk zn ]e(u, z) are the coefficients for the bivariate generating function e(u, z). However, a closer relation exists. Let B = e−z(u−1) = ez(1−u) . Then using the geometric series, we get e(u.z) =

1−u = (1 − u) ∑ u j B j = 1 − u + (1 − u) ∑ u j+1 B j+1 , 1 − uB j>0 j>0


296

E(u, z) = B

1−u = (1 − u) ∑ u j B j+1 . 1 − uB j>0

Therefore 1 + uE(u, z) = 1 + u(1 − u) ∑ j>0 u j B j+1 = 1 + (1 − u) ∑ j>0 u j+1 B j+1 = 1 + e(u.z). def Hence e(u, z) = 1 − u+ u E(u, z). Extracting the coefficients for n > 0, we obtain e(n, k) = n n! [uk zn ]e(u, z) = k−1 , and for n = 0 we have e(k, 0) = 1.

6.2.3 Recurrences from Generating Functions Normally, a generating function (GF) passes through a recurrence that its terms satisfy. In §6.4, we even show how to obtain a GF from a given recurrence, but this is not always the case. Later we shall see situations where the symbolic method allows us to obtain a GF directly. We then have the terms, right? Well, implicitly. If we can extract the coefficients and get a useful expression, then yes, we are done. If not, we show in Chapter 11 a way to extract information from a GF about the behavior of the terms for large index value (in fact, as the index value grows to infinity), and sometimes this information is valid for index values as small as 5 or 10. A good example here is the Stirling approximation. But sometimes we have a generating function and cannot extract coefficients usefully, and the asymptotics are not helpful one way or another. . .. Then we want recurrences, assuming we can figure out some initial values; these are usually easy to obtain since we know the context. One way that often works is differentiation. The basic idea that the derivative of GF ϕA (z) = ∑k ak zk is given by ϕA′ (z) = ∑k k ak zk−1 , and comparing coefficients of equal powers of z completes the process. As is often the case, an example is best here. For the GF ϕS (z) = (1 + z)n , differentiation yields

∑ kak zk−1 = n(1 + z)n−1 k>0

=⇒

(1 + z) ∑ kak zk−1 = n(1 + z)n = n ∑ ak zk . k>0

k>0

Equating coefficients of zk on both sides we have (k + 1)ak+1 + k ak = n ak , which can be written as ak+1 = (n − k) ak /(k + 1). We know that a0 = ϕ (0) = 1, and this is enough for a first-orderrecurrence; now we can compute a1 = n/1 × 1 = n, a2 = (n − 1)/2 × n = n(n − 1)/2 = n2 , and so on, we can calculate to our heart’s desire.

Exercise 6.61 [4] Let f (z) and g(z) have the expansions f (z) = ∑k>0 fk zk and g(z) = ∑k>0 gk zk ; develop an expression, as explicit as you can, for [zn ] f (g(z)). This exercise calls for the Faàdi-Bruno formula, given in Eq. (F.20), page 723. Hint: try f (u) = eau . Exercise 6.62 [2] In §6.4, we show methods for solving recurrences using GFs. Here you are asked to go in the reverse direction: Show that the coefficients in the exponential generating function 2 zn a(z) ˆ = ∑ an = ez+z /2 , n! n>0 satisfy the second-order recurrence an = an−1 + (n − 1)an−2 .


297

When we need to determine a power series for the reciprocal of a high-degree polynomial, 1 (n = 0, 1, 2, . . .) is required. One way to get at p(z), the extraction of coefficients [zn ] p(z) these coefficients is to use partial fraction decomposition, an approach that requires us to solve the equation p(z) = 0. When the polynomial is a quadratic, this is simple. However, when the polynomial is of higher degree, the solution promptly gets very difficult. For a polynomial of fifth degree or higher, there is typically no general way to solve the equation, except numerically. If we deal with a numerical solution, there is another approach based on the division procedure described briefly on page 674. It does not give us the general term of the expansion, but provides an efficient and numerically stable recursive scheme for any number of coefficients we need. Here is the underlying mechanism. Let p(z) = ∑rk=0 pk zk . We assume p0 6= 0 and define ek = −pk /p0 . We want un = [zn ] (p(z))−1 for n in the range n ∈ [0..m]. The idea is to express the reciprocal to p(z) as 1 1 1 1 = = r k p(z) p0 1 − ∑k=1 ek z p0

r

∑ ∑ ek zk

j>0

k=1

!j

.

Hence u0 = 1/p0 , and for n > 1, 1 1 = un = [z ] p(z) p0 n

r k

n

∑ [z ] ∑ ek z j>1

k=1

!j

1 = p0

n

∑ bn, j , where

def

r n

bn, j = [z ]

j=1

k

∑ ek z k=1

!j

.

Finally we create a recursive scheme for the partial coefficients bn, j . We initialize, bn,1 = en , 1 6 n 6 min(m, r), and compute recursively r

bn, j = [zn ] ∑ ek zk k=1

r

∑ el zl l=1

! j−1

r

=

∑ ek bn−k, j−1 ,

1 < j 6 n.

(6.43)

k=1

Exercise 6.63 [3] Produce a similar recurrence needed to calculate 1/p(z) when the assumption p0 6= 0 does not hold, and pd is the lowest order nonzero coefficient in p(z). (It looks much easier when you realize that while this is given here in the context of extraction of coefficients of GFs, it is merely a procedure to calculate long division of power series.) We have seen on page 48 that the Newton formula (2.42) specifies the coefficients for a simple polynomial (1 + z)n = ∑k nk zk . Moreover, its coefficients an,k = nk satisfy the addition formula (2.62): an,k = an−1,k + an−1,k−1 . It is interesting to note that the coefficients an,k in the series representation of the product of two polynomials f (z) = (1 + α z + β z2 )(1 + z)n = ∑k an,k zk , where α and β are some constants, also satisfy the addition property (2.62).

Exercise 6.64 [2] (a) For any positive integers m and n, m < n, show that the double sequence def A(n, k) = [xk ] xm (1 + x)n = [xk−m ] (1 + x)n satisfies the recurrence A(n, k) = A(n − 1, k) + A(n − 1, k − 1),

m < k < n + m,

(6.44)


298

which is similar to the addition formula (2.62) on page 59 for the binomial coefficients. def (b) For any positive integers m and n, m < n, show that the double sequence B(n, k) = [xk ] xm (1 + x)−n = [xk−m ] (1 + x)−n satisfies the recurrence B(n, k) = B(n − 1, k) − B(n, k − 1),

m < k.

Exercise 6.64 allows us to make the following observation. The coefficients, A(n, k) = [xk ] f (x) of the generating function f (x) = P(x) (1 + x)n , obtained from the multiplication of an arbitrary polynomial P(x) by (1 + x)n , satisfies the same recurrence (6.44). Therefore the function f (x) generates a “Pascal triangle” similar to one on page 60, but instead of “1” on the top row, there would be coefficients of the polynomial P(x). For instance, the coefficients A(n, k) = [xk ](1 + 2x + 3x2 )(1 + x)n generate the truncated triangle 0 row 1st row 2nd row 3rd row

1 1 1 1

2 3

5

4 5

3

8 12

3 8

16

3 11

3

Coefficients A(1, k) A(2, k) A(3, k)

On the other hand, the function g(x) = (a + bx)n , where a and b are real numbers, generates (this means that the coefficients [xk ]g(x) fill k-th row of) the triangle 0 row 1st row 2nd row 3rd row

1 a a2 a3

b b2

2ab 3a2 b

3ab2

b3

Coefficients A(1, k) A(2, k) A(3, k)

according to the recurrence def

A(n, k) = [xk ](a + bx)n = a A(n − 1, k) + b A(n − 1, k − 1). def

Exercise 6.65 [2] (a) Show that the double sequence A(n, k) = [xk ] xn (1+x)n = [xk−n ] (1+x)n satisfies the recurrence A(n, k) = A(n − 1, k − 1) + A(n − 1, k − 2),

n < k < 2n.

def

(b) Show that the double sequence B(n, k) = [xk ] xn (1 + x)−n = [xk−n ] (1 + x)−n satisfies the recurrence B(n, k) = B(n − 1, k − 1) − B(n, k − 1), n < k. Note that the generating function for xn (1 + x)n is (1 − x − x2 )−1 , hence ∑n>0 xn (1 + x)n = ∑n>0 Fn+1 xn , where Fn is the Fibonacci number.

6.3. COUNTING BINARY TREES Roots −→

299 . ... ... .... .. .. .. .. ..

(a)

... ... .. ... ... .. ... ... .

(b)

... ... ... ... .... ... .. .. ... .. ... .. .. ... .. . .. ...... .. . ... .. . ... ... .. . ... .. ... ... ... .. . ..

(c)

(d)

Figure 299: Binary trees def

Exercise 6.66 [2+] (a) Show that the double sequence A(n, k) = [xk ] (1 + x + x2 )n satisfies the recurrence A(n, k) = A(n − 1, k) + A(n − 1, k − 1) + A(n − 1, k − 2),

k < 2n.

def

(b) Show that the double sequence B(n, k) = [xk ] (1 + x + x2 )−n satisfies the recurrence B(n, k) = B(n − 1, k) − B(n, k − 1) − B(n, k − 2).

6.3 Counting Binary Trees Binary trees are among the most frequently used data structures. They help to represent a rich variety of relations and therefore the binary trees are embedded in many algorithms. In this section, we introduce them and do some related combinatorial work. Binary trees are defined recursively as follows: • a tree of size zero (also called the empty tree), • a tree of size one (a naked root, illustrated in part (a) of Figure 299), • a tree consisting of a root + left subtree + right subtree (the left and right subtrees have to be, of course, binary trees, and each may be empty—if both are, we have the previous case). A botanist would claim we draw trees upside down, with the root at the top and growing downward. An important detail is that even when there is a single subtree, it is classified as the right- or left-subtree; that is why the trees (b) and (c) in Figure 299 are considered different. Question: How many binary trees are there with n nodes? Let this number be denoted by bn . According to the definition, there exists a tree of size zero, hence b0 = 1. Similarly, there is a single tree with a childless root, hence b1 = 1, which is case (a) in the figure. Beyond that, we observe that a tree with n nodes must have one at the root, some number r, 0 6 r < n, at the left subtree—and the rest, n − r − 1, in the right subtree. The crucial observation is that the subtrees can assume each of the br and bn−r−1 shapes independently, and the two rules of counting discussed in §2.3 tell us then that for n > 1. bn = ∑n−1 r=0 br bn−r−1


300

Since the full-history recurrence is nearly a convolution, we shall use the ordinary generating function to solve it: b(z) = ∑n>0 bn zn . From the recurrence, it follows n−1

b(z) − b0 =

∑

n>1

zn ∑ br bn−r−1 = z ∑ br zr r>0

r=0

∑

zn−r−1 bn−r−1 .

(6.45)

n>r+1

Carrying out the summations we find the quadratic equation b(z) − 1 = zb2 (z),

(6.46)

which we rearrange and solve: 2

zb (z) − b(z) + 1 = 0

=⇒

√ 1 ± 1 − 4z . b(z) = 2z

(6.47)

We need to choose between the two solutions, and use the knowledge that 1 = b0 = b(z) z=0 (in view of the solution, we need to evaluate b(0) as limz→0 b(z)). Obviously this cannot hold √ 1+ 1−4z since it is manifestly infinite at z = 0, but the other solution comes for the function 2z through, which we show by doing some of the work we need in any case: ! ! √ 1/2 1/2 1 1 1 − 1 − 4z j j 1− ∑ −∑ = (−4z) = (−4z) . j j 2z 2z 2z j>0 j>1 √

Clearly limz→0 1− 2z1−4z = 1, and we select this solution. Hence 1/2 1 (−4) j z j−1 , leading to (compare with the extraction in Exercise 6.43) bn = −[zn ] ∑ 2 j>1 j 1 2n 2n 1 4n+1 1 1 1/2 n+1 × × (−4) = = . (6.48) bn = [z ]b(z) = − n 2 n+1 2 2×4 n+1 n n+1 n 2n 1 The values n+1 n are called the Catalan numbers; these numbers arise naturally in many problems of combinatorics [36]. Their conventional symbol is Cn . n

Example 6.67 (Binary search) Suppose that we have a sorted array with n keys: A = hk1 , k2 , . . . , kn i with k1 < k2 < · · · < kn . Given a key k in A, we want to find the index i for which k = ki (successful searching). A wellknown binary search algorithm consists in comparing k against median element k⌊(n+1)/2⌋ . If k = k⌊(n+1)/2⌋ , we are done; otherwise if k < k⌊(n+1)/2⌋ , we proceed by using the left subarray, while if k > k⌊(n+1)/2⌋ , we proceed with the right subarray. We are interested in finding the average number, bn , of comparisons necessary to finish the binary search for any k ∈ A. There is only one element that can be found with a single comparison (the median element in A). There are two elements that can be found with two comparisons; four elements can be found with three comparisons, and so on up to a maximum of 1 + ⌊log2 n⌋ comparisons. Hence, summing the number of comparisons corresponding to each of the n keys, we get bn =

Bn 1 = (1 + 2 + 2 + 3 + 3 + 3 + 3 + · · ·+ (1 + ⌊log2 n⌋)) . n n

6.3. COUNTING BINARY TREES

301

We consider the infinite sequence S = h0, 1, 2, 2, 3, 3, 3, 3, 4, . . .i obtained by summing shifted sequences h0, 1, 1, 1, . . .i, h0, 0, 1, 1, 1, . . .i, h0, 0, 0, 0, 1, 1, . . .i, and so on, the jth sequence starting with 2 j zeroes. This means that the generating function of S is z z2 z4 + + + ··· = ϕS (z) = 1−z 1−z 1−z

j

z2 ∑ 1 − z. j>0

The total number, Bn , of comparisons for the array A is given by the rule (9) in Table 275: j

j

z2 z2 j 1 n = [z ] ∑ 1−z ∑ (1 − z)2 = ∑ [zn−2 ] (1 − z)2 j>0 j>0 j>0 ⌊log2 n⌋ ⌊log2 n⌋ n − 2j + 1 j 1 = = (n + 1)⌊log2 n⌋ + n − 2⌊log2 n⌋+1 + 2. = ∑ [zn−2 ] ∑ 2 j (1 − z) n − 2 j=0 j=0

1 Bn = [z ] 1−z n

Example 6.68 (Binary summation tree) It is known that the accuracy of floating-point arithmetic with rounding depends on the order of computation. However, we consider here only combinatorial aspects of the operation, not the numerical issues, but these are the source of the interest. Let {xk } be n numbers in floating-point representations that we want to add: sn = ∑nk=1 xk . The following algorithm is a natural way to compute sn : 1. while n > 0 do 2. Pick two numbers from the set {xk }, say x and y 3. Compute z ← x + y 4. Replace x and y by z in the set {xk } and n ← n − 1 The two numbers in line 2 could be chosen according to a special rule or at random. The summation procedure is associated with a binary tree: the leaves contain the values xk , k = 1, 2, . . . , n, and the internal nodes contain the intermediate sums sm . The number of the inner nodes depends on the selections in line 2; you may want to build all possible summation trees for n = 5. The total sum, sn , is at the root. By making different choices in the algorithm, we may get different binary summation trees. Let g(n) be the number of different binary summation trees with n leaves. By simple counting, we have the full-history recurrence 1 n−1 n g( j)g(n − j) g(n) = ∑ 2 j=1 j

for n > 2.

The factor 12 comes from the commutative law of summation (x + y = y + x). Checking for the initial values, we have g(1) = 1, g(2) = 1, so for the exponential generating function g(z) ˆ = ∑n>1 g(n) zn /n! we obtain zn n1 n 1 zn g(z) ˆ = z + ∑ g(n) = z + ∑ ∑ j g( j)g(n − j). n! 2 n>2 n! j=1 n>2

302


Changing the order of summation, we derive n 1 zn 1 g(n − j) = z + gˆ2 (z). g(z) ˆ = z + ∑ g( j) ∑ 2 j>1 n! 2 n> j+1 j

√ ˆ we get the number of difTherefore g(z) ˆ = 1 − 1 − 2z. Extracting coefficients from g(z), ferent representations of a real number as a sum of n (n > 1) summands to be n! [zn ]g(z) ˆ = (2n − 3)!!. Example 6.69 (Random walk with a stopping condition) Imagine a particle moving on a horizontal axis, starting at the origin, and moving back and forth on the line by random steps of unit length. We considered a similar process in Example 4.127 on page 178 when a particle moves to the right with probability p, and to the left with probability q = 1 − p. However, now we assume that there is an absorbing position x = k (k > 0) so that the process stops once it reaches this point. What is the probability, denoted by Pk , that the particle eventually would be absorbed?

The process is equivalent to flipping a bias coin until the number of heads exceeds the number of tails by k. The problem of determining the value of Pk for this situation can be solved using stochastic process technique described in Chapter 9. However, we are going to show that Pk can be obtained by solving the second order difference equation Pk+2 =

p 1 Pk+1 − Pk , q q

P0 = 1,

k > 0.

(6.49)

Indeed, the first step of a random walk could be either to the left or to the right. Suppose it starts off to the left, an event with probability q, and so the particle should make k + 1 steps to the right in order to be absorbed. The probability that it reaches the stopping point is Pk+1 . If the first step is to the right (with probability p), then the particle will be one step closer to the terminating point. Therefore Pk = q Pk+1 + p Pk−1 , from which we get Eq. (6.49). Its solution we obtain either from the formula (5.8), page 211, if p 6= q, or from the formula (5.9) if p = q = 1/2 because the corresponding characteristic equation, λ 2 = λ /q − p/q, has two roots:

λ1 = 1

and

λ2 =

p q

since 1 − 4pq = (p − q)2 . Let us consider first the symmetric case p = q = 1/2; then Pk = (P1 − P0 )k + P0 ,

k>0

When p 6= q, the solution of Eq. (6.49) becomes " k # p p q P0 − P1 + (P1 − P0) , Pk = p−q q q

(p = q).

k>0

(6.50)

(p 6= q).

(6.51)

6.3. COUNTING BINARY TREES

303

We can obtain the same results by extracting coefficients from the corresponding generating function: P0 (q − z) + zq P1 def , P0 = 1. P(z) = ∑ Pk zk = (z − 1)(pz − q) k>0 The obtained formulas, (6.50) and (6.51), still contain the unknown value P1 . The values of Pk (k > 1) can be found by summing all probabilities: Pk =

∑ Cn(k) qn pn+k ,

k = 1, 2, . . . ,

n>0

where Cn (k) is the total number of moves with 2n+k steps from the origin that takes n steps to the left and it takes n + k steps to the right. To each such move corresponds a lattice path that has n + k horizontal steps and n vertical steps. Therefore, Cn (k) is the number of northeastern lattice paths (see Example 2.78 on page 51) from the origin to the point (n, n + k) that cannot cross the diagonal y = x + k. We compute P1 by considering the probability that the process is absorbed at 1 in exactly 2n + 1 steps (it must be an odd value), and summing over all such n. The probability of absorption in 2n + 1 steps is qn pn+1 , times the number of such possible paths. Fortunately, Cn (1) can be recognized as the nth Catalan number (from Example 2.106, page 60), which leads to √ n+1 n 1 ± 1 − 4pq 1 − |p − q| 2n p q =p = p b(pq) = , q = 1 − p, P1 = ∑ n + 1 2pq 2q n n>0 where b(z) is given in Eq. (6.47). The correct choice of sign depends on whether p is greater than 1/2 or less. When p > 1/2, we choose minus to obtain P1 = 1 because the other sign (+) would give a probability greater than 1. For p < 1/2, we choose plus in front of the root. Therefore, ( 1, if p > 1/2, (6.52) P1 = p p if p < 1/2. q = 1−p , Now we consider the second probability: P2 =

∑ Cn(2) pk qn+2,

n>0

q = 1 − p,

where Cn (2) is the total number of northeastern lattice paths from (0, 0) to (n, n + 2) that stay below the diagonal y = x + 2. Note that Cn (2) is the same as Cn+1 (1) because an additional step to the right reduces the problem to counting lattice paths from (0, 1). So we get P2 =

n>0

n>0

where C j =

1 2j

1

1

∑ Cn(2) pk qn+2 = p ∑ Cn+1 pn+1 qn+2 = p ∑ C j p j q j+1,

2 j

. This yields ( 1, P2 = P12 = q2

j>1

j

p2

if p > 1/2, =

(1−p)2 p2

,

if p < 1/2;

=⇒

Pk = P1k .

(6.53)


304 Roots −→

. ... ... .. .. .. .. .. ... .. .. .. ..

(a)

... ... ... ... ... .. ... ... ... .. .

(b)

... ... ... ... ... ... ... ... ... ... ...

. ....... ... ... ... ... .. .. .... .. ... . .... ... .. ... ... . ... . ... .

(c)

······

.. ........ .. .. ... .. ... ... ... ... ... .. .. .... .... ... ... .. ... ... ... ... ... ... ... .. .. ... ... . . ... . ... .. .. .. .. .. .. .... . .. .. ... .. . .. ... .... .. .. ... ... .. .. ... .. .. .. ... . .. ... .. .. .. .

(d)

Figure 304: Ternary trees There are many related formulations of this problem and the gambler’s ruin problem is one of them (see Exercise 9.110 on page 541). Now we consider ternary trees. As seen in Figure 304, ternary trees are formed with the same approach that led us to binary ones, and are counted similarly. There is one tree of size 0, one tree of size 1, and three trees of size 2. Let us denote the number of ternary trees with n nodes by b3,n . Using the initial condition and the derivation shown in Eq. (6.45), we find the equation b3 (z) − 1 = z b33 (z), (6.54) for the ternary tree generating function b3 (z), which is the OGF of the sequence {b3,n }. When we counted binary trees we had a quadratic equation, with an immediate solution. Now we have a cubic equation. Since it has no squared term, it is not too hard to write a similar solution for it. However, a careful look at the derivation reveals that we can perform the same derivation for a k-ary tree, in much the same way (this is a generalized binary tree, in which each node can have up to k children in k designated positions). The OGF bk (z), which counts the number of such trees, satisfies the equation bk (z) − 1 = z bkk (z). It is known that for k > 4 there is no explicit solution, even a complicated one. How do we find the coefficients when we cannot solve the equation? The Lagrange Inversion Theorem, presented in §6.7, comes to our aid. In Example 6.188 on page 353, we found precisely what we need, and itsays that the number 3n 3n+1 3n 1 1 = 3n+1 = 2n+1 of ternary trees with n nodes is given by b3,n = 1n n−1 n n , with a similar expression for the number of k-ary trees. We return to the enumeration of further tree varieties in §8.1. Exercise 6.70 [3] Consider the set of all binary trees in which the left subtree is limited to have at most 10 nodes—without any other limitation. How many such trees with n nodes exist? (a) When the restriction is only at the root level. (b) When the restriction holds at every internal node. Use the series command of MAPLE to compare their number with Cn , n ∈ [0..100].

Exercise 6.71 [2] Let rn be the fraction of binary trees of size n in which the root has two nonempty subtrees; compute this fraction (a) using the binary search tree (BST) statistics (see the discussion preceding Exercise 5.90 on page 235), and assuming that the tree was formed by inserting a uniformly random permutation into an empty tree;

6.4. SOLVING RECURRENCES

305

(b) using the Catalan statistics. 1 2n n Exercise 6.72 [2] (L. W. Shapiro) Let C(z) = ∑n>0 n+1 n z be the ordinary generating function for the sequence of Catalan numbers. Prove the identities: 2 C(z) 1 1 n 2n+1 n 2n+2 = (z) = ; (b) ∑ (2n+1)z C . (z) = (a) ∑ (n+1)z C 1 − 4z 2 −C(z) 1 − 4z n>0 n>0

6.4 Solving Recurrences The objective of this section is to demonstrate effectiveness of generating functions in solving recurrences. Generating functions are perfectly suited to treat linear recurrences with constant coefficients; while the restriction of linearity may appear serious, it is not: linear equations cover such a vast terrain that the limitation7 will be rarely noticed. The restriction to constant coefficients is more serious. And while it still leaves a large domain—many of the recurrences we saw and will see have constant coefficients—we often need to venture beyond. The rules in Tables 275 and 278 enable us to resolve many cases when the coefficients are not constant. A standard way to perform the analysis of algorithms is to go through the following sequence of steps. 1. Develop a recurrence for the counting function(s). 2. Translate the recurrence into an equation for a suitable generating function. 3. Solve the equation for the generating function. 4. Extract coefficients—the reversal of evaluating a generating function. In Chapter 5, we considered some methods for implementing step 1. The properties of the OGF and EGF described in §6.1 are used to select the suitable generating function, based on the structure of the recurrence. Thus, if the recurrence contains a regular or binomial convolution, the desirability of an OGF or EGF, is suggested, respectively. In Example 6.90 on page 311, we show that choosing the right type of a generating function can reduce the recurrence to a trivial equation instead of an intractable one. What about step 3, solving the equation? It is sometimes possible; it depends, naturally, on the type of the equation, and it is hard to make any other general statement. When the equation can be solved, the last step listed above can be used. When this is not the case, we typically resort to asymptotic methods, which are the topics of Chapters 10 and 11. Later in this chapter, we go through the entire process for several representative situations.

6.4.1 Ordinary Recurrence Relations The word ‘ordinary’ in the name of the subsection does not mean that the relations are necessarily simple; it is used to contrast with situations considered later, when we deal with 7 Later,

we could solve a variety of nonlinear equations that were kind enough to be cast as convolutions.


306

coupled recurrences. We begin with a general statement, and then show some illustrative examples. The general linear difference equation for the sequence {xi }i>0 is presented in Theorem 5.28. We modify it slightly; the highest indexed xn is placed separately from the sum, and we use the constant coefficients b j , rather than the variable coefficients a j (n)/a0 (n) there: p

xn =

n > p ∈ Z+ .

∑ b j xn− j + gn ,

j=1

(6.55)

The first p elements of the sequence {xi }i>0 , that is, x0 to x p−1 , are assumed known (or can be seen as free parameters, which will ultimately drive the rest of the sequence). We now define the ordinary generating functions x(z) = ∑n>0 xn zn and g(z) = ∑n>0 gn zn , and do the usual algebra—multiplying Eq. (6.55) by zn and summing for n > p, we find on the p−1 xi zi . On the right-hand side, we have left-hand side x(z) − ∑i=0 p

∑ zn

n>p

∑ b j xn− j + gn j=1

!

p

=

p−1

∑ b jz j

j=1

∑ xn− j zn− j + g(z) −

n>p

∑ gi zi .

i=0

p− j−1 xk zk . Hence, The inner sum on n > p can be written as ∑k>p− j xk zk = x(z) − ∑k=0 p−1

p−1

g(z) +

j=1

i=0

x(z) =

p− j−1

∑ (xi − gi ) zi − ∑ b j ∑

i=0

p

1 − ∑ b jz

xi zi+ j .

(6.56)

j

j=1

In particular, the ordinary generating function for the sequence {xn } that satisfies the recurrence xn+2 = p xn+1 + qxn , n > 0, is

∑ xk zk = k>0

x0 + (x1 − px0 )z . 1 − pz − qz2

(6.57)

Exercise 6.73 [1] Use Eq. (6.57) to derive the ordinary generating functions, which correspond to two sequences that are solutions of the following recurrence: yn − 2tyn−1 + yn−2 = 0, n > 2, subject to two distinct initial conditions: y0 = 1, y1 = t and y0 = 0, y1 = 2t. Note: This leads to the OGF for Chebyshev polynomials of the first kind (denoted by Tn (t)) and second kind (denoted by Un (t)), respectively. According to Eq. (6.56), we see that the ordinary generating function x(z) is a ratio of two known polynomials. The rules we have seen in §6.2 enable us to extract coefficients from it. In fact, the process is so simple that there are CAS programs that automate much of it. Let us continue with examples. First, we look at a recurrence we have met a few times before.


307

Example 6.74 (Fibonacci numbers with OGF) These numbers are generated using the following rule: Fn+2 = Fn+1 + Fn ,

n > 0,

and

F0 = 0 , F1 = 1.

The corresponding ordinary generating function is F(z) = ∑n>0 Fn zn , and, using rule (3) of Table 275, the recurrence translates into F(z) − F0 − zF1 F(z) − F0 = + F(z), z2 z and with the initial values of F0 and F1 . Hence we find F(z) − z = zF(z) + z2 F(z) =⇒ F(z) =

z . 1 − z − z2

(6.58)

We want to extract the coefficients, Fn , from this solution. We do this by performing a partial fraction decomposition as follows. The denominator of F(z) vanishes at the roots of the √ √ −1− 5 −1+ 5 2 equation 1 − z − z = 0; that is, at z1 = 2 and z2 = 2 . We can then write: F(z) =

−z

z2 + z − 1

=

B A −z −z = . = z + z z (z1 − z)(z2 − z) z z 1 − 1 − z1 1 − zz2 1 − z2 1 2 z1

The right-hand side is in an expression from which √ we know how to find the coefficients immediately. Note that z1 z2 = −1 and z1 − z2 = 5. In order to obtain the values of the numbers A and B we repeat the last relation, A B −z = z + z , z z 1 − 1 − z1 z2 1 − z1 1 − z2 z1 z2

and continue to multiply both sides by (1 − zz1 ), and then set z = z1 . This yields the equation −z 1 = A, z1 z2 1 − zz21

=⇒

A=

1 1 =√ , z1 − z2 5

√ and similarly we find B = 1/(z2 − z1 ) = −1/ 5. Therefore ! 1 1 1 , F(z) = √ z − z 5 1 − z1 1 − z2 and now we can extract Fn using formula 3 from page 288: 1 n 1 1 1 1 Fn = [zn ]F(z) = √ [zn ] z − √ [z ] z = √ 1 − 1 − 5 5 5 z1 z2

1 1 − . zn1 zn2


308

Since z1 = − z12 , we may simplify it somewhat: " √ !n 1 1+ 5 1 n n − Fn = √ ((−z2 ) − (−z1 ) ) = √ 2 5 5

√ √ !n # (1 + 5)n 1− 5 ∼ √ , 2 5 × 2n

(6.59)

where the last estimate results from the fact that |z2 | > 1 while |z1 | < 1. Note: The number −z2 is commonly denoted by φ , and known as the golden ratio. Its value, √ √ −z2 = φ = 1+2 5 is approximately 1.618034. . . . We also denote φˆ = − φ1 = 1 − φ = 1−2 5 . Eq. (6.59) can be rewritten in terms of the golden ratio as

1 1 Fn = √ φ n − (−φ )−n = √ φ n − φˆ n . 5 5

(6.60)

See Appendix D for further details about Fibonacci numbers and the golden ratio φ . Example 6.75 (Fibonacci numbers with EGF) The same recurrence can be handled with an exponential generating function as well, and the ˆ = ∑ Fn zn . differences are interesting. Let F(z) n! n>0 ˆ = DF(z)+ ˆ Rule (3) in Table 278 translates the recurrence into the differential equation D2 F(z) 2 ˆ ˆ F(z), which can be rearranged to (D − D − 1)F(z) = 0, with D = d/dz. The characteristic equation z2 − z − 1 = 0 is similar to the denominator of F(z), but it is not the same. However, we notice that the two equations have closely related roots! Explicitly: z2 − z − 1 = (z − φ )(z − φˆ ), where φ = −z2 , φˆ = −z1 , the roots we saw before in Example 6.74. So we ˆ = 0. can write the differential equation as (D + z1 )(D + z2 )F(z)

This means that the general solution of the differential equation is any linear combination of solutions to the two simpler, first-order differential equations: ˆ =0 (D + z1 )F(z)

ˆ = 0. (D + z2 )F(z)

and

ˆ which can be written as Fˆ ′ = a, where the left-hand Both equations have the form DFˆ = aF, Fˆ ˆ ˆ = az + C =⇒ side is clearly a logarithmic derivative of F(z). Integration yields ln(F) az Fˆ = k e , and we get two linearly independent solutions: Fˆ1 (z) = k1 e−z1 z

Fˆ2 (z) = k2 e−z2 z ,

with the so-far undetermined constants, k1 and k2 . ˆ The complete solution is then F(z) = k1 e−z1 z + k2 e−z2 z . To solve for ki (i = 1, 2), using two initial values F0 and F1 , we find that ˆ F0 = F(0) = k1 + k2 = 0;

ˆ F1 = DF(0) = −k1 z1 − k2 z2 = 1.

The solution of these two equations is immediate, k1 =

1 1 = −√ ; z2 − z1 5

1 k2 = −k1 = √ 5

ˆ = √1 eφ z − eφˆ z . =⇒ F(z) 5

(6.61)


309

ˆ : Now we can obtain Fn , using Fn /n! = [zn ]F(z) ˆ = √1 ((−z2 )n − (−z1 )n ) = √1 φ n − φˆ n , Fn = n![zn ]F(z) 5 5

(6.62)

to our general lack of surprise—compare with Eq. (6.59).

The following two exercises present properties of the Fibonacci numbers that are convenient in applications. Exercise 6.76 [2] Having the Fibonacci numbers in terms of φ , flip Eq. (6.62) around to express powers of φ and φˆ in terms of the {Fk }:

φ n = Fn−1 + φ Fn ,

φˆ n = Fn−1 + φˆ Fn .

(6.63)

Exercise 6.77 [3] (a) Use the representation of Exercise 6.76 (or directly, the OGF) to show that the convolution of the Fibonacci numbers can be expressed using terms of the sequence: n

∑ Fk Fn−k = k=0

2n n−1 Fn + Fn−1 . 5 5

(6.64)

(b) Show, similarly, that the binomial convolution of the Fibonacci numbers can be expressed in closed form by elements of the sequence: n 2 n 2n (6.65) ∑ k Fk Fn−k = 5 (Fn−1 + Fn+1) − 5 . k=0 Exercise 6.78 [2] Find the ordinary generating function for each of the following sequence (b) {Fn2 }. of numbers involving Fn , the Fibonacci numbers: (a) {∑nk=0 Fk };

Exercise 6.79 [2] Let wn be the number of ways of rolling a die n times such that no consecutive sixes occur. Derive and solve a recurrence for the sequence {wn } using its OGF.

Exercise 6.80 [2] (a) Extracting coefficients from the ordinary generating function obtained in Exercise 6.73, find explicit values for the Chebyshev polynomials. (b) For what values of n and t does the Chebyshev polynomial Un (t) equal a Fibonacci number? Exercise 6.81 [3] Derive the ordinary generating function for the sequence {xn (t)} of the Legendre polynomials that are determined by the recurrence (n + 1)xn+1 (t) = (2n + 1)t xn (t) − n xn−1 (t),

n > 1,

x0 = 1, x1 (t) = t.

Exercise 6.82 [2] Suppose that the sequence { fn }n>0 satisfies the full-history recurrence: n

fn =

∑ j fn− j ,

n > 1,

f0 = 1. Find the OGF of this sequence and its closed form.

j=1

Note: This sequence is related to the Fibonacci numbers.


310

Exercise 6.83 [3] Given that the sequence { fn }n>0 satisfies the full-history recurrence fn = n ( j − 1) fn− j for n > 1, find the corresponding ordinary generating function. ∑ j=2 Example 6.84 Consider the following recurrence an+2 − an = 1 for n > 0

with a0 = 0,

a1 = 2.

Let a(z) be the corresponding ordinary generating function. Then

Hence

1 1 (a(z) − a0 − a1 z) − a(z) = 2 z 1−z

=⇒

a(z)(1 − z2 ) =

z2 + 2z. 1−z

2z 1/2 1/4 3/4 z2 + + , = − 2 2 2 (1 − z )(1 − z) 1 − z 1 − z (1 − z) 1+z where the right-hand side shows the partial fraction decomposition of a(z). We can now extract coefficients to obtain the desired sequence: 1 2n + 3(1 − (−1)n ) 3 1 1 −2 . (−1)n − (−1)n = × [2(n + 1) + 1 − 3(−1)n ] = an = + 4 2 n 4 4 4 a(z) =

The only difference here from the Fibonacci example is that this recurrence is inhomogeneous. This changes, naturally, the final generating function, but does not impact the solution procedure. You may want to redo Exercise 5.36 using our new technique! Exercise 6.85 [3] Using generating functions, solve the following difference equations: (a) an+2 − 3an+1 + 2an = 7n; n > 0; a0 = −1; a1 = 3. (b) xn+2 − 5xn+1 + 6xn + 2n+3 = 0, x0 = 2, x1 = 7. (c) xn+2 − 4xn+1 + 4xn = 0, x0 = 3, x1 = 10. (d) xn+2 − 6xn+1 + 9xn = 0, x0 = 2, x1 = 3. (e) xn+3 − 8xn+2 + 21xn+1 − 18xn = 0, x0 = 0, x1 = 2, x2 = 13. (f) xn+3 − 7xn+2 + 16xn+1 − 12xn = 0, x0 = 0, x1 = 5, x2 = 19. (g) xn+3 − 3xn+2 − 4xn+1 + 12xn = 0, x0 = 3, x1 = −7, x2 = 7. (h) xn+4 − 13xn+2 + 36xn = 0, x0 = 7, x1 = −6, x2 = 38, x3 = −84.

The previous examples are typical of handling linear recurrences with constant coefficients. Any deviation from this pattern introduces some special wrinkles. Exercise 6.86 [3] The Pell numbers Pn and their associated numbers Qn satisfy Pn+2 = 2Pn+1 + Pn ,

P0 = 0, P1 = 1

and

Qn+2 = 2Qn+1 + Qn ,

Q0 = 1, Q1 = 1.

Find the numbers Pn and Qn using the generating function method. Exercise 6.87 [3−] (E. Deutsch) Show that the Pell number can be represented as (i + j + k)! n−1 n−i−1 n − k − 1 =∑ ∑ Pn = ∑ i! j! k! i, j k=(n−i− j−1)/2∈N i=0 j=0 i+ j+2k=n−1

The summation extends over all nonnegative integers i, j, k satisfying i + j + 2k = n − 1.


311

Exercise 6.88 [3] Given that the sequence of numbers {an } satisfies the difference equation an+1 = an + (−1)n an−1 , (a) Prove that an+2 = an + an−2 .

a0 = a1 = 1.

n > 1,

(b) Find a simple formula for an .

So far we applied ordinary generating functions (except Example 6.75) to solve given recurrences. However, rules (4) and (6) in Table 278 suggest that difference equations with polynomial coefficients could be solved sometimes with less effort using exponential generating functions as the following examples show. Example 6.89 Consider the second order difference equation an = (n − 1)an−2 ,

n = 2, 3, . . . .

Multiplying both sides by zn−1 /(n − 1)! and summing results with respect to n from 2 to infinity, we get zn zn−1 d zn−1 a a = = . a n n−2 n ∑ (n − 1)! dz ∑ n! ∑ (n − 2)! n>2 n>2 n>2 If we denote by a(z) ˆ = ∑n>0 an zn /n!, the exponential generating function for the required sequence {an }, the latter equation becomes d (a(z) ˆ − a0 − a1 z) = z a(z) ˆ dz

or

d a(z) ˆ − z a(z) ˆ = a1 . dz 2

Multiplying both sides of the differential equation by e−z /2 , we obtain Z i 2 2 2 2 2 d h ˆ = a0 ez /2 + ez /2 a1 e−z /2 dz. a(z) ˆ e−z /2 = a1 e−z /2 =⇒ a(z) dz

Hence the exponential generating function is the sum of two functions, one of them aˆ0 (z) = R 2 2 2 a0 ez /2 , is a series in even powers of z, and another, aˆ1 (z) = a1 ez /2 e−z /2 dz (a constant of integration is assumed to be included into aˆ0 (z)), is a series in odd powers. Extracting coef2 ficients from aˆ0 (z), we get n! [zn ]aˆ0 (z) = a0 qn , where the value qn = n! [zn ]ez /2 was obtained in Example 6.55 on page 292. For odd indices, we get the binomial convolution n! [zn ]aˆ1 (z) = a1 {qn } ⋆ {qñ },

where qñ = (n − 1)! [zn−1 ] e−z

2

/2

.

Example 6.90 We start with a recurrence from Example 5.4 on page 201, but this time we use a different procedure: an+1 = (n + 1)an , n > 0, a0 = 1. We begin with the exponential def generating function, call it a(z) ˆ = ∑n>0 an zn /n!, and from the second and fourth lines of Table 278 we find the differential equation aˆ′ (z) = (zD + 1)a(z) ˆ = a(z) ˆ + zaˆ′ (z). We can regroup this equation to read aˆ′ (z)/a(z) ˆ = 1/(1 − z). Recognizing the logarithmic derivatives on both sides, we integrate, and find ln a(z) ˆ = c + ln(1 − z)−1 , with an arbitrary

312


constant c. Exponentiation leads to a(z) ˆ = C/(1−z), where C = ec . From the initial condition a(0) ˆ = 1, we find C = 1. 1 = n!, and we have the expected answer. Finally, an = n![zn ]a(z) ˆ = n![zn ] 1−z

If we want to use OGFs the situation is not as simple. Setting up the equation for the OGF poses no difficulty. Using Table 275, we obtain z2 a′ (z) + (z − 1)a(z) + 1 = 0. This equation is a standard linear, first-order differential equation. The canonical form for such an equation, and its solution, are R R R (6.66) f ′ (z) + h(z) f (z) = g(z) =⇒ f (z) = e− h(z)dz C + g(z)e h(z)dz dz .

Here we identify a(z) with f (z), (z − 1)/z2 with h(z), and −1/z2 with g(z). However, this is a solution of the equation on condition that there is an open neighborhood of the origin where f (z) is regular. But we happen to know the function we have here: it has the power series expansion ∑k>0 k!zk . This series converges for no z 6= 0. Therefore, a(z) can be nowhere regular.

Note that we did not really need to know the solution a-priori to realize that this procedure is not working for us: when trying to use the solution part of Eq. (6.66) we find that we need to integrate e1/z , a function with an essential singularity at the origin (see Example 12.56). Here we see a situation where one GF is more preferable to another. Exercise 6.91 [2] Solve the recurrence yn = n yn−1 + 2n − 1 EGF.

n > 2,

y1 = 0, using the

Exercise 6.92 [1+] Determine the differential equation for the exponential generating funcn+1 + tion generated by the sequence that satisfies the following difference equation: xn+2 + xn+1 xn = n, x0 = 1, x1 = 2. Exercise 6.93 [3] Solve the difference equation Dn = (n− 1)Dn−1 + (n− 1)Dn−2 , n = 2, 3, . . ., under two kinds of initial conditions: (a) D0 = 1, D1 = 0; (b) D0 = 1, D1 = 1. ˆ Hint: Use the exponential generating function, D(z) = ∑ Dn zn /n!, and the substitution y(z) = ˆ D(z)(1 − z) to solve the differential equation obtained.

Exercise 6.94 [2] The following problems ask you to find the EGF of the sequences {ai }i>0 : n (a) an+1 = (n + 1)an − an−1 , n > 0, a−1 = 0, a0 = 1. 2 n (b) 2an+1 = ∑ ai an−i , n > 0, a0 = 1. i i n (c) 2an+1 = ∑ ai an−i , n > 1, a0 = 1, a1 = 1. i i Exercise 6.95 [2] Find the solution and the exponential generating function for the third order difference equation xn+3 − 5xn+2 + 8xn+1 − 4xn = 0 subject to the initial conditions x0 = x1 = 0, and x2 = −1. Exercise 6.96 [3] Let 0, x1 , x2 , . . . be a sequence of numbers (x0 = 0) satisfying the difference zn ˆ be its exponential generating function. equation xn+1 = pxn + qxn−1 and X(z) = ∑n>0 xn n!


313

Show that ˆ ˆ X(z) = −e−pz X(−z). Exercise 6.97 Solve for the general solution of the following recurrences using exponential generating functions (EGFs). All the recurrences hold for n > 0: (a) [2] an+2 − nan = 0; (c) [4] an+2 − 2nan+1 + n(n − 1)an = 3n ; (e) [4] an+2 − 2nan+1 + n2 an = 0;

(b) [4] an+2 − n(n − 1)an = 1; (d) [4] an+1 − 2nan = n2 (−2)n ; (f) [4] an+2 + nan+1 + nan = 3n+1 .

Some of the recurrences do not have closed explicit solutions. In this case, consider the exponential generating function, a(z), ˆ in the final answer. We show in §11.2.2 how to get useful information (asymptotic estimates) about an directly from the EGF. MAPLE note: You can use the series command of MAPLE to find an arbitrary prefix of a power series development of the EGF. For the special case of rational GFs, the package genfunc provides further tools, including the function rgf expand that computes the general term of the sequence corresponding to the generating function. Exercise 6.98 [3] Let an be the number of symmetric n × n matrices with exactly one 1 and n − 1 zeroes in each row and each column. Prove that an+1 = an + n an−1 , for n > 1, with initial values a0 = 1, a1 = 1; show that the corresponding exponential generating function is 2 ez+z /2 . The following exercises demonstrate the power of generating functions in solving full-history recurrences. Exercise 6.99 [3] (Log piles, continuation of Example 5.17 on page 206) In the cited example, a recurrence is obtained for f (k), the total number of piles with k logs in the bottom layer; f (k) = 1 + ∑k−1 j=1 (k − j) f ( j). Solve the problem by using the fact that the recurrence is a convolution. Show how the solution relates to the Fibonacci sequence. Exercise 6.100 [3] Unusual recurrences sometimes need unusual generating functions to be tamed and solved. Here is an example for you to see this through. In the sequence {an } only the terms with odd indices play a role, and they satisfy 2n−2 j+1 2 j 1/2 n− j j a2n−2 j+1 , n > 0. a2n+1 = 1 + ∑ 2n+1 j>1 2 j − 1 n Note that this also fixes a1 to be 1. Define the generating function A(z) = ∑n>0 Use it to derive the result a2n+1 = 2(n + 1)/(n + 2).

2n+1 n

a2n+1 zn .

Exercise 6.101 [3] (Adapted from [131]) Suppose we need to multiply n quantities (numbers or matrices) without changing their order. Determine pn , the number of ways this can be done. For example, for n = 2 there is only one way to do the one multiplication, hence p2 = 1. For n = 3, we see the possibilities (AB)C and A(BC), hence p3 = 2. Find a recurrence for pn , then derive an equation for their ordinary generating function (OGF), and solve it. Hint: You should recognize the OGF (and the numbers that can be extracted from it). You


314

may want to establish a 1-to-1 correspondence between parenthesizing of products and Dyck paths (see Example 4.127, page 178) by assigning to every open parenthesis “(” a move ր, and to every closed parenthesis “)” a move ց. Exercise 6.102 [2+] Consider a combination lock with n buttons numbered 1 through n. A valid combination consists of a sequence of button-pushes, in which each button is pushed exactly once, but some of them are allowed to be pushed simultaneously. Let an be the number of combinations for a lock with n buttons. If n = 0 then the only valid combination is the empty sequence, so a0 = 1. For n > 0, we have the following full-history recurrence: n n an = ∑ an−k , for n = 1, 2, . . . , a0 = 1. k=1 k Dividing the formula above by n! yields the following recurrence relation for bn = an /n!: n

bn =

bn−k , k=1 k!

∑

for k = 1, 2, . . . ,

b0 = 1.

Solve both recurrences for {an } and {bn }.

Exercise 6.103 [3−] Solve the recurrence cn = cn−1 + 22 cn−2 + · · · + n2 c0 , for n > 0.

Exercise 6.104 [4] Find explicit expressions for the sequences that satisfy the following fullhistory recurrences. n−m

(a) an =

∑ ai an−i ,

n > m + 1, a0 , . . . , am are given.

i=m n−1

(b) an =

∑ 3i an−i ,

i=1

n−1

a0 = 1, a1 = 6.

(c) an =

∑ 5i ai ,

a0 = a1 = 1,

n > 2.

i=1

Note: Assume that the sum is zero if the lower bound of summation is greater that the upper bound. Exercise 6.105 [3] (a) The Stirling numbers of the second kind are denoted by nk , and traditionally defined as the number of ways in which a set of size n can be partitioned into k disjoint and exhaustive subsets. They are therefore also called subset numbers. Use their sum representation (C.20) to calculate their upper-argument exponential generating function: 1 n zn = (ez − 1)k , k > 0. sˆk (z) = ∑ k! n>k k n! (b) Show that the definitionofthe Bell numbers8 as sum of the Stirling numbers of the n zn second kind, that is, Bn = ∑ , provides them with the EGF: ∑ Bn = exp (ez − 1). n! k n>0 k

Exercise 6.106 [3] (Bell numbers) Use the EGF of the Bell numbers Bn you derived in Exercise 6.105(b) to answer the questions:

8 After Eric Temple Bell (1883, Peterhead, Scotland – 1960, Watsonville, California), a mathematician and science fiction author who lived in the USA for most of his life (since 1884).


315

(a) obtain from this EGF the representation Bn =

rn 1 ; ∑ e r>0 r!

(b) derive the binomial full-history recurrence Bn+1 = ∑ i

n Bi . i

6.4.2 Vector Recurrence Relations In many applications (for instance, in the predator-prey model discussed in Example 5.2) we come across coupled recurrences. A common term for them is vector recurrences. For example, suppose we need to find the Laplace transforms of the functions sinn t and sinn t cost, for which we introduce the notations: an =

Z ∞ 0

e−λ t sinn t dt,

bn =

Z ∞ 0

e−λ t sinn t cos t dt

(n = 0, 1, 2, . . .).

Integration by parts leads to the following second order vector difference equation with variable coefficients: an =

n bn−1 , λ

λ bn = nan−1 − (n + 1)an+1 ,

a0 = 1/λ ,

a1 =

1 , λ2 +1

b0 =

λ . λ2 +1

More generally, two sequences of numbers {an } and {bn } may be related via two equations an = ϕ (an , an−1 , . . . , a0 , b0 , . . . , bn ) and

bn = ψ (an , . . . , a0 , b0 , . . . , bn ).

Again, this is a vector or a coupled recurrence. This definition can be naturally extended for three and more sequences. In this subsection, we consider only linear vector difference equations of the first order. For two sequences {an } and {bn }, such coupled difference equation has the following form: an = α (n)an−1 + β (n)bn−1 + fn ,

bn = γ (n)an−1 + δ (n)bn−1 + gn ,

n > 1.

(6.67)

where α (n), β (n), γ (n), δ (n), fn and gn are given sequences, and the initial values a0 and b0 are assumed available. Eq. (6.67) can be rewritten in the vector form: # " # " # " " # α (n) β (n) fn an−1 an . (6.68) , n > 1, where A = + =A γ (n) δ (n) gn bn−1 bn If the coefficients of the matrix A in Eq. (6.68) do not depend on n, we have a constant coefficient vector difference equation of the first order. To solve the vector difference equation (6.67), we use the same technique—generating functions: multiply both sides of Eq. (6.67) by zn and sum over n > 1. This yields

∑ an zn = ∑ α (n)an−1 zn + ∑ β (n)bn−1 zn + ∑ fnzn ;

n>1

n>1

n>1

n>1


316

∑ bn zn = ∑ γ (n)an−1 zn + ∑ δ (n)bn−1 zn + ∑ gn zn , n>1

n>1

n>1

n>1

which we can rewrite as

∑ α (n)an−1 zn + ∑ β (n)bn−1 zn + f (z) − f0,

a(z) − a0 =

n>1

n>1

b(z) − b0 =

∑ γ (n)an−1 zn + ∑ δ (n)bn−1 zn + g(z) − g0, n>1

n>1

where a(z) =

∑ an zn,

b(z) =

n>0

∑ bn zn ,

n>0

f (z) =

∑ fn zn,

n>0

g(z) =

∑ gn zn

n>0

are ordinary generating functions for the sequences {an }, {bn }, { fn }, and {gn }, respectively. Sums like ∑n>1 α (n)an−1 zn can be expressed via a(z) in many particular cases as Table 275 suggests. For definiteness, we here consider the simplest case, when the coefficients in the matrix A are all constants, then we have ( a(z) − a0 = α za(z) + β zb(z) + f (z) − f0 , b(z) − b0 = γ za(z) + δ zb(z) + g(z) − g0 . This algebraic system of equations can be rewritten in the vector form: # " # " # " # " α β a(z) f (z) a0 − f0 . (I − zA) = + , A= γ δ b0 − g0 b(z) g(z) Multiplication by (I − zA)−1 , the inverse of the matrix I − zA, gives the solution: # " # " # " a − f f (z) a(z) 0 0 . + (I − zA)−1 = (I − zA)−1 b0 − g0 g(z) b(z)

(6.69)

Of course, we need to assume that the matrix I − zA is invertible (that is, 1/z is not an eigenvalue of the matrix A). Now we can find a solution of the vector difference equation (6.68) (with constant matrix A) by extracting coefficients from the vector function in the right-hand side of Eq. (6.69). The following example demonstrates this approach. Example 6.107 Suppose that we have three sequences {an }, {bn } and {cn }, n > 0, defined by the simultaneous recurrences an = an−1 + 5bn−1 − 2cn−1 ,

bn = 2n − cn−1 ,

cn = 5n − bn−1 ,

a0 = 0, b0 = c0 = 1.

Note: The solution, using the three obvious OGFs is entirely routine, and the point is to note that it is desirable to use all three OGFs, and solve for them, rather than eliminate one or two of the sequences by substituting in the original recurrences.


317

Let xn = han , bn , cn iT be the vector-column of the unknown sequences. We denote by a(z), b(z), and c(z) the OGFs of the sequences {an }, {bn } and {cn }, respectively. Then a(z) = za(z) + 5zb(z) − 2zc(z), 1 − 1 − zc(z), b(z) − 1 = 1 − 2z 1 − 1 − zb(z). c(z) − 1 = 1 − 5z

(6.70)

While solution of the linear system (6.70) does not look promising, MAPLE can help with the command: solve((1-z)*a=5*b-2*c, b+z*c=1/(1-2*z), c+z*b=1/(1-5*z), {a,b,c});

We obtain the same result by using MAPLE directly in Eq. (6.69):      1 − 8z 0 a(z) 10       (1 − z)(1 − 6z + 2z2 ) , b(z) = (I − zA)−1 (1 − 2z)−1  = ∆(z) (1 − z)(1 − 3z + 5z2 ) (1 − 5z)−1 c(z) 

where ∆(z) = (1 + z)(1 − z)2 (1 − 2z)(1 − 5z) and the resolvent is   z(5z+2) z(5+2z) 1 − 2 2 (1+z)(1−z)   1−z (1+z)(1−z) z 1 . (I − zA)−1 =  − 0 (1−z)(1+z) (1−z)(1+z)   1 z 0 − (1−z)(1+z) (1−z)(1+z)

We used the following MAPLE procedure: > > > >

with(LinearAlgebra): M:= ; A:= matrix(3,3,[1,5,-2,0,0,-1,0,-1,0]); K:= M - z*A; R:= MatrixInverse(K);

Now partial fraction decomposition (see §12.5) completes the task. A computer algebra system, like MAPLE, is very helpful. For example, MAPLE’s command convert(a(z), parfrac, z) provides the following result: 1 219 1 3 1 16 125 1 21 − + + − , 2 8 (1 − z) 32 1 − z 8 1 + z 1 − 2z 32 1 − 5z 1 1/4 4/3 5 3 1 + + − , 8 z − 1 z + 1 1 − 2z 24 1 − 5z 1/4 2/3 25 1 3 1 + − + . 8 1 − z 1 + z 1 − 2z 24 1 − 5z

a(z) = − b(z) = c(z) =


318

Extracting coefficients, we obtain the solution: 1 219 3 21 −2 + (−1)n + 2n+4 − 5n+3 , − an = − 8 32 8 32 n 4 5 3 1 bn = − + (−1)n + 2n − 5n , 8 4 3 24 1 1 3 1 cn = + (−1)n − 2n+1 + 5n+2 . 8 4 3 24 Exercise 6.108 [3+] Solve the following simultaneous recurrences for two sequences. (a) an = 2n + bn−1 , bn = 3n + 4an−1 , n > 1, with the initial values a0 = b0 = 1. (b) an+1 = bn − an , bn+1 = bn + an , n > 0, a0 = b0 = 1. (c) qn = 2rn−1 + 1, rn = qn + qn−1 + 1, n > 0, q0 = r0 = 0. (d) sn+1 = sn + dn , dn+1 = 2sn + dn , (n > 1), d1 = s1 = 1.

Another source of coupled recurrences is the transformation of higher order difference equations, as the following example shows. Example 6.109 Let us consider the non-homogeneous Fibonacci recurrence: an = an−1 + an−2 + 3n−1 ,

n > 2,

a0 = α , a1 = β .

We set un = an+1 and vn = an , n > 2; this allows us to rewrite the given second order difference equation as the vector equation of first order " # # " " # " # " # " # β 1 1 3n−1 un−1 u0 un , A= . (6.71) + = =A , n > 1, α 1 0 vn−1 v0 0 vn Let u(z) and v(z) be the OGFs of the sequences {un } and {vn }, where of course, v(z) = zu(z). We can determine these functions from the following vector equation: " # # " " # 1 u −1 β − 1 −1 1−3z + (I − zA) , = (I − zA) α 0 v where the resolvent is −1

(I − zA)

" # #−1 1 z 1 − z −z 1 = = 1 − z − z2 z 1 − z −z 1

"

Therefore u(z) =

(β − 1)z + α z 3z + 4 1 9 1 (β − 1)z + α z 1 + = − − . + 1 − z − z2 1 − 3z 1 − z − z2 5 1 − z − z2 1 − 3z 1 − z − z2

Extracting coefficients, we obtain 4 3 3 an = un−1 = − Fn−1 − Fn + 3n + (β − 1)Fn + α Fn−1 . 5 5 5


319

Exercise 6.110 [3] This exercise deals with a modification of a classical problem, considered by L. Fibonacci in 1202, that starts out with a pair of rabbits, male and a female. A female will begin bearing after two months and will bear monthly thereafter. The first litter a female bears is twin males, thereafter she alternately bears a pair of females and a pair of males. Assume that each offspring pair also breeds in the same manner. Find the total rabbit population at the end of nth month by deriving and solving with the aid of generating functions a recurrence relation for the number of males and females. Exercise 6.111 [3] Let r and s be the distinct roots of the quadratic equation x2 − px − q = 0 (p2 6= −4q). Let un = (sn − rn )/(s − r) and vn = sn + rn . Derive the vector recurrence for un , vn and obtain the generating functions for these sequences {un } and {vn }. Hint: See Exercise 5.129 on page 258. Exercise 6.112 [3] Solve the vector equations. (a) sn = 2sn−1 + sn−2 + 4tn−1 , (b) pn = qn − pn−1 ,

tn = sn−1 + tn−1 ,

qn = 4pn−1 − qn−1 ,

(c) an = an−1 + 2bn−1 ,

n > 2,

n > 1,

bn = 2an−1 + 3bn−1 ,

n > 1,

t0 = 0, t1 = 1, s0 = 1, s1 = 2. p0 = 1, q0 = 1. a0 = 0, b0 = 1.

Show also that an + bn = F3n+2 , the Fibonacci number.

6.4.3 Partial Difference Equations We here briefly consider methods involving GFs for solving recurrences with two indices (also called partial difference equations or bivariate recurrences). Such recurrences require the usage of multivariate generating functions (see §6.1.4). Let us reconsider the familiar recurrence (5.65): an,k = an−1,k + an−1,k−1 , for n > 1 and 1 6 k < n, with the boundary values an,0 = an,n = 1. Define an (z) = ∑ an,k zk . Using the left k>0

shift rule (rule (4) in Table 275), we obtain an (z) − 1 = an−1 (z) + zan−1 (z) − 1

=⇒

an (z) = (1 + z)an−1 (z), n > 1,

a0 (z) = 1.

The solution is clearly an (z) = (1 + z)n . Looking back at the recurrence we started with, you should recognize the addition formula for the binomial coefficients, shown on page 59. We can therefore compute the bivariate generating function for the binomial coefficients: n n k 1 . a(u, z) = ∑ ∑ u z = ∑ (1 + z)n un = 1 − (1 + z)u n>0 k=0 k n>0 n

Now we look at a double sequence {an,k } that satisfies a slight generalization of the above linear recurrence, an,k = α an,k−1 + β an−1,k + γ an−1,k−1

for n, k > 1,

(6.72)


320

with the initial conditions an,0 = β n , a0,n = α n for n > 0. Let a(x, y) =

∑ ∑ an,k xn yk

n>0 k>0

be the bivariate generating function for the sequence {an,k }. Using the recurrence (6.72), we “generate” successively over n and over k, and find a(x, y) to be a(x, y) =

2 . 1 − α x − β y − γ xy

(6.73)

Exercise 6.113 [3] Using bivariate generating functions, find explicit expressions for the sequences that satisfy the following recurrences. In all cases an,0 = 1, and a0,k = δ0,k . (a) an,k = an,k−1 + an−1,k , n, k > 1. (b) an,k = 2an−1,k − 3an,k−1 , n, k > 1. (c) an,k = an,k−1 − 2an−1,k−1 , n, k > 1. Exercise 6.114 [3+] Let {an,k } be a sequence (which we met in Exercise 5.139 on page 260) that satisfies the following recurrence: an,k = an−1,k + an−1,k−1 + an−1,k−2 ,

k = 2, 3, . . . 2n,

n > 1,

subject to the initial and boundary conditions: an,0 = 1, an,1 = n, and an,k = 0 for k > 2n. Find the univariate generating function, Tn (z) = ∑k>0 an,k zk , and the bivariate generating function, A(u, z) = ∑n>0 Tn (z) un = ∑n,k>0 an,k un zk . By differentiating Tn (z) (n > 2) twice, derive another recurrence k(k − 1)an,k = 2n an−1,k−2 + n(n − 1)an−2,k−2 + 4n(n − 1)an−2,k−3 + 4n(n − 1)an−2,k−4 , which is valid for k = 4, . . . , 2n. Note: the coefficient tn = [zn ]Tn (z) is called the central trinomial number, which we discuss in Example 6.185 on page 352. Example 6.115 (Convolution of Catalan numbers) 1 2n , n ∈ N, be the sequence of Catalan numbers. For all k > 1, we define Let bn = Cn = n+1 n (k)

bn =

n

(k−1)

∑ b j bn− j

,

n > 0,

j=0 (0)

(k)

with bn = δ0,n , n > 0. The sequence {bn } is the k-fold convolution of the sequence {bn }. Since the OGF of the Catalan numbers (computed on page 300) is √ 1 − 1 − 4t n , B1 (t) = ∑ bnt = 2t n>0 √ (k) rule 5 in Table 275 allows us to deduce that Bk (t) = ∑n>0 bn t n = (2t)−k (1− 1 − 4t)k , k > def (k) 1. Hence the bivariate generating function, B(x,t) = ∑k>1 ∑n>0 bn xk t n = ∑k>1 Bk (t)xk , can


321

be directly computed: k x k √ 1 √ −1 1 − 1 − 4t = 1−4t x−x 1− k>1 k>1 2t 2t √ √ 1 − 2x − 1 − 4t x − x 1 − 4t √ = 2tx . = x2 − x + 4t 2 2t − x + x 1 − 4t

B(x,t) =

∑ Bk (t) xk = ∑

The univariate GFs Bk (t) show a curious recurrence satisfied by the Catalan numbers convolutions. We begin with the expression for Bk+1 (t): √ √ √ (1 − 1 − 4t)k+1 (1 − 1 − 4t)k−1 (1 − 1 − 4t)2 = Bk+1 (t) = (2t)k+1 (2t)k+1 √ √ √ √ (1 − 1 − 4t)k−1 [2(1 − 1 − 4t) − 4t] (1 − 1 − 4t)k (1 − 1 − 4t)k−1 = − = (2t)k+1 2k t k+1 t(2t)k−1 = t −1 Bk (t) − t −1 Bk−1 (t). Extracting coefficients from both sides, we obtain (k+1)

bn

(k)

(k−1)

= [t n ]Bk+1 (t) = [t n ]t −1 Bk (t) − [t n ]t −1 Bk−1 (t) = bn+1 − bn+1 .

(6.74)

(k)

Do the convolutions bn have an explicit form? The answer is that they do, but the “obvious” road to get it, by extracting coefficients from the GF, turns out not to be simple. Consider the coefficient √ k √ 1 (k) n n (1 − 1 − 4t) = k [t n+k ](1 − 1 − 4t)k bn = [t ]Bk (t) = [t ] k (2t) 2 r/2 k k 1 n+k 1 n+k r r/2 r (−4) j t j (−1) (1 − 4t) = k [t ] ∑ (−1) ∑ = k [t ] ∑ 2 2 j r r r r j k r/2 = 2k 4n (−1)n+k ∑ (−1)r . r n+k r As r takes integer values, increasing from zero, let us separate the even ones, r = 2 j, from the odd ones, r = 2 j + 1, and let j > 0: k j k j + 1/2 (k) k n n+k bn = 2 4 (−1) ∑ − . 2j n+k 2j+1 n+k j The first sum, on the even values, makes no contribution, for any value of j; the second sum is not obvious, due to the ‘ j + 1/2’ in the second binomial coefficient, which means it is not necessary that j be “large enough.” We can use a standard transformation for the term with the 1/2, this leads to a marginally better formula: n+k k 1 2n + 2k j+1 (k) (−1) j 2n+2k . (6.75) bn = k 2 n+k ∑ 2 j + 1 j 2 j+2


322

Is this the end of the road? No! Performing the first few convolutions directly leads us to the guess: 2n + k − 1 k (k) ? . bn = n+k n

This formula can be proved by induction (over k), but a more interesting way is to show that these values satisfy the same recurrence, (6.74), as well as the initial values. This amounts to a simple substitution. By an unintended turn of events, these numbers are closely related to another bivariate sequence that we examine briefly in Example 2.106, the ballot numbers. (k) According to the notation there, we can write bn = ψn+k−1,n .

Incidentally, we have again shown indirectly an intricate summation formula by equating this value with the right-hand side of Eq. (6.75). Exercise 6.116 [3] (Convolution of Fibonacci numbers) Let {Fn }, n = 0, 1, . . ., be the sequence of Fibonacci numbers with F0 = 0, F1 = 1. The sequence (2)

Fn

n

=

∑ Fj Fn− j ,

n = 0, 1, . . . ,

j=0

is the convolution of the sequence {Fn } with itself. In general, the sequence n

(k)

Fn =

(k−1)

∑ Fj Fn− j

,

n = 0, 1, . . . ,

k = 2, 3, . . . ,

j=0 (1)

with Fn = Fn , n = 0, 1, . . ., is called k-fold convolution of the sequence Fn , n = 0, 1, . . .. (a) Derive the generating function fk (t) =

(k) n

t = (1 − t − t 2 )−k t k

∑ Fn

n>0

and deduce the expressions (k) Fn

=

⌊n/2⌋

∑

r=0

n−k r+k−1 n−r+k−1 r+k−1 r =∑ . r n − 2r r n−k−r r=0

(b) Show that f (t, u) =

(k) n k

∑ ∑ Fn

k>1 n>0

t u =

ut 1 − t − t2 −1 = . 2 1 − t − t − tu 1 − t − t 2 − tu

Exercise 6.117 [3+] Find a compact expression for the series ∞

(a)

∑

m,n=0

F2m−2n xm yn ,

∞

(b)

∑

F2m−2n+1 xm yn ,

m,n=0

where Fk is the kth Fibonacci number. Hint: See Exercises 5.19 and 5.37.


323

Exercise 6.118 [3+] (Log piles, continuation of Exercises 5.121, and 6.99, pages 252 and 313) In Exercise 5.121, you were asked to derive a recurrence for f (k, n), the number of n-log piles with k logs in the bottom layer. Find the GF of this counting function (OGF in both arguments). Compare it with the GF you derived in Exercise 6.99. Exercise 6.119 [3] In §5.4.2, page 224, we calculated E[V (n)], the expected value of key comparisons the binary search procedure, RBS, makes in an array of size n = 2r + s, 0 6 s < 2r . The value found was vn = r + 2s/n. In this exercise you need to calculate its distribution, w(n, k). The basic recurrence for the random variable V (n) is V (n) = 1 + V (Rn ), where Rn , the size of the subinterval RBS continues to search is either ⌊ n2 ⌋ or ⌈ 2n ⌉, in probabilities (⌊ n2 ⌋)/n and (⌈ 2n ⌉)/n, respectively. Define w(n, k) =Pr[RBS uses k key comparisons to locate a key in a sorted array of size n]. Use the process developed in §5.4 to solve the recurrence you obtain.

6.4.4 Walks on the Integer Grid This subsection utilizes generating functions to count the number of lattice paths subject to some constraints. We have already seen walks on the plane integer grid before, as in §2.4 and in Exercise 4.130, but our approach now is analytical. It is based on the observation that the specified number of lattice paths satisfies a partial recurrence, which is solved using corresponding bivariate generating functions. Example 6.120 On the plain grid, we consider paths that start from the origin, (0, 0), and end at some point (m, n). We want to find the number of paths, denoted by Sm,n , from (0, 0) to (m, n), m > n, using steps that carry one square to the right (→), or up (↑), or the diagonals from (k, j) to either (k + 1, j + 1) (ր) or (k + 1, j − 1) (ց). We put additional constraints: the walk may touch, but not cross either of the main diagonals, y = ±x.

To find the recurrence for Sm,n , we observe that the path can arrive to the point (m, n) from either South, (m, n − 1), or West, (m − 1, n), or South-West, (m − 1, n − 1), or North-West, (m − 1, n + 1). However, at the point (m, −m) (m > 0) it can come only from North-West; at the point (m, m) the path can arrive only from either (m, m − 1) or (m − 1, m − 1); at the point (m, −m + 1) the path can come either from (m, −m) or from (m − 1, −m + 1) or from (m − 1, −m + 2); and at the point (m, m − 1) the path can arrive only from either (m, m − 2) or (m − 1, m − 1), or (m − 1, m − 2). Therefore, the number of paths, Sm,n , satisfies the recurrence Sm,n = Sm,n−1 + Sm−1,n + Sm−1,n−1 + Sm−1,n+1 ,

m > |n| + 1,

m > 0;

with the boundary conditions Sm,−m = 1,

m > 0;

S1,0 = 2;

Sm,m = Sm,m−1 + Sm−1,m−1 ,

Sm,−m+1 = Sm,−m + Sm−1,−m+1 + Sm−1,−m+2 ,

m > 2;

Sm,m−1 = Sm,m−2 + Sm−1,m−1 + Sm−1,m−2 ,

m > 2.

m > 1;


324 m

∑ Sm,n xm yn = ∑ Gm (y)xm be its bivariate generating function, where

Let S(x, y) = ∑

m>0 n=−m

m

Gm (y) = ∑ Sm,n

yn ,

m>0

m > 0. In particular

n=−m

1

G0 (y) = S0,0 = 1,

G1 (y) =

∑

S1,n yn = S1,−1 y−1 + S1,0 + S1,1 y = y−1 + 2 + 3y

n=−1

because S1,−1 = 1, S1,0 = 2, and S1,1 = S1,0 + S0,0 = 2 + 1 = 3. Let us denote by tm the coefficient Sm,−m+1 . Then from the boundary condition, Sm,−m+1 = Sm,−m + Sm−1,−m+1 + Sm−1,m+2 , it follows that tm = 2 + tm−1 , m > 0, t0 = S0,1 = 0. This recurrence has the solution tm = Sm,−m+1 = 2m, m > 0. For m > 2, we use the main recurrence to obtain Gm (y) = Sm,−m y−m + Sm,−m+1 y−m+1 + Sm,m ym + Sm,m−1 ym−1 +

m−2

Sm,n yn

∑

n=−m+2 −m

=y

−m+1

+ 2my

m−2

∑

m

m−1

+ Sm,m y + Sm,m−1 y

+

[Sm,n−1 + Sm−1,n + Sm−1,n−1 + Sm−1,n+1 ] yn .

n=−m+2

We calculate each sum separately: m−2

∑

m−3

Sm,n−1 yn = y

Sm,i yi

∑

i=−m+1

n=−m+2

= y Gm (y) − Sm,m−1 ym−1 − Sm,m−2 ym−2 − Sm,m ym − Sm,−m y−m

m−2

∑

∑

m−1

Sm−1,n yn =

n=−m+2

m−2

= yGm (y) − Sm,m−1 ym − Sm,m−2 ym−1 − Sm,m ym+1 − y−m+1 ;

∑

n=−m+1

∑

Sm−1,n yn − Sm−1,−m+1 y−m+1 − Sm−1,m−1 ym−1

= Gm−1 (y) − y−m+1 − Sm−1,m−1 ym−1 ; m−3

Sm−1,n−1 yn = y

∑

i=−m+1

n=−m+2

m−2

Sm−1,i yi = y Gm−1 (y) − Sm−1,m−1 ym−1 − Sm−1,m−2 ym−2

= yGm−1 (y) − Sm−1,m−1 ym − Sm−1,m−2 ym−1 ; m−1

Sm−1,n+1 yn = y−1

∑

Sm−1,i yi

i=−m+3

n=−m+2 −1

=y

Gm−1 (y) − Sm−1,−m+2 y−m+2 − Sm−1,−m+1 y−m+1

= y−1 Gm−1 (y) − 2(m − 1)y−m+1 − y−m .

Summing the results, we obtain Gm (y) = y−m +2my−m+1 +Sm,m ym +Sm,m−1 ym−1 +yGm (y)−Sm,m−1 ym −Sm,m ym+1 −y−m+1 −


325

Sm,m−2 ym−1 +Gm−1(y)−y−m+1 −Sm−1,m−1 ym−1 +y Gm−1 (y)−Sm−1,m−1 ym −Sm−1,m−2 ym−1 + y−1 Gm−1 (y) − 2(m − 1)y−m+1 − y−m . Cancellation yields Gm (y)(1 − y) = Gm−1 (y)(1 + y + y−1 ) − Sm,m ym+1

m > 1,

G0 (y) = 1.

Substituting this expression into the generating function, we get S(x, y) = ∑ Gm (y)xm = 1 + ∑ Gm (y)xm m>0

= 1+ =

m>1 1+y+y−1 1−y

2 1 + x 1+y+y y(1−y)

1 ∑ Gm−1 (y) xm − 1−y ∑ Sm,m ym+1 xm

m>1

S(x, y) −

y 1−y

∑ Sm,m

m>1 ym xm .

Let C(z) = ∑ Sm,m zm , then

m>1

m>1

yC(xy) 1 + y + y2 , S(x, y) = 1 − 1−x y(1 − y) 1−y

=⇒

S(x, y) = y

yC(xy) + y − 1 . x(1 + y + y2 ) − y + y2

y(1 − y) , hence the numerator must 1 + y + y2 vanish at this value. So yC(xy y) + y − 1 = 0, or C(xy y) = −1 + 1/y. By setting u = y xy we obtain C(u)= −1 + xy /u. Substitution of C(u) into the equation yC(xy y) + y − 1 = 0 yields

The denominator in the last equation is zero at x = xy =

x

yx

y −1 + yy + y− 1 = 0 or uy = 1, which gives the cubic equation y3 − y2 + u(1+ y+ y2 ) = 0. This cubic equation has one real root for small u. We denote this root as R(u), which can be approximated by R(u) = 1 + O(u) for small u. Then C(u) = 1/R(u) − 1. Finally we get S(x, y) = y

y/R(xy) − 1 . x(1 + y + y2 ) − y + y2

Exercise 6.121 [3] Consider a subset of lattice points ( j, k), k = 0, 1, j = 0, 1, 2, . . ., and grid paths that start at the origin and end at (n, 0) and can go either East (→), North (↑), or SouthEast (ր) one unit. Thus, the legitimate steps are ( j, 0) → ( j, 1),

( j, 0) → ( j + 1, 0),

( j, 1) → ( j + 1, 1),

( j, 1) → ( j + 1, 0).

Find the number of grid paths from (0, 0) to (n, 0) by constructing the corresponding generating function. Exercise 6.122 (Walks on the plane grid) In this exercise, we consider different variations of lattice paths by adding new kinds of steps, and limiting the accessible regions of the grid. In each of the parts below, consider moves from (0, 0) to (m, n) ∈ N×N. The main goal here is to (k) find the number of paths, denoted by Sm,n (which are defined below for k = 1, 2, . . . , 6), from the origin to (m, n). To achieve this objective, you need to derive recurrences with the initial (k) condition S0,0 = 1 and appropriate boundary conditions and solve it by extracting coefficients (k)

from the corresponding bivariate generating function S(k) (x, y) = ∑m>0 ∑n Sm,n xm yn . Only in the simplest cases it is possible to get a closed expression for the coefficients of these GFs, but as Chapter 11 shows, there exist different methods we can use on the GF to wrest from it

326


useful information. (a) [2–] Re-derive the two-dimensional result of Example 2.78 on page 51, by finding a (1) recurrence for Sm,n , the number of northeastern steps, from the point (0,0) to (m, n) ∈ N × N, and solving it. So the admissible steps are →, ↑. (2) (b) [3–] Redo part (a), and derive the count Sm,n of such walks, when we also allow diagonal (2) moves, as from (k, l) to (k + 1, l + 1). Elements of the sequence Sm,n are called Delannoy numbers. So the admissible steps are →, ↑, ր. (3) (c) [3+] Repeat part (b): compute Sm,n , when we also allow diagonal steps that take us to the right and down: from (k, l) to (k + 1, l − 1). So the admissible steps are →, ↑,ր, ց. (d) [2] In the three parts above we do not allow steps that simply take us one unit down, from (k, l) to (k, l − 1), nor any steps that take us to the left. Why? (4) (e) [4] Now we change part (c): using the same types of steps, compute Sm,n , with the restriction that the walk may not get into negative territory: beginning at (0,0), you may pass on the way to (m, n) ∈ N × N through any (i, j) ∈ N × N only. Hint: The generating function must be regular in a neighborhood of the origin. (5) (f) [4–] For m > n, compute Sm,n , which only counts northeastern walks that do not cross (5) the diagonal i = j. Such walks only pass in points (i, j) ∈ Z × Z with i > j. The number Sm,n was found in Example 2.106 on page 60 where it was denoted by ψm,n and called the ballot number. (6) (g) Calculate Sm,n , the number of northeastern paths that cross the main diagonal (that is, go through a point (k, k) for some k). We do not count paths that just touch the main diagonal y = x, but only those that cross it. The next exercises contain a few of the amazing number of path counting problems that have been developed over time, as models for a variety of processes. New ones are published continually. We conclude this collection with two more that give a very different variation. Exercise 6.123 [3+] This is a riff on part (a) of Exercise 6.122; we allow the same steps as there, but they may be of length of either one or two units. Thus, to go from (0, 0) to (1, 2), we now have two additional walks, (0, 0) → (0, 2) → (1, 2) and (0, 0) → (1, 0) → (1, 2). Similarly, from (0, 0) to (2, 1), we also have two additional walks (0, 0) → (2, 0) → (2, 1) and (0, 0) → (0, 1) → (2, 1). Count the number of paths, with the new types of steps.

Exercise 6.124 You are given a deck of well shuffled cards of two colors, n red and m black. You guess the color of the top card in a deck, after that the top card is removed from the deck. Your score is the number of correct guesses that you make when all cards are gone. It is assumed that you count the number of removed red and black cards so that your next guess is the color that is dominant in the remaining cards, otherwise (when the number of red and black cards is the same) you guess whimsically. Using card counting, what is the expected score, E(n, m), for a deck of n red and m black cards? Note: Each game of “Red and Black” can be viewed as a northeastern lattice path with end points (0, 0) and (m, n) traveling backwards.

6.5. SNAKE OIL SUMMATION

327

6.5 Snake Oil Summation In this section, we demonstrate a way of using generating functions to evaluate combinatorial sums in a closed form. This method, called the Snake Oil Method9 by Herbert Wilf in [137] (which is closely related to the method of coefficients [37]) could handle a variety of sums, but we use it here only for sums involving binomial coefficients10 . Like its name suggests, it cannot cure them all because many combinatorial sums simply have no closed form! The basic idea consists of the following three steps: (1)

To evaluate a summation that depends on an integer parameter, write the generating function of the sum with respect to this parameter.

(2)

Whether the OGF or EGF is the right one depends on circumstances; particularly, on the feasibility of the following step: Evaluate this generating function in closed form (which nearly always requires change of order of summation).

(3)

Extract coefficients to regain the desired value for the initial sum.

Naturally, sometimes the extraction will lead you back to the initial form, but sometimes not! It is true that methods live by the problems they solve, not those they fail at. Example 6.125 The first example is taken from Riordan’s book [115, p.5]. If you read that example together with the author’s comments, you will appreciate the elegance of the present method. Let n n−k k n + k + 1 fn = ∑ (−1) 4 . 2k + 1 k=0 Nothing is immediately obvious, except that the snake oil cries for use: n n−k k n + k + 1 n F(z) = ∑ z ∑ (−1) 4 2k + 1 n>0 k=0 n + k+1 n k changing order of summation = ∑ (−4) ∑ (−z) 2k + 1 n>k k>0 (−z)k using the result of Exercise 2.132 = ∑ (−4)k (1 + z)2k+2 k>0 (4z)k 1 rearranging. . . = ∑ ((1 + z)2 )k (1 + z)2 k>0 1 1 1 = geometric series, Eq. (2.13). = 4z 2 (1 + z) 1 − (1+z)2 (1 − z)2 9 The

term “snake oil” is used to suggest a remedy for all ailments, and carries opprobrium. [37] and Wilf’s [137] books contain a plenty of examples and formulas that can be obtained using the snake oil method. 10 Egorychev’s


328

Finally, we can do the last step by calling upon the binomial theorem: n+1 n n −2 n −2 fn = [z ]F(z) = [z ](1 − z) = (−1) = = n + 1. n n Try to change the dummy index in the formula for fn , as in Example 2.8; you should get 2n−k+1 n k n−k , which looks less convenient, but is of course equivalent. How fn = ∑k=0 (−1) 4 k would you use the snake oil method with it? (Two answers, please.) Example 6.126 Let us look at

∑

i>0

i . n−i

Choosing the ordinary generating function, we multiply this series by zn and sum the result to obtain the generating function g(z): i n g(z) = ∑ z ∑ n−i n>0 i>0 i = ∑∑ zn changing order of summation n − i i>0 n>0 i i = ∑z ∑ zn−i adapting the power to the BC lower parameter n − i i>0 n>0 =

∑ zi (1 + z)i

binomial theorem

1 1 = 1 − z(1 + z) 1 − z − z2

geometric series.

i>0

=

What did we get? We got F(z)/z, where F(z) is the ordinary generating function of the Fibonacci numbers, Eq. (6.58). Hence [zn ]g(z) = [zn+1 ] F(z) = Fn+1 . i . It appears similarly doable, but that i outside the Let us look at another sum hn = ∑i>0 i n−i binomial coefficient is going to complicate matters. It begins very much the same; let the generating function this time be h(z): i n h(z) = ∑ z ∑ i n>0 i>0 n − i i = ∑ izi ∑ zn−i changing order and preparing for binomial summation n − i n>0 i>0 =

∑ i[z(1 + z)]i

binomial theorem

z + z2 (1 − z − z2 )2

differentiated geometric series.

i>0

=

What did we get this time? This looks close to the derivative of the generating function 1+2z g(z). But g′ (z) equals (1−z−z 2 )2 , so they are not quite the same. The derivative of F(z) is

6.5. SNAKE OIL SUMMATION

329

2

1+z F ′ (z) = (1−z−z 2 )2 , which is not quite right either. A linear combination of the two functions does not make the cut—it is not enough to satisfy three equations. 1+z z × ≡ h1 (z)h2 (z). However, here is another way of writing h(z): it equals 2 1−z−z 1 − z − z2 Here we find again that h1 (z) is the OGF of the Fibonacci numbers, and that [zn ]h2 (z) = Fn+1 + Fn = Fn+2 . Hence we have a convolution of Fn with itself—twice shifted: hn = ∑n+1 i=1 Fi Fn+2−i . So we only got another sum, which looks even more intimidating!

When this happens to be the case, one goes back to basics to obtain a usable partial fraction decomposition (PFD). The idea of producing a PFD of a fourth degree polynomial need not alarm us since we know all the roots of this polynomial: 1 1 = z2 + z − 1 (z + φ )(z + φˆ )

√ 1+ 5 , φ= 2

√ 1 − 5 . φˆ = 2

Hence we shall have each of these factors repeated, and the PFD has the following form:

d b c a z + z2 + + = + . 2 2 2 2 (z + z − 1) (z + φ ) z + φ (z + φˆ ) z + φˆ Careful calculation and use of the arithmetic properties of the φ and φˆ provide 1 a=c= ; 5

−3 b = −d = √ . 5 5

Since [zn ](z + φ )−1 = (−1)n /φ n+1 , [zn ](z + φ )−2 = (n + 1)(−1)n /φ n+2 , and φ φˆ = −1, we find,

[zn ]h(z) =

n + 1 n+2 3 3 n + 1 n+2 ˆ n+2 φ φ +φ + φˆ n+2 − Fn+1 . − √ φ n+1 − φˆ n+1 = 5 5 5 5 5

Finally, we have a closed form. Note that, using the representation from Exercise 6.76, one can write all coefficients in terms of the Fibonacci numbers. Also, a computer algebra system may be helpful with algebraic calculations. Remark: We have mentioned before the role of generating functions as a gateway to asymptotics. This is the way to go when the Snake Oil fails at step 3 outlined on page 305—when we do not succeed in getting a useful result for the coefficients of the generating function we obtained at step 2. The following exercises depend on the Snake Oil method [137]. n n−i Exercise 6.127 [3] Show that the sum ∑ i is equal to the nth term of the convolution i i=0 F ⋆ F of the Fibonacci numbers (you computed its value in Exercise 6.77 on page 309).


330

Exercise 6.128 [3] p (a) Let n ∈ N and x = y2 − 4. Show

∑ k6n/2

k

(−1)

" # y − x n+1 y + x n+1 n − k n−2k 1 − . y = x 2 2 k

(b) In part (a), compute the limit of both sides, as y → 2. (c) For part (a), let

p √ t + √1t = y2 − 4. Show

∑ k6n/2

(d) Use the sum given in part (a) to show that ( 1, n − 2k k n−k = ∑ (−1) k m − k 0, k6n/2

(−1)k

(6.76)

n−k 1 − t n+1 . (t + 1)n−2k t k = 1−t k

0 6 m 6 n; otherwise.

(e) Use part (a) or show " directly: # √ x + 1 2n+1 x − 1 2n+1 n + k n−k 1 + , n ∈ N, where x = 4a + 1. ∑ 2k a = x 2 2 k k ( −m m 2 m 2k 1 m/2 , m even; = m ∈ N. Exercise 6.129 [2] Prove ∑ − 2 k k 0, m odd; k Note: See [58, p. 8–9]. n−1 2k n + k (−1)k = , m, n ∈ N. Exercise 6.130 [3−] Show that ∑ m−1 k m + 2k k + 1 k 2 2n k n = ; (b) ∑ = Fn+1 , n ∈ N, Exercise 6.131 [2] Prove (a) ∑ n k k>0 n − k k where {Fn } is the sequence of the Fibonacci numbers. n (−1)k+1 = Hn , n ∈ N. Exercise 6.132 [1+] Show that ∑ k k>1 k This formula was derived in §5.1, as Eq. (5.1). The re-derivation requested here is simple, if you remember Exercises 6.4, 2.132, and 2.15. Exercise [4] Exercise 6.132 is a particular case of the equality 6.133 1 n (−1)k ∑ k x + k = x n+x , obtained in Example 2.139 on page 71. Show that this is the case k n by taking the limit x → 0 in the right-hand side. n + k n−k 22n+1 + 1 , n ∈ N. Exercise 6.134 [3] Show that ∑ 2 = 3 2k k 2n + 1 m + k 2m + 1 Exercise 6.135 [4] Show that ∑ = , n ∈ N. 2k 2n 2n k

6.6. APPLICATIONS IN PROBABILITY

331

n n+m+k n−m −k −p n ∈ N. Exercise 6.136 [3] ∑ (−2) = (−4) , p= 2 m+k k p k Hint: Change index of summation first, m + k → j, so you can use the snake oil method on the value m > −n. Note: See Riordan’s book [115]. j+n m n k + m k 1 − zn+1 ∑ j=0 n (1 − z) j , m, n ∈ N. Exercise 6.137 [2] Show that ∑ z = (1 − z)m+1 m k=0 Reminder: if the summation on the left-hand side is allowed to extend unbounded, you have the binomial development of (1 − z)−m−1 . Exercise 6.138 [3] Find the unique sequence {ai }i>0 of real numbers, with a0 = 1, satisfying ∑nk=0 ak an−k = 4n . Exercise 6.139 [3+] Using the snake oil method, find ∑ j>0 2mj (−a) j (see Exercise 6.37). Exercise 6.140 [2+] Using the generating function Fn (z) = (2 + z2 + z−2 )2n , prove that n (4n)! 22k ∑ (2k)! [(n − k)!]2 = [(2n)!]3 , n ∈ N. k=0 Hint: Use the fact that 2 + z2 + 1/z2 = (z + 1/z)2 .

6.6 Applications in Probability This section illustrates the usefulness of generating functions in the probability theory. As we have seen, many probability derivations require extensive calculations. This section shows that many of these tasks can be greatly simplified by using probability generating functions. In particular, all moments can be obtained by differentiation of the PGF.

6.6.1 Definition of Generating Functions Used in Probability We have seen generating functions (GFs) as a tool to connect sequences, that is, discrete functions, to functions of a continuous variable. We did not make any assumption about the values of the sequence members. Nothing should stop us from picking very special kind of sequences: the probability mass functions (PMF) of discrete random variables. The fact that they are special, since the terms are nonnegative and sum to 1, will imply various properties of the GFs. So we introduce specific terminology for them. For simplicity, we now consider only integer-valued random variables, but any discrete variable can be similarly accommodated (see discussion on page 97). Definition 6.141 Let X be an integer-valued random variable and pk = Pr[X = k], k > 0, be its probability mass function. Then its probability generating function (PGF) is (6.77) p(z) = ∑ pk zk = E zX . k>0

Many of the properties of the PGF can be viewed as a result of the representation p(z) = E zX . For instance, p(1) = 1. ⊳

332


Example 6.142 The polynomial whose coefficients are the probabilities associated with the different outcomes of rolling a fair die is p(z) =

1 1 1 1 z + z2 + · · · + z6 = 6 6 6 6

6

z 1 − z6 . 1−z

∑ zk = 6

k=1

(6.78)

When two dice are thrown simultaneously, the generating functions associated with them are 1 (z + z2 + · · · + z6 ) 6

and

1 (x + x2 + · · · + x6 ) . 6

The probability that the first uppermost die is r and the second face reveals s will be the coefficient of zr xs in the expansion of xz(1 − x6 )(1 − z6 ) , 36(1 − x)(1 − z) for r, s = 1, 2, . . . , 6, because the number falling uppermost on the first die is independent of the number falling uppermost on the second. If one is interested in just the sum r + s = a, then the probability generating function of the sum of the numbers falling uppermost on two dice will be p2 (z) =

z2 z2 (1 − z6 )2 = (1 + z)2 (1 − z + z2 )2 (1 + z + z2 )2 . 36 (1 − z)2 36

(6.79)

This example shows that the probability generating function is actually a polynomial if the random variable has a finite number of outcomes. The PMF get more interesting when we deal with the moments. For example, the mean of X , E[X ] =

∑ k Pr[X = k] = ∑ k pk = p′ (1),

(6.80)

k>0

k>0

is the derivative of the PGF at z = 1 since p′ (z) = ∑k>0 k pk zk−1 . The second derivative of the probability generating function is p′′ (z) =

∑ k(k − 1) pk zk−2 = ∑ k2 pk zk−2 − ∑ k pk zk−2 . k>0

k>0

k>0

Hence, p′′ (1) = E[X (X − 1)] = E[X 2 ] − E[X ], which is called the second factorial moment of X . This formula can be extended to any order: µ[r] , the rth factorial moment of a random variable. We visited it before, in Definition 3.88 on page 113; now it is calculated by evaluating the rth derivative of its PGF at z = 1:

µ[r] = p(r) (1) = lim ∑ pk k(k − 1)(k − 2) · · · (k − r + 1)zk−r z→1

k>r

= E[X (X − 1)(X − 2) · · · (X − r + 1)] = E [X r ] .


333

Exercise 6.143 [1] Let p(z) be the probability generating function for a discrete random variable X and assume that E[X 2] exists. Show that its variance can be computed as 2 V [X ] = p′′ (1) + p′ (1) − p′ (1) .

(6.81)

Exercise 6.144 [1] Suppose that a random variable X has possible values in N = {0, 1, 2, . . .}, and its generating function is φ (z). What is the generating function for the random variable Y = aX + b, where a and b are constants? Exercise 6.145 [3] We know from Eq. (C.39) that the polynomial f (z) = zn = (z + n − 1)n is the ordinary generating function for the sequence of the Stirling cycle numbers. Using this generating function and remembering the combinatorial meaning of the signless Stirling numbers of the first kind, show that on average a permutation of n elements has Hn cycles. Definition 6.146 Let X be a discrete random variable. The function mX (t) = E etX is called the moment generating function (MGF) of X . This function is defined only for those values of t for which the expected value summation is convergent. ⊳ While mX (t) carries, in principle, the same information as the PGF of X (i.e., everything), it is packaged in a way that is more convenient for certain applications. Exercise 6.147 [1] Express the moment generating function in terms of the probability generating function (PGF). Hint: Recall that the PGF is E[zX ]. Definition 6.148 The natural logarithm of the moment generating function, kX (s) = ln mX (s), is called the cumulant generating function: it is the exponential generating function of j the cumulants: kX (s) = ∑ j>0 sj! κX, j . So the j-th cumulant is the j-th derivative of ln mX (s) evaluated at s = 0. ⊳

Exercise 6.149 [3] As with the MGF, we get no new information from the cumulant generating function, but it presents the information about the random variable in a special way. The cumulants have several properties, which endear them to statisticians. Prove by calculation: (a) [1] E[X ] = κX,1 = kX′ (0); (b) [2] V [X ] = κX,2 = kX′′ (0). m X j , the sum of m independent (but not necessarily identically distributed) (c) [2] Let Y = ∑r=1 random variables, then the cumulants of the sum are simply sums of the cumulants, unlike m κX j ,r (r = 0, 1, 2, . . .). any other variety of moments: κY,r = ∑ j=1 X Exercise 6.150 [2−] Let p(z) = E z be the probability generating function of a random variable X . Show that the generating functions of the tail probabilities are as follows (a)

∑ Pr[X > k]zk = k>0

1 − p(z) ; 1−z

(b)

∑ Pr[X > k]zk = k>0

1 − zp(z) . 1−z

Exercise 6.151 [3] Since coefficient extraction is the inverse operation of generating function calculation, once a solution of a problem provides a PGF, the road is open to find the PMF. All the tools we discussed in §6.1 are available for this purpose. Find the PMFs giving rise to the following PGFs, with p ∈ [0, 1], p + q = 1:


334

p 1 + p(1 − z) ; (b) ; (c) exp(qz2 + pz − 1). 2 1 − qz 2 − pz − qz2 Exercise 6.152 [3] Let a random variable X defined on the natural numbers have the known probability generating function p(z). (a) Write, in terms of p(z), the PGFs of the following random variables: (1) 1 + X ; (2) 2X ; (3) X 2 . (b) Find, in terms of this PGF, the OGF of each of the following sequences (which are all probabilities, but not necessarily PMFs): (1) Pr[X 6 n]; (2) Pr[X < n]; (3) Pr[X > n]; (4) Pr[X = 2n]. (a)

Sometimes it is convenient to consider generating functions that involve negative powers as the following example shows. Example 6.153 (Two-man competition) The World Chess competition consists of the match of n games between two participants. One can easily find other examples of games played by two competitors; for instance, Deep Blue and Gary Kasparov played 6-games match a couple of times. So let A and B play a match of n games. For each game, score 1 point when A wins with probability p, –1 points when B wins, with probability q, and 0 points for a tie (with probability 1 − p − q). A similar system is used, for example, in ice hockey to evaluate a player’s impact on the ice: a player is given +1 rating if his team scores while he is on the ice, and –1 if the other team scores. The PGF for the match score in such a system is def

Q(x) = (qx−1 + 1 − p − q + px)n =

n

∑

(n)

p j (p, q) x j ,

(6.82)

j=−n

(n)

(n)

where p j (p, q) is the probability that an n-game match ends with a score j. Thus p0 is the probability that a match of n games ends in a draw and we obtain its value from the multinomial theorem (page 48): (n)

p0 (p, q) =

⌊n/2⌋

∑

j=0

n 2j

2j (1 − p − q)n−2 j (pq) j . j

(6.83)

(n)

The probability for A to win all n games is pn = pn and the probability to lose all n games (n) is p−n = qn .

Exercise 6.154 [1+] There are 3n possible outcomes in a two-man competition match of n (n) (n) games. Let N j = 3n p j be the number of ways that a score of j can arise. Derive the (n)

recurrence for the double index sequence {N j }, | j| 6 n, n ∈ N.

Exercise 6.155 [3−] In the two-man competition, Example 6.153, suppose that all three probabilities (a win for either of the competitors, or tie) are the same and equal 1/3. Show that the probability of a drawn match becomes less as the match length increases: (n)

(n+1)

p0 > p0

for each n > 2;

(1)

(2)

p0 = p0 =

1 . 3

6.6. APPLICATIONS IN PROBABILITY (n)

(n+2)

Does the inequality p0 > p0

335

hold for arbitrary probability values of p and q?

6.6.2 Examples and Problems The simplest non-degenerate random variable11 is the Bernoulli random variable, B(p), which we introduced in §4.1.1. It can assume only two values: 0, with probability q, and 1, with probability p = 1 − q. Its PMF has only two nonzero values, hence the PGF is B(z) = q + pz. Example 6.156 (Geometric distribution) We have met the geometric distribution before (§4.1.3); it is also an off-shoot of the Bernoulli distribution: in a sequence of independent and identically distributed Bernoulli trials, such as coin flips, it is the number of trials between successive heads, for example. Hence, with the traditional choice of parameters, for all positive integers k, Pr[X = k] = pqk−1

=⇒

gX (z) = pz ∑ (qz)k−1 = k>1

pz , 1 − qz

|qz| < 1.

(6.84)

Since g′X (z) = p(1 − qz)−1 + pz q(1 − qz)−2 = p(1 − qz)−2 and g′′X (z) = 2pq(1 − qz)−3 , we find its expected value and variance to be E[X ] = g′X (1) =

1 , p

V [X ] = g′′X (1) + g′X (1) − (g′X (1))2 =

and

q . p2

As you probably noticed, there are two versions of the geometric distributions. In the context of the above waiting time to flip heads, we counted all the flips, including the success. The other version only counts the failures, the tails, and hence the support of that distribution includes the value zero too. Let us denote such a random variable by Y , and then Pr[Y = k] = pqk

=⇒

gY (z) = p ∑ (qz)k = k>0

p . 1 − qz

(6.85)

Since X = Y + 1, we have the “automatic” result E[Y ] = gY′ (z)|z=1 = 1/p, whereas a similar calculation for gY (z) provides V [Y ] = V [X ]. Exercise 6.157 [1+] Use the PGFs of X and Y from Example 6.156 to calculate the higher order moments. Explain why it is clear, without computing, that both have the same variance. Example 6.158 (Poisson distribution) The Poisson distribution is used to model numerous real-life situations, described in §4.1.6. Here is its probability mass function with a rate parameter conventionally denoted by λ :

pX (r) = Pr[X = r] = e−λ

λr , r!

r > 0,

11 We say that a random variable is degenerate if it has a single possible value: its single point.

PMF

is concentrated in a


336

which leads to the probability generating function gX (z) = e−λ

∑

r>0

(λ z)r = e−λ (1−z) = eλ (z−1) . r!

(6.86)

Hence, the very compact form for the factorial moments is found to be (r)

µ[r] = gX (1) = λ r ,

(6.87)

(r)

where gX is the rth derivative of gX . Now simple differentiation yields E[X ] = g′X (1) = λ ;

E[X 2 − X ] = g′′X (1) = λ 2

=⇒

V [X ] = λ 2 + λ (1 − λ ) = λ .

We can use our prowess in extracting coefficients for other purposes than solving recurrences, such as reading off the moments from an expansion of the PGF of a random variable. We saw that factorial moments µ[r] are provided by the derivatives of the PGF gX (z) at the point z = 1. To obtain them by coefficient recovery, we use the Taylor series expansion at z = 1: (r)

gX (1 + u) =

gX (1) + g′X (1) u +

g (1) r g′′X (1) 2 u + ···+ X u + ··· . 2! r!

(6.88)

Therefore, µ[r] = p(r) (1) = r![zr ] pX (1 + z).

Example 6.159 (Uniform random variable) A discrete random variable X is uniform if its probability mass function has the same values wherever it is nonzero. The standard discrete uniform random variable is defined on a finite range of integers from 0 to, say, R − 1 with the probability mass function pX ( j) =

1 , for 0 6 j 6 R − 1, and 0 otherwise. R

Hence its probability generating function equals gX (z) =

R−1

∑

pX ( j) z j =

j=0

Since we want to use Eq. (6.88), we rewrite gX (z) in the form gX (1 + u) =

1 R

R−1 j=0

1 1 − (1 + u)R (1 + u)R − 1 = , R 1 − (1 + u) uR

and find that for all j > 0, ( j)

pX (1) = =

j! j! [u j ] pX (1 + u) = [u j+1 ] (1 + u)R − 1 R R R k (R − 1) j j! j! j+1 [u ] ∑ . = u = R R j+1 j+1 k>1 k

Thus, we have the following values: R−1 1 R , and = E[X ] = R 2 2

1 − zR

∑ z j = R(1 − z) .

2 R (R − 1)(R − 2) E[X (X − 1)] = , = R 3 3


337

from which the second central moment is simply derived E[X 2 ] =

(R − 1)(R − 2) R − 1 (R − 1)(2R − 1) + = , 3 2 6

and the variance becomes R2 − 1 (R − 1)(2R − 1) (R − 1)2 R − 1 − = (2(2R − 1) − 3(R − 1)) = . 6 4 12 12 Exercise 6.160 [1] A fair die is thrown repeatedly until a 6 is obtained. Let X be the number of throws required. Show that the probability generating function of X is z/(6 − 5z). V [X ] =

Exercise 6.161 [2+] Let dn be the number of ways of rolling a die n times such that no consecutive sixes occur. Find the generating function for this sequence and determine dn by extracting coefficients. What is the probability that no consecutive sixes are obtained when a die is rolled n times? Exercise 6.162 [3] A true die is rolled repeatedly until two successive ones appear for the first time. Find the PGF of the number of times you need to roll and its expectation.

6.6.3 Convolution An important way in which generating functions enter applied probability is through the fact we learned in Chapter 4: the distribution of a sum of independent, integer-valued random variables is a convolution of the probability mass functions of the added variables. For example, let X and Y be independent random variables and W = X +Y , then Pr[W = k] = Pr[X +Y = k] = ∑ Pr[X +Y = k|X = j] Pr[X = j] = ∑ Pr[Y = k − j] Pr[X = j]. j

j

We also know that the GFs of a convolution is a product of the corresponding GFs—hence gX+Y (z) = gX (z)gY (z). We can also easily derive the last equation from Definition 6.141 (recall that X and Y are independent random variables) gX+Y (z) = E zX+Y = E zX · zY = E zX · E zY = gX (z) gY (z).

Let us see the consequences with some of the distributions. In the Bernoulli case, we saw gX (z) = q + pz. Now suppose we look at a sum of n independent samples of this random variable, and inquire about the sum: Sn = X1 + X2 + · · · + Xn . Since they are all independent, we immediately know that gSn (z) = (q + pz)n , from which we extract coefficients, and we get n j n− j j Pr[Sn = j] = [z ]gSn (z) = pq . j Naturally enough, this is the binomial distribution. Now using Eqs. (6.80) and (6.81), we re-derive the expected value and the variance of the binomial random variable X to be d n ′ (q + pz) = np(q + pz)n−1 = np(q + p)n−1 = np 1n−1 = np; E[X ] = pX (1) = dz z=1 z=1


338

V [X ] = p′′ (1) + p′ (1) − (p′ (1))2 = n(n − 1)p2 (q + pz)n−2

z=1

+ np − (np)2 = npq.

Exercise 6.163 [1+] Find factorial moments of the number of 0’s in the random word w ∈ {0, 1}n , when the digits 0 and 1 are selected in equal likelihood. The sum of uniformly-distributed random variables does not lead to such easy insights. Still, compared with other approaches, it is the route of choice to prove that the normalized sum of many such variables approaches the Gaussian distribution (this is a particular case of the central limit theorem 10.136, §10.7). In Example 4.117 on page 176, we considered the convolution, W = X +Y , of two fair dice X and Y (with uniform distributions). As we saw, the probability distribution of W is not uniform; the probability generating function (PGF) of the result of rolling two dice is the square of the PGF we showed in Eqs. (6.78), (6.79), z2 1 + 2z + 3z2 + 4z3 + 5z4 + 6z5 + 5z6 + 4z7 + 3z8 + 2z9 + z10 . (6.89) 36 This is an opportunity to observe the following fact: if we have two pairs of random variables, say X1 and X2 , and the other Y1 and Y2 , with their respective PGFs, and it happens that pX1 (z)pX2 (z) = pY1 (z)pY2 (z), then the distributions of X1 + X2 and that of Y1 +Y2 are identical, regardless of the individual distributions taking part in this equality. Obvious as this fact is, it can give rise to surprising situations.

p2 (z) =

Consider the polynomial (6.89), it is reducible, of course, since it is the square of p(z). However, p(z) itself can be factored, (disregarding for a moment the multiple 1/6), into z(1 + z)(1 + z + z2 )(1 − z + z2 ). We again disregard for the moment the isolated z, but still, p2 (z) allows us to mix and match six factors, and some experimentation shows that four of them give rise to the PGF, (1 + z)(1 + z + z2 )(1 − z + z2 )2 = 1 + z2 + z3 + z4 + z5 + z7 , and the remaining two do likewise: (1 + z)(1 + z + z2 ) = 1 + 2z + 2z2 + z3 . Therefore, these two distributions have the same convolutions as p(z) with itself. In other words, if we take two fair dice and change the numbering of the sides, so that one has 1,3,4,5,6,8, and the other has 1,2,2,3,3, and 4, rolling these dice will produce results (the sum of two faces) identically distributed to rolling two fair dice. Do not try to show this in a casino. Example 6.164 Can we “uniformize” the distribution of the result of rolling the fair dice? That is, can two dice be loaded so that the result of rolling them is uniformly distributed: Pr[W = k] = 1/11, for 2 6 k 6 12? If we could do this, then their convolution should have the PGF (x2 + x3 + · · · + x12 )/11, that has to satisfy the identity 1 2 (x + x3 + · · · + x12 ) = (p1 x + p2 x2 + · · · + p6 x6 )(q1 x + q2 x2 + · · · + q6 x6 ), 11 where pi and qi (i = 1, 2, . . . , 6) are the probabilities of ith faces of the two dice. Factoring out x2 from the last equation, we obtain 1 + x + x2 + · · · + x10 = 11(p1 + p2 x + · · · + p6 x5 )(q1 + q2 x + · · · + q6 x5 ).

(6.90)


339

Now observe that the polynomial 1 + x + x2 + · · · + x10 has no real roots but only complex roots that are symmetrically spread on the unit circle in the complex plane. This is a result of the left-hand side of the previous equation being equal to (1 − x11 )/(1 − x). Since 1/11 = Pr[W = 2] = p1 q1 = Pr[W = 12] = p6 q6 , it follows that none of p1 , p6 , q1 , and q6 vanishes. Hence each factor on the right-hand side of Eq. (6.90) is a fifth-degree polynomial. An odd degree polynomial with real coefficients has at least one real root. So we cannot “uniformize” the two dice. For Poisson random variables, we again get a very simple expression, with an additional useful property. The addition of Bernoulli variables produces a simple distribution only if all the summands have the same parameter. The Poisson distribution displays a more tolerant behavior: when we consider the sum of independent Poisson random variables, Y = ∑i Xi , where Xi has the Poisson distribution with parameter λi , then Eq. (6.86) tells us that gY (z) = exp (∑i λi (z − 1)), which is just a Poisson PGF itself, with a parameter that is the sum of those of all the components! With the geometric distribution and its convolution, which gives rise to the negative binomial distribution, we return, in a way, to the situation with the Bernoulli distribution. As Eq. (6.85) shows, the PGF of a geometric distribution G(p) is p(1 − qz)−1 . Convolving k of these, we obtain the PGF to be pk (1 − qz)−k , and the binomial theorem provides pk k −k n k n+k−1 ⋆k n =p (−q) = p qn . (6.91) Pr[G (p) = n] = [z ] (1 − qz)k n n Exercise 6.165 [1+] Compare the last result with Eq. (4.7), and explain the difference. Note how significant is the requirement for the geometric random variables to be identically distributed. What do we get if we convolve k1 independent random variables with the probability generating function (PGF) given in Eq. (6.84) with k2 of those with the PGF given in Eq. (6.85)? Exercise 6.166 [1] The probability generating function of a random variable X is p(z) = 1 2 3 2 64 (3z + 2z + 3z ) . Find the probability generating function of X1 + X2 + · · · + Xn , where Xi (i = 1, 2, . . . , n) are independent observations of X . Exercise 6.167 [1+] Suppose that person A throws a die repeatedly until a 6 appears. Another person B does the same until 1 or 6 will be on the uppermost face. Let X be the number of rolls required by both A and B together. Find the generating function of X . Exercise 6.168 [1+] A box contains four balls, numbered 0, 1, 1, and 2. Suppose n balls are successively drawn, with replacement between drawings. Let X be the sum of the n numbers drawn. Show that the generating function of X is (1 + z)2n /22n . Exercise 6.169 [2] (W. Feller, [41]) Let X and Y be random variables with PGFs p(z) and q(z), respectively. Show that Pr[X −Y = j] = [z j ]p(z)q(1/z), for j = 0, ±1, ±2, . . ..

Exercise 6.170 [2+] (W. Feller, [41]) Consider a sequence of n independent Bernoulli experiments, with the probability of success, p. Let un be the probability that the total number of

340


successes is even. (a) Show that the values un satisfy the recurrence un = qun−1 + p(1 − un−1 ). def (b) Find the generating function U (z) = ∑n>0 un zn that solves the recurrence and obtain un . (c) Use the idea of multisection of series (§6.1.5) to re-obtain the result of part (b) from the PGF of the underlying binomial distribution. Exercise 6.171 [4] Again we consider a sequence of n independent Bernoulli trials, with the parameter p = 1/2. Unlike the procedures leading to the binomial or geometrical distributions, we do not count the total number of successes, or failures, but look at the probability of getting three successive successes, and define vn as the complementary probability that in a sequence of n trials no three successive trials succeed. Note: Other approaches exist for such problems. We show one in Example 9.71 on page 517 using finite state machines. An even more natural one is to look at a four-state Markov chain, with the states corresponding to the number of recent successive successes (see Example 9.101). (a) Clearly v0 = v1 = v2 = 1. Find relations that hold for vn , when 3 6 n < 6, and show that 1 1 1 the following recurrence vn = vn−1 + vn−2 + vn−3 is valid for n > 6. 2 4 8 (b) Define v(z), the ordinary generating function of the vn , and solve for it. (c) The v(z) of part (b) is not a probability generating function since {vn } is not a probability mass function (PMF). Explain why it is not. Then define another sequence, wn , having the PMF that is the probability mass function when the first time three successive successes occurred ended with trial number n. Find a (simple) relation between elements of {vn } and elements of the sequence {wn }. (d) This part takes you back to the generating function v(z) of part (b). Use it to find a closed expression for vn . Since the denominator of the GF does not factor nicely, you need to use a numerical procedure to find the roots and obtain a partial fraction decomposition. Show that the vn are written in terms of the tribonacci numbers. The justification for the name will be obvious once you see v(z).

6.6.4 Quicksort and Binary Search Analysis We collect here several developments of the analysis of the Quicksort algorithm, started in §5.5, and related topics, using generating functions. Much of the discussion is cast as calculations of binary tree parameters under distributional assumptions. Example 6.172 (Quicksort: mean values via GFs) We return to the average case analysis of the Quicksort algorithm that was done in §5.5. Let xn be the expected number of comparisons Quicksort needs to sort n distinct numbers; we denote by an the expected number of comparisons in one stage when the array is of size n. Let Pn,k be the probability of the pivot being k-th smallest out of n entries. We assume, as it is usual in the analysis of comparison-based sorting algorithms, that any permutation of the


341

given distinct n elements is equally likely. Then Pn,k = 1/n and we have n

n

xn = an + ∑ Pn,k (xn−k + xk−1 ),

or

nxn = nan + 2 ∑ xk−1 ,

k=1

n > 1,

x0 = 0,

k=1

Let X (z) and A(z) be ordinary generating functions for sequences {xn } and {an }, respectively. Then we obtain 2 X (z). X ′ (z) = A′ (z) + 1−z Setting an = n + b, b is a constant, we get the non-homogeneous differential equation for X (z): 2z b+1 2 + . X (z) = X ′ (z) − 2 1−z (1 − z) (1 − z)3 Multiplying both sides by an integrating factor (1 − z)2 , we obtain an exact equation: 2z d , (1 − z)2 X (z) = b + 1 + dz 1−z

=⇒

(1 − z)2 X (z) = z(b + 1) − 2z − 2 ln(1 − z).

From Exercise 6.4, we know that the OGF of the harmonic numbers, Hn = ∑nk=1 1k , is given by H(z) = ∑n>1 Hn zn = −(1 − z)−1 ln(1 − z), and we can rewrite the solution in a form easier for the extraction of coefficients: X (z) = (b − 1)

ln(1 − z) z 2 z −2 = (b − 1) − + 2H ′ (z). 2 2 2 (1 − z) (1 − z) (1 − z) (1 − z)2

Extracting coefficients, we have xn = [zn ]X (z) = 2(n + 1)Hn+1 + (b − 3)(n − 1). Example 6.173 (Moments of tree parameters) The purpose of this example is to display a general approach for computing the moments of data structure parameters by forming a bivariate generating function of the distributions of the parameters and the size of the structures. We consider the number of leaves in binary trees, but the technique is quite general. We start with leaf distribution in binary search trees. Then we repeat the analysis for the case of Catalan statistics.

Leaves in binary search trees (BSTs) Leaf count is a convenient variable to investigate because of the simple relation it satisfies. Let the tree be of size n and denote the value of the root12 by ρ . All the leaves are in the subtrees, except n = 1, when the root also serves as the only leaf. Hence, considering the value at the root for the moment to be a random variable R, we get

L(n) = L(R − 1) + L(n − R) + δn,1,

n > 1,

L(0) = 0, L(1) = 1,

(6.92)

12 We can assume that either the root has actual value ρ when all nodes are from the set of n first integer numbers, [1..n], or it is simply the ρ th smallest among the keys stored in the tree.


342

where L(n) is the number of leaves in the binary tree with n nodes and δn,1 is the Kronecker delta, Eq. (2.5). We observe that the two L values in this relation are only dependent through their sharing the value of R, and once we randomize on it they are independent. Let pn,k = Pr[L(n) = k] be the PMF; with the indicated randomization, we have pn,k =

1 n k 1 n Pr[L(r −1)+L(n−r) = k] = ∑ ∑ Pr[L(r −1) = j] Pr[L(n−r) = k− j], ∑ n r=1 n r=1 j=0

n > 2.

Introducing now the probability generating function λn (z) = ∑k>0 pn,k zk , and observing that the probabilities provide a complete convolution, we obtain

λn (z) =

n

1 n

∑ λr−1(z)λn−r (z),

n > 2, λ0 (z) = 1, λ1 (z) = z.

(6.93)

r=1

If we differentiate this full-history relation and set z = 1, then we obtain the same recurdef rence, l(n) = E[L(n)] = 2 ∑n−1 k=0 l(k), that we have seen in Exercise 5.89, page 234. A second differentiation, also at the point z = 1, would lead to a very similar, but much more wieldy recurrence for the second factorial moment. We approach it differently by defining a bivariate generating function, G(u, z) = ∑n>0 λn (z)un . Multiplying Eq. (6.93) by nun−1 and summing over n, we find the following relation n

n

Gu (u, z) = λ1 (z) + ∑ un−1 ∑ λr−1 (z)λn−r (z) = z − 1 + ∑ un−1 ∑ λr−1 (z)λn−r (z) (6.94) n>2

r=1

n>1

r−1

= z − 1 + ∑ λr−1 (z)u r>1

n−r

∑ λn−r (z)u

n>r

r=1

2

= G (u, z) + z − 1.

The equation Gu (u, z) = G2 (u, z) + z − 1 encapsulates the entire probabilistic characterization of the leaf count parameter in BSTs. The moments are obtained by differentiating with respect to z and evaluating at z = 1. Because we solve differential equations instead of difference equations, this approach is better than the method used in Exercise 6.176. The first derivative yields Guz (u, z) = 2G(u, z)Gz (u, z) + 1. At the point z = 1, we see G(u, 1) =

1

∑ λn (1)un = ∑ un = 1 − u .

n>0

n>0

Gz (u, 1) =

def

∑ λn′ (1)un = ∑ ln un = v(u),

n>0

n>0

where v(u) is the OGF of the expectations E[L(n)]. The crucial point is to view Guz (u, 1) simply as v′ (u). We have then the equation v′ (u) = 2v(u)/(1 − u) + 1, which fits the template of Eq. (6.66), providing the solution 3u − 3u2 + u3 1 1 1 3 (1 − u) , v(u) = . C − , v(0) = 0 =⇒ C = v(u) = (1 − u)2 3 3 3(1 − u)2 Extracting coefficients and using the Iverson bracket notation (2.4), page 29, we find ln = [un ]v(u) = [un−1 ]

1 n−3 1 1 1 − [un−2 ] [u ] 2 2 (1 − u) (1 − u) 3 (1 − u)2


343

   0, 1 = n [n > 1] − (n − 1) [n > 2] + (n − 2) [n > 3] = 1,  3  n+1 3

n = 0, n = 1, 2, , n > 3.

(6.95)

The process to secure the second moments is similar, but requires more work. The function we manipulate is Gzz (u, z), which at z = 1 is the OGF of the second-order factorial moments, and it is denoted by f (u). Differentiating the equation for Guz (u, z), we have Guzz (u, z) = 2G2z (u, z) + 2G(u, z)Gzz (u, z)

z=1

=⇒

f ′ (u) = 2v2 (u) +

2 f (u). 1−u

Again, the solution follows the template in Eq. (6.66), and provides f (u) =

2 u6 − 6u5 + 15u4 − 15u3 + 5u − 5 C − (1 − u)2 45 (1 − u)3

f (0)=0

=⇒

2 C=− . 9

Following the same procedure as in Eq. (6.95), we find the GF and its coefficients to be  0, n 6 2,   2 3 3 2u 15 − 15u + 6u − u n and fn = [u ] f (u) = f (u) = 2/3, n = 3, (6.96)  45 (1 − u)3  (n+1)(5n−8) , n > 4. 45

It is now possible to compute the second moment itself, which excepting the initial values is (n+1)(5n+7) 5n−8 , and the variance then becomes given by E[L2 (n)] = n+1 3 15 + 1 = 45 n+1 2 2 (n + 1)(5n + 7) = (n + 1). − V [L(n)] = 45 3 45

(6.97)

The root of the variance is in Θ(n1/2 ), which tells us that the number of leaves is typically quite close to its expectation for trees with big number of nodes.

Leaves in Catalan binary trees We now consider the distribution of leaves in binary trees following the Catalan statistics: def 2n 1 each of the Cn = n+1 n trees of size n is equally likely. The basic relationship is the same shown in Eq. (6.92), except that we now denote the size of the left-hand subtree with v leaves as Lv to obtain Ln = Lv + Ln−v−1 + δn,1 , for n > 1, with the initial values L0 = 0, L1 = 1. As before, we use this to write a recurrence for the probability mass function (PMF) of Ln , randomizing of the subtree sizes, where the size v has the probability CvCn−v−1 /Cn . We denote now the corresponding PMF by qn,k and find

qn,k =

1 n−1 k 1 n−1 C C Pr[L + L = k] = v n−v−1 v n−v−1 ∑ ∑ ∑ Cv qv, jCn−v−1 qn−v−1,k− j , Cn v=0 Cn v=0 j=0 n−1

n−1 k

an,k =

∑

∑ av, j an−v−1,k− j

v=0 j=0

(6.98)

=⇒

αn (z) =

∑ αv (z)αn−v−1 (z)

v=0

n > 2,

(6.99)


344

def

where we make the suggestive definition an,k = Cn qn,k . This is still a PMF, but not a normalized one, as the marginal sum gives the indicated Catalan number, rather than 1. Therefore, αn (z) = ∑k>0 an,k zk is its generating function, while a probability generating function (PGF) carries the lack of normalization. We continue with a bivariate generating function, H(u, z) = ∑n>0 αn (z)un . Using the relation (6.99), established for the PGFs, we find that we need to work around its range, n > 2, by adding α0 (z) + uα1 (z) = 1 + uz: n−1

H(u, z) = 1 + uz + ∑ un ∑ αv (z)αn−v−1 (z)

(6.100)

v=0

n>2

= 1 + uz − u + u ∑ αv (z)uv v>0

∑

n>v+1

αn−v−1 (z)un−v−1 = 1 + uz − u + uH 2 (u, z).

This is a pleasing result since the bivariate generating function is obtained directly: p 1 − 1 − 4u(1 − u(1 − z)) . (6.101) H(u, z) = 2u We have selected the (−) sign in the numerator to make the function H(u, z) regular at u = 0. Since we have an explicit expression for the function, obtaining the moments is a matter of differentiation and extracting the coefficients. We observe the following values at z = 1: √ 1 − 1 − 4u n n , αn (1) = ∑ an,k = Cn ∑ qn,k = Cn ; H(u, 1) = ∑ αn (1)u = ∑ Cn u = 2u n>0 n>0 k>0 k>0 where the last value was obtained from Eq. (6.101), and it is identical to the one defined in Eq. (6.47), page 300. For the derivatives, we get def

αn′ (1) = Cn ∑ kqn,k = Cn vn , k>0

def

αn′′ (1) = Cn ∑ k(k − 1)qn,k = Cn E[Ln (Ln − 1) = Cn µ[2] (n). k>0

Mean values: Differentiating H(u, z) with respect to z, we find Hz (u, z) = p

u 1 − 4u(1 − u(1 − z))

Hence Cn vn = [un ]Hz (u, 1) =

−1/2 n−1

z=1

=⇒

(−4)n−1 =

find the expected number of leaves to be vn = Exercise 6.176(b).

Hz (u, 1) =

u . (1 − 4u)1/2

(6.102)

2n 2n−2 1 n−1 and dividing this by Cn = n+1 n , we n(n+1) 2(2n−1) , the same value you are asked for in

The variance: Re-differentiating Hz (u, z), as given in Eq. (6.102), and substituting z = 1 directly shows that Hzz (u, 1) = 2u3 /(1 − 4u)3/2 . Hence n n−3 −3/2 n−3 −3/2 = 2(−4) . Cn µ[2] (n) = [u ]Hzz (u, 1) = 2[u ](1 − 4u) n−3

6.6. APPLICATIONS IN PROBABILITY 2n n

1 Division of this expression by Cn = n+1 number of leaves in a binary tree to be

µ[2] (n) =

345 produces the second factorial moment of the

(n + 1)4 . 4(2n − 1)(2n − 3)

This allows us to find the second moment: E[L2n ] =

n(n + 1)(n2 + n − 4) 4(2n − 1)(2n − 3)

=⇒

V [Ln ] =

(n + 1)4 (2n − 1)2 (2n − 3)

and for large n we get V [Ln ] ≈ n/8, which tells a similar tale: in binary search trees, the variance of a distribution of leaf numbers tightly clumped near its mean value, though this variance is larger than (6.97) we found for the binary search statistics. Exercise 6.174 [3] (Number of function calls) How many times does Quicksort call itself when sorting an array of size n? While the required calculation is simple, the answer turns out to depend on details in the implementation that do not matter for other considerations. Consider the code given in page 229, which we reproduce here: 1. QS (A, i, j) 2. { if (i >= j) return; pivot (A, i, j, k); 3. QS (A, i, k − 1); 4. QS (A, k + 1, j); } 5.

The test in line 2 is reasonable: there is no point in doing anything with a single (or no) entry in the array, however, why not do this test before the recursive call? It makes no difference to the number of term comparisons made—or, for that matter, to the number of times pivot is executed—but here we are interested in the number of calls to QS itself, denoted by Vn . For each of the following varieties, compute the mean number of these calls and their variance: (a) As above, no test before any call, and hence V0 = 1. (b) A test is made to verify the array is not empty, but a call is performed even for a single entry (this variety probably lacks any justification, but compute the implications). Here V0 = 0, but V1 = 1. (c) We can remove the test in line 2 since the original size of the array A is presumably at least 2, and before each recursive call, in lines 4 and 5, we test: 1. QS (A, i, j) 2. { (a place holder) pivot (A, i, j, k); 3. 4. if (k > i + 1) QS (A, i, k − 1); 5. if ( j > k + 1) QS (A, k + 1, j); } In this case V0 = V1 = 0.


346

Exercise 6.175 (Analysis of Quickselect [104]) Consider again the analysis of Quickselect, as done in Exercise 5.125, except that this time you are asked to (i) use the tools of this section, which lead to a somewhat more pedestrian solution procedure than the one needed there; and (ii) calculate not only the expected values of Cn.k and Rn,k , but their variances as well. Although articles [74, 104] may be useful, they consider another type of a problem. Note: The use of a computer algebra program is essential for this exercise. Exercise 6.176 In Exercise 5.89 on page 234, we considered tree parameters as random variables over the set of binary search trees: D(n), F(n), and L(n) (see there for definitions). Since the set of binary search trees (BSTs) with n nodes is the same set we counted in §6.3, all these variables are well defined here, and satisfy the same recurrences. We now pose a 2n 1 different distributional assignment: each of the n+1 n trees of size n has the same probability. We denote by Dn , Fn , and Ln the corresponding parameters under the Catalan statistics, that is, Ln is the number of leaves, Fn is the number of nodes in the tree that are full, and Dn denote the internal path length. (a) [2] Show for all trees of sizes 3 and 4 how the distribution generated by the definition in Exercise 5.89, page 234, differs from the current uniformity assumption. (b) [3+] Write recurrences for the expectations of the variables with this distributional assumption and solve them. You should obtain the following formulas:

E[Ln ] =

n(n + 1) ; 2(2n − 1)

E[Fn ] =

(n − 1)(n − 2) ; 2(2n − 1)

E[Dn ] = 4n

n+1 − (3n + 1). 2n

(6.103)

n

(c) [3+] Determine the extremal values (minima and maxima) of the parameters Ln , Dn , and Fn , and the likelihood these that values are realized. (d) [3] Repeat part (c) for the distribution over BSTs. We deferred this part from Exercise 5.89 because its solution is better suited for the tools we have demonstrated in this chapter. Note: It is a curiosity that the generating functions you derive in this part are trigonometric functions! Exercise 6.177 [4] This exercise complements Exercises 5.89 (page 234) and 6.176 by considering the variance of the parameter D(n) discussed there, using the approach displayed in Example 6.173 to compute the moments of L(n). Note that since the difference between F(n) and L(n) is fixed, at 1, it is considered no further. (a) Compute the variance of D(n) under the BST distributional assumptions. Adapt this result to compute the variance of the number of comparisons in Quicksort that uses n + b comparisons to partition an array of size n, but no threshold larger than 2. (b) Compute the variance of Dn in the model of Catalan statistics.

6.7. THE LAGRANGE INVERSION THEOREM

347

6.7 The Lagrange Inversion Theorem The Lagrange Inversion Theorem (LIT), also called Lagrange Expansion Theorem, is useful in extracting coefficients from generating functions for which we have an equation, but not an explicit solution. A detailed presentation of the expansion and its evolution is available in Whittaker and Watson [135], Goulden and Jackson [55], and Henrici [62]. In this section, we concentrate our attention on ideas and present applications of the material, with few proofs. The Lagrange Inversion Theorem (LIT) originates from the problem of finding power series for inverse functions. More precisely, let g(z) be an analytic function in a neighborhood of the origin where it has a zero. Assuming, for simplicity, that g′ (0) 6= 0, its corresponding Maclaurin series becomes g(z) = a1 z + a2 z2 + a3 z3 + · · · =

a1 = g′ (0) 6= 0.

∑ ak zk , k>1

(6.104)

While the implicit function theorem 12.83, page 694, guarantees that in a sufficiently small neighborhood of the origin there exists an inverse function, g[−1] (w), it stays mute on definition of this function. Fortunately, when g(z) has a power series expansion (6.104), the inverse function has a similar property: z = g[−1] (w) = b1 w + b2 w2 + · · · =

∑ bk wk .

(6.105)

k>1

This series defines the (unique) inverse function z = ϕ (w) = g[−1] (w) of g(z). Moreover, ϕ (w) is an analytic function inside a circle |w| < ε , where ε > 0. Substitution of Eq. (6.104) into Eq. (6.105) leads to z = b1

k

∑ ak z k

!

k

∑ ak z

+ b2

k

!2

+ b3

k

∑ ak z k

!3

+ ··· .

Equating terms with like-powers of z, we find recurrence formulas for the coefficients bk , which can be successively resolved. However, we can find coefficients bk (k = 1, 2, . . .) in Eq. (6.105) directly without solving these recurrences, and there are two options to do this. The first approach uses the Cauchy formula, as given in Eq. (12.23) on page 679: bn =

1 ϕ (n) (0) = n! 2π i

I

δ

ϕ (ζ ) dζ , ζ n+1

n = 1, 2, . . . , H

where δ is a closed loop around the origin in the ζ -domain. We use the notation γ to denote integration along a closed curve γ in the complex plane. The critical step is a change of variable in the integration: setting ζ = g(z) in the latter integral, we obtain bn =

1 2π i

I

γ

z g′ (z) dz. gn+1 (z)

(6.106)


348

Now we use the integration by parts formula 1 2π i

I

u(z)v′ (z) dz = −

γ

1 2π i

I

γ

u′ (z)v(z) dz

for the closed contour γ . If we note that 1 d g′ (z) =− gn+1 (z) n dz

1 g(z)

n

,

we see that using the Cauchy formula (12.23), the relation (6.106) can now be rewritten in the form n I I 1 d n−1 z n z 1 dz 1 1 . dz = = (6.107) bn = 2π in γ gn (z) 2π in γ zn g(z) n! dzn−1 gn (z) z=0

In other words, bn is the coefficient of zn−1 in the power series expansion of the function (z/g(z))n , that is, z n 1 , n = 1, 2, 3, . . . . (6.108) bn = [zn−1 ] n g(z) We can derive this formula in another way by implementing the idea of Lagrange13 (1770), which is based on the following extension of the Cauchy formula (12.23).

Let γ be a closed Jordan contour14 such that F(z) and ψ (z) are analytic functions within and on γ . We assume for simplicity that ψ (z) has only one zero at z = a inside γ of multiplicity 1, that is, ψ (a) = 0 and ψ ′ (a) 6= 0. Then

1 2π i

I

γ

F(z)

ψz′ (z) dz = F(a). ψ (z)

(6.109)

Intuitively, ψ (z) is approximately (z − a)ψ ′ (a) inside γ , taken in counterclockwise direction. The Rouché theorem (page 682) guarantees that the functions ψ (z) and (z − a)ψ ′ (z) have only one zero at z = a if γ is a closed loop inside a circle of a small radius. Therefore, the ratio ψz′ (z)/ψ (z) is equal to (z − a)−1 + Φ(z), where Φ(z) is an analytic function within and on the closed contour γ . Actually (z − a)−1 + Φ(z) is the Laurent’s representation around z = a of the ratio ψz′ (z)/ψ (z). And the result follows from the Cauchy formula (12.23): 1 2π i

I

γ

F(z)

ψz′ (z) 1 dz = ψ (z) 2π i

I

γ

1 F(z) dz + z−a 2π i

I

γ

F(z)Φ(z) dz =

1 2π i

I

γ

F(z) dz = F(a) z−a

since the integral over a closed curve of any analytic function is zero—the function F(z)Φ(z) is analytic inside γ . 13 Joseph-Louis Lagrange, born Giuseppe Lodovico Lagrangia (1736 – 1813) was an Italian mathematician and astronomer. 14 See Definition 12.38 on page 678.


349

We choose a small circle, γ , around the origin so that the function g(z) has no zeroes other than z = a in the interior of γ nor on γ itself. Let m denote the minimum of |g(z)| as z ranges over γ . Then |w/g(z)| < |w|/m < 1 (or |w| < m) for z ∈ γ . Then from the Rouché theorem, it def follows that the ψ (z, w) = g(z)− w possesses only one zero inside γ and therefore Eq. (6.109) yields I I 1 ζ ∂ ψ /∂ ζ ζ g′ (ζ ) 1 dζ = dζ . z= 2π i γ ψ (ζ , w) 2π i γ g(ζ ) − w The denominator can be represented as a uniformly convergent series on γ : 1 1 wk 1 = = ∑ k+1 . g(ζ ) − w g(ζ ) 1 − w/g(ζ ) k>0 g (ζ ) Hence 1 z= 2π i

I

γ

′

ζ g (ζ )

∑ k>0

wk dζ = ∑ gk+1 (ζ ) k>0

1 2π i

I

γ

ζ g′ (ζ ) dζ gk+1 (ζ )

wk .

The first term in this series is zero, 1 2π i

I

γ

ζ g′ (ζ ) dζ = 0 g(ζ )

since the integrand is an analytic function inside γ and we obtain the same formula (6.107) for bn . The integrand is a product of two analytic functions: g′ (ζ ) and ζ /g(ζ ). The latter one is a reciprocal of the power series g(ζ ) = a1 + a2 ζ + a3 ζ 2 + · · · ζ

(a1 = g′ (0) 6= 0)

and therefore is an analytic function. Example 6.178 (The Lambert function) Let us consider the equation w = z e−z , which has numerous applications (see, for example, §8.1.2). Its solution subject to the condition w(0) = 0 is usually referred to as the Lambert function (see §12.6) or the omega function. However, we can represent z as a power series z = b1 w + b2 w2 + b3 w3 + · · · in |w| < e−1 , with coefficients that are given by Eq. (6.107): I nn−1 1 d n−1 nζ 1 dζ 1 = e · . = bn = 2π in γ ζ n e−mζ n (n − 1)! d ζ n−1 n! ζ =0 n z = (−1)n−1 . Exercise 6.179 [1] Use the LIT to prove that [zn−1 ] z e −1 Hint: Use w = g(z) = ez − 1 in Lagrange’s expansion. Now we are ready to generalize Lagrange’s formula (6.107). Let F(z) be any analytic function on and inside the closed curve γ . Using ψ (z, w) = g(z) − w in Eq. (6.109), we obtain F(z) =

1 2π i

I

γ

F(ζ )

1 ∂ ψ /∂ ζ dζ = ψ (ζ , w) 2π i

I

γ

F(ζ )g′ (ζ ) dζ g(ζ ) − w


350

= =

1 1 F(ζ )g′ (ζ ) · dζ 2π i γ g(ζ ) 1 − w/g(ζ ) I wk 1 1 dζ = ∑ F(ζ ) g′ (ζ ) ∑ k+1 2π i γ (ζ ) k>0 g k>0 2π i I

w 1

where w = g(z) and for n = 1, 2, . . ., bn = =

1 F(ζ )g′ (ζ ) F ′ (ζ ) 1 d dζ ζ = (n = 1, 2, . . .) 2π i γ gn+1 (ζ ) 2π in γ gn (ζ ) n−1 ′ z n F (z)zn d 1 n−1 ′ 1 . = [z ] F (z) n! dzn−1 gn (z) n g(z) z=0 I

I

(6.110)

Example 6.180 Let us again consider the equation w = ze−z from Example 6.178. The power series of eα z (α is a constant) in terms of w is

eα z = 1 + w +

α (2 + α )w2 α (3 + α )2 w3 + + · · · = 1 + ∑ bn wn , 2! 3! n>1

where 1 bn = [zn−1 ] α eα z n

z ze−z

n

=

1 n−1 (n + α )n−1 [z ] α e(n+α )z = α . n n!

Exercise 6.181 [2] Show Abel’s identity (see Eq. (2.84)) n n αβ ∑ (α + k)k−1 (β + n − k)n−k−1 = (α + β )(α + β + n)n−1 . k k=0

Hint: Apply the result from Example 6.180 to the functions eα z , and use the binomial convolution rule.

eβ z ,

e(α +β )z = eα z · eβ z

Eqs. (6.107) – (6.110) contain the ratio of ζ and g(ζ ). To simplify the results, let us set g(z) to be z/φ (z). In other words, let ψ (z, w) = wφ (z) − z. The zero (if it is unique) of the function ψ defines the function w = g(z) = z/φ (z) and  F(z) = F(wφ (z)) = F(0) + ∑n>1 bn wn , o n n−1 n H (6.111) d ′ (z)φ n (z)) bn = 1n [zn−1 ] F ′ (z)φ n (z) = 2π1in γ F ′ (ζ ) φ ζ(nζ ) d ζ = n!1 dz (F . n−1 z=0

In particular, if F(z) = z we have  z = wφ (z) = ∑n>1 bn wn , bn = 1n [zn−1 ] φ n (z) =

1 2π in

H φ n (ζ ) γ

ζn

dζ =

1 n!

n

d n−1 dzn−1

(φ n (z))

o

z=0

.

(6.112)

6.7. THE LAGRANGE INVERSION THEOREM Exercise 6.182 [2] Let w = tan 2z . Find (a) sin z,

351 (b) cos z in terms of w.

This series (6.111) is known as Lagrange’s series. These results can easily be extended to the case when the function w = g(z) has a zero at any point a and the following theorem serves these needs.

Theorem 6.183 (The Lagrange inversion theorem — LIT) Let F(z) and φ (z) be functions that are analytic on and inside a closed Jordan contour γ , which surrounds a point a in the z-plane; let m(a, γ ) denote the minimum of the function φz−a (z) as z ranges over the closed curve γ . It is an immediate result of Rouché’s Theorem 12.58 that the equation ζ = a + wφ (ζ ), (6.113) regarded as an equation in ζ , has exactly one root ζ (w, a) in the interior of γ . Furthermore, the function F(z) can be expanded at ζ as a power series in w: wn d n−1 ′ n [F (z)φ (z)] (6.114) , |w| < m(a, γ ). F(ζ (w)) = F(a) + ∑ dzn−1 z=a n>1 n! A restatement of Eq. (6.114) in terms of coefficients reads as follows 1 [wn ]F(ζ (w, a)) = [(z − a)n−1 ]{F ′ (z)φ n (z)}, n

n > 1.

(6.115)

def

The further notation f (z) = F ′ (z)φ (z) leads, on differentiating Eq. (6.114), to the suggestive relation f (ζ (w)) = ∑ wn [xn ]{ f (x)φ n (x)}, (6.116) |w| < m(a, γ ). 1 − wφ ′ (ζ (w)) n>0 It is convenient to regard F, f , and φ as being defined in two different complex planes, with z and a located in the z-plane, and ζ in the w-plane. P ROOF : We need to show only the validity of Eq. (6.116). Differentiation of both sides of Eq. (6.114) with respect to w yields n−1 wn−1 d d ′ n F(ζ (w)) = ∑ [F (z)φ (z)] dw dzn−1 z=a n>1 (n − 1)! n n d w [F ′ (z)φ n+1 (z)] = ∑ n n! dz z=a n>0 On the other hand, dζ 1 d F(ζ (w)) = F ′ (ζ (w)) · = F ′ (ζ (w)) · dw . dw dw dζ The derivative of w =

ζ −a φ (ζ )

with respect to ζ is

dw φ (ζ ) − (ζ − a)φ ′ (ζ ) φ (ζ ) − wφ (ζ ) φ ′ (ζ ) 1 − w φ ′ (ζ ) = = . = dζ φ 2 (ζ ) φ 2 (ζ ) φ (ζ )


352

Therefore, we have wn F ′ (ζ (w)) φ (ζ ) d = F(ζ (w)) = ∑ dw 1 − w φ ′ (ζ ) n>0 n!

dn ′ n [F (z)φ (z) φ (z)] dzn z=a

and Eq. (6.116) follows. Corollary 6.184 Taking φ (ζ ) = eζ , w = ζ e−ζ , and a = 0 in Eq. (6.114), we obtain wn d n−1 ′ f (z) enz . f (ζ ) = f (0) + ∑ n! dz n>1 z=0

Example 6.185 The nth central trinomial number is defined as the coefficient of zn in the expansion of (1 + z + z2 )n , that is, tn = [zn ](1 + z + z2 )n . Using the binomial theorem, Eq. (2.52) on page 56, we obtain n n n − i j 2i (1 + z + z2 )n = ∑ (1 + z)n−i z2i = ∑ ∑ zz . j i>0 i i>0 j>0 i Hence, the nth central trinomial coefficient is expressed via the following sum: n n−i n 2j tn = [zn ](1 + z + z2 )n = ∑ =∑ . i i 2 j j i>0 j>0 This is adequate for computing by hand the first few central trinomial numbers, and a computer algebra system, of course, can go much further; here are first 26 elements of the sequence: 1, 1, 3, 7, 19, 51, 141, 393, 1107, 3139, 8953, 25653, 73789, 212941, 616227, 1787607, 5196627, 15134931, 44152809, 128996853, 377379369, 1105350729, 3241135527, 9513228123, 27948336381, 82176836301, 241813226151. We use the LIT in the representation (6.116) to find the OGF of these numbers. In the statement of Theorem 6.183, we pick a = 0 and the functions f (z) = 1 and φ (z) = 1 + z + z2 . The righthand side of Eq. (6.116) is the desired OGF of the central trinomial numbers. To evaluate the of theequation ζ = w(1 + ζ + ζ 2), left-hand side, we use the fact that ζ (w) is the solution p p hence ζ (w) = (1/2w) 1 − w − (w − 1)2 − 4w2 = (1/2w) 1 − w − (1 + w)(1 − 3w) , having chosen the solution to be regular at the origin. The left hand side of Eq. (6.116) is 1 . 1/[1 − w(2ζ (w) + 1)] = √ (1+w)(1−3w)

While the sum for the tn does not lead to a convenient closed form, it is a starting point for asymptotic information. At a glance, the leading term of tn is close p to 3n , and with a little more work, asked for in Exercise 11.33, page 641), we find tn ∼ 3n /2 nπ /3. This is a good approximation even for quite low n, but it is a slight over-estimate.

Example 6.186 (Kepler’s Equation) Kepler’s equation

x − a = ε sin x,

(6.117)


353

occurs in the theory of the elliptic orbit of a planet. In this equation, x denotes the so-called eccentric anomaly, a the mean anomaly, and ε the eccentricity of the orbit. From Eq. (6.115), we immediately get the Taylor series representation of x in terms of ε : ∞

x = a+ ∑

n=1

(sinn a)(n−1) n ε , n!

where (sinn a)(n−1) is (n − 1)-th derivative of the function sinn a with respect to a. Example 6.187 Let z is such root of the equation z=ζ+

w 2 (z − 1) 2

z = ζ + wφ (z),

or

1 φ (z) = (z2 − 1), 2

which has value ζ when w = 0. Since this is a quadratic equation with respect to z, we can find its exact value to be p 1 − 1 − 2ζ w + w2 . z= w From Eq. (6.114), it follows that n ∞ wn d n−1 z2 − 1 z = ζ+∑ dzn−1 2 z=ζ n=1 n! ∞

= ζ+∑

n=1

wn d n−1 (ζ 2 − 1)n . 2n n! d ζ n−1

Now we differentiate both sides to obtain

where Pn (ζ ) =

1 dn 2n n! d ζ n

∞ 1 dz = 1 + ∑ wn Pn (ζ ), =p dζ 1 − 2ζ w + w2 n=1

(ζ 2 − 1)n is so called the Legendre polynomial of degree n.

Example 6.188 Consider the following equation for an OGF q(z): q(z) = 1 + zqr (z), where r is a real number. This equation is a particular case of Eq. (6.113), where a = 1 and φ (z) = zr , which is regular around z = 1, and satisfies the inequality |wzr | < |z − 1| < ρ for z close to 1 and w near its origin. Specifically, when |z − 1| < ρ , the condition holds for all |w| < η , where η = ρ (1 + ρ )−r . Substituting in Eq. (6.115) with F(z) = z, we find 1 1 nr 1 nr n−1 nr n−1 . [z ]q(z) = [(x − 1) ]x = [(x − 1) ] (1 + (x − 1)) = n n n n−1 n

Relation (6.118) produces the summation formula nr + 1 n 1 1 nr n z =∑ z = q(z), ∑ n n>0 nr + 1 n>0 n n − 1

q(z) = 1 + zqr (z).

(6.118)

(6.119)

354


For instance, in §6.3, we found that the ordinary generating functions for binary and ternary trees satisfy algebraic equations b(z) = 1+ z b2 (z) and b3 (z) = 1+ z b33 (z), respectively. Using Eq. (6.114) with F(z) = z, φ (z) = z2 , we obtain wn wn 2n wn d n−1 2n = 1+ ∑ z 2n (2n− 1) · · · (n+ 2) = 1+ ∑ . b(w) = 1+ ∑ dzn−1 n z=1 n>1 n! n>1 n + 1 n>1 n! Similarly, for ternary trees, we use F(z) = z, φ (z) = z3 and get 3n wn wn d n−1 3n z . = 1+ ∑ b(w) = 1 + ∑ n−1 dz n z=1 n>1 2n + 1 n>1 n! Exercise 6.189 [3] Use the discussion in Example 6.188 to establish the following summation formulas: nr + s szn (6.120) ∑ n nr + s = qs (z), q(z) = 1 + zqr (z). n>0 nr + s n qs+1 (z) , q(z) = 1 + zqr (z). (6.121) z = ∑ n r − (r − 1)q(z) n>0 The Lagrange expansion can be generalized in at least three different directions. First, when the function ψ (z, w) in Eq. (6.109) has more than one zero, second, when the function F(z) is a function of z in a ring-shaped region, and third, when you want to extract arbitrary coefficients of a power expansion. We present the answer for the last question in the following exercise.

Exercise 6.190 [1+] Using Eq. (6.115), establish the useful result u n m n−m m [−1] n . ] z = g (g(z)) =⇒ [z ] (g(z)) = [u n g(u)

(6.122)

Note: The result stated in Eq. (6.122) is known as the Lagrange-Bürmann inversion theorem. Exercise 6.191 [1+] (J. Lambert) Find coefficients in the Maclaurin series expansion of the function q(z) that is a solution to the equation q(z) = z + qr (z), r 6= 1.

Exercise 6.192 [3] (L. Euler) For any positive integers α and β , find a power series expansion in v of the solution of the following equation (b) ln x = v xα (α = β ). (a) xα − xβ = (α − β )v xα +β (α 6= β ),

Chapter 7

Enumeration with Generating Functions In this chapter, we will again take up the topic of enumeration that was considered in Chapter 2. Using generating functions (GFs), we are going to broaden our horizons by embracing enumeration operators on sets. This is not just a new application of GFs, it is a different point of view. We now associate a GF not with a sequence but with a set. Later it will be shown how the two approaches complement each other. This view of generating functions, like much of the mathematics we use in the analysis of algorithms, goes back to Pierre Simon de Laplace1 . Much of this theory was developed by Leonhard Euler, and had a rebirth in the 20th century with the appearance of the Symbolic Method, also called Combinatorial Calculus, since it unifies these branches of mathematics.

The topics of enumeration, combinatorics, and number theory were called higher arithmetic in previous periods, and were classified as pure mathematics. The need to analyze algorithms transformed them to engineering mathematics since nearly all such analyses are reduced at some point to an enumeration of features in combinatorial structures (such as strings, graphs, or Turing machine traces, etc.). Generating functions provide the theoretical apparatus for this field. We present and illustrate their uses in various problems on enumeration. Our exposition, and much of its content, have been inspired by the work of Philippe Flajolet and his students.

7.1 Definition of Enumerators Let S be a countable set of elements, such as numbers, words, graphs, coins, computer programs, other sets, etc. When is such a set acceptable (or well-defined)? We already encountered this question, on page 80, and answered that a proper set definition provides an 1 Pierre

Simon de Laplace (1749–1827) was one of the greatest French mathematicians.

355

CHAPTER 7. ENUMERATION WITH GENERATING FUNCTIONS

356

unambiguous answer to the membership question, that is, whether an element belongs to the set, or not. We use the notation a ∈ S to signify that a is an element in the set S, and a ∈ /S otherwise. How can we determine that a set is acceptable? Technically, this is phrased as a requirement of the set to have a characteristic function. Such a function, defined on the universe of all elements potentially in the set, specifies the membership in the set. For example, it may have the value one for the elements in the set and zero otherwise. In nearly all practical situations, the membership problem is simple. Difficulties sometimes arise with sets of sets, or with sets that are defined by self-referential means. Here is a famous example: a group of men consists of two sets: those who shave themselves, and those who are shaved by the barber (a man). To which set does the barber belong? This is in fact a case of poor terminology. But sometimes the difficulty is genuine, as dealing with sets that contain themselves. While this case is probably never relevant to the kind of enumerations we do, it serves as a good warning about the care needed in definitions.

Definition 7.1 (weight) A weight function w of a set S assigns a non-negative integer to every element of the set. Formally this is expressed by w : S −→ N. ⊳ While there is no formal reason for restriction of the range of the weight function to be the set of nonnegative integers only, the intended application, which uses these weights as powers, suggests why we never use a different set of values. It should be noted that weight functions with non-negative real values are used in many applications. For instance, the fuzzy set theory considers a characteristic function that defines membership as a function with values in the interval [0, 1] rather than in binary set {0, 1}, as we do. This theory was invented by Lotfi A. Zadeh (1965) and it is widely used, for example, in digital image processing for spacial filtering. Definition 7.2 (enumerator) Let S be a set for which a weight function w : S −→ N is defined. Let S(n) = {a ∈ S : w(a) = n} be the subset of all elements from S having the weight n, n > 0. If all these subsets of S are finite, we call the set S admissible with respect to the weight w and define the enumerator of S as (7.1) ϕS (z) = ∑ zw(σ ) = ∑ ∑ zn . σ ∈S

n>0 σ ∈S(n)

Similarly, we define the exponential enumerator as

ϕˆ S (z) =

∑

σ ∈S

zw(σ ) =∑ w(σ )! n>0

∑

σ ∈S(n)

zn zn = ∑ |S(n) | . n! n>0 n!

⊳

(7.2)

In this definition, the letter z plays a double role: it is not just a mark for distinguishing elements within the set S according to their weight, but also a variable, which could be used for algebraic operations. Furthermore, every element of weight n is marked with zn .

7.1. DEFINITION OF ENUMERATORS

357

Thus, if S = {a, b, c, d}, and these four letters have the weights 0, 1, 4, and 1, respectively, we find that its enumerator is ϕS (z) = 1 + z + z4 + z = z4 + 2z + 1. This example shows what is evident in the formula: the term zn does not depend on any aspect of the element except its weight. Hence we can lump them all together and find that ϕS (z) = ∑ |S(n) |zn . n>0

Now we show how this definition is related to a previous view of generating functions. Let ar be the number of elements σ ∈ S that have weight r ∈ N. Then

ϕS (z) =

∑ zw(σ ) = ∑ ∑

σ ∈S

r>0 σ ∈S(r)

zr =

∑ ar zr

(7.3)

r>0

since ar is exactly the number of terms in the inner sum, for fixed r. This result, being the promised connection with the previous aspect of generating functions, makes ϕS (z) to be the ordinary generating function of the sequence of weight counts {ar }r>0 . Similarly, the exponential enumerator can be written as ϕˆ S (z) = ∑r>0 ar zr /r!. We shall find this variant to be suitable for enumerating labeled sets. Revisiting the finiteness restriction in Definition 7.2, it requires all of the ar to be finite, even for an infinite set. A set that does not satisfy this is inadmissible for handling with enumerators. The rightmost sums in Eqs. (7.2)–(7.3) would then be undefined. For finite sets, we have two immediate consequences of this definition, by Eqs. (7.1) to (7.3):

ϕS (1) = ϕS′ (1) =

∑ 1w(σ ) = |S|,

the number of elements in S, and

(7.4)

∑ r ar = w(S),

the total weight of S.

(7.5)

σ ∈S r>0

That is why we call ϕS (z) the enumerator of S. Example 7.3 Let A = {a1 , a2 , . . . , an } be a finite set, and let S be the power set of A (that is, the set of all subsets of A, including A itself and the empty set). Then σ ∈ S if and only if σ ⊆ A. Now we choose to define w(σ ) = |σ |, the number of elements from A, which are in σ . For n = 3 we find A = {a1 , a2 , a3 }, S = {∅, {a1 }, {a2 }, {a3 }, {a1 , a2 }, {a1 , a3 }, {a2 , a3 }, {a1 , a2 , a3 }}. In general, there are nr ways to choose subsets of size r, and we find n r z = (1 + z)n . ϕS (z) = ∑ r r

Substitution of z = 1 gives

ϕS (1) = 2n ,

ϕS′ (1) = n 2n−1 .

This motivates the term power set and the common notation for it: S = 2A .

358


Example 7.4 Let S be a set of coins: 3 pennies, 1 nickel, 5 dimes, and 7 quarters. The weight of a coin is chosen to be its monetary value in cents. Therefore

ϕS (z) = 3z + z5 + 5 z10 + 7 z25 , ϕS (1) = 3 + 1 + 5 + 7 = 16,

ϕS′ (1) = 3 + 1 × 5 + 5 × 10 + 7 × 25 = $2.33 .

Example 7.5 Let Σ = { a, b } be a two-letter alphabet. We look at the set S of all strings, also known as “words,” constructible from the alphabet Σ. For the weight of a word we make a natural choice—its letter count. With two elements of size 1, this weight function, ϕΣ (z), becomes 2z. Let x be a string made of a’s and b’s, then w(x) = |x| is the length of the word x. Therefore the subset counter, ar , is the number of such r-letter words, which is 2r . The corresponding enumerator is

ϕS (z) =

1

∑ 2r zr = 1 − 2z .

r>0

Note that the sum contains a term for r = 0; this is for the empty word, which has no letters, and is conventionally denoted by ε (epsilon). This is our only element here of zero weight.

As a function in the complex plane, (1 − 2z)−1 is a valid representation of ϕS (z), defined as the above power series expansion only for z in the circle |z| < 1/2. This series diverges for |z| > 1/2. And indeed, since 1 is outside of the convergence region, it is not surprising to see that the values ϕS (1) = −1 and ϕS′ (1) = 2 do not have the meaning we ascribed to them in Eqs. (7.4) and (7.5). If word lengths were limited to a finite n, the generating function becomes a polynomial: n

ϕS (z) =

∑ 2r zr =

r=0

1 − (2z)n+1 , 1 − 2z

and then

ϕS (1) = 2n+1 − 1.

For infinite sets, the size and total weight have no useful meaning. It is important to keep in mind that even though the enumerators are used as formal series, convergence issues do not arise here. We still need such sets to satisfy the finiteness, or the admissibility condition, that was given in Definition 7.2: for the enumerator of an infinite set A to be well-defined it is necessary that each subset A(n) = {a : a ∈ A, w(a) = n} is finite. As we shall see, it limits the weight functions that can be used, in certain situations. Example 7.6 This admissibility condition will rarely pose any difficulty for us. To show that it is not entirely fantastic—or meaningless—consider the countable set Q+ , the positive rational numbers, and define for every x in this set the natural weight w(x) = ⌊x⌋, the floor of x. Clearly, every such subset Q+ (n) , for n ∈ N, is unbounded, and this weight function cannot be used to form an enumerator. Note that since the set Q+ is denumerable, other weight functions that satisfy the admissibility condition can be crafted for it, though the obvious ones (that are derived from its usual enumeration) do not seem to be very useful.

7.2. SUM AND PRODUCT RULES

359

We expect that the purpose of this run of definitions is apparent to the alert reader, but we still would like to state it as a proposition, or better, as the Enumeration Principle: If a given function f (z) is an enumerator of a set S, then (

if f (z) is an ordinary enumerator; sn = |{σ ∈ S : w(σ ) = n}| , n!sn = n!|{σ ∈ S : w(σ ) = n}| , if f (z) is an exponential enumerator. (7.6) Behind the Principle is the expectation that the rules we describe further will enable us to obtain enumerators by construction (whereas we have so far calculated enumerators for a given set using Eq. (7.3) by summation over their known structural parameters, the ar there). The Principle details how we obtain the parameters just as we solved recurrences in §6.4. n

[z ] f (z) =

7.2 Sum and Product Rules We saw that much of the applications of generating functions (GFs) discussed in the previous chapter depended on the relations between manipulations of sequences and operations on their GFs. Here too, we will see that certain operations on sets correspond to operations on their enumerators. Specifically, we refer to operations that combine sets, such as a union, a power set, and a few others that are more elaborate. Often a set to be enumerated can be constructed from some simple building blocks that we call atom sets (classes) or simply atoms. For example, a language is built from letters and hence in this case the atom set is the alphabet. Another typical example gives a set of graphs that are built from atoms—vertices and edges. In this section and the next two, we use ordinary generating functions (OGF) as enumerators, and later we examine the role of exponential generating functions (EGFs). It will be seen that broadly speaking, OGFs enumerate unlabeled structures, whereas the EGFs deal with labeled varieties. Construction of enumerators are based on two basic rules that we present now. As you will no doubt notice, these rules are the GF-version of the counting rules we presented in §2.3.

7.2.1 The Sum Rule Let A and . B be two disjoint sets, that is, A ∩ B = ∅. We denote the union of such sets by the symbol ∪, rather that the usual2 ∪. Both sets have weight functions defined over them. Note: The sum rule does not require that the same weight function be defined for A and B. It does not even depend on the sets being of a similar nature. However, 2 It is a curious convention: there is nothing special about the operation, we highlight a property of the operands!

360


in meaningful applications, the sets would be similar, such as both are sets of words, or sets of graphs etc. . Theorem 7.7 Let C = A ∪ B be the union of the disjoint sets A and B, and let the weight of an element of C be the weight it had in the set it came from A or B. Then, the enumerator of C equals the sum of the enumerators of A and B. The proof is by a simple calculation:

ϕS (z) =

∑ zw(σ ) = ∑

σ ∈S

zw(σ ) +

σ ∈A

∑

zw(σ ) = ϕA (z) + ϕB (z).

(7.7)

σ ∈B

Exercise 7.8 [1] Explain why the requirement that A and B must be disjoint is necessary for the sum rule to hold. Hint: Use it to justify the second equality in Eq. (7.7). Example 7.9 Let A consist of all strings of the letter a of even length, from 0 to 2m for some fixed positive integer m. Similarly, let B be the set of all strings of the letter a of odd length, from 1 to 2m + 1 for the same m. Finally, both sets use the same natural weight function: the weight of a string is the number of letters in it. Hence m

ϕA (z) =

∑

z2k =

k=0

1 − z2m+2 ; 1 − z2

m

ϕB (z) =

∑ k=0

z2k+1 = z

1 − z2m+2 . 1 − z2

Clearly these two sets are disjoint, and adding the two enumerators together, we find

ϕA (z) + ϕB (z) = (1 + z)

1 − z2m+2 1 − z2m+2 1 − z2m+2 = (1 + z) = , 1 − z2 (1 − z)(1 + z) 1−z

which is exactly what we would expect for the enumerator of the set of strings that have all lengths from 0 to 2m + 1, with this weight function. Clearly the rule extends to an arbitrary collection of sets, so long as they are all pairwise disjoint.

7.2.2 The Product Rule We again consider two sets A and B with their weight functions and form the Cartesian product C = A × B; that is, the set of all the ordered pairs (a, b), where a ∈ A and b ∈ B.

As with the sum rule, neither the two sets have to be of the same type nor do their weight functions have to be related. As we shall see, in order to be able to construct the enumerator of the product, we may need to restrict the type of combinatorial operations that we represent with the Cartesian product: it must be possible to associate uniquely each result of the operation with an element of the Cartesian product. This requirement is called unique factorization (or decomposition). It means that each element σ ∈ C arises from a unique pair of elements a ∈ A and b ∈ B. We shall see that


361

in applications, this property appears natural and is usually satisfied without imposing any noticeable restrictions. This is never a difficulty when we keep the representation of the elements σ ∈ A × B explicit, as ordered pairs, but if we mix them, or, catenate them, we could run into difficulties. Example 7.10 Sometimes difficulties with this restriction of unique factorization arise because of the representation of the multiplied sets. The most common case occurs in string processing. If A and B are sets of strings of letters, it is very natural to use the Cartesian product A × B to represent the operation that creates the set of all strings formed by attaching each string in B to the tail of each one from A. Formally, we would write in that case A × B = {(a, b) | a ∈ A, b ∈ B}. Then we define a mapping, from pairs of strings to strings, ψ (a, b) = ab, for any such pair. The string ab is called the catenation (or concatenation) of the strings a and b. We extend the definition of the map ψ to sets of strings, such as A and B, and then C = ψ (A × B). When this mapping is invertible, the unique decomposition is satisfied. But this may not always be the case. Let A = {0, 00} and B = {1, 01}. The pairs (0, 01) and (00, 1) are two different elements from the Cartesian product A × B. But ψ maps these two pairs into the same string. We must say that the product rule in this case does not have the unique decomposition property. Clearly, when the strings from A and from B have no common letters, the mapping ψ is always invertible, but this is too harsh a requirement. For example, if A = {00, 010} and B = {01, 101}, the four similarly formed strings are all uniquely factorizable. Now we can state and prove: Theorem 7.11 Let C be the Cartesian product of the sets A and B, and define the weight of an element of C as the sum of the weights of the element pair. That is, when c = (a, b), then w(c) = wA (a) + wB (b), where the subscripts denote the provenance of the weights. When the product has the unique decomposition property, the enumerator of the set C equals the product of the enumerators of A and B. The proof is by straightforward calculation:

ϕC (z) =

∑ zw(c) = ∑

c∈C

c=(a,b)∈C

zw(a)+w(b) =

∑ zw(a) ∑ zw(b) = ϕA(z) · ϕB (z).

a∈A

(7.8)

b∈B

Exercise 7.12 [2] In the calculation above, where do we use the property of unique decomposition? Example 7.13 We look at the sets over the alphabet {0, 1}, which we used in Example 7.10. For A = { 00, 010 } and B = { 01, 101 }, their product is mapped into the following four elements: C = ψ (A × B) = { 0001, 00101, 01001, 010101 }. Note that, given any of these four strings, you can readily find out which unique pair of elements from A and B came together to form it. How does this observation fare in terms of enumerators if ϕA (z) = z2 + z3 and ϕB (z) happen to have the same value? We would expect ϕC (z) to be (z2 + z3 )2 = z4 + 2z5 + z6 , which indeed corresponds to the set above.

362


Now let us add the string 001 to the set A to obtain A1 = A∪{001} = {00, 010, 001}. Defining C1 as the mapped Cartesian product of A1 and B, we now find C1 = ψ (A1 × B) = {0001, 00101, 01001, 010101, 001101}.

(7.9)

The output consists of only five strings, since two elements of the Cartesian product are mapped into 00101: ψ (00, 101) = ψ (001, 01) = 00101. These pairs (00,101) and (001, 01), with the first element from A1 and the second from B, are mapped into only one string. Hence the product A1 × B, with this mapping, does not satisfy the unique decomposition property. Here is how this is represented by the enumerators: ϕA1 (z) = z2 + 2z3 and ϕB (z) = z2 + z3 . Multiplying them we find ϕA1 (z) × ϕB (z) = z4 + 3z5 + 2z6 , which is not quite what the right-hand side of Eq. (7.9) requires—it has the enumerator z4 + 2z5 + 2z6 .

Both the sum and product rules are generalized naturally to the addition or multiplication of any finite number of terms, so long as the disjointness and unique factorization requirements hold. With that accomplished, there is one further operation we are now in a position to present, as it combines the unlimited application of both the sum and product rules. It is the Kleene3 closure, often called just closure (or sequence). Let A be a set with the enumerator ϕA (z). We shall assume that the Cartesian product of A with itself, giving A2 ≡ A × A, is well defined (has the unique decomposition property). The enumerator of A2 is then ϕA2 (z). Now we do this repeatedly, obtaining the sets A3 , A4 , . . . . We also need to assume that they are all disjoint: A j ∩ Ak = ∅, when j 6= k. Note: The requirement about disjointness of powers is not trivial! The unique factorization property may be violated in the same manner as we saw in Example 7.13. For ordered sets of elements from powers of A, the disjointness is trivial. For instance, a set of pairs is always disjoint from a set of triplets. But even when each product is well defined, the mapping we use to represent or interpret our structures may not provide this property. For example, using strings and their catenation ψ , we get ψ (00, 00, 00) = ψ (000, 000) = 000000. It turns out that in many useful applications these requirements are natural, and they are obviously satisfied, as it will be shown later. Finally, we are ready to define the closure operation. Definition 7.14 (closure) Let A be a set with the enumerator ϕA (z). If the successive Cartesian products of A with itself have the unique decomposition property, and if the collection of sets created in this way, {Ak }k>0 , are all disjoint, we define the closure (or sequence) of A as the union of these sets, denoted by A∗ . Then its enumerator can be computed as follows: . [ 1 . (7.10) ϕA∗ (z) = ∑ ϕAk (z) = A∗ = Ak , 1 − ϕA (z) k>0 k>0 3 Stephen C. Kleene (1909–1994), a logician—also known as talented raconteur—who together with Alonzo Church, Kurt Gödel, and Alan Turing established much of the foundations of computability theory. He did much of his work at the University of Wisconsin, Madison, which now houses a library named after him.


363

The union in Eq. (7.10) also includes A0 , called the neutral set that consists of a single object of weight 0, which is usually interpreted as the empty set, ∅, and denoted by ε . Excluding . S A0 from the union (7.10), we get the set A+ = k>1 Ak . ⊳ Example 7.15 Let A be a very simple set, consisting of one element, a, and let its weight be the integer d. Hence ϕA (z) = zd . Multiplying A with itself, we have a single element, which we write not as a pair (a, a) but as a string, aa. Its weight is 2d, and going higher and further simply produces rather long but very monotonous strings, aaa, aaaa, . . .. We can think of A as a single-letter alphabet, and Ak as a k-long string created from this letter by catenation. It is reasonable to write the element of Ak as ak , instead of stringing the same letter along k times. Surely all these products satisfy the unique decomposition property—there is no room for ambiguity. They are all different, and hence disjoint. Note that in our interpretation, A0 consists of a0 , which must be a word with zero letters. This word is conventionally called the empty string, and is denoted by ε , so A0 = ε . Another term is null word. Putting it all together, we find A∗ =

. [

k>0

Ak = {ε , a, a2 , . . . }

=⇒

ϕA∗ (z) =

1 1 = . 1 − ϕA (z) 1 − zd

(7.11)

The last detail raises an important observation: A∗ is only well defined when |A(0) | = 0, meaning that A itself has no elements of zero weight. Otherwise we can get an unbounded number of elements in A∗ of any weight, which breaks the admissibility requirement for A∗ . We shall see that the closure is one of our most popular operations. In particular, it is the principal operation we use to create infinite sets from finite ones (another one is the multiset, presented in §7.4.4). Exercise 7.16 [2] Let A = {ab, aba} be a two-string set from the closure, {a, b}∗ . Is A∗ well defined? We are now ready to show a few counting problems in which we use our rediscovered tool: generating functions as enumerators. The approach is uniform: given a description of a set that we want to count—typically the number of elements in it of each possible weight—we proceed to find other sets, for which we have enumerators. Since we build that set from simpler sets having known their enumerators, we find the required enumerator using the admissible operations: disjoint union, Cartesian product, and their combination—the closure (sequence). At that point we can use the Enumeration Principle, which says that two sets that contain the same elements have the same size. While the approach is uniform, details vary. The crucial part in this approach is the derivation that two sets (the given one and the constructed one), have a bijection relation, which preserves the weight of the elements. Fortunately, this often turns out to be rather easy because of the construction process. It will soon become apparent that there is no algorithm to find such constructions, so one shall need to be guided by intuition, trial and error, and exposure to examples where the approach is seen to work.

364


7.3 Counting Compositions of Integers We consider the equation a1 + a2 + · · · + ak = n,

n ∈ N,

(7.12)

and want to count the non-negative integer solutions {ai ∈ N} of this equation. An ordered list (or sequence) h a1 , a2 , . . . , ak i of non-negative integers that satisfies Eq. (7.12) is called a composition of n. The integers in the list are called components of the composition. The terminology here is not uniform in literature. Some authors require all the components to be strictly positive: no zero components allowed. We call such compositions weak compositions. Note that we are talking about counting rather than finding the solutions. Normally, the latter is much harder to do. This type of equation, important in combinatorial optimization and in number theory, is an example of a Diophantine equation4 . Another view of this problem is of counting the ways to distribute n identical balls into k bins, which we visit in Chapter 8. A composition has graphical representation using the “balls-and-bars” model. For example, the composition 1 + 3 + 2 + 4 + 4 + 2 = 16 of 16 into 6 components can be written as •|• • •|••|• • ••|• • ••|••

7.3.1 Homogeneous Compositions An example of the solutions of Eq. (7.12) for k = 2 can be seen in the binomial theorem: n n j n− j n (a + b) = ∑ ab . j j=0 The total number of terms in this expansion is n + 1. This is the number of non-negative integer solutions of the equation n1 + n2 = n. Here n1 = j is the exponent of a and n2 = n − j is the exponent of b. We extend the example to three components, and want to determine, for example, the number of compositions of 5: a1 + a2 + a3 = 5. Here are all the 21 possible compositions of 5 in 3 components from N: 1+4+0 0+1+4 0+4+1

1+0+4 4+0+1 4+1+0

0+2+3 2+0+3 3+0+2

3+2+0 2+3+0 0+3+2

0+0+5 5+0+0 0+5+0

1+1+3 3+1+1 1+3+1

1+2+2 2+2+1 2+1+2

Note that the solution a1 = 1, a2 = 0, a3 = 4 is different from a1 = 4, a2 = 1, a3 = 0 even though the same three integers are being used. We allow selections with repetition, while keeping the size k = 3 fixed. If we count weak compositions (no zero components), only six survive. 4 Of Diophantus from Alexandria as an individual we have essentially no information except that he died at the age of 84.

7.3. COUNTING COMPOSITIONS OF INTEGERS

365

Do you see why the same approach provides the number of different k-ary multinomial coefficients we can write in the form a1 ,a2n,...,an ?

Example 7.17 A group of 12 students stopped by a store when going away from a final exam. The store offers five different kinds of headache remedies. How many different purchases are possible?

Solution. This problem is a dressy version of Eq. (7.12), asking about distributing 12 identical objects among 5 distinct bins. While we can do the counting by brute force (very tedious), and show that there are 1,820 ways to select, with repetition, 12 from a set of 5 distinct objects, we are going now to show a better way. Let n be a positive integer, and cn be the number of all k-compositions of n. We construct the set Ck of compositions of n with a fixed number of k components (k > 1). Its cardinality is denoted by cn,k = |Ck |. Hence cn = ∑k>1 cn,k .

The set Ck of compositions ha1 , a2 , . . . , ak i is formed by taking the k-fold Cartesian product of the set N. Formally, we write this as Ck |= Nk . The symbol |= (which means models) denotes a one-to-one correspondence. The bijection in this case is immediate and is exhibited in Eq. (7.12). If we use the natural weight function in the set Nk —the weight of an integer is its numerical value—then the product rule tells us that we should set the weight of such a list to be equal to the sum ∑i ai . Since we want to count how many different series of this form have the sum n, we only need to find the number of elements of weight n in the set Ck . And this is just the coefficient of zn in the enumerator of the set Ck . It only remains to calculate this enumerator.

1 . Let ϕN (z) be the enumerator of N, with the natural weight function: ϕN (z) = ∑ j>0 z j = 1−z k The product rule tells us that the k-fold Cartesian product of N, which we can denote by N , and which can be viewed as Nk = N {z· · · × N} , | ×N×

k factors

has the enumerator

ϕNk (z) = ϕNk (z) =

1 1−z

k

= (1 − z)−k .

(7.13)

Hence we find the desired answer by extracting coefficients from (1 − z)−k . Actual calculations require the application of the binomial theorem 2.42 and the binomial identity 2.64, which shows how to negate the upper parameter in a binomial coefficient: −k n+k−1 n −k n n (−1) = . (7.14) cn,k = [z ]ϕNk (z) = [z ](1 − z) = n n

This formula fits all particular requirements from the two examples we discussed at the beginning of this subsection. In this case, the bijection between solutions of Eq. (7.12) and the k-fold Cartesian product is manifest and needs no further argument. We even write them in the same way!

366


As a side affect, we can find the number of solutions of a1 + a2 + · · · + ak 6 n in nonnegative integers. Introducing a slack variable ak+1 , we reduce the problem under consideration to another one (a1 + a2 + · · · + ak + ak+1 = n) for which the answer is already known from Eq. (7.14). Therefore the number of solutions to a1 + a2 + · · · + ak 6 n is n n+k j+k−1 =∑ . (7.15) n j j=0 Exercise 7.18 [2] We have already considered compositions in Example 5.9 on page 204, and produced the relation (5.2). Verify that the result given in Eq. (7.14) satisfies this relation. Example 7.19 What is the number of compositions of n with up to r components, with n, r ∈ N? Let us denote by Cr (n) the number of compositions of a positive integer n into up to r components. There are two approaches to calculate Cr (n). The first one is based on the result obtained in Eq. (7.14) and we sum the cn,k for the required range of parameter k, using Eq. (A.18) with a shift of index: r n+k−1 n+r Cr (n) = ∑ = , r > 1, n > 1. (7.16) n r−1 k=1 Another way is to observe that the sets of solutions with different numbers of components are disjoint. Then we can take the union of these sets, and get the set of all Cartesian products . r of N with itself, from 1 to r. We denote this set by Sr . To get the enumerator of Sr = ∪k=1 Nk , we just add the enumerators, using the sum rule. Doing the calculation, we find: r

ϕSr (z) =

∑ k=1

1 1−z

k

=

1 1 1 − ( 1−z )r 1 1 = (1 − z)−r − , 1 1 − z 1 − 1−z z z

and the desired coefficient is 1 −r r+n 1 n −r n+1 −r n+1 n (1 − z) − = [z ](1−z) = (−1) = , Cr (n) = [z ]ϕSr (z) = [z ] z z n+1 r−1 as expected. Example 7.20 What happens if we try to compute the number of compositions of n with any number of components? Clearly this number is infinite, because an arbitrary number of ai ’s may be zero. How would this be seen in our machinery? If we proceed formally we find 1 1 1 1 1 −j n =− =⇒ [z ] − = −[zn ] = 0!? ϕS (z) = ∑ (1 − z) = 1 1 − z 1 − 1−z z z z j>1 since [zn ] 1z = 0, as z−1 cannot be developed as a power series in positive powers of z. But the question becomes tractable if, for example, we only consider ai > 1 (i.e., ai ∈ Z+ =

7.3. COUNTING COMPOSITIONS OF INTEGERS {1, 2, . . .}): z ϕI1 (z) = ∑ z = 1−z i>1 i

=⇒ ϕS (z) =

∑

j>1

z 1−z

367

j

=

1 1 1 . z = 1 − z 1 − 1−z 1 − 2z

1 = 2n . And then immediately, if surprisingly, [zn ] 1−2z

Exercise 7.21 [1+] Compute the probability of getting a total of 37 when you roll a (true) die 12 times. Exercise 7.22 [1+] In the casino of the Red Baron the chips used for betting have the values $3, $7, $13 and $31 only. In how many ways can you pile $100 or $1000 with these chips? Exercise 7.23 [2] Find the number of compositions of a positive integer n with summands greater than one. Exercise 7.24 [3] (Alladi & Hoggatt, 1974) Prove that the number of distinct compositions of a positive integer n limiting the components to 1’s and 2’s is the Fibonacci number Fn+1 (n > 1). Exercise 7.25 [3] (a) Derive a recurrence and from it the corresponding generating function for the number of compositions with any number of components limited to 1, 2, and 3. (b) Provide an enumerator-driven solution for part (a).

7.3.2 Inhomogeneous Compositions So far we considered compositions where the components all come from the same source, and hence have common enumerators. What happens when this is not the case? From one point of view nothing changes: each component has its enumerator, the enumerator of the set of compositions is the simple product of the k (say) component-enumerators, and extracting the coefficient of zn (say) leads us to the desired value. On the other hand, the non-uniformity means that often we shall find ourselves with relatively awkward calculations. Consider the following problem: in how many ways can the equation a1 + a2 + · · · + ak = n be solved when a1 is an even number and all the other components a j , for j > 1 are divisible by 3? The first component, a1 , belongs to the set A2 = {0, 2, 4, 6 . . . }

=⇒

ϕA2 (z) = ∑ z2i = i>0

1 ; 1 − z2

and for j > 1, the other components a j belong to the set A3 = {0, 3, 6, 9 . . . }

=⇒

ϕA3 (z) = ∑ z3i = i>0

1 . 1 − z3

Creating the set of the suitable compositions, we get S = A2 × Ak−1 3

=⇒ ϕS (z) =

1 1 · . 2 1 − z (1 − z3 )k−1


368

We can view this enumerator ϕS (z) as a convolution, and use extraction Rule 4 on page 288:

ϕS (z) =

1 1 , · 2 (1 − z3 )k−1 1 − z | {z } | {z } f (z)

hence [zn ]ϕS (z) = ∑ 1 · [zn−2i ](1 − z3 )−(k−1) . i>0

g(z)

r −k + 1 −k + 1 3 j (−1) 3 , Now we compute: [z ](1 − z ) = [z ] ∑ (−z ) = r | {z } j j>0 3 (−1) j z3 j n−2i −k + 1 n (−1) 3 . It where we equated r with 3 j. Therefore we have: [z ]ϕS (z) = ∑ n−2i r

3 −(k−1)

r

i>0

3

should not be surprising that this question does not have a closed-form result. However, the last expression can be better organized. For example, for even values of n, the last sum can ⌊n/6⌋ be rewritten as ∑r=0 2r+k−2 . 2r ⌊n/6⌋ Exercise 7.26 [2] Obtain an expression, similar to the last one we derived, ∑r=0 2r+k−2 , 2r for odd values of n.

Example 7.27 (Money changing problem) This is a classic counting example: find the number of ways to make change with coins of given denominations; it is due in this form to George Pólya5 . Let us start with a particular example. Suppose we have a stash of pennies, nickels, dimes, and quarters. In how many ways can we pay $3.99? This problem is equivalent to the number of integer solutions to the equation 1 × n1 + 5 × n2 + 10 × n3 + 25 × n4 = 399.

We adopt the approach used in the example that started this subsection, and consider solutions to the equation a1 + a2 + a3 + a4 = 399, where the ai count the contributions from sets with the enumerators (1 − z)−1 , (1 − z5 )−1 , (1 − z10 )−1 , and (1 − z25 )−1 , respectively. Hence the answer is 1 1 1 1 . [z399 ]ϕC (z) = [z399 ] 5 10 1 − z 1 − z 1 − z 1 − z25

The inhomogeneity means that usually we should not expect neat general closed answers. For a specific coefficient one can always call on MAPLE, which promptly produces the answer 9,692.

In the general case, a currency is an ordered set of positive integers 1 6 d1 < d2 < . . . < dm , called denominations. If we inquire about the number of payouts with respect to the given currency, we need to determine the coefficient [zn ]ϕ (z), where ϕ (z) is the corresponding enumerator 1 1 1 · ··· . ϕ (z) = d d 1 2 1−z 1−z 1 − zdm 5 Famous mathematician, George P´ olya was born in 1887 in Budapest, Hungary and died in 1985 in California.


369

Exercise 7.28 [2] Write the enumerator for counting ways of using postage stamps with face value of 2, 5, 10 and 39 cents. In how many ways can we apply a postage of 95 cents to a letter?

7.3.3 Compositions with Restrictions In this subsection, we consider again Eq. (7.12) but with some restrictions on components that come in two flavors. They are restricted by the need to satisfy inter-component constraints, and §7.3.2 had some of this. Another restriction puts some limits on the components. We only deal with the latter kind here. First, consider the k-compositions of n ∈ N with uniformly bounded components: they can take values in the range [1..m] only. We denote this number by W (n, k, m), and write its enumerator, Φ(k, m; z) =

∑ W (n, k, m)zn =

n>0

z + z2 + · · · + zm

k

= zk

(1 − zm )k . (1 − z)k

(7.17)

The numbers W (n, k, m) appear in Example 8.18 as the number of ways to get a total of n by rolling k identical m-faced fair dice. Eq. (7.17) provides the desired coefficients explicitly, but not in closed form: 1 − zm k n = [zn−k ](1 − zm )k (1 − z)−k W (n, k, m) = [z ] z 1−z ⌊(n−k)/m⌋ k n − mr − 1 = ∑ (−1)r . (7.18) r k − 1 r=0 Now we are able to generalize the problem considered in Exercise 4.118 on page 176 and find the distribution of the convolution Wk = X1 + X2 + · · · + Xk of k m-sided fair dice: 1 Pr[Wk = n] = k m

⌊ n−k m ⌋

∑

j=0

k n − jm − 1 (−1) j n − jm − k j

for k 6 n 6 km.

(7.19)

We define the conditional probability function p(n | k) = Pr[Wk = n] that determine the probability of obtaining a sum of n given that we toss k dice (each has m sides). Its numerator, Eq. (7.18), gives the number of ways to throw a total of n when tossing k fair m-sided dice, and this is the number of weak compositions of a number n = a1 + a2 + · · · + ak where 1 6 ai 6 m for i = 1, 2, . . . , k. We will see in §8.2.2 that W (n, k, m) counts the number of ways to place n distinguishable balls into k bins so that each bin receives at least 1, but no more than m balls. There is nothing formally wrong with this expression (7.18), but it is not very transparent. As a source of numerical values it is truly distinguished, in a horrible way. This is often the case


370

with combinatorial sums of terms with alternating signs. As an example of the behavior of this sum, consider evaluating it for k = 20, n = 200, m = 10, which should clearly produce the value 1. However, on the way, the partial sum after 4 terms (out of 19), reaches values as large as 4.4 × 1027 , which would make a double-precision evaluation lose all significance! Can we do better? Yes, in more ways than one. Well, in two. The idea is to avoid the complete summation in Eq. (7.17), and split the multinomial on the last ‘+’ symbol: W (n, k.m) = [zn ](z + z2 + · · · + zm )k = [zn ][(z + z2 + · · · + zm−1 ) + zm ]k k n =∑ [z ](z + z2 + · · · + zm−1 )i zm(k−i) . i i This sum can now be reduced by using the powers of the bare zm to the following expression: k n−m(k−i) k 2 m−1 i W (n, k, m) = ∑ [z ](z + z + · · · + z ) =∑ W (n − m(k − i), i, m − 1), i i i i a full-history recurrence that is initialized by W (n, i, 1) = 1, 1 6 n 6 i 6 k, and zero otherwise. A second, even simpler, recurrence can be produced, is driven by k. We get it by peeling off one power from k: W (n, k, m) = [zn ](z + z2 + · · · + zm )k = [zn ](z + z2 + · · · + zm ) × (z + z2 + · · · + zm )k−1 ; now going over the terms in the first factor we reduce it to W (n, k, m) =

m

m

j=1

j=1

∑ [z j ](z + z2 + · · · + zm )[zn− j ](z + z2 + · · · + zm )k−1 =

∑ W (n − j, k − 1, m),

a recurrence initialized by W (n, 1, m) = 1, for 1 6 n 6 m, and zero for larger n. The array W (n, k, m) can be calculated for some particular cases. For example, W (n + k, k, 2) is the binomial coefficient nk and we met W (2n, n, 3) in Example 6.185 on page 352 as the central trinomial numbers.

Example 7.29 (Convolution of uniform variates) Let X be a random variable uniformly distributed in the interval [0, 1], that is, Pr[X 6 t] = t for any t ∈ [0, 1]. Suppose we pick up k numbers from the unit interval at random and let W = X1 + X2 + · · · + Xk be their sum. Our goal is to prove that the probability distribution of W is  1 ⌊t⌋ j k k   k! ∑ j=0 (−1) j (t − j) , if 0 < t 6 k, (7.20) Pr[Wk 6 t] = 1, if t > k,   0, otherwise; based on discrete approximation. Suppose we have a balanced m-sided die whose faces are labeled 1/m, 1/2m, . . . , 1. Let Ym be the corresponding uniform random variable, that is, Pr[Ym = j/m] = 1/n for j = 1, 2, . . . , m. Rolling such a die is a discrete approximation to


371

choosing a number at random from the unit interval, and the sum of k rolls approximates Wk . The probability distribution of rolling k such dice is known from Eq. (7.18) and by taking the limit we shall derive the desired expression (7.20). Let Um,k be the convolution of k dice with m faces, namely, the sum of k rolls of m-sided fair die. Its cumulative distribution is available from Eq. (7.18) (where we need divide the W (n, k, m) by mk ), and we have 1 Pr[Um,k 6 t] = k m

j

⌊mt⌋−k m

∑

j=0

k

k ⌊mt⌋ − jm (−1) . j k j

So we need to find the limit lim Pr[Um,k 6 t] = Pr[Wk 6 t].

m→∞

Since

⌊mt⌋ k ⌊mt⌋ ⌊mt⌋ − k = lim − = ⌊t⌋, = lim lim m→∞ m→∞ m m→∞ m m m

it is sufficient to show that

(t − j)k 1 ⌊mt⌋ − jm . = lim k m→∞ m k! k We rewrite the binomial coefficient as the product of k terms: (⌊mt⌋ − jm)k k! ⌊mt⌋ − jm = mk k mk ⌊mt⌋ − jm − k + 1 ⌊mt⌋ − jm ⌊mt⌋ − jm − 1 · ··· = m m m ⌊mt⌋ 1 k−1 ⌊mt⌋ ⌊mt⌋ −j − j− − j− × ... × . = m m m m m Each term in the right-hand side of the latter expression approaches ⌊t⌋ − j and we obtain the desired result (7.20). Exercise 7.30 [3] Let Wn be the set of all n-letter words that can be formed from a two-letter alphabet (say, heads and tails or success and failure). Define a run of the letter j in the word w to be a string of consecutive letter j’s and let ρ j to be the length of the longest run of the letter j in the word w. The maximum of each ρ j is denoted by Mn . For any fixed integer r (r > 1), let ai be the number of words from Wi having maximum run less than or equal to r.

Show that the numbers ai satisfy the recurrence of order r: an = an−1 + an−2 + · · · + an−r

for n > r + 1;

an = 2n

for ∀n 6 r.

Solve the recurrence by extracting coefficients from the corrsponding ordinary generating function.


372

7.3.4 Heterogeneous Components Now we look at compositions where the components are non-uniformly bounded. First, consider the enumerator for the number of compositions in the equation (7.12) with lower bounds on the components: ai > si , i = 1, 2, . . . , k

and

s = s1 + s2 + · · · + sk 6 n.

(7.21)

The corresponding enumerator for a component ai is zsi + zsi +1 + zsi +2 + . . . = zsi (1 + z + z2 + · · · ) =

zsi . 1−z

Note that this can be viewed as usual (unrestricted) compositions, where we initialize each component #i with si units. Multiplying enumerators for each component, we find the desired enumerator to be k zsi zs = = zs (1 − z)−k . ϕl (z) = ∏ k 1 − z (1 − z) i=1 Extracting n-th coefficient from ϕl (z), we obtain the number of integer compositions, subject to restrictions (7.21): −k n−s+k−1 n−s −k n−s n (−1) = . (7.22) [z ]ϕl (z) = [z ](1 − z) = n−s n−s

This result shows that the number of solutions of a1 + a2 + · · · + ak = n subject to the constraints (7.21) is the same as the number of solutions of a1 + a2 + · · · + ak = n − s without any restrictions. In particular, if all si = 1 and therefore s = k, then the number of weak compositions of an integer n into k positive components (of course k 6 n) is n−1 n−1 = . [Weak compositions] (7.23) k−1 n−k Exercise 7.31 [1] Show that there exist exactly nj strictly increasing j-tuples so that their last term does not exceed n. Now consider integer compositions with upper-bound restrictions on the components (the lower bound 0 is implicit): ai 6 mi , i = 1, 2, . . . , k

and

m = m1 + m2 + · · · + mk > n.

(7.24)

Its enumerator is k

φ (z, m) = ∏(1 + z + +z2 + · · · + zmi ) = i=1

∏ki=1 (1 − zmi +1 ) , (1 − z)k

m = (m1 , . . . , mk ).

The number of integer solutions of Eq. (7.12) subject to the restrictions (7.24) is then [zn ]φ (z, m). In general, nothing useful can be said about this expression.


373

Let us consider the rather limited range of n defined by m − mi 6 n 6 m, for every 1 6 i 6 k. It is possible here to obtain a closed-form result. To compute the coefficient [zn ]φ (z, m), we make the substitution x = 1/z; hence −n

n

−n

k

[z ]φ (z, m) = [x ]φ (1/x, m) = [x ] ∏ i=1

1 1 1 1+ + 2 + ··· + m x x x i

.

Factoring x−mi from every term, we obtain [zn ]φ (z, m) = [x−n ]

k 1 k mi m−n (1 + x + · · · + xmi ) , (1 + x + · · · + x ) = [x ] ∏ ∏ xm i=1 i=1

m > n,

which cannot be simplified in general. However, in the limited range for n, we can add terms with powers greater than mi ad libitum, without affecting the result: ! k

k

i=1

i=1

[zn ]φ (z, m) = [xm−n ] ∏ (1 + x + · · · + xmi ) = [xm−n ] ∏

=

∑ xj

j>0

= [xm−n ](1 − x)−k

−k k+m−n−1 k+m−n−1 m−n (−1) = = . m−n m−n k−1

(7.25)

Example 7.32 The technique based on the transformation x = 1/z used previously may appear to be a clever device to get the coefficient we just obtained. Except special cases, such transformation may not work. Let us see why. Consider a simple function f (z) = (1 − z)−1 . Obviously, [zn ] f (z) = 1 for any n > 0. The transformation x = z−1 maps the origin to infinity. Therefore to preserve coefficients we have to expand f (x−1 ) around x = ∞, but not around x = 0, since the function f (x−1 ) has the following expansion: x x 1 = =− = − ∑ x j. 1 x−1 1−x 1− x j>1 Hence [x−n ] f (x−1 ) = 0 for every n > 0. So this example shows that in general [zn ] f (z) 6= [x−n ] f (x−1 ) if f (x−1 ) is expanded around x = 0. Another example in the current context is to consider the number of 3-compositions of n = a1 + a2 + a3 subject to constraints a1 6 1, a2 6 2, and a3 6 2. Here m1 = 1, m2 = m3 = 2 and therefore m = m1 + m2 + m3 = 5. If we pick n = 3 we see that the condition m − n = 2 6 mi fails for m1 = 1. While there are 5 compositions of 3 subject to the given constraints: 0+1+2

0+2+1

1+0+2

1+2+0 1+1+1, the formula (7.25) produces = 42 = 6. On the other hand, if we set m1 = m2 = 1 3−1 and m2 = 2, then m = 1 + 1 + 2 = 4 and the condition m − n = 4− 3 = 1 6 mi holds for every i = 1, 2, 3. Now the formula (7.25) gives the answer 3+4−3−1 = 3, which is correct since 3−1 0 + 1 + 2, 1 + 0 + 2, and 1 + 1 + 1 are all compositions of 3 that satisfy the new constraints. 3+5−3−1

374


Example 7.33 In how many ways can a mother distribute 24 candies to her four children so that each child gets at least three candies, but no more than eight?

Solution. Why would a mother even think of unequal distribution is a puzzle, which is not for us to reason. The choices for the number of candies each child receives are given by the corresponding generating function z3 + z4 + · · · + z8 = z3 (1 + z + z2 + · · · + z5 ). Since there are four children, the required generating function is just the product, 4 f (z) = z3 + z4 + · · · + z8 .

To answer the question, we have to extract the coefficient of z24 in f (z). Here is one way of going about it: 4 4 [z24 ] f (z) = [z24 ] z3 + z4 + · · · + z8 = [z24 ] z12 1 + z + z2 · · · + z5 4 1 − z6 12 12 2 5 4 = [z12 ] (1 − z6 )4 × (1 − z)−4 = [z ] 1 + z + z · · · + z = [z ] 1−z −4 −4 2 −4 3 12 6 12 18 24 = [z ] 1 − 4z + 6z − 4z + z 1− z+ z − z + ··· 1 2 3 −4 −4 15 9 = −4 +6 = −4 + 6 = 125. 12 6 12 6

Another possibility is to use MAPLE—the coeftayl command does just that: coeftayl( f ,z = 0,24); returns 125. Exercise 7.34 [1+] Find the number of solutions of x1 + x2 + x3 + x4 + x5 = 100 positive integers with x5 > 22, and (b) in integers greater than −10.

(a) in

Exercise 7.35 [2] Find the number of solutions of x1 + x2 + x3 + x4 = 21 in integers satisfying 1 6 x1 6 5, 1 6 x2 6 7, 3 6 x3 6 11, and 4 6 x4 6 13. Exercise 7.36 [2−] Find the number of solutions of x1 + x2 + x3 + x4 = 1 in integers between −5 and 5 inclusive.

Exercise 7.37 [2] Find the enumerator of the number of compositions a1 + · · · + ak = n with the restrictions pi 6 ai 6 qi , i = 1, 2, . . . , k.

Exercise 7.38 [2] Find a formula for the number of solutions in positive integers of a1 + a2 + · · · + ak = n with a1 6 s1 , a2 > s2 , a3 6 s3 . Exercise 7.39 [2] A balanced die is rolled n times, what is the probability that the sum is 2n?

Exercise 7.40 [3] Here is a problem about compositions for which enumerators do not offer any advantage. Find the number of palindromic compositions of the positive integer n. For example, 2,4,4,2 and 4,2,2,4 are different palindromic compositions of 12. Exercise 7.41 [4] This exercise looks at compositions where the parts are more limited. (a) How many k-part compositions are there for n, when each part is restricted to be an


375

integer in the range [1..t]? (b) The answer you derived for part (a) is probably an unpleasant sum with no clear closed form, and which is also very inconvenient for numerical calculation, since the signs of the terms oscillate, and can be tremendously larger than the final answer. Your task in this part is to find a recurrence satisfied by the sum which allows a much safer numerical calculation of the value for any specific set of parameters. In fact, there are two relatively easy such recurrences. One driven by the bound of part values t, and the other by the number of parts, k. Write both of these, using the combinatorial interpretation (or the enumerator) rather than the explicit sum. Note: The recurrences are in terms of three indices: k, t, and n. While the formalism handles our initial counting questions very elegantly, its source of strength, which is the simplicity of obtaining the enumerator of all desired allocations, is also the source of its weakness. We obtain the simplicity by letting each component take its values independently of the others. They only interact when we want them to have a specific total value. This approach cannot help us to solve, for example, the problem of counting compositions in which the components are required to be distinct. We address this problem in the following sections, and also in the next chapter.

7.3.5 Limited Selection The methods of counting compositions we developed turn out to be useful in counting the ways of making selections of items from sets, when the selection needs to satisfy certain sideconditions. The examples and exercises in this subsection outline some useful techniques. Example 7.42 (Limited Selection: Separated samples) What is the number of ways to select k items out of n with no constraints on the choice? We know the answer from Chapter 2, but we wish to rederive the answer using our new tools. We consider creating a set of elements from the atom class I, where each element has the weight 1. Each element can be selected or not, and we agree to assign a selected item to a set X, and mark it with an x; as usual, z continues to mark the weight. Hence the set of our selections is seen as a repeated choice, modeled by the sum I + X, with the enumerator z + zx. The entire selection process is modeled by the Kleene closure S = (I + X)∗ and its enumerator is n is [zn ]ϕS (z, x) = (1 + x)n . We ϕS (z, x) = 1/(1 − z − zx). The enumerator for a fixed set-size n k n find the desired number by computing [x ](1 + x) = k , as expected. Here and in the following exercises we count the number of ways to choose k integers k , out of [1..n], with a particular type of constraint. A canonical representation of a {a j } j=1 selection is given by writing it in the form 1 6 a1 < a2 < · · · < ak 6 n, corresponding to a possible outcome of the process of repeating the selection, modeled by multiplying the sum I + X by itself, k times. The restrictions are all expressed as upper and lower bounds on differences between successive ai .

In the simplest case, we require ai − ai−1 > 1: the choices may not include successive values.

376


The approach we used before is not directly applicable since there we allowed each component to behave independently. It was the extraction operator [zn ] that made them all speak to the problem at hand. Here the constraints do not allow this independence. Instead we shall construct other sets that we can handle and that can be shown to have a bijection with the set of our constrained selections. We used this approach in Example 3.132, page 127, when lottery numbers were mapped into spacing variables. Here we proceed in a similar way. The inequality a2 − a1 > 1 can be rewritten as (a2 − 1) − a1 > 0, which means that there is no constraint for a1 and a2 − 1. We rewrite the next inequality a3 − a2 > 1 as (a3 − 2) − (a2 − 1) > 0, and so forth. Since the minimal difference between a1 and ak is k − 1, we reduce n by k − 1 to get n − k + 1 and consider the set of all unconstrained k-selections h b + 1, b2 , . . . , bk i from the set of first n − k + 1 positive integers, i.e., [1..n − k + 1]. We claim that the sets of constrained a’s and free b’s have the same cardinality. The bijection needed to prove this is as follows: ha1 , a2 − 1, a3 − 2, . . . , ak − (k − 1)i

⇐⇒

hb1 , b2 , b3 , . . . , bk i.

Every correct selection of a’s produces a suitable selection of b’s as above, and conversely, giving a j the value of b j + j − 1 produces an admissible reverse mapping. The number of unconstrained choices from the reduced set is clearly n−k+1 , giving us the desired answer. k

Let us show a somewhat more elaborate transformation which will use the method we saw to count compositions—this is the reason for discussing it here. We extend every k-selection h a1 , a2 , . . . , ak i with two ‘dummy’ entries, a0 = 0, ak+1 = n. Then we define a new list, equal to the differences between successive a j ’s: di = ∆ai = ai+1 − ai ,

0 6 i 6 k,

a0 = 0, ak+1 = n.

We list the components di (i = 0, 1, . . . , k) along with corresponding enumerators in the following table. The di are selected to satisfy: z ϕ0 (z) = 1−z , d0 > 1, z2 d j > 2, 1 6 j < k, =⇒ ϕ j (z) = 1−z , 1 6 j < k, 1 ϕk (z) = 1−z , dk > 0, where we also used our knowledge that the a j are selections from N, and disregard temporarily the limitation to [1..n]. Moreover, not only does the list h a1 , a2 , . . . , ak i determine the list h d0 , d1 , . . . , dk i uniquely, the reverse is also true: since a j = d j + a j−1 , we find recursively a1 = d1 , a2 = d2 + a1 , . . . and ak = dk + ak−1 . Hence the two sets have the same cardinality, and we can count the d vectors instead of counting a vectors. How do we find the cardinality of h d0 , d1 , . . . , dk i? We need two observations. First is that the inequalities in the table above are all that constrain the di s. Second is that we know their sum! If we add them all together we find that k

k

∑ di = ∑ (ai+1 − ai ) = ak+1 − a0 = n

i=0

since the sum is telescopic.

i=0


377

How many lists h d0 , d1 , . . . , dk i are there? There are as many as (k + 1)-compositions of n with the above constraint on their range. Hence the set of compositions, S, has the enumerator ϕS (z) = ϕ0 (z)ϕ1k−1 (z)ϕk (z), and the number of valid selections is −k − 1 n−k+1 1 z2k−1 n−2k+1 n−2k+1 = [z ] = (−1) = . [z ] n − 2k + 1 (1 − z)k+1 (1 − z)k+1 k n

While the discussion of the transformation between the selections of the a j and the differences di was entirely adequate for the purpose here, it is useful to state it more formally (with no essential change) for other more involved situations. Let Sk be the set of all k-selections from [1..n], n ∈ N, the set Z+ restricts N to the positive integers (N \ {0}). Then Sk |= Zk+ × N : {a j }kj=1

⇐⇒

h d0 , d1 , . . . , dk i,

(7.26)

where a1 < a2 < · · · < ak and di = ai+1 − ai , for i ∈ [0..k] with a0 = 0; ak+1 = n. The symbol |= means one-to-one correspondence by decomposition. The k sets Z+ generate the di for 1 6 i < k, and the extra N provides the specification of ak+1 = n needed to compute dk . Exercise 7.43 [3] (Separated samples, continued) (a) Show that the total number of subsets of [1..n] that do not contain a pair of successive integers is Fn+2 , a Fibonacci number. (b) Write an enumerator for the set of selections in the more general case, where the choices are restricted by si 6 ai − ai−1 6 ti , 1 6 i 6 k.

Exercise 7.44 [2+] (More on separated samples) This exercise generalizes n−k+1 Example 7.42. You are asked to show that fn,k ( j) = k−1 k− j , where fn,k ( j) is defined as the number of selections of k elements j out of [1..n], with exactly j successions (pairs of adjacent selections). The pairs may overlap (the choice of 1,2,3 provides two successions). Example 7.45 Our knowledge about compositions and recurrences obtained previously allows us to solve the following classical problem without explicit application of our enumerating techniques. This problem is usually referred to as Smirnov’s problem, and it is considered later in Examples 8.14 and 8.16, page 437. What is the number of ways to choose k objects, no two consecutive, from n objects arranged either in a row or in a circle? To answer this question, we consider a simple particular case of choosing three letters b, e, and i from the ten letters a through j arranged in a row: h a, b, c, d, e, f , g, h, i, j i . Starting at h, we jump one letter to reach the first letter chosen (b), then jump two positions, then three, and finally jump one to reach i. Thus our choice of b, e, and i can be encoded by the jump sequence 1, 2, 3, 1. This sequence has four terms that sum to seven, since seven integers were jumped in choosing three things from ten. Clearly all the choices of three letters, no two consecutive, from a set of ten standing in a row are in one-to-one correspondence


378

with the solutions of a1 + a2 + a3 + a4 = 7, where a1 > 0, a2 > 1, a3 > 1, and a4 > 0. The number of such solutions is known from Eq. (7.22) to be 73 = 35.

The general case can be treated in a similar way and the number of choices of k objects, no two consecutive, from a set of n objects in a row is equal to the number of solutions of a1 + a2 + · · · + ak + ak+1 = n − k, where a1> 0, ai > 1 for i = 2, 3, . . . , k, and ak+1 > 0. According to Eq. (7.22), this number is n−k+1 . k The same result can be obtained by deriving a recurrence for the desired number, denoted by f (n, k). We split the number of ways of selecting k objects, no two consecutive, into two disjoint subsets: those which include the last of the n objects and those which do not. The former is f (n − 2, k − 1) and the latter is f (n − 1, k). Hence we get the recurrence: f (n, k) = f (n − 1, k) + f (n − 2, k − 1), which has the solution f (n, k) = n−k+1 . k

Now we go to distribution of n objects around a circle. Wesplit the circle between objects 1 and n and stretch it out flat. We know that there are n−k+1 choices of k of these objects, no k two consecutive. From these we must subtract precisely those in which objects 1 and n are chosen. Such selections did not n − 1, and their count is obtained by reducing n by use 2 and n−k+1 4 and k by 2, which is n−k−1 . Since can be expanded as k−2 k

n−k+1 n−k n−k n−k n−k−1 n−k−1 = + = + + , k k k−1 k k−1 k−2

in which the last term is equal to the subtrahend; so we are left with

n−k n−k n n−k n−k−1 n−k k . = + = + n−k n−k k k k k−1 k

Exercise 7.46 [2+] Show that the number of ways to choose k objects, no two consecutive, from a set of n objects arranged in a row, with every pair of selected objects separated by at least r not chosen objects, is n − rk + r . k Exercise 7.47 [3] Show that the number of ways to choose k objects, no two consecutive, from a set of n objects arranged in a circle, with every pair of selected objects separated by at least r not chosen objects, is n − rk n . n − kr k Example 7.48 (Limited selection: Terquem problem) The following example is a classical limited-selection problem. It may remind you of the separated-samples problem, but it is stated differently; the transformation below shows how close they are. The problem is to find the number of k-selections from [1..n], denoted by (1 6 a1 < a2 · · · < ak 6 n) such that the terms in odd-indexed positions are odd, and those in even positions are even. Another way to express this requirement is to write it as ai ≡ i


379

mod 2. Hence any two successive selections differ by an odd number. We can then write the set of all such selections for a fixed k as Sk |= Ok × N : (a1 , a2 , . . . , ak )n |= (d0 , d1 , . . . , dk ), where the symbol |= denotes one-to-one correspondence by decomposition that provides a bijection (but not an equality) of the k-products of the set of all odd positive integers, O, with the relation between the list {a j }kj=1 augmented with the ‘dummy’ entries, a0 = 0, ak+1 = n, and the list {di }ki=0 , which is almost as in Example 7.42: di = ∆ai = ai+1 − ai ,

0 6 i 6 k,

a0 = 0, ak+1 = n.

On the left-hand side of the correspondence, the value for ak+1 is provided explicitly, by the subscript on the selection vector. On the right-hand side there Ok contributes the first k components di (the last one, dk is not constrained), and N provides the value n, needed to make the transformation invertible. Therefore this decomposition creates a bijection, from which the enumerator follows: !k 1 zk = . ϕSk (z) = ϕOk (z) × ϕN (z) = ∑ z2 j+1 1 − z (1 − z)(1 − z2 )k j>0 To find the number of suitable selections we need the coefficient of zn and find, conveniently 1+z replacing the 1 − z in the denominator by a 1−z 2 , that n−k

sn = [z

n−k−1 n−k −k − 1 −k − 1 1+z 2 2 + (−1) = (−1) ] n−k n−k−1 . (1 − z2 )k+1 2 2

Exactly one of the two binomial coefficients is non-zero: the one with a lower argument which is an integer; whether n − k is even or odd, the integer among the two can be written as ⌊ n−k 2 ⌋. Since n − k and n + k have the same parity, we find the answer to be ⌊ n−k 2 ⌋

(−1)

n+k ⌊ 2 ⌋ −k − 1 = . ⌋ ⌊ n−k k 2

7+3 By way of checking: n = 7, k = 3 allows the ⌊ 32 ⌋ = 10 selections {(1,2,3); (1,2,5); (1,2,7); (1,4,5); (1,4,7); (1,6,7); (3,4,5); (3,4,7); (3,6,7); (5,6,7)}.

Exercise 7.49 [2+] (From Terquem to Skolem sets) This is a direct generalization of the Terquem problem: a k-Skolem set of index p ∈ N is (a1 , a2 , . . . , ak ), a k-subset of [1..n], augmented with a0 = 0, such that ai −ai−1 ≡ 1 mod p, 1 6 i 6 k. Compute sn,k,p , the number of such subsets. Example 7.50 Consider a scheduling problem when many people make a request to use a resource—it may be a supercomputer or a lecture room. A request consists of two numbers: a start time and a finish time. Using a fixed unit for time, such request can be visualized by

380


an interval on the straight line with integer end points. Assuming that the resource can be used by many people at a time, the corresponding request intervals overlap. When there is a limitation on the number of people that can use the resource simultaneously, the number of overlapping intervals cannot exceed this fixed number. Define an r-cluster on the one-dimensional integer grid as an interval of the grid, which is created by laying ‘rods,’ subintervals of fixed size r units (which cover r + 1 grid positions), in such a way that they overlap as shown sample in the diagram, of a 27-unit long cluster made up of six seven-units long rods. The diagram is “exploded,” so you can see the order in which they came together to create the cluster. But an entirely equivalent construction allows the cluster to grow in one direction only. A rod covers r + 1 points, and rods that meet at a point share the point. Every point in the cluster is covered by one or more rods, up to r, independently of the number of rods in the cluster. The bound is a result of the requirement that no two rods may lie exactly in the same position. How many clusters of size (meaning number of grid points covered) n can be created using k rods? The description of the cluster means that its end points belong to a single rod each. Therefore a cluster can be constructed by adding one rod at a time, going, say, from left to right. The new rod protrudes by some number j of grid positions beyond the existing cluster such that 1 6 j 6 r. Hence the class C of clusters may be seen as having the construction C = C + [1..r] = [1..r]+ (see Definition 7.14 on page 362) since we do not want an empty cluster (we would use [1..r]∗ if we allow this). The weight function we want for the cluster, length, tells us to pick for [1..r] a weight function equal to the numerical values of the term; it then has the enumerator ϕr (z) = z + z2 + · · · + zr = (z − zr+1 )/(1 − z). Since we need to count the rods, we mark each with the variable u; then for the clusters we have the closure of this enumerator: ϕC (z, u) = uzr+1 /(1 − uϕr (z)). The requested answer is then given by k−1 uzr+1 z − zr+1 1 − zr k−1 n−r−1 n−r−k = [z ] . = [z ] [z u ]ϕC (z, u) = [z u ] r+1 1−z 1−z 1 − u z−z 1−z n k

n k

An explicit answer is unlikely, but for small n one can extract the needed values directly, and a recurrence is possible for higher values. Since the enumerator is a polynomial, the utility of analytic tools for it is limited. Note that the computed number can be written as W (n − r, k − 1, r), where W (n, k, m) is defined on page 369, in Eq. (7.18).

Now we want to find the mean length (=number of grid points covered) and variance of a cluster made up of k rods. This part does not need any combinatorial calculations. The length of a (k, r)-cluster is r + 1 + X1 + X2 + · · · + Xk−1 where the X j are independent, integer-valued random variables, uniformly distributed on [1..r]. The mean of such a random variable is (r + 1)/2, and its variance is (r − 1)2 /12. Hence the expected length of a cluster is r + 1 + (k − 1)(r + 1)/2 = (r + 1)(k + 1)/2, and its variance is (k − 1)(r − 1)2 /12. Exercise 7.51 [3+] How many sequences of n tosses of a coin contain k times the pair HH of

7.4. FURTHER SET OPERATIONS

381

heads, r times the pair TT of tails, and s times HT? Hint: Count overlaps (see also §9.4); for instance, the number of overlaps for the sequence THT is 1, 0, 1, respectively. Note: The essence (and relative difficulty) in this exercise is to relate the specified paircounts to the variables used in an enumerator you can construct by inspection, for this set of sequences.

7.4 Further Set Operations The sum and product operations, and the associated closure, have served us well in our discussion of integer compositions, but we need to prepare for more complicated needs ahead. In truth, all the other operations we shall see are based on these two fundamental operations, but being more complex, they often provide useful shortcuts. We discuss the next two operations in the context of sets of strings, where they appear most natural, but they can, and will be used in other contexts as well. Let A = {a} be the set consisting of a single element. This is just the alphabet we introduced in Example 7.15, page 363. We use the same weight function, which is natural for a string: its length, w(a) = 1; the enumerator is ϕA (z) = zw(a) = z. Let A∗ be the Kleene closure of A (see Definition 9.14 on page 492). We saw in the same example that its enumerator is 1 . Indeed, [zn ]ϕA∗ (z) = 1, since we have a single word of each ϕA (z) = 1 + z + z2 + · · · = 1−z ∗ size n > 0 in A . This situation is generalized in the following subsection, but we start with an example that sheds an interesting light on our discussions of string problems, and the applicable role of enumerators. Example 7.52 (Hidden patterns in words) A sequence of letters that occurs in the prescribed order, but not necessary contiguously in a text is referred to as a “hidden pattern.” There are many practical situation where detection of hidden pattern is important. For instance, in molecular biology it is common to search for a constrained hidden pattern in genes. In the area of computer security the problem is related to detecting an attack on computer systems. The hidden pattern problem is known in other related areas as data mining, bibliographic search, textual data processing, and many others. The purpose of this example6 is to show that ‘disobeying’ a rule in a disciplined way sometimes provides us with the structure we need. We consider words of any finite length over an alphabet A with r letters. Suppose we have a fixed “pattern” of k letters, β = x1 x2 · · · xk , and we consider the number of its occurrences as a subsequence in a text T , which is an element of the closure A∗ . In contrast to substring, a subsequence is an ordered subset of the letters, but not necessarily adjacent ones. Thus, the pattern abc occurs seven times in the word c b a c b c b b a c a c b, reusing some letters more than others. We add another wrinkle: we wish not just to count such occurrences, but 6 Its

generalization can be found in [46].

382


also to add a probabilistic flavor to the discussion, and for once we abandon the prevalent assumption of uniformity: some words are recognized as likelier than others. For example, we know that the frequency of letters in any natural language is very skewed. A standard frequency assignment to letters in English gives the letter e a likelihood just over a hundred times that of q. The simplest model that allows us to do so sees the letters in A as associated with fixed probabilities; the letter a j appears with probability p j . We assume these events are independent. Hence the probability of a word is equal to the product of its letter-probabilities. def We denote this probability for a string w as π (w) = pw1 pw2 · · · pw|w| .

The enumerator of A∗ , when saddled with such a model, is naturally 1/(1 − ∑ j (p j z j )), where z j marks, or enumerates, the number of appearances of the letter a j . However, if we disregard the identity of the letters, and just ask about counts of words of given size, then all z j become z, the sum ∑ j p j z equals z, since the probabilities sum to 1, and we revert to the enumerator we saw before, 1/(1 − z).

Finally, the needed construction is apparent from the following diagram, drawn for the special case of the pattern x1 x2 x3 , · · · · · · · · · · · · x1 · · · · · · · · · · · · x2 · · · · · · x3 · · · · · · · · · · · · · · · ,

where the dotted segments represent arbitrary letters from A, and each is therefore an element from A∗ . We can see that all such words, w, with cardinality k = |w| will be created by taking the Cartesian product of the pattern letters with k + 1 repetitions of A∗ : T = A∗ x1 A∗ x2 A∗ x3 · · · xk−1 A∗ xk A∗ . Note that we only consider texts that contain the pattern at least once. Such a product is associated with the enumerator, that we call a probability generating function, π (β )zk , π (β ) = px1 px2 · · · pxk . Q(z) = (1 − z)k+1

The alert reader will see that this string of products does not have the unique decomposition property! Refer back to the example with the pattern β = abc and you see how we find it several times in the given word, by associating different strings with the four segments from A∗ .

Let us state this observation precisely since it will provide us with our key relation. Look again at the preceding example as an instance of a text T in which the pattern letters appear in certain places. We call each such sequence of k (here 3) index locations a pattern position and denote it by I. Typically a given text T will have the pattern in several positions, and the set of these positions, seven in this example, is denoted by IT . The size of this set is the number of times β appears in T , and we shall denote it by Xβ (T ) = |IT |. Let Q be the collection of all possible texts with all possible placements of the pattern in them: def

Q = { (I, T ) :

I ∈ IT }.

The size of each such pair is taken to be the length of the text T . We will examine the pattern in texts of some fixed length n; they are all elements from A∗(n) , the set of all words from the


383

alphabet A having the weight n, and their probabilistic weight is given by π (T )zn . Now we use this expression for Q to write its ordinary probability generating function, first arranging the summation by choice of text, and then breaking those up by length: Q(z) =

∑

π (T )z|T | = ∑ Xβ (T )π (T )z|T | = ∑ zn T

(I,T ),I∈IT

n

∑

Xβ (T )π (T )

T ∈A∗(n)

From Eq. (3.11) on page 100, we see that the inner sum equals the expected value of Xβ on all texts of size n; it is denoted by En [Xβ ]. Hence we found two expressions for the (ordinary) probability generating function, Q(z) =

π (β )zk = En [Xβ ]zn . (1 − z)k+1 ∑ n

Extracting the coefficient of zn from the middle expression, we obtain k −k − 1 n n π (β )z n−k −k−1 n−k = π (β )[z ](1 − z) = π (β ) (−1) = π (β ) . En [Xβ ] = [z ] k+1 (1 − z) n−k k It is interesting to note that as n increases, and we keep k fixed, this grows polynomially fast (as nk ). If we let k grow as well, as nc for some c > 0, the expectation decreases to zero, because the pattern probability decreases exponentially (even faster). While this calculation appears startlingly simple, we make two observations: (1) In practice we are interested in the number of occurrences where the separations between successive pattern positions are not unbounded, but required to satisfy some constraints. (2) Computing higher moments, in particular the variance, is much more difficult. For further information, see [47].

7.4.1 Substitution We show by an example what is meant by “substituting a set B into a set A” to produce a set denoted by C = A[B]. While A could be in principle any set with a suitable structure (which will become obvious shortly), we present this operation using A ⊆ Σ∗ , a set of strings over an alphabet Σ; the same holds for B, though it could—and often would—be over a different alphabet. The subset of A consisting of all words of length n is denoted as usual by A(n) . Using the weight function of string length we have w(x) = |x| = n for any x ∈ A(n) . For the purpose of defining substitution, think of such words as having n places. When we substitute elements from B into A, all attributes are disregarded for the substituted element. In other words, elements from A contribute only their “skeleton,” not their contents. We shall use a special symbol A ←֓ B to represent the operation of B substituted into A (to produce A[B]). The set A[B] is defined as follows. Definition 7.53 Let A(n) be the subset of A consisting of elements of weight n. The substitution of B into A, denoted A[B], is well defined when B(0) = ∅, and is accomplished by

384


associating with each element x ∈ A(n) the entire set Bn : [

C = A[B] =

n>1

A(n) ←֓ Bn ∪ A(0)

In turn, A(n) ←֓ Bn is ∪x∈A(n) (x ←֓ Bn ), and x ←֓ Bn = ∪β ∈Bn (x ←֓ β ).

⊳

The need for restricting B to have no elements of weight zero will be demonstrated later, in Example 7.56. The substitution of β , which consists of a sequence of n elements from B, say (b1 , b2 , . . . , bn ), is seen as being done by filling the n places in x, element by element. (When we substitute a language into a language, letters go into letter positions, but with other combinatorial structures, different things could happen: substituting a language into graphs creates a labeling of the graph; substituting digits into well-formed formulas creates numerical expressions.) Example 7.54 Let S = {a} and B = {0, 1}. The Kleene closure A = S∗ of the one-letter alphabet S is the union of Sn , n > 0, that is, A = { ε , a, aa, aaa, . . . } .

In this case, the substitution of B into A means the replacement of an by all possible values in Bn , and leads to the set (7.27). The corresponding generating functions are

ϕS (z) = z,

A[B] =

ϕA (z) =                               

ε,

"

1 1 = , 1 − ϕS (z) 1 − z



#  00  01 0 ,  10 1  11



          ,          

000 001 010 001 011 101 110 111

ϕB (z) = 2z. 

               

       , ... .                      

(7.27)

And by inspection of the above array, the substitution produces

ϕC (z) = ϕA[B] (z) =

∞

∞

n=0

n=0

1

∑ 2n zn = ∑ (2z)n = 1 − 2z

(7.28)

since here the weight of each element from Bn is n. Once we have the enumerator of C, we can find how many elements it has of any given weight obviously: [zk ]ϕC (z) = 2k .

Theorem 7.55 Let C = A[B] where the sets A and B have weight functions wA and wB and the enumerators ϕA (z) and ϕB (z). Then

ϕC (z) = ϕA[B] (z) = ϕA (ϕB (z)).


385

P ROOF : We use Definition 7.53: C = A(0) ∪ ∪n>1 A(n) × Bn . We assume that the weight of an element of C, created by putting an element βn ∈ Bn , into an x ∈ A(n) , is just the weight of βn , and does not include wA (x). It is as if the x contributes its abstract structure, or skeleton, but no weight is added. We recall the notation an for the number of elements in A(n) , and each one of them receives all the elements in Bn . This set has the enumerator ϕBn (z) and, using Definition 7.2, we find it to be

ϕC (z) =

∑

σ ∈C

zw(σ ) = a0 z0 + ∑ an n>1

∑

zw(βn )

βn ∈Bn

= a0 + ∑ an ϕBn (z) = a0 + ∑ an ϕBn (z) = ϕA (ϕB (z)) .

(7.29)

n>1

n>1

We notice that the result obtained in Eq. (7.28) is consistent with this formula. Now we reconsider Example 7.54 with different weight functions. Example 7.56 Again, let S = { a }, A = S∗ , and B = {0, 1}. We define the weight function of B-elements as their numerical value: w(i) = i. Therefore we have w(0) = 0, w(1) = 1, and the enumerator is ϕB (z) = 1 + z. Then for C = A[B] we obtain

ϕC (z) = ϕA (ϕB (z)) =

1 1 1 = =− . 1 − ϕB (z) 1 − (1 + z) z

(7.30)

This is a function that has no positive powers of z. What has happened? In §7.1, when we introduced enumerators, we required that even for infinite sets, the number of elements of each weight must be finite. But this is not the case with this set C. For example, C(0) is not finite, since we can put the letter 0 ∈ B in all the places of an infinite number of elements from A, creating the structures 0, 00, 000, . . . ; and they all have the weight zero. This is the reason for the requirement in Definition 7.53, that B(0) = ∅ (which can be alternatively written as b0 = 0). Here B(0) = {0}. Exercise 7.57 [2+] In Example 7.56, show, by explicit construction, how it is possible to get an unbounded number of elements of C = A[B] of any weight.

Example 7.58 Once again we delve into Example 7.54, and use a different weight function. Let S = { a }, A = S∗ , and B = {0, 1}. We define the weight function of the elements of B by w(i) = numerical value + length = i + 1. Thus, w(0) = 1 and w(1) = 2. Now the enumerator of B is ϕB (z) = z + z2 and

ϕC (z) = ϕA (ϕB (z)) =

1 . 1 − z − z2

(7.31)

This is not only a proper enumerator, we actually know the coefficients: [zn ]ϕC (z) = [zn ]

1 z = [zn+1 ] = Fn+1 , 2 1−z−z 1 − z − z2

where {Fn } is the Fibonacci sequence we have seen before (e.g., in Example 1.26).

386


Let us manufacture an example showing how such a scenario can occur in the analysis of a computation. Suppose we have a program that processes strings, defined over the alphabet B = {0, 1}. The program spends most of its run-time in a computation that takes time α for every bit, and an additional time β to process a bit 1. So the cost of 0 is α and the cost of 1 is α + β . If there are i ones in the word of length n then its cost is nα + iβ . Correspondingly, we use a generating function that accounts separately for the two components of the weight. We shall use u to mark length (in bits) and z to mark numerical value. The enumerator of the set B would then be given by ϕB (u, z) = u + uz, where u is contributed by the 0, and the uz by 1. The calculation of the enumerator of substitution is done as before, ϕC (u, z) = ϕA (ϕB (u, z)), and we find

ϕA (x) =

1 1−x

=⇒

ϕC (u, z) =

1 . 1 − u − uz

(7.32)

With the aid of ϕC (u, z) we can find the number of elements in C of length n and bit-wise numerical value d: n k k d n k n d n d n d . [u z ]ϕC (u, z) = [u z ] ∑ (u + uz) = [u z ] ∑ u (1 + z) = [z ](1 + z) = d k>0 k>0

Since such a string costs α n + β d to process, the total cost of processing all words of length n is n n n d α n + β d) α n + β = 2 ( ∑ d ∑ d d d n−1 = 2n α n + β n2n−1 = n2n−1 (2α + β ). = 2n α n + β ∑ n d − 1 d To find the average time to process such a string we need to divide the total cost by 2n to obtain the entirely unsurprising value n(α + β /2). Note that this approach of summing for total cost and dividing by the number of elements (strings, here), is the same as assigning each element the same probability. We can then define a random variable C to be the processing cost of a string, which for a string of size n and d 1-bits is given by α n + β d. Therefore, the probability of a string of length n having processing cost c is given by the number of such strings with d = (c − α n)/β 1-bits, divided by 2n , that is 2−n dn .

Exercise 7.59 [3] Use the method of the last example to compute the variance of the cost of processing a random string of size n. Example 7.60 (Binary sequences with no repeated 0’s, adapted from [55, p. 43]) This example shows a use of the substitution operation in enumeration. Let S be the set of all binary strings (of 0’s and 1’s) with no repeated zeroes. The objective is to count the number of strings of any given size n in the set S. We shall see a more elaborate version of this problem (known as the Smirnov problem) in Example 8.14 on page 437.


387

Unlike previous examples, such as compositions, where we constructed the elements we wished to count from more elementary constituents, we take a reverse course here: we show that our set S is a building block for a larger set, specifically, the closure {0, 1}∗ . This will allow us to deduce its enumerator. How do we do it? We show that the closure, A = {0, 1}∗ , is in one-to-one correspondence with the set obtained by substituting the set B = 00∗ into selected places in S. This is achieved by tracking the full substitution scheme outlined in Definition 7.53: each occurrence of a zero in strings from S is replaced by one or more zeroes. For example, the element 010001100100011 from A could only be generated from one element of S: 0101101011. Hence we have a bijection between A and S[B]. Since the substitution is partial, not complete, the relation given in Theorem 7.55 is clearly untenable. In order to be able to write the relation explicitly, we need to mark the 0’s and 1’s in A and in S separately. Therefore we define the weight function used in these sets as having two components: w(σ ) = (|σ |0 , |σ |1 ), where the components simply mean the number of 0’s and 1’s in the string σ as it appears in those sets, and then the enumerators are self-evident: i+ j i j |σ |0 |σ |1 ϕA (x, y) = ∑ x y = ∑ ∑ xy , i σ ∈A i>0 j>0 where i and j are the numbers of 0’s and 1’s in a string, which we can order in i+i j different ways. We can evaluate the sum, for example, by making the substitution k = i + j; it becomes k k 1 ∑ ∑ i xi yk−i = ∑ (x + y)k = 1 − (x + y) . k>0 i=0 k>0 For ϕS we have a similar definition, but we do not know how to calculate it yet. The enumerator of the set B is clearly ϕB (x, y) = ∑i>0 xi+1 y0 = x/(1 − x). Now we represent the (partial) substitution as giving rise to an equation that will finally clarify the relationships: 1 x = ϕS ,y ϕA (x, y) = ϕS (ϕB (x, y), y) =⇒ 1−x−y 1−x Let u = x/(1 − x). Solving for x we find x = u/(1 + u), hence the required enumerator becomes 1 1+u . = ϕS (u, y) = u 1 − 1+u − y 1 − y − uy

Now what? We still want to answer: how many strings of length n does S have? Observe that the length is the sum of the number of 0’s and 1’s, and therefore we can write

ϕS (x, x) =

∑ x|σ | x|σ | ) = ∑ x|σ | +|σ | ) = ∑ S(n) xn . 0

σ ∈S

1

0

σ ∈S

1

n>0

1+x Hence all we need to do is to extract the coefficient: [xn ]ϕS (x, x) = [xn ] 1−x−x 2 . You may remember this denominator from Example 7.58. Using Eq. (7.31), we find that S(n) = Fn+1 + Fn = Fn+2 . Indeed, for n = 4 the possible strings are 1111, 0111, 1011, 1101, 1110, 1010, 0101, 0110, for a total of 8 = F6 .

388


Exercise 7.61 [3] A Ferris wheel can be modified to carry a variable number of benches. We now give the various parameters of the wheel, and in each case also a representative value in parenthesis. Benches come in small and large sizes, of which the management has U (in particular, 20 benches) and u (in particular, 40 benches) respectively, and may not be mixed in a setting. The wheel can carry from C0 (in particular, 8) to C1 (in particular, 16) large benches or c0 (8) to c1 (32) small ones. Since the benches are all different (many carry names of patrons, either inscribed or incised), changing the arrangement—a circular permutation— of the benches is considered a different setting, even if the same bench selection is used. Small benches can accommodate up to d (4) people, and large ones up to D (8). In how many different ways can n (75) people be seated in the wheel? Note that changes of order on a single bench do not count, but shifting people (which are always “labeled”) between benches does.

7.4.2 Marking The marking operation on a set A produces the set D(A) in a way that depends on the nature of the elements of A. As usual, we assume that there is a weight function w : A → N. Consider a typical scenario: each element σ ∈ A is based on a number of ‘atoms’ or elementary ingredients, such as nodes in a graph or letters in a string. Then, the number of these atoms is equal to the weight, w(σ ). There are different ways of writing down D(A), and no universal convention is known. For example, D(A) = {(σ , a), ∀a ∈ σ , ∀σ ∈ A}. The weight of each element (σ , a) is equal to w(σ ) itself; it is the same element, but with one of its atoms, a, distinguished. You may think, for example, on the selection of a node in a connected acyclic graph, making it a root of a tree. As another example, consider the set of one single string A = {absa}, with ¯ absa, the enumerator z4 . Then D(A) = {absa, ¯ absa, ¯ absa}, ¯ where the bar denotes the marked 4 ′ atom. The enumerator of this set is ϕD(A) (z) = 4z = zϕA (z). There is another way to express D(A) by numbering marked positions: D(A) = {(absa, 1), (absa, 2), (absa, 3), (absa, 4)}.

For each subset A(n) of A consisting of the an = |A(n) | elements having n atoms of weight 1, the corresponding set D(A(n) ) contains nan elements because any element from A(n) can be . [ {σ } × {weight of σ } is simply selected (or marked). Hence the enumerator of D(A) = σ ∈A

the derivative of that of A:

d ϕD(A) (z) = ∑ nan z = z ϕA (z) = zϕA′ (z). dz n>0 n

(7.33)

For this reason some authors call this operation a differentiation of the set, and that is why we denoted it with a D (the letter M was otherwise engaged, see §7.4.4). Example 7.62 We can use this quite limited operation to derive the enumerator of a set created by the closure operation. Let S = [1..n]∗ be the closure of the set of the first n positive integers.


389

For any p ∈ N, we want to find the cardinality of S(p) , the set of all strings of length p. Of course we know it, we have done so before, in Definition 7.14 and Eq. (7.10), using infinitary arguments. This is not a surprise since S is an infinite set, but let us yield to the very human tendency of preferring calculations that handle finite, “tangible” objects and arguments. We do this by showing a decomposition of D(S − ε ), the marked set derived from S (except the empty set, where there is nothing to mark). The atom we mark in S is a member of [1..n], and the element b of S is viewed as a string that we can decompose into two strings surrounding the item marked with a dot above it: b = x c˙ y, with x, y ∈ S, and c ∈ [1..n]. Hence, we see the decomposition D(S − ε ) |= S × [1..n] × S. This establishes a bijection because we can fit each element on the left-hand side with exactly one element on the right-hand side, and the same relation holds for the reverse direction. Using as the weight function the length of a string, the relation in Eq. (7.33) is valid, and we have the equation z

d [ϕS (z) − 1] = ϕS2 (z)nz dz

=⇒

ϕS′ (z) = nϕS2 (z).

The differential equation f ′ (z)/ f 2 (z) = n has the integral −1/ f = nz + c, and a constant of integration c is determined at the point z = 0. Since S includes the empty string of weight 0, the function ϕS (z) has the value 1 at the point: ϕS (0) = 1. Hence ϕS (z) = 1/(1 − nz), and the coefficient of z p is naturally n p . The marking operation can provide more functionality if we do not insist that the number of atoms in an element that can be marked be equal to the weight of the element. For example, this could occur in an acyclic graph where only nodes of degree > 2 could be selected as roots, or strings where only a certain subset of the alphabet is eligible for marking and so on. We then use two variables over the marked set: z to record its usual weight, w, and u to track its “marking weight,” or m-weight, wm , which is the number of marks that can be used on that element. Thus, it is natural to define the enumerator as ϕA (z, u) = ∑σ ∈A zw(σ ) uwm (σ ) . For instance, [um ]ϕA (z, u) gives the (enumerator of the) number of elements in A that can be marked in m positions, and for the enumerator to exist this number must be finite for each m ∈ N. We find the relation . [ ∂ {σ } × {weight of σ } =⇒ ϕD(A) (z, u) = u ϕA (z, u). (7.34) D(A) = ∂u σ ∈A More generally, suppose we want to count the number of occurrences of letters in a word built from the alphabet A = {a1 , . . . , ar } with r letters. We mark each letter a j with the variable u j ( j = 1, . . . , r) and the enumerator for a word of length 1 (an atom) is z(u1 + u2 + · · · ur ). From Eq. (7.10), it follows that the required enumerator for all possible finite words becomes E(z, u1 , u2 , . . . , ur ) =

1 . 1 − z(u1 + u2 + · · · ur )

For example, if A = {absa} is the one-word set, then its enumerator is E(z, u1 , u2 , u3 ) = 1/(1 − zu1 − zu2 − zu3 ). In particular, the number of words with n j occurrences of letter a j is


390

(n = n1 + n2 + · · · + nr ) [zn un11 un22 . . . unr r ]E(z, u) =

[un11 un22 . . . unr r ] (u1 + u2 + · · · + ur )n

n = . n1 , n2 , . . . , nr

In the following exercise, we look at k-selections from [1..n]: the weight of each element is n, but only selections can be marked usefully in the context of the problem there. Exercise 7.63 [3+] (Separated samples on a circle) This exercise continues Example 7.42 and Exercise 7.44. (a) Assume the set [1..n] is laid out around a circle; say it is used to number n seats at a circular table. In how many distinct ways can we select k seats? Use a bijection to obtain an enumerator for your answer. (b) Now we ask the same question as in Exercise 7.44: show that the number of kselections can be made around the table with exactly j pairs of neighbors selected is given by n k n−k−1 k j k− j−1 . The pair (1, n) can contribute to this number as well.

7.4.3 Power Set The power set, denoted by 2A , of a set A is the collection of all of its subsets, as seen in Example 7.3. In that example, we calculated the enumerator of 2A for a natural weight function w(σ ) to be the number of A-elements in σ ∈ 2A . According to this weight assignment, ϕA (z) = |A|z and ϕ2A (z) = (1 + z)|A| .

What can we say about the possible representations of ϕ2A (z) in terms of the enumerator ϕA (z), when the weight function is arbitrary? We shall continue to assume that the weight of a subset of A is the sum of the weights of the elements, using the same weight function w( σ ) . When these |A| factors are that prevails in A. Let us look at the product ∏σ ∈A 1 + z expanded, we get 2|A| terms, each of the form zw(σ1 )+w(σ2 )+···+w(σk ) , for 0 6 k 6 |A|. These selections are exactly the weights of the possible subsets of A; hence we write the enumerator of 2A : ϕ2A (z) = ∑ zw(σ ) = ∑ zw(α ) = ∏ 1 + zw(a) . (7.35) α ⊆A

σ ∈2A

a∈A

This still does not provide us with an explicit functional relationship, of ϕ2A (z) in terms of ϕA (z). To get one we need to work a bit harder.

We would like to use the coefficients an = [zn ]ϕA (z). Since an is the number of elements in A with weight n, we can rewrite the right-hand side of Eq. (7.35) as follows ϕ2A (z) = ∏ 1 + zw(a) = ∏ (1 + zn )an . a∈A

n>0

Taking logarithms of both sides and using the standard Taylor series for the logarithm function around the point 1, we find ln ϕ2A (z) =

(−1) j−1 n j z , j j>1

∑ an ln (1 + zn) = ∑ an ∑

n>0

n>0


391

and reversing the order of summation, ln ϕ2A (z) =

(−1) j−1 ∑ j ∑ anzn j = n>0 j>1

(−1) j−1 ∑ j ϕA (z j ). j>1

(7.36)

Exercise 7.64 [2] Use (7.36) to recover the calculations made in Example 7.3 on page 357. See the first paragraph of this subsection.

7.4.4 Multiset Operation It is instructive to discuss the multiset operation by comparing it to the power set operation, or to the closure. Like these operations, it creates a collection of sets from a given set A. This collection is denoted by M(A). The multiset adopts some features of these two set operations, and drops some of their features. Like the closure, a set may contain repeated elements of A (as in sampling with replacement), and as in the power set structure, there is no order among those elements. Given a set A = {0, 1}, any collection of zeroes and ones is an element of M(A). Since order does not matter, we can completely describe the element by saying how many it has of each of the two digits. Thus, both {0, 0, 1} and {0, 1, 0} are the same element, which we can denote by 02 11 . Definition 7.65 The multiset of a set A, finite or countably infinite, is denoted by M(A). It is the set of all samples of elements selected from A with replacement, without regard to the order of the sample. To be able to say when two samples are equal, we only need to specify the number of times each element of A appears there. Let ha1 , a2 , . . . i be the array of elements of A with fixed order. An element σ ∈ M(A) is specified by a sequence of integers jk ∈ N, k > 1, where jk is the number of times ak is in σ . Hence its canonical representation ⊳ can be written as σ = a1j1 a2j2 a3j3 . . .. Given a set A, with a weight function w(·) that induces an enumerator ϕA (z), what is the enumerator of M(A)? We ask the same question for the natural weight function over M(A), which is the same one we adopted for the closure and the power set, namely, the weight of a multiset is the sum of the weights of the elements included in it. Thus, the weight of σ as defined by its canonical representation is given by j1 w(a1 ) + j2 w(a2 ) + · · · . The canonical representation also shows the way to find the enumerator of M(A). We use the same approach that was used to find the enumerator of compositions in §7.3: there we multiplied the enumerators of the components. The enumerator of a component was simply the enumerator of the set of values that the component could assume—and here we do the same. Each ak ∈ A contributes its component and we multiply them all, getting

ϕM(A) (z) =

1

∏ 1 − zw(a ) .

a j ∈A

j

(7.37)

Note how close this is to Eq. (7.35). Just as we obtained the result given in Eq. (7.36), from


392

that equation we can find ln ϕM(A) (z) =

1 ϕA (z j ). j j>1

∑

Exercise 7.66 [1+] Complete the derivation of Eq. (7.38).

(7.38)

Let us return briefly to Eq. (7.37). Suppose w(a j ) = 0, for some element of A. What do we get? A division by zero! Since we cannot afford this, we have the same restriction as we had in the operation of substitution: no elements of weight zero may be used to form the multisets. Exercise 7.67 [2] What is a combinatorial reason for requiring |A(0) | = 0 for M(A) to be welldefined? Hint: This constraint is similar to a corresponding restriction on the substitution operation.

7.5 Partitions of Integers A partition7 of a positive integer n is a representation of n as a sum of positive integers, not considering the order of the summands. At first sight, a partition appears to be very similar to a composition of the same number n, as shown in Eq. (7.12). That is, we can represent n as a sum, a1 + a2 + · · · + ak = n, except for the following differences. • The individual summands, ai , called parts (or blocks) rather than components, and they are all positive integers (elements from I1 ≡ Z+ = {1, 2, 3, . . .}). • Order among the parts is disregarded: 5 = 2 + 3 = 3 + 2 are both representations of the same partition of 5 having two parts. • The number of the parts is not fixed.

Thus, a partition of a positive integer n having k parts is an unordered list of k positive integers that sum to n. Since weak compositions also use positive (non-zero) components, they are sometimes called ordered partitions. The study of partitions—their number and structure— has been a major activity in number theory, and they seem to appear sufficiently frequently in analysis of algorithms to deserve consideration here. In addition, partitions provide a different illustration of building enumerators, here ordinary, and in §7.6 of exponential type.

We denote by p(n) the number of partitions of n, and by pk (n) the restriction of these partitions to having exactly k parts. The expression p(n) is known as the partition function. For example, p3 (5) = 2 since 5 = 2 + 2 + 1 = 3 + 1 + 1 are the only ways to get 5 as the sum of three positive numbers, and p(5) = 7, because 5 = 5, 4 + 1, 3 + 2, 3 + 1 + 1, 2 + 2 + 1, 2 + 1 + 1 + 1, 1 + 1 + 1 + 1 + 1. In addition, let pd (n) denote the number of partitions of a positive integer n where the parts are distinct. For example, pd (5) = 3 since 5, 2 + 3, 4 + 1 are the only suitable candidates. Several other counts have been considered (see Example 5.10 on page 205). 7 Some

authors also use the word decomposition.

7.5. PARTITIONS OF INTEGERS

393

We would like to find the enumerator of partitions, as the ordinary generating function of the sequence {p(n)}. When we did this for compositions, the structure of Eq. (7.12) was very suggestive—treating the list of ai as a Cartesian product, the enumerator of each position was immediate from the sum rule. But the way a Cartesian product is defined specifies the order of components, whereas the order of parts in a partition does not matter—this means that we cannot use the product rule in the same way here. n

p(n)

n

p(n)

n

p(n)

n

p(n)

n

p(n)

1

1

11

56

21

792

31

6842

41

4483

2

2

12

77

22

1002

32

8349

42

53174

3

3

13

101

23

1255

33

10143

43

63261

4

5

14

135

24

1575

34

12310

44

75175

5

7

15

176

25

1958

35

14883

45

89134

6

11

16

231

26

2436

36

17977

46

105558

7

15

17

297

27

3010

37

21637

47

124754

8

22

18

385

28

3718

38

26015

48

147273

9

30

19

490

29

4565

39

31185

49

173525

10

42

20

627

30

5604

40

37388

50

204226

Table 393: A brief table of the partition function. Instead we use a multiset operation because a partition may include parts of the same size. Comparing with Definition 7.65 we see that the set of partitions of all integers is simply the collection of all possible multisets over those same integers, M(Z+ ), where Z+ = {1, 2, . . .}.

Thus, Eq. (7.37) provides the enumerator of all possible partitions in the form 1 1 1 1 ··· ··· 2 3 1−z 1−z 1−z 1 − zk = (1 + z + z1·2 + z1·3 + z1·4 + · · · ) × (1 + z2 + z2·2 + z2·3 + z2·4 + · · · )

P(z) =

(7.39)

× (1 + z3 + z3·2 + z3·3 + z3·4 + · · · ) × (1 + z4 + z4·2 + z4·3 + z4·4 + · · · ) × · · · ,

because the multiple (1 − z)−1 enumerates all parts containing 1, the multiple (1 − z2 )−1 enumerates all parts containing 2, and so on. Now we have in principle p(n) = [zn ]P(z). The reason for this supercilious ‘in principle’ is that, unlike the case of obtaining the number of compositions in Eq. (7.14) on page 365, there is no closed form for the coefficient p(n). Using the alternative form of the enumerator, as given in Eq. (7.38), does not help here. Note: Over the years, number theorists have developed an explicit expression for this counting function, but what they found is not closed. Its intricacy is surprising, and we shall not consider it further.8 8 The

story of this effort is developed very engagingly in Andrews’ book [6].


394

The recursive approach for calculating the coefficients p(n) = [zn ]P(z) is one of the practical ways to determine numerically these coefficients. Taking the logarithmic derivative, we obtain ! i i zi−1 i z P′ (z) =∑ =⇒ z P′ (z) = ∑ P(z). i P(z) i>1 1 − zi i>1 1 − z Using Rule 4 from §6.2, page 288, we extract coefficients [zn ]zP′ (z) = n p(n), i zi = [zn ] ∑ i zi i 1 − z i>1 i>1

[zn ] ∑

where σ (n) = [zn ] ∑ i zi i>1

∑ zi j = σ (n), j>0

∑ zi j is the sum of divisors of n. Then using the convolution rule,

j>0

we get the full-history recurrence (which obviously cannot be solved in closed form because we don’t have a formula for the number of divisors of an integer): n

n p(n) =

∑ σ ( j)p(n − j) .

(7.40)

j=1

The recurrence is not really helpful in practical calculations of the (unrestricted) partition function, p(n), because Eq. (7.40) involves another sequence of numbers, σ (n), known as the divisor numbers. In the eighteenth century, L. Euler discovered the fascinating result that these two sequences, p(n) and σ (n), share a common recursive relation (M.27), page 752.

σ (n)

n

Divisors of n

1

1

11

1, 11

12

2

1, 2

3

12

1, 2, 3, 4, 6, 12

28

3

1, 3

4

13

1, 13

14

4

1, 2, 4

7

14

1, 2, 7, 14

24

5

1, 5

6

15

1, 3, 5, 15

24

6

1, 2, 3, 6

12

16

1, 2, 4, 8, 16

31

7

1, 7

8

17

1, 17

18

8

1, 2, 4, 8

15

18

1, 2, 3, 6, 9, 18

39

9

1, 3, 9

13

19

1, 19

20

10

1, 2, 5, 10

18

20

1, 2, 4, 5, 10, 20

42

n

Divisors of n

1

σ (n)

Table 394: Divisors of n and σ (n).

A truncation of P(z) to its first n terms is denoted by Pn (z): Pn (z) =

1 1 1 1 ··· . 1 − z 1 − z2 1 − z3 1 − zn

(7.41)


395

The reason we define the truncated version is that if we wanted to compute p(n), say via MAPLE, Pn (z) suffices: it would produce the same result as using the full-fledged, infinitely long generating function P(z). Exercise 7.68 [2+] Explain why the truncated function (7.41) can be chosen as the enumerator for p(n), the number of partitions of n. r 1 is the number of partitions of n Exercise 7.69 [3] Explain why the coefficient [zn ] ∏ k k=1 1 − z into parts that do not exceed r. In other words, prove that the number of partitions of n into exactly r parts, where r is a given positive integer, is equal to the number of partitions of n into parts the largest of which is r. Also give a combinatorial proof. You were asked about these counts in Exercise 5.11, page 205. Exercise 7.70 [1+] Find an enumerator for qˆr (n), the number of partitions where the largest allowed part k is used at least once (this was also defined in Exercise 5.11), page 205. Exercise 7.71 [3] (a) Explain why the enumerator of partitions with no repeated parts, which is the generating function of {pd (n)}, is given by

ϕd (z) ≡

∑ pd (n)zn = (1 + z)(1 + z2 )(1 + z3) · · · .

(7.42)

n>1

(b) Explain why the enumerator given in part (a) is equal to the enumerator of the power set of Z+ , and why that power set is exactly the set of partitions that part (a) asks about. Exercise 7.72 [4] (M. Berman) A plus or minus sign is assigned randomly to each of the numbers from the set [1..n] = {1, 2, . . . , n}. What are the probabilities that the sum of the signed numbers is positive, negative, or zero? Exercise 7.73 [3+] (a) Write an enumerator for the total number of partitions of n which can be arranged as palindromes. For example, for n = 6, there are seven such, namely: 1,1,1,1,1,1; 1,1,2,1,1; 1,2,2,1; 2,2,2; 3,3; 1,4,1; and 6. (b) A different question asks you to count the number of palindromes which are partitions of a fixed number n. The difference from the previous problem is that for n = 6, for example, we now also have the palindrome 2,1,1,2. As a partition, it is identical with 1,2,2,1, but here we agree to count such palindromes as distinct. Example 7.74 Let us write the generating function for the number of partitions of a positive integer n into even summands. This generating function is the product ge (z) = 1 + z2 + z4 + · · · 1 + z4 + z8 + · · · · · · . 1 1 1 1 1 1 · · · ., and pe (n) = [zn ] ··· . So ge (z) = 1 − z2 1 − z4 1 − z6 1 − z2 1 − z4 1 − z6 1 1 1 = [z6 ] 1 + z2 + z4 + z6 1 + z4 1 + z6 = 3, For example, pe (6) = [z6 ] 2 4 6 1−z 1−z 1−z and indeed 6 = 6; 2 + 4; 2 + 2 + 2. Similarly, let po (n) denote the number of partitions of an integer n into odd summands, n > 1.


396

Then the corresponding generating function is go (z) =

1 1 1 ··· . 3 1 − z 1 − z 1 − z5

Example 7.75 We are going to prove the following claim, which was discovered by L. Euler. For any n ∈ N, the number of partitions of n into odd parts is equal to the number of partitions of n into distinct parts. P ROOF : Using previous results, we need to show that for any such n, [zn ]

1 = [zn ](1 + z)(1 + z2 ) · · · (1 + zn ), (1 − z)(1 − z3 ) · · · (1 − zon )

(7.43)

where on is the largest odd number that is smaller than or equal to n (that is, on = 2⌈n/2⌉ − 1).

The proof that we present is not the briefest possible one, but it is methodologically interesting. Looking at the functions in Eq. (7.43), and fixing n = 1, we see that the hypothesis ? 1/(1 − z) = 1 + z surely fails, but it is true if we qualify it with the phrase “up to powers of z of order two and higher” (which do not contribute to the coefficient of z1 ). This is obviously true since the left-hand side is 1 + z + z2 + · · · . Hence the modified hypothesis ? [z j ]1/(1 − z) = [z j ](1 + z) holds, for j = 0, 1. This will also serve as a basis for a proof by induction. To complete the proof, we introduce the notation {zk } for an arbitrary expression that has in its power series expansion no powers smaller than k. For example: x + 2x2 is in {x}, but not in {x2 }, nor in {x3 }. Thus the qualified statement above is equivalent to writing 1 = 1 + z + {z2 }. 1−z

Another way of saying this is that if we take the denominator of the left-hand side and multiply by the right-hand side, we get 1, up to {z2 }. Let us define the functions on the two sides of Eq. (7.43) def

fn (z) =

1 , (1 − z)(1 − z3 ) · · · (1 − zon )

def

gn (z) = (1 + z)(1 + z2 ) · · · (1 + zn ).

(7.44)

We show a somewhat stronger claim than in Eq. (7.43): Claim 1: [z j ] fn (z) = [z j ]gn (z) for all j ∈ [0..n]. The key fact is that gn (z)/ fn (z) = 1 + {zn+1 }.

(7.45) 2k

. Then [z j ]gn (z) = [z j ]g(z) ˜ for To show this fact, define g(z) ˜ = ∏k>1 (1 + zk ) = ∏k>1 1−z 1−zk 0 6 j 6 n. The function g(z) ˜ can be rearranged by canceling terms with even powers of z: g(z) ˜ =

(1 − z2 )(1 − z4 )(1 − z6 ) · · · (1 − z)(1 − z2 )(1 − z3 )(1 − z4 ) · · ·


=

397

(1 − zon +1 )(1 − zon +3 ) · · · 1 = fn (z) 1 + {zn+1 } . 3 o o +1 o +2 (1 − z)(1 − z ) · · · (1 − z n ) (1 − z n )(1 − z n ) · · ·

Therefore [z j ] fn (z) = [z j ]gn (z) for 0 6 j 6 n, which includes the particular case of j = n, as required by Eq. (7.43). To finish the prove of Claim 1 we need a more detailed claim: Claim 2: The functions gn (z) and fn (z), defined in Eq. (7.44), satisfy the relation n

gn (z)/ fn (z) =

∏ j=u(n)

1 − z2 j ,

u(n) = n − ⌈n/2⌉ + 1.

(7.46)

Exercise 7.76 [3] Prove claim 2 by induction on n. Note that 2u(n) = n + 1 when n is odd, and it equals n + 2 when n is even. Claims 1 and 2 together provide the required proof of Eq. (7.43). Example 7.77 Let us determine an enumerator (denote it by T2 (z)) for partitions in which no part may be repeated more than twice. Example: calling the counter of this type of partition p(2) (n), we see that p(2) (5) = 5 because 5 = 4 + 1 = 3 + 2 = 3 + 1 + 1 = 2 + 2 + 1. To construct T2 (z), we delete from each of the sums in the parentheses of Eq. (7.39) all but the first three terms, since no integer is allowed to appear in the partition more than twice. Hence it has the following form: T2 (z) = (1 + z + z2 )(1 + z2 + z4 )(1 + z3 + z6 )(1 + z4 + z8 ) · · · .

(7.47)

We would like to convert the enumerator into the form T2 (z) =

1 (1 − za1 )(1 − za2 )(1 − za3 ) · · ·

because it then tells us that we choose partitions from some fixed set of integers, which we denote by A = {a1 , a2 , a3 , . . .}, where a1 < a2 < a3 < . . .. In order to determine the values a j , j = 1, 2, 3, . . ., we multiply T2 (z) by 1 − za1 to obtain (1 − za1 )T2 (z) = (1 − za1 )(1 + z + z2 )(1 + z2 + z4 )(1 + z3 + z6 ) · · · =

1 . (1 − za2 )(1 − za3 ) · · ·

The term with the lowest exponent of z in the left-hand side must be equal to the term with the lowest exponent of z in the right-hand side. Hence a1 = 1 because the right-hand side power series has the lowest term za2 . And so we get (1 − z)T2 (z) = (1 − z)(1 + z + z2 )(1 + z2 + z4 )(1 + z3 + z6 ) · · · =

1 (1 − za2 )(1 − za3 ) · · ·

.

Now again, the term with lowest exponent of z on the left-hand side, which is z2 , must be equal to the term with the lowest exponent of z on the right-hand side, which is za2 . Therefore


398

a2 = 2 and we again multiply both sides by (1 − z2 ). Continuing in a similar fashion, we find a3 = 4, a5 = 7, a6 = 8, a7 = 10. It would appear that the set A consists of the numbers not divisible by 3. So we have to show that the generating function T2 (z) has the following form T2 (z) =

1 1 = . (1 − z)(1 − z2 )(1 − z4 )(1 − z5 ) · · · ∏ j>1 (1 − z3 j+1 )(1 − z3 j+2 )

Replacing each triple expression 1 + z p + z2p by the ratio

1−z3p 1−z p ,

we get

T2 (z) = (1 + z + z2 )(1 + z2 + z4 )(1 + z3 + z6 )(1 + z4 + z8 ) · · · 1 − z3 1 − z6 1 − z9 1 − z12 1 − z15 ··· = 1 − z 1 − z2 1 − z3 1 − z4 1 − z5 1 1 = . = (1 − z)(1 − z2 )(1 − z4 )(1 − z5 ) · · · ∏ j>1 (1 − z3 j+1 )(1 − z3 j+2 ) The next exercise asks you to generalize this result. Exercise 7.78 [1+] (J. W. L. Glaisher) Prove that the number of partitions of n, where each part appears at most m times, is equal to the number of partitions of n into parts that are not divisible by m + 1. Also show an explicit bijection between these two sets. Exercise 7.79 Determine the enumerator for the number of partitions when each part appears (a) [1] exactly twice; (b) [1+] at least twice; (c) [1+] at least once but no more than three times; (d) [1] at least twice but no more than three times; (e) [1+] no part appears exactly twice; (f) [1+] no part appears exactly two or three times. Example 7.80 We derive a remarkable formula for pd (n), the number of partitions of n (n > 0) into distinct parts, in terms of p(n), the number of (unrestricted) partitions of n: ∞

pd (n) =

∑ (−1)i p

i=−∞

∞ n − (3i2 + i) = p(n) + ∑ p(n − (3 j2 + j)) + p(n − (3 j2 − j)) , j=1

(7.48) where the former sum extends over all integers i except i = 0, for which the arguments of the partition function are non-negative. Its derivation is based on Eq. (L.88), called the Euler pentagonal number theorem [30]. The generating function for the number of partitions into distinct parts, pd (n), has been established in Eq. (7.42), and we develop it as follows: ∞

ϕd (z) =

∑

n>1 ∞

=

pd zn = ∏(1 + zi ) i=1

∞

∏(1 − z j )−1 ∏(1 − zi )(1 + zi ) = j=1

i=1

∞

∑

j=0

p( j)z j

!

∞

∏(1 − z2i ) i=1

7.5. PARTITIONS OF INTEGERS ∞

=

∑ p( j)z j

j=0

!

∞

399 (3i2 +i)/2

∑ (−1)i (z2 )

i=−∞

!

∞

=

∑ p( j)z j

j=0

!

∞

3i2 +i

∑ (−1)i z

i=−∞

The result follows by equating coefficients of zn on both sides of this convolution.

!

.

One of the varieties of partition counts we defined was pk (n), the number of partitions of n with exactly k parts. In order to count elements of such a doubly-indexed set, we need to create an enumerator with two variables, and we denote it by P(u, z), so that pk (n) = [uk zn ]P(u, z). To see how to create this enumerator, let us look first at a simpler one, 1/(1 − z j ), that enumerates parts containing j ( j > 0). Now suppose we insert a u to get 1 = 1 + uz j + u2 z2 j + u3 z3 j + . . . . 1 − uz j This is exactly what we need. The power of u tells us the number of times the part j is being used. Hence, the desired complete enumerator is P(u, z) =

1 1 1 1 ··· ··· . 2 3 1 − uz 1 − uz 1 − uz 1 − uz j

(7.49)

Example 7.81 What is the value of p2 (4), the number of partitions of 4 with exactly 2 parts? According to the last relation it can be written as p2 (4) = [u2 z4 ]

1 1 1 1 . 2 3 1 − uz 1 − uz 1 − uz 1 − uz4

The calculation can be done by MAPLE as follows. First, define the function f := 1/( (1-u*z)*(1-u*zˆ2)*(1-u*zˆ3)*(1-u*zˆ4) ); and then request mtaylor(f, [u,z],7); this produces the result 1 + uz + uz2 + u2 z2 + uz3 + u2 z3 + uz4 + u3 z3 + 2u2 z4 , which tells us that the answer to our question is 2, as expected. Extraction of such a coefficient by hand, even with these low values, is tedious, and a computer algebra system (like MAPLE) is recommended. Nevertheless, it is beneficial to see it done once in detail, with most of the intermediate steps explicit. Our process is in fact a depth-first evaluation of the expansion tree of the computation. The first step is to pick, say, the first factor and observe that when we expand it as 1 + uz + u2 z2 + · · · , only the first three terms contribute—the power of u is too large in the rest; a similar consideration of z is the reason why only the first four factors of P(u, z) are used now. Hence, 1 1 1 p2 (4) = [u2 z4 ](1 + uz + u2 z2 ) 2 3 1 − uz 1 − uz 1 − uz4 1 1 1 def = A = [u2 z4 ] 2 3 4 1 − uz 1 − uz 1 − uz 1 1 1 def 3 = B +[uz ] 1 − uz2 1 − uz3 1 − uz4

400

CHAPTER 7. ENUMERATION WITH GENERATING FUNCTIONS +[u0 z2 ]

1 1 1 2 3 1 − uz 1 − uz 1 − uz4

def

=

C.

where the A, B,C are simply identifiers that allow us to name the contributions of these three lines. Part C is the easiest: since any occurrence of z requires u to be present, and it is disallowed in this part, C contributes just 0. For B we do a similar development and find 1 1 3 1 − uz 1 − uz4 1 1 def = B1 = [uz3 ] 3 4 1 − uz 1 − uz 1 1 def = B2 . +[u0 z] 3 4 1 − uz 1 − uz

B = [uz3 ] (1 + uz2 )

Nothing fits B2 and it does not contribute a thing. For B1 we find B1 = [uz3 ] (1 + uz3 )

1 1 − uz4

1 def = B11 1 − uz4 1 def = B12 . +[u0 z0 ] 1 − uz4

= [uz3 ]

Here B12 gives 1, and B11 contributes zero. Hence B1 evaluates to 1, and B contributes just that—a total of 1. The same stepwise refinement is done for part A: A = [u2 z4 ] (1 + uz2 + u2 z4 )

1 1 3 1 − uz 1 − uz4

1 1 def = A1 3 4 1 − uz 1 − uz 1 1 def = A2 +[uz2 ] 3 4 1 − uz 1 − uz 1 1 def 0 0 = A3 . +[u z ] 1 − uz3 1 − uz4

= [u2 z4 ]

The part A3 brings a 1; nothing fits the needs of A2 , and A1 is incompatible with the remaining terms as well. Hence the total for A is 1, as it is for B, and we get p2 (4) = 2, which is true since the only two-part partitions of 4 are 1,3 and 2,2. Exercise 7.82 [3+] (D. Knuth) Suppose n people are sitting at a table, which can be either circular or straight. Let em,n (correspondingly, fm,n ) denote the number of ways to partition them into m affinity groups with no two members of a group seated at circular table (correspondingly, in a row) next to each other. (For example, e3,4 = 2, e3,5 = 5, f3,4 = 3, f3,5 = 7, and e3,6 = 10.) For m > 2, find the generating functions Em (z) = ∑n>0 em,n zn , Fm (z) = ∑n>0 fm,n zn for each sequence. Exercise 7.83 [3] Compute the following sum, which arises in considering the lengths of runs of identical characters in strings. The sum extends over all possible k-partitions of n:


401

∑{ni } ∏i1ni ! , where the ni are non-negative integers specifying the number of parts of size i, hence ∑i ni = k, and ∑i ini = n. Hint: Multiply the summand by k! and look at Eq. (2.46). Exercise 7.84 [4] (adapted from Bender & Williamson [15]) This exercise is about counting paths in the plane grid of integers (lattice points), as shown in Figure 51. In particular, we find an , the number of paths from (0,0) to (n, n) that stay above the diagonal i = j, where we use i and j to mark the horizontal and vertical axes, respectively, except touching at the two end points. (a) Show that an is also the number of such paths that remain under the diagonal, except touching at the end points. Prove such equality by showing a one-to-one correspondence between the two sets of paths. (b) Show that an is also the number of different sequences of ±1s, which we denote by {si }16i62n , and which contain exactly n ‘+1’s and n ‘−1’s, restricted by the requirement that k si > 0, for 1 6 k < 2n; that is, the first 2n − 1 partial sums of the si are strictly positive. ∑i=1 (c) Looking at the sequences of part (b), we see that there is one sequence of length zero (corresponding to n = 0). Any other sequence starts with s1 = 1. Hence you can view the rest of the sequences as all possible ways to bring first the sum down by −1 in 2n − 1 steps. By partitioning all sequences according to k, the size of the prefix of the sequence that is all +1s, show that the enumerator of the set of all such sequences satisfies A(z) = 1 + ∑k>1 zk z(A(z)z)k−1 = 1 + z2 1−z21A(z) . (d) What is an , using the last relation, in part (c)? This is a Catalan number! Hence the number of the sequences described in part (b) is also the number of n-node binary tree. Show a one-to-one correspondence between the two sets, of sequences and trees. Exercise 7.85 [4] (adapted from Bender & Williamson [15])

6

-

Define a pile to be an arrangement of identical square tiles on the plane. The arrangement of a pile of r rows is determined by the 2r numbers 0 = a1 6 a2 6 · · · 6 ar−1 6 ar < br 6 br−1 6 · · · 6 b2 6 b1 . Row i has bi − ai tiles. You can imagine the tiles to be arranged regularly, in rows and columns, and stacked so that row i begins in column ai (but this is just for easy visualization. Note that the inequalities mean that the lengths of successive rows cannot increase).

Pile of 16 tiles for Exercise 7.85. The enclosed figure describes a pile for a = (0, 1, 1, 3), b = (6, 5, 5, 5). Thus two piles of n r (bi − ai ). Let tiles in r rows differ only if their {ai } and {bi } sequences differ, while n = ∑i=1 tn,r be the number of distinct piles of n tiles in r rows, and tn be the total number of n-tile piles. Develop one- and two-variable enumerators for the set of all piles. Every partition can be uniquely identified as the following definition suggests; we just list the parts in sorted order, and increasing order appears to be natural. Definition 7.86 A canonical representation of a partition, which includes p1 times the part


402

1, p2 times the part 2, and so on, is written as n = (1 p1 , 2 p2 , 3 p3 , . . .).

(7.50)

⊳

Parts of zero size are omitted, For example, the partitions of 5 listed at the beginning of this section would be, in this notation: 5 = (5), (1, 4), (2, 3), (12 , 3), (1, 22 ), (13 , 2), (15 ). It is also possible to use an exponential enumerator (which we consider in the next section) for partitions: 2 3 ˆP(z, u) = exp u1 z + u2 z + u3 z + · · · , 1! 2! 3!

where u = (u1 , u2 , . . .). We mark by u j the number of parts of size j in the canonical representation (7.50). Extracting the coefficient, we find the number of partitions with p1 parts of size 1, p2 parts of size 2, and so on, to be ˆ u) = [zn ][u1p1 ][u2p2 ] · · · P(z,

n! . p1 ! p2 ! · · · pk (1!) p1 (2!) p2 · · · (pk !) pk

Example 7.87 (A Related birthday problem) Let λ = {λ1p1 , λ2p2 , . . . , λkpk } be a (canonical) representation of a partition of the number n. For example, suppose that in a class of n students, we have two persons who were born on April 29, three on June 11, four on September 8, and another four on December 15 with the remainder all having different birthdays. Then these data can be represented by the partition λ = { 1n−4 , 2, 3, 42 }.

Therefore, any distribution of birthdays is associated with a partition of n; in particular, a distribution of all distinct birthdays is represented by {1n } and one in which all birthdays are the same by {n}. For any partition λ , let π (n, λ ) be the probability of birthday distributions, which are represented by the partition λ of n. We use the notations from §7.6.4 and assume that there are m days in a year; the number of people in a group, n, is considered to be n 6 m. We denote by Pr (m, n) the probability that at least r out of n randomly selected people have the same birthday. Let nr be the least number of n such that Pr (m, n) > 1/2 (known as the median). Then for r = 2 and r = 3 we get P2 (m, n) = 1 − π (n, {1n }) , P3 (m, n) = P2 (m, n) − Pr[at least one pair but no triple share a birthday] and so P3 (m, n) = P2 (m, n) − π (n, {1n−2 , 2}) − π (n, {1n−4 , 22 }) − . . . − π And in general,

o n . 1n−2⌊n/2⌋ , 2⌊n/2⌋

Pr (m, n) = Pr−1 (m, n) − Pr[at least one set of r − 1 people share a birthday, but no r] .

7.6. EXPONENTIAL ENUMERATORS

403

Recall that pn denote the total number of partitions of n. Exercise 7.89 shows that Pr (m, n) = Pr−1 (m, n) −

∑

π (n, λ ) .

(7.51)

λ ∈p(n) λr =n−1

Eq. (7.51) gives the basis for numerical calculations that lead to known results for m = 365: n2 = 23,

n3 = 88,

n4 = 187,

n5 = 313,

n6 = 460

because P2 (365, 22) ≈ 0.475695, P3 (365, 87) ≈ 0.499455, P4 (365, 187) ≈ 0.495826, P5 (365, 311) ≈ 0.496200,

P2 (365, 23) ≈ 0.507297, P3 (365, 88) ≈ 0.511065, P4 (365, 188) ≈ 0.502685, P5 (365, 312) ≈ 0.501074.

Calculations of nr for r > 6 should be done with special care because in this case the number of people in a group exceeds the number of days in a year. Exercise 7.88 [3] If λ = {λ1p1 , λ2p2 , . . . , λkpk } is a partition of n = p1 + p2 + · · · + pk , then prove that m! n! · . mn π (n, λ ) = p p 1 k (λ1 !) . . . (λk !) p1 ! p2 ! . . . pk !(m − ∑ki=1 pi )! Exercise 7.89 [3] Prove Eq. (7.51).

7.6 Exponential Enumerators Following Definition 7.2, page 356, where the exponential enumerator was introduced, we said that it would prove useful for the counting of labeled structures. Now we begin to make good on that promise. In truth, this only holds in the (restricted, but still important) case where the weight of each element, in the sets we count, is equal to the number of labels it carries. This is certainly a limitation, but it turns out that in many situations arising in our analyses, this is natural. Without any loss of generality, we can assume that the set of labels coincides with the set of positive integers. Then permutations, surjections, set partitions, and words from some language can be viewed as labeled structures. There are many examples when atoms of an object are distinguishable by bearing labels; a typical class of such objects is the set of graphs. In this section, we deal mostly with labeled constructions of words. Nevertheless, we mention several other examples of structures with labels. (1) A graph; the labels are on nodes, and the weight of the graph is its node count. (2) A collection of sets of distinct elements; their identities provide the labels, and the weight of a set is equal to its cardinality. (3) Finally, consider the set of permutations of n distinct elements, also referred to as the symmetric group. We can see such permutations as linear graphs, with labels from the set

404


[1..n]. Since there are n! members in the symmetric group, its exponential enumerator is 1 . n! × zn /n! = zn , and the enumerator of the entire class is then ∑n>0 zn = 1−z (4) For a ‘misleadingly similar’ example, consider the class P of circular permutations— permutations arranged along a circle—which we consider invariant under rotations. Therefore such an arrangement of n atoms has only (n − 1)! different possible labels, and its enu1 . merator is then ϕˆ P(z) = ∑n>1 (n − 1)!zn /n! = − ln(1 − z) = ln 1−z

7.6.1 The Sum and Product of Labeled Structures We note that the sum rule for unlabeled sets rolls over into labeled ones with little change because either ordinary or exponential enumerators can be used, and they just need to be added up. The proof used for Theorem 7.7 on page 360 goes through without change for exponential enumerators. However, the union of disjoint labeled sets is the structure that is bestowed by distinguishable labels. Therefore, their union requires an additional mechanism of redistribution of labels because labels of elements need to be distinct. For example, if a finite set A uses labels from [1..n] (the cardinality of A could be less than n because some elements may have more than one label) and the finite set B has its collection of labels in [1..m], then their union A ∪ B should be labeled by a set of cardinality n + m (which may be chosen from [1..n + m]), preserving the original label order for A and B. The important issue to keep in mind is that enumerators are tools. A tool is good if we can use it to produce results that are hard to get otherwise. In particular, the usefulness of enumerators depends on their being able to capture natural set operations. Looking at Cartesian products of labeled sets, which leads to convolutions of the set counts, we find that things change in a significant way. Which enumerator fits which kind of convolution? The OGF (and hence the ordinary enumerator) corresponds to the Cartesian product in Eq. (7.8) because it can handle a convolution of sequences, as seen in Table 275. Similarly, the exponential enumerator captures the product of labeled structures because it handles the binomial convolution of sequences, as given in Table 278. Sets that satisfy the weight-to-label restriction, produce such a convolution when we create their Cartesian product. Let A and B be sets of such labeled elements, and C = A × B their Cartesian product. We denote by ak , bk , or ck the number of elements in A, B, or C with weight k, respectively. How do we calculate ck ? We count elements in C of weight k, and each such element is created by combining an element from A(i) (the subset of A containing elements of weight i) and an element from B(k−i) , for 0 6 i 6 k. Now, there are i! elements in A(i) , which are identical except for the arrangements of their i labels.9 Each element can come with i! different arrangements of labels, and similarly, we get the same the B-element (k − i)! times, once labels are removed. Then we have k! ways to distribute the labels on the newly created C-element, but we only use ki of them since in the redistribution of labels we

9 We make an implicit assumption that there are no symmetries in those sets that reduce the element count. The issue of counting sets with symmetries is somewhat more involved; it is often called Pólya theory of counting, and we do not deal with it now.


405

keep the original order! Hence, for a fixed i, there are ki ai × bk−i different elements in the set C(k) , and their total number is: k ai bk−i k! = ∑ ai bk−i =⇒ ϕˆC (z) = ϕˆ A (z)ϕˆ B (z), (7.52) ck = ∑ i i i i! (k − i)! as was shown in the derivation of Eq. (6.11). Multiplication of two sets with labels requires the special definition: the labeled Cartesian product, an old term for it is the partitional product. Each of its element is an ordered pair with all possible labels preserving the order. For instance, if an element a ∈ A has two labels and an element b ∈ B has three labels, then their partitional product has 2 × 3 = 6 elements (a, b) with all possible labels taken for a and for b. Consider another example: we multiply sets of circular graphs, which we would 4draw as 3 polygons. For example, here is a square from A(4) (note the order of labels): 1 2 and a 3 triangle from B(3) : 1 2 . Their products needs 7 labels and in one of the 73 allocations of labels, the6 square gets 2,4,5,6 and the triangle gets 1,3,7; then a proper relabeling would 5 7 produce 2 4 . 1 3; these allocations of labels can be seen as partitions of the set [1..7]. Our treatment of the Cartesian product assumes that the pairs are ordered: ha, bi 6= hb, ai. Frequently, the different positions are used for elements from different sets, in which case the reversed order is not even well-defined. When this is not the case, then both orders represent two ways of writing the same element (especially common when we are multiplying a set with itself, as in Exercise 7.99); we need to recognize this and divide the product count we obtained by 2; if it is a k-fold product, divide by k!. Such a product is called the labeled Abelian product. There is no standard notation to distinguish between Abelian and Cartesian products, and we continue to use the same terminology for both, expecting context to make the meaning clear in each case. A prime example of an Abelian labeled product is as follows. We consider the class of atoms T, allow an atom to carry a label and have weight 1. The enumerator of an atom is then a(z) ˆ = z. Then we create Sk , a set of k atoms by using the Abelian product of such k single-atom sets. The product will have the desired unordered k distinct labels, and the enumerator φˆSk (z) = aˆk (z)/k! = zk /k!. The enumerator of the class S, the (disjoint) union of all possible sets is then ∑k>0 zk /k! = ez (there is one of each size).

Life gets more interesting when we consider sets of structures with richer inner complexity than a solitary atom, which we substitute into the sets, exactly in the manner outlined in §7.4.1. Just as in the unlabeled world, the Cartesian product gives rise to the closure, when labeled structures are considered. The distinction between the simple labeled product and the Abelian labeled product is important. The k-fold convolution of a set A produces the enumerator ϕˆ Ak (z), and therefore

ϕˆ A∗ (z) =

1

∑ ϕˆ Ak (z) = 1 − ϕˆ A(z) ,

k>0

Kleene closure.

(7.53)


406

For the Abelian product, however, the k-fold convolution of a set A produces the enumerator ϕˆ Ak (z)/k!, and therefore

ϕˆ A∗ (z) =

ϕˆ Ak (z) = eϕˆ A (z) , k! k>0

∑

Abelian closure.

(7.54)

Example 7.90 (Sets and products of sets) A function from some domain into a discrete range—which can in principle be finite or infinite, but in the current context of counting it is only meaningful for a finite range—is called a surjection if every element of the range is the image of at least one input entry. (A bijection is the special case when each element of the range is the image of exactly one input.) In the set of functions from A to B (with cardinalities |A| = n, |B| = m) we consider its subset, fn,m , that consists of surjections from A on B. In the next chapter, we look at such functions in the context of placing balls in bins, but here we keep the abstract view. Let an,m = | fn,m | be the number of such functions. Clearly an,m = 0 for n < m, and am,m = m!. We construct the collection of functions we need by concentrating on the set B, and saying that each element in it ‘selects’ a set of elements (of size at least one) from the set A, without replacement. Let f be such a function, then it is determined by listing the sets that are mapped to each bi ∈ B in order: h f −1 (b1 ), f −1 (b2 ), . . . , f −1 (bm )i. For example, if m = 4, then ({3, 7}, {1}, {4, 8, 9}, {2, 5, 6}) specifies such a function from [1..9] to B. What we have constructed is a 4-fold Cartesian product of nonempty sets from N. Since elements of N are labeled as are the elements of B, both need exponential enumerators. For the sets we need the Abelian variety; there we get the enumerator ez − 1, deleting the contribution of the empty set from the enumerator, and its m-fold product will give us ϕˆ m (z) = (ez − 1)m . Then n![zn ]ϕˆ m (z) is the number of surjections from a set of size n on a set of size m: m kz m n n m−k m−k an,m = n![z ](e − 1) = n![z ] ∑ e (−1) =∑ k (−1) = m! , (7.55) k k m k k n

z

m

n

using the representation in Eq. (C.20) for Stirling numbers of the second kind. Look again at the m-long ‘vector’ representation we used for a surjection. Suppose we give up the order; what do we have? A collection of sets, with no repeated labels (because the underlying mapping was produced by a function), or, in other words, m sets of non-empty sets, or a partition of a set of size n into m components. Now the m-fold Abelian product 1 (ez − 1)m . of the non-empty set enumerator, ez − 1, will have the enumerator ϕˆ m (z) = m! n Then extraction of the coefficient of z will proceed identically, and yield the same result as in Eq. (7.55), but divided by m!. Hence we get mn , which recalls the definition of those numbers in §5.8.3 (page 261). Consider what happens when we look at surjections from [1..n] on sets of any size m > 0. For any n > 1, the number of possible surjections (or partitions) is finite, since none is possible as soon as the number of partitions m > n. Therefore we can talk about, and write enumerators


407

for, the sets of all surjections S and all set partitions P, and they will have the enumerators

ϕˆ S(z) =

1

∑ (ez − 1)m = 2 − ez ;

ϕˆ P(z) =

m>0

1 z z (e − 1)m = ee −1 . m>0 m!

∑

(7.56)

Exercise 7.91 [2] Using the results in Eq. (7.56), obtain the total possible number of surjections from the set [1..n], and the number of ways to partition it to any number of non-empty sets (the Bell number). Example 7.92 Collision resolution interval duration. The following problem arises in the analysis of the capacity of random access channel, where collisions are resolved using a “tree algorithm.” The following treatment is adapted from [64, Exercise 2.2.11], which contains more information about the network and protocol. The analysis is bases on the following recurrence, for the length of time, measured in time-slots, required to resolve a collision of n broadcasters: Ln = 1 + LI + Ln−I ,

n > 2,

L0 = L1 = 1,

(7.57)

where I is a random variable, with the binomial distribution B(n, p). The probability p is a parameter of the protocol. The given recurrence can be used to write a PGF for Ln , but this example only looks to recover def the expected collision resolution duration, that is, ln = E[Ln ]. Since the random variable I in Eq. (7.57) is the only stochastic element in it, we get a recurrence for the means ln by randomizing on I and taking expectation on the occurances of L there, to get n i n−i ln = 1 + ∑ p q (li + ln−i ), n > 2, q = 1 − p, l0 = l1 = 1. (7.58) i i The binomial coefficient in the recurrence suggests the use of EGF in solving it, and we define ˆ = ∑n>0 ln zn . Using it over the last recurrence leads to the unusual equation for this GF: l(z) n! ˆ = ez − 2 − 2z + e pz l(pz) ˆ ˆ l(z) + eqz l(qz).

(7.59)

The unusual feature of this equation is that it is not a local equation. It connects values of ˆ in three different locations. We do not know how to solve a non-local equathe function l(z) tion, but our true interest is not in the function itself, but only in its power series expansion ˆ coefficients, and these are available with the help of a small device. Let the function b(z) be −z ˆ ˆ the Poisson GF of the sequence {ln }, defined as b(z) = e l(z). Introducing this new function into Eq. (7.59) leads to the somewhat simpler equation ˆ − b(pz) ˆ ˆ b(z) − b(qz) = 1 − 2(1 + z)e−z ,

ˆ b(0) = 1,

bˆ ′ (0) = 0.

(7.60)

It does not seem we have gained much since this is a non-local equation as well. . . This is misleading, since we can now extract coefficients from both sides of Eq. (7.60)to obtain  1 n=0   n n n (−1)k (k − 1) def k n ˆ . bn = [z ]b(z) = 0 n = 1 =⇒ ln = ∑ bk n = 1 + 2 ∑  1 − pk − qk  2(−1)n (n−1) k=0 k=2 k n!(1−pn −qn ) n > 2

408


It is not easy to see from this expression that ln is essentially linear in n; this is the topic of Exercise 10.52, page 575.

7.6.2 Permutations and Cycles Every permutation has a cycle structure. Here is an example: the permutation h 5, 2, 4, 6, 1, 3 i of [1..6] specifies that 2 stays in its place, 1 is moved to the 5th place, while 5 occupies the first position, and then 3 is moved to the 6th spot, 6 to the 4th, and 4 lands in position 3. We have described the cycles (2), (1,5), (3,6,4). More generally, given a permutation σ of [1..n], pick a value j1 , which is placed by the permutation in position j2 = σ ( j1 ); continue to form the sequence jk = σ ( jk−1 ) until you reach a jr such that j1 = σ ( jr ). Thus we complete an r-long cycle. If r < n you can pick another value which was not in the first cycle and let it trace its own cycle under σ . In general we continue until [1..n] is exhausted. Exercise 7.93 [1] Explain why, in the description of the sequence { jk } just given, j1 must be the first repeated value, rather than, say, j3 . Using the cycles thus discovered we say that we have a cycle representation of the permutation, for example, (2)(15)(364) for the one above. However, for any combinatorial purpose we need a unique form, and since cycles stay the same when rotated and since disjoint cycles commute, (51)(436)(2) would still represent the same permutation while looking quite different. The canonical form of this representation is obtained by rotating each cycle until it is led by its smallest element, and sorting the cycles by these leading values (in any order, but by convention, in descending order). For the last example this provides (364)(2)(15). Note that we can remove the parentheses and still be able to interpret it correctly and unambiguously by observing the drops that are less than all of its predecessors. In the sequence (364), the last term, 4, is not a drop because it is bigger than 3, the leading number. Given such a representation, how many cycles are there? We can simply scan it, from left to right, and count the number of maxima we encounter, exactly as in Example 4.159, on page 190. There we only calculated the moments of the number of cycles in a random permutation of [1..n], and now we want to look for their distribution. One way to do it is by developing a recurrence and solving it (see [64, §4.1]). However, enumerators provide an interesting and more direct way. Under this view, a permutation is a set of cycles. The class of cycles, P, was briefly considered earlier in this section, page 404, where we 1 . So, n! [zn ]ϕˆ P(z) = (n − 1)! is saw that its exponential enumerator is given by ϕˆ P(z) = ln 1−z the number of circular permutations. Then its Abelian product, P, will have the enumerator 1 1 ) = 1−z , as expected. But to count the number of cycles in ϕˆ P(z) = exp(ϕˆ P(z)) = exp(ln 1−z a permutation we have to use an additional variable, u, to mark the number of cycles, and therefore we find

ϕˆ P(u, z) = exp(uϕˆ P(z)) = e−u ln(1−z) = (1 − z)−u .

(7.61)

This function, which was also derived in Exercise 6.29 on page 285, tells all about relations


409

between cycles and permutations. Thus, the exponential generating function (EGF) of the number of permutations consisting of exactly d cycles is given by d![ud ](1 − z)−u , and now we retrace the steps above and find d![ud ](1 − z)−u = d![ud ] exp(−u ln(1 − z)) = (− ln(1 − z))d . Looking at Eq. (C.40) in Appendix C, we discover that this is the EGF (over the upper argument) of the signless Stirling numbers of the first kind. Coefficients extracted from it are quite naturally called the cycle numbers: this tells us how manypermutations of n distinct numbers we can generate with d cycles: n![zn ](− ln(1 − z))d = d! dn . On the other hand, recovering from ϕˆ P(u, z) the coefficient of zn /n! will provide us with the EGF of the number of cycles of any given size in the symmetric group: (−u)n n −u n −u = un = (u + n − 1)n , n![z ](1 − z) = n!(−1) = n!(−1)n n! n where we have used the relations in Eq. (2.40) on page 47. Example 7.94 Our next example is concerned with counting permutations of [1..n] with a specified number of cycles of a given length 1. A cycle of unit length is also called a singleton or a permutation with a fixed point (see Definition 5.131 on page 258). Let cn,k be the number of permutations that contain exactly k such cycles (singletons). For k = 0 we called such permutations in Example 5.132 (page 258) derangements, and the number of such permutations was denoted by cn,0 = Dn . Let us go over some initial values for n = 4. The number of permutations that contain exactly 1 singleton cycle is c4,1 . A fixed point can be chosen in 41 ways. The only way to avoid additional singletons is to have the remaining elements form derangements (why?). We can do this in c3,0 = D3 = 2 ways, for a total of 4 c4,1 = c1,1 × c3,0 = (4 × 1) × 2 = 8 1 permutations that have exactly 1 singleton cycle since c1,1 = 1. Similarly, there are 3 1 4 c1,1 × c1,1 × c2,0 = 6 c4,2 = 2! 1 1 permutations that contain exactly 2 fixed points. We notice that a 4-permutation cannot contain exactly 3 singleton cycles (the fourth element would be perforce a singleton as well). Indeed, if we use the same template to compute their number we find 3 2 1 4 c1,1 × c1,1 × c1,1 × c1,0 = 0, c4,3 = 3! 1 1 1 because c1,0 = 0. Finally, 3 2 1 1 4 c1,1 × c1,1 × c1,1 × c1,1 × c0,0 = 1, c4,4 = 4! 1 1 1 1

410


a single permutation is made of all singleton cycles—it is the identity transformation. This analysis is generalized to permutations of [1..n]. The number of permutations: n n cn,1 = c1,1 × cn−1,0 = cn−1,0 with exactly 1 singleton cycle; 1 1 n−1 n 1 n c1,1 × c1,1 × cn−2,0 = cn−2,0 with 2 singleton cycles. cn,2 = 2! 1 1 2 And in general, we have the formula for k-singleton permutation counts, which provides a relation for the derangement count, n n n cn,k = cn−k,0 =⇒ cn,0 = n! − ∑ cn−k,0 . k k=1 k The corresponding EGF for the sequence {cn,0 } of derangements is n n zn zn zn zn ˆ cn−k,0 C0 (z) = ∑ cn,0 = ∑ Dn = ∑ n! − ∑ ∑ n! n>0 n! n>0 n! n>0 k=1 k n! n>0 =

cn−k,0 zn−k 1 zk zk 1 −∑ ∑ = − Cˆ0 (z) ∑ . 1 − z k>1 k! n>k (n − k)! 1−z k>1 k!

Solving this equation for Cˆ0 (z), we obtain the exponential enumerator for the number of derangements: zn e−z zn . (7.62) Cˆ0 (z) = ∑ cn,0 = ∑ Dn = n! n>0 n! 1 − z n>0 Using the relation cn,k = nk cn−k,0 , we find the exponential generating function of the sequence {cn,k } to be Cˆk (z) =

∑ cn,k

n>0

zk e−z n zn zn =∑ . cn−k,0 = n! n>0 k n! k!(1 − z)

(7.63)

Note that we can do better: the relation cn,k = nk cn−k,0 , tells us that the singleton counts are a binomial convolution of derangement counts and the all-1s sequence. The all-1s sequence has the exponential generating function ez . Since we want the bivariate generating function, denoted by G(u, z), on length of the permutations (marked with a z) and the singleton count ˆ z)Cˆ0 (z), where the EGF for singletons is C(u, ˆ z) = euz . (marked with a u), then G(u, z) = C(u, Let Rn (u) be the ordinary generating function for the sequence {cn,k } with respect to the second index. Then Rn (u) = ∑ cn,k uk , and the bivariate generating function becomes k>0

G(u, z) =

∑

∑ cn,k

n>0 k>0

e−z zn k u = n! 1−z

e−z(1−u) (zu)k = . ∑ 1−z k>0 k!


411

We differentiate this bivariate generating function with respect to u and obtain

∑ R′n(u)

n>0

zn z e−z(1−u) zn = = z ∑ Rn (u) n! 1−z n! n>0

and then comparing coefficients of zn , we get the relation R′n (u) = n Rn−1 (u)

or

Rn (u) = n

Z

Rn−1 (u) du + constant,

(7.64)

where the constant of integration is obtained from the fact that Rn (1) = ∑k>0 cn,k = n!. From these formulas, we immediately derive that R′n (1) = n Rn−1 (1) = n (n − 1)! = n! ,

R′′n (1) = n R′n−1 (1) = n (n − 1)! = n! .

From equations (6.80) and (6.81), it follows that the mean number of cycles in a permutation and its variance are

µn =

R′n (1) n! = = 1 and Rn (1) n!

σn2 =

R′′n (1) + R′n (1) − µn2 = 1. Rn (1)

n! suggest that the distribution of singleton cycles in k! e permutations is approximately Poisson with λ = 1.

These results together with cn,k ≈

Definition 7.95 In mathematics, an involution is a bijection of a set that is its own inverse, so that f ( f (x)) = x for all x in the domain of f . In combinatorics, an involution is a permutation that does not contain any permutation cycles of length > 2 (it consists exclusively of fixed points and transpositions). A transposition is a permutation of two elements (i.e., swapping of two elements). ⊳ Example 7.96 Let Jn be the number of permutations of the first n positive integers having only cycles of length 2, and let In be the of all permutations of [1..n] with cycles of length either 1 or 2—such permutations are called involutions. These numbers are related (for n > 1) by n n In = ∑ Jn , J0 = I0 = I1 = 1, (7.65) k=0 k because fixing the number of cycles of length 2 defines the permutation with cycles of length 6 2 uniquely—all other points must be singletons. Note that Jn = 0 for odd n since a cycle of length 2 involves two elements from the set [1..n]. Since any number from the set [1..n] can be paired with (n − 1) choices, we conclude that the Jn satisfy the second order difference equation Jn = (n − 2)Jn−2 ,

=⇒

In = In−1 + (n − 1)In−2 ,

n = 2, 3, . . . ,

J0 = 1, J1 = 0. (7.66)

The exponential generating function for the sequence {Jn } was found in Example 6.89 on 2 page 311 to be J(z) = ez /2 = ∑n>0 Jn zn /n!. Eq. (7.65) is actually a binomial convolution of 2 1’s with Jn , so I(z) = ez+z /2 is the exponential enumerator for the number of involutions.

412


Exercise 7.97 [2] When selecting a permutation of [1..n] at random, show that the probability that the cycle containing 1 and having length k (1 6 k 6 n) is 1/n (independent of k).

Permutations with restricted cycle structure Constructing permutations from cycles gives us a direct and easy approach for answering questions that are hard to tackle when dealing with simple sequences of atoms, as done on page 404. For example, the topic of derangements has come up on several occasions, and was tackled via recurrences in §5.8.2. But now we observe that a derangement, a permutation without stationary points, is a permutation with no cycles of length one. The enumerator of 1 − z, hence exponentiating this enumerpermutations with the size-one cycle deleted is ln 1−z 1 −z e , identical ator, which is equivalent to taking all sets of such cycles, provides us with 1−z to the EGF given in Eq. (L.44) for the number of derangements. Suppose we want to count the number of permutations without cycles of lengths 1 and 3. By marking cycles of size 1 and 3 with new variables u1 and u3 , respectively, we find its enumer1 − u1 z − u3 z3 /3. Therefore, its Abelian product will have the enumerator ator to be ln 1−z

z3 1 − u1 z − u3 P(z, u1 , u3 ) = exp ln 1−z 3

=

1 −u1 z −u3 z3 /3 e e . 1−z

Then the number of derangements of [1..n] not having cycles of size 3 is n![zn ]P(z, 1, 1) = n![zn ]

1 −z−z3 /3 e . 1−z

In general, if we want to count permutations with cycles of arbitrary length, we need infinitely many mark variables u j ( j = 1, 2, 3, . . .) to represent cycles of length j. The corresponding exponential enumerator becomes 2 3 z2 z3 z P(z, u) = exp u1 + u2 + u3 + · · · = eu1 z eu2 z /2 eu3 z /3 · · · , 1 2 3 where u = (u1 , u2 , u3 , . . .). The number of permutations with p1 cycles of size 1, p2 of size 2, and so on, is n! , (7.67) n![zn ][u1p1 ][u2p2 ] · · · P(z, u) = p1 ! p2 ! · · · pk ! 1 p1 2 p2 · · · k pk

provided ∑kj=1 j p j = n. In particular, the number of permutations of [1..n] having cycle of length n is n!/n = (n − 1)!. It is a custom to say that a permutation belongs to the cycle class [1 p1 2 p2 . . . n pn ], provided 1 p1 + 2 p2 + · · · + n pn = n, if it contains p1 singletons, p2 cycles of length 2, and so on. Exercise 7.98 [2+] Write the enumerator for the number of permutations of [1..n] that satisfy the following constraints: (a) Self-inverses: such a permutation σ satisfies σ (σ (i)) = i, for all i ∈ [1..n]. (b) Permutations restricted to have cycles of even size.


413

Exercise 7.99 [2+] Let On be the number of partitions of a set with n labeled elements into subsets with an odd number of elements in each. Show that O(z) = ∑n>0 On zn /n! = exp(sinh z), where sinh z = (ez − e−z )/2 is the hyperbolic sine function of z. Hint: Consider the labeled Abelian closure of a class U that contains a single element of each odd weight.

7.6.3 Shuffle Product In this subsection, we introduce the shuffle product of two sets of strings. Formally, this is just the Cartesian product of labeled structures described in §7.6.1, but the fact that we limit ourselves to shuffling strings over disjoint alphabets provides this operation with a specific power and usefulness. We utilize a special symbol, x , for this operation. This symbol is the Russian letter sha, which also resembles the Hebrew shin. Two sets L1 and L2 are shuffled by creating all the word pairs from both, just as we would do for the Cartesian product, and shuffling each such pair. Two words are shuffled by merging their letters in all possible manners, while retaining the original order in each. For example, shuffling the pair (abc, de) produces abc x de = {abcde, abdce, adbce, dabce, abdec, adbec, dabec, deabc, adebc, daebc} . This recursive definition can be formally expressed as follows, for an alphabet Σ: w x ε = ε and ε x w = ε , a.w1 x b.w2 = a.(w1 x b.w2 ) ∪ b.(a.w1 x w2 ), L1 x L2 = ∪(w1 ,w2 ) w1 x w2 ,

ε is the empty word, a, b ∈ Σ, w1 ∈ L1 , w2 ∈ L2 , w1 ∈ L1 , w2 ∈ L2 ,

where dots denote catenation, and the parentheses denote order of operations, but they are not part of the word. Given two sets of strings, A and B, let C = A x B; suppose we know the enumerators of A and B (with string length as the weight function). What can we say about the enumerator of C? The answer is immediate, if somewhat surprising. Consider a pair of words from these sets, of lengths n and m. Shuffling them produces all possible permutations of n + m letters, except that the order of the two sets of letters remains fixed, and therefore we “lose” factors n+m of n! and m!, and end up with a total of (n+m)! n!m! = n . Is this true? We need to be careful. In the example, with abc and de, we indeed got 10 strings. But if we shuffle abc and cd—some of the ten strings would be identical! We would have a smaller number of distinguishable strings. This is different from the case of repeated letters in some of the words, precisely because the order among the letters of each word is kept intact. In the example, if we used abb instead of abc, we would still be able to trace the ‘first’ b and the ‘second’ b, and hence such repetitions do not impact the shuffle count. Since we want the pairs to cover all possible selections from the two sets, the number of common letters may vary and we would not be able to make a useful statement, except in the


414

special case: the words in the sets A and B are defined on different alphabets (alternatively— and this is the more common case—on disjoint subsets of the same alphabet). When this is the case, the binomial coefficient is the number of words that are created by shuffling strings of lengths m and n. This suggests that we compute the exponential enumerator of C, as given in Eq. (7.2):

ϕˆC (z) =

zw(σ ) zk = c ∑ k . ∑ σ ∈C w(σ )! k>0 k!

To find ck , we look at all word pairs of total size k and change the order of summation k zk k k k z ˆ =∑ ∑ ∑ ∑ . ϕC (z) = ∑ ∑ ∑ ∑ n n k! k! n=0 a∈A b∈B n>0 a∈A k>n b∈B k>0 |a|=n |b|=k−n

|a|=n

|b|=k−n

and rearrange the summations to cover the sets A and B separately:    n k−n z z  = ϕˆ A (z)ϕˆB (z). ϕˆC (z) =  ∑ ∑   ∑ ∑ n>0 a∈A n! k−n>0 b∈B (k − n)! |a|=n

|b|=k−n

We have shown

ϕˆ A x B (z) = ϕˆ A (z)ϕˆB (z).

(7.68)

Exercise 7.100 [2] The DNA consists of four nucleotides A (Adenine), C (Cytosine), G (Guanine), and T (Thymine). The order of these four nucleotides in a strand does matter. Determine the exponential enumerator and count the number of distinct DNA strands that consist of (a) 5A, 3C, 1G ; (b) 3A, 2C, 1G, 2T . Exercise 7.101 [4] Write the enumerators for the following sets that consist of strings over the 3-letter alphabet Σ = {0, 1, 2}, with certain restrictions and qualifications. (a) A is the set of all words over Σ with up to r 0’s, s 1’s and t 2’s in each word. The weight is the number of digits in a word. (b) B is the set of all words over Σ with equal number of each of the letters and this number, denoted by q, is in the range q ∈ [r..s]; the weight function is as in part (a). (c) C is the set of all words over Σ with length not exceeding t. The weight of a word is sum of its digits. Example 7.102 Suppose we want to know the number of 64-bit words (strings) that contain no more than 10 zeroes. Then we have two exponential enumerators z2 z2 z10 + ···+ , ϕˆ 1 (z) = ez = 1 + z + + · · · 2! 10! 2! for zeroes and ones, respectively. Then the enumerator for our problem is their product: ϕˆ (z) = ϕˆ 0 (z)ϕˆ1 (z) and the number of 64-bit words that have at most 10 zeroes is

ϕˆ 0 (z) = 1 + z +

64! [z64 ]ϕˆ 0 (z)ϕˆ 1 (z) =

10

∑ (64)k = 5.59841 . . . × 1017 .

k=0


415

Example 7.103 In how many ways can three of the letters in the word RUSSIA be arranged? Some elements are distinguishable—two are not!

Solution. As you were asked to explain in Exercise 8.10, the correct approach here is of “putting positions in letters.” In four of the letters, R-U-I-A we can put zero or one position (= use them at most once), but the S can be used twice, hence we get the rather truncated exponential enumerator z2 (1 + z)4 . g(z) ˆ = 1+z+ 2! The required answer is the coefficient of

z3 3!

in g(z), ˆ which is 72.

Example 7.104 A ship carries 60 flags, 12 each of the colors red, green, blue, white, and black. To communicate a signal to other ships, ten flags are placed on a vertical pole. We consider different enumeration problems related to distribution of four distinct objects—flags— into 10 ordered bins. How many of these signals use either no red flags or at least three red flags, but no more than five? To answer the question, we first write the enumerator for the red flags: ϕr (z) = 3 z4 z5 + 5! . Since there is no constraint on other colors, the required enumerator is 1 + z3! + 4! z ϕ (z) = ϕr (z) (e )4 . The coefficient of z10 /10! in ϕ (z) yields 10! [z10 ]ϕr (z) (ez )4 = 4132864.

How many of the signals use at least three red flags, even number of green flags, odd number of blue flags, at most two white flags, and exactly three black flags? We find the required enumerator, denoted by ϕ (z), as the product of enumerators for red flags—ϕr (z) = ez − 1 − z − z2 /2, for even number of green flags—ϕg (z) = cosh(z), for odd number of blue flags—ϕb (z) = sinh(z), for at most two white flags—ϕw (z) = 1 + z + z2/2, and z3 /6, the enumerator for three black flags. The number of signals sought here is the coefficient of z10 /10! in ϕ (z), which results 10! [z10 ]ϕ (z) = 614880. At the end, we find the enumerator so that the total number of red and blue flags is even. Then the sum of two variables (that correspond to red and blue flags) must be even, so the enumerator becomes ϕ (z) = cosh(2z) (ez )2 , and the number of signals in this case is 10! [z10 ]ϕ (z) = 102480. Exercise 7.105 [2] Find an exponential enumerator for the number of distinguishable ways to arrange n letters, 1 6 n 6 6, selected without replacement from the word PEPPER. Exercise 7.106 [3] Write the enumerators for the number of letters in each of the following sets, using the same weight function for the strings. Each set consists of all the strings you can create from the letters of the name “worcester,” under the following restrictions: (a) without repeating a letter; (b) with the letter e repeated twice at most, just as in “worcester;” (c) all strings of lengths up to 9, 12, and n, with arbitrary repetitions. Exercise 7.107 [1] An 8-digit ternary (0, 1, 2) sequence is randomly generated. What is the probability that the total number of 0’s and 1’s is even? The term “shuffling” appears in a very different context as well, card shuffling, and the fol-

416


lowing interesting exercise illustrates it. It is a fascinating topic, although it is not really related to the business at hand (good sources are [4, 83]). Here is a taste of that world: Exercise 7.108 (R. DeSario) The usual procedure for shuffling a deck of n cards involves first splitting the deck into two blocks and then merging the two blocks to form a single deck in such a way that the order of the cards within each block is not changed. A trivial cut in which one block is empty is allowed. (a) [3] How many different permutations of a deck of n cards can be produced by a single shuffle? (b) [4] How many single-shuffle permutations can be inverted by another such single shuffle to put the deck back in order?

7.6.4 The Birthday Problems The birthday problem is one of the most famous problems in combinatorial probability. Much of its fame is due to its counterintuitive results as shown in Example 3.19 on page 90. We were asked there about the probability that among n people at least two share a birthday. Now we generalize this classical problem in a few directions. In particular, we consider the number of people it takes to find a date of birth shared by r > 2 people. We allow a “year” to have m days, and we ask not only about median probabilities but also about expectations (higher moments are computationally much harder). The one assumption we hold is that the days of birth are uniformly and independently distributed throughout the year, and we ignore the existence of leap years. While this assumption appears natural in our context, it is curious to note that at least in the United States, statistics for the years 1978–1987 show a 12% increase of births in September compared with January. The birthday problems are naturally embedded into so called “coincidence problems” that have enormous applications (for example, in hashing procedures, cryptography, and combinatorial problems such as coupon collecting). This example demonstrates the power of our new technique—shuffle product. We note that the scenario of having a birthday on a given date is similar to a scenario in which unmarked balls are deposited with equal likelihood into m labeled bins, a problem that we take up again in the next chapter. We use the following notations: • m is the number of possible birthdays, and all birthdays are assumed equally likely; • n is the number of people in the group including possible new arrivals, according to context; • Pr (m, n) is the probability that at least r out of the n randomly chosen people share their birthday; • nr is the least value of n such that Pr (m, n) > 1/2 (known as median). There are several questions that one can ask; here are four examples. Problem 1: Compute Pr (m, n). In particular, for a given r and m, what is the smallest group size n for which the probability Pr (m, n) is greater than 1/2? Problem 2: A dynamic version of Problem 1: people join the group and their birthdays are


417

sampled on arrival. What is the probability that we find our rth match when the nth person joins? Problem 3: In the context of Problem 2, what is the expected number of people we need to sample in order to observe r-matches of their birthdays? Problem 4: Again consider a given group of size n. What is the expected number of subsets of at least r people who share their birth date? In the balls & bins model, we consider m (m = 365 if you wish) bins and we record birthdays by placing a ball for each person’s birthday into the indicated bin. Once we distribute all the n balls, we observe bins that contain at least r balls. This problem asks for their expected number. We start with Problem 1 and consider a few ways of analyzing it. In Example 3.19. we looked at placing n balls in m bins (for the original birthday problem m = 365), numbered 1, 2, . . . , m, and each toss of a ball is equally likely to end up in any bin. When done, we check to see if any bin’s occupancy reaches or exceeds r. Here is another view: assign the letter ai to the bin numbered i (i = 1, 2, . . . , m), and create an n-long string of these letters. When a ball lands in the i-th bin, we write the letter ai . Hence the process of tossing balls into m bins is equivalent to writing words from an alphabet of m letters. This view is further developed in §9.1, but for our present needs these concepts suffice. The event “the occupancy of a bin reaches r” above corresponds to the event “a letter has occurred at least r times in the word.” The two views define the same abstract probability space. For the most part we adopt the string view of our problem. So let us look a little deeper into letters and words. Suppose an alphabet consists of one letter, say a; we denote by a∗ the somewhat limited language of all finite words (including the empty one) containing a’s. Moreover, a3 a∗ is the set of all strings of the letter a with at least three a’s; we denote this set by a>3 . Its complement in a∗ , denoted by a4 x d r 0 mn!n [zn ]em r−1 (z), and manipn n n ulate it. First we move the m to be with z : when we find z , we shall also get the mn , hence zi m E[Tr (m)] = ∑n>0 n![zn ]em r−1 (z/m). Suppose we wrote er−1 (z/m) = ∑i ai i! , which is simply the exponential generating function a(z) ˆ for the sequence {ai }. This is a finite expansion, a polynomial, although we make an important observation that our last expression for E[Tr (m)] can be rewritten in terms of this sequence: E[Tr (m)] =

zi

∑ n![zn ] ∑ ai i! = ∑ [zn ] ∑ ai zi , i

n>0

n>0

i

where we have reduced the factorial, just as we reduced the power of m above; the last sum simply equals ∑n an = a(1). It is the ordinary generating function (OGF) of the sequence {ai }, at the point 1. How do we connect the known exponential generating function of the sequence em r−1 (z/m) with this OGF?R Via the Laplace-Borel transform introduced in Eq. (6.41), −t dt, substituting z = 1 meets our need: page 293. Since in general a(z) = 0∞ a(zt)e ˆ E[Tr (m)] = a(1) =

Z ∞

−t

a(t)e ˆ dt =

0

Z ∞ 0

−t em r−1 (t/m)e dt.

Integrating such a function over an unbounded interval requires some care, but is relatively simple with an adaptive integration routine. This is best done within MAPLE or a similar system that can handle the huge demands the function will place on its numerical range capability. We calculate Tr for the same values as it was done in the table of medians from Problem 1 and obtain the following table r

2

3

4

5

6

7

8

Tr

24.6166

88.7389

187.0494

311.4468

456.0163

616.6169

790.2997

r

9

10

11

12

13

14

15

Tr

974.8939 1168.7567 1370.6135 1579.4531 1794.4590 2014.9607 2240.3999

The next problem also asks about expectation, but it is of a different flavor. Problem 4: Again consider a given group size n. What is the expected number of subsets of at least r people who share their birth date? In the language of strings over the alphabet Σ, we are asking for the number of different letters which have recurred at least r times in an n-long prefix. We already developed the tools to answer this question in the analysis of Problem 1. There we created languages denoted by Lk . Such a language is a collection of all where exactly k letters recur r or more times. kwords m−k (z)ϕˆr

422


just lumping together all these enumerators; but here we need to keep track of those k-values, and we do the usual thing when computing with enumerators. We use an additional variable, u, to mark the number of properly repeated letters, and we shall write m k m−k ϕˆ k (z) = ϕˆ (z)ϕˆr (z) + ϕˆ r Hence the last term, ϕˆ (u, z) = (uϕˆ >r (z) + ϕˆ r (z) + ϕˆ r (z/m) + ϕˆ r (z/m) and ϕˆ r (z/m) and ϕˆ 5, for that matter), but the Lagrange Inversion Theorem, using Example 6.188, allows us to write immediately kn + 1 kn 1 1 1 kn n = = . (8.2) [z ]bk (z) = n n−1 kn + 1 (k − 1)n + 1 n n Exercise 8.1 [2] Consider a “mixed” tree variety: the root may have up to k children, just as in the above k-ary tree (for some k > 2), but each of the subtrees rooted in these k places is a ternary tree. Write an enumerator for this set of trees; it is not clear that a closed expression can be found for the number of such trees with n nodes for any k > 3.

x

(a)

(b)

(c)

(d)

(e)

(f)

(g)

Figure 425: Unlabeled trees, roots at the top.

Plane trees Recall that a plane tree T can be defined recursively as a finite set of vertices such that one distinguished vertex is called the root of T , and the remaining vertices are put into an ordered partition (T1 , T2 , . . . , Tm ) of m disjoint non-empty sets, each of which is a plane tree. Plane trees are also called ordered trees because the order of the subtrees at every node is specified. We denote the set of such trees by T p . They differ from binary (or k-ary) trees mainly because their children do not have specified positions, and therefore the trees (a) and (b) in Figure 425 which were distinct binary trees, are indistinguishable when considered as plane trees, and would be both typically drawn as tree (c) there. But the adjective “ordered,” or their being “in the plane” means that the order of subtrees is significant, and therefore trees (d) and (e) in Figure 425 are distinct as plane trees. The structure equation for plane trees differs from one for binary trees in two related ways: there is no bound on the number of offspring of a node, and an empty tree is not counted

CHAPTER 8. FURTHER ENUMERATION METHODS

426

as a subtree. Thus, the set of subtrees of any node, including the root, is an element of (T p − ∅)∗ , the closure2 of nonempty plane trees. Note that we use the Kleene closure since order between the subtrees matters. Hence we find T p = ∅ + ρ × (T p − ∅)∗

=⇒

T p − ∅ = ρ × (T p − ∅)∗ .

(8.3)

Denoting by T p (z) ≡ T p − ∅, the enumerator of the set of all plane trees except the empty tree, we see that it satisfies the equation from which we can read-off the symbolic relation: T p (z) =

z 1 − T p (z)

=⇒

2

T p (z) = z + T p (z).

(8.4)

Comparing this to the equation for the enumerator of binary trees, b(z) = 1 + zb2 (z), we see that zb(z) and T p (z) satisfy the same equation. Hence T p (z) = zb(z), and [zn ]T p (z) = [zn ]zb(z) = [zn−1 ]b(z) = 1n 2n−2 n−1 , the Catalan number (see its definition in Eq. (6.48) on page 300). For this reason the term Catalan tree is often used instead of rooted unlabeled plane tree. This relation between T p (z) and b(z) holds for n > 1 because T p (z) is the enumerator of the set T p without the empty tree. Hence, Tp (z) = 1 + T p (z). Exercise 8.2 [1] We have shown that the number of plane trees on n nodes is the same as that of binary trees with n − 1 nodes. Show a bijection between Tb , the set of binary trees, and the set of forests of plane trees (if you remove the root of a plane tree you get a forest of plane trees, and removing the root reduces the node count of the tree by 1).

Oriented trees Oriented trees are also called rooted trees. Sometimes we refer to such trees as unordered trees to distinguish them from ordered trees. Oriented trees are trees in which one node is marked as root; all nodes have a unique path to the root, and these paths induce an orientation on the edges, towards the root. They are the “loosest” variety of considered trees since we allow two trees to be called identical if one can be obtained from the other by rearranging the order of the children, at any node. The trees (d) and (e) in Figure 425 are the same when representing oriented trees. The tree (f) there has the same number of nodes as (d) and (e), but it is not equivalent to them: the root has two subtrees here too, but they are identical, and in (d) and (e) they differ. We denote the set of oriented trees by To . Note that there is, in fact, an even less restricted kind of trees, suitably called free trees or just trees, which differ from the above by having no root specified. Consider tree (g) in Figure 425. As a free tree, it is identical with (d) and (e). To see this, look at tree (e). One node is marked there with an ‘x’; pull it up so it becomes the top node, and the others nodes hang down from it, which forms tree (g)! While free trees have their own share of interesting combinatorial properties, they would take us far afield of algorithmic needs. 2 See

definition of closure on page 492.

8.1. ENUMERATION OF TREES

427

How do we count oriented trees? The new symmetry means that we cannot use the Cartesian product to form a tree from root and subtrees. We need a different set operation, and we shall use the multiset defined in §7.4.4. To see how this operation fits our needs, consider the trees in To . Any oriented tree, an element of To , is either the empty tree ∅, or a root ρ to which we attach, as subtrees, a number of nonempty oriented trees. As before, let us denote the set of nonempty trees with a bar over the set symbol: To ≡ To − ∅. We can say that the subtrees of the root of an oriented tree form an element of the multiset of To , which we denote by M(To ). Formally: To = ∅ + ρ × M(To )

=⇒

To = ρ × M(To ).

(8.5)

From Eq. (7.38) on page 392, we get a relation for the enumerator of To , denoted by T o (z): ! 1 j T o (z) = z exp ∑ T o (z ) . (8.6) j>1 j What can one learn from such an equation? It does not look very encouraging, does it? But in fact, it characterizes the sequence tn ≡ [zn ]T o (z) completely, and we should be able to find everything we ever wanted to know about oriented tree enumeration from it. First, how would you find a prefix of the sequence {tn }? Clearly, just equating coefficients on both sides does it, but there is even a somewhat more disciplined approach. [h]

Exercise 8.3 [3] Let T o (z) denote the enumerator of oriented trees, with height of up to [0] h. The height of a tree that consists of ρ only is zero. Hence T o (z) = z. Explain why the derivation of Eq. (8.5) implies the following recurrence ! 1 [h] j [h+1] T o (z) = z exp ∑ T o (z ) . j>1 j [h]

Explain why knowing T o (z) provides all the coefficients tn for n 6 h + 1. Use it to derive the first few coefficients manually; show how you would compute the value of any tn for a specific n. As a check, the first few coefficients, beginning with t1 , are 1, 1, 2, 4, 9, 20, 48, 115, 286, . . . . Exercise 8.4 [2+] Show, by differentiating Eq. (8.6) and extracting coefficients, that the tree counts, tn = [zn ]T o (z), satisfy a recurrence that allows them to be calculated successively, tn+1 = ∑ iti ∑ tn+1−i j /n. i>1

j>1

Asymptotic analysis of Eq. (8.6) given in [42], which goes beyond our treatment, shows that as n grows, the tn have the asymptotic estimate tn ∼ c α n n−3/2 , where c ≈ 0.4399237, and α = limn→∞ tn+1 tn ≈ 2.9557649.

Degree-restricted trees


428

Trees can be restricted in various ways. The most common ones are either by degree— limiting the number of children a node may have (unlike the plane and oriented trees, where the degree is unlimited), or by height. The height of a tree is the depth of its furthest leaf. Recall that the depth of a node is the length of the path from the root to the vertex. The problem of enumerating height-restricted oriented trees is only partially answered in Exercise 8.3, where we have a recurrence for the height-constrained trees. Although, this is not an explicit answer, we can calculate the enumerator for any finite height we wish. A similar argument would hold for the other tree varieties considered previously. The rest of our treatment here is restricted to plane trees when every node has at most k children. Then from Eq. (8.3), it follows 2 k (8.7) T p,k = ρ × 1 + T p,k + T p,k + · · · + T p,k ,

where the parenthesized polynomial of trees is simply the k-degree prefix of the Kleene closure series. Calling this polynomial Q(z) = 1 + z + z2 + · · · + zk = (zk+1 − 1)/(z − 1), we have an immediate equation for T p,k (z), the enumerator of T p,k : T p,k (z) = zQ T p,k (z) . (8.8) For k = 2, we get an easily solvable equation, T p,2 (z) = z

2 1 + T p,2 (z) + T p,2 (z)

=⇒

√ 1 − z ± 1 − 2z − 3z2 . T p,2 (z) = 2z

(8.9)

Selecting the sign to be negative ensures the regularity of the function T p,2 (z) at (minus) p z = 0. So we find T p,2 (z) = 1 − z − (1 + z)(1 − 3z) /2z; the coefficients of T p,2 (z) are known as Motzkin numbers3 (see Appendix for relevant formulas).

Exercise 8.5 [1+] Obtain an expression for pn,2 ≡ [zn ]T p,2 (z), using the last result for it, and write it as a convolution of the coefficients from the two factors. The prefix of the series for T p,2 (z) is given by z + z2 + 2z3 + 4z4 + 9z5 + 21z6 + 51z7 + 127z8 + 323z9 + 835z10 + 2188z11 + 5798z12 + · · · . Exercise 8.6 [2] While it is hard to find an explicit, closed expression for pn,k ≡ [zn ]T p,k (z), it is easy enough to find a derivation rule. Show, starting with Eq. (8.8) and using the Lagrange inversion theorem with Eq. (6.115) on page 351, that 1 n+ j−1 n n− j−1 1 n−1 n . (8.10) (−1)q , q ≡ pn,k = [z ] Q (z) = ∑ n n j>0 k+1 j q Note that the number of nonzero terms in the sum is ⌊ n−1 k+1 ⌋. 3 These numbers, named after Theodore Samuel Motzkin (1908 – 1970), have many applications in combinatorics, and they are related to the Catalan numbers; see, for instance, [34, 106].


a

a

b

c

c

b

(i)

(ii)

429

a

c

a

b

c

b

(iii)

(iv)

Figure 429: Labeled trees; roots at the top.

8.1.2 Labeled Trees Labels on the nodes of a tree are used as identifiers, and therefore they all must be different. In the context of algorithms, the values or data associated with the nodes are not labels. We use labels as a tool to distinguish the nodes. Rather such values may break symmetries that would otherwise lead us to say that two trees are identical. The vertices of a labeled tree on n nodes are typically given the labels 1, 2, . . . , n. Thus, when we consider binary trees, where there are no possible symmetries since each node has its unique location, which serves as an identifier, then adding the labels breaks no symmetries. Hence the number of labeled binary trees of n nodes is simply n! times the number of such unlabeled trees (and the same holds for k-ary trees of any order). In Figure 429, we see two pairs of trees i–ii and iii–iv, which are identical as unlabeled trees, but the given labeling changes things. The labeled i and ii are distinct. However, trees (iii) and (iv), when unlabeled are identical, would be considered different as labeled plane trees, but identical as labeled oriented (= unordered) trees.

Labeled plane trees The approach to counting labeled trees is identical with our approach to counting unlabeled trees. In other words, the relation (8.3) holds here as well. The trees are labeled structures and operations on them, in particular, the products and the closure, are expressed via the exponential enumerators, as shown on page 405. Since the structure relation for both labeled and unlabeled sets is the same, we get the same equation for the exponential enumerator, √ Tˆ p (z). Consequently, we obtain the same result Tˆ p (z) = zb(z) = (1 − 1 − 4z)/2, which yields the number of such labeled plane trees to be (2n − 2)! 1 2n − 2 ˆt p,n = n![zn ]Tˆ p (z) = n![zn−1 ]b(z) = n! . (8.11) = n n−1 (n − 1)!


430

Exercise 8.7 [1+] We have found that there are n! as many labeled plane trees of size n as there are unlabeled ones. Explain the similarity between labeled and unlabeled plane k-ary trees.

Labeled oriented trees Labeled oriented trees (also called Cayley trees, for historical reasons, after Arthur Cayley4 who enumerated them more than 150 years ago) have different symmetries than those of labeled plane trees. The children of each node are unordered. The subtrees of the root of an unordered tree form a multiset of such trees. When the tree is labeled, we need to preserve the symmetry: the tree stays the same when the order of subtrees is changed. When forming the exponential enumerator, we must divide each set of j subtrees by j!, in exactly the same manner as we did when deriving Eq. (7.52) on page 405. It is instructive to view the equation for the enumerator for plane trees, which as said previously, is essentially like Eq. (8.4): 2 j z ˆ ˆ ˆ ˆ = z 1 + T p (z) + T p (z) + · · · + T p (z) + · · · . T p (z) = 1 − Tˆ p (z) For the oriented trees we now find 1 2 1 j Tˆ o (z) = z 1 + Tˆ o (z) + Tˆ o (z) + · · · + Tˆ o (z) + · · · = z exp Tˆ o (z) . 2! j!

(8.12)

This is a very fortunate result since the equation f (x) = xe f (x) is well known. Its solution is called the omega function, see §12.6. While we do not know how to represent it in a closed form using simple functions, the Lagrange inversion theorem provides an immediate power series solution, which is entirely adequate for our needs here. Using, for example, Eq. (6.114) on page 351, we find that tô,n = n![zn ]Tˆ o (z) = (n − 1)![zn−1 ]enz = nn−1 .

(8.13)

This surprisingly compact result is sometimes called simply the tree function.

8.1.3 Counting Alternating Permutations Characteristics of permutations are probably as commonly used in analysis of algorithms as are trees. We shall see here a relationship between these two. Later in this chapter, we enumerate other configurations of permutations using different techniques; but here we restrict ourselves to a particular configuration, alternating permutations. A permutation π = π1 π2 · · · πn of a set of n distinct integers is alternating provided either π1 < π2 > π3 < π4 > · · · or π1 > π2 < π3 > π4 < · · · . For example, h 4, 5, 1, 7, 2, 6, 3 i and h 4, 3, 7, 1, 6, 2, 5 i are such 4 Arthur Cayley (1821–1895) was a brilliant British mathematician. He published more than 200 mathematical papers but worked for 14 years as a lawyer.


431

permutations—they are also called up-down and down-up permutations, respectively. Sometimes this kind of permutation is also called a zig-zag permutation because every number in its even position is either larger or smaller than its predecessor. Our exposition follows [46] closely. All the alternating permutations either start with an increasing pair, or a decreasing pair. The latter permutation h 4, 3, 7, 1, 6, 2, 5 i is of the second kind (down-up): 4 to 3. If in the given alternating permutation of [1..n] we replace each element k by n + 1 − k, then we obtain a permutation of the another kind. For instance, such transformation yields h 4, 5, 1, 7, 2, 6, 3 i 7→ h 4, 3, 7, 1, 6, 2, 5 i. So we shall count now only down-up permutations having the entry in even position to be less than its predecessor, and double the number of such permutations at the end. Let A0 = 1. For n = 1, 2, . . ., define An to be the number of alternating permutations of the set [1..n] starting with a decreasing pair. We will show that this sequence has a fantastic exponential generating function z π zn . (8.14) = sec z + tan z = tan + A ∑ n n! 2 4 n>0 To prove the formula (8.14), we establish a relationship between all permutations of [1..n] and decreasing, or descending, binary trees; each such tree carries n labels, from [1..n], on their internal nodes, so that the labels along each path to the root are monotonically decreasing. Such a tree forms a loose min-heap: it satisfies the min-heap property—each parent has a smaller label than its children’s—but may flout the heap structure requirement to be an essentially-complete binary tree (which has all its levels full, except possibly the bottom one, and if that one is not, the nodes in it are right-justified). Figure 432 shows such example. The transition from a permutation, π , to a binary tree, T , is done with a decomposition using the pattern T = (TL , R, TR ), for root and its subtrees. The minimal element (1, initially) is picked as the root yielding T (π ) = (T (πL ), T (min), T (πR )), augmented with the specification that the tree of the empty permutation is an empty tree. Figure 432 also shows diagrammatically how to associate a permutation with the tree; algorithmically we obtain the sequence of labels by an inorder traversal of the tree. The definition of this tree implies that every subtree is a descending tree as well. Clearly each such tree determines a permutation uniquely.

Let us count descending binary trees, denoting their class by Td . Their structure equation is similar to one in Eq. (8.1), Td = ∅ + Td × ρ × Td . The symbol ρ stands for the root, naturally, but there is an aspect of this structure equation that distinguishes it from an ordinary (labeled) product: the root is constrained to carry the minimal label.

Digression: label-restricted products. The restriction on the degree of the root requires us to choose an operation that differs, although not by much, from the labeled products presented on page 405. To handle this situation, we follow [46] and introduce the box product C = A × B, which is defined as an ordinary labeled product, except that the minimal label is constrained to be carried by an element from A. Note: This product requires that A has no elements of weight 0, or, in other words, we need a0 = 0, which implies c0 = 0 (here a0 and


432

c0 represent the number of elements of zero weight in sets A and C, respectively). 1

4

2

3

5

6

7

3

2

1

6

4

7

5

Figure 432: Descending tree and its corresponding permutation. What is now ϕˆC (z)? While it may not be obvious a-priori, it turns out that we have found another EGF-admissible operation. For the box-product to be admissible, the exponential enumerator of C needs to be computable from those of A and B. Recall that C(n) denotes a subset of C that contains all elements with weight n. The calculation is entirely analogous to the proof of Eq. (7.52): the number of ways to generate the elements of C(n) is a convolution of all {ai } and {bi }, except that now we only have n − 1 labels to distribute freely since ‘1’ is known to be given to the A contingent, in addition to i − 1 more: n

cn = ∑

i=1

n−1 1 n n (iai )bn−i . ai bn−i = ∑ n i=1 i i−1

Hence ncn is a binomial convolution of the sequences {kak } and {bk }. From Table 278, we find that the EGF of the first one is given by zϕˆ A′ (z) according to rule (4). Then, following in order rules (9) and (5), we find that

C = A× B

=⇒

ϕˆC (z) =

Z z 0

ϕˆ A′ (t)ϕˆB (t) dt.

Box product.

(8.15)

The same calculation will handle the maximal box product C = A × B, which requires the largest label in each C term to be attached to an element of A. End of digression: label-restricted products. Now we return back to the descending trees. We know their structure equation to be Td = ∅ + ρ× Td × Td since there are no further restrictions on distributing the labels between the subtrees. We observe that the exponential enumerator for the root is ϕˆ ρ (z) = z, hence

ϕˆ T (z) = 1 +

Z z 0

1 · ϕˆ T2 (t)dt

=⇒

ϕˆ T′ (z) = ϕˆ T2 (z)

=⇒

ϕˆ T (z) =

1 , 1−z


433

just the exponential generating function of all permutations. This implies that the relationship between permutations and descending trees is a bijection. But we are looking at particular permutations! So not all such trees are suitable. Some experimentation suggests that it is best to handle separately odd- and even-sized permutations, beginning with the former.

Odd-size Permutations Let us denote the class of such permutations by Po and the minimal element, by ρ ; then we write odd-size alternating permutations, (8.16) Po = ρ + {Po × ρ× Po }, where the first ρ is for the permutation of size 1. The reason for the choice of odd-sized permutations is that when limited to permutations starting with a decreasing pair (which look like a W), the above correspondence with trees fits them with a binary tree where both subtrees have odd number of nodes, and all the nodes have even degrees, zero (at the leaves) or two (at internal nodes).

Exercise 8.8 [1] Prove by induction that there is a bijective transformation between zig-zag permutations of odd size starting with a decreasing pair and descending binary trees that has no single-child nodes. It becomes clear now the reason for introducing this bijection: while alternating permutations appear difficult to characterize combinatorially, the binary descending tree corresponds to an odd-sized permutation, which we know how to count. An equation for the enumerator of these restricted trees (or permutations) follows directly from Eq. (8.16),

ϕˆ Po (z) = z +

Z

1 · ϕˆ P2o (z)dz

=⇒

ϕˆ P′ o (z) = 1 + ϕˆ P2o (z).

This equation is known as the differential equation satisfied by the tangent function, ϕˆ Po (z) = tan z. The coefficients do not have a simple form, though they can be written in terms of the Bernoulli numbers, see Eq. (L.11). To get the total number of the alternating permutations we still need to double the above expression (except for the permutation of size one), and then the first few terms of the sequence are as follows: 1, 4, 32, 544, 15872, 707584, 44736512, 3807514624. The last value is the number of alternating permutations of [1..15], which is about 1 in 433 (the total number of permutations is 15! = 1307674368000).

Even-sized permutations Once we have handled the odd-sized permutations, the rest is easier. Observe that when we perform the bijective transformation of an even-sized alternating permutation to a binary tree, picking the minimal element, we must have on its right and left sides one odd-sized and one even-sized such permutation (zero is of course an even number). In other words: Pe = ∅ + ρ× Po × Pe, from which we read the relation among the enumerators:

ϕˆ Pe (z) = 1 +

Z z

t=0

1 · ϕˆ Pe (t)ϕˆ Po (t)dt =⇒ ϕˆ P′ e (z) = ϕˆ Pe (z) tan z =⇒ ϕˆ Pe =

1 = sec z. cos z

434


On the differential equations in this section The two ordinary differential equations we saw in this treatment are typical of what we find when manipulating enumerators; they can be solved by a method called separation of variables. Let us use plain notation, where the equations are for y(x) dy . which we simply write y, and y′ is written as dx dy The first equation we had was in this notation dx = y2 subject to the initial condition = dx and then y(0) = 1. Separation of variables implies that we write it as dy y2 integrate each side using the implied differential. One integration constant C takes 1 . Using the initial condition care of both sides and we find − 1y = x +C, or y = − x+C 1 y(0) = 1 yields C = −1, hence y = 1−x . dy = 1 + y2 , y(0) = 0, are exactly the The preliminaries for the second equation, dx dy same, giving us 1+y 2 = dx, but the integral on the left-hand side may require you to consult a table of integrals (even the limited one in [1, Chapter 3] is adequate for R dy this). It will tell you that 1+y2 = arctan y +C, hence y = tan(x +C), and substitution of the initial value reveals that C = 0 (in fact, it could be any integer multiple of π ), providing us with the second solution above, y = tan x. The power series development of the secant function is superficially similar to that of the tangent, except that the other known sequence that can be used to express them are the Euler E2n . Their rate of growth is similar. numbers (not the Eulerian numbers): [z2n ] sec z = (−1)n (2n)! Exercise 8.9 [1] We say that a permutation π of the set of the first n (n > 1) integers has a local maximum at k ∈ [1..n] if π (1) > π (2) for k = 1, π (n − 1) < π (n) for k = n, and π (k − 1) < π (k) > π (k + 1) for 1 < k < n. What is the average number of local maxima of a permutation of [1..n], averaging over all permutations?

8.2 Occupancy Enumeration Counting the ways of distributing objects into bins or cells is a basic combinatorial activity. Formally this may be seen as the counting of functions that map objects to bins with various symmetry constraints. There are several ways in which such scenarios arise in analysis of algorithms. An important example arises with strings over alphabets—since a word of length n, using m-character alphabet, may be viewed as a function mapping the integers [1..n] into [1..m]. Exercise 8.10 [1+] Explain the last example: why do we have to consider a function of the form positions −→ letters, whereas the converse—putting characters in positions—appears more natural? Another example is a hash function, which maps keys into slots of a hash table. A customary unifying framework for the study of occupancy problems is the ball-and-bin model. We

8.2. OCCUPANCY ENUMERATION

435

assume that there are n objects (items, balls) and m cells (places, bins, urns, boxes), and we use these terms interchangeably. What we are counting is always “distinguishable occupancy configurations.” What is considered distinguishable depends on the following considerations: • Balls can be identical or distinct (which is the same as saying they are numbered, labeled). • Bins may be ordered (= numbered) or not. • In all standard situations, we consider the items in a bin unordered, even if they are distinct and labeled. When the bins are ordered, the configurations we count are represented as occupancy vectors, lists of m nonnegative integers. Their interpretations depend on the specific scenarios of the counting. There are four principal variants of the occupancy problem, depending on whether we consider items and/or cells distinct or not.

8.2.1 Distribution of Identical Balls into Distinguishable Bins We start with the distributions of n identical balls into m distinct urns. There is a one-to-one correspondence between the integer solutions of the Diophantine equation (7.12), page 7.12, and the distributions of n indistinguishable balls into m distinct boxes. Since the bins are distinguishable, we can order them—call them by number, 1 through m— and represent the number of balls in each bin using the occupancy vectors n = hn1 , n2 , . . . , nm i, where ni is the number of balls stored in the ith urn. Comparing this with the enumerations we did in §7.3, we see that this case is identical with the problem of counting compositions of n with m nonnegative components. In that section we saw that the enumerator for homogeneous compositions (for all values of n, but for fixed m) is given by !m

ϕ (z) =

∑ zi

i>0

= (1 − z)−m .

The number of placing n identical balls into m distinct bins is [zn ] ϕ (z), namely, −m m+n−1 m+n−1 n −m n [z ] (1 − z) = (−1) = = . n n m−1 According to Eq. (7.23), the same number when no urn is empty is n−1 n m −m n−m −m [z ] z (1 − z) = [z ] (1 − z) = . m−1

(8.17)

(8.18)

(8.19)

A different verbal description, which leads to the same mathematical structure, a multiset, asks about the number of ways to make n choices from a set of m distinct items, possibly


436

with repetition. Note that this is the same as string counting (see, for instance, §7.6.4). And just as there we use a construction that may create a cognitive dissonance: to describe the allocation of balls into bins, we consider a multiset over the bins since an element of this multiset is captured by an occupancy vector n. Again, let A be the set of bins. To count these vectors, we give each bin the weight 1, and have ϕA (z) = mz. Then from Eq. (7.36), we find m 1 1 zj j ln ϕM(A) (z) = ∑ (mz ) = m ∑ = −m ln(1 − z) =⇒ ϕM(A) = 1−z j>1 j j>1 j as in Eq. (8.17). Let us now probe somewhat deeper. In §7.3, we looked at counting compositions when not all components were the same. For example, we require some components to be even and others odd, or satisfy inequalities, etc. Such inequalities can represent constraints on bin occupancies. Also, in some applications, we may want to ask questions about occupancies where different bins play different roles. To do this using enumerators, we need to distinguish the occupancies of the bins. A conventional way of doing it is to mark each bin with its own variable. Thus, instead of one plain z we have m variables zi (i = 1, . . . , m), and we collect them all together in the row vector z = hz1 , . . . , zm i.

The unrestricted enumerator for the total number of occupied urns (some of them may be empty) is now m 1 . (8.20) ϕ (z) = ∏ j=1 1 − z j By setting all z j equal z, we return to the composition case: n+m−1 n −m ϕ (z, z, . . . , z) = (1 − z) = ∑ z . n n>0

The corresponding enumerator for all m urns, none of which may be empty, is m

∏

j=1

m z j + z2j + · · · = ∏

zj = z1 z2 . . . zm ϕ (z) . j=1 1 − z j

This detailed enumerator (8.20) allows us to find the number of occupancies when distinct cells play distinct roles. Note that it provides the coefficient m

1 = 1. j=1 1 − z j

[zn11 zn22 · · · znmm ]ϕ (z) = [zn11 zn22 · · · znmm ] ∏

(8.21)

This is an illustration of a fact that we shall see more of: a complete specification, which in this case is a particular occupancy vector n = hn1 , n2 , . . . , nm i, is realized in exactly one way. All the questions we ask about this model concern particular collections of such occupancy vectors.


437

Example 8.11 To find the number of possible occupancies where some bin, say #r, may not have a balls, or an odd number of balls, we would replace the factor (1 − zr )−1 by the factor (1 − zr )−1 − zar , or by the factor (1 − zr )−1 − zr (1 − z2r )−1 = (1 − z2r )−1 .

To find the number of possible occupancy vectors where n1 = nm = a, out of a total of n balls, we extract the coefficient [za1 zam zn−2a ]ϕ (z)|z2 =···=zm−1 =z

m + n − 3 − 2a = m−3

(8.22)

since the factors separate completely in this particularly simple case. Exercise 8.12 [2] (a) Perform the extraction of coefficients in Eq. (8.22). (b) Repeat the derivation for the case where we only require that at least two bins contain exactly a balls (but do not specify which ones). Note: We discuss in §8.3 how to approach harder problems, such as counting the allocations where exactly two (unspecified) bins contain a balls (Exercise 8.46 on page 459). Exercise 8.13 [3] A pseudo-random generator selects numbers from the set [1..n] with uniform probability, and naturally, with possible repetitions. As integers are generated, a running sum, X , is kept. Calculate Pr[X = k], the probability that the value k is obtained at some point, for integer k 6 n. Example 8.14 (Smirnov’s problem) We present a classical problem, adapted from [11]. Consider arrangements in a row of three letters A, B, and C with no two identical letters adjacent. Let there be a times the letter A, b times B, and c times the letter C. How many such arrangements are possible? The problem is known as Smirnov’s problem and it comes up in numerous applications [11]. Let L(a, b, c) be the set of such arrangements, called the Smirnov set, and denote its size by L(a, b, c). Similarly, let T (a, b, c) be the set of all arrangements of a letters A, b letters B, and c letters C without the adjacency constraint, and denote its cardinality by T(a, b, c), which equals a+b+c a,b,c . For example, the string CCCBAAAC is in T (3, 1, 4) but not in L(3, 1, 4). On the other hand, the string CBCACACA is in both sets. The following treatment only holds for values of a, b, and c that are positive, but it can be adapted to allow zero-counts as well. We define a map from T (a, b, c) to L(a, b, c) as follows. Given an arrangement in T , replace any run of identical letters with one of them (e.g., CCCBAAAC → CBAC). Then an element from T (a, b, c) will be mapped to some element from L(i, j, k), with 1 6 i 6 a, 1 6 j 6 b, and 1 6 k 6 c. This mapping is the identity mapping on L(a, b, c). We now need to know the number of elements of T (a, b, c) that will be mapped to a fixed element from L(i, j, k). Note that this is the result we obtained when discussing boundedcomponent compositions in §7.3.1, in Eq. (7.23), as we need the number of ways todistribute a identical balls in i cells so that each cell gets at least one ball, which equals a−1 i−1 . The partitions of the three letters among their runs occur independently, and hence, applying


438

this result to each letter, we find that a−1 b−1 c−1 L(i, j, k) i−1 j−1 k−1 gives the number of ways that a row of a + b + c positions can be filled with a letters A in i runs, b letters B in j runs, and c letters C in k runs. Thus we have two different formulas expressing the number of ways of arranging a string of a, b, and c counts of distinguishable letters A, B, and C: a

b

c

∑∑∑

i=1 j=1 k=1

a+b+c a−1 b−1 c−1 (a + b + c)! = . L(i, j, k) = a! b! c! a, b, c i−1 j−1 k−1

(8.23)

Using three times the binomial inversion formula (as in Eq. (2.72) on page 65), we obtain a

b

L(a, b, c) = ∑ ∑

c

∑ (−1)a+b+c−i− j−k

i=1 j=1 k=1

a − 1 b − 1 c − 1 (i + j + k)! , i! j! k! i−1 j−1 k−1

(8.24)

where a, b, and c are positive integers. Exercise 8.15 [4] (a) Develop the analogies of Eq. (8.24) for words over the same alphabet with runs of only two different letters, and of a single letter. (b) Show by direct summation that the enumerator of the set {L(a, b, c) ∪ L(t1 ,t2 ) ∪ L(t3 )}, tk ∈ {a, b, c} is

y z x − − L(x, y, z) = ∑ L(a, b, c)x y z = 1 − x+1 y+1 z+1 a,b,c>0 a b c

−1

.

(c) In the generating function you obtained in part (b), replace all identifiers by the same symbol, say t, then [t n ]L(t,t,t) is the number of such Smirnov’s words of length n. Extract the coefficient and give it a combinatorial interpretation. Example 8.16 (Smirnov’s problem, continuation) It is instructive to approach Smirnov’s problem, and, in particular, to derive the answer to Exercise 8.15, without any calculations. Let Σ be an alphabet—a finite set of distinguishable symbols (also called letters). Its closure, the set of all possible finite strings formed by letters from the alphabet, is denoted by Σ∗ . Let L(Σ) be the Smirnov set over the alphabet Σ—the set of all finite strings of length at least 1 that do not contain adjacent letters. For simplicity and to be consistent with Example 8.14, we consider only a three-letter alphabet Σ = {A, B,C}.

The idea of using the symbolic method in Smirnov’s problem consists in generating the closure, Σ∗ , from the Smirnov set, L(Σ), which can be done as follows. In each word from L(Σ), we replace any letter in it by all possible finite strings of arbitrary positive length of the same letter. For example, if letter A is in the word, it is replaced by any string from AA∗ . The set of all strings of A’s containing at least one time the letter A is conventionally denoted by


439

def

· · · A}. Its enumerator is za /(1 − za ), while A+ = AA∗ = {An : n = 1, 2, . . .}, where An = AA | {z the enumerator of A∗ is 1/(1 − za ).

n

times

Let L(za , zb , zc ) be the ordinary enumerator of the Smirnov set, L(Σ), over the three-letter alphabet Σ = {A, B,C}. According to the substitution rule, Eq. (7.29) on page 385, the well known enumerator, T (za , zb , zc ), of the closure, Σ∗ , must satisfy the relation za zb zc 1 =L , , . T (za , zb , zc ) = 1 − za − zb − zc 1 − za 1 − zb 1 − zc

Since the solution of t = y/(1 − y) is y = t/(1 + t) we have found that −1 zb zc zb zc za za , , , − − = 1− L(za , zb , zc ) = T 1 + za 1 + zb 1 + zc 1 + za 1 + zb 1 + zc as expected. In a similar way, the above formulas could be extended to arbitrary (finite) number of letters.

8.2.2 Distribution of Distinct Objects into Ordered Cells Consider now n distinct objects (balls) and m distinct, or ordered, cells. The number of ways of putting n different balls into m distinct cells (or urns) is mn because each ball has m choices to be placed. Since the balls are identifiable, we can order these balls by labeling them with, say, the numbers 1, 2, . . . , n. This labeling differentiates between occupancies that in §8.2.1 are lumped together. In our case of distinct objects, an occupancy vector n that provided a complete specification before, would become an n × m matrix: n = [n1 , n2 , . . . , nm ], where each column nk (k = 1, 2, . . . , m) is a binary n-vector (consisting of zeroes and ones), and each row has a single “1.” A “0” in the ith place of nk indicates that the ith ball is not in the kth cell, and “1” there tells us that the ith ball is in the kth bin. Equivalently, we can specify directly what is the content of each bin. This is best demonstrated by an example. Let m = 3 and n = 2, with the two balls labeled with “a” and “b.” The occupancy vector (1,0,1) from §8.2.1 corresponds here to two configurations (a, 0, b) and (b, 0, a), when the two balls are labeled with “a” and “b.” We list all such configurations with one or two balls: a:

(a; 0; 0), (0; a; 0), (0; 0; a).

b:

(b; 0; 0), (0; b; 0), (0; 0; b).

ab:

(a, b; 0; 0), (0; a, b; 0), (0; 0; a, b), (a; 0; b), (b; 0; a), (a; b; 0), (b; a; 0), (0; a; b), (0; b; a).

440

CHAPTER 8. FURTHER ENUMERATION METHODS "

# 100 The first configuration in the line ab is represented by the occupancy matrix . 100 This is the most detailed occupancy description. Notice that if we went further and ordered balls in each bin according to the order they reached the bin, we would be describing the allocation process, not just the resultant occupancy. However, we do not usually need the complete specification. A complete specification is such that can be realized in one way only, see Eq. (8.21). There it specifies how many balls end in each cell, whereas here it even details which cell receives each ball. Every application of such occupancies we encountered is satisfied by the higher level description, the occupancy vectors used in §8.2.1. To count the number of ways in which this specification is realized, we need to aggregate many of the current details. We will use the vector notation n = hn1 , n2 , . . . , nm i, where ni (i = 1, 2, . . . , n) is the count of balls in each i-th cell. Often, even with balls that are distinct, we do not really need this level of detail, as further applications show, but it is still true that the count of distinguishable configurations determines the statistical properties of the process. We proceed to construct the appropriate enumerators. Let us start with S1 , the set of occupancies created with a single ball. Dropping it into each one of m cells produces m configurations: (1; 0; 0; . . . ; 0), (0; 1; 0; . . . ; 0), (0; 0; 1; . . . ; 0), . . . , (0; 0; 0; . . . ; 1). As in §8.2.1, we associate the variables zk , k = 1, 2, . . . , m, with the corresponding bins. Note that we introduce an asymmetry between balls and cells. While both are distinct and numbered (labeled), our treatment will continue to give bins a preference, and we do not associate variables with balls; their labels are only used to distinguish between arrangements that can be fold apart, as in the previous example with balls “a” and “b.” Thus, the ultimate result of the present calculation is to find formulas for the number of occupancy vectors n = hn1 , n2 , . . . , nm i. We ask about the number of balls in each bin, mindful of the way in which the labeling of the balls influences the counting. However, we are not interested in which balls are in each place. A term used conventionally in this context is indicator; it has exactly the same meaning as enumerator. Having zk as the indicator of placing a ball in the kth cell, we invoke the sum rule to obtain the enumerator of one ball in our array of m urns:

ϕ1 (z) = z1 + z2 + · · · + zm , where as before z stands for the vector h z1 , z2 , . . . , zm i.

Now suppose that the balls are all distinguishable, so we can create Sn , the set of occupancies of n balls, as an n-fold Cartesian product: (1)

(2)

(n)

Sn = S1 × S1 × · · · × S1 ,


441

(k)

where S1 is the set of occupancies of the kth ball in m bins. This set has the same enumerator ϕ1 (z), for all k ∈ [1..n]. Hence the enumerator (OGF) of Sn is the product of all the individual ones: ϕn (z) ≡ ϕn (z1 , z2 , . . . , zm ) = ϕ1n (z) = (z1 + z2 + · · · + zm )n . (8.25)

Here zn is a conventional symbol for zn11 zn22 · · · znmm , and the summation is over all vectors n = hn1 , n2 , . . . , nm i, restricted to a sum of components equal to n = n1 + n2 + · · · + nm . We have seen the right-hand side of Eq. (8.25) before—in §2.4, Eq. (2.46) and in §7.3.3, Eq. (7.17). It is a multinomial, and what we get when we extract coefficients from it are the multinomial coefficients: n . [zn ] ϕn (z1 , z2 , . . . , zm ) = n1 , n2 , . . . , nm This provides us with the number of ways in which the configuration n = hn1 , n2 , . . . , nm i, with ni distinct balls in bin number i can arise. What do we get if we want to find the total number of configurations with n balls—regardless of where and how many balls land in each bin? We need to sum over all the multinomial coefficients, but this sum is immediate: n n n n (z1 + z2 + · · · + zm ) = ∑ z =⇒ ∑ = (z1 + z2 + · · · + zm )n |z=1 = mn . (8.26) n n n n We have found now, under two different assumptions about the balls to be distinguishable or not, two formulas for the number of ways to distribute n objects in m bins, Eqs. (8.18) and (8.26). The remarkable feature of these formulas is that we find them in nature! More precisely, physicists have found that to explain observed phenomena they sometimes need to assume that all mn configurations of their n particles in m possible states are equally likely, whereas other systems behave as if they have only m+n−1 distinguishable, and equally likely n states. In statistical physics, one says that distinguishable (or indistinguishable, respectively) particles that are not subject to the Pauli exclusion principle5 obey Maxwell-Boltzmann statistics (or Bose-Einstein statistics, respectively). However, if the particles are identical and are subject to the exclusion principle, they obey Fermi-Dirac statistics. For example, electrons, protons, and neutrons obey Fermi-Dirac statistics, whereas photons and pions obey BoseEinstein statistics. Exercise 8.17 [2] Compare the values produced by Eq. (8.26) with the number of distinguishable occupancies of n identical balls in m bins, as given in Eq. (8.18), by calculating their ratio. In addition to using the Stirling approximation, you may want to use MAPLE and calculate them for a large range of values m and n up to a few hundreds (which will help you to develop intuition about these results). Example 8.18 (Convolution of rolling k-sided dice) What is the number of ways to place n distinguishable balls into m unlike urns so that each urn receives at least 1 ball, but no more than k balls? We discussed this problem in §7.3.3. 5 At

most one particle in each cell.


442

The problem is equivalent to counting the number of ways that m (ordered) values, chosen from the set [1..k], can sum to n. We denote this number by W (n, m, k). Another view of this is the number of ways to get a total of n by rolling m times a true k-sided die with faces labeled 1, 2, . . . , k. If there is no upper limit on the number of balls in an urn (k > n), then the number of ways to put n balls into m urns is n+m−1 , see Eq. (8.18) on page 435. We know the value of n W (n, m, k) from Eq. (7.18) on page 369 and therefore the probability of getting the sum of m rolls of k-sided dice to be n is W (n, m, k)/km . It is of certain interest to find the cumulative distribution of the corresponding random variable because it gives a finite approximation to the convolution of the uniformly distributed random variables. Let Xm,k be the sum of m rolls of a balanced k-sided die. The probability that this sum is less than or equal to n is  ⌊ n−m  k ⌋ 1 j m n− jk ,  (−1) if m 6 n 6 km, ∑ m  n j=0 k j m W (n, m, k) = (8.27) Pr[Xm,k 6 n] = ∑ 1, if n > km,  km  j=m  0, otherwise. The proof of Eq. (8.27) is almost trivial and it is based on changing the order of summation: ⌊ n−m k ⌋

∑

j=0

⌊ n−m n k ⌋ s − jk − 1 m n − jk j m (−1) . ∑ s − jk − m = ∑ (−1) j j s=m+ m j=0 jk j

Using bin enumerators We obtain additional information, and probably a clearer view of the scene when we abandon the fixed value of n, and instead look at the generating function of the multinomial enumerators over all n. Since the balls are labeled, we are led to use their exponential generating function:

ϕ (z,t) =

∑ (z1 + z2 + · · · + zm )n

n>0

tn = et (z1 +z2 +···+zm ) = etz1 × etz2 × · · · × etzm . n!

(8.28)

This allows us to recover the result given in Eq. (8.26): the number of all configurations with n balls is n m n mt n = mn . n! [t ] ϕ (1,t) = n! [t ] e = n! n!

This displays what we referred to as a clearer view: the enumerator given in Eq. (8.28) “allocates” a separate factor to each bin. This is similar to the ordinary enumerators for the number of compositions when the balls were not distinguishable; now we obtain this separation by using exponential enumerators. This is one more example of the general working rule that labeled structures are enumerated better with the EGF, rather than the OGF. Here is the enumerator associated with the occupancy of bin number i: etzi = 1 + t zi +

t 2 z2i t 3 z3i + + ··· . 2! 3!

(8.29)


443

The first term (1) corresponds an empty urn, the second term (t zi ) corresponds to one element in the ith urn, the third term (t 2 z2i /2!) represents two elements there, and so on. This knowledge may be used to find the number of ways to put n distinct objects into m ordered cells subject to any desired specification. For example, if the ith cell may not be empty, its enumerator is etzi − 1. If we have m bins, the enumerator under the requirement that the first three may not to be empty would be etz1 − 1 etz2 − 1 etz3 − 1 etz4 · · · etzm . and if the first cell must contain at least 2 and at most 3 balls, it contributes the enumerator 22 z31 t 3 z1 t + × etz2 × · · · × etzm . 2! 3!

If the first bin must contain even number of balls and the second one only odd number of balls, the corresponding enumerator becomes cosh (tz1 ) × sinh (tz2 ) × etz3 × · · · × etzm . Once the enumerator, which consists of the desired “bin-enumerators,” has been created, we need to calculate n![t n ] ϕ (1,t) in order to find the total number of configurations with n items that can arise in the scenario that enumerator provides.

Thus, the number of configurations of n distinct balls into m different bins, with no bin empty, is tz m n tzm n t n tz1 2 = ∆m 0n , (8.30) n! [t ] e − 1 e − 1 . . . e − 1 z=1 = n![t ] e − 1 = m! m where the symbol mn denotes a Stirling number of the second kind, presented on page 261. The sum of these numbers over the lower parameter forms the sequence, known as the Bell numbers, discussed in Exercises 6.105 and 6.106 on page 314. Without any problem, we find the probability that r specified (for simplicity, they are assumed to be the first r) bins are empty when n distinct balls are randomly distributed into m distinguishable bins to be n! n! m−r m tzi [t ] ∏ e (8.31) = n [t m−r ] e(m−r)t . n m m i=r+1 zi =1

tzi t(zr+1 +···+zm ) does not prohibit that some other than Note that the enumerator ∏m i=r+1 e = e specified cells could be empty.

Exercise 8.19 [2] Use an appropriate enumerator to show that the number of configurations n of n labeled balls in m ordered bins that leave exactly r bins empty is given by m! r! m−r . How does this result lead to the formula (C.20) in Appendix C?

444


Exercise 8.20 [1] Let an be the number of ways to place n people (assumed distinct; no clones admitted) into four different rooms, with at least one person in each. Write the exponential generating function of this sequence.

Counting bin occupancy profiles Let us change the point of view we have been using, and remove the asymmetry between balls and bins. We have done this in part in the last few examples, but we shall go now and create a new complete specification. We say that a configuration of n (labeled) balls in m (ordered) bins satisfies the specification a = ha0 , a1 , . . . , an i, when ai is the number of bins which contain exactly i balls each. Naturally we must require that the sum of the ai is m, and that ∑i i ai = n. We want to count these configurations. Naturally, we do this by producing the appropriate enumerator. By the discussion following Eq. (8.29) we are led to consider the enumerator a n (tz)i i ∏ i! . i=0 Note that here all the z j are represented by a plain z since this type of specification obliterates the identities of the bins. Further, we are yet free to choose which a0 bins will be the empty ones, which a1 bins will be handed a single ball and so on. The numberof ways to produce the complete allocation is simply the multinomial coefficient a0 ,a1m,...,an , and the complete enumerator for this specification is n a m (tz)i i . (8.32) ϕ (t, z) = i! a0 , a1 , . . . , an ∏ i=0 Extracting from this the number of corresponding configurations is immediate: n m t iai m t ∑i iai n n n = n![t ] n![t ] ∏ n![t ]ϕ (t, 1) = ai a0 , a1 , . . . , an a0 , a1 , . . . , an ∏ni=0 (i!)ai i=0 (i!) m n! , (8.33) = a 0 a0 , a1 , . . . , an (0!) (1!)a1 · · · (n!)an since ∑i i ai = n, giving us a formula we find notable for its relative simplicity. Exercise 8.21 [3] A calculation that comes up in hashing: sending v keys into a table of d slots, we define the random variable X as the number of keys which take no part in collisions. In the language of this subsection, X is the value of a1 . What are the distribution and expected value of X ? Note: It looks tempting to approach this problem using Eq. (8.32), but in fact the approach that led, e.g., to Eq. (8.30) is easier. Exercise 8.22 [2+] In the analysis of a simple Leader Election protocol, we need to calculate the probability, p(n, r), that placing n balls (processes) in r bins (signal types) results in at least one bin having exactly one ball.


445

(a) Compute an explicit expression for p(n, r) (an r-term sum, for which we do not have a closed form). Note: This question is very similar to Exercise 8.21. (b) Consider the probability you computed as the probability of success of a Bernoulli trial. We are looking to minimize the expected duration of the related geometrical random variable, with a twist: the time to conduct each experiment is linear in r1/2 . Hence the intent is to minimize r1/2 /p(n, r). Find numerically how the optimal values of r vary with n. What are the values of p(n, r) for those optimal r? Can you explain why this is a reasonable result (or argue otherwise). Exercise 8.23 [3] (M. Dwass) Suppose n labeled balls are randomly distributed among m ordered boxes. Let Y be the number of boxes occupied by two or more balls. Find its enumerator and show 1 n−2 n k+1 1 n n n− j+1 m . m = n∑ Pr[Y = 1] = n ∑ m j=2 j m k=0 k

8.2.3 Distribution of Identical Objects into Identical Cells Consider now placing n identical balls in m identical bins. How can these distributions be counted? What does it mean for two arrangements to be different? Let m = 3, n = 2 and we denote each ball by b, and a bin by a pair of parentheses. Then the occupancies ( ), (b, b), ( ) and ( ), ( ), (b, b) are identical, and differ from (b), ( ), (b). In terms of the classes and types of combinatorial structures considered in Chapter 7, we create by such an allocation an m-element multiset over N, with the weight n, assuming each ball carries the weight 1. This gives us the following result: the number of ways of putting n identical objects into m like cells, with no cell empty, is pm (n), the number of partitions of n into m parts, and the number without restriction is pm (1) + pm (2) + · · · + pm (n) . Exercise 8.24 [1+] Explain how this situation is identical to the problem of counting partitions. You need to consider the fact that we can have here empty bins, and their total number is fixed, whereas we did not use zero parts in §7.5. Hint: Consider partitions of n + m to at most m parts, and show a suitable bijection.

8.2.4 Distribution of Distinct Objects into Identical Cells Finally, we consider briefly the situation when we place distinct balls into identical bins. Like the previous case we do not dwell on this one; not only because of paucity of explicit results, but because both are less natural (and useful) for analysis of algorithms. Note that

446


in this case the balls, because they are labeled, distinguish between the bins, bringing back a situation discussed in §8.2.1, so long as we insist that no bin be left vacant. The number of ways that n different objects can be put into m like urns is the same as the number of partitions of a set of n elements into m subsets. Let G(n, m) denote the number of distributions of n distinct balls into m like bins, i.e., bins that are cannot be distinguished in any way. In other words, G(n, m) is the number of separations of n distinct objects into m or fewer batches because one or more of the batches may be empty. Now let g(n, m) denote the number of distributions of n different balls into m like bins, with no bin empty. In general, we can separate the G(n, m) distributions into those where no bin is empty, those where exactly one bin is empty, those where exactly two bins are empty, and so on, to get G(n, m) = g(n, m) + g(n, m − 1) + g(n, m − 2) + · · · + g(n, 1) . However, there is a simple relation between g(n, m) and m! mn , the number of distributions of n different balls into m distinct bins, with no bin empty, determined by Eq. (8.30). It parallels the relationship between combinations and permutations in the elementary theory. To see this, consider any distribution counted by g(n, m); since there are m! ways of numbering the m bins to change them from like to unlike bins, this leads to the formula g(n, m) = mn .

8.3 The Principle of Inclusion and Exclusion ( PIE) The principle of inclusion and exclusion (PIE) is one of the most common and intuitive enumeration procedures. It is a straightforward generalization of the situation presented in Eq. (3.1), page 81: if we form the set C as the union of the sets A and B, which include |A| and |B| elements respectively, then C consists of |A| + |B| elements if the two sets are disjoint; otherwise, any terms common to both sets, of which there are |A ∩ B|, are counted twice. To correct this we use the following relation |A ∪ B| = |A| + |B| − |A ∩ B|.

(8.34)

Example 8.25 In a zoo, we observe a group of 100 different species and find that 60 are mammals, that 75 are predators, and that of those, 45 are mammals. How many are in the union class: the set of predators or mammals? Solution: This can be found from direct substitution in Eq. (8.34), once we identify set A with the mammals, set B with the predators, and observe that the third data point tells us that |A ∩ B| = 45. Hence we have 60+75-45=90 species that are either mammals or predators.

Recall that S′ denotes the complementary set to S. One of De Morgan’s laws states the set equality, A′ ∩ B′ = (A ∪ B)′ , which translates to equality of sizes. Hence we find immediately that |(A ∪ B)′ | = 100 − |A ∪ B|, which we have just found to be 10, this is the number of species that are neither predators nor mammalian.

8.3. THE PRINCIPLE OF INCLUSION AND EXCLUSION (PIE)

447

Example 8.26 In how many ways can we order the 10 digits, 0–9, so that the first digit is at least 2, and the last is at most 8?

Solution: Each condition excludes one or two digits in a specified position. This exclusion defines a subset of the “permitted” permutations, out of all 10!. We can identify the sets of permutations created by the restrictions in positions 1 and 10, with the sets A and B, respectively. Hence the answer is given by |A ∩ B|, the right-most term in Eq. (8.34). There is no direct way to count the strings in the set A∩B since the conditions interact. This means that there are digits that are candidates for both positions, while satisfying the above conditions. Hence, we need to know all other sizes that appear in that relation to find the desired size. It turns out that they are all easier to tackle by considering their complements. First, for A: if we forbid 0 and 1 in the first position we give up on 2 × 9! permutations. Hence, |A| = 10! − 2 × 9! = 8 × 9!, and similarly |B| = 10! − 1 × 9! = 9 × 9!. How many strings satisfy either condition? Again we go for the complement, and consider the 8! strings that have 0 and 9 at the end positions, and a similar number that have 1 and 9 there. Together this is the set A′ ∩ B′ = (A ∪ B)′ , hence |A ∪ B| = 10! − 2 × 8! = 88 × 8!, and finally, the desired set has the size |A ∩ B| = |A| + |B| − |A ∪ B| = (72 + 81 − 88) × 8! = 65 × 8! = 2, 629, 800, which is 65/90 ≈ 0.7222 of the total.

Now suppose we add in the last example the condition that the digit in position 4 must be even. How many permitted permutations exist now? The new condition interacts with the previous conditions—clearly with the definition of the set A. It can be done directly, but this approach gets rough, and with additional conditions the calculation can get easily out of control. For such calculations, a unifying framework is needed, and the PIE provides it. Consider the following abstract set-up: given a finite set, S, of elements and a list, P, of d properties or conditions Pi , i = 1, 2, . . . , d. Each element σ ∈ S may or may not satisfy6 any property Pi. Consequently, each property carves out a subset of S, containing just those elements that satisfy the property. It is convenient (and common) to use the same symbol Pi to denote the subset as well as the property. Hence we can write Pi = {σ ∈ S : σ satisfies the property Pi }. In other words, the properties define d subsets of S that we name Pi , i = 1, . . . , d. These sets play the same role below as the sets A and B above. Note that elements from the set Pi may satisfy other properties. The intersection of these sets can be written as Pi1 Pi2 . . . Pik = {σ ∈ S : σ satisfies each of the properties Pi1 , Pi2 , . . . , Pik }, instead of Pi1 ∩ Pi2 ∩ . . . ∩ Pik , omitting the symbol ∩. We denote by Pi (or Pi′ ) the subset of the set S that consists of all elements not having the property Pi (the complement of Pi ). 6 We

also say that the element has the property.


448

Let α = {i1 , i2 , . . . , ik } be a list of k indices, specifying a k-subset of properties. For every such list we create two enumerating functions: e(α ) – the number of elements from S that satisfy each of the properties Pi , i ∈ α , and no others (“exact” enumeration); n(α ) – the number of elements from S that have each of the properties Pi , i ∈ α , and possibly others as well (“at-least” enumeration). Note: Enumeration (number of elements) is a particular case of a general weight function that assigns to every element of a set a non-negative number (see Definition 7.1 on page 356), not necessary 1. Therefore all formulas in this section hold when the cardinality of a set, |A|, is replaced by its weight, w(A). Sometimes we will denote by e(Pi1 , . . . , Pik ) and n(Pi1 , . . . , Pik ) the numbers e(α ) and n(α ), respectively, when the list α is short. In particular, n(Pi Pj ) = |Pi Pj | because n(α ) = Pi1 Pi2 . . . Pik when α = {i1 , i2 , . . . , ik }. For the exact count, e(α ), we have e(α ) = Pi1 Pi2 . . . Pik Pik+1 . . . Pid .

The essence of the PIE is that it relates the exact counts, which are of interest, to the at least counts that are easier to obtain, usually. In its basic form the PIE is expressed as follows: to compute e(α ) we need to sum on all the lists of properties β , that include α : e(α ) =

∑

(−1)|β \α | n(β ),

(8.35)

β : α ⊆ β ⊆P

where β \ α is the set difference, the properties in β that are not in α . S 2 5 6 7 1 3

4

Figure 448: Venn diagram.

The Venn diagram in the accompanying figure illustrates Eq. (8.35) for α = {1, 2}. The diagram is isomorphic with the one on page 126 but tells a different story. The properties P1 , P2 , and P3 are associated with circular subsets of S. The set P1 consists of regions 1,4,5,7, the set P2 consists of regions 2,5,6,7, and the set P3 —the regions 3,4,6,7. The left-hand side of Eq. (8.35) consists of the region 5. The function n(α ) = n(P1 , P2 ) counts all the elements that satisfy P1 and P2 , which consist in the regions 5 and 7. We write the properties lists that include α in the following table:

β Set Regions β \ α |β \ α | n(β ) 1, 2 P1 ∩ P2 5, 7 ∅ 0 |P1 ∩ P2 | 1, 2, 3 P1 ∩ P2 ∩ P3 7 3 1 |P1 ∩ P2 ∩ P3| Substitution into Eq. (8.35) yields e(α ) = n(P1 , P2 ) − n(P1 , P2 , P3 ) ≡ |P1 ∩ P2 | − |P1 ∩ P2 ∩ P3 |,

region 5.


449

Example 8.27 Here is a set of 22 words collected more or less at random: {ability, arid, boar, chin, crab, dauphine, devour, dumb, durable, grain, great, next, note, pause, prior, rebuttal, sable, shot, spout, suit, tuna, vine}. We select five properties, Pa , Pe , Pi , Po , Pu , for the usual five vowels, and adopt the natural meaning that a word satisfies a property if it has the corresponding vowel. Let us write out a few examples: e(Po ) = |{ shot }| = 1, e(Pa , Pu ) = |{ tuna }| = 1, e(Pa , Pi ) = |{ ability, arid, grain }| = 3. Also, n(Po ) = |{ boar, devour, note, prior, shot, spout }| = 6, n(Pa , Pu ) = |{ dauphine, durable, pause, rebuttal, tuna }| = 5, and n(Pa , Pi ) = |{ ability, arid, dauphine, grain }| = 4. To find the number of words that contains a and u and no other vowel, we would need to compute the at-least counts n(a, u), n(a, u, e), n(a, u, i), n(a, u, o), n(a, u, e, i), n(a, u, e, o), n(a, u, i, o), n(a, u, e, i, o), which are, in this order, 5, 4, 1, 0, 1, 0, 0, 0. Summing, with the signs, we find 5 − 4 − 1 + 1 = 1, as it should be. One can not say much about these counts for an arbitrary list α . So we make an “averaging” of sorts: we sum over all selections α of a fixed list size k ∈ [1..d] to obtain the counts ek ≡

∑

e(α );

nk ≡

α : |α |=k

∑

n(α ),

k = 1, 2, . . . , d.

(8.36)

α : |α |=k

The ek is the number of elements having exactly k properties since each α in the sum (8.36) is different from the other ones, and no element is either missed or counted more than once: ek =

∑

Pi1 Pi2 . . . Pik Pik+1 . . . Pid .

(8.37)

16i1 k − e>k+1 (k = 0, 1, 2, . . . , d − 1), we get e>0 = e0 + e>1 . Hence e0 = |P1 P2 . . . Pd |,

n0 = e>0 = |S|,

e>1 = |P1 ∪ P2 ∪ · · · ∪ Pd |.

(8.38)

The last values ed = e>d = nd are all equal because there are no more properties than d; therefore the items that satisfy at least d properties also satisfy exactly d properties. Exercise 8.28 [2–] For the data in Example 8.27, compute nk and ek for 0 6 k 6 5.

That nk is typically much larger than e>k , which can be readily seen when we observe that they satisfy the following equations: d j j−1 def nk = ∑ e j =∑ (8.39) e> j , where e>k = ∑ e j . k k − 1 j>k j=k j>k

Proof: The relations follow from a careful reading of the definitions of nk , and of ek , in Eq. (8.36): every list α of size k contributes to nk when it consists of all elements that possess at least k properties. This means that nk can be divided intodisjoint subsets of elements having exactly j properties, where j > k. Since there are kj different lists where it can happen, the first relation in Eq. (8.39) follows. To prove the second part, observe that e j = e> j −e> j+1 , j = 0, 1, . . . , d −1, and e>d = ed . Substituting these difference equations into the first relation (8.39), which just has been proved, we get d d d−1 j j j+1 j nk = ∑ e j =∑ [e> j − e> j+1 ] = e>k + ∑ − e> j+1 . k k k j=k j=k k j=k Using the computational identity (2.59) on page 58, we obtain j j j j k j+1 j j+1 − = = . − = j+1−k k j+1−k k k k−1 k k Hence

d−1

nk = e>k + ∑

j=k

d j i−1 e> j+1 = e>k + ∑ e>i , k−1 i=k+1 k − 1

from which the second part follows. The sequences ek and nk , k ∈ [1..d], are associated with the ordinary generating functions E(z) =

∑ ek zk , k>0

N(z) =

∑ nk zk . k>0

(8.40)


451

Theorem 8.29 (Principle of Inclusion and Exclusion.) The ordinary generating functions of the exact and at-least counts satisfy the relation N(z) = E(z + 1).

(8.41)

P ROOF : Having shown Eq. (8.39), the rest is immediate: j j k N(z) = ∑ nk zk = ∑ zk ∑ e j = ∑ ej ∑ z = k k j>0 j>k k>0 k>0 k

∑ e j (1 + z) j = E(1 + z).

j>0

A well-known text on combinatorics refers to this statement of the PIE as an elementary lemma. Surely elementary, we have just given it a one-line proof! And yet we think that a “celebrated theorem” reflects its significance better, and reinforce our attitude to it by proving the PIE in yet another way, less technical, and possibly more revealing. For an item σ ∈ S, let I(σ ) be the subset of properties that σ has, and let i(σ ) be its size, that is, i(σ ) = |I(σ )|. Then ( 1, if σ has exactly k properties; ek = ∑ δk,i(σ ) , where δk,i(σ ) = 0, otherwise. σ ∈S Therefore we have E(z) =

∑ ek zk = ∑ zk ∑

σ ∈S

k

k>0

δk,i(σ ) =

∑ ∑ δk,i(σ ) zk = ∑ zi(σ ) .

σ ∈S k

σ ∈S

On the other hand, N(z) =

∑ nk zk = ∑ zk ∑ k>0

k>0

n(α ).

α : |α |=k

By definition n(α ) =

∑ [ σ satisfies α ] = ∑ [ α ⊆ I(σ ) ],

σ ∈S

[C ] =

(

1, 0,

if the condition C is satisfied, otherwise,

N(z) =

is the Iverson bracket. Thus,

∑ zk ∑ ∑ [α ⊆ I(σ )] = ∑ ∑ zk ∑

k>0

=

where

σ ∈S

α :|α |=k σ ∈S k

∑ ∑z

σ ∈S k>0

because, for every k, there are

σ ∈S k>0

α :|α |=k

[α ⊆ I(σ )]

i(σ ) = ∑ (1 + z)i(σ ) = E(1 + z) k σ ∈S

i(σ ) k

ways to select suitable subsets in I(σ ).


452

Corollary 8.30 The exact counts are given in terms of the at-least counts via ek =

∑ n j (−1)

j−k

j>k

j , k

d

def

e>k =

∑ e j = ∑ n j (−1) j>k

j−k

j=k

j−1 , k−1

k > 0,

(8.42)

with the boundary condition ed = e>d . In particular, for k = 0 we obtain the sieve formula7 : e0 =

∑ n j (−1) j .

(8.43)

j>0

P ROOF : Ii immediately follows by extraction of the coefficient of zk on both sides of the equation E(z) = N(z − 1) to obtain j ek = [zk ] E(z) = [zk ] N(z − 1) = [zk ] ∑ n j (z − 1) j = ∑ n j [zk ] (z − 1) j = ∑ n j (−1) j−k . k j>0 j>0 j>k The sieve formula is derived by substituting there k = 0. Actually we don’t need to prove the first formula that relates ek and nk because together with Eq. (8.39) it is an example of inverse relations for the binomial coefficients, Eq. (A.60). The second part follows from the first one and Eq. (A.19): d

e>k =

d

∑ ej = j=k

d

∑ ∑ ni (−1)i− j j=k i= j

d d d i i i−1 = ∑ ni ∑ (−1)i− j = ∑ ni (−1)i−k . j j k−1 i=k j=k i=k

The sieve formula (8.43) can be written in a notation that evokes the properties directly: e0 = |P1 P2 . . . Pd | = |S| − ∑ |Pi| + ∑ |Pi Pj | − . . . + (−1)d |P1 P2 . . . Pd |, i

(8.44)

i< j

where P (also denoted as P′ ) is the complement of the set P and Pi Pj ≡ Pi ∩ Pj is the intersection of Pi and Pj . Recall that S is a finite set and P1 , P2 , . . . , Pd are some subsets (or properties that identify the subsets) of S. Using the De Morgan formula (P1 ∪ P2 ∪ . . . ∪ Pn )′ = P1′ P2′ . . . Pn′ and the equation |P′ | = |S| − |P|, we obtain another variant of the sieve formula: e>1 = |P1 ∪ P2 ∪ . . . ∪ Pd | = |S| − e0 = ∑ |Pi | − ∑ |Pi Pj | − . . . + (−1)d−1 |P1 P2 . . . Pd |. (8.45) i

i< j

We present another proof of the latter result based on the properties of the characteristic function χA : S → {0, 1}. Recall that χA (x) = 1 if x ∈ A and χA (x) = 0 otherwise. The cardinality of A can now be written as

|A| = 7 Sometimes

∑ χA (x).

x∈S

this formula is called after James Sylvester (1814 – 1897).


453

Also we need the formulas

χA∩B = χA χB

χA = 1 − χA ,

and

where 1 = 1(x) is the unit function: 1(x) = 1 for all x ∈ S. Now we start with the left-hand side of Eq. (8.45):

χP1 ∪P2 ∪...∪Pd = 1 − χP1 ∪P2 ∪...∪Pd = 1 − χP1 ∩P2 ∩...∩Pd

= 1 − (1 − χP1 )(1 − χP2 ) . . . (1 − χPd )

= ∑ χPi − i

=

∑

16i6d

∑

16i< j6d

χPi −

∑

χPi χPj +

∑

16i< j6d

16i< j0 αn n! Fˆα (z) = ez Fˆβ (−z),

Fˆβ (z) = ez Fˆα (−z),

(8.53)

due to the convolution rule (6.11) on page 279. The sequences {αn } and {βn } (or functions α (C) and β (C)) that satisfy Eq. (8.52) (or Eq. (8.51)) are called symmetric inclusionexclusion pairs. These formulas are naturally extended to probabilities (when dividing by the total number of elements); so the functions α (z) and β (z) can be defined as β (C) = Pr [∩i∈C Ai ] , α (C) = Pr ∩i∈C Ai , for homogeneous set of events Ai (i = 1, 2, . . . , d). More applications of PIE to probabilistic problems are given in §8.5.

Example 8.41 A very simple example of homogeneous properties gives an experiment of flipping a coin, which results heads with probability p and tails with probability q = 1 − p. Let Ai be the property (event) of seeing a head on the i-th flip. Then for any finite set of indices C, we have β (C) = Pr [∩i∈C Ai ] = p|C| , α (C) = Pr ∩i∈C Ai = q|C|. Then the formula (8.51) gives the Binomial Theorem (2.52): n n n k n (1 − p) = q = ∑ (−1) pk . k k=0


457

Example 8.42 (Pólya urn model) The idea to use urns containing balls of different colors in order to model physical phenomena seems to be due to G. Pólya. Some simple variants of the model were discussed in Exercise 3.74 (on page 111) and Example 4.176 (on page 196). Here we consider another particular case of such model that is based on a classical probabilistic model called the Pólya or Pólya-Eggenberger urn model. More general problem together with many examples can be found in [41, 96]. Initially a box contains w white balls and b black balls from which a ball is drawn at random equiprobably (so on first stage, the probability of choosing a white ball is w/(w + b) and the probability of picking a black ball is b/(w + b)). When a ball is drawn, its color is observed and the ball is replaced by two balls of the same color. This procedure is repeated forever. By ball j, we shall understand that the ball drawn at the j-th step. For any two disjoint set of positive integers W = {i1 , i2 , . . . , im } and B = { j1 , j2 , . . . , jn } from the set [1..n + m], we define the event, En+m , of choosing m + n balls in the sequences W and B. We claim that the probability of the event En+m depends only on the cardinality of the sets W and B, so the corresponding properties are homogeneous: Pr [En+m ] =

wm bn . (w + b)n+m

(8.54)

Its proof can be done by induction argument as follows. The basic step, m + n = 1, is satisfied because the procedure of choosing a ball is uniform. Assuming that the inductive hypothesis is valid up to m + n, we consider the next step of picking (m + n + 1)-st ball. At this stage, we have w + m white balls and b + n black balls in the box, that make tally to be w + b + m + n. Then the probability to choose a white ball is (w + m)/(w + b+ m + n) and the probability to pick up a black ball is (b+ n)/(w + b+ m + n). The events Ei (i = 1, 2, . . .) are not independent, so we use the conditional probability (3.30): Pr[Ei+1 ] = Pr[Ei ∩ Ei+1 ] = Pr[Ei ] Pr[[Ei+1 | Ei ] when i = m + n. Substitution into the above equation the given formula (8.54) (it is true due to the inductive hypothesis) and the conditional probabilities ( (w + m), if the next ball is white, 1 × Pr [Em+n+1 | Em+n ] = w+b+n+m (b + n), if the next ball is black, completes the proof. Now let Pi be the property (event) that the ball i is white, so Pi is the property that it is black. For any set C of positive integers, α (C) is the probability that all the balls with numbers in C are white, and β (C) is the probability that all the balls with numbers in C are black. Hence

α (C) =

w|C| (w + b)|C|

,

β (C) =

b|C| (w + b)|C|

,

458


where wk is the k-rising factorial, see Eq. (2.50) on page 52. Taking cardinality of the set C to be n, we get from Eq. (8.52) that n n wn wk bk bn k n k n , . = (−1) = (−1) ∑ ∑ (b + w)n k=0 (b + w)n k=0 k (w + b)k k (w + b)k The corresponding exponential generating function identity (8.53) yields wn zn zn bn n z (−1) = e , ∑ ∑ n (b + w)n n! n>0 n>0 (b + w) n! which is the well-known 1 F1 transformation for the hypergeometric functions: b w z z = e 1 F1 −z . 1 F1 b + w w + b

Example 8.43 Using the Inclusion-Exclusion Principle, we recover the result in Eq. (7.18), page 369, for the number of compositions a1 + a2 + · · · + ak = n in positive integers not exceeding m. We say that a composition satisfies the property Pi if ai > m, for i ∈ [1..k]. These k properties are homogeneous, and so we can use the formula (8.48). According to Eq. (7.22) on page 372, taking into consideration that ai > 1, we find the number of compositions that j−1 . Hence the number of compositions with no property satisfy j properties to be β j = n−m k−1 satisfied is n n−mj−1 j k . (8.55) |P1 P2 . . . Pk | = ∑ (−1) j k−1 j=0 n 3n Example 8.44 (D. Clark) The following binomial identity 3 ∑ = 8n +2(−1)n can 3i j=0 be established not only with the multisection method (see §6.1.5), but also with an approach that has the same flavor as the PIE, while not using it directly. We want to add every third binomial coefficient in the last (3n-th) row. Now, these terms are connected to the row above by the basic two-term recurrence (2.62) for the binomial coefficients (addition formula). So, add all the elements in the next to the last row (the sum of the binomial coefficients in the row 3n − 1 is 23n−1 ). Remember that the black circles indicate terms which are added. But this adds too much—in the second-to-last row, the 2nd, 5th, 8th, . . . terms are not involved in the sum of every third term in the last row. So we have to subtract these away. Remember that white circles indicate terms which are subtracted. How to do it? Look up one more row and subtract all the terms (total 23n−2 ). But now this subtracts too much, namely the 3rd, 6th, 9th, . . . terms in the third row from the bottom. So we have to put these back. Hence we go up another row back and add the terms, but this adds too much, etc., etc. The process finally ends when we have added back the two binomial coefficients in the second row from the top. The result is a simple geometric series with trivial evaluation: 3n−1 3n−1 3n 1 j 8n + 2(−1)n j−1 3n− j 3n = . = −2 ∑ − = ∑ (−1) 2 ∑ 2 3 06 j6n 3 j j=1 j=1


−

− +

−

− +

−

− +

−

− +

+

−

−

−

+ −

+ −

+ −

+

+

+

+

−

−

−

− +

+

+

+

−

−

−

− +

+

+

+

−

− +

− +

− +

459

+ −

+

+

− +

−

− +

+ −

+

− +

+

Figure 459: Example 8.44. Exercise 8.45 [2] Use the PIE to prove the identities:

∑

a1 ,a2 ,...,ak >0

n a1 , a2 , . . . , ak

n = k! = k

k k− j

∑ (−1)

j=0

k n j . j

Recall that the left-hand side gives the number of ways to distribute n (identical) balls into k distinguishable cells with no cell empty. Exercise 8.46 [2] Show that the number of distributions of n identical balls into m distinguishable bins so that exactly two (unspecified) bins contain a balls equals m−2 m+n−2a−3−(a+1)k m (−1)k . k 2 ∑k m−3−k Exercise 8.47 [2] How many permutations π : [1..n] → [1..n] (n > 1) satisfy π (1) 6= 2 and π (2) 6= 1.

Let V be the set of all m-dimensional arrays with integer values from the set [1..n]. For any pair (x, y) ∈ V ×V , define the Hamming distance, ρ (x, y), as the number of positions in which the strings differ. For fixed array u = hu1 , u2 , . . . , um i, the set {x ∈ V | ρ (x, u) = m − r} is called the sphere of radius m − r with the center at u and it is denoted by Sr (u). Since the number of m-vectors in Sr (u) coincides with the number of m − r coordinates of m-vector, we conclude that the cardinality of this sphere is mr (n − 1)m−r .

Exercise 8.48 [2] Using the PIE, show that hmr (n), the number of m-vectors such that ρ (x, y) = m − r, is m! m−r (−1) j n − r − j , r = 0, 1, . . . , m. hmr (n) = r! ∑ j! m−r− j j=0


460

8.4 Extensions and Further Applications of the PIE The Principle of Inclusion and Exclusion (PIE) is one of those methods that looks quite different under various circumstances. Hence further examples of its use should enhance its appeal. Takács in [130] provides a good discussion of the Principle in more abstract settings, but it is not expected we would need those in our analyses. Instead, we incorporate the symbolic method with the PIE. Choosing some n so that 1 6 k + 1 6 n 6 d, where d is the number of properties, we get from Eq. (8.42): n−1 j−k j ek − ∑ (−1) n j = (−1)n−kU (n, k), (8.56) k j=k where U (n, k) can be transformed as follows d

U (n, k) =

∑ (−1) j−n

j=n

j nj = k

d

j

∑ e j ∑ (−1)i−n

j=n

i=n

i j = k i

j j−k−1 e j > 0. ∑ n−k−1 j=n k d

Depending on the parity of n − k, the expression (−1)n−kU (n, k) can be either positive or negative. Disregarding this expression in Eq. (8.56), we make the rest either bigger or smaller. In this way, we obtain so called the Bonferroni inequalities: k+2ν k+2ν −1 j−k j j−k j (8.57) ∑ (−1) k n j 6 ek 6 ∑ (−1) k n j , j=k j=k where 0 6 ν 6 (d − k)/2. They are equivalent to the following two inequalities: j k + 2ν + 1 nj > − nk+2ν +1 , ek − ∑ (−1) k k j=k k+2ν −1 k + 2ν j−k j nk+2ν . nj 6 ek − ∑ (−1) k k j=k k+2ν

j−k

(8.58)

Since the series (8.42) is alternating, the error of disregarding the terms (−1)n−kU (n, k) does not exceed the first missing term (see Theorem 10.5 on page 549). Example 8.49 The variable x is called a boolean variable if it assumes logical values— true and false—usually denoted by 1 and 0, respectively. Let B = {0, 1} and let Bn be its Cartesian product. An element hx1 , x2 , . . . , xn i from Bn is said to be a boolean array, and a function f : Bn → B is referred to as a boolean function of n variables. Two boolean functions, f and g, are called equal or equivalent if and only if f (x1 , x2 , . . . , xn ) = g(x1 , x2 , . . . , xn ) for all entries. For a boolean function f (x1 , x2 , . . . , xn ), the variable xi is said to be not essential if f (x1 , . . . , xi−1 , 1, xi+1 , . . . , xn ) = f (x1 , . . . , xi−1 , 0, xi+1 , . . . , xn ), for all other coordinate values, that is, the function f does not depend on the variable xi , i ∈ [1..n]. A variable is either essential or not-essential for a given boolean function.

8.4. EXTENSIONS AND FURTHER APPLICATIONS OF THE PIE

461

The number of functions that have two possible values, true and false, and a domain of n size 2n (since each xi can be either 0 or 1), is 22 . If we only count functions for which r n−r specific variables, out of the n, are not-essential, there can only be 22 distinct functions. We denote by Ai the set (or property) of all boolean functions on Bn for which the variable xi , i ∈ [1..n], is not essential. The number of boolean functions having (at least) r such properties is n−r |Ai1 Ai2 · · · Air | = 22 . This is “at-least” counts because the intersection includes functions for which additional coordinates are not essential; in particular, it includes the two constant functions, f (x) = true and n−r f (x) = false. Since there are nr ways to select r coordinates, we found that nr = nr 22 .

Let Mn be the set of all boolean functions having all their n boolean variables essential. They have none of the properties Ai , and their number is given by the sieve formula (8.48): n n−1 n n−k k n |Mn | = ∑ (−1) 22 = 22 − n22 + · · · . k k=0 This is not a sum for which we have a closed form, but clearly the first term is the dominant one. In fact, the Bonferroni inequalities (8.58) yield n n n−1 n 22 − n22 6 |Mn | 6 22 =⇒ |Mn | 22 −→ 1. n→∞

“Almost all” the functions have no inessential coordinate.

Example 8.50 The following problem needs some preparatory work to fit in the framework of the PIE. We have 4 words, each with 5 letters. Every two words share 2 letters, and every 3 words have one common letter. However, no common letter is in all four words. Question: how many different letters are used in these words?

Solution. Let S be the set of letters that we use. As usual, n0 = |S| is the number of letters. Let Pi be the property of a letter to be in the ith word, 1 6 i 6 d = 4. From Eq. (8.36), we have n1 = ∑ n(α ) = 4 · 5 = 20. α : |α |=1

Since the number of letters that satisfy any given property is 5 (which we write in the PIE notation as n(α : |α | = 1) = 5), and there are four size-one lists. Then we recall that every two words share 2 letters, and there are 42 pairs of words, so 4 · 2 = 6 · 2 = 12. n2 = ∑ n(α ) = 2 α : |α |=2 Similarly, n3 =

∑

α : |α |=3

4 n(α ) = · 1 = 4, 3


462

and,

∑

n4 =

α : |α |=4

4 n(α ) = · 0 = 0 = e4 . 4

Then the sieve formula produces e0 = n0 − n1 + n2 − n3 + n4 . Finally, since e0 = 0 we obtain n0 = n1 − n2 + n3 − n4 = 20 − 12 + 4 − 0 = 12, hence we need 12 letters to play this game. The following two lists of words satisfy the above conditions, can you find more? trash

sixty

eight

chips.

clear

brash

coast

crisp.

Since we know all the ni , there is no difficulty in obtaining all of the ei as well. The interpretation of the exact counts in this example is the usage frequencies of the letters: it is immediate to compute e1 = 8, e2 = 0, and e3 = 4. Therefore we obtain the curious result that no letter appears in exactly two words. Exercise 8.51 [3] The following problem extends Example 8.50. (a) Suppose the same data is given, except that there are five words (of five letters). What is the number of different letters needed then? What is wrong with this scenario? (b) An obvious extension: four words, six letters in each, three letters are shared by each couple of words, two in each set of three, and there is a common letter to all four. How many distinct letters? Can you find an instance? Exercise 8.52 [3] A man has three friends; he has met each of them at dinner 12 times, every two of them 5 times, and all three — twice. Could this have taken place in a single month? Example 8.53 (The ménages problem) A classical enumeration problem formulated and solved by E. Lucas in 1891 asks: in how many ways can n married couples (in French, ménages) be seated at a round dinner table with men and women alternating, so that no woman sits next to her spouse? This is clearly a problem about counting permutations with restrictions, and as such is directly tractable using the mechanism of rook polynomials, developed in detail in Chapters 7 and 8 of Riordan’s book [114]. We proceed here differently, and use the results derived in Exercise 7.43 on separated samples. Let us begin by assuming that the 2n chairs are distinct, such as each chair has a number. We fix the order of the wives, who can be seated in any of the chairs in n! ways on either the even- or odd-numbered chairs. Note that we have no rotational invariance here because of the


463

seat numbering! Once they are seated, we count the couples, starting arbitrarily with some woman as 1, and continue clockwise along the seating. We then renumber the still-empty chairs, so that those adjacent to the wife from couple #i are the chairs i and i + 1, mod n. Now we consider the so-called reduced ménage problem that is of interest in itself: determine the number of ways to place the men in the remaining seats, so that man #i (his seat we denote by mi ) avoids seats i and i + 1. This number is denoted (following Comtet [30]) by µ (n). Let P2i−1 be a property satisfied by any seating that puts mi in seat i, and similarly, P2i is the property satisfied by any seating where mi is in seat (i + 1 mod n). The PIE now tells us that we need the sizes of all k-fold intersections, 1 6 k 6 2n, of these offending sets of seatings. Any such intersection is empty if it tries to put a man in two different seats; and also, if it shoehorns two men into the same seat. Therefore, it may not contain two adjacent properties from the list {P1 , P2 , P3 , . . . , P2n , P1 }. This immediately limits k: it can only extend from 0 to n. A k-fold intersection (which has no conflicts) fixes the seats of k men out of the n and the rest are free to form (n − k)! permutations, with possibly other prohibited seat assignments among them. How many ways to select k conflict-free properties exist? Part 2n−k−1 . Hence the at-least count nk (b) of Exercise 7.63 gives the answer for k > 0 as 2n k k−1 2n 2n−k−1 is nk = k k−1 (n − k)! for k > 0 and n0 = n!, the number of all possible seatings (on numbered chairs). The number of ways to seat the men properly is given by Eq. (8.42) as 2n−k−1 (n − k)!, and the total number of arrangements needs yet µ (n) = n! + ∑nk=1 (−1)k 2n k k−1 to be multiplied by 2n!/(2n) = (n − 1)!, taking into account the actual rotational invariance of the seating. Example 8.54 (The bridge couple problem) Suppose n married couples are invited to a bridge party where 2n people are randomly paired without regard to gender. In how many ways can it be done so that no one will be chosen as a bridge partner with his or her spouse? We denote this number by bn —the number of ways to separate 2n people into n pairs without married couples. The total number of assignments to choose n distinct pairs is (2n)!/2n n! because we need to divide the total number of permutations by 2n (two choices for each pair and there are n of them), and by n! (the number of permutations of n couples). Let P1 be the homogeneous property (set of permutations) that a particular married couple has been assigned as bridge partners. This could happened in

|P1 | =

(2n − 2)!

2n−1 (n − 1)!

ways. Similarly, the number of ways to assign first k (1 6 k 6 n) married couples as bridge partners is |P1 P2 . . . Pk | =

(2n − 2k)! . 2n−k (n − k)!

464


Application of the sieve formula (8.48) yields bn = P1 P2 . . . Pn =

n

∑ (−1) j

j=0

n (2n − 2 j)! . j 2n− j (n − j)!

Also, using Eq. (8.46), we find the number of assignments to have exactly k (1 6 k 6 n) paired married couples: P1 P2 . . . Pk Pk+1 . . . Pn = ∑ (−1) j−k n (2n − 2 j)! . j 2n− j (n − j)! j>k

Exercise 8.55 [2] Show that the bivariate generating function for Dn (k), the number of nn 1 ex(y−1) . permutations that leave k elements unmoved, is D(x, y) = ∑n>0 xn! ∑nk=0 yk Dn (k) = 1−x Exercise 8.56 [2] (Variegated strings) Consider strings created from the 2m elements in two identical sets [1..m]. A string is called variegated if no pair of identical elements appear in adjacent positions. Thus, 123123 is variegated, but 123321 is not. (a) Use the PIE to calculate the fraction of strings which are variegated. A possible choice of properties: Pi is satisfied by strings that has the pair i i in adjacent positions. (b) Generalize the result of part (a): what is the fraction of strings in which k pairs appear in adjacent positions?

8.4.1 The PIE via the Symbolic Method It is time to take advantage of Eq. (8.41), relating the exact and at-least counts using their (ordinary) enumerators. It usually saves us a step on the way to find exact counts. Once the atleast enumerator of the easier generating function N(z) is obtained, we do not need to extract coefficients from it and later go through the summation (8.42), which could be challenging. Instead we extract directly coefficients from the enumerator we want, E(z) = N(z − 1), to obtain the exact counts. Much of the following was influenced by Philippe Flajolet and [46]. To obtain N(z), we use the atom classes, denoted by T or Z, which can be labeled or not, and the operations of sum, product, and Kleene closure; sometimes substitution and multiset are also applied. Properties will often be marked with the class M. Then we construct a combinatorial structure involving at once the set S, with the variables z and t, and the set P, with the variable v to mark some of the properties. The best way to explain the concept is to go by examples. We begin with problems that can be equally done by either symbolic method or without it, and then provide examples where the enumerator approach will give significant advantage. Example 8.57 (Full occupancy placements) Let us return to Example 8.34 and ask about the number of placements of r items in d bins that leave no bin empty. We also look at the more general case, and count placements that leave a specified, but possibly positive, number of bins empty. A closer look reveals that the


465

question we asked bears two interpretations, and we shall deal with them separately. Note: The two interpretations were in fact already encountered, in §8.2, where we saw two distinct distributions for labeled and unlabeled items into labeled bins (in both cases). (1) The first alternative talks about maps from [1..r] to [1..d]; a mapping that leaves no bin empty is a surjection8 from [1..r] to [1..d]; we want to count them. We pick for Pi the property of leaving bin i empty. As it is shown in Example 8.72, there are dk (d − k)r maps that leave (at least) k bins empty, hence this is nk ; and the number leaving no bin empty, e0 , k d (d − k)r . You were asked to show (−1) is then given by the sieve formula to be e0 = ∑k k in Exercise 5.145 on page 263 that its value is d! dr . Note that the above formula for e0 is true for any values of r and d. We know that if r < d the number of surjections is zero and e0 = 0.

To tackle this question via enumerators, we create a more elaborate structure, in which we not only distribute items (marked by z) into bins, we also mark (or place marks into) a subset of the empty bins. This is rather backwards logic: once we mark a bin as empty, it is no longer available for ball-placement. The distribution of balls constrained in such a way may still leave some other bins empty, and hence the corresponding enumerator is that of the at-least counts. It only remains to write this enumerator. Since we are counting mappings, the balls are assumed labeled. The choice made at each bin, “mark or allocate any number of balls” is expressed by using the sum rule, with the classes mark M and the labeled closure of the class ball (or, the generic term, atom), T, which we associate with the variable9 z. For a single bin we have M + T ∗ , and for d ordered bins the enumerator is N(v, z) = (v + ez )d . If we have an arbitrary number of bins, a third marking variable would be needed. In our case, we only need two variables. Now E(v, z) = N(v − 1, z) = (v + ez − 1)d , hence k r z d r d ek = r![v z ](v + e − 1) = r![z ] (ez − 1)d−k k z d−k r d! d r (e − 1) = , (8.59) = (d − k)! × r![z ] (d − k)! k! d − k k where in the last step the EGF of the Stirling numbers of the second kind, Eq. (L.51), is used. For k = 0, we recover the result obtained previously. The general case is discussed later, in Example 8.80. (2) The second interpretation is asking us about the number of possible configurations of the mappings (the balls are no longer labeled), or using the terminology of §7.3, the number of weak compositions of r in d strictly positive components, or the number of occupancy vectors that leave no bin empty. And again, we shall approach it using both methods, and consider a general problem of a specified, possibly non-zero number k empty bins. Using the same obvious set of properties as in part (1), we find the n j in Eq. (7.14), which 8A

surjection (also called onto mapping) is a function where each point in the range is the image of at least one point in the domain. 9 Normally, we would say here that we mark the atom with z, but we wanted to avoid using this word in two different senses.


466

assures that there are r+d−r j−1 configurations of placing r balls into d − j bins, and we have yet to account for the possible choices of such j taboo bins. Hence d r+d−k−1 r+d−k−1 r−1 j d nk = =⇒ e0 = ∑(−1) = , (8.60) k r k r d −1 k where we have availed ourselves of Eqs. (A.44) and (A.7). Note that this is the result we obtained when discussing bounded-component compositions in Eq. (7.23) on page 372. An r−1 entirely similar calculation, using the n j in Eq. (8.42), provides ek = dk d−k−1 .

Since we have already generated composition counts via enumerators and the closure operation in Sections 7.2 and 7.3, we only need to add the set M, which marks some distinguished (meaning, not necessarily all) empty bins, and the closure of the set that contains one atom, T. Taking their sum, (M + T ∗ ), accounts for a single bin. It again shows explicitly that in each bin we make a choice: mark it, or allow the allocation to use it. Since we have d ordered (=labeled) bins, we need to raise it to the d power. Moving to enumerators, we use v to mark bins that M selects, and with T ∗ having the enumerator 1/(1 − z), we find that the enumerator 1 d . Hence of the at-least counts, reusing the notation of Chapter 7, is ϕN (v, z) = v + 1−z d−k d 1 d r+d −k−1 = , nk = [v z ]ϕN (v, z) = [z ] k r 1−z k k r

r

just as we found in Eq. (8.60), and the same result for the exact counts could follow, naturally. But the point of the method is that if we only want the exact counts, which is the normal need, we do not even need to find the nk since Eq. (8.41) tells us that the enumerator of the exact 1 d z d . Then the appearance of = v + 1−z counts is ϕN (v − 1, z), which equals v − 1 + 1−z positive components is manifest and d d−k z z r d r−d+k d k r = [z ] = [z ] (1 − z)k−d . ek = [v z ] v + k k 1−z 1−z r−1 Hence ek = dk d−k−1 , as expected.

Example 8.58 (Separated samples revisited) In how many ways can we select k numbers out of [1..n] without any pair of adjacent values? With exactly10 j adjacencies? The first part of the question is exactly the problem of separated samples, posed in Example 7.42 on page 375, and the second part was the topic of Exercise 7.44; in both a nice trick helped us out. The problem seems to be natural for the PIE, but it is not simple. Try to write down a value for the easier at-least counts and you will see the difficulty: each adjacency “uses” two items, but two “adjacent adjacencies” use only three. Going via the symbolic method makes it easier to see a way out. Let us first consider selection without any adjacency constraints. We view it as a closure operation over the sum of an element of each of the classes atom T and select S, giving us 10 Both

terms adjacency and succession are used to describe such a pair of entries in a sequence.


467

(T + S)∗ . The sum shows it explicitly that every element may be either skipped or selected. This is formalized with the enumerators t and st (since a selection consumes an atom), leading to the closure enumerator 1/(1 − (t + st)). Taking the indicated coefficient, we find n 1 k n n k = [s ](1 + s) = , [t s ] 1 − t(1 + s) k as expected. Now we add adjacencies to the mix, and mark them with a variable v. Because the number of selections (and atoms) consumed by a cluster of j adjacencies is j + 1 of each, not proportional to j, we decompose our structure to a closure over a sum of three objects: an atom, a selected atom, and a cluster of adjacent atoms, which are selected. Such an adjacency cluster, A∗>1 , has the enumerator ∑ j>1 v j (ts) j+1 = v(ts)2 /(1 − vts), where the three variables mark the various components we need to keep track of. Notice that the sum begins at j = 1: it is meaningless to use this enumerator for zero adjacencies; that is, the role of the bare term S, ⋆ a selection that is not part of an adjacency. Now we get our enumerators of T + S + A⋆>1 : N(t, v, s) =

1

1 − t − st −

v(ts)2 1−vts

=⇒

E(t, v, s) =

1 − st(v − 1) . 1 − t − tsv − t 2 s(1 − v)

Getting the coefficients out of this needs some care, and is then quite mechanical: 1 1 + ts(1 − v) def D = v + t(1 − v) = t + v(1 − t); D , 1−t 1 − ts 1−t k k−1 Dk−1 Dk 1 k n j t D k n j + t (1 − v) = [t v ] t = [t v ] 1−t (1 − t)k (1 − t)k−1 (1 − t)k+1 k − 1 k− j−1 1 (t + v(1 − t))k−1 = [t n−k ] t (1 − t) j = [t n−k v j ] k+1 k+1 (1 − t) (1 − t) j k − 1 n−2k+ j+1 k−1 n−k+1 = [t ](1 − t) j−k−1 = . j j k− j

[t n v j sk ]E(t, v, s) = [t n v j sk ]

For n = 7, k = 4, j = 2 we get the exact count 18, as we should. The number of samples , as in the solution to Exercise 7.43, page 377. without any adjacencies is n−k+1 k

One could change the question and ask not about the number of adjacencies but about the number of continuous blocks of adjacencies. Thus making 4 selections out of [1..7], we find in 2,3,6,7 two adajcencies in two distinct blocks, while the selection 2,3,4,7 shows two adjacencies in a single block. It is a question we shall return to in §8.6, as some reflection reveals that the new question is of a different character. Exercise 8.59 [2] Show that the number of ways k objects, from n arranged on a of selecting n−k n . circle, with no two consecutive is bn,k = n−k k Note: The problem is a particular case of Exercise 7.47, page 378, but you are asked to use results of the previous example.


468

Exercise 8.60 [1] Suppose that n people have lunch, and later supper together at a counter (a straight table). For a given seating arrangement at lunch, how many ways are there to seat these persons for supper such that no one has the same neighbor at both meals? Exercise 8.61 [1] The next week, the n people of Exercise 8.60 have lunch and later supper together at a round table. For a given seating arrangement at lunch, how many ways are there to seat these persons for supper such that no one has the same neighbor at both meals?

8.5 Probabilistic Inclusion – Exclusion Principle The following two theorems recast the PIE in a general probabilistic framework and tie it to the combinatorial context. The last one is just reformulation of the sieve formula (8.43). Recall that the intersection of two sets (events), A ∩ B, is denoted simply by AB.

Theorem 8.62 Consider a random variable N that may take integer values 0, 1, 2, . . . , d with probabilitiesh p0 , ip1 , p2 , . . . , pd , so pk = Pr[N = k] for k = 0, 1, 2, . . . , d. Define the expecta tion sk = E Nk = ∑ j>k kj p j , also known as kth binomial moment of the random variable N. If N is the number of events among ω1 , . . . , ωd , which occur (counting multiplicity), then sk =

∑

i1 k] are related via the following equations: j j−1 sk = ∑ pj = ∑ p> j , k = 0, 1, 2, . . . ; (8.62) j>k k j>k k − 1 j−k j pk = ∑ (−1) s j, k = 0, 1, 2, . . . ; (8.63) k j>k j p>k = ∑ (−1) j−k s j+1 , k = 0, 1, 2, . . . . (8.64) k j>k Theorem 8.63 Let ω1 , ω2 , . . . , ωd be any d events in a sample space. Then # " Pr

d [

k=1

d

ωk =

∑ Pr[ωk ] − ∑ Pr[ωi ω j ] + ∑ i< j

k=1

∑ i< jk /|Ω|, sk = nk /|Ω| (k = 0, 1, . . . , d), and Pr [ωi1 ∩ ωi2 ∩ . . . ∩ ωik ] ≡ Pr [ωi1 ωi2 . . . ωik ] = |ωi1 ωi2 . . . ωik | /|Ω| (k = 1, 2, . . . , d), the formulas in both theorems follow from Eqs. (8.39), (8.42), (8.45), (8.46), and (8.47). Example 8.64 Suppose that we randomly distribute n distinct balls among four ordered cells. What is the probability of at least one empty cell? Solution. The sample space consists of the 4n possible descriptions of assignments of n balls, each sample point has the probability 4−n . We define ωk to be the event that cell k is empty, k = 1, 2, 3, 4.

We need to find Pr[ω1 ∪ ω2 ∪ ω3 ∪ ω4 ]; for this purpose we use Eq. (8.65). Some preliminary computations show that n 3 , k = 1, 2, 3, 4; Pr[ωk ] = 4 n 2 , i < j; Pr[ωi ∩ ω j ] = 4 n 1 , i < j < k; Pr[ωi ∩ ω j ∩ ωk ] = 4 Pr[ω1 ∩ ω2 ∩ ω3 ∩ ω4 ] = 0. Therefore n 4 3 , s1 = 4 1

n 4 2 s2 = , 2 4

n 4 1 s3 = , 3 4

s4 = 0,

and from Eq. (8.65), it follows that Pr[ω1 ∪ ω2 ∪ ω3 ∪ ω4 ] =

4 · 3n − 6 · 2n + 4 . 4n

Exercise 8.65 [2] Suppose that n true dice are rolled. What is the probability that each of the six different numbers will appear at least once? Exercise 8.66 [2] (Le problème des rencontres) In this exercise, we discuss an application of PIE to a generalization of classical problem, known as “le problem des rencontres” (literally, the question on encounters, and sometimes translated as the matching problem), which was first proposed in 1708 by Pierre Remond de Montmort (1678 – 1719). It asks about probability to get a derangement when you choose an n-permutation at random. Originally, this problem was formulated as follows. Suppose that n people checked their coats, with an absentminded check-room attendant who returned


470

coats at random. What is the probability that nobody gets his or her own coat? The problem has been recast in a variety of amusing guises including the random switching of n letters in n envelopes by a secretary apparently having a bad day, and even in terms of a wager on the ordering in which the horses finish in a given race. We treated this problem before, using recurrences—see Definition 5.131 on page 258 and Examples 2.122 and 5.132. However, it is probably the most common combinatorial problem cited as a classical application of the PIE. It is surprisingly simple, compared with the solutions in §5.8.2. Now we go to its generalization: an n-permutation is chosen at random (with uniform distribution). What is the probability that the permutation you pick (a) Is a derangement? (b) Has one fixed point? (c) Has k fixed points? Note: Using the PIE here via the symbolic method is not obviously possible since the combination of fixed points and segments of permutations does not correspond to any admissible operation. Exercise 8.67 [2] A man addresses n envelopes and writes n checks in payments of n bills, so each envelope gets a bill and a check. (a) If the n bills and the n checks are placed at random in the n envelopes, one of each in each envelope, what is the probability that no envelope contains either correct check or correct bill? (b) What is the probability that each check and each bill will be in a wrong envelope? Exercise 8.68 [2+] Cards are exposed one at a time, without being replaced, from an ordinary deck of 52 cards. The draw ceases as soon as four cards of the same kind are exposed. Find the probability that the k-th card exposed finishes the draw. Exercise 8.69 [3] Consider the lottery 6/49 (see Example 3.132 on page 127) and let X j ( j = 1, 2, . . . 5) be the individual spacing random variable X j (t) = t j+1 − t j , 1 6 j 6 5, on the set of all lottery tickets t = ht1 ,t2 , . . . ,t6 i. Let Xmax = max{X1 , X2 , X3 , X4 , X5 } be the maximum spacing. Show that Pr[Xmax > k] =

49 6

−1

5

∑ (−1) j−1

j=1

5 49 − j(k − 1) , j 6

1 6 k 6 44, j(k − 1) 6 49,

and derive the expected value, variance, and the mode for the random variable Xmax .

Exercise 8.70 [3] Suppose that in a randomly chosen permutation of [1..n], some numbers are eliminated with probability q, 0 < q < 1. Find the probability that the remaining set has k fixed points, for any 0 6 k 6 n. Note: In the permutation 1243, if the 1 is selected for deletion, 2 is the only fixed point. Example 8.71 Suppose that m integers are chosen at random (with replacement) from the set [1..n]. We want to find the probability, Pk (n, m), that the sum of these numbers is k. This problem was actually solved in §7.3.3, but we want to give a combinatorial meaning to Eq. (7.19) based on the PIE. Let S be the set of all weak compositions a1 + a2 + · · · + am = k,

8.5. PROBABILISTIC INCLUSION – EXCLUSION PRINCIPLE

471

ai > 1, i = 1, 2, . . . , m. Denote by Pj ( j = 1, 2, . . . , m) the subset of S for which a j > n. Then from Eq. (7.22), it follows that k − 1 − rn |Pi1 Pi2 . . . Pir | = provided rn + m 6 k. m−1 Application of Eq. (8.66) yields k − 1 − rn 1 r m . Pk (n, m) = Pr P1 P2 . . . Pm = m ∑ (−1) r n r>0 m−1 Example 8.72 A question one asks when placing labeled balls into distinguishable bins concerns the number of unoccupied bins. This is formally identical with the problem of the coupon-collector, which was introduced in Example 4.46 (page 150). In §8.2.2 we saw one approach, which led to Eq. (8.30). However, the question is a natural one for the PIE. Consider a sequence of independent trials in which n distinct balls are distributed randomly and independently into m ordered bins so that each arrangement has probability m−n . Let Ak represent the event that the preassigned bin k is still unoccupied after n trials, 1 6 k 6 m. Its complement, Ak , means that the bin #k contains a ball. Then 1 n . Pr[Ak ] = 1 − m The probability, Pr[∪16k6m Ak ], that at least one binis not occupied can be estimated using n Boole’s inequality (H.4): Pr[∪16k6m Ak ] 6 m 1 − m1 . However, this is only estimation, but not exact value that can be determined with the PIE: m k n (8.68) Pr [∪16k6m Ak ] = ∑ (−1)k−1 1− m k 16k6m because the probability that all k preassigned bins are unoccupied is Pr [A1 ∩ A2 ∩ · · · ∩ Ak ] = n 1 − mk , 1 6 k 6 m. Then the probability that all bins are occupied is k n k m . 1− Pr ∩16k6m Ak = ∑ (−1) m k 06k6m Denote by X the number of unoccupied bins when n balls are randomly distributed among m bins. This random variable has the following probability distribution: m n m! n def k m (m − k) = n , (8.69) pn,m = Pr[X = 0] = ∑ (−1) mn m m k k=0

where

n m

m (m − k)! n m m−k n pn,m−k = , Pr[X = k] = k m mn m−k k

(8.70)

is the Stirling number of the second kind (see Appendix C) and k = 0, 1, . . . , m − 1.

472


P ROOF : When i balls are distributed among j bins, let Ni, j denote the number of allocations which result in no empty bins; then pi, j = Ni, j / ji . This is the scenario of the second part of Example 8.34 and Exercise 8.35: according to it nk = kj ( j − k)i , and the number of allocations that satisfy no property (= avoid no bin) is given by the sieve formula. With n balls and m bins, we then obtain Eq. (8.69), where we divided by mn , the total number n is the probability that a of allocations. To show Eq. (8.70), we observe first that m−k m specified set of m − k bins receives all n balls, leaving the other k bins empty, and there are m k such sets of m − k bins. Finally, we need to multiply by the probability pn,m−k , of the n balls leaving no bin in this limited set of m − k bins empty. Next we show that the expected value of X is

(m − 1)n . (8.71) mn−1 P ROOF : The PMF we obtained for X is quite unwieldy, but for the first moment, the indicatorrandom variable approach is quite handy. For each bin i, we define the indicator variable Xi as follows: Xi is 1 if the i-th bin is empty; it is 0 otherwise. Then X = X1 + X2 + · · · + Xm and E[X ] = mE[X1 ] since all the Xi are identically distributed (even though not independent!). Since E[X1 ] = Pr[X1 = 1] = (m − 1)n /mn , the result follows. E[X ] =

Example 8.73 (Continuation of Example 8.72) We can now ask other questions about full occupancy. For example, how many balls need to be distributed before all m bins are occupied? Let Z denote the random variable that counts the number of balls required to achieve a full occupancy. Obviously, the smallest possible value of Z is m, the number of bins. The following observation makes the direct connection between the random variables X and Z. Saying Z 6 n is the same as saying that by the time the first n balls have been distributed, all m bins are occupied. Therefore Pr[Z 6 n] = pn,m , using the probability defined in Eq. (8.69), and the probability mass function of Z is then Pr[Z = n] = Pr[Z 6 n] − Pr[Z 6 n − 1] = pn,m − pn−1,m .

(8.72)

But we can do more. Let Tk , 1 6 k 6 m, be the number of trials required to occupy exactly k distinct bins. The event {Tm > n} is just the event ∪16k6m Ak that at least one bin remains empty after n trials. By setting T0 = 0, we get 0 = T0 < 1 = T1 < T2 < · · · < Tm . The differences Tk − Tk−1 count the number of additional trials necessary to occupy some previously empty bins. These differences are independent geometric random variables with means E[Tk − Tk−1 ] = m/(m − k + 1) and variances V [Tk − Tk−1 ] = m(k − 1)/(m − k + 1)2 . Therefore, Pr[Tm > n] = Pr [∪16k6m Ak ]. Since Pr[Tm = n] = Pr[Tm > n − 1] − Pr[Tm > n], Eq. (8.68) produces the waiting time probability mass function k n−1 k−1 m − 1 . Pr[Tm = n] = ∑ (−1) 1− m k−1 16k6m−1 This allows us to find the expected value E[Tk ] = m ∑16i6k (m − i + 1)−1 and the variance V [Tk ] = m ∑16i6k (i − 1)(m − i + 1)−2 .

8.5. PROBABILISTIC INCLUSION – EXCLUSION PRINCIPLE Exercise 8.74 [1+] Show that the expected value of Z is 1 1 1 = m Hm . E[Z] = m 1 + + + · · · + 2 3 m

473

(8.73)

Example 8.75 As an application of Eq. (8.73), consider a game when a fair die is rolled repeatedly until each of the six faces appears at least once. The expected number of throws required is 6(1 + 1/2 + · · · + 1/6) = 6 H6 = 14.7 So, on average, it takes almost 15 tosses to obtain all six faces. Exercise 8.76 [2+] (M. Dwass) Let X be the number of unoccupied cells when n distinct balls are randomly distributed among m ordered cells. Show that the PGF of X is m m m−i n (z − 1)i . ∑ i m i=0 The next exercise shows that it is possible to compute the mean value of a random variable without ever knowing its probability distribution. Exercise 8.77 [2+] (Birthday problem) Let Xn denote the number of different birthdays among n people that use a calendar with m days per “year.” Apply the PIE to find its expected value m (m − 1)n 1 n 1 k+1 n = m− = m 1− 1− . E[Xn ] = ∑ (−1) mn m k mk−1 k=1 Example 8.78 Let a box contain n different balls. We perform a sequence of d experiments. An experiment consists of drawing at random m balls, recording their identities and returning them (selection with replacement). What is the probability that every ball is picked at least once in the d experiments?

Solution. Denote by A the successful event: every ball is recorded at least once during the d experiments. The description implies that every ball, in each drawing, is equally likely to be selected, and that the experiments’ outcomes are independent and equally likely as well. Hence, the probability of the event A is the number of experiment outcome sequences in the event, |A|, divided by the number of such sequences in the sample space, |S|. The number of various possible outcomes in one drawing of size m from n balls is mn . From the Fundamental Principle of Counting 2.3, page 43, it follows that the number of sequences of experiment outcomes in the sample space is d n . |S| = m To calculate |A|, we introduce the set of properties Pi , i = 1, 2, . . . , n. We say that an experiment sequence σ satisfies the property Pi if it does not record ball number i, that is, it misses


474

that ball in each of the d experiments. In these terms, e0 , the number of elements that satisfy no property, is the number of sequences we need. The interpretation in this context of n(α ) (which was defined on page 448) is the number of experiment sequences that manage to miss the balls listed in α , and nk = ∑α : |α |=k n(α ) collects this counts on all nk lists of k balls out of the n possible ones. Clearly n(α ) does not depend on the specific list: the same number of sequences can be designed to miss any set of size k. Hence we compute nk with the same expression used for |S|, but with n − k balls instead. Therefore n n n−k d n−k d nk = =⇒ e0 = ∑ (−1)k k k m m k>0 and Pr[A] =

∑k>0

n k

n−k d (−1)k m . n d m

This expression can be simplified slightly by using the formula obtain the probability of drawing every ball 1−d n Pr[A] = m

k

∑ (−1) k>0

n n−k k m

=

n − m n − k d−1 . k m

n n−m m k

to

A bonus we get when we work directly with enumerators is that we can find moments of the quantities of interest with, usually, less labor than from the counting functions themselves. Corollary 8.79 Let S be a finite set for which we have a list of (finite) properties and let E(z) be an “exact” enumerator defined by Eq. (8.40). Then the mean number of properties satisfied by an element of S, and its variance are given by

µp =

E ′ (1) ; E(1)

Vp =

E ′′ (1) + E ′ (1) E ′′ (1) − µ p2 = + µ p − µ p2 . E(1) E(1)

(8.74)

P ROOF : We interpret ek /|S| as pk , the probability that an element of S has exactly k properties, and then write, using E(1) = ∑k ek = |S|,

µ p = ∑ k pk = k

k ek E ′ (1) = , E(1) E(1)

and the second moment follows in the same way. Often it is not the mean number of properties we are after, but other variables defined over a set of elements that satisfy a given number of properties. An example might be the mean number of balls in a bin, for the distributions of balls into bins which leave no bin empty, or the expected largest bin occupancy in such a distribution.

8.5. PROBABILISTIC INCLUSION – EXCLUSION PRINCIPLE

475

Example 8.80 Let us look at the probabilistic interpretation of occupancy distribution of r distinct balls into d ordered cells. We want to answer the question “What is the expected number of bins left empty if we use a “random” placement, where we pick one with equal def likelihood from among the d r possible placements?” We can define pk (r) = ek /d r as the probability that an allocation leaves exactly k bins empty, and summation does the rest. But we already have the enumerator, which incorporates exactly this summation! Since the desired expected value is ∑k k pk (r), and we know already from Eq. (7.5) that for any sequence {an }, we get the value of ∑n n an by computing the derivative of the OGF at v = 1. It is v, since this is the variable marking the empty bins. We still need to take account of our looking at placements of r balls since the enumerator E(v, z) = (v + ez − 1)d includes def all numbers of balls (which was found in Example 8.57). So let us use the notation B(r) (v) = r![zr ]E(v, z), and then we can express the expected number of bins left empty as (r) Bv (1) ∂ B(r) (v)/∂ v ∑k k ek ≡ (8.75) = (r) µr = ∑ k pk (r) = . dr B (1) B(r) (v) k v=1

We know the denominator is just d r , but we wrote it like this to have a more general form for further use. An important computational note is that although we wrote this equation as if we first operate on E(v, z) with r![zr ], and later differentiate with respect to v, this needs not be done in this way: since the two operations commute we can do them in any order. Often the reverse order leads to simpler expressions. Here we find first that E(1, z) = ezd , and r![zr ]ezd = r!(d r /r!), as expected; then Ev (v, z) = d (v + ez − 1)d−1

=⇒

Ev (1, z) = dez(d−1)

=⇒

(r)

Bv (1) = d r!

(d − 1)r . r!

Therefore, the expected number of bins left empty is d−1 r . µr = d d For the variance we need Vr = ∑k k2 pr (r) − µr2 . The second derivative of B(r)(v) with respect to v gives us the second factorial moment, the sum ∑k k(k − 1)ek . Hence we correct it by adding the sum µr = ∑k k ek , and finally get the variance to be (r)

Vr =

(d − 1)(d − 2)r + (d − 1)r (d − 1)2r Bvv (1) 2 − 2r−2 , − + µ µ = r r d r−1 d B(r)(1)

(8.76)

which coincides with the result obtained in Example 4.23, page 143. Exercise 8.81 [3] Consider a game when n students from some group put their names on slips of paper, which are deposited in a box and shuffled thoroughly. Each member then draws a slip of paper from the box and is asked to give a small gift to the person whose name is on the slip. Should someone draw their own name, they replace the slip and pick again, so only


476

derangements of names are considered. It is possible that two students would have slips with each other name (we call this case a two-cycle, see definition in §5.8.3, page 262). What is the fraction of all derangements of n names contained no two-cycle? Exercise 8.82 [3] Redo the previous exercise, and write directly the enumerator of those constrained permutations, using the representation of permutations as the set of cycles. We now generalize this problem in the following Example 8.83 (Continuation of Exercise 8.81; adapted from G. Sanchis [119]). Now we consider the following generalization of Montmort’s matching problem considered in Exercise 8.66. What is the probability, which we denote Pm (n), that an n-permutation contains at least m-swap? We reformulate the problem in the form of banquet’s attendance by n men who check their hats. When they leave, everybody takes a hat at random. What is the probability, for any m from 1 to n, that m is the size of a smallest subset on n men who exchange hats among themselves? So Pm (n) is the probability that among n men, m is the size of the smallest subset of men that swap hats. In mathematical language, Pm (n) is the probability that a permutation of [1..n] contains the smallest cycle of length m. Let Ei be the event that the i-th man gets his own hat back. Obviously, P1 (n) is the probability of at least one coincidence. From Eq. (8.65), it follows Pr[∪i Ei ] =

∑ Pr[Ei] − ∑ Pr[EiE j ] + ∑ i

i< j

i< j0

The rate of the convergence is determined by the second-largest (in absolute value) eigenvalue of the matrix Q, which is guaranteed to be less than 1, but could be close. In Eq. (9.19), the matrix P˜ deals with the process after it has reached an ergodic set. The matrix S concerns the transition from transient to ergodic states. If the transient classes consist of single states (or absorbing states), then the transition matrix becomes (when after possible reordering all transient states precede the recurrent states) # " Q S . (9.21) P= 0 I

CHAPTER 9. COMBINATORICS OF STRINGS

530

Theorem 9.95 In any finite Markov chain, there are numbers c > 0 and 0 < b < 1 such that (m) pi, j 6 c bm , for any transient states i, j. Moreover, Qm → 0, as m → ∞, exponentially. P ROOF : It is sufficient to prove the statement for the case when transient states i and j belong to the same communication class. Suppose that the process starts in a transient state. Since there are only finite number of states, it is possible to reach an ergodic state in no more than r steps (r is less than the number of states). Hence there is a positive number ω such that the probability of entering an ergodic state in at most r steps is at least ω , from any transient state. Therefore, the probability of not being absorbed within r transitions is at most 1 − ω , which is less than 1. The probability of not reaching an ergodic state in mr steps is less than or equal to (1 − ω )m as m increases. Understanding the long-term behavior of a Markov chain boils down to understanding the behavior of Pm for large m values. The crucial role in this issue is played by an invariant probability distribution for the corresponding transition matrix.

Definition 9.96 A stationary distribution (also called an equilibrium distribution) of a Markov chain is a probability distribution ~π = hπ0 , π1 , . . . , πn i such that

∑ πk = 1, k

~π = ~π P

or

~π (I − P) = 0,

where I is the identity matrix (which has all entries zeroes except 1’s on the diagonal).

(9.22) ⊳

The above equation shows that an invariant probability vector, ~π = hπ0 , π1 , . . . , πn i, is a left eigenvalue of P corresponding the eigenvalue 1. One way to compute the stationary distribution of a finite Markov chain is to solve the homogeneous system of n + 1 linear equations (9.22). Since there are n + 1 unknowns, the system ~π = ~π P has infinite many solutions, but the additional constraint ∑nk=0 πk = 1 leads to the unique solution. If a chain ever reaches an equilibrium distribution, then it maintains that distribution for all future time. In particular, a Markov chain with an absorbing state will be in its stationary distribution once it reaches the absorbing state. If k is an absorbing state, and the process starts in state i, the probability of ever going to state k is called the probability of absorption into state k, given that the system started in state i. It is possible to give a probabilistic interpretation of the elements of the fundamental matrix. Define Ni j to be the number of times the process passes through the transient state j starting at another transient state i till absorption, i, j = 1, 2, . . . , s. If i = j, the initial sojourn is already counted as one visit. The expected number of steps until the chain enters a recurrent class, assuming that it starts at state i, is the sum ∑ j ni, j over all transient states j. Theorem 9.97 The entries of the fundamental matrix are expectations of Ni j : def ni, j = E[Ni, j ] = (I − Q)−1 i, j , 1 6 i, j 6 s, where s is the number of transient states. P ROOF : We show two proofs, which while making similar statements do it in usefully different manners. The first is straightforward. Since Qk i. j is the probability that a path from state i visits state j in step k (using Q0 = I), we view Ni, j as a sum of indicator random

9.5. ALGORITHMS AND MARKOV CHAINS

531

variables, for each possible step, and then ni, j =

∑ 1 · Qk i. j = (I − Q)−1i, j ,

(9.23)

k>0

where we add up the contributions of all steps. The second proof furnishes an example of a useful computational method for Markov chains: randomization on the first step from state i to, say, a: s

ni, j = δi, j + ∑ Qi,a na. j .

(9.24)

a=1

Theorem 9.98 Suppose the transition matrix for a finite Markov chain is in the canonical form (9.19). If bi, j is the probability of absorption into state j, that is, the probability that the process starting in transient state i ends up in absorbing state j, then def

B = [bi, j ] = N S = (I − Q)−1 S.

(9.25)

P ROOF : Starting in transient state i, the process may be absorbed in j in one or more steps. The probability of capture on a single step is pi, j . If this does not happen, the process may move either to another absorbing state (so it will be impossible to reach state j), or to a transient state k. In the latter case, there is probability bk, j of being captured in the right state. Hence we have bi, j = pi, j + ∑ pi,k bk, j , k

which can be written in matrix form as B = S + Q B. Solving for B, we get the required formula. The number of transitions made by the process in going from state i to state j for the first time is usually referred to as the first passage time. When i = j, this first passage time is just the number of transitions until the process returns to the initial state. Once it happens, the system recommences the process from scratch, forgetting that it was in the same state before. (m) We denote by ri, j the probability that, starting at state i, the first transition to state j occurs at time m: (m) ri, j = Pr[ Xm = j and for 1 6 k < m, Xk 6= j | X0 = i ]. (9.26) (m)

Let ri, j = ∑m>1 ri, j be the probability that, starting from state i, the system ever reaches state j. Then we can claim that state i is recurrent if

(m)

∑ ri,i

= 1, and it is transient if

m>0

(m)

∑ ri,i

< 1.

m>0

(m)

Whenever ∑m>0 ri,i = 1, the expected time to first reach state j from state i, denoted by (m)

µi, j = ∑m>1 m ri,i , uniquely satisfies the equation µi, j = 1 + ∑ pi,k µk, j . k6= j

(9.27)


532

So µi,i is the expected time to return to state i. Since we consider only finite Markov chains (with finite number of states), all expected values µi,i are finite for all recurrent states, but not for transient ones. A finite Markov chain has at least one recurrent state. We summarize the properties of stationary distributions in the following statement [99]. Theorem 9.99 A finite, irreducible, ergodic Markov chain has the following properties: 1. the chain has a unique stationary distribution ~π = hπ0 , π2 , . . . , πn i; (t) 2. for all i and j, the limit limt→∞ pi, j exists and is independent of i; (t)

3. πk = limt→∞ pi,k = 1/µk,k . (n)

(n)

4. State i is recurrent if ∑n>1 pi,i = ∞ and transient if ∑n>1 pi,i < ∞. Property 3 tells that the expected number of steps to return to state k, assuming that the chain starts at k, is given by the reciprocal of the kth component of the stationary distribution. There is a typical way to relate Markov chains to analysis of algorithms [64, 99]. An algorithm starts at some fixed initial state. Then, according to the input, it moves till completion through a finite number of states, which we again denote by vi , i = 0, 1, 2, . . . , n. A state in this context reflects the values of the variables that the algorithm uses to record the processing that it has performed so far. If a sufficiently large number of states is allowed, we could probably represent in this way any algorithm, but this approach is not really meaningful and effective unless the state space is either small or has a simple structure. Such operation is represented formally as traversal of a directed graph: nodes represent states of the algorithm, edges represent possible transitions. The crucial step in the model formulation is the assignment of probabilities to the edges. Usually, there are two sources of probabilities in an algorithm. One is induced by the distribution over the input, as would be typically the case in a sorting or searching algorithm. Another scenario arises when the algorithm itself contains a random element, taking action based on sampling a built-in random event generator. We say that this representation has the Markovian property when the probability that the algorithm goes from a certain state vi into some other state v j does not depend on the history of the algorithm execution before it reaches state vi . It is then equal to the probability assigned to the edge (i, j). There are two different mathematical descriptions of such a chain: a directed, labeled graph with weights, and a stochastic matrix of dimensions equal to the number of nodes, with entries that are positive and sum, in each row, to 1. Definition 9.87 is equivalent to saying that the set of states supports a first-order Markov chain, or provides a Markovian description of the algorithm. If the algorithm had in fact a memory of the last two, or some fixed number of states (say k), we would say it is a kth-order Markov chain. It is relatively easy to translate any such higher-order chain to a first-order one, at the cost of a significant increase in the size of the state space. Suppose that the chain has the only one absorbing state, the last one, vn . Since ni, j is the expected number of visits of the chain in v j till absorption, starting in state vi , then sum-


533

ming over the ith row of the fundamental matrix would provide us with di , defined as the expected time (counted in state transitions) till absorption, from state vi . Let us show this by independent calculation, invoking randomization on the first transition: n

n−1

n−1

di = 1 + ∑ pi, j d j = 1 + ∑ pi, j d j = 1 + ∑ Qi, j d j , j=0

j=0

0 6 i < n.

(9.28)

j=0

We change the sum above to terminate at n − 1 without changing its value, since dn = 0 is the only value which is meaningful. With our convention that v0 is the initial state, the first component, d0 , is the expected run-time of the algorithm. Note that Eq. (9.28) is identical to Eq. (9.11) that we derived in §9.4. Using the fundamental n × n matrix, N, we can express the solution of the system of algebraic equations (9.28) in the vector form: d = (I − Q)−1 e = N e,

(9.29)

where dT = hd0 , d1 , . . . , dn−1 i and eT = h1, 1, . . . , 1i are n-dimensional vectors.

Example 9.100 (Simple random walk on a graph) Consider a Markov chain whose states are the vertices in a finite, simple, connected graph (each edge connects two different nodes and any two vertices are connected by at most one edge). At each integer time, the process chooses a new state randomly from among the states adjacent to the current state. The transition matrix for this chain is given by p(vi , v j ) = 1/d(vi ),

where d(vi ) is the number of vertices adjacent to vi ; if d(vi ) = 0, we set p(vi , vi ) = 1. Such chain is called a simple random walk on the graph. The chain is irreducible if and only if the graph is connected, i.e., if any two nodes can be connected by a path of edges in the graph. In a connected graph with at least two nodes, every vertex is adjacent to at least one other vertex. Therefore, such chain has the period 2 if and only if the graph is bipartite, namely, if and only if the vertices can be partitioned into two disjoint sets V1 , V2 such that all edges of the graph connect one node of V1 and one vertex from V2 . Otherwise it has period 1. Example 9.101 We can translate the scenario of Exercise 6.171, page 340, into an algorithm that performs a sequence of steps by invoking the Bernoulli trials with parameter p = 1/2 until 3 successive successes are achieved. We may even consider a more complex situation than the one provided there: the probabilities of success (s) and failure ( f ) depend on the state (si and fi in state i, where i is the length of the current success run). The two descriptive modes will then produce the following graph and the transition matrix. We assume v0 to be the initial state to which the process returns on each failure. The weights on the arrows are the probabilities of the indicated transitions: si + fi = 1, i = 0, 1, 2. States v0 , v1 , v2 are transient states (and belong to one communication class), and v3 is the absorbing state (another communication class). The fundamental matrix is   −1  1 s0 s0 s1 1 − f0 −s0 0 1     N =  − f1 s1 (1 − f0 )  . 1 −s1  =  f + s f 1 − f0 1 − f0 − s0 f1 − s0 s1 f2 1 1 2 f2 s0 f2 1 − f0 − s0 f1 − f2 0 1


534

f0

s1

v0

 f0 s0 0 0    f1 0 s1 0   P=  f 0 0 s . 2  2 0 0 0 1 

1

f1

s0

s2

v1

v3

v2

f2

The matrix entries, [pi, j ], of P provide the probabilities of one-step transitions, and are the edge weights in the graph. The 3 × 3-matrix Q is extracted from the stochastic matrix P by eleminating the last column and the lowest row. Let S be the 3-vector in the last column of P after deleting the “1.” Multiplication of matrices N = (I − Q)−1 and S (where ST = h0, 0, s2 i) gives the probabilities to reach the absorbing state from each of transient states { v0 , v1 , v2 }:  s0 s1 s2 1 def   B = NS =  s1 s2 (1 − f0 )  . 1 − f0 − s0 f1 − s0 s1 f2 s2 (1 − f0 − s0 f1 ) 

State v3 in the figure for Example 9.101 has a self-loop that is not part of the corresponding algorithm formulation, but it is necessary to make its transition matrix stochastic. Markov chains are processes that do not terminate, and such a self-loop at the absorbing state simulates termination since it keeps the chain in the same state indefinitely. Example 9.102 (Coin tossing) Consider independent tosses of a coin that, on each toss, lands on heads (H) with probability p and on tails (T) with probability q = 1 − p. What are probabilities and expected number of tosses needed to either of the patterns THTH or HHTT to appear?

Solution. First, we draw the diagram for 8 states: v1 = T, v2 = H, v3 = TH, v4 = HH, v5 = THT, v6 = HHT, v7 = THTH, and v8 = HHTT. Note that states vk , k = 1, 2, . . . , 6, are transient, and v7 , v8 are absorbing states. T

T, q

q

p

q

v0

v3

v1 q

v2

p

v5 p

p

p

T,H

v4 H, p

p q

v7 q

v6

v8 T,H

Using the diagram, we build the transition probability 9 × 9-matrix, from which we extract the last two columns without the last two entries—denote this 7 × 2-matrix by S. Similarly, we denote by Q the 7 × 7-matrix in the upper left corner of the probability matrix. The


535

fundamental matrix, N = (I − Q)−1 , and the matrix S are read from this diagram to be



2

 0   0 1 N=  0 2  0  0 

0

1+p−p2 −p4 p2 1+p p2 1−p3 p2 q p2 q p q(1+p) p2 q p

2p 0 2 0 0 0 0

1−p2 q pq 1 pq 1−pq pq 1 pq 1 q 1 p 1 q

1+p2 q q2 1 q2 1+pq q2 1 q2 1+q q2 1 q p q2

1−p2 q p2 1 p 1−pq p 1 p

1 1+p p

1



1+p2 q q 1   q  1+pq  q   1 , q  1+q   q 

1  

1+q q



0  0  0   S = 0  0   p 0

 0  0  0   0 .  0   0 q

The entries of the product NS contain the probabilities to reach one of the absorbing states (v7 or v8 ) from each transient state. Though all these probabilities are at our disposal, we need to determine only two of them—probability to get v7 or v8 starting from v0 . Therefore, the entries in the first upper row of the product NS give us the required probabilities to be Pr[THTH] = (1 − p2 q)/(2p) and Pr[HHTT] = (1 + p2 q)/2. Adding the entries in the first row of the matrix N, we obtain the expected running time to be (1 + pq − 3p4 q − p3 + p6 )/(2p2 q2 ). Example 9.103 We reconsider Example 9.79, page 523. The transition probability matrix, P, can be written in the canonical form (9.20), where Q is the transient portion of the system so that limn→∞ Qn = 0; N = (I − Q)−1 is the fundamental matrix that is used to determine the number of times that you would be in each state before going to one of the absorbing states (KILL or LIVE).



4/6 3/6 3/6 3/6 3/6 4/6 3/6

        P=         0 0

 0 0 1/6 0 0 1/6 0 0  0 0  1/6 1/6 0 1/6 0 0  0 0  1/6 0 1/6 1/6 0 0   1/6 0 0 1/6 0 0 1/6 0   0 0  1/6 0 0 1/6 1/6 0   0  0 0 0 1/6 0 1/6 0  0 1/6  0 1/6 0 1/6 0 0   0 0 0 0 0 0 1 0  0 1 0 0 0 0 0 0

Figure 535: The state transition matrix in canonical form (9.21).


536

Then the matrix (9.25) becomes 

215  216  221 1   B = N∗S = 251 431  214   209 180

  216   215     210     180 ≈     217     222  251

0.4988 0.5012 0.5128 0.5824 0.4965 0.4849 0.4176

0.5012 0.4988 0.4872 0.4176 0.5035 0.5151 0.5824



      .     

In Figure 535, this first state is the starting state, so the probability of rolling LIVE is 0.4988 and the probability of rolling KILL is approximately 0.5012. The state transition matrix for the case LIVE vs DEAD is as follows    4/6 1/6 0 0 1/6 0 0      3/6 1/6 1/6 0 1/6 0 0        3/6 1/6 0 1/6 1/6 0 0       S= Q =  3/6 1/6 0 0 1/6 0 0 ,      3/6 1/6 0 0 1/6 1/6 0        3/6 1/6 0 0 1/6 0 1/6    4/6 1/6 0 0 0 0 0

0 0 0 0 0 0 1/6 0 0 0 0 0 0 1/6



      .     

It is obvious that starting from the first state, there is a slightly higher probability that LIVE will be rolled over DEAD. The expected number of rolls to get LIVE is 649.50 and the expected number of rolls to DEAD is 649.49. Again, the expect number of rolls to get to DEAD is smaller (albeit barely) even though it is more probable that you will roll LIVE. The probabilities of absorption become 

217  218  223 1   −1 B = (I − Q) S = 253 433  216   211 181

  216   215     210     180 ≈     217     222  252

0.5012 0.5035 0.5150 0.5843 0.4988 0.4873 0.4180

0.4988 0.4965 0.4850 0.4157 0.5012 0.5127 0.5820



      .     

To calculate the variance of Ni, j , we define the following probabilities def

wi, j = Pr[Ni, j > 0],

i 6= j,

def

q j = Pr[N j, j = 1],

(9.30)


537

where i and j are transient states. So wi, j is the probability that, starting at state i, the chain visits transient state j before absorption, and q j is the probability that, starting from state j, this state is not revisited before absorption (no return). It is surprising, at first glance, that such probabilities can be expressed in terms of the fundamental matrix, containing only expected values. ni, j 1 , (ii) wi, j = , where ni, j = E[Ni, j ]. Theorem 9.104 (i) qj = n j, j n j, j P ROOF :

(m)

R(m) = [ ri, j ],

For each m > 1, we define an auxiliary sequence of matrices (m)

m = 1, 2, . . ., where ri, j is the probability that, starting at state i, the first transition to state j (1)

(1)

occurs at time m (ri, j = pi, j = pi, j ). For consistency with the definition of Ni, j , we say that (0)

when i = j, it is the second visit we observe. We define ri, j = δi, j when m = 0. Randomizing on m, we obtain the key relation between the matrices R(m) and P: (k)

def

pi, j = Pk i, j =

k

(m)

∑ ri, j

k

Pk−m j, j =

(m) (k−m) p j, j ,

∑ ri, j

k > 1.

(9.31)

m=1

m=1

For each pair (i, j) of transient states, we define the ordinary generating functions def

ai, j (z) =

(m)

∑ zm Pm i, j = ∑ zm pi, j

def

,

bi, j (z) =

.

(9.32)

m>1

m>0

m>0

(m)

∑ zm ri, j

Multiplying Eq. (9.31) by zk and summing over k > 1, we get ai, j (z) − δi, j =

∑ zk k>1

k

(m)

∑ ri, j

Pk−m j, j =

m=1

(m) m

∑ ri, j

z

m>1

∑ zk−m Pk−m j, j , k>m

which can be rewritten as ai, j (z) = δi, j + bi, j (z)a j, j (z).

(9.33)

The definition of wi, j implies that it equals bi, j (1), and an expression for ai, j (1) has already been computed: ai, j (1) =

∑ Pk i, j = ∑ Qk i, j = (I − Q)−1i, j = ni, j ,

k>0

1 6 i, j 6 s;

k>0

where s is the number of transient states. Hence, for i 6= j, we can write ni, j /n j, j , as claimed, and when i = j, the probability of no return is q j = 1 − b j, j (1) = 1 − completing the proof.

n j, j − 1 1 = , n j, j n j, j

wi, j = bi, j (1) =


538

Exercise 9.105 [3] For a finite Markov chain with one absorbing state, let Ni, j be the number of times node j is visited in a path from node i till absorption in the final state n, (a) Compute the probability Pr[Ni, j = k], using Theorem 9.104. def (b) Show that ni, j = E[Ni, j ], as computed in Theorem 9.97, is consistent with the result of part (a), and use it to show that the variance of random variable Ni, j is given by V [Ni, j ] = ni, j (2n j, j − 1) − (ni, j )2 ,

where N = [ ni, j ] = (I − Q)−1 .

(c) In addition, show an explicit expression for the MGF of Ni, j : Mi, j (z) ≡ E ez Ni, j = 1 + ni, j

1 − e−z . 1 − n j, j (1 − e−z )

In the rest of the discussion, we assume that the algorithm has n + 1 possible states, denoted by v0 through vn , with v0 being the initial state, and vn the terminating (or absorbing) one. We can do better and obtain the PGF of the running time of the algorithm. Let Ti be the number of steps from state vi till absorption at vn . We have seen already that E[Ti] = ∑ j ni, j . By our structural assumption (start at v0 , finish at vn ), T0 is the run-time. Randomizing as we did for Eq. (9.24), we get Pr[Ti = k] = ∑ pi, j Pr[T j = k − 1] = ∑ Qi, j Pr[T j = k − 1] 0 6 i < n, j

k > 1.

(9.34)

j

def

Let gi (z) = E[zTi ], i = 0, 1, . . . , n − 1, with Tn = 0. Remembering that Q is a submatrix of dimensions n × n of the transition matrix, we obtain gi (z) =

∑ zk Pr[Ti = k] = z ∑ pi, j ∑ zk−1 Pr[Tj = k − 1], j

k>1

0 6 i < n.

(9.35)

k>1

In other words, gi (z) = z (P g(z))i

0 6 i < n,

(9.36)

where g(z) = hg0 (z), g1 (z), . . . , gn (z)iT is a column vector of all n + 1 functions gi (z). Note that gn (z) = 1 by assumption that Tn = 0. To rewrite Eq. (9.36) in a matrix form, we observe that for i = n the right-hand side expression would equal z (P g(z))n = z since the only nonzero term in the (n + 1)th row of the transition matrix is pn,n = 1. Hence, using the column vector en+1 = h0, 0, . . . , 0, 1iT of length n + 1, we get our matrix equation: g(z) = zP g(z) + (1 − z)en+1

or

gn (z) = z Q gn (z) + z S,

1 6 i < n,

(9.37)

where gn (z) = hg0 (z), g1 (z), . . . , gn−1 (z)iT is n-vector column obtained from g(z) by eliminating the last entry. Direct inversion yields g(z) = (1 − z)(I − zP)−1 en+1

or

gn (z) = z (I − zQ)−1 S.

(9.38)

Unless P has a very special structure this expression is of limited computational utility. However, we really need only the first entry of the n-vector gn (z), which yields the following result.


539

Theorem 9.106 The run-time of the algorithm, expressed as number of state transitions, has the probability generating function g0 (z) = (1 − z)(I − zP)−1 1,n = z (I − zQ)−1 S 1 , (9.39) where index 1 in the right hand-side vector indicates that we need to take the first element from the vector-column. Example 9.107 (Continuation of Example 9.101) To find the generating function, we need   −1  1 zs0 z2 s0 s1 1 − z f0 −zs0 0 1     (I−zQ)−1 =  −z f1 z(1 − z f0 )s1  , 1 −zs1  = z( f1 + z f2 s1 ) 1 − z f0 ∆(z) z f2 z2 f2 s0 1 − z f0 − z2 s0 f1 −z f2 0 1 where ∆(z) = det(I − zQ) = 1 − z f0 − z2 s0 f1 − z3 s0 s1 f2 . Multiplying by zS = zh0, 0, s2 iT and extracting the first entry, we obtain the generating function: g0 (z) =

z3 − z2 f

z3 s0 s1 s2 . 0 − z f1 s0 − f2 s0 s1

Differentiating of g0 (z) and setting z = 1 yields the expected number of transitions to be s0 s1 s2 ( f0 + 2s0 f1 + 3 f2 s0 s1 ) 3s0 s1 s2 + . 1 − f0 − s0 f1 − s0 s1 f2 (1 − f0 − s0 f1 − s0 s1 f2 )2 Exercise 9.108 [2] For the board game considered in Exercise 9.58 on page 513, find expected number of flips and the variance required to finish the game. Since the eigenvalue 1 of a stochastic matrix is simple (of multiplicity one), the resolvent matrix (I − zP)−1 has a simple pole at z = 1; hence the right-hand side function in Eq. (9.39) is analytic at this point, and this tells us that all the moments of T1 exist (= are finite), which implies the finiteness of the moments of all the other T j . Example 9.109 (Reflecting & absorbing random walk) Consider a random walk on the finite integer interval, limited to [0..n]. A corresponding Markov chain has integer states 0, 1, . . . , n. Each time-unit, the process changes its position by one: either move one step to the right with probability p or one step to the left with probability q = 1 − p. The extreme states are special: state 0 is reflecting (the process returns to the point from which it comes), and state n is absorbing. As a result, we have a first-order Markov chain having the tri-diagonal transition probability matrix, with the off-diagonal terms   1, i = 0, j = 1,     p, 1 6 i < n, j = i + 1, (9.40) pi, j =  q, 1 6 i < n, j = i − 1,     1, i = j = n,


540

and zero otherwise. According to Eq. (9.29), starting from the expected time till absorption, −1 e T = h1, 1, . . . , 1i. Even n = (I − Q) , where e the initial position 0, is d0 = ∑n−1 j=0 0, j 1 though this P is quite sparse and simple, the calculations are not trivial. For n = 5, the matrix I − Q is as follows   1 −1 0 0 0   −q 1 −p 0 0     I−Q =   0 −q 1 −p 0  .   0 −q 1 −p 0 0 0 0 −q 1 A similar tri-diagonal matrix was considered in Example 5.41, page 214. We need the sum of entries in the first row of the inverse of I − Q. This is equivalent to determination of d0 , the first element in the n-dimensional vector dT = hd0 , d1 , . . . , dn−1 i, which is the solution to the difference equation of the second order dk = 1 + qdk−1 + pdk+1 ,

k = 1, 2, . . . , n − 2,

d0 = 1 + d1 ,

dn−1 = 1 + qdn−2 .

(9.41)

The boundary conditions on both ends (for k = 0 and k = n − 1) make finding its solution intriguing. Though direct solvers like Gauss elimination and LU decomposition can be used to solve these systems, there are some special schemes available to solve the tri-diagonal systems. One of them exploits the tri-diagonal nature of the coefficient matrix, which is a simplified form of Gauss elimination. To solve the vector equation (I − Q) d = e with tridiagonal matrix, we use so called forward elimination and back substitution (FEBS) method, which is widely used in numerical simulations. In engineering, the FEBS method is associated with Llewellyn H. Thomas from Bell laboratories who used it in 1946. Let us show in detail the FEBS procedure. Substituting d0 = 1 + d1 into the difference equation (9.41) for k = 1, we get d1 = 1 + q(1 + d1 ) + pd2

or

d1 = d2 + β (2),

1 q 1 q + = + β (1), with β (1) = 1. Eliminating d2 from p p p p 1 q the recurrence (9.41) for k = 2, we get d2 = d3 + β (3), where β (3) = + β (2). In p p general, we have where we use the notation β (2) =

dk = dk+1 + β (k + 1),

β (k + 1) =

1 q + β (k), p p

k = 1, 2, . . . , n − 2.

(9.42)

The procedure is continued until dn−2 is eliminated from the last equation: dn−2 = dn−1 + β (n − 1). Using the boundary condition, dn−1 = 1 + q dn−2 , we solve last two equations to obtain 1 1 1 q dn−2 = + β (n − 1). dn−1 = + β (n − 1), p p p p


541

The value of dn−2 we get not from the above formula, but from the relation (which is actually back substitution) dn−2 = dn−1 + β (n − 1). Continuing in such manner, we can find all values dk , k = 0, 1, 2, . . . , n − 1. However, we need only d0 : n−1

n−1

d0 =

1

q

n−2

1

1

∑ β (k) + dn−1 = ∑ β (k) + p + p β (n − 1) = ∑ β (k) + p + p β (n − 1), k=1

k=1

k=1

where β (k) is the solution of the first order recurrence (9.42). It turns out that the ratio between p and q is crucial for the form of the solution. Let initially p = q = 1/2. The recursion for β (k) is then

β (k + 1) = 2 + β (k),

β (1) = 1,

k = 1, 2, . . . , n − 2.

Since its solution is immediate, β (k) = 2(k − 1) + 1, we get n−1

d0 =

∑ (2(k − 1) + 1) + 2 + 2(n − 2) + 1 = n2 .

k=1

When p 6= q, we have 1 2q − β (k) = p−q p−q

k−1 q , p

k = 1, 2, . . . , n − 1.

Then " k−1 # n−2 # 1 1 1 q q 2q 2q 1 + + − − d0 = ∑ p−q p p p p−q p−q p k=1 p − q q n−1 2pq 2q2 n − + . = 2 p − q (p − q) p (p − q)2 n−2

"

So there is asymmetry in the process, and it is not hard to see why. Observations suggest that if q > p, the process hovers around the reflecting state most of the time because the expected time of absorption is exponential in n. On the other hand, if p > q, the main term is linear in n. Exercise 9.110 [2] (Gambler’s ruin problem) Consider the random walk on [0..n] similar to the one in Example 9.109, but having two absorbing states at 0 and n. Let α ( j) = α ( j, n) be the probability that the chain starting at state j eventually ends up absorbed in state n. Find α ( j) and determine limn→∞ α ( j, n). Note: This random walk is a reformulation of the Gambler’s ruin problem when a gambler has probability p of winning one unit and probability q = 1 − p of losing one unit. If the gambler has j units initially, the game is terminated either when the player reaches n > j units, or when he is ruined (out of money).

Exercise 9.111 [2] Derive a Markov chain description of the algorithm Binary Search, presented in §5.4.2 as version RBS, page 224, under the assumption that the searched value is in

542


the array. (a) How would you use it to get the number of comparisons the algorithm performs? (b) Modify your model to handle the possibility of search for an absent element. Hint: Use as state descriptor the size of the interval in which the element is searched. Exercise 9.112 [2] Rolling a fair die, find the mean waiting time for a pattern of n identical 6’s in a row. Exercise 9.113 [3] Example 9.76, page 522, shows that some triplets of a fair coin are more likely than others. When a biased coin is flipped successively, it generates strings of two letters—T’s with probability q and H’s with probability p = 1 − q. Consider all possible pairs of triplets of two letters, T and H, generated by flipping a biased coin and determine the stopping probabilities for each such pair of patterns. Check your answers with Exercise 9.77, page 522. Example 9.114 (Parrondo Paradox) The following example shows that probabilistic situations bear careful analysis. It is possibly even more puzzling at first sight. We define two gambling games, show that each is a losing game if you try it, and then show that a third game, which simply alternates those games (or chooses them with equal probability), ends up being a money-maker for you! The phenomenon was noticed and presented in 1996 by a Spanish physicist, Juan Parrondo, and has seen much interest8 . Now we describe the games. In each of them you either gain a dollar, or lose one, in each round. Game A is simple: flip a biased coin. In probability pa < 1/2 it shows heads and you win a dollar. You lose one in the complementary probability qa = 1 − pa ; clearly qa > pa and this is a losing proposition. Game B uses one of two coins, and the choice depends on your accumulated winnings so far, which we denote by w; it is an integer, the difference between the number of games in which you gained, and those in which you lost. If in the balance you have gained (or lost) a number of dollars evenly divisible by 3, you flip coin b; otherwise, it is coin c. They are not the same: coin b is very unfavorable; for example, it could have a win probability pb = 0.09, but coin c is much nicer, and could possibly have pc = 0.75 or higher. It may not be entirely obvious that this is a losing game; it appears we would use coin b a third of the time, and coin c for 2/3 of the times, so the expected gain probability would be, with these numbers, 0.09/3 + 0.75 × 2/3 = 0.53 > 1/2. The problem with this line of argument is that the phrase “expected gain probability” has not yet been well defined. To do that we could, for example, find out the fraction of games played when w is divisible by 3. Is there a reason to believe that every wining-score is equally likely? In fact, no. Exercise 9.116 asks you to calculate this since that is where we develop suitable tools to pursue this question. Here we can take an indirect route to compute the value of this game. We observe that it has a period (in the w dimension): we start at w = 0, which is divisible by 3, and once we return to this value, or reach ±3, whichever comes first, the game behavior “forgets” its past. Technically, it is a renewal point for the process. In particular, if we could show which of 3 or −3 is likelier to be reached first we would know if in the long-run this is a winning or 8 Much of the activity and information about publications and conferences can be found from the official page of Parrondo Paradox, http://www1.eleceng.adelaide.edu.au/Groups/parrondo/


543

losing game. qc

qc

−3

−2

pc

qb

−1

qc

0 pc

qc

1 pb

3

2 pc

pc

To do this calculation, we define a set of unknowns; let xa , for a ∈ {−2, −1, 0, 1, 2}, be the probability that our winnings reach +3 before they reach −3, if our current funds equal a mod 3. Since we start at w = 0, we want to know x0 : if it is larger than 1/2 then in the long-run we shall be making a mint. We set up the ‘balance equations’ that relate these probabilities by randomizing on the current state and the outcome of the first toss. In addition, we use, without writing them explicitly, x−3 = 0 and x3 = 1. Below we have the equations, which are self explanatory, and to the right its matrix form, which we used (gave it to MAPLE) to find the value of x0 , x−2 x−1 x0 x1 x2

= = = = =

pc x−1 ; pc x0 + qc x−2 ; pb x1 + qb x−1 ; pc x2 + qc x0 ; pc + qc x1 ;



    1 −pc 0 0 0 x−2 0      −qc 1 −pc 0 0  x−1   0        0     −qb 1 −pb 0     x0  =  0  ,      0 −qc 1 −pc   x1   0   0 0 0 0 −qc 1 x2 pc

pb p2c . 1 − pc qc − qb pc − pb qc Setting pb = 0.09 and pc = 0.75, x0 becomes 0.470930. . . , which is less than one-half. As a way to justify this, consider the possible trajectories of the winnings on the interval (−2, 2) and you will realize that it is likely to visit 0 quite a bit more than a third of the time.

which came out x0 =

So both games, A and B, are losing games. Now we define game C, which combines these two. It uses still another coin, coin d, which can be chosen to be fair! Game C is simple: we flip the coin d, if it shows heads we pick game A, and otherwise we pick game B (which would be played according to the value of w then). So we are playing either of two losing games with the same probability; what could this be good for? Does it look like a choice between the frying pan and the open fire? We need to calculate, and it is not hard to see that we have here the same structure as game B alone, but with different probabilities to get heads or tails. When w ≡ 0 mod 3, we have instead pb the value p∗b = (pa + pb )/2, and otherwise we use p∗c = (pa + pc )/2. That is the only change; the same equation and matrix hold when the various probabilities are starred, and this goes for the solution of x0 too, but plugging in it the same numerical values (choosing here pa = 0.49) we find x∗0 = 0.520916 . . .! That was what we promised. But what happened? A hint of what happens is given by the functional form of x0 previously; its nonlinearity in the mixing probability is the key feature. To make the situation better visible, let us assume that coin d is not necessarily fair: it has a probability pd to show heads; and then the starred

544


probabilities are p∗b = pd pa + qd pb , p∗c = pd pa + qd pc , and MAPLE produces for the key probability x0 the following expression: x0 :=

p3d p3a + 2p2d p2a qd pc + pd pa q2d p2c + p2d p2a qd pb + 2pd pa q2d pc pb + q3d pb p2c  3p2d p2a − 3pd pa + 2pd pa qd pb + 4pd pa qd pc + 1 − 2qd pc + 2q2d pc pb − qd pb + q2d p2c

It is fairly complex, but the important fact is that the nonlinearity in pd is not an artifact of the format of the expression for x0 . The diagram alongside shows the dramatic dependence of x0 on pd . The other coins were biased to produce pa = 049, pb = 009, pc = 075. The diagram shows that x0 > 05, making game C a winning one, for pd ∈ (017 06), but also that as pd gets much larger, the success probability of game C plunges far below game A or B. Exercise 9.115 [2] Consider a random walk of an “agent” A on the n nodes of a connected undirected graph. It is placed initially at a random node and makes one transition across an adjacent edge each time unit. Each of these edges is equally likely to be chosen. (a) Show that for a complete graph, questions about the time till all the nodes are visited are equivalent to questions in the coupon-collector problem, presented in Example 4.46. How many states would we need, to represent this walk as a Markov chain? (b) How would you formulate the Markov chain model when the graph is not complete? Exercise 9.116 [3] In the scenario of Game B and Game C, as described in Example 9.114, compute the following for each game: (a) A PGF, and first two factorial moments for the number of transitions between successive “absorptions,” conditioned on its being a winning absorption (at +3) or a losing one. How is this related to the value of the games? (b) The fraction of time (counted in transitions) that the winnings process spends in each of its possible values in the interval (−2 2).

Chapter 10

Introduction to Asymptotics The results of analyses of algorithms are rarely given by closed-form expressions. Usually, we get a difficult sum or integral. Even when we are lucky or resourceful, and derive closedform results, they tend to be hard to use; the formulas are too complicated to “see through” and get insight into the dependence of the algorithm performance on the parameters. Sometimes we can derive a formula for the answer to a problem and yet be unable to evaluate it numerically because the number crunching simply becomes overwhelming. Modern computers have come a long way to make such concerns far less serious than they once were. Nevertheless, computers fail to solve problems involving vast operations and we are forced to find another approach to overcome this obstacle. Nearly always, a typical problem in analysis of an algorithm contains a large parameter that represents the problem size. When the problem size is small, the performance issues are of small interest—even a poor algorithm delivers the goods. The relative quality of algorithms is judged by their handling large problems, where the differences between the good and the bad can be staggering. Therefore, the context of the present treatment is real-valued (positive) functions depending on a positive parameter, and the main interest is to determine the behavior of the functions for large values of this parameter. We plan to produce approximations, in a sense we shall see below, which is called asymptotic. The word asymptotic stems from a Greek expression meaning ”not meeting,” and is associated with the image of a function approaching, but not reaching (or meeting) a limit as its argument increases. Those unmet lines are called asymptotes.

This chapter contains many methods and techniques to obtain and analyze asymptotic behavior of different expressions involved. Insights into the character of random phenomena are customary obtained by looking at the asymptotic distributions of the random variables discussed (which is the topic of §10.7). Deriving asymptotic formulas usually require a solid mathematical background and good knowledge about many intermediate steps. Understanding impossibility to present all details, we use many results without proofs (but giving a reference) and then show other derivations. 545

CHAPTER 10. INTRODUCTION TO ASYMPTOTICS

546

10.1 Asymptotic Notations and Applications All the functions we look to approximate asymptotically are performance measures, or combinatorial counts; in other words, the functions are considered to be positive. This allows us to simplify the definitions, compared with situations where negative functions need to be considered as well. Here is our first notation, “similar” or “equivalence:” f (n) → 1, as n → ∞. f (n) ∼ g(n) means g(n) In this sense f (n) = 2n + 7 sin(n2 ) ∼ g(n) = 2n.

Sometimes we cannot find such a close approximation, and to that end we introduce the “big Oh” notation: f (n) = O(g(n))

[a more precise notation:

f (n) ∈ O(g(n)) ]

if there exist an integer n0 and a positive constant K such that f (n) 6 K g(n) for all n > n0 . The reason we treat the symbol ∈ as more precise here, is that the above relation is not a true equation—more on this below—it expresses the statement “ f (n) belongs to a collection of functions, all of which satisfy this relation.” The big Oh notation tells us that ultimately (a word we use in the sense “for large enough n”) f (n) is bounded by a multiple of g(n), or that f (n) increases no faster than g(n) does. However, it says nothing about the value of K (which may be minute or whopping large), and it leaves open the possibility g(n) to be a poor bound. √ Thus, it is true that 1/ n = n−1/2 = O(n), but this is a bound with little merit. Sometimes we can do no better. There is a history of complex algorithms whose cost function evaluation has gone through a sequence of improvements of bounds which differ very much. Note: Some computer scientists use the big Oh notation in the sense of “order of.” We do not do this since we have another symbol for this purpose, Θ, which is introduced in a moment. There is another notation that carries a similar but stronger message: f (n) increases more slowly than g(n): the “little oh” symbol: f (n) = o(g(n))

[or

f (n) ∈ o(g(n))]

if

f (n) →0 g(n)

as n → ∞.

(10.1)

Using the little oh notation, we can claim that the similar notation (∼) is equivalent to f (n) ∼ g(n)

⇐⇒

f (n) ∼ g(n) + h(n)

for any

h(n) ∈ o( f (n)).

(10.2)

Exercise 10.1 [1] Justify the assertion that the little oh notation makes a stronger claim than the big Oh, by showing that f (n) = o(g(n)) =⇒ f (n) = O(g(n)), but there are function pairs { f (n), g(n)} such that f (n) = O(g(n)) and f (n) ∈ / o(g(n)). The little oh makes a stronger claim than its big brother, and hence turns out to be somewhat less useful. While it is less frequently useful, little oh has a place under the sun, and one such use is in the idiom shown in the following exercise.

10.1. ASYMPTOTIC NOTATIONS AND APPLICATIONS

547

Exercise 10.2 [2] (a) Show that f (n) = g(n)[1 + o(1)] ⇐⇒ f (n) ∼ g(n). (b) Explain why the similar idiom (1 + O(1)) never caught on.

For the converse statement that f (n) is ultimately bounded from below by g(n), or that it increases at least as fast as g(n), we have the omega (Ω) notation, which also requires the existence of an integer n0 and a positive constant K: f (n) = Ω(g(n))

[or

f (n) ∈ Ω(g(n))]

if

f (n) > K g(n),

∀n > n0 .

(10.3)

You may well ask: what is the point of this additional notation since it is clear that whenever f (n) ∈ Ω(g(n)) ⇐⇒ g(n) ∈ O( f (n))? The answer is that we need to consider how we actually apply asymptotic notations. This is also the reason why the notation f (n) ∈ O(g(n)) is used as well as the generally accepted form f (n) = O(g(n)): unlike a true equation, the roles of the two sides are different. It provides information about f (n), the ‘complicated’ function, in terms of a typically much simpler g(n). This is the main purpose of this approach, and it makes sense to maintain this convention, that is, in asymptotic relations, we always keep the complicated expression on the left-hand side. Finally, for our last notation, we consider the situation where f (n) ∈ O(g(n)) and also f (n) ∈ Ω(g(n)). This is captured by the theta (Θ) notation: f (n) ∈ Θ(g(n))

[or

f (n) = Θ(g(n))] ⇐⇒ f (n) ∈ O(g(n))

and

f (n) ∈ Ω(g(n)).

Exercise 10.3 [1] Show that f (n) ∈ Θ(g(n)) implies the existence of an index n0 and two f (n) 6 C, ∀n > n0 . positive constants c and C such that 0 < c 6 g(n) Exercise 10.3 tells us that the theta notation is not quite as strong as the similarity notation (∼), except in the particular case where c = C = 1. Exercise 10.4 [2] Show that f (n) = n log n + O(n) =⇒ f (n) ∈ Θ(n log n).

10.1.1 Properties of the Big Oh Notation Of all the asymptotic relations we defined, the most commonly used is the big Oh. The reason it is more popular than the omega relation is that an algorithm user is more interested in an upper bound (a guarantee) than a lower bound. Here we discuss some further aspects of this type of approximation. Let us first visit the dual notations, = and ∈, we used previously. It is related to another point already mentioned that the function on the right-hand side of the relation needs to be simpler than the one on the left-hand side. It expresses the observation that we use on the right-hand side “representative” functions; in other words, such a function denotes a set of functions (and in fact, an infinite set of functions). Thus, if we write there O(n2 ), this stands for a multitude of functions, such as n2 + 3n or 5n2 − 2e−n/2 , and 7.2, and 3n1.73 , or (n sin n)2 , and many others—they are all members of the set O(n2 ), and hence the notation ∈ is more accurate in a purist sense. On the other hand, many prefer the notation =; while it conveys


548

here the same information, it reminds the user that the relation behaves in some ways like an equation. Let us now list the basic properties of the big Oh notation, including the equation-like properties/manipulations. Most of the relations below (except the first and seventh) are ordinary equalities, but between sets! The other asymptotic notations satisfy similar relations, which can in part be inferred from those we present now. 1.

f (n) = O( f (n)).

2.

C · O( f (n)) = O(C f (n)) = O( f (n)),

3.

where C is a constant (in n).

O( f (n)) + O( f (n)) = O( f (n)).

4.

O(O( f (n))) = O( f (n)).

5.

O( f (n))O(g(n)) = O( f (n)g(n)).

6.

g(n) O( f (n)) = O( f (n)g(n)).

7.

O( f (n)) + O(g(n)) = O( f (n) + g(n)), where the positivity simplified our notation.

8.

n p = O(nq ) for p 6 q.

9. O(lg n) = O(ln n) = O(logb n) for any b > 1. log a The property follows from the relation: logb a = logc b . Recall that we denote by lg n the c logarithm with base 2, and use ln n to denote the natural logarithm (to base e). Hence we could replace the big Oh by a theta. 10. g(O(nα )) = O(g(nα )). This relation only holds when g(·) satisfies certain constraints; the most important case is when it is an increasing polynomial (it suffices that g(n) be ultimately monotone increasing, and this only requires a positive leading coefficient). When we use asymptotic results as approximations, for values of the argument that are not necessarily very large, but in the range for practical problems, we find the need to consider additional issues, such as the accuracy of the approximations, and the rate of convergence. Also the asymptotic expression f (n) ∈ O(g(n)) may provide an estimate for values of n that we use in practice. This is one of the reasons for interest in improving the quality of asymptotics. Suppose we know a few terms of the given function, and a lower order term is bounded, as in log n . (10.4) a(n) = 2n + 3 log n + O n Such an expression is said to have an absolute error of O logn n . A statement about the relative error is possible as well: log n , a(n) = (2n + 3 log n) 1 + O n2 n . When the right-hand side from which we see the relative error: compared to 1, it is O log 2 n


549

has merely an O term, as in f (n) = O(n2 ), then the possible absolute error is a multiple of n2 , which is in O(n2 ), and the relative error is O(1). For alternating series, the absolute error can be easily estimated. We present the result known from calculus and attributed to G. Leibniz. Theorem 10.5 (Error Estimate for Alternating Series)

Let

∑ (−1)n an be an alternating

n>0

series with coefficients that satisfy the following conditions • an > 0

for all n > 0,

• |an+1 | 6 |an | for all n, and • limn→∞ an = 0. n

∑ (−1)k ak = L n→∞

Then the series converges to finite L: lim

(0 6 L 6 a0 ). Moreover,

k=0

n ∑ (−1)k ak − L 6 |an+1 |. k=0

That is, the error in approximating L by the sum of first n terms of the series is at most the magnitude of the next term. Note that the condition about monotonic convergence of an cannot be relaxed. There are known divergent alternating series with the terms that are not decreasing monotonously to zero. For instance, the series 112 − 11 + 212 − 21 + 312 − 31 + · · · diverges.

Exercise 10.6 [2] The convergent expansion for the logarithm function ln(1 + x) = − ∑ j>1 (−x) j / j satisfies the requirements of Theorem 10.5, for 0 6 x < 1. To what order do we need to expand the function ln(n + 2) to get a relative error smaller than 10−6 for n ∈ [100, ∞)?

Example 10.7 Assume we have two algorithms for a certain problem. Their costs have been determined to equal T1 (n) = 2n2 ln n + 2n2 ln ln n

and

T2 (n) = n2 ln n + 10n2 ln ln n.

Under the simplistic asymptotic view, both algorithms would be seen to have costs in O(n2 ln n), giving us no way (or reason) to distinguish between the two. Looking at the leading coefficients, however, gives a clear preference to the second algorithm. A short calculation reveals that T1 (n) < T2 (n) for all n < 2.15 × 108 , which is well beyond the practical range for many applications of “quadratic algorithms.”. You could claim that from an “infinitely remote” asymptotic point of view there is never a reason to keep anything beyond the leading term in an asymptotic expression, and the expression in Eq. (10.4) can be written as a(n) = 2n + O(log n), or even a(n) = 2n(1 + o(1)). It does not matter really what is the coefficient of log n since this term provides ultimately


550

a vanishingly small contribution. This is not the case when we need to know how fast (or slowly!) all but the leading terms diminish. Certainly we do not want to lose that information in an engineering context, where it is an approximation, and n may be large, but surely finite.

10.1.2 Asymptotic Expansions A typical way to describe an asymptotic estimate (or approximation) is as a characterization of the behavior of the function as its argument grows, or that it captures the main dependence of the function on a large argument. Example 10.8 Consider the following function: f (n) = 2n + 7 sin(n2 ). We know that |7 sin(n2 )| 6 7. Hence f (n) is close to 2n, and for large enough n it becomes relatively very close. The simplest situation in which it is easy to exhibit a sequence of better-and-better asymptotic estimates is when we have a power development series for the given function, say T (n), in terms of the problem size n; such a development is called an asymptotic series: T (n) = c0 +

ck c1 c2 c3 + + + ···+ k + ··· n n2 n3 n

(10.5)

Every prefix of this infinite series provides an approximation to the function since ) ( m ck for any m > 0. lim T (n) − ∑ k = 0 n→∞ k=0 n In fact, we can make a much stronger claim: the difference goes to zero even if we multiply it by nm , which gives us a measure of how fast this difference goes to zero. ) ( m c k for any m > 0. (10.6) lim nm T (n) − ∑ k = 0 n→∞ k=0 n In this case, we write

m

T (n) ∼

ck

∑ nk

(10.7)

k=0

and say that T (n) has the asymptotic expansion ∑nk=0 ck n−k or that series represents T (n) asymptotically as n → ∞. This expansion, due to Poincaré1 (1886), has an important property of being unique if it exists. The coefficients ck of the Poincaré asymptotic expansion are determined by the recurrence relations: # " k−1 c i c0 = lim T (n), ck = lim nk T (n) − ∑ i . n→∞ n→∞ n i=0 1 Jules

Henri Poincaré (1854 – 1912) was a famous French mathematician, a student of Charles Hermite.


551

We look at the topic of asymptotics to provide us with two types of results. One is practical approximations, for engineering applications; the other is of more theoretical interest: ‘behavioral’ information about functions when the argument is ‘sufficiently large,’ a phrase which may mean different things in different circumstances. Example 10.9 Let us consider the function f (n) = ln(n + 2). What do we know about its behavior as n increases? Clearly, we know it behaves like ln n: increases slowly (slower than any positive power of n). Also, when n is large, then n and n + 2 are relatively very close, so it is essentially just ln n. In the meantime, we may want to see if we could use the notation of the series in Eq. (10.5). A result, which you know, is the Maclaurin series for ln(1 + x), that is: ln(1 + x) = x − x2 /2 + x3 /3... (for |x| < 1). So we write ln(n + 2) = ln[n(1 + 2/n)] = ln n + ln(1 + 2/n), and find, that to order n−4 , 8 4 2 2 ln(n + 2) = ln n + − 2 + 3 − 4 ± · · · . n n 3n n Note that this series is not an asymptotic expansion of the function f (n) = ln(n+ 2), however, the difference, ln(n + 2) − ln n, has the Poincaré series representation as n → ∞.

Exercise 10.10 [1] Use the same approach to develop an asymptotic series for the function ln(n + n2/3 ). The key is to notice that when n is large—which is when we expect asymptotic developments to be interesting—it is still much larger than n2/3 ; by a factor of n1/3 , and this will be the ratio of successive terms in your series. As we saw in the previous exercise, asymptotic expansions need not be limited to negative powers of the integers. The definition of Poincaré series representation can be extended to series other than power series. Definition 10.11 Suppose that {φn (z)}, n = 0, 1, 2, . . ., is a sequence of functions defined on some interval (a, b) ⊂ R, and that for every n

φn+1 (z) = o{φn (z)}

as

z → c ∈ (a, b).

(10.8)

Then the set of functions {φn (z)} is said to be asymptotic sequence or scale. We say that f (z) admits the asymptotic expansion as z → c, which is expressed by writing m

f (z) ∼ if

∑ ak φk (z) k=0

(z → c ∈ (a, b)),

(10.9)

m

f (z) =

∑ ak φk (z) + O(φm+1(z))

as

k=0

z → c ∈ (a, b)

(10.10)

and say that ∑k ak φk (z) is a generalized asymptotic expansion with respect to the scale {φn (z)}. ⊳ Property (10.10) is equivalent to one of the following conditions: lim

z→c

f (z) − ∑nk=0 ak φk (z) =0 φn (z)

or

lim

z→c

f (z) − ∑n−1 k=0 ak φk (z) = an . φn (z)


552

An important example of general asymptotic expansion is given in §11.3.2. Exercise 10.12 [1] Expand ln(1 + ax + bx2 ), where a, b are real numbers not necessary positive, for x → 0 to within O(x5 ). The asymptotic expansion (10.5) may diverge for all values of n as the following example shows. T (n) =

Z ∞ −nt e

e−nt (1 − t + t 2 − t 3 + · · · ) dt dt = 1+t 0 1 1! 2! 3! − + − + ··· . n n2 n3 n4 0

=

Z ∞

Moreover, if the asymptotic series converges, its sum need not to be equal to T (n). The partial sum tN (n) =

(N − 1)! 1 1! 2! 3! − + − + · · · + (−1)N−1 n n2 n3 n4 nN

is the asymptotic approximation of the function T (n) and its truncation error can be estimated to be Z ∞ N −nt t e N! N dt < N+1 . εN = T (n) − tN (n) = (−1) 1+t n 0 When the asymptotic expansion diverges we may not be able achieve arbitrarily high accuracy; nevertheless, the relative error can be—and often is—extremely small. Next we consider the notion of improving an asymptotic bound. An example is all we need here: we analyze a complicated algorithm, and can only find that its expected cost, tn , is at most proportional to n2 , at least for large n; this is exactly the information conveyed by the statement tn ∈ O(n2 ). After a while, further, more careful analysis shows that in fact tn is smaller, and we can claim tn = O(n2 / ln n). This is of course in O(n2 ), but it is a better bound and we say that O(n2 / ln n) is an improvement on O(n2 ). Finally, the following theorem gives a connection between regular functions and asymptotics. Theorem 10.13 Let f (z) be regular in the disk Dr = {|z| 6 r}, for some r > 0, then f (z) has a power series expansion ∑n>0 fn zn that is absolutely convergent in the disk Dr . In that case, for any positive integer k, we have |z| 6 r. f (z) = f0 + f1 z + · · · + fk zk + O zk+1 , P ROOF : This holds since k

f (z) − ∑ fn zn = zk+1 ( fk+1 + fk+2 z + · · ·) , n=0

when |z| 6 r. The assured convergence of the expansion implies that the infinite summation in parentheses is convergent; since this holds on a closed set, it is uniformly convergent, hence bounded by some constant. The claim follows.


553

From this general property, we get, for example, that a na/n = ea ln n/n = 1 + ln n + O [(ln n)/n]2 , n for any a since the exponential function is an entire function (with a power series expansion that converges everywhere). The application of logarithms is a common technique in finding asymptotic expressions. However, it needs some care. The following result has proved useful in such situations. f (n) = K > 0, then Lemma 10.14 Let both f (n) and g(n) approach +∞ as n → ∞. If lim n→∞ g(n) ln f (n) = 1. In particular, if f (n) ∼ g(n), then ln f (n) ∼ ln g(n). lim n→∞ ln g(n) Exercise 10.15 [2] Prove Lemma 10.14. The requirement in Lemma 10.14 on functions f and g to be unbounded cannot be relaxed. K +n1 For example, Lemma 10.14 is not applicable for functions g(n)= 1 + n1 and f (n) = 1 1 1 1 = ln K + ln 1 + ∼ ln K + O . because ln g(n) ∼ and ln f (n) = ln K 1 + n Kn Kn n Therefore, despite lim we have n→∞ f (n)/g(n) = K, 1 1 1 ln f (n) = lim n ln K + n ln 1 + = ∞ since lim n ln 1 + = . lim n→∞ n→∞ n→∞ ln g(n) Kn Kn K

If ln f (n) ∼ ln g(n), then there exists a function h(n) such that ln g(n) = ln f (n)+ h(n) and h ∈ o (ln f (n)) = o (ln g(n)). Exponentiation yields g(n) = eln g(n) = eln f (n)+h(n) = eln f (n) eh(n) = f (n) eh(n) . Therefore, the equivalence relation f (n) ∼ g(n) follows from ln f (n) ∼ ln g(n) if eh(n) tends to 1 as n → ∞. But this happens if and only if h(n) → 0. The next exercise gives an example that the statement of Lemma 10.14 cannot be reversed without additional restrictions on the asymptotic behavior of functions f and g; that is, from ln f (n) ∼ ln g(n) does not always follows that f (n) ∼ g(n).

Exercise 10.16 [1+] The Stirling formula (2.87) can be “written logarithmically,” as ln(n!) ∼ 1 1 2 ln(2π n) + n ln n − n. Since 2 ln(2π n) ∈ o(n, n ln n), it is also true that ln(n!) ∼ n ln n − n. Explain why it is incorrect to conclude from this that n! ∼ nn /en .

Example 10.17 The binomial coefficients take part in many of the expressions we use in analysis of algorithms. We develop the formula, which will be used later, and its importance will become clearer as well; it is also an example of asymptotics at work. We consider the n relationship between ns and s+t . In Chapter 2, we saw formulas for all n and s and for t = ±1. Here we look at cases where n is large, and t, while small compared with n and s, could still be quite a large number. The derivation will make these observations explicit. s (n−s)t n ns+t = ns!(s+t) Using falling factorials we can write s+t = (s+t)! t . We separate from this ratio the n ns factor s! , which equals s and consider the rest: 1 2 t−1 (n − s)t 1 · 1 − n−s 1 − n−s · · · 1 − n−s (n − s)t . · = (s + t)t st 1 + 1s 1 + 2s · · · 1 + st


554

t , and look at the remaining expression, which we Again, we separate the first factor, n−s s denote by T . It is clearly a number which is smaller than 1; when is it close to 1? Can we estimate it? Assuming for the time being that t = o(s), we estimate the logarithm of each of the terms in the numerator and denominator of T using a two-term prefix of the logarithm function expansion, ln(1 + x) ∼ x − x2 /2 for small x. Hence, we can write t k j − ∑ ln 1 + ln T = ∑ ln 1 − n−s s j=1 k=1 t t−1 j2 k2 k j − − 2 − ∑ =∑ − n − s 2(n − s)2 2s j=1 k=1 s 1 t3 t3 1 t 1 + + , − +Θ = − t(t − 1) 2 n−s s s (n − s)2 s2 t−1

using standard summations. Hence 1 1 t(t − 1) t(t + 1) 4 − +O t + , T = 1− 2(n − s) 2s (n − s)2 s2 where we also used the three-term expansion of the exponential function, ex = 1 + x + x2 /2 + · · · . Pulling it all together, we have that

n s+t

n n−s t , ∼ s s

when t 2 = o(s)

and t 2 = o(n − s).

(10.11)

Exercise 10.18 [3] (a) What happens if you substitute s = n/2 in Eq. (10.11)? Explain. (b) The calculation leading to Eq. (10.11) assumes that t > 0. What changes when this does not hold? Show how the above derivation would be affected. Alternatively,you could do a n similar calculation, of the same form as above, again assuming t > 0, for s−t , with the same assumptions about relative sizes of n, s,t.

Example 10.19 (The birthday problem) We consider a minor generalization of Example 3.19, page 90. Assume a group of n people, that use a calendar with m days per “year.” What is the probability of the event A, that at least two in the group share a birthday?

Again, as in Example 3.19, the probability of the complementary event, A′ , that they all have different birthdays is Pr A′ =

mn m! = (m − n)! mn mn

=⇒

Pr[A] = 1 −

m! mn = 1 − . (m − n)! mn mn

How large does n need to be to make Pr[A] > 0.5? The answer is determined by the equation 1 m! = , n (m − n)! m 2


555

where n is used as a continuous variable, and then taking ⌈n⌉. We assume that n is smallcompared to m. For the following calculation to hold we need m to be large, and n ∈ o m3/4 . We √ m use the Stirling formula, Eq. (2.89) on page 77 for m! in the form m! = 2π m me eθ /12m , where 0 < θ < 1. Hence we get √ m 2π m me eθ /12m m! =p m−n (m − n)! mn eθ /12(m−n) mn 2π (m − n) m−n e θ θ r m−n 12m − 12(m−n) e m m = m−n m−n en r − θn e 12m(m−n) 1 m . = m − n 1 − n m−n en m

n . We similarly replace In this expression, the first factor, to first order in (n/m), equals 1 + 2m the factor exp {−θ n/12m(m − n)} by the first two terms in its power expansion, and find θn ; hence, the product of both terms is 1 + O(n/m). 1 − 12m(m−n)

Next, we look at the denominator, and “logarithmize” it, o n n n m−n n ln(1− mn ) m−n n def +n . e = e e = exp (m − n) ln 1 − D = 1− m m 2

3

Using a prefix of the Taylor expansion for logarithm: ln(1 − x) = −x − x2 − x3 , we obtain o n n n2 n3 n + n = exp (m − n) − − 2 − 3 + n D = exp (m − n) ln 1 − m m 2m 3m 2 3 2 3 4 2 n n n n n n3 n4 n − + + + + n = exp + + = exp −n − 2m 3m2 m 2m2 3m3 2m 6m2 3m3 n2 n , = e 2m 1 + O m where we assumed n ∈ O m1/2 . Therefore, our equation is currently h n i n2 , n ∈ O m1/2 . 2 = e 2m 1 + O m Applying the natural logarithm to both side of the equation, we obtain the value ln 2 =

n2 2m

=⇒

n2 = 2m ln 2

=⇒

n≈

√ √ 2m ln 2 = 1.1774 m

for the required threshold number of people in the group. Naturally, this number is not an integer, and since we prefer our people whole, you need to take the ceiling of this value. If you do this for m = 365 you get n = 22.495..., which agrees with Example 3.19. Can this rather elaborate calculation be simplified? Not much, the way we went at it, but here is another try: we attack the same problem by finding an upper bound for mn /mn from which the lower bound for Pr[A] will follow.


556

First, we rewrite this ratio as a product j mn m(m − 1)(m − 2) . . .(m − n + 1) n−1 = = ∏ 1− . mn m · m · m · ...· m m j=0

(10.12)

As we know from Example 1.38 (page 18), the geometric mean is strictly less than the arithmetic mean if all terms are distinct. So n 1/n ∑n−1 1 − j j n − ∑n−1 j=0 m n−1 m j=0 m < = = 1− . mn n n 2m The desired lower bound for Pr[A] is n−1 n . Pr[A] > 1 − 1 − 2m

(10.13)

For example, for n = 23 and m = 365, approximation formula (10.13) yields Pr[A] > 0.5053 (compare with exact value Pr[A] = 0.5073). 2

We stress that 1 − e−n /2m is only approximation to the actual probability that there is a match of birthdays in the group of n people. For example, suppose we have 35 license plates, each ending in a 3-digit number. We have m = 103 = 1, 000, the number of possible 3-digit n2 ≈ 0.6125, numbers, and n = 35, the number of license plates under consideration. Since 2m the approximate probability of a match is 1 − e−0.6125 ≈ 0.458 . The correct answer is obtained by calculating 2 34 1 1− ··· 1− = Q0 (1000, 35) ≈ 0.452264 . . . , 1− 1− 1000 1000 1000 where Q0 (m, n) is the Knuth-Ramanujan Q-function introduced in Exercise 2.21 on page 36. Exercise 10.20 [1] Repeat the calculations of√Example 10.19, to get the value of c in the result for the threshold count of the form n = 2m ln 2 1 + cn/m + O(n3 /m2 ) .

Example 10.21 (From binomial to Poisson distribution) A binary random variable is said to be binomially distributed if the probability of j successes def in n trials is expressed as p j = Pr[B(n, p) = j] = nj p j (1 − p)n− j (see Eq. (4.2) on page 136). We consider a limiting form of the binomial distribution, which is usually referred to as the law of small numbers. When the probability of success, p, is small and the number of trials, n, is reasonably large, we observe a rare event (success) and the binomial distribution can be approximated within a reasonable remainder term by the Poisson distribution. Let n → ∞ and p → 0 while keeping the product np constant, at a value we denote by λ . What happens to


557

j

p j ? We write it in the form p j = nj! qn (p/q) j . Then replace p by λ /n and let n increase. This yields nj λ nλj 1 (10.14) 1− pj = j! n n j (1 − λ /n) j n n j · 1 · 1 − 1 1 − 2 · · · 1 − j−1 j n n n λ λ . × 1− = j j! n n j 1 − λn n As n → ∞ we have 1 − λn → e−λ , and after cancellations, both numerator and denominator j

in the large fraction above go separately to 1. So we have shown that p j → e−λ λj! , the Poisson probability mass function that we saw in §4.1.6. For this reason, the Poisson distribution is sometimes called the distribution of rare events, and the above approximation can be rewritten as d

B(n, p) −→ Poisson(np). Exercise 10.22 [3] The proof in Example 10.21 was for the limiting value, with a fixed, finite j. Here we ask for two variations. (a) Develop an asymptotic expression for p j as n → ∞ and p = λ /n to relative error in O(n−3 ). (b) Develop an asymptotic expression for p j , with a non-constant j = θ n, where θ ∈ (0, 1).

Exercise 10.23 [3+] (Limits Nof distributions) The hypergeometric probability mass function h(x, n; N, m) = mx N−m / n−x n is defined in Eq. (4.9), page 151. Show that as N → ∞ while d

mn/N → β , then h(x, n; N, m) → Poisson(β , x), as defined in Eq. (4.15) on page 155. Note: Since this bound relates to the situation where m ≪ N, it does not pertain to the setup considered in §4.1.5, where m and N − m are assumed to play the same role. In that scenario, as N, m → ∞, n remains finite, and the distinction between sampling with or without replacement vanishes. If we allow m/N → p there, the situation considered in Example 10.21 emerges.

Exercise 10.24 [4] (Limits of distributions) In Exercises 8.19 and 8.36 we found the distribution called in the latter t(m, n, m − r), the probability that distributing n labeled balls uniformly over m ordered bins leaves exactly r bins empty. This is equivalent to “sampling bins with replacement” n times, and never visiting r of them. Its value is given as n ; let us denote this probability by pr (n, m). Show that as both n t(m, n, m − r) = mm! n r! m−r and m increase without bound, so that me−n/m → α , then pr (n.m) → Poisson(α , r). Guidance: (1) The given relation between n and m can be rewritten as n ≈ m ln(m/α ). (2) The direct approach seems to be hard. Instead, use the given formula for the probability of having no empty bins, p0 (n, m), and replace the Stirling number mn by its sum representation, Eq. (C.14). (3) Show that as n and m grow, in the manner specified, p0 (n, m) → e−α . (4) m m−r n p0 (n, m), and Show that the probability pr (n, m) can be rewritten as pr (n, m) = r m


558

that the added factors converge to the desired result. This is essentially the proof suggested by [41, §IV.2]. Exercise 10.25 [3] Estimates of factorials are often essential in analysis of algorithms. (a) For which range of k ∈ N, in terms of n, does the following relation hold: (n ± k)! ∼ n±k n!ek

2 /2n

,

as n → ∞?

(10.15)

(b) Show that for 0 6 k < n/2 we have the two-sided bound k2

k

k3

e− 2n + 2n − 3n2 6

2 k nk n! − k2n + 2n = 6 e k k n (n − k)! n

as n → ∞.

Exercise 10.26 [2] Show that (n + 1 + O(n−1 ))n = enn + O(nn−1 ), n−i .

n → ∞.

a(n)n

Compute to within a relative error of Exercise 10.27 [2] (a) Let a(n) = ∑i>0 ai −2 O n . (b) Let a(n) be the same as in part (a), and let b(n) = ∑i>0 bi n−i . Compute a(n)b(n) , to the same relative error, O n−2 . 1 2n Exercise 10.28 [2−] Let Cn = n+1 n be the nth Catalan number. Compute an asymptotic expansion for the Cn th harmonic number, HCn , to within O(1/n). Definition 10.29 The support of a function is the set of points where the function is not zero. ⊳

Exercise 10.30 [3+] (A. Boneh) Let {ai } be a set of n positive real numbers. For any t ∈ R, 1/t def . define their t-mean as mt {ai } = 1n ∑ ati (a) Show that mt {ai } is monotonic increasing in t : t1 < t2 =⇒ mt1 {ai } 6 mt2 {ai }. (b) As t increases, from −∞ and up, show that mt {ai } has, at selected points, the following listed types of means. t

−∞

−1

0

1

2

+∞

mt {ai } min{ai } Harmonic Geometric Arithmetic Quadratic max{ai } (c) Show that the t-norm, k{ai }kt = (∑ ati )1/t , is also monotonic in t > 0, but it is monotonicdecreasing! More precisely, as t increases from −∞ towards 0 the values of k{ai }kt decrease, from min{ai } to zero. At t = 0 the norm is undefined, and as t increases from 0 towards ∞, the norm decreases from ∞ to max{ai }. Note: The reason this exercise appears in this chapter is the calculation needed in part (b) for the points t = 0, ±∞. One way of doing part (a) uses Jensen’s inequality (F.25) in Appendix F, which states that if f (·) is convex on the support of the random variable X , then E[ f (X )] > f (E[X ]). It is possible to generalize on the claim made in part (a), and replace the uniform weights, 1/n, by giving to each ai the positive weight wi , which sum to 1. This def makes Mt {ai } = (∑ wi ati )1/t the weighted mean of the sequence. def


559

Exercise 10.31 [2+] (adapted from [122]) Consider t-component heterogeneous compositions, a1 +a2 +· · ·+at = n, where the component ai is a multiple of the integer di . In addition, 1 = d1 < d2 < · · · < dt . Let tn be the number of such compositions, for n > 0. (a) Write an explicit, simple expression for the ordinary generating function Tt (x) = ∑n>0 tn xn for the sequence {tn }. (b) Prove by mathematical induction on t the claim in the text that the leading asymptotic term in tn is given by nt−1 . βn,t = d1 d2 · · · dt (t − 1)! Exercise 10.32 [4] We construct a binary search tree (BST) from scratch by inserting in it the content of a permutation of [1..2r − 1]. This number of integers could end up forming a complete tree of r levels. What is the probability this happens, given that all permutations are equiprobable? Find a meaningful asymptotic expansion for the answer. Exercise 10.33 [2] For a real x, show that xi+ j ∑ i! j! ∼ e−2x − I0(2x) i>1 06i6 j−1

2∑

as

x → ∞,

2n

where I0 (z) = ∑n>0 n!z n! is the modified Bessel function of the first kind. n−s n sk s2 k + sk2 , Exercise 10.34 [2+] If s + k = o(n3/4 ), show that ∼ exp − − n 2n2 k k as n → ∞. n−s n Exercise 10.35 [1] If s2 + k2 = o(n), show that ∼ 1, as n → ∞. k k 1 n ∼p λ −λ n (1 − λ )−n(1−λ ) Exercise 10.36 [2] Let 0 < λ < 1, prove that λn 2πλ (1 − λ )n as n → ∞ by showing that

n λ −λ n (1 − λ )−n(1−λ ) −1/(12n λ (1−λ )) λ −λ n (1 − λ )−n(1−λ ) p e . < p < λn 2πλ (1 − λ )n 2πλ (1 − λ )n

10.1.3 Limits for Indeterminate Forms One of the first basic results of calculus presents a way to compute the limit of a ratio of functions, when both approach zero or infinity, since the ratios 0/0 and ∞/∞ are both called indeterminate. Sometimes this is simple to avoid. For instance, the ratio (2x + 3)/(x − 2+3/x 1) seems to approach the indeterminate ∞/∞ as x → ∞, but since 2x+3 x−1 = 1−1/x → 2 as x increases, there is really no ambiguity there. However, when life is not this simple we have the rule called after French mathematician2 Guillaume de l’Hôpital (1661 – 1704). We shall prove it in some detail since this proof presages the other theorems we have in this subsection. 2 The rule was first derived by Bernoulli who generously

allowed his result to be published in l’Hôpital’s book.

560


Theorem 10.37 (The l’Hôpital rule) [118] Let f (x) and g(x) be real and differentiable in the interval (a, b), and g′ (x) 6= 0 for all x ∈ (a, b), where −∞ 6 a < b 6 ∞, and finally, let f ′ (x) → L, as x → a. g′ (x) If (i) f (x) → 0 and g(x) → 0 as x → a,

or

f (x) → L, g(x)

(10.16)

if (ii) g(x) → ∞ as x → a, then as x → a.

(10.17)

The same claims hold for x → b, and also when the functions g(x) and f (x) approaches −∞. The statements are valid for any L, which can be either finite or infinite. P ROOF : We consider only |L| < ∞, our likelier case. Then there exist two numbers r and q such that L < r < q. The limit in Eq. (10.16) implies the existence of c ∈ (a, b) such that for x ∈ (a.c) we have f ′ (x) < r, x ∈ (a, c). (10.18) g′ (x) A form of the mean value theorem guarantees now that for any pair of points x and y such that a < x < y < c there exists a t ∈ (x, y) such that f ′ (t) f (x) − f (y) = ′ < r, g(x) − g(y) g (t)

x ∈ (a, c).

(10.19)

Now, if condition (i) above holds, we let x → a in Eq. (10.19), and find that f (y) 6 r < q, g(y)

y ∈ (a, c).

(10.20)

In the same way we can show that for any p < L there is a d ∈ (a, b) such that f (y) > p, g(y)

y ∈ (a, d).

(10.21)

and the claim Eq. (10.17) follows. For the condition (ii) the calculation that produces the equivalent of Eq. (10.20) (and (10.21)) is somewhat more involved. For a fixed y in Eq. (10.19), we use the existence of a v ∈ (a, y) such that g(x) is positive and larger than g(y) for x ∈ (a, v). Then multiply the inequality in Eq. (10.19) by g(x) − g(y) to get f (x) − f (y) < r[g(x) − g(y)], for x ∈ (a, v); dividing both sides by the positive g(x) and rearranging, we find g(y) f (y) f (x) < r−r + , x ∈ (a, v). (10.22) g(x) g(x) g(x)


561

As we let here x → a, g(x) increases, and at some point u ∈ (a, v), we shall find that f (x) 6 r < q, g(x)

x ∈ (a, u),

(10.23)

similar to Eq. (10.20). The analog of Eq. (10.21) follows in the same way. Notes: (1) The same proof holds when x approaches the other end of its range, b. (2) If the functions f and g are generating functions, they typically can be developed in power r+ f r+1 +··· r+1 x series around zero. We then have in the left-hand side of Eq. (10.17) the ratio gfr xxs +g s+1 +··· , s s+1 x and then clearly L = 0 when r > s, it is infinite when r < s, and it is fr /gr when they are equal. When the power-series are not that evident from the available functional forms, the differentiations bring it out. Exercise 10.38 [2] Prove Theorem 10.37 for the case that L = ∞. Hint: Use the reasoning that led to Eq. (10.21).

So far we used the convenience of continuous arguments. What can we do when the functions have discrete arguments? What is the limit of u(n)/n, when n → ∞? It is clearly zero when u(n) is bounded, but if it is not? Or we do not know? We need a “discrete version” of l’Hôpital’s rule! [67]. Theorem 10.39 Assume that functions f (x) and g(x) are defined on the infinite interval [a, ∞), for which the following three conditions hold. (1) Either (i) lim f (x) = lim g(x) = 0, or (ii) lim g(x) = ∞. x→∞

x→∞

x→∞

numbers3

(2) There exist h > 0 and x0 > a such that ∆g(x) = g(x + h) − g(x) does not change sign for x > x0 . ∆ f (x) = L (finite or infinite). x→∞ ∆g(x)

(3) lim Then

f (x) = L. x→∞ g(x) lim

(10.24)

Note: If, in addition, the functions f and g are bounded on any finite interval, and if the three def def conditions above hold for f = ∆k f and g = ∆k g, then the limit (10.24) exists and equals L. P ROOF : Without loss of generality, assume that ∆g(x) > 0 for x > x0 > a. First, let L be finite. Given any ε > 0, there exists N > x0 such that for x > N:

L−ε
x0 such that ∆ f (x) > M∆g(x), x > N; this and condition (2) imply ∆ f (x) > 0 for x > N. This allows us to use the result for finite L. The non vanishing of ∆ f (x) > 0 for x > N implies ∆g(x) ∆ f (x)

−→ n→∞

1 = 0+ L

=⇒

g(x) = 0+ x→∞ f (x) lim

=⇒

f (x) = ∞. x→∞ g(x) lim

Example 10.40 (a) Given the ratio an /n, what can we say about its limit as n → ∞? Using h = 1, Theorem 10.39, says that if the fraction ∆an /∆n = √ an+1 − an has a limit, this limit is n the same as the original ratio. If, for example, a = ∑ n k=1 k + 1, then the differenced ratio √ gives n + 1, which has the infinite limit. (b) Here is a case where the answer is self-evident, but we cannot apply the theorem: it says the limit of sinn n equals that of sin(n + 1) − sin n , if it has a limit. . . , but this difference has no limit, meandering in the interval [−1, 1], whereas the limit of the original ratio is obviously 0. (c) What is limn→∞ (an )1/n ? Here we need a devious path; let f (n) = log an , and g(n) = n. Then an+1 f (n) log an 1/n 1/n = lim [log an+1 − log an ] = lim log =⇒ lim log an = = log an . n→∞ n→∞ n→∞ g(n) n an def

def

Again, if the last limit exists, that is the value we are looking for.

(d) In the following example, the variable x is continuous and tends to infinity, but the function of interest contains the nondifferentiable term ⌊x⌋. Therefore Theorem 10.37 cannot 2π x √ ? be used, and we must appeal to Theorem 10.39. What is the value of limx→∞ ⌊x⌋(√sinx+1− x) Conditions √ (1i) and (2) of our second theorem are satisfied if we define f (x) = sin(2π x)/⌊x⌋ √ and g(x) = x + 1 − x. To verify condition (3), we observe that 1 1 − sin 2π x sin 2π (x + 1) sin 2π x − = sin 2π x − ∈ O x−2 = ∆ f (x) = ⌊x + 1⌋ ⌊x⌋ ⌊x⌋ + 1 ⌊x⌋ (⌊x⌋ + 1)⌊x⌋ and

∆g(x) =

√

p √ √ √ p x+2−2 x+1+ x = x 1 + 2/x − 2 1 + 1/x + 1 .

Using the binomial theorem to develop the two square roots and the obvious cancellations, √ f (x) is in O x−1/2 , we have ∆g(x) = x − 4x12 + O x−3 ∈ O x−3/2 . Hence the ratio ∆∆g(x) which has the limit zero; this must then be the limit of the original function.

10.2. THE CRITICAL RANGE METHOD

563

∑nj=1 (2 j − 1)2 = 1 twice: by direct calculation and Exercise 10.41 [2] (a) Show that lim n→∞ ∑nj=1 (2 j)2 by using Theorem 10.39. ∑nj=1 j p 1 . (b) For p ∈ N, show that lim p+1 = n→∞ n p+1 n p ∑ j=1 (2 j − 1) 2p . = (c) Show that lim n→∞ n p+1 p+1 i ∑ni=1 (n + 1 − i) 1 + 1i . Exercise 10.42 [3] (a) Evaluate lim 2 n→∞ n + n sin (nπ + π /4) Hint: It is best to take second order differences of the numerator and denominator here and use Note to Theorem 10.39. n p n 1 ∑i=1 i − = . Hint: Bring the difference to a (b) For p ∈ N, show that lim p n→∞ n p+1 2 common denominator.

10.2 The Critical Range Method Many problems in algorithmic analysis lead to sums for which we know no closed form. The problem of developing asymptotic estimates for such sums has been given much attention. In this section, we consider sums with fast-varying summands, involving factorials and exponentials. If the dominant contribution to the sum comes from a relatively small portion of the summation range, then this part of the sum provides an asymptotic representation of the whole sum. This technique of finding fast-varying terms in a sum that can serve as its asymptotic expansion is referred to as the critical range method. Many sums needed for analysis of algorithms have the following two properties that help in obtaining usable asymptotic values: • The summands in the subinterval that provides the main contribution need to be uniform in sign and unimodal in the index of summation (that is, they consist of monotonic sequences of terms). • Since the terms are fast-changing in size, nearly the entire value of the sum comes from the contribution of a small fraction of the summation range. Suppose we want to find the value of the sum that depends on a big parameter n: Sn =

∑

an (k) ,

k∈Dn

where Dn is a domain of summation (that may also depend on this parameter n). The following procedure, which may be viewed as an adaptation to the discrete case of the Laplace method (§11.3.3), is suggested to handle such sums:


564

1. Look for the value (or several values) of k = k0 that maximizes an (k) in the given domain: an (k0 ) = max an (k) . (10.25) k∈Dn

2. Evaluate the sum over the critical value (values) k0 . Sometimes this can be done directly, often various asymptotic approximation methods are naturally applied to estimate this sum. 3. It is convenient to change the index of summation, so that the maximal term obtains at or near zero: j = k0 − k, and rewrite the sum as Sn = ∑ j an (k0 − j). We demonstrate the critical range method with an elaborate example, but it is appropriate to precede it with a suggestion (and a warning) that pertains to almost all work in asymptotics: never trust a formula or a method ‘blindly.’ It is necessary that you get to understand the characteristics of the terms of the sum to judge the suitability of the critical range used by the formula or method. We consider the following sum: r n ∑ k , k=0 λn

An =

for some 0 < λ 6 1, r > 0, and large n.

(10.26)

There seems to be no closed form for An except when λ = 1 or 1/2 and r is a small integer. According to Exercise 2.114, the maximal value of the summand is obtained at ( n max = 06k6n k

n ⌊n/2⌋

n n/2 ,

=

n is even,

n ⌈n/2⌉

,

n is odd.

The derivation of the sum proceeds differently in the cases λ = 1/2, λ < 1/2, and λ > 1/2. Let us start with λ = 1/2. If we assume that n is even, the sum goes up to the maximum value at n/2 (and Exercise 10.43 asks you to show that we obtain the same result for odd n, when the sum goes to ⌊n/2⌋). We shift the summation index: j = 2n − k, and then n/2

An =

∑

j=0

n n − j 2

r

.

(10.27)

This sum is as opaque as the original one, but is easier to manage since the main contribution occurs when the index j is small, allowing to truncate expansions. We approximate the binomial coefficient nk using Stirling formula (§2.7), and this will suggest the continuation: √ n k n n−k 1 1 n n + 1+O . =p n−k k n−k k 2π k(n − k) k

(10.28)

10.2. THE CRITICAL RANGE METHOD n − j, 2 Then the square root term becomes

n−k =

With our change of variables, k = √

n p =q 2π k(n − k) 2π

√

n n 2− j

n 2

565

+j

=r

n + j, 2 √

n 2n = , k n−2j

n

2 2π 1 − 4nj2 · 2n

n 2n = . n−k n+2j

√ 1 2 . = √ ·q πn 4 j2 1− n2

Since the contributing range of j is for small values only, we can use the binomial theorem to write for the last denominator −1/2

(1 − x)

3 1 = 1 + x + x2 + · · · 2 4

Therefore

=⇒

4 j2 1− 2 n

−1/2

4 j 2 j2 = 1+ 2 +O 4 . n n

√ 2 j2 n 2 p = √ · 1+ 2 + ··· . n πn 2π k(n − k) √

We shall neglect the second term in the brackets since it is below the relative error threshold we already established by selecting the Stirling formula truncated as in Eq. (10.28). For the next two factors in the binomial coefficient, we write:

2n n−2j

and then

n/2− j

2n n+2j

n/2+ j

n

=2

2j 1− n

−n/2+ j

2j 1+ n

−n/2− j

n 2j 2j n + j ln 1 + − . ln Tn = − + j ln 1 − 2 n 2 n

def

= 2n Tn ,

(10.29)

Now we use Maclaurin expansion of the logarithm function to second order, ln(1 + x) = 2 x − x2 , which holds for |x| < 1; here the role of x is played by j/n which is in o(1). It follows that 2j 1 2j 2 1 2j 3 2j ± − ··· . =± − ln 1 ± n n 2 n 3 n Substituting these expansions into Eq. (10.29), we find after extensive cancellations that j4 2 j2 ln Tn = − n + O n3 , where the O term is supported by Theorem 10.13. Hence,

4 4 4 j j j 2 j2 2 j2 2 j2 +O 3 = exp − = exp − . ·exp O 3 1+O 3 Tn = exp − n n n n n n n with Since Tn = 1 when j = 0, the initial factor, (2/(nπ ))1/2 2n , is our approximation of n/2 the agreement of Eq. (2.92). Collecting all the pieces, we find 4 r 4 j n j 2 n −2 j2 /n n −2 j2 /n = = . (10.30) e 1+O 3 2 e 1+O 3 n nπ n n/2 n 2− j


566

2

1 1 + n−k ) = O( N1 + nj 3 ), is within the considered For small j, the error term in Eq. (10.28), O( nk accuracy. We have been justifying our retaining a single “error term” by citing our intention to sum over relatively small values of j only. The underlying reason is our knowledge of the behavior of the binomial coefficients: nk increase very fast as k increases from 0, but slow down as k approaches n/2, and on the “shoulder” the rate of change is slow.

Since j = 0 corresponds to the middle of the shoulder, we need to sum until we effectively go off it; how far do we have to go? We need a quantitative estimate, and it is provided 2 n n ∼ n/2 e−2 j /n , we clearly need to go further than by the last relation (10.30). Since n − 2 j √ n, but in view of the largest remaining error term, j4 /n3 , we need not exceed n3/4 . If we pick, say, n0.6 as the upper limit of the sum, then at that point we find the exponential factor is exp(−2n0.2 ), which even for moderate n is a very small fraction, and it approaches zero exponentially fast as n increases. We don’t need to settle on any particular value, and just assume that we sum up to ns , with s ∈ (0.5, 0.75). ........ ... ... ... .... ... .... ... n .. Introducing this approximation into the sum (10.27), we ... .. ... .. k ... ... ... .. ... need to evaluate: ... .. ... ... ... ... ... .. .. . s s . ... r n r n .... ... . 2 n n 2 .. . −2 j r/n −2 j r/n ... .. . ... . e . e = . . ∑ ∑ n/2 ... .. . ... n/2 .. ... . j=0 j=0 .. How can this sum be evaluated? It seems that we escape one hurdle only to be faced with another one, seemingly as high! Well, here we have a way. In §10.4, we go in some detail into the relationship between sums and integrals (and you may remember from the treatment of Riemann integrals in calculus the elements of this relationship), but at this stage we only observe the similarity of the above summand to the density function of the standard normal random variable, denoted by N(0, σ 2 ). It has the density function

ϕ (x) = √

1 2 2 e−x /2σ 2πσ

and

Z ∞

−∞

. ... ... .... .. ... .. .. ... .. ... ... .... .. .. .. .. .... .. ... .. .. ... .. .. .. .. .... . .. .. .. .. . . . . . .............................

... ... ... ... ... .. ... ... .. ... ... .. ... .. ... ... .. .. ... .. .. .. ... .. .. .. ... .. .. .. .. .. .. ... ... .... .......... ......................

k

Figure 566:

ϕ (x) dx = 1 .

Approximating the series by the corresponding Riemann integral of ϕ (x), we have ns

−2 j2 r/n

∑e

j=0

≈

Z ns 0

− 2rn x2

e

dx ≈

Z ∞ 0

2 − 2r n x

e

dx =

r

r √ n π nπ · = . 2r 2 8r

(10.31)

In the second transition, we replaced the upper limit ns by ∞; the reason is that this integrand, behaving like the summand we examined above, has vanishing contribution beyond

10.2. THE CRITICAL RANGE METHOD

567

ns . Hence, changing the limit does not change the result, but allows us to use the fact that the density integrates to 1. The relative error introduced by extending the range of the sum is exponentially small in n, and is negligible in comparison with the relative error we have been carrying all along, O(n−1 ), as seen in Eqs. (10.28) and (10.30). But what about replacing the sum by an integral? In §10.4, we present the tool, the Euler summation formula, which can answer just such questions. Exercise 10.74, page 589, asks you to carry out the computation, which shows that we need to add 1/2 to the value given in Eq. (10.31). Note that omitting this half would have contributed a relative error of O(n−0.5 ), far larger than any we incurred so far! Hence, we finally obtain r r n 1 π n 1/2 n 1 + O n−1 . ∑ k = n/2 2 + 8r k=0 n/2

(10.32)

The relative error term is the one we used in the Stirling formula: all other terms contribute much smaller errors. n

r

16 64 256 1024 4096 16384 65536

0.5

1

1.75

6.5

25

100

−.00134470 −.00006100 −.00000374 −.00000024 −.00000001 −.00000000 −.00000000

.01295039 .00354617 .00092976 .00023817 .00006028 .00001516 .00000380

.01754972 .00492026 .00130802 .00033759 .00008578 .00002162 .00000543

.01888521 .00572733 .00159774 .00042365 .00010920 .00002773 .00000699

.04874049 .00509321 .00153027 .00042533 .00011260 .00002900 .00000736

.24934293 .04279589 .00129960 .00038876 .00010799 .00002858 .00000736

A numerical illustration of the relative error in the calculation in Eq. (10.32) The table above shows how well (and otherwise) this calculation performs, when viewed as an approximation. It shows the relative error of the estimate given in Eq. (10.32) for each of the indicated parameter pairs, (n, r), and the table supports the claim that the error is in O(n−1 ). The error generally increases with r; this is not reflected in the calculation. The dependence is probably complicated, but some information can be gleaned from the table: in most of the columns the error decreases proportionally to n−1 , except the first column! In that column, where r = 1/2, the relative error behaves like O(n−2 ). This tells us that the coefficient of the term n−1 in the asymptotic expansion is proportional to r − 1/2 (but the dependence almost disappears beyond the third column). Exercise 10.43 Show how the entire treatment above holds for odd n as well (except that we need to replace n/2 by ⌊n/2⌋ = (n − 1)/2 in the binomial coefficient). Exercise 10.44 [1+] Can you apply the result of Exercise 10.18(b) directly to the sum in Eq. (10.27)? What do you expect to happen if you do, and why?


568

Figure 566 attempts to show the form of the function nx , for n = 100, to illustrate the claims about the steep parts at the sides and the relatively flatter ‘shoulder’ near the center. In fact, in order for the diagram to show any detail it is substantially “squashed,” and the actual values 0.2 presented are 100 . In addition, to motivate further the selection of the value s above, x √ √ two points are added, at x = n/2 − 0.5 n and x = n/2 − 1.5 n. Note the ratio between the leading term in the result (10.32) and the leading term of the sum, at j = 0. It is large, having the value (π n/8r)1/2 . How to explainthis, in view of the aimed ‘steepness” of the function? The reason is that the function, nx , is not steep at all near this maximal term, and changes very slowly as formula (10.11) on page 554 shows. For λ > 1/2, the maximal term, at k = n/2, occurs well within the range of the summation. Everything proceeds as previously, except that we have to use the integral on both sides of the origin j = 0. For the same reason the sum on one side above was completed for half the real line—now it would be allowed to cover the entire real line. The final result is then just double the value given in Eq. (10.32), independently of the precise value of λ (to see why such a blank statement makes sense, let a = λ − 1/2, and use the fact that for large n, (1/2 + ε )n > n/2 + ans , with the s used previously in Eq. (10.31), for arbitrarily small ε > 0). Now look at λ < 1/2. The maximum term falls again at the summation boundary, at the last term: k0 = ⌊λ n⌋. This, however, is not on the ‘shoulder’ of the curve of the function nk, as before, but in the region where the function is steep indeed, and the maximal term, kn0 , dominates the sum. Let j = k0 − k, and again we have a partial sum of binomial coefficients, for which there is no closed form, so we need to develop an alternative. It will prove to be much simpler than before, when we needed to sum binomial coefficients close to n/2. Now, t n when t 2 is very small using Eq. (10.11) on page 554, which states that k+t ∼ nk n−k k compared to both k and n − k, we can write j n n λ ∼ . α− j α 1−λ Then the sum on j reduces to a geometric series: r n ∑ k ∼ k=0 λn

r r r j 1 λ n n . = ∑ α 1−λ ⌊λ n⌋ 1 − [λ /(1 − λ )]r j>0

(10.33)

Exercise 10.45 [4] For each of the following sums, determine whether it is amenable to the “critical range method.” If it is, perform the summation; if it is not, explain why that is the case. n/2 r n (a) ∑ ak . Discuss separately the cases of a > 1 and a < 1. k k=0 n

n

(b)

(d)

(k + n)4 ∑ 2n − k . k=0 n/4

∑

k=0

n sin( 4πn k ). k

(c)

∑ k!. k=1

Compute to relative order O n−5 .

Finding here the maximal point needs care; you may want to

10.3. RICE’S METHOD

569

compute some numerical examples. Note that k = n/4 is an inflection point for the binomial coefficients. n/2 n n n 4π k ). (f) (e) ∑ sin( n ∑ k sin k. k k=0 k=0 (g) Example 3.69, page 108, derives a variant sum related to the birthday problem: µ (n, m) = n−1 r 1 n−r . Consider separately different relations of n and m, such as (i) n is fixed mn ∑r=1 m (m − r) √ (and large), (ii) n = α m, for some fixed 0 < α < 1, (iii) n = m. Exercise 10.46 [3] The answer to Exercise 3.24(b), page 92, is a sum for which we have no n+k−1 k p , where p + q = 1. Find an estimate of Pr[Am,n ] for closed form, Pr[Am,n ] = qn ∑m−1 k=0 k large fixed n, as m → ∞.

Exercise 10.47 [3] Let p be the probability of success of a Bernoulli random variable, and let2 m be the number of successes in n independent trials. Show that Pr mn − p > ε < 2 e−nε if a positive number ε is less than the smaller of p and q = 1 − p.

Exercise 10.48 [3+] (J. Griggs) Let P(n, m) denote the probability that in a group of randomly chosen n people no two of them share the same birthday of a year with m days. (a) For C > 1, show that P(n,Cn) → 0 as n → ∞. (b) For C > 0, show that P(n,Cn2 ) → e−1/2C as n → ∞. (c) For C > 0, find limn→∞ P(n,Cnα ), where 1 < α is a real number.

10.3 Rice’s Method There are known many combinatorial problems that lead to sums with alternating terms. In particular, all problems treated by the Principle of Inclusion and Exclusion in §8.3 involve such sums. Direct numerical evaluation of sums with alternating coefficients may be challenging because of subtraction operation involving large numbers that may result in losing significance. One of the ways to avoid subtractions is to reduce the sum to a recurrence. However, as it was shown in Example 7.20 (page 366), this recurrence could be a full-history one, which again cannot be considered a numerically-friendly road to efficient calculation. Our goal in this section is to find approximations of the following sums def

Sn ( f ) =

∑ (−1)k k>0

n fk . k

(10.34)

We met this sum in §8.3.1 when we used the PIE for homogeneous properties. The sum can be viewed as the nth order difference of f0 due to Eq. (5.81): Sn ( f ) = (−1)n ∆n f0 . This yields elegant relationship (have known as Euler transformation, see Exercise 6.58 on page 294) 1 F S(z) = 1−z

−z , 1−z

ˆ = ez F(−z) ˆ S(z)


570

between the generating functions S(z) =

∑ Sn ( f )zn ,

n>0

ˆ = S(z)

∑ Sn ( f )

n>0

zn , n!

F(z) =

∑

fn zn ,

ˆ = F(z)

n>0

∑

n>0

fn

zn . n!

Instead of using generating functions, we apply another technique attributed to S. O. Rice [113] (see also [45]). The idea of Rice’s method is to treat the coefficients fk in the sum (10.34) as residues of a suitable complex-valued function f (z), defined in appropriate domain containing integer points {0, 1, . . . , n} under summation. The Cauchy residue theorem 12.57 (page 682) is often used to convert a line integral in the complex plane to such a sum. Here we turn the tables: for a given series, we shall find the line integral and the contour, and then use an alternative approach to compute the integral. First, we need to find an analytic function having residues at positive integer values equal to n the binomial coefficients, k . Our candidate for the role is the beta function (see §12.6) B(−z, n + 1) =

Γ(−z) n! Γ(−z) Γ(n + 1) = . Γ(n − z + 1) Γ(n − z + 1)

Using the gamma function recurrence Γ(z + 1) = z Γ(z), we rewrite the above expression as (z is a complex number) B(−z, n + 1) =

Γ(−z) n! Γ(−z) n! Γ(−z) n! = = ··· = . Γ(n − z + 1) (n − z) Γ(n − z) (n − z) (n − z − 1) · · · (−z) Γ(−z)

Hence, we get B(−z, n + 1) =

n! Γ(−z) n! = , Γ(n − z + 1) (n − z) (n − z − 1) · · · (1 − z)(−z)

(10.35)

which is valid for complex values z ∈ C except integers 0, 1, . . . , n. Evaluating residues at these points, [0..n], we obtain n! Γ(−z) n! k n = Res Γ(−z) = (−1) , (10.36) Res B(−z, n + 1) = Res z=k z=k Γ(n − z + 1) (n − k)! z=k k k

because Resz=k Γ(−z) = Resz=−k Γ(z) = (−1) k! . Now we outline the connection between sums (10.34) and the line integrals in the following lemma, which formalizes this observation. Lemma 10.49 Let f (z) be a regular function is a domain D containing integers {a, a + 1, . . . , n}, and no other integers. Then I n 1 k n (10.37) ∑ (−1) k fk = − 2π i ∂ D f (z) B(−z, n + 1) dz, k=a

where B(−z, n + 1) is given in Eq. (10.35) and ∂ D is the boundary (oriented in counterclockwise direction) of the domain D ∋ [a..n], which is assumed to be a closed Jordan curve.


571

P ROOF : We start with the right-hand side integral, assuming that the function f (z) is analytic in some domain D that includes the integer points [a..n]. Using the Cauchy residue theorem 12.57, we calculate the integral as sum over all simple poles inside ∂ D: 1 − 2π i

Z

∂D

f (z) B(−z, n + 1) dz = −

n 1 n! f (z) × 2π i ∑ Res 2π i k=a z=k (n − z)(n − 1 − z) · · · (−z)

Since every pole z = k is simple, we calculate the residue as follows: Res z=k

n! f (z) n! f (z) = − lim z→k (n − z) · · · (k + 1 − z)(k − 1 − z) · · · (−z) (n − z)(n − 1 − z) · · · (−z) n! n! f (k) = (−1)k+1 f (k) =− (n − k) · · · 1 (−1) · · · (−k) k! (n − k)! H

because the denominator in B(−z, n + 1) has the factor (k − z). Recall that g(z) dz/(k − z) = −2π i Resz=k g(z).

To calculate the integral over closed curve, one may use its property not to change its value when the line of integration is smoothly changed without crossing singular points (recall that the beta function has simple poles at negative integers). However, when the curve of integration embraces a new singular point, we need to subtract the residue at this point to compensate the original value. Stretching the boundary of the domain D to infinity usually affects the original integral in two ways: one needs to evaluate the integral over infinite line and subtract residues over all singular points that were crossed during this stretching. The additional residues usually give the main contribution to the asymptotic estimate of the sum (10.34) when the line integral over stretched line has a small value. By enlarging the original contour, we get Z n 1 k n ∑ (−1) k fk = 2π i ∂ Dl f (z) B(−z, n + 1) dz + ∑ Res f (z) B(−z, n + 1), (10.38) k=a z∈(D \D) l

where the boundary, ∂ Dl , of the larger domain Dl is a closed contour oriented in clockwise direction, and summation is over all singular points of f (z) that are inside Dl , but outside D. If the contribution of the line integral over ∂ Dl is small to compare with the sum of residues, Eq. (10.38) gives an asymptotic estimate of the sum (10.34). Exercise 10.50 [1+] Using Eq. (10.38), prove the formula (10.36) without actual evaluation of residues. We illustrate the Rice method in the following two examples. Example 10.51 Series like the following one arise in several communications protocol evaluations: n n (−1)k n k n fk , (10.39) = ∑ (−1) Sn = ∑ −k k k k=3 k=3 1 − 3 where fk = (1 − 3−k )−1 = 3k /(3k − 1), k = 3, 4, . . . , n, are the values of the analytic function f (z) = 3z /(3z − 1) at the integer points k = 3, 4, . . . , n. We take a closed contour of


572

integration to be the boundary of the rectangle ABCD with corners A C n + 12 − iω , and D n + 21 + iω .

5 2

+ iω , B

5 2

− iω ,

Then we enlarge the original contour into A′ B′C′ D′ , which embraces a new singular point z = 2. Hence I B(−z, n + 1) 1 B(−z, n + 1) = − dz. Sn + Res −z z=2 1−3 2π i A′ B′C′ D′ 1 − 3−z Since z = 2 is a simple pole, we have 1 B(−z, n + 1) × Res B(−z, n + 1) = Res x=2 1 − 3−z 1 − 3−z z=2 z=2 Γ(−z) Γ(n + 1) 9 1 × Res = n(n − 1) Res Γ(−z), = −2 z=2 z=2 1−3 Γ(n − 1) 8

because B(−z, n + 1) = Γ(−z) Γ(n + 1)/Γ(n + 1 − z) and Γ(n + 1) = n(n − 1)Γ(n − 1). Since k the gamma function has residue at z = −k (k = 1, 2, . . .) to be Resz=−k Γ(z) = (−1) k! , we find 1 9 9n(n − 1) B(−z, n + 1) 2 n . = Res B(−z, n + 1) = (−1) = Res −z −2 z=2 1−3 1 − 3 z=2 8 16 2 ℑ

A′′

D′

A′ A

4π i ln 3 2π i ln 3

− 2lnπ3i

D′′

D ℜ

0

1

2

3

4

5

6

.

.

.

B

− 6lnπ3i

n

n+1

C

B′

C′

B′′

C′′

Figure 572: Contours of integration. Shifting the left edge of the contour one more unit to include the pole at z = 1, so it passes through the point z = 1/2, we add to the contour integral the value Res z=1

1 3n B(−z, n + 1) = Res B(−z, n + 1) = − , −z −1 1−3 1 − 3 z=1 2


573

which is of lower order than the previous residue at z = 2, and will be seen to represent the next term of the asymptotic expansion of the sum Sn . Now we can express it as Sn = −

1 9n(n − 1) 3n + − 16 2 2π i

I

A′′ B′′C′′ D′′

B(−z, n + 1) dz. 1 − 3−z

Now we deform the closed loop A′′ B′′C′′ D′′ into another one by moving the vertical line (1/2 − iM, 1/2 + iM) one unit left and stretching the rest into a semicircle. So our new contour of integration consists of the straight vertical line An Bn and semicircle of radius M connected these two points (see Figure 574). Such deformation crosses some singular points of the function f (z)B(−z, n + 1). From Eq. (10.35), we see that the beta function contributes with the factor 1/(−z), at z = 0. However, f (z) is singular wherever 3−z = 1, and since we can write 1 as e2iπ k , for all k ∈ Z, this happens along the imaginary axis, at all the points z = 2iπ k/ ln 3, as seen in Figure 572:

3−z = 1 = e2π ik

(k = 0, ±1, ±2, . . .)

=⇒

zk = 2π ik/ ln 3.

All these zk are simple poles of the integrand, but z = 0 is a double pole because the gamma function also has the pole at the origin. Each simple pole zk , k 6= 0, contributes the additional term: n! B(−z, n + 1) B(−zk , n + 1) B(−zk , n + 1) B(−z, n + 1) = ∈ O = = Res z=zk 1 − 3−z 3−z ln 3 z=zk 3−zk ln 3 ln 3 nn+1 since 3−zk = 1. This relation suggests that the residues at z = zk , k 6= 0, decrease exponentially as n → ∞. The exact values are quite hard to characterize beyond that; the real and imaginary parts are quite close (rarely differing by more than a factor of 3–4); their small size may appear quite surprising. The residues at zk and z−k are complex conjugates, the imaginary parts cancel out; in addition, the evident very fast decline of the residues with k, for any fixed n, suggests strongly that the very few poles, which are closest to the real line produce the bulk of the (however small) contribution. This sum oscillates, as n increases, between positive and negative values, with the number of n values between successive changes in sign nearly doubling at each repeat. The decay of the amplitude of this oscillation is quite slow.

The accompanying table (see Figure 574) shows the first 13 extrema of an = ∑nk=−n Resz=zk [ f (z)B(−z, n + 1)] as these values oscillate between negative and positive values, and appear to approach a limit. Calculation of the residue at z = 0 requires some work. Since we need to find the coefficient of z−1 in the Laurent expansion of the integrand, we write first the Maclaurin series of expo2 nential term: 1 − 3−z = z ln 3 − z2 ln2 3 + O(z3 ). Then we expand the beta function around z = 0 as follows: B(−z, n + 1) =

n! Γ(−z) Γ(n + 1) = Γ(−z) , ′ Γ(−z + n + 1) n! − z Γ (n + 1) + O(z2 )


574

where Γ′ (n + 1) is the derivative of the gamma function at the point z = n + 1. From Eq. (G.30), we find that Γ′ (n + 1) = Ψ(n + 1) Γ(n + 1) = n! (Hn − γ ), where Hn is nth harmonic number and γ is the Euler constant. Hence, we get B(−z, n + 1) = Γ(−z)

1 2 = Γ(−z) 1 + z (H − γ ) + O(z ) . n 1 − z (Hn − γ ) + O(z2 )

We obtain the Laurent series for Γ(−z) by expanding Γ(−z) + 1/z = (1 + z Γ(−z)) /z into the Maclaurin series: 2 π γ2 1 + + ··· . Γ(−z) = − − γ − z z 12 2 Multiplication of series for B(−z, n + 1) and f (z) and extraction of the coefficient of z−1 yields 1 Hn B(−z, n + 1) . =− − Res z=0 1 − 3−z 2 ln 3 ℑ

Bn

n

An

2 3 8 15 27 48 84 147 256 444 769 1333 2309

ℜ

0

n

n+1

Figure 574: (a) New contours of integration. n Res [ f (z)B(−z, n + 1)] ∑k=−n z=zk

an −9.621094 × 10−3 5.290547 × 10−3 −1.396605 × 10−3 6.550908 × 10−4 −4.261746 × 10−4 3.336226 × 10−4 −2.898224 × 10−4 2.673021 × 10−4 −2.551285 × 10−4 2.483027 × 10−4 −2.445095 × 10−4 2.422877 × 10−4 −2.410692 × 10−4 (b)

The values of an =

It becomes simple arithmetic to add all terms together to obtain Sn ∼ −

9n2 33 n Hn 1 + − − + O(1) as n → ∞. 16 16 ln 3 2


575

Finally, we estimate the remaining term, the contour integral. When |z| is large enough, the beta function (10.35) can be estimated as n! 6 K n! (10.40) |B(−z, n + 1)| = (n − z) (n − z − 1) · · · (−z) |z|n+1

for some positive constant K. This allows us to conclude that the integral in Eq. (10.38) over semi-circle approaches zero as its radius goes to infinity. Therefore, the main contribution to the error gives the integral over the vertical interval, An Bn . For z ∈ An Bn , we have −z = d + iy, where d is the distance (which can be considered as a constant) to the imaginary line. So we need to estimate B(d + iy, n + 1) as y → ±∞.

Factoring out n!, we see that B(d + iy, n + 1) = n!Γ(d + iy)/Γ(n + 1 + d + iy). For any fixed real x, the gamma function Γ(x + iy) is in O(|y|x−1/2 e−π |y|/2 ) as |y| → ∞. There Γ(d+iy) fore, Γ(n+1+d+iy) ∈ O(|y|−n−1 ), for and all |y| larger than some y0 . This guarantees that the integral over vertical interval An Bn converges to some number as its ends go to infinity, for which no useful approximation is available. Note that moving the vertical edge An Bn to the left further, no new singularities are introduced.

Exercise 10.52 [2] In Example 7.92 on page 407, we obtained an expression for a collision resolution interval duration quite similar to the series the previous example deals with: ln = k (k−1) , where p + q = 1 (0 < p < 1). Show that ln grows linearly as n → ∞. 1 + 2 ∑nk=2 nk (−1) 1−pk −qk

Example 10.53 (adapted from [57]) Consider a biased random generator that produces binary numbers, say 1 with probability p (0 < p < 1) and 0 with probability q = 1 − p. Suppose that for some reasons the generator produces digits one at a time until two sequential 1’s come out. The sequences produced by such generator form words in a regular language generated by the regular expressions {0, 10}∗ 11.

In Example 5.39, page 213, we obtained the recurrence for the probabilities pn that a word in the language is of length n: pn = qpn−1 + pqpn−2 , for n > 3, with the initial values p0 = p1 = 0 and p2 = p2 . This is readily converted to the PGF:

ϕ (z) =

p2 z2 , 1 − qz − qpz2

where the probability to see a word of length n is pn = [zn ]ϕ (z). Factoring the denominator in ϕ (z), we get p p 2 + 4pq − q q + q q2 + 4pq def def (z) = 1 − qz − qpz2 1 ak (a + p) + (−b)k (b − p) . Pr[X > k] = [zk ]ϕ> (z) = p q2 + 4pq


576

Let X1 , X2 , . . . , Xn be a sequence of n such words, generated by the biased random number generator, and let Xmax = max {X1 , X2 , . . . , Xn } be the maximal word length produced. Then using Eq. (3.18) on page 106, its expected value becomes ! # " n 1 def ak (a + p) + (−b)k (b − p) En = E[Xmax ] = ∑ 1 − 1 − p 2 + 4pq q k>0 m n m−1 n k k = ∑ (−1) , (10.41) A a − B (−b) ∑ m k>0 m=1 where a2 a+ p = p A= p q2 + 4pq q q2 + 4pq

b2 = p . q2 + 4pq q q2 + 4pq √ For example, if p = q = 1/2 (an unbiased generator), we √ have a = (1 + 5)/4 ≈ 0.809, √ √ b = ( 5 − 1)/4 ≈ 0.309, A = (5 + 3 5)/10 ≈ 1.17, B = (3 5 − 5)/10 ≈ 0.17. and

B = −p

b− p

Our objective is to find an asymptotic formula for En as n → ∞. Surely, we are going to use Rice’s method and express En as a contour integral in the complex plane En = −

1 2π i

where f (z) =

∑ k>0

I

γ

(−1)n n! f (z) dz, z(z − 1) · · · (z − n)

A ak − B (−b)k

z

,

ℜz > 0,

(10.42)

(10.43)

and γ is a smooth Jordan loop encircling the points 1, 2, . . . , n (and no other integer points), oriented counterclockwise. To simplify further calculations, we introduce the following notation: p b2 q + 2p − q2 + 4pq B b2 def b = ∈ (0, 1), =⇒ = = d2. d= = a pq 2p A a2 This allows us to rewrite f (z) in the following form: z j z 2 k z kz = A ∑ a ∑ (−1) f (z) = A ∑ a 1 − d (−d) (−d)k j+2 j j j>0 k>0 k>0 z (−1) j d 2 j j z z 2j kz kj z , = A ∑ (−1) d ∑ a (−d) = A ∑ z j j j>0 j>0 j 1 − a (−d) k>0 z

kz

(10.44)

where uniform convergence for ℜz > 0 justifies the exchange of summations. The singular points of the function f (z) are found by equating the denominator to zero and finding roots of 1 − az (−d) j = 0, which yields z j,k = − j

ln d 2π ki + , ln a ln a

k = 0, ±1, ±2, . . . ;

j = 0, 1, 2, . . . ,

(10.45)


577

where i is the unit vector in vertical direction (along the imaginary axis), i2 = −1. Since both a and d are in the interval (0, 1), their logarithms are negative, and for each fixed j all the singular points defined in Eq. (10.45) are located on the vertical line with ℜz = − jw, where def w = ln d/ ln a > 0. (Note that w need not be small; it is 4.511 for p = 0.5 and around 46,600 for p = 0.01). Therefore, the series in Eq. (10.44) converges in the entire plane, except the isolated singularities (10.45), and is the analytic continuation of f (z) into the left half plane. These singularities then pepper the entire left half of the complex plane; we shall be able, however, to make do with the singularities z0,k only. The fact that this set is the only one with a singularity on the imaginary line turns out to be significant. Now we transform the contour of integration into another one that encircles these singular points (it is similar to one in Figure 574). Initially, the contour is just a counterclockwise loop around the points z ∈ [1..n]. We open it to a half circle on the straight L1 , parallel to the imaginary line through z = 1/2 (similar to A′′ B′′ of Figure 572). Now L1 is moved just left of the imaginary line, to ℜz = −ε , where 0 < ε < w, and then we only need to account for the singularities from the set {z0,k }k∈Z only. We show first that the line integral along the half circle z = Meiθ for −π /2 6 θ 6 π /2 must vanish.

The value of B(−z, n + 1) on such semicircles was bounded in Eq. (10.40). How does f (z) behave there? Using the initial form in Eq. (10.43), we observe that the first two terms, k = 0, 1, are 1’s (since A − B = 1 and Aa + Bb = 1); let us then write f (z) = 2 + ∑k>1 (Aak − B(−b)k )z . All this terms, for p > 0, are positive and strictly less than 1, hence for z with positive, large real part, the sum for f (z) converges to some finite number, and the bound on the beta function there is enough to make it arbitrarily small (as M → ∞).

In order to be able to transform the contour of integration, we need an estimate z j for f (z) along z j the vertical line ℜz = −ε . For any such z, we have 1 − a (−d) > 1 − a d = 1 − a−ε d j , and by choosing ε < min(1, w), we can keep this value close to 1 even for small j (such as 0). Therefore, along the line ℜz = −ε , we have an estimate z 1 z (−1) j d 2 j z z j 2 j (−1) d = | f (z)| = A ∑ 6 A j>0 j 1 − az (−d) j ∑ |A(1 − d 2 )|ε j>0 j that leads to a conclusion that the integral (10.42) converges along the line ℜz = −ε : Z −ε +iM 1 Z −ε +iM (−1)n n! f (z) dz dz n! −ε 6 2π i −ε −iM z(z − 1) · · · (z − n) 2π |A|ε |1 − d 2 |ε −ε −iM |z(z − 1) · · · (z − n)| ∈ O(n ).

Careful numerical experimentation has shown very conclusively that the values become promptly extremely small for quite moderate n and M, but as p gets to be small, this is no R −0.5+iM longer true. For p = 0.5, the value of the integral −0.5−iM B(−z, n + 1) f (z)dz came to 10−10 for n = M = 25. The computations do not suggest anything clearer than a value in O(n−ε ) for some positive ε , and no better or more detailed bound seems evident. Now we deform the contour of integration into another closed loop that consists of the vertical straight line and enclosed it in a semi-circle of radius M, which embraces a finite number of


578

singular points (10.45) and integers 0, 1, 2, . . .. The integral over semi-circle can be made less than any positive number for large R. Hence the entire loop only contributes the integral on the vertical line, which we denote by In (ε ). The original sum, En , equals then En = In (ε ) + ∑ Res k∈Z

z0,k

(−1)n n! f (z) , z(z − 1) · · · (z − n)

n

R

+i∞ (−1) n! f (z) where In (ε ) = 21π i −−εε−i∞ z(z−1)···(z−n) dz and summation includes all poles that are not counted by the original sum. All the poles, except the one at the origin, are simple, and z0,0 is a second order pole. (−1) j d 2 j Consider first the simple ones. For f (z) = Az ∑ j>0 zj 1−a z (−d) j , only the j = 0 term is singular at z0,k , and the rest of the sum has a zero residue. Hence def

z − z0,k Az0,k . = − z→z0,k 1 − az ln a

Res f (z) = Az0,k lim z0,k

Note that |Az0,k | = 1 for any of these purely imaginary points. The beta function is regular at z0,k = 2π ki/ ln a when k 6= 0, and can be estimated, for large n using Eq. (G.22) as n! Γ(−z0.k ) Γ(n+1)(1+O(z/n) 2 ) . From Eq. (G.10) on page 725, we find for it the absolute value 1/2 2 close to 1k ln ae−2π k/ ln a , which also means that nearly the entire contribution is obtained for k = ±1, and even then is extremely small.

Hence, these simple poles make no material contribution to evaluating the series. The authors of [57] observe its smallness, and that it is an oscillatory function of log n, but no specific bound seems easy to formulate.

It is at the double pole, z = 0, where we finally hit pay-dirt, that is, meaningful estimates. In a neighborhood of the origin, the main contribution to the function f (z) makes the first term in (−1) j d 2 j Az Az +g(z) = +g(z), where g(z) = Az ∑ j>1 zj 1−a the sum (10.44): f (z) = z (−d) j z −z 1−a 1 − (1/a) is a regular function since g(0) = 0. Hence def

R0 = Res z=0

(−1)n n! Az (−1)n n! f (z) = Res . z(z − 1) · · · (z − n) z=0 (1 − az ) z(z − 1) · · · (z − n)

To find R0 , we expand the integrand into the Laurent series. Multiplying the series i 1 h 1 + z ln A + O(z2 ) z Az −1 = = ln a + · · · , 1 + z ln A + 1 − az z ln a−1 − z2 ln2 a−1 + O(z3 ) z ln a−1 2 2 1 1 1 z z2 1 = − = − k = 1, 2, . . . , n, 1+ + 2 + ··· , z−k k 1 − z/k k k k and extracting the coefficient of z−1 , we obtain R0 =

1 ln A Hn − − , 2 ln a ln a

(10.46)

10.4. THE EULER SUMMATION FORMULA

579

where Hn = 1 + 1/2 + · · · + 1/n in the nth harmonic number. Adding all terms together, we obtain the asymptotic expansion of the expected value to be En = −

Hn 1 ln A + − + O(n−ε ). ln a 2 ln a

The following table shows the values of En , computed from Eq. (10.41), and δn , the difference between En and the value of R0 , computed from Eq. (10.46); the calculations are exact to within the size of the given error. The dependence of the quality of this approximation on p is notable, and clearly related to the quality of our approximating In (ε ) by O(n−ε ). p

0.2 (w = 50.483715) En δn 10 85.23910 −8.480295 10−15 100 149.9182 2.224152 10−56 1000 215.7338 7.743354 10−112 10000 281.6652 3.679386 10−122

n

0.5 (w = 4.54112) En δn 15.06422 −1.884814 10−6 25.72034 −4.998023 10−14 36.56371 2.821361 10−18 47.42615 −2.559275 10−20

0.8 (w = 0.7400044) En δn 5.572457 −1.504484 10−3 8.950219 −4.278568 10−4 12.38664 −2.724793 10−5 15.82874 8.929479 10−6

Exercise 10.54 [4] This is a generalization of the previous example. Suppose that a biased random generator terminates on k 1’s in a row. Let X1 , X2 , . . . , Xn be a sequence of n applications of this generator, and let Xmax = max {X1 , X2 , . . . , Xn } be the maximal word length produced. Find the first two terms in the asymptotic expansion of the expected value def En,k = E[Xmax ]. Exercise 10.55 [2] (The Coupon-collector problem) In applying the PIE to the coupon-collector problem, we derived Eq. (8.69) for the probability of completing the entire collection of n coupon types in m drawings: n m def k n (n − k) , (10.47) pm,n = Pr[X = 0] = ∑ (−1) nm k k=0 where X is the number of coupons still missing (mλ = n and 0 < λ < 1, naturally). How would you find an approximation of pm,n as n → ∞? Explain why the Rice method is ineffectual for this estimate.

Exercise 10.56 [3] Asymptotically approximate the sums n n (−1)k n (−1)k n , (b) ∑ ak , (a) ∑ k k=2 e + 1 k k=2 3 − 1 k

a is a constant.

10.4 The Euler Summation Formula In most first-year courses of calculus, the definite integral is presented as the limit of its Riemann sums. In fact, a simple geometrical argument shows that the sum ∑n−1 k=1 f (k) is a Rn lower bound of the integral 1 f (x)dx if f increases monotonically or an upper bound if f decreases monotonically. When a function is not monotone, it is not obvious what the relation


580

between the sum and the integral may be. Our current interest is to utilize the fact that we have many more closed-form formulas for integrals than for sums. The following formula is referred to as the Euler summation formula (ESF). It is called after Leonhard Euler4 , and gives an explicit way to compute the difference between a sum and the corresponding integral. Since the proof does not shed much light that could be helpful in applying it, we do not include it (possible references are [7], [84], or [135]). Nevertheless, a formal proof of the ESF is given in Example 12.78, page 690, to demonstrate the operator method. Theorem 10.57 [Euler Summation Formula] Let m be a positive (fixed) integer, and f (x) be R a smooth function (having continuous derivatives up to 2m) such that the integral 1n f (x) dx exists for each integer n > 2. Then n

∑

f (k) =

k=1

Z n 1

−

i m B2 j h (2 j−1) 1 f (n) − f (2 j−1) (1) f (x) dx + [ f (n) + f (1)] + ∑ 2 j=1 (2 j)!

1 (2m)!

Z n 1

B2m ({x}) f (2m) (x) dx,

(10.48)

where f (2 j−1) (x) is the derivative of the function of the order 2 j − 1 at the point x, Bk (x) are the Bernoulli polynomials, Bk = Bk (0) are the Bernoulli numbers (see Appendix B), and the notation { x } = x − ⌊x⌋ is used for the fractional part of a real number x. The function B2m ({x}) is also called the periodic Bernoulli polynomial. The term of the second line of Eq. (10.48) is called the error term. When m = 0, we get: Z n Z n n 1 1 ∑ f (k) = 1 f (x) dx + 2 [ f (n) + f (1)] + 1 x − ⌊x⌋ − 2 f ′ (x) dx. k=1 Remark: The formula (10.48) does not necessarily provide better estimates as m increases. There is usually an optimal value of m, but we know of no general rule to find it: we must examine each particular case on its own. There is nothing special about the lower bound of the summation index being 1; if we have at hand another lower value, we can replace the 1 by it throughout. For example, n

∑ f (k) −

k=a

Z n

f (x) dx =

a

where the error term is

1 1 [ f (n) + f (a)] + [ f ′ (n) − f ′ (a)] + ρ ( f , n, a), 2 12

1 ρ ( f , n, a) 6 120

Z n a

(10.49)

| f ′′′ (x)| dx.

The Euler summation formula itself is not an asymptotic. It will become so only if the functions f (2 j−1) (n), j = 1, 2, . . ., form an asymptotic scale, and if the remainder integral 4 L. Euler was the first to publish the formula in 1732. The idea is sometimes also associated with Colin Maclaurin (1698 – 1746) whose result was published later.

10.4. THE EULER SUMMATION FORMULA k

Bk (x)

Bk

0 1 1 − 21 1 2 6 3 0 1 4 − 30 5 0 1 6 42 7 0 1 8 − 30 9 0 5 10 66

581 k

1 x − 21 x2 − x + 61 x3 − 23 x2 + 21 x 1 x4 − 2x3 + x2 − 30 x5 − 25 x4 + 35 x3 − 61 x 1 x6 − 3x5 + 25 x4 − 21 x2 + 42 x7 − 27 x6 + 27 x5 − 67 x3 + 6x 7 4 1 6 2 x8 − 4x7 + 14 3 x − 3 x + 24 x − 30 3 5 3 x9 − 29 x8 + 6 x7 − 21 5 x + 72 x − 10 x 3 2 5 8 6 4 x10 − 5 x9 + 15 2 x − 7 x + 180 x − 2 x + 66

Bk

11 0 −691 12 2730 13 0 7 14 6 15 0 3617 16 − 510 17 0 43867 18 798 19 0 174611 20 − 330 21 0

Table 581: Values for the Bernoulli numbers Bk and Bernoulli polynomials Bk (x). tends to zero. L. Euler did not discover the error term for this formula. The first to do so was S. D. Poisson [108] in 1823. Using integration by parts (see Exercise 10.58), the remainder integral can also be written in odd-order derivative version: −

1 (2m)!

Z n 1

B2m ({x}) f (2m) (x) dx =

1 (2m + 1)!

Z n 1

B2m+1 ({x}) f (2m+1) (x) dx.

(10.50)

Some of the Bernoulli numbers and polynomials needed in Eq. (10.48) are given in Table 581. Additional values may be found in the book by M. Abramowitz & I. Stegun [1] or determined from their exponential generating functions (L.32) and (L.33). Other properties of these numbers and polynomials are given in Appendix B. The Bernoulli numbers are usually calculated n−1 n+1 1 n = 2, 3, . . .. from the recurrence Bn = − n+1 ∑ j=0 j B j ,

The Bernoulli polynomial Bn (x) of degree n is the unique polynomial solution of the integral R1 equation 0 Bn (x + t) dt = xn . They are binomial convolutions of the Bernoulli numbers and the power functions, Eq. (B.2). The Bernoulli numbers Bk are zeroes for odd indices k > 3, and those of even index are related to the Riemann zeta function ζ (2m) = ∑k>1 k−2m (see Eq. (B.16), page 707):

B2m = (−1)m+1

2 × (2m)! 2 × (2m)! ζ (2m) = (−1)m+1 2m (2π ) (2π )2m

∑

j>1

1 j2m

,

(10.51)

which leads to the asymptotic relation: 2 1 B2m ∼ (−1)m+1 (2m)! (2π )2m

as

m→∞

(10.52)


582

because ζ (2m) ∼ 1 as m → ∞.

To estimate the error term (10.50), we need to know the extreme values of the Bernoulli polynomials Bk (t) in the unit interval 0 6 t 6 1 because {x} = x − ⌊x⌋ is the periodic function with period 1. Using formula (B.5), we get its derivatives: B′2m (t) = 2m B2m−1 (t),

B′2m+1 (t) = (2m + 1) B2m (t).

For m > 1, B2m−1 (t) has precisely three simple roots within the unit interval at 0, 1/2, and 1. Hence, the extreme values of B2m (t) are attained at these points and they have the following values: B2m (0) = B2m (1) = B2m = (−1)m−1 |B2m |, B2m 1/2 = (−1)m 1 − 21−2m |B2m |.

For odd index, there is no exact formula for the roots B2m (t) = 0. It is known [87] that |B2m (t)| 6 |B2m |

and

|B2m+1 (t)| 6 (2m + 1) |B2m |,

0 6 t 6 1.

(10.53)

Hence, the remainder in Euler’s summation formula has the following upper bound: Z n Z n (2m) 1 def −2m (2m) B2m (x − ⌊x⌋) f (x) dx ∈ O (2π ) (10.54) (x) dx. Rm = − f (2m)! 1 1

Exercise 10.58 [2] (a) Show that Eq. (10.50) holds. (b) Some authors write the ESF as follows: n−1

∑

j=1

f ( j) =

Z n 1

m

(−1)m Z n Bk (k−1) f (n) − f (k−1) (1) − Bm ({y}) f (m) (y)dy. k! m! 1 k=1

f (x)dx + ∑

Show that this is identical to the version given in Eq. (10.48), except in one detail; which one is it? Actually several generalizations of the approach that leads to the ESF has resulted in a number of other similar formulas with somewhat different properties (see [23, 54]). The following one, from [54], is especially interesting since the values it produces, together with Eq. (10.48), usually bracket the exact answer: Z n+1/2 n ∞ 1 1 (2 j−1) (2 j−1) −f , (10.55) a− n+ ∑ f (k) = a−1/2 f (x) dx − ∑ C j f 2 2 j=1 k=a where the coefficients C j ( j = 1, 2, . . .) are obtained either from the full-history recurrence Cj =

j−1 Ck 1 − ∑ 4 j−k (2 j − 2k + 1)! , 4 j (2 j + 1)! k=1

C1 = 1/24;

or explicitly, in terms of the Bernoulli polynomials or Bernoulli numbers: B2 j (1/2) −1 2 j 2 j k = j Cj = − ∑ k 2 Bk . (2 j)! 4 (2 j)! k=0

(10.56)

(10.57)

10.4. THE EULER SUMMATION FORMULA j Cj

583

1

2

3

4

5

1 24

−7 5760

31 967,680

−127 154,828,800

73 3,503,554,560

0.041667 −0.0012153 3.20354 × 10−5 −8.20261 × 10−7 2.08360 × 10−8

Let us use the Euler summation formulas in a few cases. Example 10.59 We begin with a familiar sum, for which we have a closed form value: n

∑ k2 = k=1

n(n + 1)(2n + 1) . 6

We apply the ESF and truncate it at m = 1 since the second derivative of the function x2 is a constant, and the integral of any Bk (x) over [0,1] vanishes. The right-hand side (RHS) of Eq. (10.48) in this case is RHS =

Z n

=

Z n

= =

1

1 1 Z n B2 ′ 1 2 ′ f (n) − f (1) + B2 ({x}) f (2) (x) dx x dx + (n + 1) + ∑ 2 2! 2! 1 j=1 2

1 1 x2 dx + (n2 + 1) + (2n − 2) + 0 2 12 1 x=n n − 1 n3 − 1 1 2 n−1 1 2 x3 = + (n + 1) + + (n + 1) + 3 x=1 2 6 3 2 6 n(n + 1)(2n + 1) n 2 (2n + 3n + 1) = , 6 6

since f (x) = x2 and f ′ (x) = 2x. In this case the ESF gives the exact value of the sum for m = 1. Using another formula (10.55), we obtain Z n+1/2 n 1 1 1 (n + 1/2)3 (1/2)3 1 1 2 2 2 − − n+ − ∑ k = 1/2 x dx − 24 n + 2 − 2 = 3 3 12 2 2 k=1 =

2n3 + 3n2 + n n(n + 1)(2n + 1) = , 6 6

which is the same expression. Example 10.60 Now we consider a more complicated example, summing f (k) = k2.5 . In this case we have no closed form, and the error term does not vanish as well, for any m. Let us again denote the right hand side of the ESF for f (x) = x2.5 by RHS. Then x=n Z n m B2 j 2.5 (2 j−1) 1 2.5 2.5 (x ) n +1 + ∑ RHS = x dx + + Rm . 2 1 j=1 (2 j)! x=1

We need the derivatives, f (x) = x5/2 ,

f ′ (x) =

5 3/2 x , 2

f ′′ (x) =

5 3 1/2 15 1/2 x = x , 22 4

f ′′′ (x) =

15 1 −1/2 15 x = √ . 4 2 8 x


584

Stopping at m = 2, and we find n5/2 + 1 1/6 5 1 15 1 2 7/2 3/2 √ − 1 + R2 n −1 + n −1 − + · · RHS = 7 2 2 2 30 · 4! 8 n 1 2 1 5 1 5 1 2 √ + − − + + R2 . = n7/2 + n5/2 + n3/2 − 7 2 24 384 n 2 7 24 384 5 1 23 + 384 = 2688 ≈ 0.0085565. At n = 10, the difference The value in parenthesis is 12 − 72 − 24 between the sum—approximately 1068.217558—and the RHS is about 0.000040; the relative error is then less than 4 × 10−8 .

Exercise 10.61 [1+] Repeat calculations from the previous example using the formula (10.55).

Exercise 10.62 [1+] Derive a five-term approximation of the sum:

∑nk=1 k3/4 .

def

Exercise 10.63 [3+] Consider the sum Φ p (n) = ∑nk=1 k p , with an integer p larger than 1. (a) To what order would you need to use the ESF to have a vanishing error term? (b) Obtain the following formulas n

1 ∑ k = p+1 k=0 p

p

∑

j=0

B p+1 (n + 1) − B p+1 p+1 . (n + 1) p− j+1 B j = j p+1

(10.58)

Is this a closed form? The right-most expression may be one, if we consider the Bernoulli polynomials “closed” (which we do, for example, when using MAPLE for computing since it has a primitive function that computes these polynomials). However, even the sum over j, while surely not “closed,” contains only about p/2 terms (the Bernoulli numbers are zeroes for odd indices), and is quite convenient in practice, for any n. Exercise 10.64 [1+] Using the ESF, find the next three terms in the asymptotic expansion def

Φ p (n) =

n

n p+1

∑ kp = p + 1 +

k=1

np + a n p−1 + b n p−3 + c n p−5 + · · · , 2

as n → ∞.

What would be an asymptotic formula for the sum ∑nk=m k p ? Exercise 10.65 [4−] Let Φ p (n) = ∑nk=1 k p , with p be a real number. Show that [Φ p (n + m)]1/(p+1) − [Φ p (n)]1/(p+1)

−→

n→∞

m(p + 1)−1/(p+1) .

Example 10.66 Let X denote a discrete uniform random variable with values in [0..n] = {0, 1, . . . , n}. Hence Pr[X = k] = 1/(n + 1), for k = 0, 1, 2, . . . , n. Let f be a bounded realvalued function in the unit interval [0, 1]. We want to estimate a finite sum ∑nj=0 f ( j/n) as n → ∞. Keeping only two first terms in the ESF, we obtain E[ f (X (n)] =

1 n+1

n

∑

j=0

f ( j/k) =

1 n+1

Z n 0

f (x/n) dx +

1 [ f (1) + f (0)] + R2 . 2(n + 1)


585

Changing the variable of integration, 1 n+1

Z n 0

Z 1

n f (x/n) dx = n+1

0

f (u) du =

n E[ f (U )], n+1

where U is the uniform distribution on the unit interval [0, 1]. Hence 1 1 ∑ f ( j/n) = n E[ f (U )] + 2 [ f (1) + f (0)] + O n j=0 n

as n → ∞.

Example 10.67 (Convergent sum asymptotics) Let us apply Euler’s summation formula to a sum, Sn = ∑nk=1 ak , that converges when the range of summation is extended to infinity. To make our exposition more specific, we con(2) sider the incomplete zeta function, ζn (2) ≡ Hn = ∑nk=1 k−2 , which has the finite limit: limn→∞ ζn (2) = ζ (2) = π 2 /6 ≈ 1.644934068. Using Eq. (10.48) with f (x) = x−2 , we get

1 1 ζn (2) = 1 − + n 2

1 1 + 2 2 n 1

where R2m (n) = −(2m + 1)

m B2 j + ∑ B2 j + R2m(n), 2 j+1 j=1 j=1 n m

−∑

Z n 1

B2 j ({x}) x−2m−2 dx

because f (m) (x) = (−1)m (m + 1)! x−m−2 . Collecting the terms, we have

1 1 1 1 1 1 1 + · · · + R2m (n). ζn (2) = 1 + + − + · · · − + 2 − 3 + 2 6 30 n 2n 6n 30n5 Denoting the sum of constant terms by C = C(m) = 1 + 12 + ∑mj=2 B2 j , we get m B2 j 1 ζn (2) = C(m) − − ∑ 2 j+1 + R2m (n). n j=1 n

(10.59)

The constant C(m) depends on m, the number of terms in the ESF, but not on n. Do you expect C(m) to be close to ζ (2) = π 2 /6? Not really, because the remainder (which is the integral over the interval [1, n]) also contributes a non-zero—and hard to evaluate—value. 49 ≈ 1.63333. However, for m = 4 we have C(4) = Indeed, choosing m = 2, we get C(2) = 30 341 210 ≈ 1.6238. So increasing the number of terms in the ESF does not necessarily lead to a better estimate. Therefore, we cannot use Eq. (10.59) na¨ıvely for estimation of ζ (2) (and in general, S∞ ) whenever n is large. So, if the ESF cannot be used to estimate S∞ , and it does not give us the error bound O(n−2m−1 ) for arbitrary large m as well, what is the point?

There is a way out, and this escape route will be used in other applications of the ESF, as the next example shows. Since the series ∑k>1 k−2 converges to a value we know (π 2 /6) through other mathematical techniques, we have ζ (2) = ζn (2)+ o(1), where o(1) tends to 0 as n → ∞.


586

Expressing ζn (2) from the latter equation and substituting into the Euler summation formula, we get m B2 j ζn (2) = ζ (2) − o(1) = C(m) − ∑ 2 j+1 + R2m (n). j=0 n Now let n goes to infinity in both sides of the last equation, this yields C(m) = ζ (2) − R2m (∞), R

where R2m (∞) = −(2m + 1) 1∞ B2 j ({x}) x−2m−2 dx exists because B2 j ({x}) is a (periodic) bounded function. Substituting C(m) into Eq. (10.59), we obtain m

m B2 j B2 j + R (n) = + (2m + 1) ζ (2)− 2m ∑ 2 j+1 2 j+1 j=0 n j=0 n

ζn (2) = ζ (2)− R2m (∞)− ∑

Z ∞ B2 j ({x}) n

x2m+2

dx.

The remainder term in the latter equation can be estimated as follows Z ∞ Z ∞ B2 j ({x}) 1 B2m (2m + 1) dx dx = 2m+1 . 6 (2m + 1)B 2m 2m+2 2m+2 x x n n n

Example 10.68 (Harmonic numbers asymptotics) We consider the harmonic numbers: Hn = ∑nk=1 k−1 , n = 1, 2, . . .. Besides the interest in this as a commonly used sequence, it shows a case where the basic mechanism of the Euler approximation is not effective since the remainder term is increasing in m! Fortunately, we have some knowledge of the limiting behavior of these numbers, which helps us circumvent this inconvenience of diverging error terms. To estimate the Hn , we use the function f (x) = 1/x and its derivatives: k! d k f (x) = (−1)k k+1 , k dx x

1 f (x) = , x

d 2 j−1 f (x) (2 j − 1)! =− . 2 j−1 dx x2 j

Substituting in Eq. (10.48), we find Hn =

m B2 j 1 1 1 + 1+ +∑ 1 − 2 j + Rm , x 2 n n j=1 2 j

Z n dx 1

(10.60)

where Rm =

Z n 1

B2m ({x})

dx x2m+1

,

Eq. (10.53)

=⇒

|Rm | 6 |B2m |

Z n 1

dx x2m+1

|B2m | 1 = 1 − 2m . 2m n

The Bernoulli numbers B2m increase fast! For example, B30 = 6.0158 × 108

and

B60 = −2.13999 × 1034 .

The error terms are fast increasing in m. We have the bound for them from Eq. (10.52): Rm = O (2m)!/(2π )2m , which appears to track their actual growth quite well. We take


587

advantage of our knowledge, obtained by means other than this development, that Hn = ln n + γ + O n−1 , where the constant γ is called Euler’s constant, and approximately equals γ ≈ 0.57721566 . . .. Replacing the left-hand side of Eq. (10.60) by this estimate, we find Zn m B2 j 1 1 dx 1 −1 B2m+1 ({x}) 2m+2 . (10.61) 1− 2j + = ln n + + + ∑ ln n + γ + O n 2 2n j=1 2 j n x 1 This should be viewed as a way of stating that γ − 1/2 is our estimate, to within o(n), of the difference between the sum involving the Bernoulli numbers, and the error term. We nail this difference down when we let n → ∞, and find

1 m B2 j + γ= +∑ 2 j=1 2 j

Z ∞ 1

B2m+1 ({x})

dx x2m+2

m

=⇒

∑

j=1

B2 j 1 =γ− − 2j 2

Z ∞ 1

B2m+1 ({x})

dx . x2m+2

Turning right back and substituting this sum into the right-hand side of Eq. (10.60), we have (note the subtraction of the integral ranges!) Hn = ln n + γ +

m B2 j 1 −∑ − 2n j=1 2 j n2 j

Z ∞ n

B2m+1 ({x})

dx x2m+2

(10.62)

and we note that for any fixed m the last integral is in O n−2m−1 . For m = 3 we have

Hn = ln n + γ +

1 1 1 1 − + − + O(n−8 ). 2 4 2n 12n 120n 252 n6

(10.63)

Using m = 3 and n = 10, we obtain 8 correct digits of the Euler constant (it is still unknown whether γ is a rational number or not). D. Knuth [75] used (10.61) with m = 250 and n = 10, 000 to calculate the value of γ to 1271 decimal places.

Exercise 10.69 [1] Continue the derivation to improve the estimate (10.63) of the harmonic numbers to order n−22 : O n−8 =

1 691 1 3617 43867 174611 1 − + − + − + + ··· . 8 10 12 14 16 18 240n 132n 32760n 12n 8160n 14364n 660n20

This expansion was obtained by D. H. Bailey et al. in [12]. Exercise 10.70 [2+] The following sum defines the zeta5 function: ∑k>1 k−s = ζ (s), for ℜs > 1. The finite sum, ∑nk=1 k−s , is called the incomplete zeta function and denoted by (s) ζn (s) or Hn . Show that when n → ∞ this sum has the asymptotics " # n B 1 j + s − 2 def j −∑ , s 6= 1. (10.64) ζn (s) = ∑ k−s = ζ (s) + n1−s 1 − s j>1 jnj j−1 k=1 5 The

notation, ζ , was introduced in 1838 by Lejeune Dirichlet (1805 – 1859).


588

Example 10.68 covers the complementary case, s = 1, and the approach used there can be followed for the current derivation. The interesting fact in this result is that although the zeta function is represented by the sum ∑k>1 k−s only for ℜ s > 1, the estimate above holds in a wider region (since the function can be defined in the entire plane, except the point s = 1, by analytic continuation). Thus, for example, ! n √ B j j − 3/2 1 1 ∑ √ = ζ 2 + n 2 − ∑ jn j j − 1 j>1 k=1 k √ 1 1 ± ··· . = −1.4603545088 . . . + n 2 + − 2n 24n2 For n = 106 , the sum, ζn (1/2) = ∑nk=1 √1k , has the value of 1998.54014. . . , whereas the first √ two terms (ζ 21 + 2 106 ) of the asymptotic formula give approximately 1998.539645. Exercise 10.71 [2+] Derive four-term approximations of the following sums: n

∑ k3 =

(a)

k=1

n2 (n + 1)2 , 4

n

(b)

(2)

∑ k−2 = Hn ≡ ζn (2),

k=1

n

(c)

∑ ek = k=1

en+1 − e , e−1

using the Euler formulas (10.48) and (10.55). Determine whether these approximations are underestimates or overestimates of the exact values of these sums. Exercise 10.72 [2+] Use the Euler summation formula to obtain the following estimates n 3 1 1 1 1 = n ln 2 + + − +O 5 . (a) ∑ k 3 4 16n 128n n k=0 1 + n n 1 1 πn 3 1 = + − +O 2 . (b) ∑ k2 4 4 24n n k=0 1 + 2 n

Exercise 10.73 [3] Use the ESF to obtain the following approximations involving logarithms: n 1 1 1 1 1 (a) ∑ k ln k = n2 ln n − n2 + n ln n + ln n + + O n−1 . 2 4 2 12 4 k=1 n 1 5 n(n + 1) ln n + n2 ln 4 − (1 + ln 2) + O(n−2 ). − (b) ∑ k ln(2n − k) = 2 4 12 k=1 n 1 1 1 11 1 ln k 1 2 = ln n − γ1 + (ln n) − + − + O(n−5 ), where + (c) ∑ 2 4 2 2 2n 12n 120n 12n 720n4 k=1 k γ1 is the first Stieltjes constant, Eq. (12.56) on page 696. n

(d)

1

1

∑ k (ln k) ln(ln k) = ln(ln ln n) +C + 2n ln n ln(ln n) − · · · , where C is a constant.

k=3

1 1 2 1 1 1 =K− − + + · · · , where K is a constant. + (e) ∑ 2 ln n 2n ln2 n 12n2 ln2 n ln3 n k=2 k ln k Note: The constant C in part (d) indicates the contribution that comes from the constant terms n


589

in the Euler summation formula. Therefore, its actual value depends on the number of terms in the asymptotic formula but not on n. The same words can be said about the constant K in part (d). Although the ESF cannot provide the value of K (it only establishes the existence of such a constant), we know this constant to be ∑k>2 k ln12 k ≈ 2 because of its convergence. Exercise 10.74 [1+] In §10.2, we replaced the sum in Eq. (10.31) by an integral without ado. Use the Euler summation formula to find the error term that substitution introduces. Exercise 10.75 [2+] Develop the Stirling formula to order O n−4 for the factorial, by noticn ln k, and using the Euler summation formula for this sum. ing that ln(n!) = ∑k=1

Exercise 10.76 [2] The number of permutations of [1..n2 ] that have no monotone subsequence longer than n (compare with Exercise 1.19 on page 14) is known to have a value 2 2 given by the unwieldy expression (2n−1)(2n−2)2n···n! n (n−1)n−1 ···11 . Find an asymptotic expression for this surprising formula (taken from [79, p.607]), to relative order O(n−2 ). Hint: Use the results of Exercises 10.73 and 10.75.

Exercise 10.77 [2+] We know the value of the geometric sum ∑k>0 e−k/n = 1/(1 − e−1/n ). Use the ESF to estimate it to order O n−2 , and show how to get the same estimate from the right-hand side as well.

Exercise 10.78 [3] (a) Show that if t is “small enough” compared to n then (n + t)! ∼ 2 n!nt et /2n , and determine how large t may be for this still to hold. (b) Extend the above relation to the first correction term.

Exercise 10.79 [3] This problem is concerned about an asymptotic property of the nth Bernoulli (an+b) polynomial, Bn (x). For constants a and b, find an asymptotic formula for Bn+1 (an+b)n as n → ∞.

10.4.1 Alternating Series Even though our calculations concern almost always expressions that are positive, we may get them in terms of alternating sums, as we saw when the principle of inclusion and exclusion was discussed in Chapter 8. Consider a simple sum, similar to the one handled in Example 10.60: Sn = ∑nk=0 sk , where sk = (−1)k k p , for some noninteger p. Application of the ESF leads to inserting the factor (−1)x into the function f (x), which introduces immediately an obstacle. Here is a direct way around this difficulty: assume for simplicity that n is even, n = 2r, and def r write S2r = Tr −Ur , where Tr = ∑rj=0 s2 j = ∑rj=0 (2 j) p , and Ur = ∑r−1 j=0 (−1)s2 j+1 = ∑ j=1 (2 j − 1) p . A somewhat more general formulation, from Gonnet [49], is as follows: Let ∑n0 f (ak) = F(a, n), This helps not only with alternating summation but also multisections (for simplicity, we write for a bisection only) as the following example shows: n

∑ (−1)k f (k) = 2F(2, ⌊n/2⌋)−F (1, n);

k=0

∑ 16k

odd 6n

f (k) = F(1, n)−F (2, ⌊n/2⌋). (10.65)


590 ⌊n/2⌋

These sums, F(2, ⌊n/2⌋) = ∑k=0 f (2k) and F(1, n) = ∑nk=0 f (k), are not alternating series and they can be evaluated straightforward, possibly using the ESF. However, as part (a) of Exercise 10.80 suggests, getting useful results here may not be easy. Frequently, S2r is much smaller than either of Tr and Ur . In other words, there is a significant cancellation between the two partial sums corresponding odd and even indices. If, for example, just the leading term in each sum is the same, they cancel out, and finding the leading term in S2r , would require us to find two exact terms in each asymptotic formula of Tr and Ur . When the summand is more complicated, the following approach is more effective, usually. The idea is very simple: instead of splitting the sum into two series and evaluating the asymptotics of each series, we pair off adjacent terms and sum the differences. Thus, we write S2r = s0 + ∑rk=1 vk , where vk = s2k + s2k−1 . If these differences are unimodal and fastchanging, then the technique of §10.2 may be applied, and usually we can utilize the ESF as well. For the special case of sk = (−1)k k p , we find p j! p −1 1 = (2k) p 1 − ∑ vk = (2k) p − (2k − 1) p = (2k) p 1 − 1 − 2k 2k j>0 j p p 1 1 p − + ∓ ··· . = −(2k) p 2 2k 2 (2k) 3 (2k)3 Now, we handle separately each of these terms. Since they are of the same functional type, the calculation is far simpler, and although their signs oscillate here as well, they are of different sizes, and will not cancel. There is the alternating version (taken from [22, 23]) of the Euler summation formula, known as the Boole summation formula: i n−1 1 m−1 E j (h) h n−1 ( j) a ( j) k (−1) f (n) + (−1) f (a) + Rm , (10.66) (−1) f (k + h) = ∑ j! ∑ 2 j=0 k=a where f (t) is a smooth function on the closed interval [a, n], 0 < h < 1, E j (x) are the Euler polynomials, Eq. (C.54) on page 714, and the error term ( Z n −E˜ m (x − 1), 1 (m) when 0 6 x < 1, f (x)E˜m−1 (h−x) dx, E˜ m (x) = Rm = 2(m − 1)! a Em (x) is expressed through the periodic Euler polynomials E˜ r (x) = (−1)⌊x⌋ Er ({x}), having period 2. If f is a function with m continuous derivatives defined on t > x and f (k) (t) → 0 as t → ∞ for all k = 0, 1, . . . , m, then for 0 6 h 6 1 m−1

∑ (−1)k f (x + h + k) = k>0

∑

j=0

E j (h) ( j) f (x) + 2 ( j!)

Z ∞ ˜ Em−1 (h − t) (m) f (x + t) dt. 0

2(m − 1)!

(10.67)

def k p Exercise 10.80 [3] To within a relative error of O r−4 , evaluate the sum S2r = ∑2r k=0 (−1) k for p = 2.5

10.5. FINDING PRIMES

591

(a) by splitting the sum to the two series as suggested above, S2r = Tr − Ur , and using for each of the partial sums the Euler summation formula (ESF); (b) by using the second suggestion, and summing the difference between adjacent terms; (c) by applying the ESF directly for the function f (x) = x p eiπ x and using the identity eiπ = −1.

Exercise 10.81 [2] For a positive parameter t, find an asymptotic expression for the following k 2 2 −1/2 , by splitting it into two series: S (t) = alternating sum, S2n (t) = ∑2n 2n k=0 (−1) k + t Tn (t) −Un (t). Exercise 10.82 [3] For a real number p > 1, find an asymptotic expression for the sum k−1 ln p k . Note: Convergence of the series can be very much accelerated by S2n = ∑2n k=1 (−1) k using Euler’s transformation, Theorem L.88 on page 741. Exercise 10.83 [3] Using the version of the Boole summation formula as given in Eq. (10.67), show that 1 1 (−1)k , n → ∞. = +O ∑ 2 log n n log2 n k>0 log(n + k)

10.5 Finding Primes Prime numbers (that cannot be written as a product of two smaller numbers) are the very atoms of arithmetic. Distribution of primes remains the most mysterious object studied in mathematics because of impossibility to predict when the next prime will appear. Let us consider the table n

Number of primes from [1..n], On average, how many numbers are needed before seeing a prime often referred to as π (n)

10 4 2.5 100 25 4.0 1,000 168 6.0 10,000 1,229 8.1 100,000 9,592 10.4 1,000,000 78,498 12.7 10,000,000 664,579 15.0 100,000,000 5,761,455 17.4 1,000,000,000 50,847,534 19.7 10,000,000,000 455,052,511 22.0 that gave C. Gauss the ground to model the number of primes in [1..n] by the sum 1 1 1 1 + + + ··· + . ln 2 ln 3 ln 4 ln n


592

He could then estimate the number of primes from 1 to n as π (n) ∼ n/ ln n when n → ∞. This result is known as the Prime Number Theorem. Gauss actually produced a special function, called the logarithmic integral. It is convenient to have two notations for it, namely, li (x) and Li (x) = li (x) − li (2), which are defined as follows li (x) =

(R x

dt 0 lnt ,

limε →+0

hR

1−ε dt 0 lnt

+

Rx

dt 1+ε lnt

i

for 0 < x < 1, , for x > 1;

Li (x) =

Z x dt 2

lnt

.

(10.68)

Examining π (n), the number of primes less than or equal to n, is surely one of the oldest and fascinating problems in the long history of mathematics. In 1852, P. Chebyshev proved that there are constants A and B so that, for all natural numbers n > 1,

Bn An < π (n) < . ln n ln n

(10.69)

It is known that constants A and B can be chosen very close to 1 (for example, A = 0.922 and B = 1.105) for big enough n. While the logarithmic integral function does a fairly good job of counting primes, Bernhard Riemann (1826 – 1866) introduced a new estimate for the number of primes, which substantially improves Gauss’s guess. His conjecture, known as the Riemann hypothesis (1859), is still waiting for a rigorous proof. The hypothesis establishes a relation between distribution of primes and zeroes of the zeta function; it is equivalent to the following asymptotics of π (x): √ π (x) = Li (x) + O( x ln x) as x → ∞. (10.70) We take an opportunity to present two classical algorithms used to find small primes. The word small may look odd since there is no difficulty in using these algorithms to find all the primes smaller, say, than 108 . However, this is small compared with the needs of typical modern utilization of primes that are the linchpin for security in banking and e-commerce. In their applications, much larger numbers have to be used, and different algorithms are required. Modern cryptography assumes that the primes will retain their secrets for quite a bit of time. The best known algorithm to find primes is old. In addition to computing the circumference of the earth, Eratosthenes of Cyrene (276 – 194 BCE) introduced the algorithm, called now the sieve of Eratosthenes, to determine all primes up to a given number n. While it is very likely that similar sieves (removes) were known before, Eratosthenes is said to be the discoverer of some refinements. The algorithm assumes the availability of an array of n positions, initially all unmarked:

sieve(n) mark position 1; do {let k be the next unmarked position; mark all positions k ∗ j, for k 6 j 6 n/k; } until k2 overflows the array;

// 1 is not a prime // not primes


593

All the unmarked positions in the array identify the primes. How much work is involved? As the various primes are used in order, k = 2, 3, 5, 7, . . . , the algorithm needs to find k, which involves scanning the array from the position of the last prime used, and mark ⌊n/k⌋ − k + 1 √ positions. The ideas of starting the inner loop at j = k and ending the outer one with k = ⌊ n⌋ are probably due to Eratosthenes. The scanning for the next prime, overall, amounts to testing √ the first ⌈ n ⌉ array positions, once. The number of markings of positions is Sn = 1 +

jnk 2

−1+

jn k 3

−2+

jnk 5

−4+

jn k 7

− 6+ ··· ,

(10.71)

where the initial 1 is for the marking of position 1, and the highest prime number we use— √ that generates any new markings—is bounded by ⌊ n⌋. What is this sum? If the algorithm marked each position it touches exactly once, then we would have an immediate evaluation, as Sn = n − π (n), where π (n) is the conventional symbol for the number of primes less than or equal to n (also called the prime number distribution function).

However, this is surely not the case since many places are marked multiple times. Position 12 is marked by both primes 2 and 3, j and kposition 70by 2, 5 and 7. So we need to look further, √ we need the sum Sn = 1 + ∑ p6 n np − (p − 1) . It turns out that there are two auxiliary sum (over primes, p) affiliated with Sn : ′ def

Sn =

n ∑√ p , p6 n ′′

and

′′ def

Sn =

∑√

p6

n . p n

′

The sum without the floors, Sn , has the same asymptotics as Sn because it introduces only a fast-decreasing relative error, as n increases. The calculation, which we adapt from [58, §4.2.3], is instructive. i ′ h ′ Exercise 10.84 [1+] Show the above claim, Sn − ∑ p6√n np /Sn → 0 as n increases, and give a lower bound on the rate of convergence (use the estimate (10.73)). A well known result about the distribution of primes says (see [78, §4.5.4]) that for some c>0 Z n √ dx + O ne−c ln n , π (n) = (10.72) a ln x

where a reasonable value for a is in the semi-open interval (1, 2]; we use 1.5 below, but its actual value does not matter. The error term can be slightly improved, and it is clearly significant, unless c can be shown to be not “too small,” as the following exercise asks you to do. Exercise 10.85 [2] Use √ the sieve of Eratosthenes and MAPLE to estimate c in the relation π (n) = Li(n) − ne−c ln n . Let n range up to at least 107 . Note: Gauss conjectured that Li(n) always appeared to overestimate the number of primes— numerical evidence supports this claim up to 1016 . However, it is proved that eventually π (n) must sometimes overtake Li(n).


594

The integral in the formula (10.72) can be expanded, using integration by parts, to n n 2!n r!n n + + ··· + +O + . π (n) = ln n (ln n)2 (ln n)3 (ln n)r+1 (ln n)r+2

(10.73)

This divergent series, when stopped at any fixed r ∈ N, provides an asymptotic series for π (n). Note that each term in this series is of a higher order than the big-Oh term in Eq. (10.72), ′′ hence it does not appear here. How does this help us with evaluating Sn = ∑ p6√n np ? We can think of π (x), with a continuous parameter x, as a staircase right-continuous function that increases by one as x reaches the value of a prime number. Now we consider a related sum and using Eq. (10.72), we write Z t Z x Z x Z x √ du d π (t) d π (t) 1 1 def −c lnt = = d + O te . (10.74) s(x) = ∑ = t t 1.5 ln u 1.5 2 1.5 t p6x p We split the right-hand side on the ‘+’ in the distribution. Property (h) of Stieltjes integral on R t du 1 page 698 helps with the first part there, and yields dtd 1.5 ln u = lnt . Hence, this part contributes R x dt the immediate integral 1.5 t lnt = ln ln x + O(1).

For the second integral (10.74), we usepart (f)of Exercise 12.84 in√theright-hand √side of Eq. √ √ Rx 1 Rx 1 −c −c −c −c lnt ln 1.5 ln x lnt =O e +O e + O 1.5 t d te . The that gives 1.5 t dO te first two parts contribute O(1) and o(1), respectively, as x increases. Forhthe √latter one,√weiuse R

−c lnt

−c lnt

x e − c e2t √lnt dt. part (a) of Exercise 12.84, which shows that the integral there equals 1.5 t Both integrals converge to constants as x increases (the values depend on c, naturally); hence, the total contribution of this part is O(1). Thus, s(x) = ∑ p6x 1p = ln ln x + O(1).

Exercise 10.86 With the procedure we just used to compute s(x), show that (ln p)m (ln x)m ∑ p = m + O(1), m > 1. p6x j k ′′ ′ While we need Sn = ∑ p6√n np , Exercise 10.84 shows that we can handle Sn = ∑ p6√n np , √ which in terms of s(x) is given by n s( n). Using asymptotic expansion of s(n), we find ′ Sn = n ln ln n + O(n). def

sm (x) =

′

The next component of the cost Sn , which we need to subtract from Sn , as given in Eq. (10.71), √ √ is Tn = ∑ p6√n (p − 1) = ∑ p6√n p − π ( n). To find the sum of prime numbers up to n, we R Rx ′ td π (t) = 2x td π (t). However, we proceed exactly as before and evaluate Tx = 1.5 √ shall take advantage of a different form of the correction term for π (t): π (t) = Li (t) + O t lnt . R x R t du ′ To find the value of Tx , we need to consider two contributions. One is 1.5 td 1.5 ln u = Rx t 2 ) + O(1), where Li (x) is the logarithmic integral. How does this function dt = Li (x 1.5 lnt

behave for large values of x? The answer is known to be Li (x) =

x ∞ k! ∑ (ln x)k . ln x k=0

(10.75)


595

We have here a divergent series, which, truncated at any finite k, provides an asymptotic √ series for this function, as in Eq. (10.73). When we evaluate it at x = n, we find that Li (x2 ) = (n/ ln n) 1 + O(ln n)−1 . √ Rx t dO t lnt . Using Eq. (12.63), we find for the three The second component of π (t) is 1.5 terms on the right-hand side there O(1), O x3/2 ln x , and an integral within a big-Oh we suppress while calculating: Z x √ Z x √ 1 d √ t lnt dt = t lnt + t dt. t 2 a a dt The first integral provides one more contribution to the dominant term, on integrating by parts it gives: 23 x3/2 ln x + smaller terms. Hence O x3/2 ln x is the contribution of this √ √ part, and with the substitution x → n, we get O n3/4 ln n . Finally, we note that π ( n) = √ (2 n/ ln n) 1 + (ln n)−1 , and we can collect the components of the cost: 2√n 1 1 n 3/4 1+ + O n ln n + 1+ . Sn = 1 + n ln ln n + O(n) − ln n ln n ln n ln n ′

It turns out that the first component of Sn , the sum Sn , is the dominant term here; the scanning merely adds another O(n). Hence, the cost of the sieve of Eratosthenes for finding all primes up to n is given by n (ln ln n + O(1)), when the benchmark operation is the cost of marking a single position in the table. This cost can be as small as a single assignment where each position is a byte or a word. On the other hand, the cost can also involve several shifts and a logical operation that leads to increase in the number of positions we can handle. The second method to find primes is more of a brute-force approach: a program that checks divisibility. Naturally, it is slower, as we shall see, and has an interesting property of allowing some trade-offs between storage use and running time. Again, we need to reserve an array, but this time it is used to store the prime numbers we find, as we go along, not mere marks. We name it p, and each entry holds an integer. What is the required size of p? As Eq. (10.73) shows, if we plan to search for primes up to n, we need to reserve π (n) ≈ lnnn (1 + a), where a is a small number, on the order of 2/ ln n. Therefore

π (n) ln n = 1 . n→∞ n lim

The following algorithm allows us to find all primes up to a certain value, or up to a predetermined number of primes. It is slightly simplified, to avoid handling boundary cases, in a C-like structure:

Primes by brute force (n, np) int n, np; { int k, lim p = 1, j; double sqrtk; p[0] = 2; p[1] = 3;

// n – upper bound on desired primes // np– upper bound on # of primes


596

for k from 5 by 2 while (k < n) {sqrtk = sqrt(k); for j from 1 while (p[ j] < sqrtk) if ((k%p[ j]) == 0) break; if (p[ j] > sqrtk ) p[+lim p] = k; if (lim p == np) break; }}

// divisibility check

Let us analyze the run-time of this algorithm, C(n). We only count here the main operation: the number of divisibility checks the algorithm makes. Assume that the argument np is set at a value larger than π (n), so that all the primes less than or equal to n are found. How many divisibility checks are made? We need to consider separately the handling of primes and composite numbers. For each prime p, the number of divisibility checks is π (p1/2 ) − 1 (the ‘−1’ is there since we do not bother with the prime 2 since we skip even numbers). Suppose p = pk , the kth prime, √ then it requires π ( pk ) − 1 checks, and the total number of such checks used by the primes is (we start with the third prime) π (n)

π (n) √ √ ∑ (π ( pk ) − 1) = ∑ π ( pk ) − (π (n) − 2) .

k=3

k=3

We can change the lower bound to k = 1 without any change in value, and then use a procedure similar to the one we used for Eq. (10.74). Concentrating on the sum, we need to calculate π (n)

s(n) =

∑

√ π ( pk ) =

k=1

=

Z π (n) t =1.5

Z π (n)

√ π ( t)d π (t)

1.5

Z √t

! Z t q √ √ −c 1/ lnt du dx −c lnt 2 +O + O te te d u =1.5 ln u x =1.5 ln x

Splitting the calculation on the two ‘+’s in this expression we have four terms to consider: S1 =

Z π (n) Z √t dx t =1.5

S2 =

du d ; ln x ln u u =1.5 x =1.5

Z π (n) Z √t dx t =1.5

x =1.5

Z t

ln x

√ d O te−c lnt ;

S3 =

Z π (n) t =1.5

S4 =

Z π (n) t =1.5

The integral S1 is processed as follows S1 =

Z π (n) Z √t dx t =1.5

Z t

du d = u =1.5 ln u x =1.5 ln x

Z π (n) Z √t dx dt t =1.5

x =1.5

ln x lnt

Z q √ −c 1/ lnt 2 te d O

t

du ; ln u u =1.5

q √ √ −c 1/ lnt 2 te d O te−c lnt . O

=

√ Z π (n) dt 2 t t =1.5

lnt

√ 4 t + 2 + ... . lnt ln t


597

We need now an integration formula, √ tuting y = t t, and leads to S1 =

Z (π (n))3/2 2

R b √t

a lnk t dt

=

3 k−1 R b3/2 dy a3/2 lnk y , 2

which follows on substi-

(r − 2)!3r−1 4 (π (n))3/2 1 + O [ln π (n)]−1 . = r 2 ln y 3 ln π (n) r>2

dy ∑

This is the leading term, asymptotically; as it is evident from the sum, the numerical coefficients of lower order terms increase fast, which suggests very slow convergence of the asymptotics. It is a tedious but straightforward calculation, using Eqs. (12.62) and (12.63) on page 698, to show that Sk = o(S1 ), k = 2, 3, 4. Finally, writing S1 directly in terms of n, we 3/2 find S1 = 43 n7/2 (1 + O((ln n)−1 )). This part of the cost function alone is significantly larger ln n than the cost of the sieve, as promised. Next, we calculate how many divisibility tests are needed to weed out the non-primes. They are more numerous, but are quicker to dispose of. Which of these two factors dominates? The program saves some time by considering only odd numbers, and therefore one third of the (n/2) numbers end their test when divided by three, one fifth need two checks, one seventh—three, and in general, the kth prime disposes of (1/pk ) of all numbers, at the cost √ of k − 1 tests. Since when √numbers up j tok n are checked, the first π ( n) primes are used, π ( n) we need to add the cost ∑k=2 (k − 1) pnk . The argument needed for Exercise 10.84 is also effective here √ to show that for large n we may neglect the floor function, and sum instead π ( n) An = n ∑k=2 k−1 pk . Here is a simple way to get a rough estimate of this sum√ (Exercise 10.87 asks for a refine√ π ( n) ment). We first note that An can be written as An = n ∑k=2 pkk − n(s( n) − 1/2), where s() is defined in Eq. (10.74), and the 1/2 appears there since the sum starts at k = 2. We found s(x) ∼ ln ln x, but we still need the first part. The value of the function π (n) ∼ n/ ln n suggests that since n is close to the π (n)th prime, the kth prime is approximately k ln k. Checking with MAPLE suggests that the ratio pk /(k ln k) gets very quickly to about 1.15, and as k increases it goes down, with erratic oscillations, extremely slowly. Hence the sum we still need to do is Z π (n1/2 ) π (n1/2 ) π (n1/2 ) √ 1 k k ≈n ∑ ≈n dt = n Li (π ( n)) + O(n). n ∑ lnt t=2 k=2 pk k=2 k ln k Using the information in Eq. (10.75), we find that the value above has the leading term n3/2 / ln2 n. This is larger, by a factor ln3/2 n, than the value we found for the cost of testing the primes, hence this is also the leading term of the total cost. Exercise 10.87 The calculation of C(n), the run-time of Primes by brute force algorithm, includes several rough estimates and approximations: even the coefficient of the leading term is not exact. Refine the calculations to a relative error of O(n−1 ). Since we saw the leading term is of order n2 / ln2 n, you need to find the correct coefficients of all terms larger than n/ ln2 n. What about the storage usage? Keeping only primes, rather than an indicator for each number,

598


as the sieve does, turns out to provide no savings: while the sieve needs n positions, many more positions than the second method, which only needs slightly over n/ ln n locations, the sieve can use a single bit per position, whereas here we need to store integers. If we use 32 bits per position, the sieve is space-cheaper for all n such that ln n 6 32, which corresponds to n = 7.8963 × 1013 . Moreover, already for n = 232 ≈ 4.925 × 109 numerous prime numbers exceed the largest number processors can hold in a 32-bit word. However, suppose we do not store the prime numbers. Instead, let us print them, or push them off to external storage, where it does not make sense to access them for each new candidate. So we save a lot of space, but naturally there is price to pay: we need to check divisibility by 2 and every odd number, such as 9 and 15, not just the smaller primes. Hence, to determine √ that p is a prime, we need to test its divisibility by p/2 values. The calculation for the nonprimes is similar to what we did above, but instead of summing over primes, we sum over all odd numbers, smaller than the square root of the tested value. Exercise 10.88 Compute the cost of the variant we just described that uses no storage beyond a small number of indices and constants (which allows us to call this method an O(1)-storage algorithm). Exercise 10.89 (a) Compute the cost of a variant of the algorithm (page 595) Primes by brute force, which for some given input n needs only to report pn , the nth prime. Hint: Estimate the value of pn , and an upper bound on the values of the primes that √ need to be stored: you only consider the primes smaller than pn . (b) Now consider another variant that is designed to find the largest prime not larger than √ n. An efficient approach may be to find all primes smaller than n, and then—suppose the value of n is even—test successively n − 1, n − 3, . . ., until a prime is found.

10.6 Asymptotics from Recurrences We have seen that solving recurrences often leads to expressions that provide us with little insight; so we need to look for their asymptotic behavior to understand the results we derived. Can this two-phase process, of exact solution followed by the search for asymptotics be abbreviated? Several approaches have been taken. Two are shown here. The first one is applicable to divide-and-conquer recurrences, a particular case of which has been popularized under a rather ambitious title, the “Master Theorem.” The second approach is the more general Birkhoff-Trjitzinsky method [18, 139] that answers this question in the affirmative way, in certain cases.

10.6.1 Divide-and-Conquer Asymptotics The run-time of a recursive algorithm does usually satisfy a recurrence. An important part of recursive algorithms consists of algorithms based on the well-known divide-and-conquer approach. Such algorithms often lead to recurrences with integer-functions, and some of

10.6. ASYMPTOTICS FROM RECURRENCES

599

them were considered in §5.4. For instance, the merge sort algorithm yields the following recurrence: an = a⌊n/2⌋ + a⌈n/2⌉ + bn , (10.76) where bn is the cost of the divide and combine steps. This term is customarily called the toll function. A technique for handling these recurrences and finding asymptotic solutions by integral transform methods was developed by Flajolet and Golin [43]. They suggested to get rid of the floor and ceiling operations by applying ∇, the backward finite difference operator: ∇an = a⌊n/2⌋ + a⌈n/2⌉ + bn − a⌊(n−1)/2⌋ + a⌈(n−1)/2⌉ + bn−1 .

For even indices n = 2k, this simplifies to

∇a2k = (ak + ak + b2k ) − (ak + ak−1 + b2k−1 )

= (ak − ak−1 ) + b2k − b2k−1 = ∇ak + ∇b2k ,

a recurrence without ceilings and floors. For odd indices n = 2k + 1, we get in similar way that ∇a2k+1 = ∇ak+1 + ∇b2k+1 . However, it does not quite help us in solving the divide-and-conquer recurrence (10.76) because the right-hand side contains the combination of ak with odd and even indices. To separate them, we apply the forward difference operation to obtain ∆∇a2k = ∇a2k+1 − ∇a2k = (∇ak+1 + ∇b2k+1 ) − (∇ak − ∇b2k ) = ∆∇ak + ∆∇b2k ,

(10.77)

and for odd indices ∆∇a2k+1 = ∆∇b2k+1 .

(10.78)

Let D(s) be the Dirichlet generating function for the sequence {∆∇an }n>1 . Then using Eqs. (10.77) and (10.78), we obtain def

D(s) =

∞ ∞ ∆∇a2k ∆∇a2k+1 ∆∇an = + ∑ ∑ s s s n=1 n k=1 (2k) k=0 (2k + 1)

∞

∑

=

∞ ∆∇b2k+1 ∆∇ak + ∆∇b2k + ∑ s s (2k) k=0 (2k + 1) k=1

=

1 2s

∞

∑

∞

∞ ∆∇ak 1 ∆∇bk + ∑ ks ∑ ks = 2s D(s) + η (s), k=1 k=0

where η (s) is the Dirichlet transformation of the known sequence {∆∇bk }. So we find its solution: η (s) . (10.79) D(s) = 1 − 2−s This equation together with the Mellin-Perron formula (2.42) (page 41) provides an exact solution to the recurrence (10.76).


600

Theorem 10.90 (Flajolet and Golin, 1994) Let b1 , b2 , . . . be a sequence of numbers that grow as O(n). The recurrence (10.76) has the solution an = n a1 +

n (V.P.) 2π i

Z c+i∞ c−i∞

η (s) ns ds, s(s + 1)(1 − 2−s )

(10.80)

where η (s) is the Dirichlet transform of the sequence {∆∇bk }, c is any number greater than 2, and (V.P.) signifies that the integral is understood in special sense called principal value after Cauchy (who called it valeur principale in French), see definition (6.21) on page 282. When the Dirichlet transform η (s) is meromorphic, all the singularities in the integrand are poles, the integral (10.80) is evaluated as the sum of its residues.

Example 10.91 In §5.4.3, we discussed the mergesort algorithm and have shown that its cost, in number of comparisons, satisfies the canonical divide-and-conquer recurrence (10.76). The actual values of the bn in Eq. (10.76) depend upon whether it is the worst, best, or average case that is being analyzed. Exact solution formula, which corresponds to the worst case k behavior of the mergesort, was found n in Eq. (5.25) for n = 2 , and in Exercise 5.79 for general case. It is known [43] that bn = 2 for the best case. Let us consider the worst case when bn = n (n > 1). For simplicity, we ignore constant terms in this formula. Adding the initial conditions a0 = b1 = b0 = 0, we obtain ∆∇a1 = ∆(a1 − a0 ) = ∆a1 = a2 − a1 = b2 = 2,

∆∇bn = ∆(bn − bn−1 ) = ∆(n − n + 1) = ∆1 = 0,

n > 2.

Then the Mellin transform of the sequence {∆∇bn } is 2/1s = 2, and from the formula (10.80), we obtain Z 3+i∞ 2 ns n (V.P.) ds, (10.81) an = −s 2π i 3−i∞ s(s + 1)(1 − 2 ) because a1 = 0. To evaluate the integral in Eq. (10.81), we use the approach as in Example 10.51: move the vertical line (3 − i∞, 3 + i∞) to the left, to (−2 − i∞, −2 + i∞). Such deformation crosses the singularities of the integrand, f (s) = 2ns /s(s + 1)(1 − 2−s ); therefore, we need to add 2 , the residue of f (s) at the double pole s = 0, 1. 2 ln n + 1 − ln 2 2. 2/n, the residue of f (s) at a simple pole s = −1, n2π ik/ ln 2 , the sum of residues at simple poles sk = 2π ik/ ln 2, k = 2π ik ln 2 − 4π 2 k2 ±1, ±2, . . ..

3. 2 ln2 2 ∑

Note that the function f (s) has thefollowing Laurent representation around s = 0: 1 2 1 ln n − 1 1 2 + + O(1) = 2 + +2 + ···. f (s) = 2 1 + s(ln n − 1) + O(s ) s2 ln 2 2s s ln 2 s s ln 2 The residue at s = 0 provides the main contribution to the asymptotic behavior of an .


601

In subsection 5.4.2, we considered other more complicated divide-and-conquer recurrences, where coefficients also depend on floors/ceiling of indices. This requires to analyze recurrences of different type, and we start with the following recurrence (which occurs in parallel divide-and-conquer algorithms): T (n) = a T (n/b) + f (n),

(10.82)

where a, b are positive constants and f (n) is a given function. In the algorithmic context, b is almost always an integer, the number of fragments that the input data is divided into, and the toll function, f (n), represents the cost of the dividing and recombining, for the recursive steps. When n is a power of b, that is, n = bm , for some m ∈ N, the recurrence (10.82) is reduced to a linear first order difference equation, tm = atm−1 + f (bm ), by changing the variable: tm = T (bm ). Assuming that the recurrence holds for all m > 1, and the initial value T (1) = t0 is known, the general solution is obtained from Eq. (5.13) to be

∑

f (b j+1 )

j=0

∏

i= j+1

m−1

m−1

m−1

m−1

tm =

a + t0 ∏ a = i=0

m−1

Since m = logb n, we have obtained T (n) =

∑ k=0

f

∑ k=0

f

n bk

ak + t0 am .

(10.83)

n ak + T (1)nlogb a . bk

If n is not divisible by b, it is assumed that the function T (n/b) reduces the ratio n/b to an integer in a certain way (for instance, using floor or ceiling). In §5.4.3, we split n to ⌊n/2⌋ and ⌈n/2⌉ for b = 2. In general, for larger b, the relation would involve some division of n into b parts; however, we assume the following: (i) all the parts are either ⌊n/b⌋ or ⌈n/b⌉, and (ii) the function T (n) is monotonic increasing in n. This is sufficient for the following claim to hold: Claim 10.92 The following two interpretations of Eq. (10.82), for all n ∈ N, l n m j n k + f (n) and T (n) = aT + f (n), T (n) = aT T (1) = T (1) = T (1), b b

provide, respectively, upper and lower bounds on T (n): T (n) 6 T (n) 6 T (n),

and

T (bm ) 6 T (bm ) 6 T (bm ),

n, m ∈ N.

⊳

The claim, in turn, suffices to prove the following theorem, which can be seen simply as estimating which of the two terms in the solution (10.83) is dominant. Theorem 10.93 (The Master Theorem) Let a > 1 and b > 1 be constants, and let f (n) be a nonnegative function defined on exact powers of b. Then for n > 1, the solution to the recurrence (10.82) can be bounded asymptotically as follows.

1. If f (n) ∈ O nlogb a−ε for some positive constant ε , then T (n) ∈ Θ nlogb a .


602

1a If f (n) ∈ O nlogb a / lg1+ε n for some positive constant ε , then T (n) ∈ Θ nlogb a . 1b If f (n) ∈ O nlogb a / lg n , then T (n) ∈ Θ nlogb a lg lg n . 1c If f (n) ∈ O nlogb a lgq n , where q > 0, then T (n) ∈ Θ nlogb a lgq+1 n . 2. If f (n) ∈ Θ nlogb a , then T (n) ∈ Θ nlogb a lg n . 3. If f (n) ∈ Ω nlogb a+ε for some positive constant ε , and if a f (n/b) 6 K f (n) for some constant K < 1 and all sufficiently large n, then T (n) ∈ Θ ( f (n)). P ROOF : The proof is essentially direct, via the claim 10.92, but quite tedious, due to the numerous cases that need to be checked; it is, however, a good exercise in performing algebra within asymptotic expressions. See [32, §4.4] and [134] for details. Corollary 10.94 Let ε > 1. Using the notation of Theorem 10.93, suppose f (n) ∈ O(nlogb a g(n)), where g be any of the following functions, then T (n) ∈ Θ nlogb a . 2. g(n) = 1/ (log n (log log n)ε ). 1. g(n) = 1/(log n)ε .

Lemma 10.95 Suppose a < b and h(x) is a twice-differentiable function such that h′ (x) 6= 0, x ∈ [a, b]. Then 1.

Rb

′ ′ a 1/h(x) dx ∈ O(ln h(b)/h (b) − ln h(a)/h (a)) ′′ ′ 2 h (x) ln h(x)/(h ) 6 c/h(x) over (a, b) for some

if h′′ (x) ln h(x) 6 0 over (a, b), or c < 1, and

2.

Rb

if h′′ ln h > 0, or constant c.

′ ′ a 1/h(x) dx ∈ Ω(ln h(b)/h (b) − ln h(a)/h (a)) ′′ ′ 2 h (x) ln h(x)/(h ) > c/h(x) over (a, b) for some

Lemma 10.96 Suppose a < b and h(x) is a twice-differentiable function such that h′ (x) 6= 0, x ∈ [a, b]. Then 1.

Rb

2.

Rb

h(x) dx ∈ O(h2 (b)/h′ (b) − h2 (a)/h′ (a)) if h′′ (x) 6 0 over (a, b), or h(x) h′′ (x)/(h′ )2 6 c over (a, b) for some c < 2, and a

a

h(x) dx ∈ Ω(h2 (b)/h′ (b) − h2 (a)/h′ (a)) if h′′ > 0 over (a, b), or > c over (a, b) for some constant c.

h h′′ (x)/(h′ )2

Corollary 10.97 If nondecreasing function satisfies the inequality f (x+1) 6 C f (x) for some positive constant C and a 6 x 6 b, then b

∑ f (k) ∈ Θ

k=a

Z

a

b

f (x) dx .

If nonincreasing function f satisfies for some positive constant C the inequality f (x) 6 C f (x + 1), a 6 x 6 b, then Z b b f (x) dx . ∑ f (k) ∈ Θ k=a

a


603

Example 10.98 Let 1 < n; from Lemma 10.95 for −1 < p < 0 and from Lemma 10.96 for p > 0, we have n n 1 p p+1 p if p > −1, ∑k =Θ n ∑ k = Θ − 2p if p < −1. k=1 k=1 Exercise 10.99 [2] Consider the following recursive implementation of the fast discrete Fourier transform. It is applied to an array of length that is a power of 2. RECURSIVE - FFT (a) 1. n ← length[a] 2. if n = 1 return a ωn ← e2π i/n ; ω ←1 3. [0] 4. a ← ha0 , a2 , . . . , an−2 i 5. a[1] ← ha1 , a3 , . . . , an−1 i 6. y[0] ← RECURSIVE - FFT(a[0] ) 7. y[1] ← RECURSIVE - FFT(a[1] ) 8. for k ← 0 to n/2 − 1 [1] [0] 9. do yk ← yk + ω yk [1] [0] 10. yk+(n/2) ← yk − ω yk ω ← ω ωn 11. 12. return y

Assume that the program needs not to calculate the primitive roots of 1, ωk = e2π i/k , which are available in a table. Derive a recurrence for T (n), the number of arithmetic operations in lines 9 and 10. Then obtain its asymptotic growth as n → ∞. The values depend on whether [1] the product ω yk is computed twice, or only once and remembered. Exercise 10.100 [2] Use the Master theorem (or its refinements) to find asymptotic bounds for the following sums: n

(a)

1

∑ k p (ln k)q ,

k=2

n

q < (1 − p) ln 2,

∑ k p qk ,

(b)

p > 0, q > 2.

k=1

Improved Master Theorems are summarized by S. Roura [117], one of them is stated here. Theorem 10.101 Let T (n) be a solution to the recurrence T (n) = a T (Sn ) + fn , where Sn = n/b + O(1), b > 1, fn > 0. Then  log a  and α < logb (a); if fn ∈ O (nα )  Θ n b , c log a b T (n) ∈ Θ ( fn log n) , if fn ∈ Θ n log n and c > 0;   α and α > a. Θ ( fn ) , if fn ∈ Ω (n )

(10.84)


604

Corollary 10.102 Let b > 1 and take fn = nα with α > 0. If T (n) and V (n) satisfy the recurrences T (n) = a T (n/b + c) + fn , V (n) = aV (n/b) + fn , (10.85) then T (n) ∼ V (n) as n → ∞.

Exercise 10.103 [1] Using Theorem 10.101, find the asymptotic bounds for the recurrence T (n) = 2 T (n/2 + 3) + 5n.

10.6.2 Direct Asymptotics The method of Birkhoff and Trjitzinsky enables us to find the exact asymptotics of sequences defined by a large class of linear difference equations without deriving a complete solution, even in generating function form. There is one drawback in this method—it cannot identify exactly the coefficients but only up to a constant multiplier. Our presentation is limited to homogeneous linear difference equations. Also, we demonstrate how the Birkhoff-Trjitzinsky method works by considering three examples. The general case is considered in the articles [18], [139] and in Hofri’s book [64, §6.1.4]. The homogeneity does not restrict the generality because a non-homogeneous n-th order linear difference equation can be reduced to the homogeneous (n + 1)-th order linear recurrence. For example, in the first order non-homogeneous equation yn+1 = r(n)yn + f (n), we divide by f (n) (assuming that f (n) 6= 0 for all n in the relevant range) and replace n by n + 1 to obtain r(n) 1 r(n + 1) 1 yn+1 = yn + 1 and yn+2 = yn+1 + 1. f (n) f (n) f (n + 1) f (n + 1) Subtraction leads to the homogeneous difference equation of the second order:

f (n + 1) yn+2 − r(n + 1) + f (n)

yn+1 +

f (n + 1) r(n) yn = 0. f (n)

Example 10.104 We start with the famous recurrence of the first order yn+1 = (n + 1)yn ,

n > 0,

y0 = 1,

(10.86)

that is needed, say, to count the number of permutations, yn , of n elements. The recurrence has the solution yn = Γ(n + 1) = n!, the gamma function. Note that the initial condition in (10.86) does not effect the asymptotic behavior of the solution (this is almost always the case in the recurrences we deal with). Direct substitution of a polynomial trial solution i h c1 c2 yn = nµ c0 + + 2 + · · · n n


605

into the given recurrence shows that the product of a polynomial and Poincarè series cannot satisfy the recurrence (10.86). So we add exponential factors: i h c1 c2 yn = nan bn nµ c0 + + 2 + · · · . n n

(10.87)

We shall need the following expansion for the left-hand side of the recurrence: a(n+1)

(n + 1)

an

a

a an

= (n + 1) (n + 1) = (n + 1) n

1 1+ n

an

2

= (n + 1)a nan ean ln(1+1/n) = (n + 1)a nan ean(1/n−1/2n +··· ) 1 a2 a a a an a + − ··· = (n + 1) n e 1 − + 2 2n n 8 3 1 a2 a 1 a a2 a3 a 1 a a + + + e 1− + 2 − 3 + ··· = nan+a 1 + n 2n n 8 3 n 4 6 48 2 k 1 a a a a 1 an+a a + 1− + 2 − ··· =n e ∑ n 2n n 8 3 k k a 1 a2 a 7a2 a3 1 a an+a a − − + + 3 + ··· . =n e 1+ + 2 2n n 8 6 n 12 12 48 So the trial solution for yn+1 would be c2 c1 + + ··· yn+1 = (n + 1) b (n + 1) c0 + n + 1 (n + 1)2 c1 c2 + + · · · = nan+a bn+1 nµ ea c0 + n + 1 (n + 1)2 µ 1 a2 a 1 a − + ··· × 1+ 1+ + 2 n 2n n 8 6 1 an+a n+1 µ a =n b n e c0 + (c0 (µ − a/2) + c1 ) n 2 a c0 a a 1 − + ··· . + 2 c2 − c1 + c1 µ + µ (µ − 1) − (c1 + c0 µ ) + c0 n 2 2 8 6 a(n+1) n+1

µ

Since the bracketed expression must be equal to nyn , we remove the factor nan bn nµ from both sides and equate the coefficients of like powers. This leads to the equations

a = 1,

b = e−a ,

µ = 1/2,

etc.

The constant c0 cannot be determined by this approach since a linear homogeneous recurrence can only be solved up to a multiplier, which an initial value may settle, but the asymptotic expression naturally can not be directly related to an initial value. All other coefficients,


606

ck , (k = 1, 2, . . .) can be determined uniquely (up to a multiple of c0 ). Eq. (10.86) is therefore satisfied formally by the series c1 /c0 c2 /c0 n −n √ (10.88) + 2 + ··· , n c0 1 + yn = n e n n where c1 /c0 = 1/12, c2 /c0 = 1/288 and so on. n 3 n Example 10.105 The sequence yn = ∑ , k=0 k

n = 0, 1, 2, . . ., satisfies the recurrence

(n + 2)2 yn+2 − (7n2 + 21n + 16)yn+1 − 8(n + 1)2 yn = 0. We seek an asymptotic expansion of the solution to the above recurrence in the form i h c1 c2 c3 (10.89) yn = bn nµ c0 + + 2 + 3 + · · · . n n n Using the binomial theorem (2.52), we get 1 1 1 1 1 1 1 = = · 1− + 2 − 3 + ··· , 1+n n 1 + n1 n n n n 1 1 1 4 2 3 1 = = · 1− + 2 − 3 + ··· , (1 + n)2 n2 (1 + n1 )2 n2 n n n µ µ µ (µ − 1) µ (µ − 1)(µ − 2) 1 + + ··· . = 1+ + 1+ n n 2n2 3! n3 Therefore c2 c1 + + ··· yn+1 = b (n + 1) c0 + n + 1 (n + 1)2 h i c1 c2 µ µ (µ − 1) n+1 µ c + + · · · (1 − 1/n + · · · ) + − · · · = b n 1+ + 0 n 2n2 n n2 1 µ (µ − 1) 1 + ··· = bn+1 nµ c0 + (c0 µ + c1 ) + 2 c2 − c1 + c1 µ + c0 n n 2 h i c1 c2 2µ 2µ (µ − 1) c + + + · · · (1 − 2/n + · · · ) + + · · · yn+2 = bn+2 nµ 1 + 0 n 2n2 n n2 1 1 = bn+2 nµ c0 + (2µ + c1 ) + 2 (c2 − 2c1 + 2c1 µ + 2c0 µ (µ − 1)) + · · · . n n n+1

µ

Substitution yn , yn+1 , and yn+2 into the recurrence and cancellation out bn nµ yields b2 − 7b − 8 = 0,

µ = −1,

c1 (b2 − 8) + c0 (2b2 − 21b + 16) = 0,

etc.

Since the quadratic equation b2 − 7b− 8 = 0 has two real roots b1 = 8 and b2 = −1, we obtain the dominant series for b = 8 as its asymptotic expansion: i c1 c2 8n h c0 + + 2 + · · · . yn ∼ n n n


607

Example 10.106 Our next example presents the asymptotic behavior of the simplest linear homogeneous difference equation of the second order with linear coefficients: (a2 n + b2 )yn+2 + (a1 n + b1 )yn+1 + (a0 n + b0 )yn = 0

(a2 6= 0,

a0 6= 0).

(10.90)

This equation is usually referred to as the hypergeometric difference equation because its solution can be expressed in terms of the hypergeometric series (see §2.6 for detail). The important role in our analysis is played by the roots λ1 and λ2 of the characteristic equation a2 λ 2 + a1 λ + a0 = 0,

(10.91)

which we assume, for simplicity, to be distinct (a21 6= 4a0 a2 ).

The difference equation (10.90) is satisfied formally by power series of the form (10.87), but without the factor nan because the coefficients of the recurrence (10.90) are all of the same degree. Indeed, if we substitute asymptotic expansion (10.87) into the given recurrence (10.90), we come across to the equation for the leading coefficient: a2 n2a e2a b2 + a1 na ea b + a0 = 0, which can be satisfied only when a = 0 and b = λ1,2 . Now we substitute the trial series (10.89) into Eq. (10.90), divide by bn nµ , and equate coefficients of successive powers of 1/n to obtain c0 [a2 b2 + a1 b + a0 ] = 0

=⇒

µ b(2a2 b + a1 ) = −(b0 + b1 b + b2 b2 )

=⇒

b = λ1

or b = λ2 , b0 + b1 λi + b2 λi2 , i = 1, 2; µ =− λi (2a2 λi + a1 )

etc. All the coefficients c1 , c2 , . . . (but not c0 ) can be obtained in this way. Since the coefficients b and µ have two different values depending on the distinct roots λ1 and λ2 of the characteristic equation (10.91), the coefficients c1 , c2 , . . . in the formal series (10.89) also depend on these roots and we obtain two asymptotic series:  h i S1 (n) = λ n nµ1 c0 (λ1 ) 1 + k1 (λ1 ) + k2 (λ2 1 ) + · · · , 1 n n h i (10.92) yn = S2 (n) = λ n nµ2 c0 (λ2 ) 1 + k1 (λ2 ) + k2 (λ2 2 ) + · · · , 2 n n where ki (λ j ) = ci (λ j )/c0 (λ j ), j = 1, 2, i = 1, 2, . . ..

As usual, the solution of a linear recurrence is the (possibly weighted) sum of separate solutions, but in the asymptotic realm other considerations apply: If the characteristic equation (10.91) has two roots and, for instance, λ1 > λ2 , then the formal series S1 (n) in Eq. (10.92) dominates and we conclude that yn ∼ S1 (n) as n → ∞. These two formal series (we call them “formal” because the series (10.92) may diverge) solutions S1 (n) and S2 (n) are linearly independent in the sense of the following


608

Definition 10.107 A set of functions z j (n), j = 1, 2, are called linearly independent if the Wronskian " # z1 (n) z2 (n) det z1 (n + 1) z2 (n + 1) does not vanish for any n = 0, 1, 2, . . ..

⊳

The following exercises ask you to find asymptotic expansions for recurrences with linear coefficients and therefore the use of series (10.89) is essential. Exercise 10.108 [2] Find formal power series representation for a solution of the following difference equation: 2(n + 1)yn+1 = (2n + 1)yn , n = 0, 1, 2, . . .. Since its solution is known to be yn = c0 41n 2n n , derive the asymptotic formula, a generalization of Eq. (2.92):

1 1 5 21 399 2n 4n 1 . =√ 1 − + 7 2 + 10 3 − 15 4 − 18 5 + O 6 8n 2 n 2 n 2 n 2 n n n nπ

(10.93)

Exercise 10.109 [2] Derive the asymptotic expansion for the sequence {yn } that is defined by the difference equation (x + n)yn = nyn−1 ,

n = 1, 2, . . . ,

y0 = 1 (x 6= −1, −2, −3, . . .).

Exercise 10.110 [1+] Find the asymptotic expansion for the sequence of Catalan numbers 1 2n (n = 1, 2, . . .) from the recurrence (n + 2)Cn+1 = (4n + 2)Cn . Cn = n+1 n

Exercise 10.111 [3] Let Dn be the number of derangements of the set [1..n] = {1, 2, . . . , n}. Then the sequence {Dn } satisfies the recurrence Dn = nDn−1 + (−1)n ,

n > 1,

D0 = 1.

Reduce the above non-homogeneous difference equation of the first order to the second order homogeneous recurrence and obtain its asymptotic series for n → ∞.

Exercise 10.112 [3] Find formal power series representation for solutions of the following difference equations: (a) yn+3 = n2 yn+2 − (7n + 2)yn+1 − (12n + 5)yn ,

(b) nyn+1 = (n + 1)yn .

Exercise 10.113 [3] Find asymptotic series for the sequence 4 n yn = ∑ , k=0 k n

n = 0, 1, 2, . . . .

Hint: Use the fact that yn is one of the solutions of the homogeneous recurrences: (n + 2)3 yn+2 − 12(n + 3/2)(n2 + 3n + 7/3)yn+1 − 64(n + 3/4)(n + 1)(n + 5/4)yn = 0 or n3 yn = 2(2n + 1)(3n2 − 3n + 1)yn−1 + 4(4n − 3)(n − 1)(4n − 5)yn−2 = 0.

10.7. LIMIT LAWS IN PROBABILITY

609

Exercise 10.114 [2+] Derive the asymptotic series for the sum of inverse binomial coefficients n −1 n , n = 0, 1, 2, . . . . yn = ∑ k=0 k Hint: Use the fact that yn is one of the solutions of the homogeneous recurrence: 2nyn = (n + 1)yn−1 + 2n, n = 1, 2, 3, . . .. Exercise 10.115 [2] Undergraduate courses on elementary matrix theory usually mention the following three methods for computing the determinant of a square n-by-n matrix A = [ai j ]: (a) direct computation using the definition via permutation (requires n! n − 1 computations); (b) reduction by Gaussian elimination (requires 2n3 /3 − n2 /2 + 5n/6 − 1 computations); (c) expansion by minors: det A = a11 A11 + a12 A21 + · · · a1n An1 , where Ai j are the appropriate cofactors. Let Nn be the number of computations required by method (c), then Nn satisfies the recurrence Nn = nNn−1 + 2n − 1,

n = 2, 3, . . . ;

N1 = 0.

Find the rate of growth for Nn as n → ∞. (This recurrence was solved in Exercise 6.108.) Exercise 10.116 [3+] (D. Doster) Define two sequences {yn } recursively: (a) yn+1 = (2n + 3)yn − 2nyn−1 + 8n,

(b) 4nyn = (6n − 1)yn−1 − (2n − 1)yn−2 ,

n > 1,

y0 = 1,

y1 = 3.

n > 1,

y0 = 1,

y1 = 5/4.

For both sequences, modify the recurrences to obtain a suitable formulas for their solutions, and then find an asymptotic formula for yn . Exercise 10.117 [2] (D. Newman) Let a0 , a1 be arbitrary and an = an−1 + an−2 /n(n − 1) for n > 1. Using Birkhoff-Trjitzinsky method, find an asymptotic formula for an .

10.7 Limit Laws in Probability In a typical experimental setup, which includes computer simulations, we generate a series of samples from a population, and we assume they all follow the distribution of a random variable X . Let X1 , X2 , . . . , Xn denote the observed values. By assumption, these outcomes are independent, identically distributed (i.i.d.) random variables, and their average (called the sample mean) is also a random variable, denoted by Xn =

1 (X1 + X2 + · · · + Xn ). n

(10.94)

Actually, the ratio (10.94) was used in the frequency interpretation of the probability (Theorem 3.17, page 88). It has been long observed that usually the values of X n are less volatile that those of the samples, and their dispersion seems to depend little on details of the distribution of X , beyond its variance.


610

These two perceptions have been formalized in several statements, all called laws of large numbers, that consider how close X n is to the expected value, followed by the central limit theorem, that addresses the dispersion of X n . These behavior patterns are subject to various additional assumptions on the distribution of X .

10.7.1 Laws of Large Numbers In this section, we discuss two of these theorems—the Kolmogorov Strong Law and the Chebyshev Weak Law. The derivation of the latter one is based on the Chebyshev inequality (4.44), page 188. There are two main versions of the laws: one requires the random variables in the sequence {Xn } to be independent and identically distributed; the other concerns sequences where the samples are independent, but may have distinct distributions. We state the two laws and bring a few special cases, tighter results, variants and corollaries. Theorem 10.118 Chebyshev’s (Weak) Law of Large Numbers. Let X1 , X2 , . . . , Xn be independent random variables with a common finite expected value, µ , and finite variance6 σ 2 , then for any positive number ε , Pr[|X n − µ | > ε ] → 0 as n → ∞.

An equivalent statement of the weak law of large numbers is ( 0, if t < µ , X1 + X2 + · · · + Xn 6t = lim Pr n→∞ n 1, if t > µ .

(10.95)

If the random variables in the sequence {Xn } are independent and identically distributed, then X1 + X2 + · · · + Xn 1 6µ = . we can complement Eq. (10.95) for t = µ with lim Pr n→∞ n 2

Theorem 10.119 Kolmogorov’s (Strong) Law of Large Numbers. Let X1 , X2 , . . . , Xn be a sequence of independent random variables, having the same distribution as a random variable X. h i 1. If E[X ] exists, then lim X n = E[X ] almost surely, that is, Pr lim X n = E[X ] = 1. n→∞

n→∞

2. If E[X ] does not exist, then the sequence X 1 , X 2 , X 3 , . . . fails to have a limit almost surely. ⊳

Theorem 10.120 (A. Khinchin) Suppose X1 , X2 , . . . , Xn , . . . is an infinite sequence of independent, identically distributed random variables with the same (finite) expected value µ = E[Xn ]. Then the sequence of sample means, X n = n1 (X1 + X2 + · · · + Xn ), converges to µ in probability (even if the variance, V [Xn ], does not exist). Chebyshev’s law assures that the average of n observations of a random variable converges in probability to the mean of the random variable as n → ∞. This law can be generalized to 6 This version of the law does not assume identical distributions, but identical expectations. The variances may vary, but need be finite with bounded variability.


611

the case when the first moments—not only the probability distributions of observations—are not the same, though the variances need then be uniformly bounded. Theorem 10.121 (P. Chebyshev) Let X1 , X2 , . . . , Xn be a sequence of independent random variables with expected values µ1 , µ2 , . . . , µn . If their variances V1 , V2 , . . . , Vn are uniformly bounded from above, Vk < K < ∞, for 1 6 k 6 n, then the average of X1 , X2 , . . . , Xn converges in probability to the average of their expected values: # " n K 1 n Pr ∑ Xk − ∑ µk > ε < 2 . n k=1 nε k=1

This claim is more informative than the initial weak law of large numbers, since it specifies a lower bound on the rate at which the deviation probability converges to zero. The claim of the weak law of large numbers can hold for dependent random variables as well.

Theorem 10.122 (A. A. Markov) If for (possibly) dependent random variables X1 , X2 , . . . , Xn , the following ratio converges to zero, 1 n2

n

∑ V [Xk ] → 0

as

k=1

n → ∞,

def

where V [X ] is the variance, then their average, X n = n1 (X1 + X2 + · · · + Xn ), converges in probability to the average of their expected values (so the sequence {Xk } satisfies the weak law of large numbers). Corollary 10.123 Let X1 , X2 , . . . , Xn be a sequence of uncorrelated random variables with def E[Xk ] = µ and V [Xk ] 6 K < ∞, for k = 1, 2, . . .. Then X n = n1 (X1 + X2 + · · · + Xn) converges to µ in L2 and in probability as n → ∞.

P ROOF : Using property of the sum of uncorrelated variables, Eq. (3.47) on page 131, we get E

h

Xn − µ

2 i

Kn 1 n = V X n = 2 ∑ V [Xk ] 6 2 → 0. n k=1 n

Convergence in probability follows from Eq. (4.26), page 160.

Theorem 10.124 (A. N. Kolmogorov) Let {Xn } be a sequence of independent random variables with variances σn2 , respectively. The convergence of the series

σn2 1 n

∑

(10.96)

is a sufficient condition, known as Kolmogorov’s condition, for the sequence {Xn } to satisfy the strong law of large numbers. Note that the condition limn→∞ n12 ∑nk=1 σk2 = 0, called Markov’s condition is sufficient for validity of the weak law of large numbers.

Theorem 10.125 (Weak law for triangular arrays with truncation) For each n ∈ N, let Xn,k , 1 6 k 6 n be independent random variables. For a sequence of


612

positive truncation values, {bn }, we define X n,k = Xn,k [Xn,k 6 bn ], where [Xn,k 6 bn ] is the Iverson bracket (Eq. (2.4) on page 29). Assume that as n → ∞, the following conditions hold: n

n

1. bn → ∞,

2.

∑ Pr[|Xn,k | > bn ] → 0,

k=1 n

Then, using the notation Sn =

2

3. b−2 n ∑ EX n,k → 0. k=1

n

∑ Xn,k , and an =

∑ E[X n,k ], we have

k=1

k=1

Sn − an → 0, bn

in probability. n

P ROOF : The proof is direct, and combines the known tools in a clever way. Let Sn =

∑ X n,k , k=1

then an = E[Sn ]. Since |Sn − an | > |Sn − an | in sample points where Sn 6= Sn , we have the bound Sn − an Sn − an > ε 6 Pr[Sn 6= Sn ] + Pr >ε . Pr bn bn

The first term on the right-hand side is bound with the Boole inequality: n

Pr[Sn 6= Sn ] 6 Pr[∪nk=1 {X n,k 6= Xn,k }] 6

∑ Pr[|Xn,k | > bn ] → 0, as provided by condition (2)

k=1

of the theorem. For the second term, we use Chebyshev’s inequality for X n,k because, in general, the variance is smaller than the second moment

" # Sn − an 2 Sn − an 1 >ε 6 E = 1 V [Sn ] Pr 2 bn ε bn (bn ε )2 =

n n 1 1 2 V [ X ] 6 ∑ n,k (bn ε )2 ∑ E[X n,k ] → 0, (bn ε )2 k=1 k=1

as given by condition (3), and the proof is complete. Theorem 10.126 Suppose that X1 , X2 , . . . are independent random variables with E[Xn ] = def 0 and finite variances σn2 = V [Xn ], n = 1, 2, . . .. If ∑n>1 σn2 < ∞, then the series ∑n>1 Xn converges almost surely.

Corollary 10.127 For a sequence of random variables {Xn }, let µn = E[Xn ] and σn2 = V [Xn ] < ∞. If for some sequence of positive numbers, {bn }, the ratio σn2 /b2n → 0, then Xn − µn bn

−→

n→∞

0

in probability.

P ROOF : It follows from E [(Xn − µn )/bn ]2 = b−2 n V [Xn ] → 0.


613

Theorem 10.128 (The law of the iterated logarithm) Let X1 , X2 , . . . , Xn , . . . be uniformly bounded independent random variables with common expectation, µ , and standard deviation, σ . Let Yn = (X1 + X2 + · · · + Xn − nµ )/σ , then7 # " |Yn | 61 =1 (almost surely). Pr lim sup p n→∞ 2π ln(ln n)

Theorem 10.129 (Erdös and Rényi [38]) Consider an infinite sequence X1 , X2 , . . . , Xn , . . . of independent, identically distributed and nondegenerate randomvariables, with zero expecta tXn tion. Assume that the moment generating function, φ (t) = E e , of the common distribution of the Xn exists for every t in some open interval I around the origin. Let a positive number α be defined as the solution of the equation min φ (t) e−α t = φ (τ ) e−ατ = e−1/C . t∈I

Then C = C(α ) > 0 and defining Sn = X1 + X2 + · · · + Xn for n > 1, S0 = 0, we have Sn+⌊C ln N⌋ − Sn = α = 1. Pr lim max N→∞ 06n6N−⌊C ln N⌋ ⌊C ln N⌋ Note: The number α = α (C) is uniquely determined by the common distribution of the summands X1 , X2 , . . . . It can be found as α = supx {x : J(x) 6 1/C}, where J(x) = supt {tx− ln φ (t)} and J(x) = ∞ for all those x’s where the former expression is meaningless.

Example 10.130 (Longest run) Consider a sequence, {Xn }, of independent, identically distributed random variables that take the values 1 and −1 with constant probabilities p and q = 1 − p, respectively (see Example 4.127 on page 178). Let Mn be the longest run at time n. In the special case when p = 1/2, tX we have the moment generating function to be φ (t) = E e n = cosh(t). Applying Theorem 10.129 for the sum Wn = X1 + · · · + Xn , we get

τ=

1 1+α ln 2 1−α

and

1 1+α = ln(1 + α ). C 2

Passing to logarithms with base 2, it can be seen that Mn / log2 n → 1 almost surely. For biased coin, we have8 Mn / log1/p (nq) → 1 almost surely.

The Erdös-Rényi theorem claims that if a fair game is played N times, the maximum average gain of a player over ⌊C log2 N⌋ consecutive games (C > 1), tends with probability 1 to α , where α is the only solution in the interval 0 < α 6 1 of the equation 1+α 2 2 1−α 1 = 1− log2 log2 − . C 2 1+α 2 1−α 7 The

original statement (1924) of the law of the iterated logarithm is due to A. Y. Khinchin (Khintchine). article “The longest run of heads” by Mark Schilling in The College Mathematics Journal, 21, No. 3, 196 – 207, 1990. 8 See the


614

Example 10.131 (Coupon collector problem) In Example 4.46 on page 150, we introduced the random variable Tn , the time (=number of attempts) to collect a complete set of n coupons. It was shown that Tn is the sum of n independent random variables: Tn = ∑k>1 Xn,k , where Xn,k represents the time (= the number of purchased coupons) to get a new coupon other than previously bought k − 1 coupons. When each coupon is equally likely to turn up at any purchase, the Xn,k are geometrically distributed with parameter 1 − (k − 1)/n. The expected value of Tn and its variance was given in Exercise 8.85, page 478, to be def

µn = E[Tn ] = n Hn ,

def

(2)

σn2 = V [Tn ] = n2 Hn − nHn ,

(2)

where Hn and Hn are the first- and second-order harmonic numbers, respectively. Since (2) Hn ∼ ln n and Hn ∼ ζ (2) = π 2 /6 as n → ∞, we choose bn = n ln n in Corollary 10.127 to obtain Tn /n ln n → 1 in probability. This result of relative tightness of a distribution about its expected value holds whenever V [Xn ]/E 2 [Xn ] → 0. For instance, if there are n = 100 coupons, then it will take about 100 ln(100) ≈ 461 attempts to get a complete set.

Example 10.132 (Cycles in permutations) Circular permutations were introduced in §2.4, page 54. We apply the weak law of large numbers to the number of cycles in a random permutation. Every permutation of [1..n] can be decomposed into finite number of cycles. The number of elements in a cycle is called its length. This decomposition can be achieved as follows. For a given permutation π , starting with 1, we consider the sequence of values 1, π (1), π (π (1)) = π 2 (1), . . . until eventually π k (1) = 1. When it does, we say that the cycle starting with 1 has the length k. Then we take another number not included into the first cycle, and proceed in a similar manner. We repeat such approach until all elements in [1..n] are accounted for. For example, the permutation h 2, 5, 6, 7, 1, 4, 3 i has the following cycle decomposition: (1, 2, 5)(3, 6, 4, 7), with two cycles of lengths 3 and 4. Actually, every permutation can be represented as a union of cycles. By choosing a specific order of cycles, called the canonical representation (see Definition 5.8.3 on page 262), we uniquely identify the permutation. Let Sn be the number of cycles in a permutation, a random variable, with distribution induced by the assumption that each of the n! permutations is equally likely. We concern about the cycle structure of a random permutation. To apply the weak law of large numbers, we represent Sn as the sum of indicator random variables: Sn = Xn,1 + Xn,2 + · · · + Xn,n , where Xn,k is 1 if and only if a left parenthesis appears immediately before position k in the canonical representation; otherwise Xn,k = 0. In the above example, X7,1 = 1 and X7,4 = 1, all other indicator random variables are zeroes. (Note that always Xn,1 = 1.) These indicator random variables are independent and Pr[Xn,k = 1] = 1/k. To prove this, we observe that every permutation can be read as the canonical representation of the cycle structure of exactly one permutation. Hence there are n! such representations, and the same


615

distributional assumptions hold for both sets. In particular, the probability that position k holds a value larger than the k − 1 positions to its left (which would cause Xn,k to be 1) is 1/k since every possible ordering of that prefix is equally likely. Furthermore, this happens independently of the arrangement of the values to its left and right, which leads to the claimed independence of the indicator random variables. 2 ] = 1/k =⇒ V [X ] = (k − 1)/k2 . This yields the needed moTherefore, E[Xn,k ] = E[Xn,k n,k ments for Sn : n

E[Sn ] =

n

k=1 n

V [Sn ] =

1

∑ E[Xn,k ] = ∑ k = Hn, k=1 n

∑ V [Xn,k ] = ∑ k=1

k=1

1 1 − k k2

(2)

= Hn − Hn .

(2)

Then the variance V [Sn ] = Hn − Hn ∼ ln n as n → ∞. Hence Sn / ln n → 1 in probability. √ However, we can improve the estimate by applying Corollary 10.127: (Sn − Hn )/( ln n)1+ε → 0 in probability, for ε > 0.

Exercise 10.133 [2] Given two sequences of independent random variables 1 (a) Pr[Xk = ak ] = Pr[Xk = −ak ] = , 2

1 (b) Pr[Yk = ka ] = Pr[Yk = −ka ] = . 2

For what values of parameter a does the weak law of large numbers hold? For what set of values does the strong law of large numbers hold? Exercise 10.134 [2+] (Occupancy problem) In Example 8.72, page 471, we considered distribution of n distinct balls into m ordered bins. Let X be the number of unoccupied bins when n balls are randomly distributed among m bins. It was shown in Eq. (8.71) that E[X ] = (m − 1)n /mn−1 . Assuming that the ratio of the number of bins, m, and the number of distributed balls, n, approaches a constant as n increases: n/m → λ , apply Corollary 10.127 to find the limit of X /m as n → ∞.

Exercise 10.135 [4] (The St. Petersburg paradox) Reconsider the St. Petersburg paradox from Example 3.70 on page 109. For the sequence of independent random variables {Xk } having the probability mass function Pr[Xk = 2k ] = 2−k , how much should you pay to make a sequence of n games fair? The underlying idea is that in spite of the infinite expected gain, “windfall” games are very rare, which may be easy to visualize when we realize that the number of steps in a game has a mean of 2 tosses per game, and its variance is 2 as well. Alternatively observe that the probability to gain at least some large number 2N equals ∑ j>N 2− j = 21−N . Theorem 10.125 can be applied to a sequence of such games; each row in the triangle of length n corresponds to a run of n completed games. The need to find a suitable sequence {bn } drives the entire calculation.

616


10.7.2 The Central Limit Theorem This law of large numbers clarifies the “frequency interpretation of probability,” given in Theorem 3.17 on page 88. There are known several variations of the strong law of large numbers depending on conditions imposed on random variables Xk , k = 1, 2, . . . , n. One of them assesses that α X1 + X2 + · · · + Xn − µ → 0 = 1 (10.97) Pr n n

for every α < 1/2, where X1 , X2 , . . . , Xn are independent random variables with the same expectation (µ ), so that the variances are bounded by some positive constant C for all n: σ 2 (Xn ) 6 C. As a consequence (see Eq. (4.24) on page 160), we get d nα X n − µ → 0,

√ where X n is the average (10.94). The situation changes for α = 1/2 because n X n − µ converges in distribution to a normal random variable, but not to zero. This statement is known as the Central Limit Theorem (CLT), considered by many as one of the most remarkable results in all of mathematics. Theorem 10.136 If X1 , X2 , . . . is a sequence of mutually independent, identically distributed random variables with finite expected value µ and variance σ 2 , then, when n is large, the def sample average, X n = (X1 + X2 + · · · + Xn )/n, has a distribution that is approximately normal: Z z 1 2 X1 + X2 + · · · + Xn − nµ def √ 6z −→ Φ(z) = √ Pr e−x /2 dx. (10.98) n→∞ σ n 2π −∞ P ROOF : See Feller [41] or Shiryayev [124]. The remarkable feature of this result is the single limiting distribution, which is independent of the summands themselves. In particular, the summands can be discrete or continuous, it does not matter! A natural question: how big must n be for the approximation to be good? This question is nearly impossible to answer satisfactorily in general. Such approximation depends on the nature of the actual distributions of summands and in many cases that distribution is unknown. The central limit theorem exhibits the fact that the sum of large number of random variables behaves simpler than individual summands. They may compensate for each other (when there are no outliers) so that their sum distribution becomes a smooth function. This observation plays a fundamental role in many theorems about limiting distribution of the sum of random variables. Diverse problems arising in mathematics, computer science, engineering, physics, medicine, chemistry, and other areas can be modeled in such a way that the central limit theorem comes into play. The origin of the theorem can be traced back to 1733, when Abraham de Moivre used it in some gambling problems. Later, in 1812, P. Laplace and then C. Gauss extended de Moivre’s result substantially. The first rigorous proofs of the central limit theorem were obtained at


617

the end of the nineteenth century by the Russian mathematician P. Chebyshev [27] and his students A. M. Lyapunov [95] and A. A. Markov [97]. Since the beginning of twentieth century, the following statement, credited to A. M. Lyapunov, got the name—the central limit theorem—due to its generality: it does not require the sequence to be of identically distributed random variables. Theorem 10.137 (A. M. Lyapunov) Let X1 , X2 , . . . , Xn be the sequence of independent random variables, having finite moments up to the third order, and X n = n1 ∑16k6n Xk be their average. If i h n 1 3 →0 as n → ∞, E |X − E[X ]| k k ∑ n3 σ 3 (X n ) k=1 X n − E[X n ] < z tends to the distribution function of standard normal random varithen Pr σ (X n ) able: Z b 1 X n − E[X n ] 2 0. More general results [73] were obtained by A. Khinchin9 in 1929. In the 1930s, the Finnish mathematician10 Jarl Waldemar Lindeberg [90] introduced some conditions that were weaker than others but still imply convergence to a normal distribution. Later William Feller11 [40] proved that Lindeberg’s conditions could not be improved on. We present a famous corollary of the central limit theorem. As seen in §4.1.1, a binomial random variable can be expressed as a sum: B(n, p) = B1 + B2 + · · · + Bn , where the B’s are independent and each is a Bernoulli variable with parameter p (see §4.1.1). Hence, Pr[B(n, p) = k] is the probability of k successes in n Bernoulli trials with probability p for success on each trial. Moreover, E[B(n, p)] = np and V [B(n, p)] = np(1 − p), so we have the following result, known as de Moivre-Laplace limit theorem # " Z z 1 2 B(n, p) − np def 6 z = Φ(z) = √ e−x /2 dx . (10.99) lim Pr p n→∞ 2π −∞ np(1 − p)

Sometimes, it is convenient to rewrite Eq. (10.99) as ! ! a − np b − np −Φ p . lim Pr [a < B(n, p) < b] = Φ p n→∞ np(1 − p) np(1 − p)

(10.100)

Exercise 10.138 [2] Using Lyapunov’s theorem 10.137, prove the de Moivre-Laplace limit theorem. 9 Aleksandr

Yakovlevich Khinchin (1894 – 1959) was a famous Russian mathematician, who also made a great contribution to quantum statistics and information theory. 10 Jarl Waldemar Lindeberg (1876 – 1932) was a Finnish mathematician who seems to have rediscovered Student’s t-distribution. 11 Vilim Feller (Zagreb, 1906 – New York, 1970), outstanding Croatian - American mathematician.


618

Originally, the proof of the de Moivre-Laplace limit theorem was made by careful application of Stirling’s approximation to factorials (similarly to derivation of Eq. (10.11) on page 554). This leads to exponential approximations of the binomial coefficients, which we summarize in the following statement, known as the local limit theorem: def

Theorem 10.139 Let 0 < p < 1 and q = 1 − p; then 2 n k n−k 1 e−(k−np) /(2npq) pq ∼ √ k 2π npq uniformly for k such that |k − np| ∈ o (npq)2/3 .

as n → ∞

(10.101)

The sign ∼ in Eq. (10.101) indicates that the ratio of the two sides approaches 1 as n → ∞. √ The speed of convergence in Eq. (10.101) is of order O(1/ npq) due to the Berry-Esseen bound (see [124]): " # p2 + q2 B(n, p) − np . (10.102) 6 z − Φ(z) 6 √ sup Pr p npq np(1 − p) z

Therefore, the normal approximation of the binomial distribution can be poor for values of p that are close to 0 or 1 because B(n, p) is asymmetrical for such p. The bound (10.102) cannot be improved. However, for small values of p, Example 10.21 suggests that the Poisson distribution provides a good approximation instead. Because of the uniformity condition in the local limit theorem, we can rewrite Eq. (10.101) as Z b+1/2 b 2 n j n− j 1 e−(x−np) /(2npq) dx, (10.103) Pr[a < B(n, p) 6 b] = ∑ pq ∼ √ 2π npq a+1/2 j=a+1 j

where a and b are assumed to be integers, and q = 1 − p. Using p = q = 1/2 in Eq. (10.103) and appropriate bounds of the summation, we obtain the following binomial approximation: Z bk 2 n 2n e−t /2 dt, (10.104) ≈√ k 2π ak where k− n − 1 2k − n − 1 √ ak = p2 2 = n n/4

k− n + 1 2k − n + 1 √ bk = p2 2 = . (10.105) n n/4 p This approximation is valid for all k such that −a 6 k − n2 / n/4 6 a as n → ∞. and

Formula (10.99) can be used to estimate the parameters of a binomial random variable, B(n, p). Suppose that we have n observations of a Bernoulli random variable. Let X be the number of successes in these trials. Then the ratio X /n is the frequency of success in n observations, which can be used to estimate the probability, p. How big n should be chosen


619

so that, with probability 0.95, the frequency will the prescribed tolerance, ε ? In be within other words, For what n does the inequality Pr Xn − p 6 ε > 0.95 hold? Using the de Moivre-Laplace limit theorem, the above inequality is equivalent to # " √ X − np ε n > 0.95. Pr p 6 p np(1 − p) p(1 − p)

(10.106)

p For big n, the random variable Z = (X − np)/ np(1 − p) is approximately standard normal. Due to Eq. (4.18), the inequality |Z| 6 2 is satisfied with probability 0.95. Therefore, if we p √ choose n so that ε n = 2 p(1 − p), the inequality (10.106) holds. Hence n should be at least 4p(1 − p) . (10.107) n= ε2 The inequality (10.106) can also be used for estimation of the parameter p (probability of success in a binomial distribution) because in many applications p is usually not known. In this case, n is given and we need to establish the tolerance, ε . Since the function p(1 − p) does not exceed 1/4 on the interval [0, 1], we estimate ε from the equation ε 2 = 4p(1 − p)/n to obtain 1 ε6√ . n Hence from the inequality (10.106), we get that X − p 6 √1 n n

with probability 0.95.

Example 10.140 It is interesting that the central limit theorem, which we view as a “law of nature,” can be shown to imply the synthetic result called Stirling theorem; this is actually the first order of the approximation given in Eq. (2.87) on page 77. Let Wn = X1 + · · · + Xn be the sum of n independent and identically distributed random variables of zero mean and unit variance. Then Eq. (10.98) is equivalent to Z x 1 2 Wn e−u /2 du. (10.108) lim Pr √ 6 x = (1/2π ) /2 n→∞ n −∞

An additional technical result12 is a particular case of a moment convergence theorem: if a sequence of random variables {Wn }n>1 converges in distribution to a random variable W , then limn→∞ E[|Wn |r ] = E[|W |r ], for 0 6 r < 2. Using this formula in Eq. (10.108) with r = 1 and Eq. (H.9)(a) on page 729, we find that √ p (10.109) lim E |Wn |/ n = 2/π . n→∞

12 This

topic is developed in great detail in [92, Chapter IV].


620

A suitable distribution for the Xi that fits our needs is the standard exponential; it has the density f (x) = e−x for x > 0, and vanishes for x < 0. Then E[Xi] = V [Xi ] = 1, and the conditions for Eq. (10.109) are met if we use Xi − 1 instead, which provides 1/2 2 |Wn − n| √ = lim E n→∞ n π

=⇒

lim

n→∞

√

|Wn − n| √ = 2. 2π E n

(10.110)

h i √ = The rest is calculations; we need the expectation of the last left-hand side, 2π E |W√n −n| n √ Wn 2π nE n − 1 . The sum Wn is the n-way convolution of standard exponentials, hence it has the Gamma distribution Γ(n, 1) with the density (see Appendix E) xn−1 e−x /Γ(n), for x > 0: Z ∞ Wn 1 x − 1 = E − 1 xn−1 e−x dx n Γ(n) x=0 n Z n Z ∞ 1 x n−1 −x n−1 −x x = − 1 x e dx . − 1 x e dx + Γ(n) x=0 n x=n n def

Let u = x/n, then the last two integrals are evaluated as follows. With integration by parts of the first and last integrals leading from line 2 to line 3, we get Z 1 Z ∞ nn n−1 −nu n−1 −nu (1 − u)u e du + (u − 1)u e du Γ(n) u=0 u=1 Z 1 Z 1 Z ∞ Z ∞ nn n−1 −nu n −nu n −nu n−1 −nu u e du − u e du + u e du − u e du = Γ(n) u=0 u=0 u=1 u=1   1 ∞ un e−nu  2nn e−n 2nn e−n nn  un e−nu = . = − = Γ(n) n n nΓ(n) n! 0

We have found that

1

h i √ = 2π E |W√n −n| n

√ 2 2π nnn e−n , n!

and then Eq. (10.110) gives us that

√ 2π nnn e−n = 1, lim n→∞ n!

as required.

The next example relates probabilities associated with rolling dice and the continuous normal distribution. In Example 7.29 on page 370, the relation between putting balls in bins, rolling a balanced n-sided die, and sums of continuous uniform random variables was established. Example 10.141 It now becomes a simple exercise to show that there are 2,247 ways to get the sum n = 26 in 6 tosses of ordinary (m = 6) die. However, instead of calculating binomial coefficients in Eq. (7.18), page 369, we introduce the idea of normal approximation to the distribution of the sum W6 = X1 + · · · + X6 . Since the mean, µ ≡ E[W6 ] = 27 · 6 = 21 and the 35 · 6 = 17.5 can be found without a problem (see Example 3.94 on variance, σ 2 ≡ V [W6 ] = 12


621

page 114), we pick up two bounds Wmax = 26.5 and Wmin = 25.5 and calculate the corresponding Z values: Z1 = (Wmax − µ )/σ ≈ 1.3148 and Z2 = (Wmin − µ )/σ ≈ 1.0757. Then we obtain the normal approximation to be Pr[W6 = 26] ≈ Φ(1.3148) − Φ(1.0757) = 0.0468,

where 2, 247/66

Φ(z) =

R z −t 2 /2 √1 e dt, 2π −∞

an error of nearly 3% to the exact value Pr[W6 = 26] =

= 0.0482. Of course, with 6 tosses, you cannot expect to get a good approximation using the CLT. However, if we roll a fair die 600 times and ask for the probability to get the sum of n = 2600, we would need to calculate 2599 − 6 j 1 333 j 600 ≈ 0.146912 × 10−33 , Pr[W600 = 2600] = 600 ∑ (−1) j 599 6 j=0 according to Eq. (7.19) on page 369. This is a very small number, which is expected because when the number of trials increases the probability to reach any particular number is close to zero. To see how the de Moivre-Laplace limit theorem works, we change the question and ask for the probability that the number of 1’s will not exceed 70. Then we have µ = np = √ 500 1 5 600 2 6 = 100, σ = npq = 600 6 6 = 6 ≈ 83.3333, and (70 − np + 1/2)/ npq ≈ −3.231563. Applying Eq. (10.103), we have 1 Pr[X 6 70] ∼ √ 2π

Z −3.231563 −∞

2 /2

e−x

dx ≈ 0.0006155759.

Example 10.142 In §10.6.2, we considered some examples of sums of powers of binomial coefficients. For instance, Example 10.105 was devoted to find an asymptotic expansion 3 of the sum ∑nk=1 nk , which is known to have no explicit formula [107]. The BirkhoffTrjitzinsky method does not allow us to determine the exact value of the leading coefficient, but to express all other terms through this unknown one. On the other hand, the central limit theorem provides the leading approximation, but it is not easy to extend the theorem in order to find other terms in the approximation. Therefore, the combination of these two methods may lead to more accurate approximations. The objective of this example is to prove the asymptotic formula (which was first obtained by Pólya and Szegö in 1925) r !p r n p 2 nπ n as n → ∞ (10.111) ∑ k ∼ 2n nπ 2p k=0 for arbitrary positive p. The presented derivation is adapted from [39]. For any real constant a > 1 and a given integer n, let √ In,a = k : |2k − n| 6 a n .

As n becomes large, we may replace the integral in Eq. (10.104) with midpoint rule as r 2 −(2k−n)2 /2n def n 2n 2 −(2k−n)2 /2n n =2 = b(n, k), (10.112) e ≈√ √ e nπ k 2π n

622


which is valid for k ∈ In,a . It should be noted that the approximation (10.112) is only good when k is near 2n and n tends to infinity. Raising each side of Eq. (10.112) to the pth power and again approximate b p (n, k) with the integral, we obtain n p √ Z b p n k −(p/2)t 2 2 n p e dt, ≈ b (n, k) ≈ √ 2 ak k 2π

where values of ak and bk are given in Eq. (10.105). Summing over k ∈ In,a yields p n p √ Z bk 2 n n 2 √ e−(p/2)t dt. ≈ ∑ ∑ k 2 k∈In,a ak 2π k∈In,a

Now we expand summation in the right-hand side to all k (the error is small and decreases with n), which leads to the integral over all real line, and the desired formula (10.111) follows. Exercise 10.143 [1] Let Xn denote the number of singles in n tosses of a fair coin. Then ⌊(n−k)/2⌋ k+ j n−k− j−1 Pr[Xn = k] = S(n, k) 2−n , where S(n, k) = 2 ∑ j=1 . It was shown in Exk j−1 ercise 5.124, page 254, that its expected value is µ = (n + 2)/4, and the variance is σ 2 = (5n + 4)/16. Apply the central limit theorem to estimate Pr[Xn = k] for large n.

Exercise 10.144 [1+] (Rounding errors in addition) Let xk (k = 1, 2, . . . , n) be a sequence of real numbers, and let yk be their values rounded to the nearest integers. That is, yk = xk + zk . Assuming that {zk } is a sequence of independent, uniformly distributed random variables on [− 12 , 21 ), find the asymptotic distribution of the average error Z = ∑nk=1 zk . Use it to √ estimate √ for n = 400 the probability that |Z| < 3. What is the approximate probability that |Z| < 10 3? Exercise 10.145 [1+] Let {Xn } be a sequence of independent, identically distributed binomial def random variables with Pr[Xn = 1] = p, Pr[Xn = 0] = q = 1− p. The empirical proportion, pˆn is defined by pˆn = (1/n) ∑nk=1 Xk (the average of n observations). Find the least positive integer n such that Pr[| pˆn − p| 6 0.01] > 0.9. Hint: Use the normal approximation.

Exercise 10.146 [1+] In Exercise 10.145, let p = 0.3, find n such that Pr[| pˆn − p| > 0.01] < 0.05, using Chebyshev’s inequality, and compare the estimate of n with the result given by the central limit theorem. Exercise 10.147 [2] In Exercise 10.133, for what set of values of a does the central limit theorem apply? Exercise 10.148 [2] (Continuation of Exercise 3.79) In Exercise 3.79 on page 111, we found the expected value of choosing numbers with replacement from [1..n] until a digit appears def n! (k+1) twice. Find asymptotic formula for this expected value, E(n) = ∑n+1 k=1 (n−k+1)! nk as n → ∞. (n)

Exercise 10.149 [2+] In Example 6.153 on page 334, let p0 be the probability that a match of n games ends in a draw with probability of advance for each player to be 1/3. Find the (n) approximation of p0 as n → ∞. n−1

(nt)k . ∑ n→∞ k=0 k! Hint: Apply the central limit theorem for a sum of Poisson random variables.

Exercise 10.150 [4] (A. Moy) For a fixed t > 0, find lim e−nt


623

10.7.3 Random Walks The scenario for the classical random walk problem was discussed previously in Example 4.127 on page 178, where we treated a random walk as a convolution of simple binary random variables. Another interpretation of a random walk can be viewed as a model for a coin-tossing game when a gambler bets one dollar on each of a series of tosses of a coin. The total gain (or loss) of such game is a sequence of integers, Wn , n = 0, 1, 2, . . ., with W0 = 0, so that each of the next value differs by ±1 from the previous one. Hence, Wn+1 = Wn ± 1, where the sign is chosen randomly. A random walk is naturally described by a Markov chain (see §9.5) having a single communication class—the set of all integers. There are known two common ways to visualize a random walk. The oldest one of them—at least a few hundred years—considers a walker, starting at the origin, that takes a unit move on the integer grid Z. At every integer time n, the walker flips a coin and moves one unit step to the right if it comes up heads and one unit step to the left if it comes up tails. The position of the walker at time n will be denoted by Wn , with W0 = 0. To see the progress of the walker, it is convenient to draw the path associated with the walk by connecting the points (0, 0), (1,W1 ), (2,W2 ), . . . , (n,Wn ) in the plane. One of such zigzag path corresponding to 18-length word, THHHTTHHHTTTTHTHHH, is presented on the graph:

A

B

There are many problems that could be modeled with random walks, for example, they play a central role in modeling polymer chains (physical chemistry). To mention a few more, they are also used in simulation of Brownian motion and electrical networks. The ballot problem (considered in Example 2.106 on page 60), which concerns with the number of lattice paths that no step ends on the horizontal axis, is a particular case of random walks. In real life, we come across problems that are modeled by random walks with probability of winning that may not be 1/2. For instance, when one bets on red in American roulette (see Example 3.49 on page 99), the probability of winning is 18/38 ≈ 0.477.

A walk of length n (or of n steps) is an n-tuple hX1 , X2 , . . . , Xn i, where each Xk is either +1 or −1. The sum over these Xk = ±1 increments, where the subscript identifies the discrete time when (or horizontal coordinate where) Xk was sampled, creates the value Wn = ∑nk=1 Xk , which is the “position” of the random walk. Its distribution was found in (n+k)/2 (n−k)/2 def def n Eq. (4.36), page 4.36: Pn,k (p, q) = Pr[Wn = k] = (n+k)/2 p q , where q = 1 − p. The de Moivre-Laplace theorem tells us that √ √ Pr [ a npq 6 Wn − np 6 b npq ]

−→ n→∞

1 √ 2π

Z b a

e−t

2 /2

dt.


624

From the definition of Pn,k (p, q), we see that a random zigzag path can return back to 0 only when the number of steps is even. Using an approximation, due to Stirling, we get √ P2n,0 (p, q) ∼ (4pq)n / π n as n → ∞. Hence ∑n>0 P2n,0 (p, q) will converge if and only if √ ∑n>0 (4pq)n / π n does. The latter series converges when 4pq < 1 and diverges when 4pq = 1. Therefore, ∑n>0 P2n,0 (p, q) = ∞ if and only if p = q = 1/2. Thus, the chain is recurrent when p = 1/2 and transient if p 6= 1/2. When p = q = 1/2, the random walk is called k symmetric and we have Pn,−k (p, q) = qp Pn,k (p, q) = Pn,k (p, q). Later on, we consider only the case when p = q = 1/2. Let f2k be the probability of the first return at time 2k: f2k = Pr[W j 6= 0 for 1 6 j < 2k;

W2k = 0 ].

It is convenient to introduce another function, though it brings nothing new: the tour with length distribution described by f2k begins with a move up or down. Consider the former one, and the second is accounted for by symmetry. Following that move up to level 1, the tour concludes when the walk goes down, below level 1 for the first time. Let g2k+1 be the probability that the walk needs 2k + 1 moves to lose height by 1 for the first time. The same is true for gaining height of one following an initial drop, and with the symmetry, we have just shown that 1 1 (10.113) f2k = g2k−1 + g2k−1 = g2k−1 . 2 2 On the other hand, such a period governed by g consists of either an immediate drop by one, or of an increase, to level 2, say, a first return (if needed) to level 2, and then another g-duration to return to level 1. This story is told as follows:   k = 0,  12 , k g2k+1 = 1   2 ∑ f2 j g2(k− j)+1 , k > 0. j=0

def

uk

Let φ (u) = ∑k>0 g2k+1 be the ordinary generating function of the sequence {g2k+1 }. Replacing the f2 j in the recurrence by g2 j−1 , we have

φ (u) −

k 1 1 u = ∑ uk ∑ g2 j−1 g2(k− j)+1 = 2 2 k>1 j=1 2

u

∑ g2( j−1)+1 ∑ g2(k− j)+1uk− j = 2 φ 2 (u).

j>1

k> j

The quadratic equation uφ 2 (u) − 2φ (u) + 1 = 0 has two solutions φ (u) = (1 ± and because this function needs to be regular at u = 0, we pick the sign “−:” √ √ 1− 1−u def =⇒ F(u) = ∑ f2k uk = 1 − 1 − u. φ (u) = u k>0

√

1 − u)/u, (10.114)

The value for the generating function F(u) results from the equality (10.113). We can extract coefficients and find that f2k = (−1)k−1 1/2 k . This result is useful since it allows us to check easily on theshape of the distribution, which decreases fast initially, and then very slowly, as √1 . So we observe the phenomenon called decreasing failure rate, which (−1)k−1 1/2 3 k ∼ 2 πk


625

claims that the longer it takes the random walk to return to zero, the smaller becomes the probability to get there in the next13 step: Failure rate =

f2k 1 − ∑k−1 j=1 f2 j

as we used the summation formula ∑nj=0 (−1) j

1/2 j

=

1 , 2k

= 2(n + 1)(−1)n

1/2 n+1

= (−1)n

−1/2 . n

Let N2n be the number of upsteps that lie above the horizontal line during 2n steps; let also def pr (n) = Pr[N2n = 2r]. Initial values are p0 (0) = 1 and pr (0) = 0 for all r > 0. We distinguish between a first step up and a first step down, and condition on the first return to level zero in 2k steps. Unless r = 0 or r = n, there is a smallest k such that W2k = 0 for 1 6 k < n. All W j , with 1 6 j 6 2k, are either all positive or all negative. This considerations yield the full-history recurrence pr (n) =

1 n 1 n f p (n − k) + 2k r−k ∑ ∑ f2k pr (n − k), 2 k=0 2 k=0

n > 0,

r 6= 0, n.

(10.115)

For r = n (in which case the walk is positive for the entire phase), we must add the probability that W2k > 0 for all 1 6 k 6 2n, which equals to 1 2n 1 1 −1/2 1 ∞ f = = ∑ 2k 2 n 4n 2 n (−1)n . 2 k=n+1 For r = 0, we must add the same quantity that represents the probability that W2k < 0 for 1 6 k 6 2n. def

Introducing the ordinary generating function gn (u) = ∑nr=0 pr (n)ur , we derive from Eq. (10.115) that −1/2 1 n 1 1 n k n (−1)n . (10.116) gn (u) = ∑ f2k u gn−k (u) + ∑ f2k gn−k (u) + (1 + u ) n 2 k=0 2 k=0 2 def

With the bivariate PGF G(t, u) = ∑n>0 gn (u)t n , Eq. (10.116) leads to G(t, u) =

1 n 1 n k n−k f (tu) g (u)t + 2k n−k ∑ ∑ ∑ f2k (t)k ∑ gn−k (u)t n−k 2 k=0 2 n>k k=0 n>k −1/2 1 (−1)n . + ∑ t n (1 + un ) n 2 n>0

(10.117)

All the summations are straightforward, and we get, using the explicit form of F(u), a surprisingly malleable expression, G(t, u) =

1 (1 − t)−1/2 + (1 − ut)−1/2 = , (1 − t)1/2 + (1 − ut)1/2 (1 − t)1/2 (1 − ut)1/2

(10.118)

13 This term originated in reliability theory, which justifies the use of the word failure. The failure rate at time j is defined as Pr[failure at time j | the life time is at least j].


626

where the last equality is obtained by multiplying both parts by (1 − t)12 − (1 − ut)12 . Now the extraction of coefficients is again immediate and we find pr (n) = [t n ](1 − t)−12 [ur ](1 − ut)−12 −12 −12 1 2r = (−1)n = n 4 r n−r r

2(n − r)  n−r

(10.119)

This distribution has the shape shown in the diagram on the side, and its peculiar feature naturally reflects the fact that while its expected value is in the middle of the range, this is also the least likely value. The walk is almost always spending nearly the entire time in the positive or negative region. The walk does not hug level zero and cross it from time to time. This is also the lesson of the nature of the distribution f : the walk is typically far away from level zero. Let us visit the last calculation where the level of the walk at time 2n is not left unspecified. We, in fact, want to compute the distribution of N2n when conditioned on the level of the walk, of W2n . To that end we define def

pr (n; m) = Pr[N2n = 2r W2n = 2m]

m > 0

The choice m > 0 is arbitrary, but as the continuation shows, it is desirable to fix the sign of m. The basic recurrence is similar to (10.115), but we need to deal differently with the boundary cases r = 0 and r = n: pr (n; m) =

1 n 1 n f2k pr−k (n − k; m) + ∑ f2k pr (n − k; m) 0 < r < n ∑ 2 k=0 2 k=0

(10.120)

For r = 0, both sides simply vanish, as it is impossible to have N2n = 0 while W2n > 0. For r = n, we also need W2n = 2m. We complete the recurrence in Eq. (10.120) by calculating the term we need for r = n: the probability Pr[Wk 0 for 0 k n; W2n = 2m]. A direct way to compute it is to count the number of paths on the grid from (0 0) to (2n 2m) that do not cross (but may touch any number of times) level zero. The simplest way to do it is using the reflection principle, Lemma 2.80 on page 52. This yields def

hn;m = Pr[Wk

0 for 0

n; W2n = 2m] =

k

m 2n 4−n  n n−m

This probability replaces the last term in Eq. (10.117) by its PGF, which equals 1 − (1 − u)12 ηm (u) = ∑ hn;m u = u n m n

m




627

Note that the kernel is exactly the PGF φ (u), which was determined in Eq. (10.114). Using it, we find Gm (t, u) =

(1 − ut)1/2 − (1 − t)1/2 2η (ut) = 2 . η (ut) t(1 − u) (1 − t)1/2 + (1 − ut)1/2

(10.121)

Extracting coefficients here for m > 0 is not a simple matter and not surprisingly does not lend itself to a closed expression. We quote the result of [28]: 2 j 2n−2 j r m j−m n− j . (10.122) Pr[N2n = 2r | W2n = 2m] = 2n ∑ j(n − j + 1) n−m j=m However, the formula for m = 0 seems to be more remarkable: pr (n; 0) = 2[t n ur ]

(1 − ut)1/2 − (1 − t)1/2 (1 − ut)1/2 − (1 − t)1/2 = 2[t n+1 ur ] . t(1 − u) 1−u

Since only r 6 n is meaningful, the first part in the numerator cannot contribute, and the extraction of [ur ] is the same for all r: 1/2 1/2 n+1 r (1 − t) =2 (−1)n , r ∈ [0..n]. pr (n; 0) = −2[t u ] n+1 1−u We formulate this result, as well as two others, in separate statements. Theorem 10.151 (Chung-Feller) Consider the 2n n possible zigzag paths (also called Dyck n-paths) starting from the origin and ending at the point (0, 2n), consisting on n upsteps (1, 1) and n downsteps (1, −1). The number of these paths with k upsteps above the abscissa is the same, regardless of the value k, 0 6 k 6 n. Hence, this distribution is uniform on these n + 1 values—the statement is usually referred to as the Chung-Feller theorem [28]. Let Sk denote the set of lattice paths having k upsteps, ր, above the x-axis,where each such path starts at the origin and terminates at (0, 2n). For n = 3, there are 63 = 20 total such zigzag paths. W. Feller provided in his book [41, §III.9] a direct proof of Theorem 10.151 by the reflection method for r = 0, n, and by induction for intermediate values. The article [68] gives a leisurely presentation of random walks, but allows up- and down-steps to have different probabilities, which changes the properties of the walk drastically. There are also some articles [25, 112] and books [86, 88] that consider many similar problems. Theorem 10.152 (Arcsine law) Let Tn be the number of points where zigzag paths are above abscissa: Tn = { k : Wk > 0 and 0 6 k 6 n }, which is the number of tosses where the number of heads exceeds the number of tails. For any real number α such that 0 < α < 1, the distribution of Tn has the following limiting form, called the arcsine law:

lim Pr[Tn < nα ] =

n→∞

1 π

Z α 0

dx p

x(1 − x)

=

√ 2 arcsin( n). π


628

n = 3,

n = 3,

|S0 | = 5

|S1 | = 5

|S2 | = 5

|S3 | = 5

Theorem 10.153 Let Rn = { k : Wk = 0 and 0 6 k 6 n } be the number of returns to zero during n-length path (or the number of times the walker visits the origin). For each α > 0, r Z √ 2 α −t 2 /2 e dt. Pr[ Rn < α n ] −→ n→∞ π 0 Exercise 10.154 [1] In a random walk, show that the random variable R, the time of first return to level zero, has no expected value.

Exercise 10.155 [1] Let Rn be the number of times that the walker visits the origin. Show that its expected value is unbounded. Exercise 10.156 [1+] For symmetric (when the probability to move right is p = 1/2) random walk Wn = X1 + X2 + · · · + Xn considered in Example 4.127 on page 178, show that Z x 1 2 Wn e−t /2 dt. lim Pr √ 6 x = √ n→∞ n 2π −∞ How would the formula above changed for arbitrary p? Exercise 10.157 [2+] Consider a random walk on the d-dimensional integer grid, Zd , which is the set of d-tuples hx1 , x2 , . . . , xd i, where the xi are all integers. The walker starts at the origin and at each integer time moves to one of the nearest points, each with probability (1) (2) (d) 1/2d. Let Wn = hWn ,Wn , . . . ,Wn i be the position of the walker after n steps. (a) Show that E[ |Wn |2 ] = n regardless of the dimension. (b) Show that for d = 2, the random walk is recurrent (always returns to the origin), but it is transitive for d > 2. (c) On the plain (d = 2), show that a nonsymmetric walk is always transient.

Chapter 11

Asymptotics and Generating Functions In Chapters 6 – 8, we have seen several applications of generating functions (GFs) where we disregard the issue of convergence of the defining series because it plays no role. The series were viewed as formal power series. The question whether the ordinary or exponential generating function has a meaning as an expansion of an analytical entity needs not even be considered—the answer was not essential for generating functions used so successfully. In this chapter, we change a dominant theme in our music since it is precisely the question of convergence, including analyticity, and the location and type of singularities, that play a leading role in deducing asymptotic information from generating functions. This musical change will provide some sweet harmonies. Also we present some statements without proofs (formulated as theorems) from advanced calculus and the theory of power series. More information can be found in [14, 103, 105]. Note: Much of the material in this chapter relies on the concepts developed in Chapter 12, especially §12.3. We recommend a review of functions of a complex variable.

11.1 Elementary Bounds from Generating Functions Let { fn } be a sequence of numbers and f (z) be its ordinary generating function, f (z) =

∑

fn zn .

(11.1)

n>0

What information do we get from the simple fact that the series converges? We can get the limit of fn from f (z) by using the Abel theorem L.86, page 740. Let us examine the evidence: we know that the series ∑k>0 zk converges to (1 − z)−1 only for |z| < 1. And the similar geometric series, ∑k>0 (cz)k equals (1 − cz)−1 only in the circle |z| < 1/c. In the first 629

630

CHAPTER 11. ASYMPTOTICS AND GENERATING FUNCTIONS

case the coefficients of zk in the series were all 1, and in the second case the latter series has coefficients ck . Surely, we can conclude that in order for the series in Eq. (11.1) to converge in some circle |z| < R, it is sufficient that fn < Rn , at least beyond some finite index n0 since the values of any finite number of elements do not impact the series convergence property. They definitely change the value of the limit, but not the fact that it converges. This R is called the radius of convergence of the series. Theorem 11.1 The radius of convergence of a power series F(z) = ∑k>0 ck (z − z0 )k is the distance of the singular point of the function F(z) closest to z0 . A more precise expression of the radius of convergence is Cauchy’s nth-root convergence test. It says that for the series in Eq. (11.1) to have the radius of convergence R, the following one-sided limits must exist: R = lim| fn |−1/n

or

R−1 = lim| fn |1/n .

(11.2)

The symbols1 lim and lim are also denoted by ‘lim inf’ and ‘lim sup.’ Sometimes the D’Alembert ratio test is easier to apply than the Cauchy root test. It is given as follows: the sum in Eq. (11.1) has the radius of convergence R if and only if fn+1 1 , (11.3) = lim R n→∞ fn

whenever the limit exists.

When the radius of convergence for the generating function is known, we have a rough estimate of the coefficients: Theorem 11.2 Let the ordinary generating function f (z) of the sequence { fn } converge in the circle |z| < R, but diverge for some point on the boundary |z| = R; then for any ε > 0

fn = O((R − ε )−n).

(11.4)

Notes: (a) When all the coefficients are positive, which is usually the case in calculations that result from analysis of algorithms, the smallest singularity will be at the real value z = R. (b) The fact that the series diverges on the boundary also says that the coefficients are bounded from below by (R + ε )−n, that is, fn = Ω((R + ε )−n ). Recall that a function f (z) is regular in the domain D if and only if it is differentiable in the domain D. A complex-valued function is said to be an entire function if it is holomorphic everywhere on the whole complex plane (that is, at all finite points). The functions sin z and ez are examples of an entire function, but (1 − z)−1 is not because it has a singularity at 1 The

usual names for lim fn and lim fn , without the absolute value operator, are the lower and upper limits of the sequence. The definition of the lower limit says that it is the largest number which is smaller than all but a finite number of fn (or smaller than all fn , n > n0 ). The upper limit is defined accordingly. If they are equal, the sequence is simply said to have a limit, denoted by lim fn .

11.1. ELEMENTARY BOUNDS FROM GENERATING FUNCTIONS

631

z = 1. This function is regular in any domain that does not contain z = 1; for instance, in the annulus 2 < |z| < 3. However, it does not have the same power development in this region (it can be represented in the entire ring by a Laurent expansion with non-zero principal part, or by several Taylor expansions), see Example 12.22 on page 672. Definition 11.3 Let f (z) = ∑n>0 fn zn be a regular function. If, for arbitrary ε > 0, the coefficients { fn } grow slower than ε n , that is, fn ∈ o(ε n ), then the function f is referred to as super-exponentially small. ⊳ Corollary 11.4 If a power series F(z) = ∑n>0 cn zn converges on the entire plane, then the sequence {cn } of its coefficients is super-exponentially small, namely, they grow slower than ε n for arbitrary positive real ε . Example 11.5 Suppose fn = cn + O( n1 ) then we cannot claim that the corresponding generating function f (z) =

∑

n>0

fn zn =

(cz)N

∑ (c z)n + · · · = 1 − cz + · · ·

n>N

has the radius of convergence R = 1/c. For example, the sequence fn = (1/5)n + 1/n2 satisfies the above constraint, but it only converges for |z| < 1, not |z| < 5. However, if fn = cn 1 + O( 1n ) , then the radius of convergence for the series ∑n>0 fn zn would be 1/c. Example 11.6 The ordinary generating function for the Catalan numbers (see Eq. (6.47) on page 300) is √ 1 − 1 − 4z . b(z) = 2z

This function has only one branch singularity z = 1/4 where the root becomes zero because z = 0 is a removal singularity. Therefore the Catalan numbers (6.48) grow slower than (4 + ε )n . √ 1 − z ± 1 − 2z − 3z2 Similarly, the generating function for the Motzkin numbers T p,2 (z) = 2z has two branch √ singularities z = −1 and z = 1/3. Hence the Motzkin numbers grow as 1 − z ± 1 − 2z − 3z2 = O(3 + ε )n as n → ∞. [zn ] 2z The Fibonacci ordinary generating function F(z) = z/(1 − z − z2 ) has two poles: z = (−1 ± √ n 5)/2. Therefore the radius of convergence of the Fibonacci √ series n F(z) = ∑n>0 Fk z is √ ( 5 − 1)/2 and the Fibonacci numbers grow slower than 1+2 5 + ε . In fact, these num √ n bers grow as √15 1+2 5 . √ Example 11.7 Let f (z) = a − z. Since the function g(z) = z1/2 is not differentiable at the point z = 0, it is not regular there. Such a singularity is called a branch singularity (see Definition 12.50 on page 680). Hence f (z) has a branch singularity at z = a, and it is regular


632

in the circle |z| < a with radius R = a. Calculations show that √ z 1/2 √ 1/2 z j = a∑ . − f (z) = a 1 − a a j j>0 Therefore

n √ √ 1/2 1 n 1/2 = a (−1) − a−n = P(n) a−n , fn = a a n n

where P(n) =

√

a (−1)

1/2 n ,

Exercise 11.8 [1] Show, by expanding

n

1/2 . n

that P(n) ∼

−1 2n

p

a nπ

(11.5)

√

= − 2√aπ n−3/2 .

Exercise 11.9 [1] Inserting the result of Exercise 11.8 into the estimate (11.5), we find √ a a−n −n fn = P(n) a ∼ − √ 3/2 . 2 π n So all coefficients are negative! As we mentioned previously, our analysis usually turns up positive ones. Explain. Do you recall a combinatorial generating function very similar to the estimate (with a = 1/4)? Exercise 11.10 [1+] Given an entire function, for instance, ez or z31 − 12, what does the discussion in this section allow us to conclude about the coefficients of a GF? Exercise 11.11 [2] In each of the following cases, an OGF ∑n fn zn equals the given function f (z). Identify its radius of convergence and estimate the coefficients fn : (a)

z3 + 1 , z3 − 1

(b) tan z,

(c)

1 ez − 3

,

(d) ln(z − 3),

(e)

z3 + 1 , z3 − 1

(f)

1 . (z4 − 1)1/3

11.1.1 The Lagrange Inversion Formula We have seen previously that many generating functions are defined through functional equations, rather than explicit expressions. For instance, enumeration of trees in §8.1 yields some algebraic equations for ordinary or exponential generating functions. If some equation F(x, w) = 0 has the only one root (in some region), then this root can be denoted by x = ϕ (w). More precisely, according to implicit function theorem from calculus, this solution can be represented as a power series x = ∑k>1 ck wk if F(0, 0) = 0 and first partial derivatives of the analytic function F(z, w) do not vanish. This series can be used to determine the asymptotic behavior of ϕ (w) as w → 0. In general, not too much is known for arbitrary function F(x, w), so we turn our attention to a particular case analyzed in §6.7. Therefore, we consider applications of the Lagrange inversion theorem 6.183 to asymptotic problems by presenting some examples.

11.1. ELEMENTARY BOUNDS FROM GENERATING FUNCTIONS

633

Example 11.12 Consider the positive solution of the equation x e2x = w, when w → 0. Recall that the Lagrange equation x = w ϕ (x) has a solution represented as a power series d k−1 wk where ck = k−1 ϕ k (z) . x = ∑ ck , k! dz z=0 k>1 Using ϕ (x) = e−2x , we obtain ck = (−2k)k−1 , k = 1, 2, . . .. The series gives also an asymptotic representation: x ∼ w − 2w2 + 6w3 − · · · as w → ∞.

Example 11.13 Consider the equation x = e−3wx , which is not in the form required by the Lagrange inversion theorem. So we make substitution x = z + 1, and then take the logarithm of both sides to obtain w=

z , ϕ (z)

where

ϕ (z) = −

3z(1 + z) . ln(1 + z)

Application of Eq. (6.114), page 351, gives us the asymptotic expansion of x to be ( ) 3z(1 + z) k wk d k−1 = 1 − 3w + 9w2 + O(w3 ). − x = 1+ ∑ k−1 k! dz ln(1 + z) k>1 z=0

Example 11.14 The following equation sin x = α x cos x,

where α is a positive parameter, is come across in shock problems in elastodynamics. Looking at the behavior of the function tan x = sin x/ cos x, we see that it intersects with the line describing the function f (x) = α x closer and closer, as x increases, to (2n + 1)π /2 for any integer n. Denoting this root by xn , we set x = w−1 + z, where w−1 = 2n+1 2 π , n = 0, ±1, ±2, . . .. n n Using these notations, we get sin x = sin 2n+1 2 π + z = (−1) cos z and cos x = −(−1) sin z. Substitution into the given equation yields w=−

z α sin z = , cos z + α z sin z ϕ (z)

def

where ϕ (z) = −z(cos z + α z sin z)/α sin z. Since ϕ (z) is an even function, we do not expect odd powers of z in its Maclaurin’s expansion: 1 2 1 4 z(cos z + α z sin z) = 1+ α − z + z + O(z6 ). sin z 3 45 Application of the Lagrange inversion theorem yields 3 2 2 1 1 1 2n + 1 4α + − 2 π− − xn = + O(n−5 ). 2 α (2n + 1) π α 3 α 2α (2n + 1) π


634

Exercise 11.15 [2] (de Bruijn [24, §2.3]) Consider the equation cos x = x sin x , which has a single solution in each interval x ∈ (nπ , (n + 1)π ), for all n ∈ Z. Denote such a solution by xn , and obtain its power series development, in terms of n−1 . Exercise 11.16 [2] For a positive integer r, find an asymptotic expansion of the root (which 1 1 1 ζ r+1 = 1 + (k − 1)ζ r+1 is unique inside a small circle) of the equation ζ = + 1 − k k k as k → ∞.

11.2 Estimates from Singularities In this section, we add precision to the results of the previous one. The radius of convergence of a power series ∑ j>0 (z − z0 ) j is determined by the location of the singularity nearest to the point of development, z0 here. In most of our applications we use the origin, z = 0, as the point of development, and then the nearest singularity also has the smallest absolute value. Hence, when a function has several singularities, the one nearest to z0 is the dominating singularity (= makes the dominant contribution to the asymptotics of the coefficients).

11.2.1 Poles Recall from Definition 12.51 on page 681 that a singularity of a function at z = a is a pole of order k when the function behaves near the singularity as u(z)/(a − z)k , where k is a natural number and u(z) is analytic in a domain containing a. If k = 1, the singularity is called a simple pole. For example, the functions f (z) = sin z/z2 and g(z) = cos z/z2 have at z = 0 a √ simple pole and a second order pole, respectively. A root like z has the branch singularity point at z = 0, and the function displays quite a different behavior, which we look into later. Theorem 11.17 Let f (z) have its smallest (in absolute value) singularity at z = a, and let the singularity be a k-order pole. We can write then f (z) ∼ u(z)(a − z)−k for some regular function, u(z), in a neighborhood of z = a, which yields def

fn = [zn ] f (z) ∼ A

nk−1 −n−k a , (k − 1)!

as n → ∞,

where

A = lim u(z). z→a

(11.6)

If a = 0, then f (z) has a k-order pole at z = 0, that is, f (z) = u(z)z−k , with a regular function u(z), then fn = un+k . P ROOF : The derivation is an interesting exercise. We use the standard rules for extraction of coefficients we have seen, and concentrate on keeping the significant terms. Since u(z) is regular at z = a, we can write u(z) = ∑i>0 ui (a − z)i , and then z i−k def , fn = [zn ] f (z) = [zn ]u(z)(a − z)−k = [zn ] ∑ ui (a − z)i−k = [zn ] ∑ ui ai−k 1 − a i>0 i>0

11.2. ESTIMATES FROM SINGULARITIES

635

according to the binomial theorem (2.52), i i−k −k i n+k−i−1 −n −k−n =a ∑ ui a (−a) = a . ∑ ui a n n i>0 i>0

(11.7)

The dependence of the last binomial coefficient on i is such that it is nonzero for 0 6 i < k, and for i > n + k. With the information provided by Exercise 11.18, we are left to consider k−1

fn =a−k−n ∑ ui ai i=0 k−1

=a−k−n ∑ ui ai i=0

(n + 1)(n + 2) · · · (n + k − i − 1) (k − i − 1)!

nk−i−1 + (1/2)(k − i)(k − i − 1)nk−i−2 + · · · , (k − i − 1)!

and as n increases, the leading term, in the i = 0 summand, provides the main contribution: =u0

nk−1 −n−k a 1 + O(n−1 ) . (k − 1)!

Finally, since u0 = limz→a u(z), Eq. (11.6) is proved. The derivation points the way for calculating successive terms, of lower order in n. Exercise 11.18 Show that the knowledge about the Laurent representation of f (z) around the nearest singularity at z = a provides sufficient information about the function u(z) to conclude that the second interval ofi values (i > n + k), contributing to the sum (11.7), satisfies a−n−k ∑i>k+n ui ai n+k−i−1 = o(1) as n → ∞. n

Exercise 11.19 [1+] Compute the next-order term for [zn ] f (z) in Eq. (11.6), that is, the coefficient of nk−2 , by inspecting the derivation. Consider now an example where a function has more than one singularity. We can see there the reason for the claim made in Theorem 11.1: the singularity closest to the origin provides the leading terms. Example 11.20 Let f (z) = ∑n>0 fn zn be the Maclaurin series for the function 2 . It has two simple poles, at z = 2 and z = 4. The coefficients fn can be f (z) = (z − 2)(z − 4) determined using the binomial theorem, following a partial fraction decomposition: z n 1 z n 1 1 1 1 1 1 1 − ∑ . − = ∑ − = f (z) = z − 4 z − 2 2 1 − 2z 4 1 − 4z 2 n>0 2 4 n>0 4 Then n

fn = [z ]

(

z n 1 z n 1 − ∑ 2 ∑ 4 2 n>0 4 n>0

)

=

1 1 1 1 − . − = 2n+1 4n+1 2(n+1) 2(n+1) 1

The term 2−(n+1) in the brackets is exponentially small compared with the ‘1’ there, and we normally disregard it. We see then that it is indeed the singularity at z = 2 that makes the dominant contribution.

636


Example 11.21 Here we look at another function with two singularities, but this time they are at the same distance from the origin: Let f (z) = (z2 − 4)−1 . Then straightforward rewriting as a series produces 2 j z 2 j 1 1 1 z 1 1 = − =− = − f (z) = 2 ∑ 4 ∑ 2 . z −4 4 1 − z2 4 j>0 4 j>0 4 Then straightforward extraction of coefficients gives ( z 2 j − 41 212k = −4−k−1 , n = 2k, 1 n n = [z ] f (z) = − [z ] ∑ 4 0, n = 2k + 1, j>0 2

∀k > 0.

Exercise 11.22 [1] Redo the extraction of coefficients of f (z) = (z2 − 4)−1 , by doing a partial fraction decomposition. Note that there were no asymptotic issues in the last example. The following theorem summarizes what we have seen in the last few examples. Theorem 11.23 Let f (z) be a meromorphic function in the disk |z| < R. Assume that f (z) has k poles inside this domain at a1 , a2 , . . . , ak , of orders m1 , m2 , . . . , mk , respectively. Then Eq. (11.6) can be generalized to provide the following leading term of the coefficients fn : k

fn =

∑

j=1

Pj (n) + O(R−n ) , anj

(11.8)

where each function Pj (n) is a polynomial in n of degree m j − 1.

It is, however, lucid enough in the quite common case of simple poles: when z = a j has multiplicity 1 (m j = 1), then Pj (n) = −

1 1 Res f (z) = − lim (z − a j ) f (z), a j z=a j a j z→a j

j = 1, 2, . . . , k,

(11.9)

which is a constant (in n). Calculating the polynomials Pj (n) is usually not a simple task, but it can be done to the order needed. Residues are further considered in Theorem 12.52, page 681. The contribution of the pole closest to the origin dominates all others, and determines the exponential rate of growth (or decline) of fn . If there are several poles closest to the origin (so they belong to the same circumference), their contributions are of the same order. Hence, all these singularities should be taken into account due to possible cancellations. Example 11.24 Consider now a function that has an infinite number of (isolated) poles: f (z) =

1 . 3 − ez

The poles are at the points where the denominator equals zero. That is, at the solution of the equation ez − 3 = 0, and there are infinitely many roots z = ak , where ak = ln 3 + 2π ik,

k = 0, ±1 ± 2, . . . .


637

The pole nearest to the origin is a0 = ln 3 = 1.098612289 . . .. The other poles get promptly further away; for example, |a1 | = | ln 3 + 2π i| ≈ 6.378508178, more than six times further off. Each ak is a simple pole since the first derivative of ez − 3 at each ak is nonzero: d z (e − 3) = eak = eln 3+2π ik = 3 · e2π ik = 3 6= 0. dz z=ak Therefore, using, for example, the formula (12.26) on page 681, we get Res f (z) = lim

z→ak

ak

1 1 1 = = −e−ak = − . (3 − ez )′ −eak 3

The values are the same for all poles ak ! Now we can apply Eq. (11.8) to estimate fn . Choosing regions |z| < R with increasing R captures more and more poles, leading to more accurate expansions for fn (as the error terms get successively smaller): 1 1 1 1 · n = ∑ n+1 fn = ∑ − − 3 ak ak 3 k ak k 1 1 1 1 + + ∑ = 3(ln 3)n+1 3 k>1 (ln 3 + 2kπ i)n+1 (ln 3 − 2kπ i)n+1

= =

2 1 (ln 3)−n−1 + ℜ ∑ (ln 3 + 2kπ i)−(n+1) 3 3 k>1 " # 2π ki −(n+1) 1 −n−1 1 + 2ℜ ∑ 1 + (ln 3) . 3 ln 3 k>1

(11.10)

In regard to the last example, note that the addition of each term improves accuracy. The reason for this is a large separation of the poles. Exercise 11.25 [2] For f (z) of Example 11.24, obtain the exact values of fn , for n = 5, 10, 20 (using MAPLE is probably the easiest), and then the estimates from Eq. (11.10) using k = 1, 2, 3. Exercise 11.26 [2] Using the bivariate generating function for the Stirling numbers of second kind zn 1 def f (x) = , F(x, z) = n ∑ 1 − x(ez − 1) n>0 n! find an asymptotic estimate of fn (x) =

∑ ∆k 0n xk k>0

as n → ∞.

The following theorem is useful in a large number of cases. As given, it provides no error terms for the estimate. Theorem 11.27 Let f (z) = ∑n>0 fn zn = A(z)B(z), where A(z) and B(z) have the power series expansions A(z) = ∑n an zn and B(z) = ∑n bn zn , respectively. Let A(z) and B(z) have the respective radii of convergence α and β , and assume the following conditions hold:


638

1. α > β > 0; bn−1 = b exists, and n→∞ bn

2. The limit lim 3. A(b) 6= 0. Then

fn = (a ⋆ b)n ∼ A(b) bn

as n → ∞.

P ROOF : See [14]. Remark. It is not hard to see the source of this result; since α > β , the factor B(z) has a smaller singularity, which is the dominating one, that determines the behavior of fn : it is essentially bn , with the factor A(z) adding only a multiplicative constant. When the radii of convergence are equal, α = β , this theorem cannot be used since the results can be wrong. Here is an extreme example: let A(z) = (1 − z)−1 and B(z) = (1 + z)−1 . Hence bn = (−1)n , and the limit b is clearly −1. A(−1) = 1/2 6= 0, and therefore using the theorem would suggest fn = (−1)n /2. However, 1/2 1/2 1 1 n n n = [z ] + = (1 + (−1)n ) , [z ] f (z) = [z ] 2 1−z 1−z 1+z 2 which is quite different and does not converge to the result above. Note that if we reversed the roles of A(z) and B(z) in this erroneous derivation, we get a different—and wrong—answer! Theorem 11.27 is effective, however, in the following case. Example 11.28 Let f (z) = ecos z (z − 2)−2 . We view f (z) as the product of the two functions A(z) = ecos z and B(z) = (z − 2)−2 . The function A(z) is entire, with radius of convergence α = ∞. The function (z − 2)−2 is analytic in the circle |z| < 2, with β = 2. Furthermore,

B(z) =

1

= z 2

4 1− 2

1 2

∑k k>1

zk−1 n+1 = ∑ n+2 zn . k 2 n>0 2

bn−1 n+1 and b = lim = n+2 n→∞ bn 2 2. Since A(b) = A(2) = ecos 2 6= 0, all the conditions necessary for Theorem 11.27 to hold are satisfied, and we have def

Hence the power series coefficients of B(z) are [zn ]B(z) = bn =

fn ∼ ecos 2 (n + 1) 2−(n+2) ≈ −0.1040367091(n + 1)/2n . Example 11.29 This example makes a more nuanced use of Theorem 11.27. Consider the function 1 − 2xz 1/2 1+ √ , x > 0, (11.11) f (z) = 1 − z(1 + x) 1 − 4xz2


639

depending on x as a parameter. The Maclaurin coefficients of f (z) are known from Eq. (L.82) ⌊k/2⌋ k r k on page 746, but not in a closed form: f (z) = ∑k>0 fk z , where fk = ∑ x . The r r=0 √ function f (z) has the two obvious singularities z1 = (1 + x)−1 and z2 = (2 x)−1 , where the two denominators in Eq. (11.11) vanish. Since we keep x positive, the singularities always satisfy the inequality z1 < z2 , and they coincide when x = 1. 1/2 with B(z) of Theorem 11.27, and It makes sense then to identify the factor 1−z(1+x) √ with A(z). We also fix α = (2 x)−1 and β = (1 + x)−1 , which yields α > 1 + √1−2xz 1−4xz2 β > 0 (x 6= 1).

Looking for the coefficients bn of B(z) and their limit ratio b, we find

bn = [zn ]B(z) =

1 1 1 n [z ] = (1+x)n 2 1 − z(1 + x) 2

=⇒

bn−1 1/2 (1 + x)n−1 1 = b, = = bn 1/2 (1 + x)n 1+x

and we need not worry about a limiting process. Now we have, assuming first 0 6 x < 1, that 1 1 − 2x 1+x 1 + x − 2x 1−x = 1+ p = 1+ p = 1 + 1 = 2. A(b) = 1 + q 2 − 4x 2 4x (1 + x) (1 − x) 1 − (1+x) 2

and Theorem 11.27 tells us that

1 fk ∼ 2 (1 + x)k = (1 + x)k 2

as k → ∞.

(11.12)

If x > 1 things change: while z1 is still smallerpthan z2 , and the formulas for the radii of convergence, and bn and b are unchanged, we get (1 − x)2 = −(1−x) and A(b) = 1−1 = 0. Hence the third condition of the Theorem fails—and it gives us no clue for a good estimate of fn . Example 11.30 (Infinite die rolling). A die is rolled repeatedly, and we sum the scores. Let pn be the probability that this cumulative sum ever hits the value n. What is limn→∞ pn ? It is not even obvious a priori that a limit exists. Surely, every pn is not zero since we could have a (highly unlikely) sequence of rolls which are all 1’s, and thus hit every value. But can we say something more constructive? Consider first the following averaging argument: the values that the cumulative sum actually achieves, what are their successive differences? The average score of rolling a standard die is 31/2, so we can say that by the time the sum exceeded the (large value) N, the die was rolled def approximately vN = N/31/2 times. This sequence visited vN values, and the likelihood of any particular value to get hit is then vN /N = 1/31/2 = 2/7. This would be then the limit, and it seems reasonable, because we also observe that the value obtained at each roll are equally likely. . . , or does it? On a careful consideration, we find that the uniform distribution of


640

values is immaterial: the waveraging argument would hold true even if the die is horribly biased, but keeps the expected values 31/2. Let us generalize the setting: the die has s sides and it is weighted to produce the score i (i = 1, 2, . . . , s) in probability wi , which sum to 1. Some regularity considerations merit attention, now that we have a biased die, where some of the wi can be zero. If they are all zero except some one i with wi = 1, the limit of pn does not exist, unless i = 1 as well, and then the value for all pn is 1. This is similar to the more general issue of periodicity: if all the values of i for which wi > 0 have a common factor (their greatest common divisor) r > 1, then again no limit exists since zero values for every pn , where n is not a multiple of r, appear among non-zero, r-spaced values. From now on we shall assume that enough wi are non-zeroes to provide a greatest common divisor for them which is 1. The PGF of the die score is ∑si=1 wi zi , and the probability of scoring n in exactly d rolls is d [zn ] ∑si=1 wi zi . We sum over d for the probability of ever hitting the value n and find !d s 1 i n = [zn ] pn = [z ] ∑ ∑ wi z s 1 − ∑i=1 wi zi d>0 i=1 Therefore, the OGF, ∑n>0 pn zn , is 1−∑s1 wi zi . To tell about the desired limit, we need, as is i=1 often the case, to consider the asymptotics of pn , and this in turn requires examining the singularities of the generating function. An obvious singularity is at z = 1. This is a simple pole, as we shall show. Recall that a function f has a simple zero in a point a if f (a) = 0 and f ′ (a) 6= 0. Why is this? Consider the Taylor series development of f in a neighborhood of a. Then the function 1/ f (z) will have a simple pole at a. def

Let Q(z) = 1 − ∑si=1 wi zi be the denominator of GF, and then we see Q(1) = 1 − ∑i wi = 0, but Q′ (1) = − ∑i iwi = −E < 0, where we have used this opportunity to define E, the expected score of a roll of this die. There can be no other singularities with |z| 6 1 since for such a z 6= 1, we have | ∑si=1 wi zi | < ∑si=1 |wi zi | 6 ∑si=1 wi = 1. Since we required several wi to be non zero, the denominator will vanish for no z on or in the unit circle except 1. All the other singularities must be outside that circle. Let Q(z) have u such distinct singularities, at z = rk , and let mk be the multiplicity of the root rk ; we single out r1 = m1 = 1. Since Q is a polynomial of degree s, we know that ∑uk=2 mk = s − 1. Consider now the PFD of this GF: u 1 Rk (z) 1 = ′ +∑ , Q(z) Q (1)(z − 1) k=2 (z − rk )mk

where we need not concern ourselves with any details of the polynomials Rk (n) (except possibly note that the degree of each Rk (n) is smaller than sk ). Since z = 1 is the smallest root, 1 + lower order terms, in we get there the dominant asymptotic contribution, pn = [zn ] −E(z−1) o(1): " # u u 1 R (z) Sk (n) 1 1 1 k = [zn ] +∑ + = → pn = [zn ] ∑ n m Q(z) E(1 − z) k=2 (z − ri ) k E k=2 rk E


641

So the averaging argument above led us to a correct result, in spite of its informality. For the standard die, the expected score E = 31/2 = 7/2, so we find that lim pn = 2/7. Exercise 11.31 [1+] For the function (11.11), find asymptotic expression as n → ∞ for fn = [zn ] f (z) when x = 1. Exercise 11.32 [2+] In the context of Example 11.29, for what complex values of x can we apply Theorem 11.27 to obtain an asymptotic approximation of fn = [zn ] f (z)? Exercise 11.33 [2] In Example 6.185, page p 352, an alternative OGF was developed for the central trinomial numbers, as f (w) = 1/ (1 + w)(1 − 3w). Determine the leading term of an asymptotic expression for tn = [wn ] f (w), using Theorem 11.27. Exercise 11.34 [2] Let g(z) be an entire generating function, and h(z) be given in terms of g(z) as 1 g(z) ln . h(z) = 2−z 2−z

Give an asymptotic estimate of hn = [zn ]h(z) as n → ∞.

11.2.2 Difference Equations: Asymptotics from GFs The generating function approach is a powerful technique for solving recurrences (which was shown in §6.4). However, extracting coefficients from a corresponding generating function may not be possible in many cases. So we want to address the problem: how to find asymptotics of a solution to the recurrence directly from generating function avoiding extraction of coefficients? Constant coefficient difference equations were considered in §6.4.1 where it was shown that the ordinary generating function for the corresponding sequence is represented as a ratio of two polynomials, see Eq. (6.56) on page 306. In particular, the generating function of the second order difference equation an+2 = pan+1 + qan , n > 0, is A(z) =

a0 + (a1 − pa0 )z . 1 − pz − qz2

Application of Theorem 11.23 yields the following result. Theorem 11.35 Suppose that the ordinary generating function A(z) of the sequence {an }n>0 P(z) . Let zmin be is a ratio (irreducible) of two polynomials with real coefficients: A(z) = Q(z) the closest null to the origin of the denominator Q(z). • If zmin is a simple root of Q(z) = 0 then an = [zn ]A(z) ∼ −

P(zmin ) Q′ (zmin ) zn+1 min

as

n → ∞.

642

CHAPTER 11. ASYMPTOTICS AND GENERATING FUNCTIONS • If zmin is a real root of Q(z) = 0 of multiplicity m and P2 (z) (z − zmin )m P1 (z) + , m (z − zmin ) Q(z)

A(z) =

where P1 (z) = p0 + p1 z + · · · + pm−1 zm−1 and P2 (z) is some polynomial, then an = [zn ]A(z) ∼ −

(−1)m zn+m min

m−1

∑

pj

j=0

−m j z n − j min

as

n → ∞.

Example 11.36 We start with a simple recurrence an+2 = 4an+1 + 5an ,

n > 0,

a0 = a1 = 3.

Of course, we know its solution from Eq. (5.8), page 211, to be an = 2(−1)n+1 + 5n ,

n = 0, 1, . . . .

The ordinary generating function for the sequence {an } is A(z) = P(z)/Q(z), with P(z) = 3 − 9z and Q(z) = 1 − 4z − 5z2 . The denominator has two roots z1 = −1 and z2 = 1/5, the latter one is the closest to the origin. Calculations show that P(zmin ) = P(1/5) = 3 − 9/5 = 6/5 and Q′ (1/5) = 6/5, hence P(1/5)/Q′ (1/5) = 1 and we get the asymptotic estimate to be an ∼ 5n as n → ∞.

Example 11.37 In Exercise 10.117 on page 609, you were asked to find an asymptotic behavior of the solution to the recurrence an = an−1 + an−2 /n(n − 1), n > 2, using BirkhoffTrjitzinsky method. We reconsider this difference equation using generating function approach.

Let A(z) be the OGF for the required sequence: A(z) = ∑n>0 an zn . We consider the auxiliary function y(z) = (1 − z) A(z). Since its second derivative is y′′ = −2A′ (z) + (1 − z) A′′ (z), we have (1 − z)y′′ = −2(1 − z)A′ (z) + (1 − z)2 A′′ (z)

= −2(1 − z) ∑ an nzn−1 + (1 − z)2 ∑ an n(n − 1)zn−2 n 2 n−1

n

n−2

= ∑ an n(n − 1)z n

− 2 ∑ an n z n

+ ∑ an n(n + 1) zn . n

Changing the index of summation, we obtain (1 − z)y′′ = ∑ an+2 (n + 1)(n + 2) − 2an+1 (n + 1)2 + an n(n + 1) zn n

Using the recurrence, we get (1 − z)y′′ = ∑ an+1 (n + 1)(n + 2) + an − 2an+1 (n + 1)2 + an n(n + 1) zn n


643

= ∑ [−an+1 n(n + 1) + an + an n(n + 1)] zn n

= ∑ [−an−1 + an ] zn = A(z) − z A(z) = y(z). n

Hence the function y(x) = (1 − z) A(z) satisfies the second order differential equation with variable coefficients: (1 − z)y′′ = y(z). From the course on ordinary differential equations, we know that this equation has two linearly independent solutions, one of which has the power series expansion: y(z) =

∑ cn

n>0

(1 − z)n n!

=⇒

y′′ =

(1 − z)n−2 (1 − z)n−1 = ∑ cn+1 . (n − 2)! (n − 1)! n>0

∑ cn

n>0

Substitution into the differential equation, we conclude that cn should be a solution of the difference equation of the first order: cn+1 =

1 cn n

=⇒

cn =

c , (n − 1)!

n > 1,

where c is a constant. This gives us one solution, denote it by v(z), of the differential equation in “explicit form:” n (1 − z)n . v(z) = ∑ 2 n>1 (n!) To find a second linearly independent solution, denote it by u, we use the Bernoulli method and put u = vw. Substituting u = vw into the equation (1 − z)u′′ = u, we see that w must be a solution of the equation vw′′ + 2v′ w′ = 0

v′ w′′ . = −2 w′ v

=⇒

Integration yields ln w′ = −2 ln v = ln v−2 . Hence w(z) =

Z

dz v2 (z)

=⇒

u=v

Z

dz . v2 (z)

Therefore the general solution of the differential equation for y(z) becomes y(z) = C1 v(z)

Z z dz 0

v2 (z)

+C2 v(z).

The unknown constants are determined from the initial conditions: C2 v(0) = y(0) = A(0) = a0 , C1 +C2 v′ (0) = y′ (0) = A′ (0) − A(0) = a1 − a0 . v(0)


644

Using Abel’s theorem L.86, page 740, we finally get lim an = lim (1 − z) A(z) = lim y(z) = C1 = v(0) (a1 − a0 ) − a0 v′ (0) z→1 z→1 ! ∞ ∞ n (n − 1) . + a = a0 1 − ∑ 1 ∑ 2 2 n=1 (n!) n=2 (n!)

n→∞

Exercise 11.38 [2] For each ordinary generating function, a(z) = ∑n>0 an zn , of the sequence {an }n>0 given as a ratio of two polynomials, find asymptotic expansion for an as n → ∞. 6z3 − 8z2 + 32z − 10 , 6z4 − 7z3 − 8z2 + 13z − 4 1 + 5z (e) , 6z2 + 5z − 6

3+z , 2 3z − 4z + 1 4 − 2z , (d) 15z2 + 4z − 4

(b)

(a)

(c) (f)

1

4z2 − 1

,

2z . z2 − 4

Exercise 11.39 [2+] For a given difference equation, find asymptotic expansion for its solution as n → ∞. 1. an+2 + 2an+1 + 4an = 0, 2. an+2 = 2an+1 − an ,

a0 = 0, a0 = 0,

3. an+1 = pan + q(1 − an ),

a1 = 2.

a1 = 2.

a0 = 1,

4. an+3 − 9an+2 + 26an+1 − 24an = 0,

p + q = 1. a0 = a1 = 1,

a2 = −3.

11.2.3 Removal of Singularities We show here a simple method to improve the estimate provided by Theorem 11.23, by looking at the information that remains in a generating function after a pole—when it is the smallest singularity—is removed. Lemma 11.40 Let f (z) =

u(z) , (z − a)m

m ∈ Z+ = {1, 2, . . .},

where u(z) is a regular function in a neighborhood of the point z = a. We define h(z), the m-remainder of f (z), as h(z) = f (z) − g(z),

g(z) =

m−1 ( j) u (a) q(z) . q(z) = ∑ j! (z − a) j . (z − a)m j=0

Then the remainder h(z) = f (z) − g(z) is regular in a neighborhood of the point z = a. The usefulness of the lemma follows from Theorem 11.2. We know that [zn ] f (z) = [zn ]g(z) + [zn ]h(z). We also know how to get explicit coefficients from g(z), for example, using the binomial theorem, or even direct asymptotics from Theorem 11.17. Then, since h(z) usually


645

has a larger radius of convergence than f (z) does, it provides a smaller asymptotic correction. The qualifying ‘usually’ comes here to cover the possibility that f (z) has another singularity, of the same size |a|. When that is the case, one needs to work harder and remove both singularities to get the promised improvement. ecos z , with a = 2, m = 2, and u(z) = ecos z . Example 11.41 Let us reconsider f (z) = (z − a)2 Then q(z) = u(z)|z=a + u′ (z) z=a (z − a) = ecos 2 − sin 2 ecos 2 (z − a) = ecos 2 (1 − (z − a) sin 2) ,

and g(z) = Then the remainder is

ecos 2 (1 − sin 2(z − a)) , (z − a)2

h(z) = f (z) − g(z) =

ecos 2 (1 − sin 2(z − a)) ecos z − . 2 (z − a) (z − a)2

This h(z) has no other singularity, hence it is an entire function. As a consequence of Theorem 11.2, we have that for arbitrary R, hn ∼ o(R−n ). This is expressed by saying that h(z) is super-exponentially small (Definition 11.3 on page 631). Using fn = gn + hn , we find 1 − (z − 2) sin 2 (z − 2)2 z cos 2 n 1 + 2 1 − 2 sin 2 = e [z ] 2 4 1 − 2z n n −1 −2 1 1 1 cos 2 + 2 sin 2 · − e − = n 4 2 2 n 2+n−1 1 cos 2 −n e 2 + 2 sin 2 = 4 n ecos 2 (n + 1 + 2 sin 2), = 2n+2 which is certainly an improvement over the estimate we derived in Example 11.28. fn ∼ gn = [zn ] ecos 2

The following transfer theorem is a generalization of Lemma 11.40. Theorem 11.42 Let both f (z) = ∑n>0 fn zn and h(z) = ∑n>0 hn zn be generating functions with dominating singularities at z = a. Moreover, as z → a we have f (z) = O(h(z)),

or

f (z) = Θ(h(z)),

or

f ∼ h,

then fn = O(hn ),

or

fn = Θ(hn ),

or

fn ∼ hn .

646


Proving (see [44]) this theorem is not easy since it expresses relations between (high order) derivatives of functions in terms of the growth rates of the functions. The fact that it can at all be shown, for some families of functions, is another testimony to the rigidity of regular functions. p Exercise 11.43 [2] Let f (z) = − (1 + z)(1 − 4z) = ∑n>0 fn zn , find the first two leading terms in the asymptotic formula for fn as n → ∞.

11.2.4 Darboux Theorem and its Relation to the Binomial Theorem Definition 11.44 A function f (z) has an algebraic singularity at z = θ , if in some annulus around this point f (z) can be represented as a sum of two terms, one—if it exists—is regular at z = θ and the other has the form

z −w , g(z) 1 − θ where g(z) is regular inside a circle |z − θ | < ε , for some ε > 0, with g(θ ) 6= 0, and w is any real or complex number other than a negative integer. ⊳ When w is a negative integer, the entire term is of course regular at z = θ , and when it is a positive integer, the singularity of f (z) at θ is a pole. Theorem 11.45 (Darboux) Let f (z) = ∑n>0 fn zn be a regular function at z = 0 that has a finite number of singularities on its boundary of convergence |z| = R, all of them are either algebraic singularities or poles. Let f (z) has the following asymptotic behavior near the singularity θk , k = 1, 2, . . . m:

z f (z) ∼ 1 − θk

−wk

gk (z),

z → θk ,

with some regular function gk (z) in an neighborhood of z = θk . Let W = maxk ℜ(wk ), then fn =

1 n

m

∑ k=1

gk (θk ) nwk + o R−n nW −1 , n θk Γ(wk )

(11.13)

where the sum runs over all singularities on the circumference |z| = R. The simplest function with an algebraic singularity is the binomial function f (z) = (z − a)β , where β is real number, but not a positive integer. This function has a singularity at z = a only, a branch point or a pole.

From the equation (z − a)β = (−a)β (1 − z/a)β , the Darboux theorem tells us that fn = (−1)n (−a)β −n

n−β −1 + o(a−n n−β −1 ). Γ(−β )


647

On the other hand, the binomial theorem claims that Γ(n − β ) (−1)n β (−1)n n−β −1 . (−a)β −n = · = fn = n n (−a)n−β (−a)n−β Γ(n + 1)Γ(−β ) We apply Stirling approximation for the Γ-function for large values of argument keeping only the leading terms to obtain Γ(n − β ) (n − β − 1)n−β −1 e−n+β +1 = eβ +1 ∼ Γ(n + 1) nn+1/2 e−n

n−β −1 n

n−β −1/2

1 nβ +1

.

As n increases, we have lim

n→∞

n−β −1 n

n−β −1/2

−β − 1 −β −1/2 −β − 1 n 1+ = lim 1 + n→∞ n n −β −1/2 −β − 1 = e−β −1 . = e−β −1 lim 1 + n→∞ n

Therefore, 1 Γ(n − β ) ∼ β +1 Γ(n + 1) n

and we have fn ∼

1 (−1)n , n− β (−a) Γ(−β )nβ +1

which agrees with the above result. This is an example of the general observation that Darboux theorem is a restatement of the binomial theorem (for asymptotic estimates), with a bound on the needed corrections. The following example shows a limitation of the Darboux theorem: if the true coefficient fn is very small (especially when it equals a true zero), the estimate of Eq. (11.13) can generate a large relative error. Example 11.46 The exponential generating function for the Bernoulli numbers, Bn , is ˆ = B(z)

∑ k>0

Bk

z zk = z . k! e − 1

ˆ This function, B(z), like the function we considered in Example 11.24, has a countable number of simple poles, at z = ±2π ni, where n = 1, 2, . . .. The denominator vanishes at the origin, ˆ but this is a removable singularity since B(0) = 1. Therefore, this series converges inside the ˆ has two simple poles on its boundary, θ1,2 = ±2π i. We apply the circle |z| < 2π and B(z) Darboux theorem to obtain estimates of the Bernoulli numbers. In our case, we have w1,2 = 1, hence w = min{w1 , w2 } = 1 and z ˆ = lim z 1 − z/θ1 = θ1 −1/θ1 = −1. B(z) g1 (θ1 ) = lim 1 − z→θ1 z→θ1 θ1 ez − 1 eθ1

648


The same equality holds for g2 (θ2 ) = −1. Therefore (−1) · n (−1) · n Bn 1 + + o((2π )−n ). = n! n Γ(1)(2π i)n Γ(1)(−2π i)n

(11.14)

This provides for B2n the known asymptotic value: B2n = −2(2n)!(−4π 2 )−n + o((2n)!(2π )−2n ), whereas for B2n+1 we only get o((2n + 1)!(2π )−(2n+1) ), rather than the correct value zero. Exercise 11.47 [2] Improve asymptotic estimate for the Bernoulli numbers by finding the next term in Eq. (11.14). Exercise 11.48 [3] Let Bn (x) be the nth Bernoulli polynomial, and α be a fixed real number so that 1 < α < e2π ≈ 535.491656. For such value of α there exists the largest real root of the equation Bn+1 (1 + x) = α Bn+1 (x). Show that it has the following asymptotic representation: 1 α +1 n +O as n → ∞. + x= ln α 2(α − 1) n

11.2.5 Logarithmic Singularities The expansions we have seen so far are not useful for functions that display logarithmic behavior near their dominating singularities. For example, if the generating function f (z) is “purely logarithmic,” as when f (z) = ln[(z − a)β ] = β ln[z − a], then the Taylor expansion f (z) = ∑n>0 fn zn allows us to perform the extraction of coefficients and obtain

β −n a , n > 1. (11.15) n The situation becomes entirely different when a generating function has a combination of a logarithmic factor with other functions. We present below the main result obtained by P. Flajolet and A. Odlyzko in the paper [44], for generating functions in which the dominating singularity is a product of both algebraic and logarithmic terms. Theorem 11.49 (Flajolet and Odlyzko) Let ∆z (φ , η ) be the domain that has the shape of a notched disk: (11.16) ∆z (φ , η ) = {z : |z| 6 1 + η , | arg(z − 1)| > φ } f (z) = ln[(z − a)β ]

=⇒

fn = −

for some η > 0 and 0 < φ < π /2. Suppose that f (z) is a regular function (without singularities) in the domain ∆z (φ , η ) except the point z = 1. If for some real numbers a 6= 0, β , and δ f (z) = O((1 − z)a L((1 − z)−1 )), as z → 1 in ∆z (φ , η ), (11.17)

where

L(u) = (ln u)β (ln ln u)δ , then fn ≡ [zn ] f (z) = O(n−a−1 L(n)).

(11.18)

11.3. ESTIMATES FROM ENTIRE FUNCTIONS

649

11.3 Estimates from Entire Functions We have seen how the fact that a function has a finite radius of convergence gives asymptotic information, in Theorem 11.2, and the various results of §12.2 refined these results. What can we do when a function is entire? Recall that an entire function has only one singular point at infinity. We need another route. Let us recall Cauchy theorem 12.44 on page 679: H du f (z) = 21π i C f (u) u−z , for a function that is regular in some neighborhood of the point z. If we have a generating function with a power series expansion f (z) = ∑n>0 fn zn , which is regular around the origin, we can use the theorem by differentiating both sides of the contourintegral equation n times, and evaluating at z = 0. The Maclaurin coefficients of f (z) can be determined by the following formula, also called after Cauchy: 1 fn = 2π i

I

f (z)

C

dz , zn+1

Cauchy formula,

(11.19)

where C is a simple closed counterclockwise loop inside the region of regularity. This is not an asymptotic result, it is exact, and holds for all n. The reason the Cauchy formula is presented here is that even we do not know how to evaluate the contour integral, methods have been developed to produce asymptotic estimates for it when n increases, and f (z) satisfies certain common properties.

11.3.1 Integration by Parts In some cases integration by parts can provide a simple and often effective way to determine the asymptotic expansion of an integral containing a big parameter. For instance, the incomplete Gamma function admits the following asymptotic power series as x → ∞: Z ∞

def

Γ(α , x) =

−t α −1

e t

x

dt = −

Z ∞

t α −1 d e−t

x Z ∞ ∞ α −1 −t t α −2 e−t dt e + (α − a) −t t=x x ∞ α −1 −x α −2 −t e − (α − 1)t e + (α − 1)(α x t=x α −1 −x α −2 −x

= =

− 2)

Z ∞ x

t α −3 e−t dt

+ (α − 1)x e + · · · + (α − 1) . . . (α − n + 1)xα −n e−x + εn (α − 1)n α − 1 (α − 1)(α − 2) + + · · · + + εn , = xα −1 e−x 1 + x x2 xn = x

e

where n is an arbitrary integer, and the remainder term can be bound for n > α − 1, n

εn = (α − 1)

Z ∞ x

−t α −n+1

e t

n α −n−1

dt < (α − 1) x

Z ∞ x

e−t dt = (α − 1)n xα −n−1 e−x .

It is hard to produce a tighter estimate of εn . However, for n > α − 1, the integrand is monotonically decreasing, and the bound above shows it is smaller than the last retained term.


650

The same approach allows us to find the asymptotic behavior as λ → ∞ of a Laplace transform: Z ∞ def def e−λ t g(t) dt (11.20) gL (λ ) = L[g](λ ) = 0

of the function g(t). We assume that the function g(t) is smooth and the integral in the righthand side of Eq. (11.20) exists for both g(t) and its derivatives. Integration by parts yields Z 1 ∞ g(t) d e−λ t g (λ ) = − λ 0 ∞ Z ∞ 1 −λ t e−λ t g′ (t) dt = − e g(t) + λ 0 t=0 L

=

g(n−1) (0) g(0) g′ (0) + εn + 2 + ···+ λ λ λn

for any positive integer n. Therefore the Laplace transform of g(t) for large λ is fully determined by the behavior of g(t) in the neighborhood of the origin and gL (λ ) admits the following asymptotic series: gL (λ ) ∼

∞

g(n) (0) ∑ n+1 n=0 λ

as

λ → ∞.

A natural extension of the latter result gives the famous Watson’s Lemma: Lemma 11.50 (G. N. Watson, 1918) Let f (t) be a function of the positive real variable t such that ∞ f (t) ∼

∑ ck t p+kq

k=0

as t → 0,

(11.21)

where p and q are positive constants, and | f (t)| 6 C eat for some constants C and a. Then def

L[ f (t)](λ ) =

Z ∞ 0

e−λ t f (t) dt ∼

∞

1

λ p+1

ck

∑ Γ (1 + p + kq) λ kq

k=0

as

λ → ∞,

(11.22)

provided that this integral converges throughout its range for all sufficiently large λ . P ROOF : Formal proof is obtained by substitution of the series for f (t) into the integral and application of integration. See details in [62], Vol. 2. If a parameter λ in the formula (11.22) is a pure imaginary number, we come across to the finite Fourier integral Z b

eint g(t) dt,

(11.23)

a

in which a, b, and g(t) are independent of the positive parameter n. Assuming that the function g(t) is smooth, we integrate by parts to obtain Z b a

eint g(t) dt =

i i h g(a) eina − g(b) einb + ε1 (n) n

11.3. ESTIMATES FROM ENTIRE FUNCTIONS m−1

=

∑ (−in)−k−1

k=0

h

651

i eina g(k) (a) − einb g(k) (b) + εm (n),

where

εm (n) = (−in)−m

Z b

(11.24)

eint g(m) (t) dt

a

decreases as o(n−m ) with n → ∞. The formula (11.24) can be used to find the asymptotics of sine and cosine Fourier transforms: def

Fc ( f )(n) =

Z ∞ 0

def

Fs ( f )(n) =

Z ∞ 0

Z ∞

1 f ′ (t) sin(nt) dt = − Fs ( f ′ )(n), n 0 1 1 f (t) sin(nt) dt = f (0) + Fc ( f ′ )(n). n n

1 f (t) cos(nt) dt = − n

Sometimes the integration by parts technique is ineffective as the following example shows. Example 11.51 Let us consider the Fourier cosine transformation of the function 1/(t 2 + 1): Z∞ cos(nt) π 1 dt = e−n . = Fc 2 2 t +1 t +1 2 0 Fortunately we know its exact value, but the method yields 0 0 0 1 (n → ∞). ∼ + 2 + 3 + ··· Fc 2 t +1 n n n This result is useless. Exercise 11.52 Use integration by parts technique to prove the asymptotic expansion √ 2 1 k eix (1 + i) π 2 √ −i C(x) + iS(x) ∼ ∑ (ix2 )k , x → ∞ (x > 0) 2x k>0 2 2 for the Fresnel integrals: def

C(x) =

Z x 0

cos(t 2 ) dt,

def

S(x) =

Z x 0

sin(t 2 ) dt.

Exercise 11.53 [2] The Sievert integral is defined by def

S(n, θ ) =

Z θ 0

e−n/ cos t dt

(n > 0,

0 6 θ 6 π /2).

Find first three terms in the asymptotic expansion for the function S(n, π /2). Exercise 11.54 [2] (Continuation of Exercises 3.79 and 10.148) In Exercise 10.148 on page 622, you were asked to use probabilistic methods to determine the asymptotic behavior of E(n), the expected number of calls for picking integers (with replacement) from [1..n] until one of them will appear twice. In this problem, you need to represent E(n) as a definite integral, and then find three terms in its asymptotic expansion.


652

The inverse result to Watson’s lemma is known as the Tauberian theorem2 . Theorem 11.55 Let f (t) be non-decreasing function on [0, ∞) and such that its Laplace transform Z ∞ def e−λ t d f (t) f L (λ ) = 0

converges for λ > 0 and for some non-negative number γ let f L (λ ) ∼ Then f (t) ∼

A λγ

as

At γ Γ(γ + 1)

λ → ∞. as t → 0.

P ROOF : See [136]. We turn our attention to the functions I(n) =

Z b a

e−nh(x) g(x) dx

(11.25)

that appear in many applications. Here h(x) is a real-valued function that increases monotonically on some interval (a, b). Now we consider the case when the function h(x) attains the global minimum at one of the ends of the interval [a, b], say at x = a. So we assume that h′ (x) exists and that h′ (x) > 0 (recall that h(x) monotonically increases). Moreover we assume that h(x) > h(a) for x ∈ (a, b]. In this case the asymptotic expansion of the integral (11.25) is obtained by integration by parts: Z b Z b g(x) −nh(x) d e e−nh(x) g(x) dx = − ′ a a nh (x) Z g(x) −nh(x) x=b 1 b −nh(x) g(x) ′ dx e =− ′ e + x=a nh (x) n a nh′ (x) Z g(a) −nh(a) g(b) −nh(b) 1 b −nh(x) g(x) ′ e − ′ e + dx. e = ′ nh (a) h (b) n a nh′ (x) Since h(b) > h(a), the last two terms decrease faster that the first one and we obtain Z b a

e−nh(x) g(x) dx ∼

g(a) −nh(a) e . nh′ (a)

(11.26)

We are temptated to apply the Watson lemma to the integral (11.25) instead of doing tedious integration by parts. Without loss of generality, we may assume that h(a) = 0 because factoring out the exponential term we obtain −nh(a)

I(n) = e 2 After

Z b a

e−n[h(x)−h(a)] g(x) dx

Alfred Tauber (1866 – 1942). The theorem formulated below was proved by Karamata in 1931.


653

Setting t = h(x) − h(a), we get −nh(a)

I(n) = e

Z b

−nt

e

a

′

−nh(a)

g(x(t)) x (t) dt = e

Z b a

e−nt f (t) dt

with f (t) = g(x(t)) x′ (t) = g(x(t))/h′ (x(t)) (h′ (t) > 0). Since we need to solve the equation t = h(x) − h(a) for x to obtain x(t), such approach does not look promising at the first glance. Fortunately, the Lagrange inversion theorem (§6.7), and the Lagrange-Burmann formula (6.122) on page 354 allows one to find the power series expansion for f (t). Example 11.56 Let us consider the integral Z ∞

def

I(n) =

0

x

e−nxe xex dx

for big n. We set t = xex that defines x = x(t) as a function of t, the Lambert function (§12.6). Then the integral becomes Z ∞ t dt I(n) = e−nt x e +t 0 because dt = (ex + x ex ) dx = (ex + t) dx. Since ex = t/x, we can rewrite the integral as Z ∞

I(n) =

0

e−nt

x dt. x+1

To expand the function x/(x + 1) into power series with respect to t, we use Eq. (6.114), page 351, with F(z) = z/(z + 1) and φ (z) = e−z : d n−1 −nz 1 e dzn−1 (z + 1)2 z=0 n−1 n − 1 − j−1 n−1 n = ∑ (−1) t ∑ n ( j + 1)!. j j=0 n>1

tn x =∑ x + 1 n>1 n!

Application of the Watson lemma yields Z ∞ 0

−nxex

e

x

xe dx ∼

∑

k!

k+1 k>1 n

k−1

(−1)

k−1

∑

j=0

k − 1 − j−1 k ( j + 1)!, j

n → ∞.

11.3.2 Asymptotics of Hypergeometric Series Recall from §2.6 that a hypergeometric function is defined by a series, f (x) = ∑n>0 cn xn , with a signature, cn+1 /cn , being a ratio of two polynomials. We start with an important particular case: nk + α1 nk−1 + · · · + αk cn+1 . (11.27) =A k cn n + β1 nk−1 + · · · + βk


654

Theorem 11.57 Suppose the sequence of positive real numbers, {cn }n>0 , satisfies the relation (11.27) with α1 6= β1 then cn ∼ c An nα1 −β1

as n → ∞,

(11.28)

for some positive constant c. Note: There is not enough information to determine the value of c. P ROOF : Since the sequence {cn } satisfies the first-order recurrence (11.27), its asymptotics can be obtained using the Birkhoff-Trjitzinsky method (see §10.6.2), however, we prove it cn directly by showing that the limit lim n α −β is non-zero. Taking the logarithm, we see n→∞ A n 1 1 that (11.29) lim (ln cn − n ln A − (α1 − β1 ) ln n) n→∞

exists. So we have to show that for any ε > 0 there exists an integer N such that for any n > N and each positive integer m > 0 the following inequality def

In,m = | ln cn+m − ln cn − m ln A − (α1 − β1 ) ln(n + m) + (α1 − β1 ) ln n| < ε . holds. To use the ratio test, we need to evaluate 1 + α1 n−1 + · · · + αk n−k cn+1 = A f (n−1 ), =A cn 1 + β1 n−1 + · · · + βk n−k

where f (x) = Taking the logarithm, we get ln

1 + α1 x + · · · + αk xk . 1 + β1 x + · · · + βk xk

cn+1 = ln cn+1 − ln cn = ln A + ln f (n−1 ). cn

Expanding f (x) into Maclaurin series, we obtain f (x) = 1 + (α1 − β1 )x + (α2 − β2 + β12 − α1 β1 )x2 + · · · .

Application of logarithm yields ln f (x) = (α1 − β1 )x +

α2 − β2 + β12 − α1 β1 +

(α1 − β2 )2 2 x + ··· . 2

Therefore, for sufficiently small x and sufficiently large N, for n > N, we have the estimate: ln cn+1 − ln cn − ln A − (α1 − β1 ) 1 < C . n n2 Applying this inequality for n = 1, n + 2, . . . , n + m, and adding the results, we obtain In,m = | ln cn+m − ln cn − m ln A − (α1 − β1 ) ln(n + m) + (α1 − β1 ) ln n|


655

n + m = ln cn+m − ln cn+m−1 + ln cn+m−1 − · · · − ln cn − m ln A − (α1 − β1 ) ln n 1 6 ln cn+1 − ln cn − ln A + (α1 − β1 ) n 1 + ln cn+2 − ln cn+1 − ln A + (α1 − β1 ) n + 1 ··· ··· ··· ··· ··· ··· ··· 1 + ln cn+m − ln cn+m−1 − ln A + (α1 − β1 ) n + m m−1 1 n + m − ln + |α1 − β1 | ∑ k=0 n + k n ! m−1 1 m−1 n + m 1 − ln + |α1 − β1 | ∑ 6C ∑ . 2 k=0 n + k n k=0 (n + k)

n+m 1 1 Since the series ∑∞ k=1 k2 converges, any finite sum ∑k=n k2 is infinitesimally small when n is large enough. The second sum can be estimated basedonthe asymptotic behavior of the nth n 1 1 . Indeed, harmonic number (2.1): Hn = ∑ = ln n + γ + O n k=1 k n+m−1 m−1 1 1 n−1 1 1 ∑ n + k = ∑ k − ∑ k = Hn+m−1 − Hn−1 = O n . k=1 k=1 k=0

Exercise 11.58 [2] In context of Theorem 11.57, find asymptotic estimate of cn when α1 = β1 , but α2 6= β2 .

Theorem 11.59 Let ℜa > −1, and let g be a continuous, complex-valued function defined on semi-closed interval (0, 1] having expansion g(x) = xa

∑ ak xk , k>0

Then the integral f (z) =

Z 1 0

x → 0,

x > 0.

(1 − x)z−1 g(x) dx

(11.30)

(11.31)

defines a function that is analytic in ℜz > 0 and admits the asymptotic expansion with respect to the scale of rising factorials: f (z) ∼

Γ(z) Γ(z + a + 1)

∞

∑ aj

j=0

Γ(a + j + 1) , (z + a + 1) j

z → ∞.

(11.32)

P ROOF : The asymptotic formula (11.32) is obtained by formal substitution of the series (11.30) and integration. See details in [62]. We consider Gaussian hypergeometric function and determine its asymptotic behavior when one parameter (in denominator) tends to infinity.


656

Theorem 11.60 For fixed parameters a, b, and c, the Gaussian hypergeometric function admits the asymptotic expansion with respect to 1/zn : a, b an bn cn 1 , z → ∞. (11.33) c ∼∑ F (a, b; z; c) ≡ F 2 1 n! zn z n>0 P ROOF : is based on integral representation of the hypergeometric series: Z 1 a, b Γ(z) (1 − cx)−a xb−1 (1 − x)z−b−1 dx c = F Γ(b) Γ(z − b) 0 z

and then application of Theorem 11.59 yields the result. See details in [62].

11.3.3 The Laplace Method In this section, we consider the integral (11.25) for n → ∞. In Eq. (11.25) either a or b or both may be infinite. We assume that g(x) and h(x) are smooth functions in [a, b] and h(x) has a unique minimum in the interval. The conditions do not restrict the generality in analysis of algorithms because our functions are invariably well behaved, and if h(x) has a finite number of minima, we can break up the range of integration into a finite number of subintervals, in each of which h(x) attains a unique minimum. The classical result for such an integral is due to Laplace (1820) who argued that its main contribution comes from the neighborhood of the global minimum of h(x), and that its dominance increases with n. The peak value of the factor e−nh(x) occurs either at end points of the interval or at the critical point x = x0 where h′ (x0 ) = 0. This critical point may coincide with one of the ends of the interval [a, b]. In order to obtain the asymptotic behavior of the integral (11.25) as n → ∞, we need to consider separately all these cases when the critical point is inside the interval, coincides with one of the ends of the interval, or the global minimum is attained at one of the ends of [a, b]. We start with the case when the critical point x = x0 is an interior point and for simplicity we assume3 that h′′ (x0 ) > 0. For n large and positive, the integration along small interval [x0 − δ , x0 + δ ] makes the dominant part of I(n), and therefore we can continue as follows: Z x0 +δ x0 −δ

−nh(x)

e

g(x) dx ≈ g(x0 )

Z x0 +δ

e

x0 −δ

≈ g(x0 ) e−nh(x0 ) ≈ g(x0 ) e−nh(x0 ) 3 If

−nh(x)

dx = g(x0 )

Z x0 +δ x −δ Z 0∞ −∞

n

e− 2 (x−x0 ) n

e− 2 (x−x0 )

Z x0 +δ x0 −δ

2 h′′ (x ) 0

2 h′′ (x ) 0

1

e−n[h(x0 )+ 2 (x−x0 )

2 h′′ (x )+··· ] 0

dx

dx

dx

the second derivative vanishes as well we need to look at higher order derivatives at x0 . If the lowest-order derivative which is not zero is positive, this is a suitable point: it will provide the dominant contribution to the integral. Note however, that as we need to consider higher-order derivatives, the rate of change of I(n) with increasing n is slower.


= g(x0 ) e−nh(x0 )

s

657

2π . n h′′ (x0 )

Therefore Z b a

−nh(x)

e

−nh(x0 )

g(x) dx ∼ g(x0 ) e

s

2π n h′′ (x0 )

(a < x0 < b),

(11.34)

if x0 is an interior point. The dominance of the peak is used in using only g(x0 ) to represent the role of g(x), keeping only the two leading terms in the expansion of the exponent, and in letting the ultimate integration bounds recede to infinity. The reason h(x) gets to display one more term then g(x) is of course that its effect on the result is more significant, since it is multiplied by the increasing n. The formula (11.34) is slightly modified when x0 coincides with the end point x = a: and h′ (a) = 0; then only one half of the above infinite range contributes, and r Z b π −nh(x) −nh(x0 ) , x0 = a, (11.35) e g(x) dx ∼ g(x0 ) e 2n h′′ (x0 ) a For a detailed proof of these results we refer the reader to R. Wong [141]. Exercise 11.61 [3] Assuming that h(x) ∼ h(x0 ) + ∑ ak (x − x0 )k

g(x) ∼

and

k>2

∑ bk (x − x0 )k ,

k>0

with x0 inside the integration interval, find the three first coefficients, c0 , c1 , and c2 , in the Poincaré expansion s Z b h i c1 c2 2π −nh(x0 ) −nh(x) c + e + + · · · . e g(x) dx ∼ 0 n h′′ (x0 ) n n2 a Note: The calculation is straightforward, and quite messy, and you are strongly urged to use MAPLE or the like of it. Pay particular attention to the terms we ignored in the exponent when deriving Eq. (11.34). Check your result against the Stirling approximation of Γ(z + 1). Now we consider the case when the function h(x) attains the global minimum at one of the ends of interval [a, b], say at x = a, but it is not a local minimum: we assume that h′ (0) 6= 0, in which case it must be positive (for this to be a minimal point). The computation for the zero-order asymptotic result is quite similar to the previous case, although the type of integral we finally reach is different. Since the contribution of the neighborhood close to x = a is dominant as n increases, we represent the role of g(x) in the calculation by its value g(a), and that of h(x) again by the two leading terms in its development, which now are h(a) + h′ (a)(x − a): I(n) =

Z b a

g(x)e−nh(x) dx ≈ g(a)e−nh(a)

Z b a

′

e−nh (a)(x−a) dx


658

≈ g(a)e−nh(a)

Z ∞ 0

′

e−nh (a)u du = e−nh(a)

g(a) , nh′ (a)

h′ (a) 6= 0.

(11.36)

Exercise 11.62 This problem is a repeat of Exercise 11.61 for the case we just considered, where the minimum of h(x) in [a, b] is at a, and h′ (a) > 0. Again assume the developments h(x) ∼ h(a) + ∑k>1 ak (x − a)k and g(x) ∼ ∑k>0 bk (x − a)k , and find the three first coefficients c0 , c1 , and c2 in the Poincaré expansion that extends Eq. (11.36). Again, you are strongly urged to use MAPLE or another computer algebra system. The Laplace method can easily be extended to the integrals of the form Z b a

n

n+1/2

g(x) [ f (x)] dx ∼ g(x0 ) [ f (x0 )]

−2π n f ′′ (x0 )

1/2

as n → ∞ ,

(11.37)

since we only need to substitute h(x) = − ln( f (x)) into Eq. (11.34), where x0 is assumed to be an inner point of maximum for f (x).

11.4 Examples and Exercises In this section, we present many interesting examples and problems to solve that show the diversity of asymptotic methods. Example 11.63 Suppose that an urn contains n distinguishable balls, that are drawn at random (with equal probability) with replacement. Let N be the number of drawings to get the ball that has been chosen previously. To determine the probability mass function of the random variable N, we start with some simple observations. First, Pr[N = 1] = 1/n. The event N = 2 may occur if we draw any ball (with probability n/n) first, and on the second draw we choose any of the other remaining (n − 1) balls (with probability (n − 1)/n). Then on the third draw, we must draw one of the two previously drawn balls (with probability 2/n). Therefore Pr[N = 2] =

n n − 1 2 2(n − 1) × × = . n n n n2

The event N = 3 occurs when we draw arbitrary ball (with probability 1 = n/n), then on the second draw we choose any of the remaining (n − 1) balls (with probability (n − 1)/n), on the third draw we choose one of the remaining (n − 2) balls (with probability (n − 2)/n). Then, on the fourth draw, we must choose one of the three previously drawn balls, so Pr[N = 3] =

n n − 1 n − 2 3 3(n − 1)(n − 2) × × × = . n n n n n3

The pattern is obvious and therefore Pr[N = k] =

nk k(n − 1)(n − 2) · · · (n − k + 1) = k × . nk nk+1

11.4. EXAMPLES AND EXERCISES

659

Now we can find the expected value of N to be n

E[N] =

n

∑ k Pr[N = k] =

∑ k2

k=1

k=1

(n − 1)k−1 . nk

(11.38)

Let T be the number of drawings for which Pr[N = k] > 12 . The next exercise asks you to √ show that E[N] and T grows as n. √ Exercise 11.64 Referring to the previous example, show that E[N] and T grows as n. Exercise 11.65 Show that the probability given in Eq. (5.59) to win the Russian roulette when the number of participants, n, is large is pk,n ∼ −

1 p · . ln q n − (k − 1)p

Exercise 11.66 For a sequence of independent Bernoulli trials, calculate the ratio of m successes in n trials and 2m successes in 2n trials (see Exercise 4.6 on page 139) and find its limit as n increases. Exercise 11.67 [4+] For fixed n, we denote by T (n) the smallest m which gives the probability of more than 1/2 that the winning set in m/n lottery is consecutive. Usually this number, T (n),is called the threshold number for n and it is the smallest number m such that 2K(n, m) > mn , where K(n, m) is the number of consecutive m-sets in [1..n]. Find asymptotic behavior of T (n). Note that the number K(n, m) is defined in Exercise 3.134 on page 129.

Exercise 11.68 [5] Regarding Exercise 4.13 on page 139, let pn = Pr[X = 1]. Analyze the behavior of pn as n → ∞.

Exercise 11.69 [4] Let Pm (n) be the probability that an n-permutation contains at least mswap (from Example 8.83 on page 476). Show that Pm = lim Pm (n) = eHm−1 − eHm , n→∞

where Hm is m-th harmonic number. Exercise 11.70 [4] (R. Mabry) The Bernstein polynomials of degree n are defined by n k bn,k (p) = p (1 − p)n−k . k

√ Find a closed-form expression for β (p) = limn→∞ n max06k6n bn,k (p). Hint: Show that the Bernstein polynomials have a maximum value for some m and then apply Stirling’s formula. Example 11.71 Let Sm (n) = ∑nk=0 (2n + k)−1 for 1 6 m 6 n. To determine limn→∞ Sm (n) we observe that Sm (n) is an upper sum for the integral Z 3n−m+1 dx 2n

x


660

and also a lower sum for

Z 3n−m dx 2n−1

Therefore ln

3n − m + 1 2n

x

.

6 Sm (n) 6 ln

3n − m . 2n − 1

Taking the limit of all terms we obtain limn→∞ Sm (n) = ln 3 − ln 2 = ln 23 ≈ 0.4054651.

Exercise 11.72 [3] For positive p, find the limit n−m

∑

lim

n→∞

k=0

1 pn + k

(1 6 m 6 n).

Exercise 11.73 [3+] (D. Callan) At a dinner party, n wives arrive, each accompanied by her husband and a girlfriend. The 3n people are seated around a circular table. Each wife insists on being seated next to her girlfriend. If they are randomly seated subject only to this restriction, let pn denote the probability that no wife is sitting next to her husband. Find limn→∞ pn . (Without the girlfriends, this is the “relaxed manage problem.”) Exercise 11.74 Let γ = 0.5772156649 . . . be the Euler constant, show that Z λ 1 − e−t 0

t

dt ∼ γ

as

λ → ∞.

Exercise 11.75 [2] Show that

k

lim n

n→∞

"

1 1+ n+1

n+1

 n #   0, 1 − 1+ = e/2,  n  ∞,

if k < 2, if k = 2, if k > 2.

Exercise 11.76 [2] The following series Un = ∑k nk Hk does not have a simple closed form, but can be very well approximation by 2n (Hn − ln 2). Show this, and in the process you will get a simple expression for the error of this approximation. Hint: The OGF of the harmonic numbers is given by Exercise 6.4, page 276.

Chapter 12

Review of Analytic Techniques In this chapter, we recall the definitions and properties of the mathematical infrastructure needed to accommodate generating functions (GFs). While we include sufficient material for our uses here, the reader who would like to get more deeply into these topics needs additional sources; happily, excellent texts exist. We recommend P. Henrici [62] for a detailed modern coverage; all our current needs are satisfied by the first chapter there.

12.1 Complex Numbers We start with a reprise of complex numbers. Definition 12.1 A complex number1 is an ordered pair of real numbers. A standard notation is z = (x, y), where the real number x is called the real part of the complex number z, and the real number y is called the imaginary part of z. The common notation for these components is ℜz = x, ℑz = y Two complex numbers are equal when their real and imaginary components are equal separately. The set of all ordered pairs of real numbers is called the set or the field of complex numbers, denoted by C, if the following arithmetic operations hold. Let z1 = (x1 , y1 ), z2 = (x2 , y2 ) be two complex numbers; they can be manipulated as follows: 1. Sum. Addition is defined as z1 + z2 = (x1 + x2 , y1 + y2 ). 2. Difference. Subtraction is similarly defined via z1 − z2 = (x1 − x2 , y1 − y2 ). 1 Complex numbers were so named by Carl Friedrich Gauss (1777 – 1855), but were known to mathematicians for several centuries earlier. The symbol i for the imaginary unit was introduced by Leonhard Euler in 1777.

661

CHAPTER 12. REVIEW OF ANALYTIC TECHNIQUES

662

3. Multiplication. The product is given by z1 z2 = (x1 x2 − y1 y2 , x1 y2 + x2 y1 ). 4. Division. This is easily derived from the multiplication: x1 x2 + y1 y2 x2 y1 − x1 y2 z1 = , . z2 x22 + y22 x22 + y22

⊳

While the real numbers constitute the real line, complex numbers span the complex plane. We denote by 1 = (1, 0) and i = (0, 1) the unit vectors along the coordinate axes Ox and Oy. In electrical engineering, it is common to denote these unit vectors as i = (1, 0) and j = (0, 1). The abscissa is also called in this context the real axis; naturally, the ordinate then becomes the imaginary axis. The definition of the multiplication operation provides (0, 1) × (0, 1) = √ (−1, 0), or i2 = −1, √ so we see that the imaginary unit is, up to a sign, −1. In other words: −1 = ±i.

Let the projections of a complex number z on these axes be x = ℜz and y = ℑz. Then the representation of z in terms of these components is z = x1 + yi

or simply

z = x + iy.

(12.1)

This is called the Cartesian form of a complex number. In MAPLE, the imaginary unit is denoted by I, hence this number would be written as z = x + Iy. There is no difference between x + iy and x + yi; for brevity we write x instead of x + i0, and iy instead of 0 + iy. The former complex number is said to be pure real and the latter—a pure imaginary. 6

z1 + z2 = (x1 + x2 ) + i(y1 + y2 )

..................... ... y2 .. .. .. .. .. ........................... .... . ... .. .. ...... ..... ..... .... .. .. ... ... ... ... .... .... ..... ... ... .. y1 .. ......... .. .. ....... .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ................................................. z1 . .. .. ... .. .................... . .................... ... .. .. . ................................. .

y

6

z2 = x2 + iy ................................. ..................2...

x2

z = x + iy

= x1 + iy1

x1

Figure 662: (a) Addition of complex numbers

.... .......... ......... . . . . . . . . . ........ ......... .......... . . . . . . . . . . ......... ..... .......... ]....... θ x ......... . . . .. . . . . . . ........... .......... .......... .......... .......... .......... .......... .......... ..........z = x − iy .......... .........

(b) Polar representation

Definition 12.2 The complex number x − iy is called the conjugate of the complex number ⊳ z = x + iy and is denoted by z = x − iy. Multiplications or divisions are easier—and more natural—in the polar representation of complex numbers, than in the Cartesian form (12.1). We place the “pole” at the origin 0 of

12.1. COMPLEX NUMBERS

663

the Cartesian axes and the polar axis along the positive part of the real line. Let r and θ denote the polar coordinates of the point z = x + iy. Then x = r cos θ ,

y = r sin θ ,

and Eq. (12.1) becomes z = x + iy = r(cos θ + i sin θ ).

(12.2)

The relation (12.2) gives the trigonometric form of the complex number z. The quantities r and θ are called the modulus and argument 2 of z, and are defined as r = |z| =

p x2 + y2 ,

θ = arg z = tan[−1]

y y ≡ arctan . x x

The modulus is the Euclidean√distance from the origin to the point z = x + iy. Note that we can write the size of z as r = zz. The argument θ = arg z (which is only defined for z 6= 0), is the counterclockwise angle between the polar axis (the positive half of the real axis) and the vector from the origin to z.

Note: Since the argument determines a complex number via trigonometric functions, two values of the argument that differ by a multiple of 2π give rise to the same complex value. This fact is commonly expressed by saying that a complex number, (and by extension, any complex-valued function), has an infinite number of polar representations, differing from one another by multiples of 2π in the argument (the modulus is the same for all). When we let the argument θ = arg z change through an interval of size up to 2π , we say the number z has remained on the same branch. Similarly, a function is single-valued on each of its branches. If −π < θ 6 π , then such value of θ is called the principal branch of the argument of z. Theorem 12.3 The operation of raising the number e to the power z = x + iy is done via the equation ex+iy = ex eiy = ex (cos y + i sin y). Putting x = 0, we obtain Euler’s formula: eiy = cos y + i sin y.

(12.3)

(In particular, using y = π in the last relation, Euler rewrote it in the form eiπ + 1 = 0, and said this magic formula exhibits all five numbers of importance in mathematics.) 2 Other terms are absolute value, size, or amplitude for the modulus, and phase for the argument. We use tan[−1] for the inverse of the tangent function; it is also denoted by the symbol arctan.


664

To prove the theorem, all we need is to show Eq. (12.3). One way of proving that both sides are the same functions is to show that their values and all their derivatives at one point are identical. This claim holds for y = 0. Both sides are 1’s. For the derivatives, this is easy once you remember the formulas: d sin y = cos y, dy

d ay e = aeay , dy

d cos y = − sin y. dy

A more natural way is to use the Taylor series of the functions. Definition 12.4 The following form of the complex number z z = |z|eiθ , where

θ = arg z

(12.4)

is called its polar representation.

⊳

Example 12.5 Here are a few frequently used values: eiπ /2 = cos π2 + i sin π2 = i. ei2π

= cos 2π + i sin 2π = 1.

ei3π /2 = cos 32π + i sin 32π = −i.

eiπ /4 = cos π4 + i sin π4 =

1+i √ . 2

So much for algebraic approach to determine complex numbers and arithmetic operations with them. From the geometrical point of view, a complex number is a point on the plane or a vector starting from the origin. However, the four arithmetic operations from Definition 12.1 make the difference between the points on the plane and complex numbers. The polar representation of complex numbers simplifies the task of describing geometrically the product of two complex numbers or raising to power or exponent, as shown in Figure 665. Let =⇒ z1 z2 = r1 r2 ei(θ1 +θ2 ) (12.5) z1 = r1 eiθ1 and z2 = r2 eiθ2

Exercise 12.6 [1] You are given two complex values √ √ z1 = 2 eiπ /4 = 1 + i and z2 = 2 eiπ /3 = 1 + i 3. √ √ √ and also told that sin (π /6) = 1/2; sin (π /4) = 1/ 2; Show z1 · z2 = 1 − 3 + i(1 + 3). While the following result follows immediately from the multiplication formula (12.5), computing powers of complex numbers is important enough to mention the historic name for the formula: it is called de Moivre formula3 n (12.6) zn = r eiθ = rn einθ = rn (cos nθ + i sin nθ ).

This formula even suggests a graphical way to calculate powers of complex numbers! To find the power zn , first determine its modulus and argument, Then on the complex plane draw a line with angle nθ and measure the distance |z|n from the origin. This is zn . 3 What Abraham de Moivre (1667 – 1754), a French mathematician, showed was that (cos θ + i sin θ )n = cos nθ + i sin nθ , for integral n, but the relation holds, by virtue of Euler’s relation, for all powers, with minor changes.

12.1. COMPLEX NUMBERS

665

... ... z1 z2 .. .. ... ... ... .. ... ... ... 6 .. ... ... ... ... ........................................... .................... . ............. θ1 + θ2 .......... .......... ......... ... ........ ....... ... ..... ..... ... ..... .... . .... ... .... z2 .... .... ... . .... . ... .... .. . .... ... ... ... ... .. . ... . ... ... .. . ... ... ... .. ... ... . .. . .. ... . . .. .. ... ... z ... 1 . ... ... ........... . . . . . . ... . ..... θ . . . . ... ...i . . . .... 2 ..... .. .. . . . . . . .. . .......... .. ... .. .. . . . .. ..... .............. ..... M.....θ1 .. ......... .. .. ..

-

Figure 665: Multiplication of complex numbers Example 12.7 Compute (1 + i)3 .

Solution. We first re-express z = 1 + i in polar form to be z=

√ iπ /4 2e ,

since |z| = r =

√

2, arg z = θ =

π . 4

/4 to the angle 3θ = 3π /4 in counterclockwise direction. Then we rotate the ray arg z = π√ Cubing the modulus we obtain 2 2, and then, √ 3 √ 3 1 1 3π 3π 3/2 3 3 3iπ /4 + i( 2) sin =2 −√ + i√ = −2 + 2i. = ( 2) cos z =r e 4 4 2 2 Alternatively, using the Cartesian form and the binomial theorem, we write (1 + i)3 = 1 + 3i + 3i2 + i3 = 1 + 3i − 3 − i = −2 + 2i. Definition 12.8 The function w(z) defined by the equation ew = z is called the logarithm function and is given by w(z) = Ln z = ln |z| + i arg z. The principal value of the logarithm is the function ln |z| + i arg z, where arg z is limited to the interval (−π , π ]. This range is commonly called the principal branch of the argument of z. Intervals of size 2π of arg z beyond or below the principal branch define single-valued functions—each is called a branch of the logarithm and denoted by ln z = ln |z| + i arg z. ⊳

Note: A different convention gives this term to the interval [0, 2π ).


666

6

ℑz

z3 = −2 + 2i.............................................................................................................3.........θ.. = 3π /4 .. ..

......... ....... ......... ..... ......... ...... ...... ...... ..... ...... . ..... . . . ..... ..... ........ ..... ..... ....... .... .... . ... .... ..... .... ..... .... .... ..... ... ... ..... ... ..... ... z = 1 + i ... ..... . .. . ..... . ... . . . . .. ..... . . . .. . . ..... .. . . . .. . . ..... .. . ..... . .. . . .... ..... . .. . . . I ... .. ..... . . . ... .. . . ..... . . .. .. . . . . .. ..... . .. θ = π /4 . . ... . . . ..... .... .. .... . ..... ..

ℜz -

Figure 666: Example 12.7. The third power of 1 + i. Equation (12.6) also shows the way to computing the nth root of a nonzero complex number 1/n = r1/n eiθ /n . However, the division by n means z: Substitution there provides z1/n = r eiθ that as the argument of z goes through a full range of 2π , the last expression does not. In fact, we can take n branches of the argument, and fold them into the principal branch of the root function. We shall do just that and find 1/n √ √ θ + 2kπ θ + 2kπ iθ n n + i sin , k = 0, 1, 2, . . . , n − 1. (12.7) z = re = r cos n n This computation has a very simple geometric analog, which we now illustrate. Example 12.9 Find all cubic roots of −1.

Solution. We write z = −1 = eiπ . Since the absolute value of −1 is one, this is going to be the modulus of all roots. We draw a unit circle in Figure 667. With arg z = π , the roots will have the arguments (π + 2kπ )/3, for k = 0, 1, 2. Thus, the first root is √ 1 3 , w1 = cos(π /3) + i sin(π /3) = + i 2 2 the next root is w2 = cos

π + 2π π + 2π + i sin = cos π + i sin π = −1, 3 3

and the last root is √ π + 4π π + 4π 1 3 + i sin = −i . w3 = cos 3 3 2 2

12.2. REVIEW OF POWER SERIES

667

6 ℑz

i

cos u + i sin u

............................................ ............ .......... ........ ...... .. ....... .... . . .. ............ . ... . . . . . ... . . .. .. ... . . ... ... .. ... .. .. . . ... .. . ... .. . .. . ... . .. ................................................... ... . ... ... .. .. . . ... . . . . . .. ... .. .. . . ... ... .. ... ... ... ... ... ... .... ... . .... . . . ... ...... .... ..... .. ..... ....... .......... ........ .....................................................

π = 3u cos π −1

−i

u = π /3 1

- ℜz

cos 5u + i sin 5u

Figure 667: Example 12.9. Cubic roots of −1.

12.2 Review of Power Series Power series can be introduced in a number of ways; it is probably appropriate to consider them as the basis for expansion of arbitrary functions, via the mechanism of Taylor series. We first consider functions on the real line, and then allow ourselves to slip from it into the complex plane. In the following we use the letter x for the independent variable when we refer to functions on the real line, and we use z for a complex independent variable. However, many of the statements that we use with x can be stated for z with hardly any change.

12.2.1 The Taylor Series Let y = f (x) be an arbitrary function that has as many derivatives on some interval I, centered at x0 , as we need. We can compute arbitrarily good approximations of the function f (x) at any point x ∈ I by creating Taylor polynomial of degree n approximating f (x) at x near x0 : f (x) ≈ f (x0 ) + f ′ (x0 ) (x − x0 ) +

f [n] (x0 ) f ′′ (x0 ) (x − x0 )2 + · · · + (x − x0 )n . 2! n!

(12.8)

This is an approximation; to get an equality, you need to add to this polynomial a correction term, given by f [n+1] (c) (x − x0 )n+1 , (12.9) En (x) = (n + 1)! for some c (which depends on x and n) between x and x0 . Exercise 12.10 [1+] (Numerical exercise) Let f (x) be the function ex ; take x0 = 2 and evaluate f (4) using the Taylor polynomial for n = 4, 5, 6, 7. Since you know f (4), for each n value, use Eq. (12.9) to compute a suitable estimate. For a function f (x) that has infinitely many derivatives on an interval I = {z ∈ Z : |z − x0 |
0 , is convergent. Let us state this precisely, for a general (not necessarily power-) series. Definition 12.11 Let ∑∞ n=0 bn be an infinite series of real or complex numbers. The sum of the series is the limit of the sequence {sn } of partial sums sn ≡ ∑nk=0 bk , if this limit exists. When limn→∞ sn is a finite number s, then the series ∑∞ n=0 bn converges to s. When limn→∞ sn is ±∞, or is undefined, then the series is said to diverge. ⊳ There are known several accepted definitions of convergence. An attractive one relies on the notion of Cesàro4 sum of infinite series. Cesàro’s definition of the sum assigns to the infinite series the limit (in the usual sense) of the mean value of partial sums: s = lim

n→∞

s0 + s1 + · · · + sn−1 , n

Cesàro sum.

(12.12)

For example, the series ∑n>0 (−1)n diverges in the ordinary sense, but the Cesàro limit for this series exists, and hence the series converges in Cesàro sense to 1/2. Exercise 12.12 [1] Show that if the sums ∑k ak = a and ∑k bk = b converge in Cesàro sense, then ∑k (ak + bk ) = a + b in Cesàro sense. Exercise 12.13 Show that convergence in the sense of Cesàro is a true extension of the usual definition of the limit of a series, in the sense that when the usual limit exists, it also exists in the sense of Cesàro, and both have the same value. Theorem 12.14 (Abel convergence theorem) [118, page 160] Let {an } be a sequence of non-negative real numbers with GF a(z) = ∑n>0 an zn , in some circle |z| < r = 1. Then we have a necessary and sufficient condition: lim a(z) = S

z→1−

⇐⇒

∑ ak = S

(12.13)

k>0

4 Ernesto Cesàro (1859–1906) was a prolific Italian mathematician who studied with Catalan—of binary tree count fame—and held professorship in Palermo and Rome.


669

For convergence in the sense of Cesàro we have another necessary and sufficient condition: lim (1 − z)a(z) = L

z→1−

⇐⇒

1 n ∑ ak = L. n→∞ n k=0 lim

(12.14)

While the first part of the theorem is a simple result of the dominated-convergence theorem, and in the second part, the implication to the left is similar to the first part, but the implication to the right there, the Tauberian part of the theorem, is relatively deep. The sum of a convergent power series is a function of its variable, say x. This function is the sum function of the series. The set of all values of x for which the power series converges is the domain of the sum function. For a function of a real variable (on the real line) this domain is a centered interval |x − x0 | < R. We call R the radius of convergence of the series. Inside this interval the series converges absolutely. The series may either converge or diverge when |x − x0 | = R. There is no single criterion that determines convergence at the boundary.

There are many expressions that provide the radius of convergence of the power series in principle (the actual calculation may be hard). The most common one is given by p 1 = lim n |an |, R n→∞

(12.15)

where lim is the upper limit of the sequence. The source of this is clear: for convergence, it is needed that the terms of the sum decrease exponentially. Then, if |an Rn | is less than one, the requirement is satisfied. Exactly the same reasoning is behind the following common test for the absolute convergence of a power series, the ratio test: If all (except a finite number) an 6= 0, and if for some fixed z an+1 = L, (12.16) |z − z0 | lim n→∞ an

then the power series (12.8) converges absolutely at z if L < 1, and diverges if L > 1. If L = 1, the test is inconclusive. For a function in the complex plane, the convergence region is a circle |z − z0 | < R. As you just noticed, we prefer to use the symbol z when referring to complex values. The radius of convergence, R, is the distance from the center, z0 , to the nearest point at which the series diverges. Such a point is called a singular point. When the power series converges only at z = z0 , we say that the radius of convergence is zero5 . On the other hand, if the series converges for all z—in the entire plane—we say that the radius of convergence is infinite. In the later case (R = ∞), the sum function of the power series is called an entire function, for instance, the exponential function is an entire function. As you know, it is the sum function of the series ∑k>0 zk /k!. The formulas (12.15), (12.16) assume that the power expansions are known. The radius is determined by the nearest singularity, even if we do not have the expansion, but have another 5 For

example, this is the case for the series ∑k>0 k!zk , around the origin.


670

y

i

y

6

y

6

........................ ..... .......... .... ..... ... ... .. ... .. ... .. ... .. .. .. .. ... ... . ... . .... .. . . . ...... . .................................

1 x

−i

Center: z = 0 Radius: 1

....................................... ................. ........... .......... ......... ......... ...... ..... . ..... . . . .... ... . . . .... ... . .... . ... ... . . ... . .. .. ... .. .. . ... .... .. ......... .. .. ... .................. ... ........... ... ... ........... ............ .. ... ........... .. .. ........... ... ......... .. . . . . ... . . . . . .. . . ....... . . . .. . ... . . . . . . . . ....... . . . .. . . . . . . . . . .. . ........ . . . . . . . . . .. . . . .. .................... .. ..... .. .. .. ... .. .. ... . . ... .. ... ... .... ... .... .... .... .... . . ..... . ... ..... ...... ...... ......... ........ ............ ........ ...........................................................

.................................................. ......... ...... ..... ....... .... .... .... .... .. ... ... ... ......... ... .. . .......... .. ....... ... .. ....... .. ... ....... ... ....... ... ....... .. .. . ..... . ... . . . . . .. . .... .. . . .. . . . . .. .. ..... . . . . .. . . . . . .... .. . . . . . . . . . .. ...... ... ....... ... ... ... ... .... .... ..... .... . . . . ..... ..... ....... ........... ........ ..........................................

6

i

i

2

−i

Center: z = Radius: 2

√

3

-

x

3

-

x

−i

√ Center: z = 2 2 Radius: 3

Figure 670: Domains of convergence of power series for f (z) =

1 1+z2

for Example 12.15.

source of information, such as an explicit representation, that allows us to determine the singularities—we can find the radius of convergence. Example 12.15 Find the radii√of convergence √ of the power expansions of the function 1/(1 + 2 z ) centered at z0 = 0, z0 = 3, and z0 = 2 2.

Solution. The denominator of f (z) has zeros at z = ±i in the complex plane. Hence the value of the function is infinite at these points: they are singularities, of the type we call simple poles. Since the distance from the origin to both zeros is 1, its power expansion, which is a Taylor series around the origin z = z0 = 0, ∞ 1 = ∑ (−1)n z2n 1 + z2 n=0

converges when |z| < 1.

√ Now we develop the power series for f (z) at z0 = 3. If we want to write the expansion explicitly, we need the Taylor series, as shown in Eq. (12.10) to derive √ √ √ √ 2 √ 1 1 3 3 1 (z − 3) + (z − 3) − (z − 3)3 ± · · · = − 2 1+z 4 8 8 32

(12.17)

We see that the distance between the new z0 and the closest singularity (which is still, of √ course, any of ±i) is 2. Hence the power series (12.17) converges when |z − 3| < 2. √ 2 −1 Since the distance from the point z√ 0 = 2 2 to ±i is 3, the power expansion of (1 + z ) about this z0 converges when |z − 2 2| < 3. Note that the three convergence circles have a common intersection. In that lens-shaped domain all three expansions are valid and equal 1/(1 + z2 ). Example 12.22 shows a more pedestrian way to obtain these developments.


671

Example 12.16 Determine the radius of convergence of the power series f (x) =

∑ 2n xn

and

g(x) =

∑

n>0

n>0

(−1)n 2n x . 2n

Solution. We use the formula (12.15) for the series of f (x) and find easily √ 1 n = lim 2n = 2. n→∞ R Therefore, the radius of convergence R of the Taylor expansion for f (x) is 1/2. Application of the ratio test (12.16) to the series g(x) yields (−1)n+1 2n 2 = |x|2 1 . |x | lim n+1 n n→∞ 2 (−1) 2 √ √ So this series converges when |x|2 /2 < 1 or |x| < 2: the convergence radius is 2.

Exercise 12.17 [2] Determine the radius and interval of convergence of the following power series. (x − 1)n xn xn ; 2. ∑ ; 3. ∑ ; 1. ∑ n n>1 n>0 2n + 1 n>0 (2n + 1)! n(x − 2)n (x + 2)n n! 5. ∑ ; 6. ; 4. ∑ n xn ; ∑ n n2 n>0 2 + 1 n>1 n>1 n (x + 2)n n(n + 2) n n(−1)n n n! x ; 8. x ; 9. 7. ∑ ∑ 5n ∑ 3n n2 . 2n n>0 n>1 n>0 When a power series converges, it gives rise to that darling of applied mathematicians— the analytic function, one of the most fecund concepts in their tool chest. We give now a definition and a few examples of convergent sums, but defer digging deeper into this rich lode to the next section. Definition 12.18 A function f is called analytic or holomorphic in an open region if for every point z0 in this region there exists a power series of the form ∑n>0 an (z − z0 )n with a positive radius of convergence and it converges to the value of f (z) for all z in the region of convergence. ⊳ Later in this section, we show that the representation of a function as a Taylor series is unique in its region of convergence. Let us give a few examples of analytic functions. Every polynomial is an analytic (even entire) function. Indeed, since the derivatives of order higher than the degree of a polynomial are all equal to zero, the Taylor series expansion of any polynomial function has only a finite number of nonzero terms, and therefore it converges everywhere (except infinity). Example 12.19 The polynomial f (x) = (x − 1)2 is its own Taylor expansion centered at x = 1 f (x) = 0 + 0 · (x − 1) + (x − 1)2 + 0 · (x − 1)3 + · · · .


672

The Maclaurin series for this function is f (x) = 1 − 2x + x2 . The Taylor series for f centered at x = 2 can be calculated from the general form (12.10), but it is easier to do it via the binomial theorem: f (x) = (x − 1)2 = (x − 2 + 1)2 = [(x − 2) + 1]2 = 1 + 2(x − 2) + (x − 2)2 . Example 12.20 Here are some well-known Taylor expansions (in fact, Maclaurin) of entire functions—they all “inherit” their entirety from the exponential function: ex = 1 + x +

∞ xn x2 x3 + + ··· = ∑ ; 2! 3! n=0 n!

sin x = x −

∞ x2n x2n+1 x3 x5 + − · · · = ∑ (−1)n = x ∑ (−1)n ; 3! 5! (2n + 1)! (2n + 1)! n>0 n=0

cos x = 1 −

∞ x2n x2 x4 + − · · · = ∑ (−1)n ; 2! 4! (2n)! n=0

sinh x = x +

∞ x2n+1 x3 x5 + + ··· = ∑ ; 3! 5! n=0 (2n + 1)!

cosh x = 1 +

∞ x2n x2 x4 + + ··· = ∑ . 2! 4! n=0 (2n)!

Example 12.21 The function f (x) = ln x is an analytic function on the interval (0, ∞). Its Taylor series centered at x = 1 has particularly convenient form: 1 (1 − x)n 1 . ln x = (x − 1) − (x − 1)2 + (x − 1)3 − · · · = − ∑ 2 3 n n>1 It converges in the interval (0, 2). This series can be rewritten, using, say the transformation tn 1 − t = x as the useful expansion ln(1 − t) = − ∑ . n>1 n Example 12.22 The function 1/(1 − x) is analytic everywhere on the real line except one point x = 1. Its Maclaurin expansion is the obvious geometric series: ∞ 1 = 1 + x + x2 + x3 + · · · = ∑ xn , 1−x n=0

−1 < x < 1.

(12.18)

The Taylor expansion of the function 1/(1 − x) centered at x0 = 2 is found by first bringing it to a form suggested by Eq. (12.18), 1 1 1 1 = = =− , 1 − x 1 − 2 + 2 − x −1 + (2 − x) 1 + (x − 2) and as long as we keep |x − 2| < 1, we can use Eq. (12.18) to obtain

∞ ∞ 1 = − ∑ (−1)n (x − 2)n = − ∑ (2 − x)n , 1−x n=0 n=0


673

whereas the Taylor expansion near x0 = 3 is 1 1 1 1 1 =− =− =− x−3 1−x 2 + (x − 3) 2 1+ 2 2 Example 12.23 (The binomial series) saw in Chapter 2, Eq. (2.42): (1 + x) p = 1 + px +

∞

∑ (−1)n

n=0

(x − 3)n 1 =− n 2 2

∞

∑

n=0

(3 − x)n . 2n

This is a different view on the Binomial theorem we

p · (p − 1) 2 p · (p − 1) · (p − 2) 3 x + x + ··· 2! 3!

for − 1 < x < 1

This expansion can be used when the exponent, p, is any number (including a complex number). When p is a positive integer, the binomial coefficients vanish for k > p since they contain the factor zero. In this case, the number of terms is finite, and the series becomes a polynomial, which converges for all x. Exercise 12.24 [2+] (a) Complete Example 12.15 by computing the first four coefficients an in the series √ n 1 a (z − 2 = 2) . n ∑ 1 + z2 n>0 (b) Compute an explicit formula for all an . Hint: Use the partial fraction decomposition.

12.2.2 Operations on Power Series We list several important operations on power series. For simplicity, we show the operations for power series centered at x0 = 0. They hold, however, for arbitrary x0 . Addition and Subtraction. Power series can be added or subtracted term-wise in their common interval of absolute convergence. That is, ∞

∞

∑

n=0

an xn ± ∑ bn xn = n=0

∞

∑ (an ± bn) xn .

n=0

Multiplication of Power Series. Two power series can be multiplied, giving the third power series ! ! ∞

∑ an xn

n=0

∞

∞

∑ bn xn

n=0

=

∑ cn xn ,

n=0

where cn = a0 bn + a1 bn−1 + a2 bn−2 + · · · + an b0 .

(12.19)

This is of course the convolution we have seen before, in Chapter 2. In the context of power series, it is called the Cauchy product. The product converges for all x in the common open interval of convergence for the two multiplied power series6 . 6 The Cauchy product may have a radius of convergence that is larger than those of the two multiplied power 1 · (1 − x) = 1 shows. series as the example 1−x


674

Division of Power Series. If b0 6= 0, two power series can be divided, giving a third power series ∞ n ∑∞ n=0 an x dn xn . (12.20) = ∑ ∞ n b x ∑n=0 n n=0 n Multiplication both sides by the series ∑∞ n=0 bn x leads to the product ! ! ∞

∞

n=0

n=0

∑ an xn = ∑ bn xn

∞

∑ dn xn

n=0

discussed above—the coefficients dn are expressed as a solution to the full-history recurrence: an = d0 bn + d1 bn−1 + d2 bn−2 + · · · + dn b0 ,

n = 0, 1, 2, . . . ,

(12.21)

that follow from the Cauchy product (12.19). Thus, we have successively a0 = d0 b0 =⇒ d0 = a0 /b0 a1 = d0 b1 + d1 b0 =⇒ d1 = (a1 − d0 b1 )/b0 a2 = d0 b2 + d1 b1 + d2 b0 =⇒ d2 = (a2 − d0 b2 − d1 b1 )/b0 and so on, to any desired order. It is clear from the last relations why we need b0 6= 0: division by zero is always frowned upon. But what can we do if this is the case: if the first nonzero term of the series in the denominator is some br xr , with r > 0? The answer is as simple as the question: write that denominator as ∑k>r bk xk = xr ∑k−r>0 bk xk−r = xr ∑ j>0 t j x j , where t j = b j+r . Now we can continue with the division as above since, by assumption, t0 6= 0. However, the result will be a power series preceded by x−r . In the next section, we shall refer to such a series as having a principal part. Exercise 12.25 [2−] Calculate the first four non-zero coefficients of the reciprocal of each of the following power series: ez =

zk ∑ , k>0 k!

z2k+1 sin z = ∑ (−1)k , z (2k + 1)! k>0

Exercise 12.26 [2] Compute the inverse to 1 − z − z2 .

cos z =

∑ (−1)k k>0

z2k . (2k)!

Substitution into series. In many cases we can get the series of a complicated function by viewing it as a composition of two or more simple functions. When f (z) = h(zm ), we can get the Taylor series of f (z) directly from the series of h(z), whereas applying the direct procedure given in Eq. (12.8) would be more difficult and cumbersome. For example, since a power series development for sin z is known from Example 12.20, the k 6k Maclaurin series for the sine function of z3 is simply sin z3 = z3 ∑∞ k=0 (−1) z /(2k + 1)!.


675

Term-by-Term Differentiation. A power series defines an analytic function in its interval of convergence and can be differentiated any number of times there by performing term-by-term differentiation: d dx d2 dx2

∞

∑

n=0

an xn =

∞

∑ n an xn−1;

n=1

∞

∞

∞

n=0

n=2

n=0

∑ an xn = ∑ n(n − 1) an xn−2 = ∑ (n + 2)(n + 1) an+2 xn

and so on. Technically, we say that the operators D (for differentiation) and Σ (for summation) commute: DΣ = ΣD. Term-by-Term Integration. A power series has an antiderivative that can be found by using term-by-term integration, giving ! Z ∞ ∞ xn+1 n ∑ an x dx = ∑ an n + 1 +C. n=0 n=0 Vanishing of all Coefficients. If a power series has a positive radius of convergence and the corresponding sum function is identically zero in an interval, then each coefficient of the series must be zero. While this is not related to any operation on a sequence, it explains why the connection between sequences and series is so tight. An immediate corollary of the above is that the representation of an analytic function as a power series around a given point is unique: f (x) =

∑ ak (x − x0)k = ∑ bk (x − x0 )k

k>0

k>0

=⇒ ak = bk , ∀ k > 0.

(12.22)

And finally, an aspect of our dealing with power series that you would be glad to know: Summation Manipulations. Power series are merely a particular kind of sums—hence all we said in §2.1 about properties of summation applies here fully. We recommend to the reader a review of that material. Example 12.27 Let us find the Maclaurin expansion of f (x) = [ln(1 + x)]/(1 − x). There are two options how to proceed. First, we can see f (x) as the product of ln(1 + x) and (1 − x)−1 , and since we know power expansions for both, use the Cauchy product: (−1)k−1 k x2 x3 x4 x × ∑ xj f (x) = x − + − + · · · (1 + x + x2 + x3 + · · · ) = ∑ 2 3 4 k j>0 k>1 n

=

1 5 7 47 (−1)k−1 n x = x + x2 + x3 + x4 + x5 + · · · . k 2 6 12 60 k=1

∑ ∑

n>1

A second approach is to divide the power series for the logarithm by the (finite) power expansion 1 − x, which leads naturally to the same result.


676

Exercise 12.28 [1] Dividing ln(1 + x) by (1 − x), show that you get in the process the follown−1 ing recurrence for the coefficients of the series for f (x): cn = cn−1 + (−1)n , which clearly has the solution given previously. Example 12.29 This example illustrates a general approach that can simplify life greatly in 2 this kind of calculations. Suppose we want to find the Maclaurin series for ex about x = 0, 2 n that is, ex = ∑∞ n=0 an x . We could find coefficients by direct differentiation 1 d n x2 f [n] (0) = e . an = n! n! dxn x=0

The calculations, using the chain rule will be very tedious. Fortunately, there is a simpler way: substituting y = x2 into the power series

2 x4 x6 yn x2n y2 y3 + + · · · + + · · · =⇒ ex = 1 + x2 + + + · · · + + ··· , 2! 3! n! 2! 3! n! provides what we need. The situation would have been less simple if the substitution does not provide the required powers. For instance, a development around the point x0 = 2 for the 2 n function ex = ∑∞ n=0 bn (x − 2) is not easy to achieve. We address this issue in the following exercise.

ey = 1 + y +

Exercise 12.30 [1+] Complete the calculation suggested in the last example—find the coef2 n ficients {bn }n>0 in ex = ∑∞ n=0 bn (x − 2) . 2 2 2 Hint: One way of approaching this is by writing ex = e(x−2) +4(x−2)+4 = e(x−2) × e4(x−2) × e4 . Here, the last factor is a mere number, which multiplies each bn , and for the other two functions we have both of their Taylor series in the last example! Use convolution to derive a final answer, which you may leave in the form of a finite sum. Example 12.31 In this example of computing a Maclaurin series, we find a higher hurdle: we consider f (θ ) = esin θ .

We know from previous examples the Maclaurin series of ey and sin θ , ey = 1 + y +

∞ yn y2 y3 + +··· = ∑ 2! 3! n=0 n!

and

sin θ = θ −

∞ θ3 θ5 θ 2k+1 + − · · · = ∑ (−1)k . 3! 5! (2k + 1)! k=0

We substitute the series for sin θ for y. This yields 2 1 θ3 θ5 θ3 θ5 + − ··· + + − ··· + ··· . θ− esin θ = 1 + θ − 3! 5! 2! 3! 5!

Collecting terms of equal powers in θ , we obtain the first few ones easily 4 θ2 θ3 θ3 θ θ4 θ2 sin θ + − + − + 0 · θ3 + ··· . + + ··· = 1+ θ + = 1+θ + e 2! 3! 3! 4! 3! 2!

Clearly MAPLE can easily produce any prefix of the series; for example, the program f := t -> exp(sin(t)); series(f(t),t=0,10); gives esin θ = 1 + θ +

θ2 θ4 θ5 θ6 θ 7 31θ 8 θ9 − − − + + + + ··· . 2 8 15 240 90 5760 5670

12.3. FUNCTIONS OF A COMPLEX VARIABLE: BASIC CONCEPTS

677

You will not find it hard to write yourself a simple procedure that collects these coefficients, but a closed-form expression is another matter. Later we shall see techniques that help in producing such developments. Example 12.32 The error function Z x

2 erf(x) = √ π

0

2

e−t dt

is an important function in applied mathematics (probability, heat conduction, . . . ). To find its power series representation (around the origin), we start with the Maclaurin series of the 2 function e−t , which we get from Example 12.29, and use term-by-term integration to obtain: 2

e−t =

∞

t 2n

∑ (−1)n n!

n=0

=⇒

2 erf(x) = √ π

∑

n>0

(−1)n x2n+1 (2n + 1)n!

since the antiderivative of t 2n is t 2n+1 /(2n + 1). Exercise 12.33 [2] Given the geometric series sions of (1 − x)−2 and (1 − x)−3 , (a) using term-by-term differentiation, (b) using convolution.

1 1−x

= 1 + x + x2 + · · · Find the power expan-

Exercise 12.34 [1+] Find the Taylor series of the given functions about the indicated point x0 . What is the interval of convergence of the series? (a) f (x) = x + 1, x0 = −1; (c) f (x) = x + 1, x0 = 0; (e) f (x) = (x − 3)2 , x0 = 0; x , x0 = 0; (g) f (x) = 1 + x2 x , x0 = −1; (i) f (x) = 1−x

(b) f (x) = x + 1, x0 = 1; (d) f (x) = (x − 3)2 , x0 = 3; (f) f (x) = (x − 3)2 , x0 = −3; x , x0 = 2; (h) f (x) = 1−x (j) f (x) = sin x, x0 = π /4.

12.3 Functions of a Complex Variable: Basic Concepts In §12.2, we found it already necessary to refer to functions of complex variable—we say somewhat more briefly, complex functions—when we discussed convergence properties of power series. We had to account for the fact that the properties of such series, even when considered for values on the real line, depend on their behavior in the entire complex plane, which we denote by C. One could say that this plane is the natural habitat of functions of a continuous variable. In this section, we survey in brief some facts and properties of such functions that play a role in analysis of algorithms.


678

Definition 12.35 Let D be a set of complex numbers, namely, D ⊂ C. A function f defined on D is a rule that assigns to each z ∈ D a unique complex number w. The number w is called the value of f at z and is denoted by w = f (z). The set D is called the domain of the function f. ⊳ Just as when we discussed complex numbers in §12.1, we viewed them as pairs of real numbers, this view can be continued here as well: Let w = u + iv be the value of a function f at z = x + iy; that is, f (x + iy) = u + iv. Then each of the real numbers u and v depends on the real variables x and y: f (z) = u(x, y) + iv(x, y). In polar coordinates x = r cos θ , y = r sin θ we have f r eiθ = u(r, θ ) + iv(r, θ ). Thus, a function f of the complex variable z can be considered as a pair of two real-valued functions u = u(x, y)

and

v = v(x, y)

u = u(r, θ )

and

v = v(r, θ )

in Cartesian coordinates, or

in polar coordinates. Example 12.36 Let f (z) = z2 , then f (x + iy) = (x + iy)2 = x2 − y2 + 2ixy. Hence u(x, y) = x2 − y2

and

v(x, y) = 2xy.

In polar coordinates:

we have

2 f r eiθ = r eiθ = r2 e2iθ = r2 cos 2θ + ir2 sin 2θ u(r, θ ) = r2 cos 2θ

and

v(r, θ ) = r2 sin 2θ . 2

Exercise 12.37 [1] For z = x + iy, determine the real and imaginary parts of f (z) = eiz . Definition 12.38 A set of points γ : z = x + iy in the complex plane C is called an arc (also contour or curve) if they are all captured by x = x(t),

y = y(t)

(a 6 t 6 b),

where x(t) and y(t) are continuous functions of the real parameter t. The arc γ is a simple arc or a Jordan arc, if it does not cross itself. ⊳ Definition 12.39 A set S is connected if every pair of its points can be jointed by a simple arc which consists only of points of the set S. ⊳ Definition 12.40 A point z0 is said to be an interior point of a set S whenever there is some neighborhood |z − z0 | < ε , ε > 0, that contains only points of S. We say that z0 is an exterior point of S when there exists a neighborhood of z0 containing no points of S. A point z0 is said to be a boundary point of S if it is neither an interior point nor an exterior point.


679

Definition 12.41 A set is open if it contains none of its boundary points. A set is closed if it contains all of its boundary points. An open, connected set is called domain. ⊳ We have already seen a definition of analytic (or holomorphic or regular) function of a complex variable in Definition 12.18. There are three equivalent definitions of this concept; they are due to Riemann, Weierstrass, and Cauchy, respectively. Definition 12.42 A one-valued function f of the complex variable z in the domain D is called analytic or holomorphic if and only if it satisfies one of the following equivalent conditions: Riemann: f has the derivative f ′ (z) at every point z ∈ D defined as f ′ (z) =

f (z + ∆z) − f (z) d f (z) = lim ; ∆z→0 dz ∆z

Note that the limit may not depend on the way ∆z goes to zero. Weierstrass: For any point z0 ∈ D, f has a power series expansion centered at z = z0 : ∞

∑ an (z − z0)n ,

f (z) =

n=0

which converges in a neighborhood of z0 . Cauchy: For any closed Jordan contour γ , which is entirely in D, Z

γ

f (z) dz = 0.

Such an integral is called contour integral and denoted by

H

γ.

⊳

Exercise 12.43 [2] With the notation z denoting the complex conjugate of z (Definition 12.2, page 662), prove that the function z¯2 is not analytic anywhere since it violates the Riemann condition in Definition 12.42. The following theorem suggests the rigidity of an analytic function: its values on a (onedimensional) curve determine its values on a (two-dimensional) domain. Theorem 12.44 (Cauchy) Let f be analytic everywhere within a single closed contour γ , taken in counterclockwise direction. Then for any point a interior to γ , we can evaluate the function and its derivatives there by contour integrals

1 f (a) = 2π i

I

γ

f (z) dz , z−a

and

f

(n)

n! (a) = 2π i

I

γ

f (z) dz . (z − a)n+1

(12.23)

Definition 12.45 If a function f fails to be analytic at a point z0 , then z0 is called a singular point, or singularity, of f . If there exists an annulus 0 < |z − z0 | < R, for some positive R ∈ R, in which f (z) is analytic then the point z = z0 is called an isolated singularity of the function f. ⊳


680

In applications of complex functions, a contour integral of zn , for any integer n, over a circle with a center in the origin, plays a special role. The integral is not hard to evaluate (using the substitution z = re2π it ), and we obtain the following result: ( Z 1 I 2π i, n = −1, 2t π i(n+1) n n+1 e dt = (12.24) z dz = r 2π i 0 |z|=r 0, otherwise; where |z| = r is the contour traversed in counterclockwise direction.

Theorem 12.46 Let a function f be holomorphic in an annular domain r < |z − z0 | < R. Then at each point z in the domain f (z) has the Laurent series representation ∞

∞

f (z) =

∑

n=0

an (z − z0 )n + ∑

n=1

bn . (z − z0 )n

(12.25)

n The series ∑∞ n=0 an (z − z0 ) is the regular part of the Laurent series, because it defines an analytic function in |z − z0 | < R. The series ∞

∑

n=1

bn (z − z0 )n

is called the principal part of Laurent series, and it converges to an analytic function in the unbounded domain |z − z0 | > r.

Definition 12.47 The residue of a function f at the isolated singularity z = a is the coefficient of (z − a)−1 in its Laurent series expansion at z = a. It is denoted by Res f a

or

Res f (z) z=a

or

Res [ f (z); z = a].

⊳

Definition 12.48 The isolated singularity z = z0 of a function f is called removable if and only if lim f (z) exists. z→z0

This implies that f has a Laurent series representation about z0 that does not have a principal part, and its residue there is zero. ⊳ Example 12.49 The function f (z) = sinz z has a removable singularity at z = 0 since its Laurent representation about this point, which can be seen from Example 12.20, ∞

f (z) =

z2n

∑ (−1)n (2n + 1)!

n=0

does not have a principal part. It is also evident from this expansion that limz→0 sin z/z = 1. Definition 12.50 A multivalued function at an isolated point is said to have a branch singularity at that point. It is also called a branch point. ⊳ For example, this is the case for the function zs at z = 0, when s is rational, but not an integer. If s is a positive integer, the singularity is removable, but clearly not when s < 0.


681

Definition 12.51 The isolated singularity of f at z = z0 is called a pole if there is a positive integer m such that z0 is not a singularity of (z − z0 )m f (z). The smallest such m is called the order of the pole. This implies that f has a Laurent expansion at z = z0 that has the form ∞

f (z) =

b1

b2

bm

∑ an (z − z0)n + z − z0 + (z − z0 )2 + · · · + (z − z0)m ,

n=0

where bm 6= 0, the residue of this function at z0 is b1 . A pole of order m = 1 is called a simple pole. ⊳ Theorem 12.52 An isolated singularity z0 of a function f is a pole of order m if and only if f (z) can be written in the form g(z) , f (z) = (z − z0 )m where g(z) is analytic and nonzero at z0 . Moreover,

Res f (z) = lim (z − z0 ) f (z) = g(z0 ) when z=z0

and in general

z→z0

d m−1 1 [g(z)] . Res f (z) = z=z0 (m − 1)! dzm−1 z=z0

m=1

(12.26)

(12.27)

Example 12.53 Consider the function f (z) =

z z3 z5 sin z 1 − + − + ··· . = z2 z 3! 5! 7!

Here z = 0 is a simple pole, and the residue there is 1. Definition 12.54 An isolated singularity of f at z = z0 is an essential singularity if and only if the principal part of the Laurent expansion of f at z = z0 contains an infinite number of terms. This is equivalent to the limit limz→z0 f (z) not existing, not even “improperly” (which is the case when limz→z0 f (z) = ±∞). ⊳

Definition 12.55 A single-valued function without essential singularities is called a meromorphic function. In other words, a meromorphic function is a function of the form f (z) = g(z)/h(z), where g(z) and h(z) are entire functions with h(z) 6= 0. Example 12.56 The condition for essential singularity via an infinitely-long principal part is useful since it is frequently difficult to verify whether a function has a limit at a singularity. That condition certainly implies that z = 0 is an essential singularity of the function ∞ 1 1 1 1 1 1 1 = 1+ + + · · · (0 < |z|), =∑ exp n z n! z 1! z 2! z2 n=0

but in this case, it is easy to see from the function itself that it must have an essential singularity at z = 0: consider the behavior of e1/z as z approaches the origin. Its limit depends on the direction from which the origin is approached, 1/z will have different phase, and there is no way to define a limit as z → 0. This differs from the situation on the real line, where we can define in such cases one-sided limits, even improper limits.


682

The following theorem contains one of the most frequently used formula for evaluation of integrals over closed loops. Theorem 12.57 (Cauchy Residue Theorem) If f is analytic within and on the closed simple (Jordan) curve γ except, possibly, a finite number of isolated singularities a1 , a2 , . . . , an inside γ , then I

γ

n

f (z) dz = 2π i

f, ∑ Res a

k=1

(12.28)

k

where the contour γ is traversed counterclockwise. The next theorem is called after the French mathematician Eugène Rouché (1832 – 1910): Theorem 12.58 (Rouché) Let the functions f (z) and g(z) be regular on a common region D and satisfy the inequality | f (z)| > |g(z)| on the boundary of D. Then the functions f (z) and g(z) + f (z) have the same number of zeroes in D, counting multiplicities. Exercise 12.59 [3] For each of the following functions, determine all their finite singularities, and which types they are. If it is a pole, determine its order, and find the corresponding residue. 2 z z1/4 exp z z ; 3. ; 4. ; 2. ; 1. (z2 + 1)2 z2 + π 2 2z + 1 z+1 ez − z2 /2 − z − 1 ; 6. tan z ; 7. z2 sin (1/z) ; 8. cot z. z5 Example 12.60 Application of the Cauchy residue theorem yields 5.

1 2π i

I

γ

n (1 + z)n dz = , r zr+1

(12.29)

where γ is any circle centered at the origin. Let us consider the sum " # I 1+z k (1 + z)n−1 m n+k−1 1 dz. = ∑k z ∑k 2π i γ z k k=1 k=1 m

(1)

The sum in the right-hand side is Gn (x) for x = (1 + z)/z and its value was found in Eq. (2.14) on page 32 to be 1 2π i

m(1 + z)n+m−1 (1 + m)(1 + z)n+m − dz, (1 + z) + zm+1 zm

I γ

n

which we evaluate using the residue theorem to obtain

n+k−1 n+m+1 n+m n+m =m − (m + 1) =n . ∑k k m m−1 m−1 k=1 m


683

Example 12.61 Let A(z) = ∑k>0 ak zk be the ordinary generating function for the sequence {ak } and let cz 1 A . T (z) = (1 − bz)d 1 − bz

To find coefficients, tn = [zn ]T (z), of the Maclaurin series to T (z), we apply the residue theorem: tn = = = = =

I

T (z) 1 dz [z ]T (z) = 2π i zn+1 I u c + bu n+1 c 1 T du u = cz/(1 − bz) 2π i c + bu u (c + bu)2 I c + bu d c + bu n+1 c 1 A(u) du 2π i c u (c + bu)2 I (c + bu)d+n−1 1 1−d c A(u) du = c1−d [un ](c + bu)d+n−1 A(u) 2π i un+1 n n + d − 1 n−k k ∑ n − k b c ak . k=0 n

Similarly, we can relate the coefficients {an } and {bn } through n n n+d −1 n + d − 1 n−k k −n n−k n−k an = c ∑ (−1) b tk , tn = ∑ b c ak , n−k n−k k=0 k=0

(12.30)

which gives another pair of inversion formulas. n

Exercise 12.62 [1] Find the generating function for the sequence sn =

∑ k=0

n n−k

2k k c. k

12.3.1 Diagonalization of Series Suppose we are given a bivariate ordinary generating function A(x, y) =

∑

an,m xn ym .

(12.31)

n,m>0

Our goal is to determine the coefficient bk (n) = an,n+k for a fixed k and the corresponding generating function (12.32) bk (z) = ∑ an,n+k zn = ∑ bk (n)zn n>0

n>0

without trying for all the terms in the double sequence {an,m }. If k = 0 we get the diagonal term b0 (n) = an,n . Let us start with an example of the generating function (GF) m+n n m xn n −n − 1 A(x, y) = ∑ x y = ∑ x (−y)m = ∑ n+1 m m n,m>0 n,m>0 n>0 (1 − y)


684

=

1 1 1 . x = 1 − y 1 − 1−y 1−x−y

As a function in the complex planes of x and y, the summation suggests that A(x, y) exists for |y| < b = 1 and |x| < |1 − y|, so we need to constrain |y| to a smaller circle, or the function would have no convergence region in the x plane. For example, if we decide on b = 1/2, we can allow x also to be in the circle |x| < 1/2 and be guaranteed a convergence; there is something to be said for symmetry too since clearly A(x, y) is symmetric in the two variables. Also note that we could deduce all this from just observing the function itself: it is regular so long as |x + y| < 1. Calculations show (we repeat all calculations later) that √ 2n + 1 n 1 − 1 − 4z 2n n 1 √ . and b1 (z) = ∑ z = b0 (z) = ∑ z =√ n n 1 − 4z 2z 1 − 4z n>0 n>0

In general, the coefficients bk (n) can be determined based on the following theorem. Theorem 12.63 Let the generating function (12.31) converge for all x and y such that |x| < a, |y| < b. Then for all z such that |z| < ab we have I ∞ z k−1 1 (12.33) s ds = ∑ an,n+k zn+k , A s, Fk (z) = 2π i γ s n=0 where γ is a closed loop inside the circular annulus D = {s : |z|/b < |s| < a}. Furthermore, if the function f (s) = A(s, z/s) sk−1 has only isolated singularities inside γ , then the integral can be evaluated by summing the residues of f (s) at these singularities.

P ROOF : The function f (s) can be rearranged to form the Laurent series f (s) = sk−1

∞

∑

an,m sn zm s−m =

∑

an,m sn−m+k−1 zm ,

n,m>0

n,m=0

which converges absolutely and uniformly for all s ∈ D. Now we integrate both sides of the last equation along the closed Jordan contour γ to obtain 1 2π i

I

γ

f (s) ds =

∑

n,m>0

zm an,m

1 2π i

I

γ

sn−m+k−1 ds =

∑

zm an,m δn−m+k,0 =

n,m>0

∑ an,n+k zn+k

n>0

since, according to the Cauchy residue theorem, I

γ

s j ds = 2π iδ j,−1

where δ j,−1 is the Kronecker’s delta given in Eqs. (2.5) or (F.1). Let us return to our example for A(x, y) = (1 − x − y)−1 with a = b = 1/2. The function z sk sk−1 = f (s) = A s, s s − s2 − z


685

has two singularities √ 1 − 1 − 4z s1 = 2

and

√ 1 + 1 − 4z s2 = . 2

To use Theorem 12.63, we restrict |z| to be less than 1/4, and then both si are regular in z, but only s1 is inside the annulus D. Therefore √ I sk sk1 (1 − 1 − 4z)k 1 k−1 √ . = = A(s, z/s) s ds = Res Fk (z) = s1 s − s2 − z 2π i γ 1 − 2s1 2k 1 − 4z Since Fk (z) = bk (z)zk , we find √ √ k (1 − 1 − 4z)k n+k (1 − 1 − 4z) √ √ = [z ] . an,n+k = [z ] k k 2 z 1 − 4z 2k 1 − 4z n

Thus we proved

√ 2n + k n (1 − 1 − 4z)k √ =∑ z . n 2k zk 1 − 4z n>0

Example 12.64 Let us consider the function a(x, y) = 2(1 − α x − β y − γ xy)−1 defined in Eq. (6.73). By inspecting the denominator, we see that if we limit x to be in the circle |x| < a, α| then y needs to be restricted to |y| < |1−a| β |+|γ a| . The Greek coefficients mask the symmetry between the variables here, but we can recover it by defining p (|α | + |β |)2 + 4|γ | − (|α | + |β |) . a= 2|γ |

and using the ranges |x| < a and |y| < a. From Theorem 12.63, we derive 2 Fk (z) = 2π i

I

sk−1 ds 2 = 1 − α s − β z/s − γ z 2 πi c

I

sk ds , 2 c s − α s − β z − γ zs

for |z| < a2 and using a contour c in the annulus D = { |z| a < |s| < a}. The denominator s − α s2 − β z − γ zs has the following two roots p 1 − γ z ± 1 − 2z(γ + 2αβ ) + γ 2 z2 , s± = 2α If only s− is inside the circle |s| < a, then using the residue theorem, we obtain k p 1 − γ z − 1 − 2z(γ + 2αβ ) + γ 2 z2 p . = Fk (z) = 2 Res s− s − α s2 − β z − γ zs 2k−1 α k 1 − 2z(γ + 2αβ ) + γ 2 z2 sk

(12.34)


686

Exercise 12.65 [3] (I. Gessel) For the given generating function ∞ 1 = ∑ an,m xn ym , 1 − x − y + 2xy n,m=0

show that (−1) j a2 j,2 j+2 is the Catalan number

2j j

/( j + 1).

Exercise 12.66 [2] Let {an } be a sequence such that the corresponding generating function a(z) = ∑n>0 an zn is analytic in |z| < a. Verify that F(x, y) =

∑ k,n>0

∇k an xn yk =

a(x) 1 − (1 − x)y

(∇an = an − an−1 ),

is the bivariate generating function for the double sequence k k fk,n = ∇k an = ∑ (−1) j an− j . j j=0 Find F(z) = ∑n>0 fn,n zn in terms of a(z), and the relation between sequences { fn,n } and {an }.

Exercise 12.67 [2+] Using the bivariate generating function A(u, z) = ∑n,k>0 E2 (n, k)un zk from Exercise 6.114 for the sequence {E2 (n, k)} (see Exercise 5.139, page 260, for detail), determine E2 (n, n).

Exercise 12.68 [4] The generating function for the sequence of Delannoy numbers (see Exercise 6.122 on page 325) is A(x, y) = (1 − x − y − xy)−1 . Find the central Delannoy numbers dn = [xn yn ]A(x, y).

12.4 Differential Operators In this section, we consider the set of continuous functions that have an arbitrary number of derivatives. Typically, we deal with analytic functions. This is the domain of the operators we discuss here. Let Dx ≡ d/dx denote differentiation with respect to x. We often suppress the subscript, when there is no danger of confusion. The effect of the operator D upon a function φ (x) is to produce its derivative φ ′ (x), for every function from the domain. We already saw this operator previously, and commented on its commutativity with the sum operator. We can form products of D, to obtain the powers Dn (n = 1, 2, . . .). These correspond to successive differentiations. For example, D2 produces the second derivative with respect to x. For any integer n ∈ N, dnφ Dn φ (x) = n = φ (n) (x). dx A linear combination of these powers

L[D] = a0 Dn + a1 Dn−1 + · · · + an−1 D + an ,

a0 6= 0

12.4. DIFFERENTIAL OPERATORS

687

is called a differential operator of order n. When applied to a function φ (x), it yields the result L[D]φ = a0 φ (n) (x) + a1 φ (n−1) (x) + · · · + an−1 φ ′ (x) + an φ (x). Note: the coefficients a0 , a1 , , . . . , an in this expression may be functions of x. With this in hand, such expressions as φ ′ (x) + φ (x), φ ′ (x) + x2 φ (x), and φ ′′ (x) + 2φ ′ (x) + 3φ (x) may be written respectively as

φ ′ (x) + φ (x) = [D + 1]φ , φ ′ (x) + x2 φ (x) = [D + x2 ]φ , φ ′′ (x) + 2φ ′ (x) + 3φ (x) = [D2 + 2D + 3]φ . The symbols D + 1, D + x2 , and D2 + 2D + 3 do not stand for quantities; they indicate operations that are to be performed. As it is obvious from the above, the operation of “operators” like 2 or x4 is merely a multiplication by the indicated value. Definition 12.69 (Equality of operators) • Two linear differential operators L1 and L2 are said to be equal if and only if L1 ϕ (x) = L2 ϕ (x) for every function ϕ from the domain. • Operators with constant coefficients commute: L1 L2 = L2 L1 . In other words, operators with constant coefficients behave as algebraic polynomials; we can add, subtract, and multiply them, and even long (synthetic) division may be used. ⊳ Example 12.70 Let A = D + 1 and B = 2D − 1. Then A + B = 3D, and (A + B)ϕ (x) = 3ϕ ′ (x).

Similarly, AB = (D + 1)(2D − 1) = 2D2 + D − 1 = BA, and ABϕ (x) = 2ϕ ′′ (x) + ϕ ′ (x) − ϕ (x).

Example 12.71 You may remember from calculus the rule for differentiating a product of functions, called the product rule, discovered by Leibniz in the 17th century: ( f (x)g(x))′ = f ′ (x)g(x) + f (x)g′ (x). How does this rule scale up for higher-order derivatives? It is simple to show that the formula for the nth derivative of a product has an uncanny similarity to the binomial Theorem: n k n d u(x) d n−k v(x) dn [u(x)v(x)] = ∑ dxn dxk dxn−k k=0 k and even closer when we use the operator D = d/dx: n n n [Dk u(x)] [Dn−k v(x)], (12.35) D (u(x)v(x)) = ∑ k k=0 where nk is the binomial coefficient (see §2.5). We may write symbolically Dn (uv) = (Du + Dv)n . Exercise 12.72 [2] Prove Eq. (12.35) using mathematical induction.

2 2 d e−x and Exercise 12.73 [1] Prove that the following two operators A1 = −ex dx d −x2 /2 x2 /2 e are identical. A2 = e x− dx


688

Example 12.74 Let us compute the fourth derivative of the product y(x) = x2 cos 2x. From the product rule, we see that derivatives of order zero to four are needed of the two terms here, x2 and cos 2x. Clearly Dx2 = 2x,

D2 x2 = D(Dx2 ) = D(2x) = 2,

D3 x2 = D4 x2 = 0.

The second term needs more work: D2 cos 2x = D(−2 sin 2x) = −4 cos 2x,

D cos 2x = −2 sin 2x,

D3 cos 2x = D(−4 cos 2x) = 8 sin 2x,

D4 cos 2x = D(8 sin 2x) = 16 cos 2x.

Finally, we apply Eq. (12.35) and obtain D4 (x2 cos 2x) = 0 + 4 · 0 · (−2 sin 2x) + 6 · 2 · (−4 cos 2x) + 4 · 2x · 8 sin 2x + 16x2 cos 2x = 16x2 cos 2x + 64x sin 2x − 48 cos 2x.

This approach is used in symbolic differentiation software, such as in MAPLE.

Let us consider now briefly differential operators with variable coefficients. The theory of such operators is largely beyond the scope of this book, and we show several points via examples and exercises. Example 12.75 Consider the two operators A1 = x D and A2 = D x. We have (A1 f )(x) = (x D) f (x) = x (D f (x)) = x f ′ (x). (A2 f )(x) = (Dx f )(x) = D(x f (x)) = f (x) + x f ′ (x). where we used, naturally, the product rule. The results of applying the two operators are different: (A2 − A1 ) f (x) = f (x) , or A2 − A1 = I , (12.36) where I is the identity operator, namely, I f (x) = f (x). This yields an expected conclusion that the operators x (multiply by x) and D (differentiate with respect to x) do not commute. The difference xD − Dx, or more generally, for any two operators A and B, the difference AB − BA is called the commutator of these operators. n Let us repeat the same calculation, using Taylor series expansion, f (x) = ∑∞ n=0 an x : ∞

A1 f (x) = x D

∑

an xn = x ∑ an D (xn ) = x n

n=0 ∞

A2 f (x) = (D x) ∑ an xn = D n=0

∞

And hence (A2 − A1 ) f (x) =

∞

∑

n=0

an xn+1 =

∞

∞

∑ an n xn ,

∑

an n xn−1 =

∑

an D(xn+1 ) =

n=0 ∞ n=0

n=1

∞

∑ an(n + 1) xn .

n=0

∑ an xn = f (x), as expected from relation (12.36).

n=0


689

Now we consider iterations of the operator A1 = xD. Using induction and recurrence relation for Stirling numbers of the second kind, Eq. (5.72), we can prove that r r k k r r x D f (x), for all integers r > 0. (12.37) (A1 ) f (x) ≡ (xD) f (x) = ∑ k=0 k n Fortunately, if f (x) is expressed via its Maclaurin power series, that is, f (x) = ∑∞ n=0 an x , we have more pleasant result:

(xD)r f (x) = (A1 )r

∞

∞

∞

n=0

n=0

n=0

∑ an xn = (xD)r ∑ an xn = ∑ an nr xn .

(12.38)

In particular, for f (x) = xm (m is a positive integer) we have r r r r k k m−k m r m =x ∑ mk = xm mr , (xD) x = ∑ x m x k k k=0 k=0 where we used Eq. (5.76), page 262. Similarly, (A2 )r f (x) = (Dx)r f (x) = (A2 )r

∞

∑

an xn = (Dx)r

∞

∑

n=0

n=0

an xn =

∞

∑ an (n + 1)r xn .

n=0

Now we can express the operator A1 via another operator A2 using the relationship (12.36): r r k (A1 )r = (A2 − I)r = ∑ (−1)r−k A2 . k k=0 n −1 yields Applying this to f (x) = ∑∞ n=0 x and making the substitution x = n r r r−k r r−k r Sn (r) = ∑ (−1) [nSn (k) − 1] = ∑ (−1) nSn (k) , k k k=0 k=0 ∞

where Sn (r) =

r −k

∑k n

Ar1

=

r−1 r−k

(n − 1)Sn (r) = n ∑ (−1)

n

∑x

n=0

k=1

k=0

∞

!

. Gathering the terms involving Sn (r), we obtain x=1/n

r Sn (k) k

or

n r−1 r−k r Sn (r) = ∑ (−1) k Sn (k) . n − 1 k=0

Exercise 12.76 [2−] Find the result of acting with the commutators of the following pairs of operators on analytic functions: (a) (c)

A1 = x2 D, and A2 = Dx2 ; A1 = D2 x, and A2 = x2 D,

(b) (d)

A1 = xD2 x, and A2 = Dx2 D, B1 = Dx−1 , and B2 = x−1 D.

In part (d), consider functions that do not vanish at the origin; use as example f (x) =

1 1−x .


690

Exercise 12.77 [1+] Show that the two difference operators, ∆, ∇, and the shift operator, E, all commute. Example 12.78 This example shows an elegant way to sum general series using summation by parts equation, established in Theorem 2.36. Such derivation employs concepts of power series and differential operators; all manipulations are made in a formal sense paying no attention to convergence or rigor. However, all derivations can be made rigorous, but it will require much more work. If we are given an arbitrary function f (k), defined on the integers, and we know F(k), for which f (k) = ∆F(k) = F(k + 1) − F(k), then Theorem 2.36 gives a simple formula: n

n

∑ f (k) = ∑ ∆F(k) = F(n + 1) − F(0).

k=0

k=0

The problem of finding the finite sum, ∑nk=0 f (k), is now reduced to finding F for a given f . hr This is easy to achieve with the aid of Taylor’s theorem: F(z + h) = ∑ Dr F(z) = ehD F(z). r! r>0 −1 D D f (z). Hence when h = 1 we get f (z) = e − 1 F(z), which leads to F(z) = e − 1 To find F(z), we start with the exponential generating function for the Bernoulli numbers (L.33): zr z B = r ∑ r! . ez − 1 r>0 Formal substitution z → D yields Dr D B = ∑ r r! eD − 1 r>0

=⇒

F(z) =

Dr f (z) Dr−1 f (z) 1 1 B B f (z) = = ∑ r r! ∑ r r! . eD − 1 D r>0 r>0

Using the above equation, we re-derive the Euler summation formula (10.48): n

∑ f (k) = F(n + 1) − F(1) =

k=1

Z n+1 1

z=n+1 Br+1 Dr f (z) . f (z) dz + ∑ r>0 (r + 1)!

(12.39)

z=1

We are going to generalize the Euler summation formula by considering a weighted sum: n

∑ wk f (k) k=1

with some weight w. To evaluate this sum using summation by part, we need to determine a function F(k) such that wk f (k) = wk+1 F(k + 1) − wk F(k) = ∆wk F(k). So for k = z, we get f (z) = w F(z + 1) − F(z), or using the relation F(z + 1) = eD F(z), we obtain f (z) = (w eD − 1) F(z)

=⇒

F(z) =

1 f (z). w eD − 1


691

In order to understand how the operator (w eD − 1)−1 acts on an arbitrary function, we expand the function into power series: zk 1 = E (w) , k ∑ w ez − 1 k>0 k!

(12.40)

where Ek (w) is some function to be determined later. The power series in the right-hand side converges within a circle, centered on the origin, with radius less than | ln w|; so we require that w 6= 1. Based on expansion (12.40), we derive the generalized Euler summation formula: n

∑ wk f (k) = wn+1 F(n + 1) − w F(1),

where

F(z) =

∑ Ek (w) k>0

k=1

Dk f (z) . (12.41) k!

Now we need just to find coefficients Ek (w). To do this, we differentiate both sides of Eq. (12.40) first by w and then by z and compare the results. This yields the recurrence: Ek+1 (w) = wEk′ (w),

E0 (w) = −

1 . 1−w

(12.42)

We multiply both sides of Eq. (12.40) by w ez − 1 and equate coefficients of like powers of z. This gives k k 1 (12.43) Ek (w) = w ∑ Ek−i (w), k > 0, E0 (w) = − 1 − w i i=0 Eqs. (12.42), (12.43) define Appell polynomials (see §6.1.3) for Eulerian numbers: 1 Em (w) = − (1 − w)m+1

1 ∑ E(m, n) w = − (1 − w)m+1 n>0 n

m ∑ n wn . n>0

(12.44)

Choosing f (z) = z, we get from (12.41) that F(z) = z E0 (w) + E1 (w) = −z (1 − w)−1 − w (1 − √ w)−2 . Taking the weight, w, √ as the golden ratio, φ = (1 + 5)/2, we obtain Fφ (z) = zφ − φ 3 . ˆ Similarly, for w = φ = (1 − 5)/2, we get Fφˆ (z) = zφˆ − φˆ 3 . So, from Eq. (12.41) we have n

∑ k φ k = φ n+1 Fφ (n + 1) − φ Fφ (1) = (n + 1)φ n+2 − φ n+4 − φ 2 + φ 4 .

k=1

Similar relation holds for φˆ . Now using power representation for Fibonacci numbers (D.3), we obtain n

∑ k Fk = (n + 1) Fn+2 − Fn+4 + 2.

k=1

Exercise 12.79 [3] Using the generalized Euler summation formula, find the sums ∑nk=1 k2 Fk and ∑nk=1 (−1)k k2 Fk .


692

12.5 Partial Fraction Decomposition When the denominator has no repeated linear factors. We consider first the case when F(s) = N(s)/D(s) is a rational function with denominator D(s) = (s − a1 )(s − a2 ) · · · (s − an ) with no repeated linear factors. We learned in a Calculus course that in this case the partial fraction decomposition of F(s) has the form F(s) =

A2 An N(s) A1 N(s) + + ··· + = = D(s) (s − a1 ) · · · (s − an ) s − a1 s − a2 s − an

The following formula may be used to compute the coefficients Aℓ : N(s) N(aℓ ) N(s) = lim (s − aℓ ) (ℓ = 1, 2, . . . , n). = ′ Aℓ = Res s→aℓ s=aℓ D(s) D(s) D (aℓ )

(12.45)

Example 12.80 Let us consider the quotient 2s + 3 A1 A2 2s + 3 = = + s2 − s − 6 (s + 2)(s − 3) (s + 2) s − 3 We have,

2s + 3 (s + 2)(s − 3)

A2 = lim (s − 3) ·

2s + 3 (s + 2)(s − 3)

s→−2

and

A1 = lim (s + 2) ·

s→3

= lim

s→−2

2s + 3 (s − 3)

2s + 3 = lim s→3 (s + 2)

(12.46)

1 = , 5

=

9 5

We can also find the values of A1 and A2 in the right-hand side of Eq. (12.46) by multiplying it by (s − 3)(s + 2). This leads to 2s + 3 A1 (s − 3) + A2 (s + 2) = . (s + 2)(s − 3) (s + 2)(s − 3) Setting s = 3 yields 5A2 = 9 and setting s = −2 yields −5A1 = −1. There is another way to determine A1 and A2 . In the equation A1 (s − 3) + A2 (s + 2) = 2s + 3 we equate coefficients of like terms either side of the above equation and solve the resulting linear system: (A1 + A2 )s + 2A2 − 3A1 = 2s + 3

=⇒

A1 + A2 = 2,

2A2 − 3A1 = 3.

12.5. PARTIAL FRACTION DECOMPOSITION

693

When the denominator has repeated linear factors. Now we look at the case when the denominator of F(s) = N(s)/D(s) has repeated factors as in D(s) = (s − a)n Q(s), where Q(a) 6= 0. The partial fraction decomposition has the form ( ) additional terms depending c1 c2 cn N(s) = + +· · ·+ + F(s) = (s − a)n Q(s) s − a (s − a)2 (s − a)n on factors of polynomial Q(s) The coefficients cℓ (ℓ = 1, 2, . . . , n) may be computed with the formula: d n−ℓ N(s) 1 , ℓ = 1, 2, . . . , n. cℓ = (n − ℓ)! dsn−ℓ Q(s) s=a

(12.47)

Example 12.81

3s − 1

s3 − 10s2 + 25s

=

3s − 1 C1 C2 A + = + 2 s(s − 5) s s − 5 (s − 5)2

The coefficient A corresponds to the factor s, which has exponent 1. Hence the formula for linear factors may be used: 3s − 1 1 3s − 1 = lim =− , A = lim s · 2 2 s→0 (s − 5) s→0 s(s − 5) 25 The coefficients C1 and C2 are associated to the factor (s − 5), which appears with exponent 2 in the denominator of the rational function. Then, 3s − 1 ′ 1 14 1 3s − 1 C2 = = 5 , and C1 = = s2 = 25 . s s s=5 s=5 s=5

Note that the coefficients A, C1 , and C2 can be determined in a similar way as in Example 12.80 from the equation A(s − 5)2 +C1 s(s − 5) +C2 s = 3s − 1.

The complex factors case. The formulas given in the previous sections apply even when the factors of the polynomials contain complex numbers. For example, 1 1 A B C 1 = = = + + . s3 + s s(s2 + 1) s(s − i)(s + i) s s−i s+i Then, A = lim s · s→0

1 1 = lim = 1, s(s − i)(s + i) s→0 (s − i)(s + i)

B = lim(s − i) · s→i

C =

1 1 1 1 = lim = 2 =− , s→i s(s − i)(s + i) s(s + i) 2i 2

lim (s + i) ·

s→−i

1 1 1 1 = lim = 2 =− . s(s − i)(s + i) s→−i s(s − i) 2i 2


694

Note that, if desired, the complex terms may be combined into a single term with real coefficients: 1 −1/2 −1/2 1 −s 1 = + + = + 2 3 s +s s s−i s+i s s +1

Exercise 12.82 [2+] Express each quotient as a sum of partial fractions. (a) (d)

4x2 −3x−25 (x+1)(x−2)(x+3) , 2−2x−2x2 , (x−1)(x2 +2x)

(b) (e)

5 , 1−x−6x2 2 1+x+2x +2x3 , 6+19x2 +15x4

(c) (f)

2+4x , 1+8x+15x2 2 3 x+2x −x . 1+x+x3 +x4

12.6 Some Special Functions We start with a simple version of the implicit function theorem for one variable. Theorem 12.83 If a function, F(x, y) of two variables is defined on an open disk containing (a, b), where F(a, b) = 0, Fy (a, b) 6= 0, and partial derivatives Fx , Fy are continuous on the disk, then the equation F(x, y) = 0 defines y as a function of x in a neighborhood of the point (a, b). The Lambert function, named after German scientist Johann Heinrich Lambert (1728 – 1777), also called the omega function or product log, is the inverse function of f (w) = w ew . It is a custom to denote the function by W . Hence for every real number x, we have x = W (x) eW (x) ,

x > −1/e.

The function is a solution of the ordinary differential equation x(1 +W )

dW = W, dx

x 6= −1/e.

Using the Lagrange Inversion Theorem, we found the coefficients of its Maclaurin series in Eq. (8.13) on pager 430: ∞

W (x) =

(−n)n−1 n ∑ n! x n=1

(|x| < 1/e).

For example, the equation y = xx has an “exact” solution: x = ln y/W (ln y). L. Euler introduced to the world two functions: Γ(z) =

Z ∞ 0

t z−1 e−t dt

(ℜz > 1),

B(x, y) =

Z 1 0

t x−1 (1−t)y−1 dt

(ℜx, y > 1), (12.48)

12.6. SOME SPECIAL FUNCTIONS

695

called the gamma function and the beta function, respectively. Actually, the beta function can expressed either via the gamma function or as product: B(x, y) =

Γ(x) Γ(y) , Γ(x + y)

B(x, y) =

1 + x+y x+y ∞ ∏ 1 + x 1k + y . xy k=1 k k

(12.49)

If the second argument is nonnegative integer, then the beta function has the following form: B(x, n + 1) =

n! n! . = n+1 x(x + 1)(x + 2) · · · (x + n) x

(12.50)

There are known two incomplete gamma functions and incomplete beta functions: Γ(a, x) =

Z ∞ x

B(x; a, b) =

Z x 0

t

a−1 −t

γ (a, x) =

e dt,

t a−1 (1 − t)b−1 dt

Z x 0

t a−1 e−t dt;

(0 6 x 6 1,

ℜa, b > 1).

(12.51) (12.52)

The function Γ(a, x) is called upper incomplete gamma function and the function γ (a, x) is known as lower incomplete gamma function. This function is closely related to the exponential integral, which is defined as a principal value of integral, Z −ε −t Z ∞ −t Z ∞ −t e e e def dt = − lim dt + dt , (12.53) Ei(x) = −(V.P.) ε →0 t t t −x ε −x through the equation 1 1 − ln z. Γ(0, z) = −Ei(−z) + ln(−z) − ln − 2 z The following holomorphic function in the plane except z = 1

1 ζ (z) ≡ Γ(z)

Z ∞ z−1 t 0

et − 1

dt =

1

∑ kz

(12.54)

k>1

is called Riemann zeta function, after German mathematician Bernhard Riemann (1826 – 1866); however, its definition and notation was invented by his adviser, L. Dirichlet. The function has a simple pole at z = 1, and it is regular in all other points. The Dirichlet series in right-hand side of Eq. (12.54) converges for all real z > 1. The zeta-function satisfies the following functional equation:

ζ (s) = 2s π s−1 sin

πs Γ(1 − s) ζ (1 − s). 2

The classical formula for ζ (2n) uses Bernoulli numbers of even order:

ζ (2n) =

(−1)n+1 22n−1 2n π B2n . (2n)!

(12.55)


696

No explicit evaluations of ζ (2n + 1) have been obtained. At the origin, the zeta function has the value ζ (0) = −1/2. The residue at z = 1 is 1:

ζ (s) ∼

1 +γ s−1

as

s → 1.

Stieltjes constants A sequence of constants that arise in calculations connects the zeta function with the next section. The following limits define the Stieltjes constants: ! m lnn k lnn+1 m − . (12.56) γn = lim ∑ m→∞ n+1 k=1 k They also appear in an expansion of the zeta function around the point z = 1,

ζ (z) =

∞ 1 (−1)n +∑ γn (z − 1)n z − 1 n=0 n!

=⇒

ζ (z + 1) −

∞ 1 (−1)n γn zn =∑ . z n=0 n!

(12.57)

As we can see from comparing the definition to the asymptotics of the Harmonic numbers, γ0 = γ , the usual Euler constant, approximately 0.5772156649.

12.7 Stieltjes Integrals Integration according to Riemann is not the only one possible—there are known many of its generalizations. In this subsection, we present the definition and properties of Stieltjes integral, which may lead to some notational and conceptual simplifications. In particular, this permits us to unify the discrete and continuous cases in that we no longer require separate definitions for each. Let g(x) be a monotone nondecreasing, continuous-on-the-right function with the finite difference, g(∞) − g(−∞) < M, for some M. We partition a semi-open interval (a, b] (a < b) into n subintervals determined by points a = x0 < x1 < x2 < · · · < xn = b. Then, for any function f (x) on [a, b], let f (xk ) =

sup

f (x),

f (x) =

xk−1 <x6xk

inf

xk−1 <x6xk

f (x),

and define the upper and lower Riemann-Stieltjes sums by n

n

Sn =

∑ f (xk )[g(xk ) − g(xk−1)],

k=1

Sn =

∑ f (xk )[g(xk ) − g(xk−1 )],

k=1

12.7. STIELTJES INTEGRALS

697

which depends on the partition of (a, b]. Now let n → ∞ in such a way that max (xk − xk−1 ) → i6k6n

0. Then if both limits (for Sn and Sn ) exist and equal, −∞ < lim Sn = lim Sn < ∞, n→∞

n→∞

we call this limit the Stieltjes (or Riemann-Stieltjes) integral of f (x) with respect to g(x) on R (a, b] and denote it by ab f (x) dg(x). The two functions f and g are respectively called the integrand and the integrator, respectively. You can view this definition as extending the Riemann integral from handling a uniform distribution to an evaluation method that allows for a richer space of distributions: in particular, distributions with atoms, that change values by jumps—just as the prime number distribution function π (x) does. This definition preserves almost all the properties of the Riemann integral, and introduces a few new ones, which the following exercise asks you to prove.

When g(x) is a cumulative distribution function, we have M = 1. This definition could be generalized by allowing g(x) to be a function of bounded variation (which means that such function can be written as the difference of two monotonic nondecreasing functions). Let us list the basic properties of the Stieltjes integrals. (a) Linearity:

m

Z b m

∑ Ci fi (x) dg(x) = ∑

Z b

i=1 a

a i=1

fi (x) dg(x) for constants Ci and integrable func-

tions fi . (b) Let c be a real number with a < c < b, then if f (x) is integrable with respect to g(x) on (a, b], we have

Z c

Z b

f (x) dg(x) =

Z b

f (x) d[g1 (x) + g2 (x)] =

f (x) dg(x) +

f (x) dg(x).

c

a

a

Z b

(c) If f (x) is integrable with respect to g1 and g2 , then a

Z b a

f (x) dg1 (x) +

Z b a

f (x) dg2 (x).

(d) If f (x) is integrable with respect to g(x) on (a, b], and m = inf f (x)

and

a6x6b

then m [g(b) − g(a) 6

M = sup f (x), a6x6b

Z b a

f (x) dg(x) 6 M [g(b) − g(a)].

(e) If f (x) is integrable with respect to g(x) on (a, b], then | f (x)| is integrable with respect to g(x) on (a, b], and Z b Z b a f (x) dg(x) 6 a | f (x)| dg(x). (f) If f (x) is continuous on [a, b], then

Z b

f (x) dg(x) exists.

a

(g) If f (x) has a continuous derivative on [a, b] then Z b a

f (x) dg(x) = f (b)g(b) − f (a)g(a) −

Z b a

g(x) f ′ (x) dx.


698

(h) Z

Z

b(x) d d b(x) f (x,t)dt + f (x, b(x))b′ (x) − f (x, a(x))a′ (x). f (x,t)dt = dx a(x) a(x) dx (i) If f (x) (a 6 x 6 b) has discontinuities only at x = a1 , a2 , . . . , am , and if g(x) is continuous at each of these points, then f (x) is integrable with respect to g(x) on (a, b].

Note that the Stieltjes integral does not exist if f and g have a common point of discontinuity. Exercise 12.84 (Properties of Stieltjes integral) (a) There is a simple connection between the two types of integrals when the ‘distribution’ g(x) has a ‘density’ g′ (x), which is continuous on the interval [a, b], then Z b

f (x)dg(x) =

Z b a

a

f (x)g′ (x)dx,

(12.58)

where the right-hand integral is a Riemann integral. Hint: You shall need to make explicit use of the fact that the Riemann integral is the limit of a sequence, and of the continuity of g′ (x). (b) If a and b are integers, and f () is continuous from the right at integer points, then Z b

b−1

f (x)dg(⌈x⌉) =

a

∑ f (k)∆g(k), k=a

∆g(k) = g(k + 1) − g(k).

(12.59)

(c) If a and b are integers, and f () is continuous from the left at integer points, then Z b

b

∑

f (x)dg(⌊x⌋) =

a

f (k)∇g(k),

k=a+1

∇g(k) = g(k) − g(k − 1).

(12.60)

(d) If a and b are integers, and g() is continuous from the left at integer points, then Z b

b−1

f (⌊x⌋)dg(x) =

a

∑ f (k)∆g(k), k=a

∆g(k) = g(k + 1) − g(k).

(12.61)

(e) The last two properties of the Stieltjes integral connect it with asymptotic estimates. However, you may wonder about the meaning of integrating over such an estimate — what is the Rb meaning of the Riemann integral a O( f (x))dx? Since both the integral and the big-Oh operators are linear, this would simply be the set of values obtained when functions in O( f (x)) (here: functions that are bounded by f (x) times a constant throughout the interval (a, b) are integrated). Show that Z b Z b f (x)dg(x) , (12.62) O ( f (x)) dg(x) = O a

a

if f () is positive, g() is monotone increasing (and both integrals exist, naturally). (f) Z b Z b f (x)dg(x) . f (x)dO (g(x)) = O ( f (a)g(a)) + O ( f (b)g(b)) + O a

(12.63)

a

Hint: The proof uses integration by parts, result (e) here, and another integration by parts, successively.

Appendices Appendix A: Binomial Coefficients Definitions For an integer k, the k-falling factorial, α k , and the k-rising factorial, α k , of the real or complex number α are defined by the equations:

α k = α (α − 1) · · · (α − k + 1),

α 0 = 1,

α k ≡ (α )k = α (α + 1)(α + 2) · · · (α + k − 1) =

α k+r = α k (α − k)r , Γ(α + k) , Γ(α )

(A.1)

α 0 = 1,

(A.2)

where (α )k is called the Pochhammer symbol. For negative integers k, truncated factorials are defined by the requirements ak−k = ak−k = 1, which leads to the relations:

α −k =

1 (α + 1)k

=

1 1 , = k (α + k) (α + k)(α + k − 1) · · · (α + 1)

(A.3)

α 6= −1, −2, . . . , −k; α −k =

1 1 1 , = = (α − 1)k (α − k)k (α − 1)(α − 2) · · · (α − k)

The binomial coefficients: n n! ; = k!(n − k)! k

t tk = = [zk ](1 + z)t , k k!

α 6= 1, 2, . . . , k.

n, k ∈ N; z,t ∈ C.

(A.4)

(A.5)

Relations The addition formula: n n = = 1; 0 n

n n−1 n−1 = + , k k k−1

Symmetry and negating the upper argument: n n = , 0 6 k 6 n, k, n ∈ N; k n−k

n > 1,

0 < k < n.

−x x+k−1 k (−1) = , k k

699

(A.6)

x ∈ R. (A.7)

APPENDIX

700

Shifting the arguments: t t −1 t −k+1 t t t t −1 = = , = k k−1 k t −k k−1 k k

−1 pn pn − p pn pk , = pk pk − p p p

t ∈ R.

p, k ∈ N.

Diagonal exchange: t m t t −k t t −m+k = = . m k k m−k m−k k Decomposition: n n+1 + = n2 , 2 2

(A.8)

(A.9)

(A.10)

kn n k+1 n k n+1 2 k =k +n = + . (A.11) 2 2 2 2 2 2 2

Fractional arguments:

k + 1/2 n

=

 −n    4    4−n

2n −1 2n n+k+1 n , n (−1) k+1 2k+2

2n n

2k+1 2n

k−1 n

,

(A.12) if k > n.

1 n − 1/2 n − /2 −n 2n = (−1) =4 . n n n

In particular, (−1)n −1/2 (−1)n 2n n (2n − 1)!! √ ∼ = (−1) = 4n n (2n)!! n nπ n−1 2n − 2 (−1) (−1)n−1 1/2 1 −1/2 √ = 2n−1 ∼ = 2n n − 1 2 n n−1 n 2n nπ

Reciprocals:

if k < n;

(A.13)

as n → ∞,

(A.14)

as n → ∞.

" # −1 −1 n − 1 −1 n n m . − = m−1 m−1 1−m m

(A.15)

Asymptotic estimate (from Stirling approximation):

1/2 n−k n k n n n × (A.16) = 2π k(n − k) k n−k k " 2 # 1 1 1 1 1 1 1 1 1 − − − − +O . + 1+ 12 n k n − k 288 n k n − k kn(n − k)

Appendix A: Binomial Coefficients

701

The binomial theorem. α ∑ k xk = (1 + x)α , k>0

n k n−k (x + y) = ∑ x y k=0 k n

n

(A.17)

Simple summations. Some are also given in the text, or are from the exercises. t +k t +n+1 ; ∑ k = n k=0 n

n

∑ km = k=0

(n + 1)m+1 . m+1

t t −1 t −1 k m ∑ k (−1) = m − 1 (−1) + n (−1)n , k=m n t t −1 k ∑ k (−1) = n (−1)n . k=0 2 n n n 2n n (2n − 1)! k n 2 ∑ k k = [(n − 1)!]2 . ∑ n − k = Fn+1; ∑ k = n ; k=0 k=0 k=0 2 n n n 2n + 1 2n 2n n 2n 2n−1 k 2n = 2 ; = (−1) . = 2 ; (−1) ∑ k ∑ ∑ n k k=0 k=0 2k k=0 m k+n m+n+1 n ∑ r = r+1 − r+1 . k=0 " # m n −1 n + m −1 k + n −1 n n n + 1 n 2j . = 1− , ∑ = n ∑ ∑ n−1 n 2 j=0 j + 1 k=0 n−1 k=0 k n n 1 n + k −k n (−1)k − ∑ k 2 = 2n . ∑ k x + k = x x+n x ∈/ Z ; k=1 k=0 n ( n 0, if 0 6 p < n, n ∑ (−1)n−k k (a + bk) p = bn n!, if p = n. k=1 n (r + s)n ∑ k (r + tk)k−1 (s − tk)n−k = r , r, s ∈ R. k

(A.18)

n

n (x − k)n for any real number x, k k=0 n 2 n n n n + 5n + 2 k n k n n+1 −k . = ∑ (−1) 2 + n − k = ∑ (−1) 2 k k k=0 k=0 n kn n n+k n n = ∑ (−1) . k n k=1

(A.19)

(A.20)

(A.21) (A.22)

(A.23)

(A.24)

(A.25) (A.26)

n

n! =

∑ (−1)k

(A.27) (A.28) (A.29)

APPENDIX

702

Binomial coefficients and harmonic numbers: n (−1)k+1 (x + ky)r = xr Hn + rxr−1 y, ∑ k k k>1

r 6 n ∈ Z+ ,

(A.30)

where Hn = ∑nk=1 k−1 is n-th harmonic number. The particular case where x = 1 and y = 0 is of interest: Z 1 n 1 − xn n (−1)k+1 = H = dx, n ∈ Z+ . (A.31) n ∑ k k 1 − x 0 k=1 n−1 n−1 n k n 1 k 1 = (Hn − Hm ), ∑ Hk = Hn − . (A.32) ∑ m m+1 m+1 k=m m k=m m n − k n n n r+n 2n+1 − 1 1 1 r+n−k . (A.33) = = (Hn+r − Hr ) , ∑ ∑k n+1 r r k=0 k + 1 k k=1 Abel’s identity:

n ∑ k x(x + k)k−1 (y + n − k)n−k = (x + y + n)n, k

n αβ ∑ (α + k)k−1 (β + n − k)n−k−1 = (α + β )(α + β + n)n−1 . k k=0

(A.34)

n

(A.35)

Other summations. n n n n−1 2 n ∑ i = 2 , ∑ i i = n 2 , ∑ i i = n(n + 1) 2n−2 , i i i n n ∑ i3 i = n2 (n + 3) 2n−3 , ∑ i4 i = n(n3 + 6n2 + 3n − 2) 2n−4 . i i In general, let G(n, k) = ∑i ik ni . Then G(n, k) satisfies the following recurrence: k−1

G(n, k) = n ∑

j=0

where

k j

k−1 G(n − 1, j) = j

k ∑ j n j 2n− j , j=0

(A.36)

k

(A.37)

are the Stirling numbers of the second kind.

The sums combining falling factorials and binomial coefficients have the simpler-looking value: n k n (A.38) ∑ i i = nk 2n−k . i=0 = n+1 (n > 1), A weighted sum of the triangular numbers, Tn = 1 + 2 + 3 + · · · + n = n(n+1) 2 2 is n+2 n n k+2 1 n 3 (A.39) ∑ k Tk+1 = ∑ k 2 = 2 ∑ (k + 2) = TTn+1−1 = 22 . k=1 k=1 k=1

Appendix A: Binomial Coefficients

703

Two Binomial Coefficients.

d p+d n r − sk = , (−1)k = sn . ∑ ∑ k n − k m + n n k k p d p+d ∑ m + k n + k = p − m + n , p ∈ N, m, n ∈ Z. k p p−k d +k p+d +1 = , n > d, m, p ∈ N. ∑ m n m+n+1 k=0 p d +k k m+p d − m (−1) = (−1) , m, n ∈ Z, d, p ∈ N. ∑ m+k n n− p k p−k d k m+p d − m − 1 (−1) = (−1) , n ∈ Z, p, m ∈ N. ∑ m k−n p−n−m k6p p k+m

p n n n− j p , p ∈ Z and pm < n. = ∑ (−1) m j m j=1 r+s r + tk s − tk r = , r, s ∈ R. ∑ k n n − k r + tk k n r k r+k−1 = 0, n > 1. ∑ (−1) k n−k k=0

(A.40) (A.41)

(A.42) (A.43) (A.44)

n−m

j+1

2 2n 2k 2n − 2k 2n . = ∑ 2k k n−k n k=0 p −1/2 −1/2 1/2 1/2 p . = − ∑ 2m(2m + 1) m − p m + p m+ j j=1 m − j + 1 −1/2 1/2 n k −n (−1) = 2 . ∑ k n+1−k ⌊n/2⌋ k

(A.45)

(A.46)

(A.47)

n

n

∑ k=0

2n − k n

2 n n n−k n . = ∑2 k k k=0

n/2 n k n! for even n > 2. ∑ (−1) k 2 = 2n/2 k=1 ( n 0, if 0 6 p < n, n x + kz = ∑ (−1)n−k k p zn , if p = n. k=1 n

n−k

(A.48)

(A.49)

(A.50)

(A.51)

(A.52)

(A.53)

APPENDIX

704

k ( 0, if 2p < n, n ∑ (−1)n−k k p2 = (2p − 1)!! = if 1 · 3 · 5 . . . (2p − 1), for n = 2p . k=1 k ( n 0, if pq < n, n ∑ (−1)n−k k pq = (pq)!p , for n = pq. k=1 (q!) p! n n k n/2 n! for even n > 2. ∑ (−1)n−k k 2 = 2n/2 k=1 n−r n n+r+k n −k (n−1)/2 r−n = (−1) 2 ∑ (−2) r + k n−r . k 2 k=0 n

n 0 1 2 3 4 5 6 7 8 9 10 11 12 13

n 0

1 1 1 1 1 1 1 1 1 1 1 1 1 1

n 1

n 2

n 3

1 2 3 4 5 6 7 8 9 10 11 12 13

1 3 6 10 15 21 28 36 45 55 66 78

1 4 10 20 35 56 84 120 165 220 286

n 4

n 5

n 6

n 7

n 8

n 9

n 10

1 5 1 15 6 1 35 21 7 1 70 56 28 8 1 126 126 84 36 9 1 210 252 210 120 45 10 1 330 462 462 330 165 55 11 495 792 924 792 495 220 66 715 1287 1716 1716 1287 715 286

(A.54)

(A.55)

(A.56) (A.57)

n 11

n 12

n 13

1 12 78

1 13

1

Inverse relations n an = ∑(−1) bk , k k n an = ∑ bk , k k k an = ∑ bk , n k ⌊n/2⌋ n an = ∑ bn−2k , k k=0 k

n bn = ∑(−1) ak , k k n−k n bn = ∑(−1) ak , k k k−n k bn = ∑(−1) ak , n k ⌊n/2⌋ n−k k n an−2k , bn = ∑ (−1) n−k k k=0 k

(A.58) (A.59) (A.60) (A.61)

Appendix A: Binomial Coefficients where

705 b0 = a0 ,

2k bn−k , an = ∑ k k=0 n+ p an = ∑ bk , k+ p k n n−1−k an = ∑ n kbk , k k n + 2k an = ∑ bn+2k , k k n

2k 1 an−k , bn = ∑ 1 − 2k k k=0 n+k n + p bn = ∑(−1) ak , k+ p k n+k n bn = ∑(−1) kn−k ak , k k k n + 2k n + k an+2k . bn = ∑(−1) n+k k k n

(A.62) (A.63) (A.64) (A.65)

Finite Differences x x n n−1 n , ∆x = , ∆x = x+1 x n n−1 x x x−1 n n − 2x + 1 n + 1 , ∇x = , ∇x = n x n n−1 x

(A.67)

∆x xm = mxm−1 ,

(A.68)

∇x xm = m(x − 1)m−1 ,

(A.69)

∆x xm = m(x + 1)m−1 , ∇x xm = mxm−1 ,

(A.66)

where ∆ is the forward finite difference operator and ∇ is the backward finite difference operator: ∆ f (x) = f (x + 1) − f (x), ∇ f (x) = f (x) − f (x − 1). A generalized hypergeometric function p Fq (a1 , ..., a p ; b1 , ..., bq ; x) is a function that can be defined in the form of a hypergeometric series, i.e., a series for which the ratio of successive terms can be written as ck+1 (k + a1 )(k + a2 ) · · · (k + a p ) x = . ck (k + b1 )(k + b2 ) · · · (k + bq ) k + 1 (The factor of k + 1 in the denominator is present for historical reasons of notation.) The function 2 F1 (a, b; c; x) corresponding to p = 2, q = 1 is the most frequently used hypergeometric function, and so it is frequently known as the hypergeometric equation or hypergeometric series or, more explicitly, Gauss’s hypergeometric function (Gauss 1812). This function is also commonly denoted as a, b ak bk xk (A.70) x = F (a, b; c; x) ≡ F 2 1 ∑ k k! c k>0 c

APPENDIX

706

Appendix B: The Bernoulli Numbers The Bernoulli numbers can be defined either through the exponential generating function (L.33) or by the recurrence relation (M.12), which leads to 1 n−1 n + 1 Bn = − ∑ k Bk , n + 1 k=0

B0 = 1.

n > 1,

(B.1)

The Bernoulli numbers vanish for odd indices beyond 1: B2k+1 = 0 for k > 1, and those with even indices have alternating signs (see Table 581). n

0

1

2

3

4

5

6

7

8

9

10

Bn 1 −1/2 1/6 0 −1/30 0 1/42 0 −1/30 0 5/66

11

12

0

−691/2730

The Appell polynomials over the sequence of the Bernoulli numbers are called the Bernoulli polynomials: n n Bn (x) = ∑ Bk xn−k , n > 0. (B.2) k k=0 The Bernoulli numbers, which are the values of the Bernoulli polynomials at x = 0 or x = 1, that is, Bn = Bn (0), n > 0 and Bn = Bn (1), n > 2, are known to have several explicit representations: n

1 Bn = ∑ k=0 k + 1

k n n n−1 (−1)k+1 k! n − 1 , ∑ (−1) j j = 2n − 1 ∑ 2k+1 k j=0 k=1 k

j

m

(−1)i−1 m!(m + 1)! i=1 i(i + 1)(m − i)! (m + i + 1)!

B2m = ∑

n > 0,

B0 = 1. (B.3)

i

∑ k2m

for each positive integer m.

(B.4)

k=1

The first ten Bernoulli polynomials and Bernoulli numbers are presented in Table 581. The Bernoulli polynomials satisfy the following relations: d Bn (x) = nBn−1 (x), dx Z y x

Bn (t) dt =

n dk B (x) = k! Bn−k (x). n k dxk

Bn+1 (y) − Bn+1 (x) , n+1

Z x+1 x

Bn (t) dt = xn ,

n > 1.

(B.5)

(B.6)

Symmetry relation: Bn (1 − x) = (−1)n Bn (x), Addition formula:

n > 0.

n ∑ k Bk (y) xn−k , k=0

(B.7)

n

Bn (x + y) =

n > 0.

(B.8)

Appendix B: The Bernoulli Numbers

707

Raabe’s multiplication formula: m−1

Bn (mx) = m

n−1

∑ Bn k=0

k , x+ m

n > 0,

m > 1.

(B.9)

The binomial convolutions, attributed to L. Euler: n n ∑ k Bk (α )Bn−k (β ) = n(α + β − 1)Bn−1(α + β ) − (n − 1)Bn(α + β ). k=0

n ∑ k Bk Bn−k = −n Bn−1 − (n − 1)Bn, k=0

(B.10)

n

Fourier series: Bn (x) = −

∞ n! k−n e2π ikx ∑ (2π i)n k=−∞

n > 1.

(B.11)

(0 < x < 1).

(B.12)

The sums Φm (n) = ∑nk=1 km are expressed via the Bernoulli numbers: m m (n + 1)m+1−k m Bm+1− j = Bk k = ∑ ∑ ∑ j−1 j m+1−k j=1 k=0 k k=1 1 m m+1 = ∑ k (n + 1)m−k+1Bk , m + 1 k=0 m+1

n

m

(B.13)

or more succinctly via the Bernoulli polynomials: n

1

∑ km = m + 1 [Bm+1(n + 1) − Bm+1(1)] .

(B.14)

k=1 n

bp

∑ (a + bk) p = p + 1 [B p+1(n + 1 + a/b) − B p+1(a/b)] ,

p > 1,

n > 0.

(B.15)

k=1

Asymptotics Since the zeta function, ζ (s) = ∑k>1 k−s , can be expressed via Bernoulli numbers for even integer values as ∞

ζ (2n) =

∑ k=1

1 (2π )2n = (−1)n−1 B2n ∼ 1, 2n k 2(2n)!

as n → ∞ ,

(B.16)

we get the following approximations: B2n ∼ (−1)n+1 2

(2n)! , (2π )2n

(2π )n nπ , Bn (x) ∼ cos 2π x + π + 2 (n)! 2

n → ∞.

(B.17)

APPENDIX

708

Appendix C: Stirling and Euler/Eulerian Numbers Generating Recurrences Stirling numbers of the first kind, sn,k or s(n, k), signless (or cycle) Stirlingnumbers of the first kind, nk = |sn,k | = (−1)n−k sn,k , and Stirling numbers of second kind, nk , are defined via following recurrences: (C.1) sn,k = sn−1,k−1 − (n − 1)sn−1,k , sn,0 = δn,0 , sn,k = 0 for n < k; n n−1 n−1 = (n − 1) + , n > 1; initial values in (C.10). (C.2) m m m−1 n n−1 n−1 =m + , n > 1; initial values in (C.10). (C.3) m m m−1 The Stirling number of the second kind, nk , equals the number of partitions of a set of n distinguishable objects into k unordered nonempty subsets. The sum of Stirling numbers of the second kind is denoted by Bn : n ∞ n n k (Dobinski’s formula), (C.4) Bn = ∑ = e−1 ∑ k k=1 k=1 k! and is called the Bell number. The Bell numbers, Bn , are defined via the recurrence relation (M.11) and they count the (total) number of partitions of a set of n distinguishable elements into unordered nonempty subsets. The Bell numbers can be expressed as the sum: Bn = ∑

k1

!(1!)k1

1 n! ∼ √ λ (n)n+1/2 eλ (n)−n−1 k k 2 n n k2 !(2!) . . . kn !(n!)

as n → ∞, (C.5)

where the summation is extended over all ki > 0, i = 1, 2, . . . , n, with k1 + 2k2 + · · · + nkn = n, and λ (n) ln λ (n) = n. They also are related to the derangement numbers: s n j s s− j s n n! ∑ (−1) n Bj = ∑ k Dk , whenever n > s. (C.6) j k j=0 k=0

Special values n n n n n n = = =0 if m > n; = = = 1; m m m n n n n n n = = ; n−1 n−1 2 n n n n n n =3 +2 ; =3 + ; n−2 4 3 n−2 4 3

(C.7) (C.8) (C.9)

Appendix C: Stirling and Euler/Eulerian Numbers n 0 1 2 3 4 5 6 7 8 9

n 0

n

n

1

n

2

3

n

709 n

4

5

n n n n 6

7

8

1 0 1 0 1 1 0 2 3 1 0 6 11 6 1 0 24 50 35 10 1 0 120 274 225 85 15 1 0 720 1764 1624 735 175 21 1 0 5040 13068 13132 6769 1960 322 28 1 0 40320 109584 118124 67284 22449 4536 546 36

9

1

Table 709: Signless Stirling’s numbers of the first kind.

Asymptotics

n n n n = (n − 1)!, = 1; = = δn,0 ; 1 1 0 0 n+1 n = n! Hn ; = 2n−1 − 1; 2 2 n 1 n−1 n+1 n! 2 (2) = Hn − Hn ; 3 − 2n + 1 . = 3 2 2 3

n (γ + ln n)m−1 , m ∈ o(ln n); ∼ (n − 1)! (m − 1)! m

n+m

lim m m→∞ m2n

Inverse and orthogonal relations n−k

an = ∑(−1) k

n bk , k

n k ∑ k m (−1)k = (−1)n δm.n ; k n k n! n − 1 = ∑ k m m! m − 1 ; k

(C.11) (C.12)

n+1 lim m = 1. (C.13) n→∞ n n m n+1 lim mn = m. (C.14)

1 = n ; 2 n!

n+m m2n ∼ n , n ∈ o(m1/2 ); m 2 n!

n 1 lim m = ; n→∞ m! m −n

(C.10)

n→∞

m

n bn = ∑ ak . k k n k ∑ k m (−1)k = (−1)n δm,n . k n+1 k k m n (−1) = (−1) . ∑ k+1 m m k

(C.15)

(C.16) (C.17)

APPENDIX

710

k m+1 m! ∑ n k + 1 (−1)k = (−1)n n! , if m > n, k n m n+1 Bm+1 (n + 1) − Bm+1 , k! = ∑ k k+1 ∑ km = m+1 k k=1

otherwise 0.

(C.18)

n, m ∈ N,

(C.19)

where Bm+1 and Bm+1 (x) are Bernoulli number and polynomial, respectively. Compare with Eq. (F.14).

Sum representations n 1 = k! k

k k−i

∑ (−1)

i=0

k n i , i

n

∑ k=1

k n+1 k= , k−r n−r

k+ p n+ p+1 , ∑ k k= n k=1

r > 0;

(C.20)

n

n 0 1 2 3 4 5 6 7 8 9 10

n n n 0

1

1 0 0 0 0 0 0 0 0 0 0

2

n 3

n 4

n 5

p > 0. n 6

(C.21)

n n n 7

8

1 1 1 1 3 1 1 7 6 1 1 15 25 10 1 1 31 90 65 15 1 1 63 301 350 140 21 1 1 127 966 1701 1050 266 28 1 1 255 3025 7770 6951 2646 462 36 1 511 9330 34105 42525 22827 5880 750

9

1 45

Table 710: Stirling’s numbers of the second kind. n k n+1 ∑ k m = m+1 ; k n 1 n+1 k 1 = ; ∑ n! m + 1 k=0 m k!

k n n+1 ∑ m k = m+1 ; k n k n+1 n−k (m + 1) = ; ∑ m+1 k=0 m

k+m n+m (−1)k = 0, ∑ k k − m k

n > m;

(C.22) (C.23)

(C.24)

Appendix C: Stirling and Euler/Eulerian Numbers

711

mn n 1 = ; ∑ m! k=1 k (m − k)! m

n ∑(−1) k = 0, n > 2; k n n+1 n k−m ∑ k + 1 (−n) = m ; k=m k

(C.25)

n ∑(−1) k! k = 1; k n n ∑ k k = n! Hn ; k=1 n−k

n n ∑(−1) (k − 1)! k = 0, n > 2; ∑ k = n!. k k m−n m+n m+k n = , n > m; ∑ m+k n+k k n − m k m−n m+n m+k n = , n > m; ∑ n+k n+k k n−m k n−r n n−k k m n ∑ k r m − r = r m , r 6 m 6 n; k=m−r n−r n n−k k m n = , r 6 m 6 n; ∑ k r m−r r m k=m−r n n ∑ k 2k = (n + 1)! ; k=0 m n k + m n − 1 k! m = ∑ k! . n =∑ k k k k k−1 n k k k

n k k n n n (zDz ) = ∑ zD, z D =∑ (−1)n−k (zDz )k . k k k k n n def d (az Dz )n = anz ∑ (ln a)n−k Dkz , Dz = . dz k=0 k n

Generating functions Lower argument (n is a positive integer): n n n n x n k n−k k n (−1) x = ∑ k! ; x =∑ x =∑ k k k=1 k k k=1 k n n (−1)n−k xk , x =∑ k k n n n k n x = ∑ |sn,k | xk = ∑ (n − 1)n−k xk . xn = (x + n − 1)n = ∑ k k k=0 k k=1

(C.26) (C.27)

(C.28) (C.29) (C.30) (C.31) (C.32) (C.33) (C.34)

(C.35) (C.36)

(C.37) (C.38) (C.39)

APPENDIX

712

Upper argument:

1 ln 1−z

n

k k z ; = n! ∑ k>0 n k!

k k z (e − 1) = n! ∑ k>0 n k! z

n

(C.40)

Eulerian and the Euler Numbers The Eulerian numbers, denoted by

n k

, are generated by the recurrence

n n−1 n−1 = (k + 1) + (n − k) , n, k > 0, (C.41) k k k−1

n

n subject to the boundary conditions n0 = 1, k = 0, 0 < n 6 k. The Eulerian number k is the

number of permutations of the set [1..n] that have exactly k ascents. In other words, the nk is the number of permutations hπ1 π2 . . . πn i of the set [1..n] that have exactly k places where πi < πi+1 . Caution: More than any other array of numbers we show, the notation and exact definitions of Eulerian numbers are not uniform7 .

Like the binomial coefficients, the Eulerian numbers show symmetry in each row: n n = , n > 0. k n−k−1

(C.42)

Beyond the recurrence Eq. (C.41), similar to the other arrays in this appendix, the Eulerian numbers can be determined from several other identities that can be viewed as recurrences, and, naturally from its GF, which is presented in Eq. (C.48). Some particular cases: n n = 1, = 2n − n − 1, 0 1

n n+1 n n = 3 − (n + 1) 2 + . 2 2

(C.43)

Identities: n−1 n k z z. n+1 ∑ k (1 − z) i>1 k=0 k n n+1 =∑ (k + 1 − j)n (−1) j = k j j=0 m m k n! =∑ . n k m−n k

∑ in zi =

(C.44) n n− j (−1)n−k− j , ∑ j! j k j>0

(C.45) (C.46)

7 You need to verify, when using “Eulerian number” from different sources, what their numbers represent. This is possible through a combinatorial definition as above; since the numbers in fact crop up in several such contexts; it is best to refer to Table 713 of values or generating functions (C.48), (C.50), all given here.

Appendix C: Stirling and Euler/Eulerian Numbers

713

n x+k x =∑ , k n k n

Worpitzky’s identity [142] :

n > 0.

(C.47)

Generating functions n k zn u−1 . = u ∑∑ n! u − ez(u−1) n>0 k>0 k n n u−1 k z z(u−1) . = 1 + u e ∑ ∑ n! u − ez(u−1) n>1 16k6n k − 1 x e−yt − y e−xt n + m xn−1 ym−1 (−t)m+n−1 1 ln =∑ ∑ . xyt x−y (n + m)! m n>1 m>1 n 0 1 2 3 4 5 6 7 8 9 10

n 0

n 1

n 2

n 3

n 4

n 5

n 6

n 7

(C.48) (C.49) (C.50)

n n 8

1 1 0 1 1 0 1 4 1 0 1 11 11 1 0 1 26 66 26 1 0 1 57 302 302 57 1 0 1 120 1191 2416 1191 120 1 0 1 247 4293 15619 15619 4293 247 1 0 1 502 14608 88234 156190 88234 14608 502 1 1 1013 47840 455192 1310354 1310354 455192 47840 1013

9

0 1

Table 713: Eulerian numbers. The inner sum in Eq. (C.48) is called the Eulerian polynomial: n−1 n k En (u) = ∑ u , n > 1, E0 (u) = 1. k=0 k n n 1 En (u) d n 1 n i = ∑i u = = u ∑ k − 1 uk . du 1 − u (1 − u)n+1 i>0 (1 − u)k+1 k=0 The Eulerian polynomial can be also calculated through the recurrence: n n uk dEn−1 (u) = ∑ k! . En (u) = nu En−1 (u) + u(1 − u) du k (1 − u)k+1 k=1

(C.51) (C.52)

(C.53)

APPENDIX

714

n Setting u = 1 in Eq. (C.53), we obtain En (1) = ∑n−1 k=0 k = n!. The Euler polynomial En (x) of degree n is the unique polynomial solution of the equation En (x) + En (x + 1) = 2 xn . The Euler polynomials, En (x), and the Euler numbers, En , n = 0, 1, 2, . . ., are related through n 1 n−k −k , Ek 2 x− En (x) = ∑ 2 k=0 k n

d En−1 (x) = n En−1 (x), dx

(C.54)

and the Euler numbers can be expressed as the values of the Euler polynomials at x = 1/2: n

En = 2

E(1/2),

E0 = 1,

n−1

E2n = − ∑

E2n+1 = 0,

k=0

2

4

6

8

10

12

2n E2k . 2k

n

0

En

1 −1 5 −61 1385 −50521 2702765 −199360981 19391512145

The Euler polynomials are related at points x = 0 and x = 1: Explicit expressions of first five Euler polynomials:

14

(C.55) 16

En (1 − x) = (−1)n E(x).

1 3x2 1 + x3 , E4 (x) = x − 2x3 + x4 . E0 (x) = 1, E1 (x) = x − , E2 (x) = x2 − x, E3 (x) = − 2 4 2

Asymptotics r 2n n 4n 480n2 + 9 as n → ∞. E2n ∼ (−1) 8 π π e 480n2 − 1 ( n+1 sin(π x), n is even, ⌊(n+1)/2⌋ π En (x) −→ (−1) n→∞ 4 (n!) cos(π x), n is odd. n

The following sequence of constants is called the Stieltjes constants: ! m lnn k lnn+1 m . − γn = lim ∑ m→∞ n+1 k=1 k

(C.56)

(C.57)

(C.58)

They also appear in an expansion of the zeta function around the point z = 1,

ζ (z) =

∞ ∞ 1 1 (−1)n (−1)n γn zn +∑ . γn (z − 1)n =⇒ ζ (z + 1) − = ∑ z − 1 n=0 n! z n=0 n!

(C.59)

As we can see from comparing the definition to the asymptotics of the Harmonic numbers, γ0 = γ , the usual Euler constant, approximately 0.5772156649.

Appendix D: Fibonacci Numbers

715

Appendix D: Fibonacci Numbers Fibonacci numbers were given their name in 1876 by the French mathematician Francois Edouard Anatole Lucas (1842 – 1891). Leonardo Pisano Bigollo (1170–1250), also called Leonardo Fibonacci or Leonardo Pisano, was born into the Bonacci family of Pisa. Leonardo was one of the greatest mathematicians in the Middle Ages known to us; he was well-known due to his book Liber abbaci (Book of Calculation), first published in 1202. His significance possibly lies mainly in popularizing the Hindi-Arabic number system we use now, as a replacement of the Roman numbers then in use. Leonardo is known now by his nickname “Fibonacci,” which is a contraction of “Filius Bonacci,” son of Bonacci. The Fibonacci polynomials, Fn (x), are defined by the recurrence relation Fn+1 (x) = xFn (x) + Fn−1 (x),

n > 2,

F1 (x) = 1,

An explicit formula for the Fibonacci polynomials: n − j − 1 n−2 j−1 Fn (x) = ∑ x , j j

F2 (x) = x.

n > 0.

(D.1)

(D.2)

The Fibonacci numbers, usually denoted by Fn , are Fibonacci polynomials with x = 1. They are defined for negative argument as F−n = (−1)n+1 Fn . The order of Fn (x) is n − 1. F3 (x) = 1 + x2 ,

F4 (x) = 2x + x3 , F5 (x) = 1 + 3x2 + x4 , F6 (x) = 3x + 4x3 + x5 . " √ !n √ √ !n # 1+ 5 (1 + 5)n 1− 5 1 ∼ √ − , n → ∞, (D.3) Fn = √ 2 2 5 5 × 2n √ √ 1+ 5 1 1 − 5 1 n ˆ n , golden ratio, φˆ = − = . (D.4) where φ = Fn = √ φ − φ , 2 φ 2 5 √ √ 1+ 5 1− 5 π π ˆ = 2 cos , φ = = −2 cos , φ= φ n = Fn−1 + φ Fn . (D.5) 2 5 2 10 Fn = 2n−1

αn − β n , α −β

π n−1 cosn π5 − (−1)n sinn 10 π π n−1 (−1)k cosn−k−1 sink . = 2 ∑ π π cos 5 + sin 10 5 10 k=0

(D.6)

p 1 β = (x− x2 + 4) = −e−θ . (D.7) 2 p sinh 2nθ cosh(2n − 1)θ , F2n (x) = , 2 cosh θ = x2 + 4. (D.8) F2n−1 (x) = cosh θ cosh θ Let wn = F2n±p , then wn satisfies the recurrence relation Fn (x) =

p 1 α = (x+ x2 + 4) = eθ , 2

wn+2 = 3wn+1 − wn ,

n > 0.

(D.9)

APPENDIX

716

the Lucas polynomials, Ln (x), are defined by the same recurrence as for the Fibonacci polynomials but with different initial conditions: Ln+1 (x) = xLn (x) + Ln−1 (x),

L1 (x) = x,

L2 (x) = 2 + x2 .

(D.10)

For example, L3 (x) = 3x+x3 ,

L4 (x) = 2+4x2 +x4 ,

L5 (x) = 5x+5x3 +x5 ,

L6 (x) = 2+9x2 +6x4 +x6 .

The Lucas polynomials satisfy three properties: Ln (x) = Fn+1 (x) + Fn−1 (x),

xLn (x) = Fn+2 (x) − Fn−2 (x),

L−n (x) = (−1)n Ln (x).

The Lucas numbers, usually denoted by Ln , are just Lucas polynomials at x = 1.

Identities Fn−1 Fn+1 − Fn2 = (−1)n ,

n > 1.

Fn+m = Fm Fn+1 + Fm−1 Fn

Cassini identity

m > 1, n > 0.

2 Fn2 − Fn−1 − Fn Fn−1 = (−1)n+1 ,

n > 1.

(D.12)

n > 1.

(D.13)

n > 2.

(D.14)

Hoggatt identity

(D.15)

F2n−1 = Fn Fn+1 − Fn−1Fn−2 , 2 2 F2n = Fn+1 − Fn−1 ,

3 3 F3n = Fn+1 + Fn3 − Fn−1 .

(D.16)

2 2 + 1 = 3F6n+7 F6n+5 . + F6n+5 F6n+7

Fn = Fk+1 Fn−k + Fk Fn−k−1 ;

(D.17)

Fn = (−1)n [Fk Fn+k+1 − Fk+1 Fn+k ].

∆(n2 Fn ) = (n + 1)2 Fn+1 − n2 Fn = n2 Fn−1 + (2n + 1)Fn+1 . 2 4 4 Fn+1 + Fn4 + Fn−1 = 2 2Fn2 + (−1)n . 2 F2n+1 = Fn2 + Fn+1 ,

F2n = 2Fn Fn+1 − Fn2 ,

F2n = Fn Ln , 2Fi+ j = Fi L j + Fj Li ,

(D.11)

n > 0.

Ln+1 = Fn + Fn+2 . 2Li+ j = Li L j + 5Fi Fj .

F2n−1 = Fn+1 Ln+2 − Ln Ln+1 . q q 2 2 2Fn+1 = Fn + 1 + 5Fn − 2Fn + 1 ; 2Ln+1 = Ln + 1 + 5Ln − 2Ln + 1 .

(D.18) (D.19) (D.20) (D.21) (D.22) (D.23) (D.24) (D.25)

Inequalities 2 Fn2 6 F2n 6 Fn+1 ,

2 F2n F2m 6 Fn+m .

(D.26)

Appendix D: Fibonacci Numbers

717

Sums n

n

∑ k=1

Fk = Fn+2 − 1,

∑ k Fk = (n − 2)Fn+1 + (n − 1)Fn + 2,

n > 0;

(D.27)

k=1

n

∑ (−1)k kFk = (−1)n (n + 1)Fn−1 + (−1)n−1Fn−2 − 2,

n > 2.

(D.28)

k=0

n

∑ k2 Fk = (n2 − 4n + 8)Fn+1 + (n2 − 2n + 5)Fn − 8.

(D.29)

k=1 n

n

∑ F2k−1 = F2n,

5 ∑ F2k+1 = (F2n+3 + F2n+1 ) + 2.

k=1

(D.30)

k=0

n

n

5 ∑ (−1)k F2k = (−1)n (F2n+2 + F2n ) − 1.

∑ F2k = F2n+1 − 1,

k=1

(D.31)

k=1

n

n

∑ F4k−1 = F2nF2n+1, k=1

2 − 1. ∑ F4k = F2n+1

(D.32)

k=1 n

n

10 ∑ Fk3 = F3n+2 + (−1)n+1 6Fn−1 + 5.

∑ Fk2 = FnFn+1,

(D.33)

k=1

k=1 n

2 5 ∑ F2k = F4n+2 − 2n − 1, k=0

n−1

2 5 ∑ F2k+1 = F4n + 2n.

(D.34)

k=0

n ∑ k Fk = F2n, k=1

n n k n+1 ∑ k Fj+n−k = Fj+2n, ∑ (−1) k Fk = 0. k k=1 2n 2n k 2n k−1 n k 2n ∑ (−1) k 2 Lk = 5 , ∑ (−1) k 2k−1 Fk = 0. k=0 k=0 n n n 2 n−1 Fk+2 = ∑ F2k+5 . ∑ k k=0 k k=0 n n n 2 n 2 n n+k n n+k n n L2k = ∑ Ln−k . F2k = ∑ Fn−k , ∑ ∑ k k k=0 k k=0 k k=0 k k=0 k n

(D.35)

(D.36) (D.37)

(D.38)

n

Fn+2 Fk−1 = 1− n . k 2 k=1 2

∑

(D.39)

n

8

4 . ∑ F2k+1F3k+2F6k+2 = F3n+3

(D.40)

k=0 n

1

n2

∑ Fk > Fn+2 − 1 ,

k=1

n > 1.

(D.41)

APPENDIX

718

Generating Functions x = ∑ Fn xn ; 1 − x − x2 n>0

1 = ∑ Fn+1 xn ; 1 − x − x2 n>0

(D.42)

2−x = ∑ Ln xn ; 1 − x − x2 n>0

1 + 2x = ∑ Ln+1 xn ; 1 − x − x2 n>0

(D.43)

1−x = ∑ F2n+1 xn ; 1 − 3x + x2 n>0 2x = ∑ F3n xn ; 1 − 4x − x2 n>0

x = ∑ F2n xn ; 1 − 3x + x2 n>0

n 1−x = ∑ ∑ Fn−k Fk xn ; (1 − x)(1 − 2x)(1 − 3x) + x4 n>0 k=0

3 − 2x = ∑ L2n+2 xn ; 1 − 3x + x2 n>0 x − x2 = ∑ Fn2 xn ; 1 − 2x − 2x2 + x3 n>0 4 − 7x − x2 = ∑ L2n xn ; 1 − 2x − 2x2 + x3 n>0

(D.44) (D.45)

x + x2 = ∑ L2n+1 xn+1 ; 1 − 3x + x2 n>0

(D.46)

x = ∑ Fn Fn+1 xn ; 1 − 2x − 2x2 + x3 n>0

(D.47)

9 − 2x − x2 = ∑ L2n+2 xn ; 1 − 2x − 2x2 + x3 n>0

(D.48)

x − 2x2 − x3 = ∑ Fn3 xn ; 1 − 3x − 6x2 + 3x3 + x4 n>0

(D.49)

2x = ∑ Fn Fn+1 Fn+2 xn . 1 − 3x − 6x2 + 3x3 + x4 n>0

(D.50)

Fm + Fm−1x = ∑ Fm+n xn ; 1 − x − x2 n>0

Lm + Lm−1 x = ∑ Lm+n xn . 1 − x − x2 n>0

(D.51)

t = ∑ Fn (x)t n ; 1 − xt − t 2 n>0

2 − xt = ∑ Ln (x)t n . 1 − xt − t 2 n>0

(D.52)

Fj + (−1) j Fm− j x = ∑ Fmn+ j xn . 1 − Lm x + (−1)m x2 n>0 ˆ eφ cos x sin(φ sin x) − eφ cos x sin(φˆ sin x) =

√

(D.53)

∞

sin nx Fn . n=1 n!

5∑

(D.54)

Appendix E: Harmonic Numbers

719

Appendix E: Harmonic Numbers The s-harmonic (or generalized harmonic) number is defined as n

(s)

Hn ≡ ζn (s) =

∑ k=1

1 , ks

n

(1)

Hn ≡ Hn ≡ ζn (1) =

∑ k=1

1 . k

(E.1)

(s)

A common term for Hn is the incomplete zeta function, and it is frequently denoted by (s) (1) ζ (n; s) or ζn (s). A hyperharmonic number hn of order s is defined as follows. Let hn = Hn (s−1) (s) (s) . and for positive integer s > 1, define hn recursively: hn = ∑ni=1 hi (s) hn

n+s−1 = [Hn+s−1 − Hs−1] , s−1

see Eq. (A.33).

(E.2)

2n

n

∑ (−1)k Hk = Hn/2,

∑ Hk = (n + 1)(Hn+1 − 1) = (n + 1)Hn − n;

k=1

(E.3)

k=1

n(n − 1) n(n + 1) 1 n(n + 1) Hn+1 − . ∑ k Hk = 2 Hn − 4 = 2 2 k=1 n

n

∑ k=1

Hk2 = (n + 1) Hn2 + 2n − (2n + 1) Hn , n

∑

h

k=1 n h

∑

k=1

(2) Hk2 + Hk (2)

Hk2 − Hk

i

i

n

= (n + 1)

h

n

(2)

∑ Hk k=1

(2) Hn2 + Hn

i

(2)

= (n + 1)Hn − Hn .

− 2(n + 1) Hn + 2n,

i h (2) = (n + 1) Hn2 − Hn − 2n Hn + 2n. (2)

2 − Hn+1 ) − 2(n + 1)Hn + 2n. ∑ Hk Hn+1−k = (n + 2)(Hn+1

(E.4)

(E.5)

(E.6) (E.7)

(E.8)

k=1

1 n (n − 1)(4n + 1) . (n + 1)(2n + 1)H − n ∑ 6 6 k=1 n 10n − 1 n(n + 1) 2 (2n + 1)(2H2n − Hn ) − . ∑ k Hn+k = 6 6 k=1 n n−1 (−1)n k 2 . ∑ (−1) k Hk = 2 n(n + 1)Hn − 2 k=1 n

n

k2 Hk =

∑ k Hk2 = k=1

n(n − 3) n(n + 1) 2 1 − n2 + n Hn + Hn + . 2 2 4

(E.9) (E.10) (E.11)

(E.12)

APPENDIX

720 n

∑ k=1 n

∑ k=1

n

i Hk 1 h 2 (2) Hn + Hn , = k 2

∑ k=1 n

Hk (2) 2 = Hn+1 − Hn+1 , n+1−k n

∑ k=2

Hk

∑ k(k − 1) = 2 −

k=2 n

∑ (−1)k kHk = k=1 n

∑

(2) (−1)k Hk

=

k=1

(

i Hk 1h 2 (2) Hn+1 − Hn+1 . = k+1 2 i 1 Hk 1h 2 (2) Hn−1 + Hn−1 + 1 − . = k−1 2 n

Hn+1 1 − n n+1

−→

1 n+1 2 Hn − 4 Hn/2 , 1 n − 2 Hn − 2 − 41 H(n−1)/2 ,

1 (2) 4 Hn/2 , (2) 1 (2) 4 H⌊n/2⌋ − Hn ,

2.

n→∞

(E.13) (E.14)

(E.15)

if n is even; if n is odd.

(E.16)

if n is even,

(E.17)

if n is odd.

k n+1 1 , Hk = Hn+1 − ∑ r+1 r+1 k=1 r n 1 r r−1 1 r−1 k n − . Hn+1 − ∑ k (−1) Hk = (−1) r n+1 r n k=1 n

1 1 n 1 − , =∑ Hn = ∑ = ∑ k k k + n k(k + n) k>1 k>1 k=1 n

Hn2 17π 4 ∑ 2 = 360 , n>1 n Hn+1 ∑ n(n + 1) = 2, n>1

H2n = 2n

1 . 2 − n2 k k>n+1

∑

Hn Hn 1 = 2 ∑ 3 = 2ζ (3) = 2 ∑ . 2 2 n>1 n n>1 n n>1 (n + 1)

∑

∞ ∞ ∞ Hn 1 Hn Hn = + = 5 ∑ n3 ∑ n4 ∑ (n + 1)3 ∑ (n + 1)3 . n=1 n=1 n=1 n=1

(E.18) (E.19)

(E.20) (E.21)

∞

1 1 < Hn − ln −γ < , 2n + 2/5 2n + 1/3

γ < Hn + Hm − Hnm 6 1,

p

(E.22)

(E.23)

1 where γ = 0.577215665 . . . = ∑ p>2 (−1) p ∑k>1 k p is Euler’s constant. For any positive integer r, i h i h (E.24) r 1 − (n + 1)−1/r < Hn < r n1/r − 1 + 1.

n=

1 2

3

4

5

6

7

8

9

10

Hn

1

3 2

11 6

25 12

137 60

49 20

363 140

761 280

7129 2520

7381 2520

1

5 4

49 36

205 144

5269 3600

5369 3600

266681 176400

1077749 705600

9778141 6350400

1968329 1270080

(2)

Hn

Appendix F: Miscellaneous Formulas

721

Appendix F: Miscellaneous Formulas Summations n

1

∑ km = m + 1 (n + 1)m+1.

(F.1)

k=0

Kronecker’s delta function is

δn,k = δn−k,0 = δ0,n−k =

if n = k, otherwise.

1, 0,

(F.2)

Abel’s partial summation formula: n

n

∑ ak bk = Anbn+1 + ∑ Ak (bk − bk+1 ),

k=1

(F.3)

k=1

where Ar = a1 + a2 + · · · + ar and Br = b1 + b2 + · · · + br (r = 1, 2, . . . , n) are r-partial sums of the sequences. The Lagrange identity: !2 ! !

∑

16k<m6n

(ak bm − am bk )2 =

a2k

∑

∑

16m6n

16k6n

n

n

∑

(ak bm − am bk )(Ak Bm − Am Bk ) =

16k<m6n

m=1

n

ℜ

∑ ak bk k=1

!

1 6 2n

,

1 2

k=1

n

(F.4)

n

n

m=1

(F.5)

k=1

n

(F.6)

! 2−1 n 4n ∑ |ak |2 + 3 ∑ |bk |2 . k=1 k=1

(F.7)

i=1

j=1

ak bk

∑ ∑ (ai − a j )(b j − bi).

∑ ai ∑ b j = n ∑ ai bi +

i=1

∑

16k6n

∑ bm Bm ∑ ak Ak − ∑ bm Am ∑ ak Bk .

n

n

n

b2m −

i=1 j=1

n

def

Some particular cases of the sums Φm (n) = ∑nk=1 km are as follows n n n+1 n(n + 1) n(n + 1) = , , (−1)k (n − k) = Φ1 (n) ≡ ∑ k = ∑ 2 2 2 k=1 k=1 n n n(n + 1) 2 n(n + 1)(2n + 1) 1 2n + 2 3 , = , k = Φ2 (n) ≡ ∑ k2 = ∑ 6 4 2 3 k=1 k=1 6 1 n(n + 1)(2n + 1)(3n2 + 3n − 1) = Φ2 (n) Φ1 (n) − , Φ4 (n) ≡ ∑ k = 30 5 5 k=1 n 4 1 n2 (n + 1)2 (2n2 + 2n − 1) = Φ21 (n) Φ1 (n) − , Φ5 (n) ≡ ∑ k5 = 12 3 3 k=1 n

4

(F.8) (F.9)

(F.10) (F.11)

APPENDIX

722 n

n(n + 1)(2n + 1)(3n4 + 6n3 − 3n + 1) 42 k=1 6 1 12 2 Φ1 (n) − Φ1 (n) + , = Φ2 (n) 7 7 7 n n2 (n + 1)2 (3n4 + 6n3 − n2 − 4n + 2) Φ7 (n) ≡ ∑ k7 = 24 k=1 1 4 2 2 = Φ1 (n) 2Φ1 (n) − Φ1 (n) − . 3 3 Φ6 (n) ≡

∑ k6 =

(F.12)

(F.13)

In general, the sums Φm (n) = ∑nk=1 km can be determined either through the Bernoulli numbers m n B2 j m nm+1 nm + +∑ nm− j+1 , (F.14) Φm (n) = ∑ km = m + 1 2 j j − 1 j=2 k=1 m−1 m m or the recurrence: ∑k=0 k Φk (n) = (n + 1) − 1, or using multisection, p 2p + 1 2∑ Φ2p−2k (n) = (n + 1)2p+1 + n2p+1 − 1, 2k + 1 k=0 (F.15) p 2p 2p 2p Φ2p−2k+1 (n) = (n + 1) + n − 1. 2∑ k=0 2k + 1 def

def

All functions Φm (n) = ∑nk=1 km can be expressed via the first two values, Φ1 (n) = ∑nk=1 k = def n(n + 1)/2 and Φ2 (n) = ∑nk=1 k2 = n(n + 1)(2n + 1)/6: ( m/2−1 if m is even, Φ2 (n) ∑ j=0 b j Φ1j (n), (F.16) Φm (n) = (m−3)/2 j 2 c j Φ1 (n), if m is an odd number, 3 or larger; Φ1 (n) ∑ j=0 where the coefficients b j and c j are called the Faulhaber polynomials. Another generalization: n an+1 − 1 , (F.17) Φa,m (n) = ∑ ak km , Φa,0 (n) = a−1 k=1 Φa,m (n + 1) − Φa,m (n) = an+1 (n + 1)m . The Bell polynomials8 , which are associated with set partitions, are defined as m 1 Bm,k (x1 , x2 , . . . , xm−k+1 ) = ∑ j =m j1 . . . jk x j1 . . . x jk , k! j1 + j2 +···+ k

(F.18)

(F.19)

ji >1

where B0,0 = 1; the sum is effectively over set partitions of [1..m] with block sizes j1 , . . . , jk , with the factor 1/k! correcting for the multiple counting inside the sum. Only m − k + 1 8 The

name “Bell polynomials” was introduced by John Riordan.

Appendix F: Miscellaneous Formulas

723

variables are necessary because no block can contain more than m − k + 1 elements. For instance, B4,2 (x1 , x2 , x3 ) = 4x1 x3 +3x22 since there are 7 partitions of the set [1..4] = {1, 2, 3, 4} into two blocks, 4 with one block of size three and one of size one, and 3 with two blocks of size two. Faà-di-Bruno’s formula9 : dr [ f (g(z))] = dzr

r

r

∑ f [ j] (g(z)) ∑ Fj,rk ∏ j=1

k

l=1

h ikl g[l] (z) ,

(F.20)

r! j r r! = (i!)−ki , and the sum are given by = r where the coefficients ∏ k i k ! (i!) j! k ∏i=1 i i=1 over k ranges over all nonnegative r-dimensional vectors that satisfy ∑i ki = j and ∑i iki = r ; that is, on all j-part partitions of r, but arranged by part count instead of the usual, explicit k k k 1 2 r way: k = 1 , 2 , . . . , r . This formula can be rewritten as !kr ′ k1 ′′ k2 (r) (z) g (z) g (z) r! g dr , (F.21) [ f (g(z))] = ∑ ··· f (k) (g(z)) dzr k1 ! k2 ! . . . kr ! 1! 2! r! k Fj,r

k Fj,r

where the sum is over all different solutions in nonnegative integers k1 , k2 , . . . kr of k1 + 2k2 + · · · + rkr = r and k = k1 + · · · + kr . For example, if r = 3 we have 3 d3 [ f (g(z))] = f ′ (g(z)) g′′′ (z) + 3 f ′′ (g(z)) g′ (z)g′′ (z) + f ′′′ (g(z)) g′ (z) . 3 dz

Faà-di-Bruno’s formula can be expressed as set partition version

dr [ f (g(z))] = ∑ f (k) (g(z))(g′ (z))k1 (g′′ (z))k2 · · · (g(r) (z))kr , dzr

(F.22)

where the sum is over all partitions of [1..r], and, for each partition, r is its number of blocks and ki is the number of blocks with exactly i elements; or via Bell polynomials: r dr [ f (g(z))] = ∑ f (k) (g(z))Br,k (g′ (z), g′′ (z), . . . , g(r−k+1) (z)). dzr k=0

Jensen’s inequality for a real continuous function f : ! 1 n 1 n 1 n or f (xi ) 6 f xi ∑ ∑ ∑ f (xi ) > f n i=1 n i=1 n i=1

1 n

n

∑ xi

i=1

!

(F.23)

(F.24)

depending whether f is concave function down (the former formula) or up (the latter one). We have equality when all xi are the same: x1 = x2 = · · · = xn . For a real convex function f , numbers xi in its domain, and positive weights ai , Jensen’s inequality can be stated as n n 1 ∑i=1 ai xi ai f (xi ). (F.25) 6 f ∑ ∑ni=1 ai ∑ni=1 ai i=1 9 Francesco

Faà di Bruno (1825 – 1888) discovered this formula in 1855.

APPENDIX

724

Finite Differences For finite difference operators ∆ f (n) = f (n + 1) − f (n), ∇ f (n) = f (n) − f (n − 1) and for the shift operator E f (n) = f (n + 1), we have the following equations. g · ∆( f ) − f · (∆(g)) f . (F.26) = ∆( f g) = f · ∆(g) + Eg · ∆( f ), ∆ g g · (Eg) E −1 g · ∇ f − E −1 f · ∇g f −1 . (F.27) = ∇( f g) = f · ∇g + E g · ∇ f , ∇ g g · (E −1 g) ∆( f × g) = f × ∆g+ ∆ f × g+ ∆ f × ∆g,

∆x ax = ax (a − x − 1),

∆x ax = ax (a + x − 1),

∆xm = m(x + 1)m−1 ,

∇( f × g) = f × ∇g+ ∇ f × g− ∇ f × ∇g . (F.28)

∇xm = m xm−1 ,

∆xm = mxm−1 ,

∆x ax = ax (a − 1),

(F.29)

∇xm = m (x − 1)m−1 .

(F.30)

The inverse difference operators: ∆−1 f (x) =

x−1

∑ f (i) +C,

i=s

x

∇−1 f (x) = ∑ f (i) +C,

(F.31)

i=s

where s < x − 1 is any index and C is any function whose first difference is zero (for instance, a function with period 1 like a constant or sin(2π x)). In particular, m ∆−1 x x =

1 xm+1 +C, m+1

m ∇−1 x x =

1 xm+1 +C. m+1

(F.32)

Summation by parts: n

n

∑ ∆ f (k) g(k) = f (n + 1) g(n + 1) − f (0) g(0) − ∑

k=0

f (k + 1)∆g(k).

k=0

∑

16k6n

∇ f (k) g(k) = f (n) g(n) − f (0) g(0) −

∑

16k6n

(F.33) f (k − 1)∇g(k).

In particular, n

∑ ∆ f (k) = f (n + 1) − f (0),

k=0

n

∑ ∇ f (k) = f (n) − f (0).

(F.34)

k=1

The kth difference operators: ∆k f = ∆(∆k−1 f ) and ∇k f = ∇(∇k−1 f ). Since ∆ = E − I and ∇ = I − E −1 , where I denotes the identity operator and E is the shift operator, we have ∆k f (n) = (E − I)k f (n) = ∑ki=0 (−1)k−i ki f (n + i), f (n) = ∑ni=0 ni ∆i f (0). (F.35) i n k n k i i k −1 k ∇ f (n) = (I − E ) f (n) = ∑i=0 (−1) i f (n − i), f (−n) = ∑i=0 (−1) i ∇ f (0).

Appendix G: The Gamma Function

725

Appendix G: The Gamma Function Definitions and Representations

Γ(z) =

Z ∞

−t z−1

e t

0

1 Γ(z) = ∏ z n>1

ℜz > 0;

dt,

1 1+ n

z z −1 1+ , n

z 6= 0, Z− . (G.1)

h z −z/n i 1 γz . e = ze ∏ 1 + Γ(z) n n>1 [Γ(z)]

±1

∓1

= z

with a0 = 1,

γ2 π2 2 ± z + ··· = 1 ∓ γz + 2 12

a± j =

ln Γ(z + 1) = −γ z +

∓γ a±( j−1) (−1) j ± j j

π2 2 z − ··· 12

(G.2)

∑ a± j z j±1,

j>0

z∈ / Z− ,

(G.3)

j−2

∑ (−1)k ζ ( j − k)a±k ,

j > 1. (G.4)

k=0

ζ ( j) (−z) j , j j>2

= −γ z + ∑

|z| < 1.

(G.5)

Relations Γ(1/2) =

√

π;

Γ(z + 1) = zΓ(z);

π ; sin π z

Γ(n + 1) = n!,

n ∈ N.

Γ(b + n + 1) = (b + n)n+1 . Γ(b) √ √ √ −2z (2n)! π π 1 1 = n (2n − 1)!! . Γ(z)Γ(z − /2) = 2 π 2 Γ(2z); Γ(n + /2) = 4n n! 2 ∆Γ(z) = (z − 1)Γ(z), ∇Γ(z) = (z − 2)Γ(z − 1). Γ(z)Γ(1 − z) =

z∈ / Z.

(G.6) (G.7) (G.8) (G.9)

Explicit representation for a complex argument, z = x + iy, x, y ∈ R: Γ(x + iy) = eiθ |Γ(x)| ∏ p j>0

| j + x|

y2 + ( j + x)2

,

y − tan[−1] k+x . For a purely imaginary argument: r Z y y π 1 ∞ −t iy iθ , . − tan[−1] θ = −γ y + ∑ e t dt = ie Γ(iy) = iy 0 y sinh π y k k>1 k

where θ = yΨ(y) + ∑k>0

(G.10)

y k+x

|Γ(z)| 6 |Γ(ℜz)|;

Γ(z) has simple poles at 0 and at Z− :

|Γ(iy)|2 =

Res Γ(z) =

z=−n

π . y sinh π y

(−1)n n!

(integer n > 0).

(G.11)

(G.12) (G.13)

APPENDIX

726

The beta function is defined as follows: Z 1

Z ∞

vx−1 dv. x+y 0 (1 + v) 0 n (−1)k (−1)n+1 n! n+1 = (−1) ∑ . B(n + 1, −z) = k z−k z(z − 1) · · · (z − n) k

B(x, y) =

Γ(x)Γ(y) )= Γ(x + y)

ux−1 (1 − u)y−1 du =

(G.14)

(G.15)

The incomplete gamma function has two versions—the upper, Γ(z, a), and the lower, γ (z, a):

Γ(z, a) =

Z ∞

−t z−1

e t

a

γ (z, a) =

dt,

Z a 0

e−t t z−1 dt,

ℜz > 0.

(G.16)

√ √ The scaled gamma function is defined as Γ∗ (z) = Γ(z) zez /( 2π zz ).

Asymptotics Stirling approximation (see also Eq. (I.13) on page 731): √ 1 1 n −n −3 +O n + 1+ Γ(n + 1) = n! = 2π n n e 12n 288n2 ( ) n n √ ∞ B2k . 2π n exp ∑ n! = 2k−1 e k=1 (2k − 1)2k n 1 en+1 1 = =p Γ(n + 1) n! 2π (n + 1)(n + 1)n

(G.17) (G.18)

1 1 −3 1− + +O n . (G.19) 12(n + 1) 288(n + 1)2

From Eq. (G.12):

|Γ(iy)|2 ∼

r−1/2 r 3 /6N 2 −π M/2

2π −π |y| e , |y|

y → ±∞.

(G.20)

r = |t| ≪ M = |N| → ∞.

(G.21)

|Γ(N + it)| = |Γ(N)| 1 + O (t/N)2 , |t| ≪ N, N → ∞. (a − b)(a + b − 1) Γ(n + a) a−b −2 =n +O n , a, b ≪ n. 1+ Γ(n + b) 2n

(G.22)

|Γ(t + iN)| = O M

e

,

(G.23)

Some inequalities for the factorial: (a)

√

n+ 21 −n

2π n

e

< n!
1 n>2

Ψ(z + 1) = −γ + ∑

|z| < 1,

(G.31)

where ζ (n) = ∑k>1 k−n . The function Ψ has the same poles as the Γ-function. All residues are −1. 1 1 ∇Ψ(z) = . (G.32) ∆Ψ(z) = , z z−1

The polygamma function, denoted by ψn , is the n-th logarithmic derivative of the gamma function:

ψn (z) =

d n+1 d n Γ′ (z) dn ln[Γ(z)] = = Ψ(z), dzn+1 dzn Γ(z) dzn

n = 0, 1, . . . ,

ψ0 (z) = Ψ(z). (G.33)

The polygamma function obeys the recurrence relation

ψn (z + 1) = ψn (z) + (−1)n n! z−n−1 . Similar to Eq. (G.32), we get ∆ψn (z) = (−1)n

n! zn+1

,

=⇒

∆−1

1 ψn−1 (z) = (−1)n−1 +C. n z (n − 1)!

(G.34)

APPENDIX

728

Appendix H: Random Variables and Distributions Name, Symbol PMF (Zero outside and Parameters the given range) Mean Variance PGF 2 a Uniform U [a, b] 1 a+b d −1 z − zd , a6i6b 2 12 d(1 − z) a 6 b, d ≡ b − a + 1 d Binomial B(n, p) n i n−i np npq (q + pz)n pq , 06i6n p ∈ (0, 1), q = 1 − p, n > 1 i Zipfian, Z(n) 1 n , 16i6n VZ — iHn Hn n>1 Geometric, G(p) 1 q pz pqi−1 , i > 1 p p2 1 − qz p ∈ (0, 1), q = 1 − p Shifted Geometric, G0 (p) q q p pqi , i > 0 2 p p p ∈ (0, 1), q = 1 − p 1 − qzr Negative Binomial r rq i − 1 r i−r pz p q , i>r>1 2 p p p ∈ (0, 1), q = 1 − p, r > 1 r−1 1 − qz r Shifted Negative Binomial rq rq r+i−1 i r p q p , i>0 2 p p 1 − qz p ∈ (0, 1), q = 1 − p, r > 1 i m N−m Hypergeometric, h(x, n; N, m) nm x n−x VH — N N m 6 N, n ∈ N n Poisson, Pois(λ ) λi e−λ , i > 0 λ λ eλ (z−1) λ >0 i! n+1 n def nm(N − m)(N − n) def n − . and V H = where V Z = Hn 2 Hn N 2 (N − 1) Discrete Random Variables Name, Symbol Density (Zero outside and Parameters the given range) Mean Variance LST \ CF Uniform U (a, b) 1 a + b (b − a)2 e−as − e−bs , a6x6b b−a 2 12 s(b − a) a 0 2 λ >0 λ λ λ + sr Gamma, Γ(r, λ ) λ r r−1 −λ x r r λ x e , x>0 Γ(r) λ λ2 λ +s r > 0, λ > 0 Beta, βa,b (1 − x)a−1 xb−1 b ab , x ∈ (0, 1) M(a, c, s) B(a, b) c c2 (c + 1) a, b > 0, c ≡ a + b 1 x−µ 2 Normal, N(µ , σ 2 ) 1 2 2 √ e− 2 ( σ ) µ σ2 eiµ s−s σ /2 σ >0 σ 2π Continuous random variables The range R, standard deviation σ , mean µ , median µ ∗ , and mode m0 (having frequency m)

Appendix H: Random Variables and Distributions

729

of a set of real numbers {x1 6 x2 6 . . . 6 xn } satisfy p ⌊n2 /4⌋ R R, (a) √ 6 σ 6 n 2n n (c) |m0 − µ ∗ | 6 σ , m

(b) |µ − µ ∗ | 6 σ , r n . (d) |m0 − µ | 6 σ m

(H.1) (H.2)

For any set of real numbers {x1 6 x2 6 . . . 6 xn }, the set of standard scores {zk = (xk − µ )/σ | 1 6 k 6 n} satisfies n √ 2n 6 ∑ |zk | 6 n . (H.3) k=1

The Bonferroni inequality (also called Boole’s inequality):

∑

Pr [∪16i6n Bi ] 6

Pr[Bi ] .

(H.4)

16i6n

Integrals over the normal distribution such as below are needed for summation asymptotics. The density φ (x) and the probability distribution function Φ(x) are defined as follows: Z x

Z

x 1 2 φ (t)dt = √ e−t /2 dt. −∞ 2π −∞ √ Z ∞ 1 π (2 j)! 2 1 . =√ x2 j e−ax dx = (2 j+1)/2 Γ j + j j! 2 a(4a) a −∞

2 1 φ (x) = √ e−x /2 ; 2π

(a)

Z ∞

2

−∞

e−ax dx =

π /a,

(b)

Z ∞

−∞

2 +bx

e−ax

dx =

p 2 π /aeb /4a ,

p b b2 /4a , Φ √ e dx = π /ae 0 2a r Z ∞ b b π b2 /4a 1 −ax2 +bx + e Φ √ , xe dx = 2a a a 0 2a r Z ∞ b π b2 /4a −ax2 +bx e . xe dx = a a −∞ 1 φ (x) −4 1 + 2 + O(x ) . 1 − Φ(x) = x x Z ∞

The error function 2 erf(x) = √ π

p

Φ(x) =

Z x 0

−t 2

e

−ax2 +bx

dt = 2

Z x√2 0

1 φ (t) dt = √ γ π

1 2 ,z 2

2z =√ F π

and the complementary error function 2 erfc(x) = 1 − erf(x) = √ π

Z ∞ x

2

e−t dt = 2

Z ∞

√ x 2

φ (t) dt.

(H.5) (H.6)

(H.7)

(H.8)

(H.9) (H.10) (H.11)

1 2 1 2

! 2 −z

(H.12)

(H.13)

APPENDIX

730

Appendix I: Combinatorics of Permutations n 1 2 3 4 5 6 7 8 9

k

1 2 3 4 5 6 7 8 9 1 0 1 3 0 1 8 6 0 1 45 20 10 0 1 264 135 40 15 0 1 1855 924 315 70 21 0 1 14832 7420 2464 630 112 28 0 1 133497 66744 22260 5544 1134 168 36 0 1 Table 730: Coincidence Numbers dn,k .

Let (i1 , i2 , . . . , in ) be a permutation of the set [1..n] = {1, 2, . . . , n}. The point ik is called fixed point of this permutation if ik = k. A permutation is called a derangement if it does not have a fixed point, that is, if ik 6= k for all k = 1, 2, . . . , n. The number of permutations of the set [1..n] with k fixed points is usually referred to as the coincidence number and denoted by dn,k . For every positive integer n > 1, the number of derangements is conventionally denoted by Dn ≡ dn,0 . n n def dn,k = dn−k,0 = Dn−k . (I.1) k k n n! n−k (−1)i n−k n = Dn−k . (I.2) dn,k = n dn−1,k + (−1) , dn,k = ∑ k k! i=0 i! k dn,k = dn−1,k−1 + (n − 1)[dn−1,k + dn−2,k − dn−2,k−1 ] . Dn = (n − 1) Dn−1 + (n − 1) Dn−2 ,

Dn = nDn−1 + (−1)n .

(I.3) (I.4)

n

(−1)k , D0 = 1, D1 = 0 . k=0 k! 1 1 n! +m , m∈ , , Dn = e 3 2   n! , if n is odd , e n! + 1 = Dn =  n! e if n is even . e + 1, Dn = ∆n x! x=0 . n n n n n 2 n D = n!, k D = (n − 1) n!, k ∑ k k ∑ k k ∑ k Dk = (n2 − 2n + 2) n! k=0 k=0 k=0 Dn = n!

∑

(I.5) (I.6)

(I.7) (I.8) (I.9)

Appendix I: Combinatorics of Permutations

731

The permutation (i1 , i2 , . . . , in ) of the set [1..n] is said to be of the rank m if im is the first fixed point: im = m and ik 6= k for k < m. A derangement is said to be of the rank n + 1. The symbol Rn,k is called the rank number and it counts the number of permutations of [1..n] having rank k. The recurrences: Rn,m = Rn,m−1 − Rn−1,m−1 , Rn,m = (n − m)Rn−1,m + (m − 1)Rn−1,m−1 ,

2 6 m 6 n; n > 2;

(I.10)

m = 2, 3, . . . , n; n = 2, 3, . . . ,

(I.11)

subject to the boundary conditions Rn,1 = (n − 1)!, Rn,n = Dn−1 , Rn,n+1 = Dn , have exact solution: m−1 k m−1 (n − k − 1)!. (I.12) Rn,m = ∑ (−1) k k=0 n 1 2 3 4 5 6 7 8

k

1 2 3 4 5 6 7 8 9 1 0 1 0 1 2 1 1 2 6 4 3 2 9 24 18 14 11 9 44 120 96 78 64 53 44 265 720 600 504 426 362 309 265 1854 5040 4320 3720 3216 2790 2428 2119 1854 14833 Table 731: Rank Numbers Rn,k .

√ √ The scaled gamma function is defined as Γ∗ (z) = Γ(z) zez /( 2π zz ). In the cut complex plane | arg(z)| < π , we have

Γ∗ (z) ∼

∑ (−1)k δk z−k ; k>0

1 Γ∗ (z)

∼

∑ δk z−k ,

(I.13)

k>0

where Γ(k + 1/ ) (2k)! δk = (−2)k √ 2 d2k = (−1)k k d2k , 2 k! π with d0 = 1. k δk k 1 1 − 12 2 139 3 4 51,840 163,879 5 − 209,018,880 6 53,470,3531 7 8 902,961,561,600 432,261,921,612,371 9 − 514,904,800,886,784,000 10

" # n + 1 dn−1 n−1 d j dn− j , dn = −∑ n+2 n j=1 j + 1

δk 1 288 571 − 2,488,320 5,246,819 75,246,796,800 4,483,131,259 − 86,684,309,913,600 6,232,523,202,521,089 86,504,006,548,979,712,000

The Stirling cofor δk efficients 1 6 k 6 10, δ0 = 1. Note that these numbers are not the same as the coefficients in Stirling formula for n!.

(I.14)

APPENDIX

732

Appendix J: Continued Fractions Rational function approximation Rm,n (x) =

m

Pm (x) , Qn (x)

Pm (x) =

∑

n

pk xk ,

Qn (x) =

k=0

∑ qk xk , k=0

is called a Padé approximation if the following requirement holds f (x)Qn (x) − Pm (x) = O xn+m+1 as x → 0.

(J.1)

For example, the Padé rational approximation, R6,6 , for the sine is (for |x| 6 π /6) # " 34911 4 479249 325523 2 x + 7613320 x − 11511339840 x6 1 − 2283996 . sin x ≈ x 1261 2623 18381 2 x + 4567992 x4 + 1644477120 x6 1 + 761332 A continued fraction is often defined as the limit of “an expression of the form:” an

xn+1 = xn +

.

an−1

xn−1 +

an−2 ..

xn−2 + x3 +

x2 +

. a2

a1 x0 In view of the unwieldiness of this notation, various authors have proposed other ways of writing continued fraction; it has become customary to write in a typographically convenient form as follows: a1 a1 a2 a3 a1 ak ∞ ak ∞ = b0 + or b0 ; ; . · · · or b0 ; b0 + b1 + b2 + x3 + bk k=1 b1 bk 2 a2 b1 + a3 b2 + .. . (J.2) x1 +

A continued fraction (J.2) is called proper if b0 = 0, otherwise we call such fraction improper. The proper continued fraction is usually denoted by K∞ k=1 (ak /bk ). The fraction ak /bk is called the kth partial quotient of the continued fraction (J.2). We call ak and bk kth partial numerator and denominator respectively, or simply coefficients of the continued fraction. The terminating continued fraction b0 +

an An a1 a2 a3 ··· ≡ = Kn b1 + b2 + b3 + + bn Bn

(J.3)

Appendix J: Continued Fractions

733

is called the nth convergent (or approximant) of the continued fraction (J.2), and An and Bn are called nth canonical numerator and nth canonical denominator, respectively. Hence, the continued fraction is the limit limn→∞ An /Bn (if it exists). The coefficients in Eq. (J.3) satisfy the following recurrence An = bn An−1 + an An−2 ,

Bn = bn Bn−1 + an Bn−2 ,

n = 1, 2, 3, . . . ,

(J.4)

subject to the initial conditions (suggested by L. Euler) A−1 = 1, A0 given,

B−1 = 0, and B0 = 1.

Continued fraction expansions for standard functions: x x 1 x 2 x 3 ...; 1−1+1−1+1−1+1− 1x 2x 3x x ... 1+ 1−x + 2−x + 3−x + 4−x + 1 x 1x 1x 2x 2x 3x 3x 4x 4x ..., 1−1+ 2 − 3 + 4 − 5 + 6 − 7 + 8 − 9 + 30240 + 15120x + 3360x2 + 420x3 + 30x4 + x5 , |x| < 1, 30240 − 15120x + 3360x2 − 420x3 + 30x4 − x5 1 z z z z z z z ... 1−1+2−3+2−5+2−7+ 1 z 1z 1z 2z 2z 3z 3z 4z 4z ... 1−1+ 2 − 3 + 4 − 5 + 6 − 7 + 8 − 9 + z 1z 2z 3z 1 ..., z ∈ C. 1 − 1+z − 2+z − 3+z − 4+z

ex = 1 + = = ≈ ez = = =

Z ∞ x

ln(1 + x) = = = =

ln

1+z 1−z

2

−t 2

e

e−x 1 c 2c 3c · + ··· , dt = 2x 1 + 1 + 1 + 1

n2 x ; 1 n + 1 − nx

∞

c=

x ∈ C, Padé approximation.

1 . 2x2

1 2x x2 = ; 1 − 2 + x + 2(2n + 1) n=1

∞

n=1

x 12 x 12 x 22 x 22 x 32 x 32 x 42 x 42 x ... 1+ 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + x 1x 1x 2x 2x 3x 3x 4x 4x ... 1+ 2 + 3 + 2 + 5 + 2 + 7 + 2 + 9 + x 1 x 1/2 z 1 x 2/3 x .... 1+x − 1 + 1+x − 1 + 1+x − 1 + 1+x − 1 + 1+x −

= 2 arctanh z ≡ 2 tanh[−1] z =

2z 12 z2 22 z2 32 z2 42 z2 .... 1 − 3 − 5 − 7 − 9 −

(J.5)

APPENDIX

734

ln x ≈ tan z =

24(x2 − 1)(x2 + 10x + 1)

(x2 + 22x + 1)(5x2 + 14x + 5)

,

z z2 z2 z2 z2 ..., 1− 3 − 5 − 7 − 9 −

(1 + z)α =2 F1 (−α , 1; 1; −z) =

Padé approximation. z ∈ C.

1 α z (1 + α )z (1 − α )z (2 + α )z ..., 1− 1 + 2 + 3 + 2 +

when | arg(1 + z)| < π ,

=

1(1 + α )z(1 + z) 2(2 + α )z(1 + z) αz 1 ...; 1 − 1 + (a + α )z − 2 + (3 + α )z − 3 + (5 + α )z −

i x−i x √ = ln 1 + 1 + x2 2 x + i n2 4 1 1 ... ... x + 3x + 5x + + (2n + 1)x + x 12 x 22 x 32 x 42 x 52 x ... 1 + 3 + 5 + 7 + 9 + 11 + x3 32 x2 22 x2 52 x2 42 x2 72 x2 62 x2 ... x− 3 + 5 + 7 + 9 + 11 + 13 + 15 + x2 9x2 (2n − 1)2 x2 x . . . .... 1 + 3 − x2 + 5 − 3x2 + + 2n + 1 − (2n − 1)x2 + x2 x x2 x2 ... ... 1 + 3 + 5 + + 2n + 1 + 945x + 105x3 + x5 Padé approximation. 945 + 420x2 + 15x4 −1/4 −1/2 −1/4 −1/4 ... . . . , Chebyshev polynomials. t + t + t + + t + 1 (1 + ν )x (1 − ν )x (n + ν )x (n − ν )x νx ... ... 1 − 1+x + 2 + 3 + + 2 + 2n + 1 + x−1 x−1 x−1 .... 1+ 2 + 2 + 2 + az az az ..., a ∈ C \ {0}. 1 + 1 + 1 + x2 x2 1 . . . . The Motzkin number generating function. 1−x − 1−x − 1−x −

arctan x ≡ tan[−1] x = 2 arctan = = = = tanh x = ≈ Tn (t) = (1 + x)ν √ √

=

x =

1 + 4az − 1 2

=

M(x) = √

ℜz > −1/2.

2 = 1+

1 1 1 ... 2+2+2+

Appendix J: Continued Fractions = √

3 = e = = = =

ln 2 =

π = π 4 4 π π2 12 √ 5−1 2 1 √ e−1

=

= = = =

735

1 2 1 2 2 2 2 .... 1 − 8 − 1 − 16 − 1 − 16 − 1 − √ 2 2 2 2 √ = 1+ .... 1+ 3−1 = 1+ 2 +2+2+ 1+ 3 1 1 1 1 1 1 1 1 ... 1−1+2−3+2−5+2−7+ 1 1 2 1 1 ... 1+ 1 + 6 + 10 + 14 + 18 + 2 3 4 5 ... 2+ 2+3+4+5+ 1 1 1 2 1 1 .... 1 − 3 + 6 + 10 + 14 + 18 + 1 1/2 1/(2 · 3) 2/(2 · 3) 2/(2 · 5) 3/(2 · 5) .... 1+ 1 + 1 + 1 + 1 + 1 + 4 12 22 32 42 .... 1+ 3 + 5 + 7 + 9 + (2n − 1)2 1 1 32 52 72 ... .... 1+2+ 2 + 2 + 2 + + 2 + 1 32 (2n − 1)2 ... .... 1+ 2+ 2 + + 2 + 1 14 24 34 1 .... ζ (2) = 2 1+ 3 + 5 + 7 + 1 1 1 1− .... 3−3−3− 2 4 6 1+ .... 3+5+7+

Regular continued fractions have an especially convenient format: K = α0 +

1 1 1 . . . = [α0 ; α1 , α2 , α3 , . . .]. α1 + α2 + α3 +

Regular continued fractions for standard constants:

π = 3.14159265358979 . . . = [3; 7, 15, 1, 292, 1, 1, 1, 2, 1, 3, 1, 14, 2, 1, 1, 2, 2, 2, . . .]; π 2 = 9.86960440108934 . . . = [9; 1, 6, 1, 2, 47, 1, 8, 1, 1, 2, 7, . . .]; π2 = 1.64493406684823 . . . = [1; 1, 1, 1, 4, 2, 4, 7, 1, 4, 2 . . .]; 6 π 4

= 0.785398163397448 . . . = [0; 1, 3, 1, 1, 1, 15, 2, 72, 1 . . .]; = 1.27323954473516 . . . = [1; 3, 1, 1, 1, 15, 2, 72, 1, . . .];

APPENDIX

736

e = 2.71828182845905 . . . = [2; 1, 2, 1, 1, 4, 1, 1, 6, 1, 1, 8, 1, 1, 10, . . .]; e2 = 7.38905609893065 . . . = [7; 2, 1, 1, 3, 18, 5, 1, 1, 6, . . .]; ln 2 = 0.693147180559945 . . . = [0; 1, 2, 3, 1, 6, 3, 1, 1, 2, 1, 1, 1, 1, 2, . . .];

γ = 0.5772156649015328 . . . , the Euler constant, = [0; 1, 1, 2, 1, 2, 1, 4, 3, 13, 5, 1, 1, 8, 1, 2, 4, 1, 1, 40]; ∞ (−1)k = 0.915965594177 . . . , the Catalan constant, G = ∑ 2 k=0 (2k + 1) 1 1 1 π2 π2 1 − − = −∑ = 2 = ∑ ∑ 2 2 2 (4k + 3) 8 8 k>0 (4k + 3)2 k>0 (4k + 1) k>0 (4k + 1) Z 1 arctan x

Z 1 ln x

Z

Z

1 ∞ x 1 π /2 x dx = dx = dx 2 x 2 0 cosh x 2 0 sin x 0 1+x 0 = [0; 1, 10, 1, 8, 1, 88, 4, 1, 1, 7, 22, 1, 2, 3, 26, 1, 11, 1, 10, 1, 9, 3, . . .]; log2 k ∞ 1 , the Khinchin constant, K = 2.685452001 . . . = ∏ 1 + k(k + 2) k=1 =

dx = −

= [2; 1, 2, 5, 1, 1, 2, 1, 3, 10, 2, . . .];

C = 0.697774658 . . . = [0; 1, 2, 3, 4, 5, 6, 7, 8, 9, . . .]; 1 1 . = [1; 2, 3, 4, . . .], I0 (2)/I1 (2) = ∑ ∑ 2 n>0 n! (n + 1)! n>0 (n!)

where Ik (x) is the modified Bessel function of the order k.

We say that a regular continued fraction K = [α0 ; α1 , α2 , α3 , . . .] is (ultimately) periodic if αn+t = αn for all n > N. In this case we write

K = [α0 ; α1 , α2 , α3 , . . .] = [α0 ; α1 , α2 , . . . , αN , αN+1 , αN+2 , . . . , αN+t ] , and say that the period of the continued fraction has length t. An overbar is used to indicate the block of integers that repeat constantly from that point on. Theorem J.85 (Lagrange) The partial quotients in the continued fraction expansion for real number K are ultimately periodic if and only if K is an irrational number that satisfies a quadratic equation with integer coefficients. Periodic regular continued fractions: √ 2 = 1.41421356237310 . . . , √

√

√

Pythagoras’ constant,

= [1; 2, 2, 2, 2, . . .]; 3 = 1.73205080756888 . . . = [1; 1, 2, 1, 2, 1, 2, 1, 2, . . .]; 5 = 2.236067977 . . . = [2; 4, 4, 4, 4, . . .]; 6 = 2.449489743 . . . = [2; 2, 4 , 2, 4, 2, 4, 2, . . .]; |{z} period

Appendix J: Continued Fractions √ √

7 = 2.645751311 . . . = [2; 1, 1, 1, 4, 1, 1, 1, 4, . . .]; | {z } period

8 √ 10 √ 11 √ 12 √ 13 √ 14 √ 15 √ 17

√ 5+1 2

p

√ 5−1 p m2 + 1 p m2 + 2 p m2 + m p m2 + 2m

= 2.828427125 . . . = [2; 1, 4, 1, 4, 1, 4, . . .] ; = 3.16227766 . . . = [3; 6, 6, 6, . . .] ;

= 3.316624790 . . . = [3; 3, 6, 3, 6, 3, 6, 3, . . .]; = 3.464101616 . . . = [3; 2, 6, 2, 6, 2, 6, 2, 6, . . .]; = 3.605551275 . . . = [3; 1, 1, 1, 1, 6, 1, 1, 1, 1, 6, 1, . . .]; = 3.741657387 . . . = [3; 1, 2, 1, 6, 1, 2, 1, 6, . . .]; = 3.872983346 . . . = [3; 1, 6, 1, 6, 1, 6, . . .]; = 4.123105626 . . . = [4; 8, 8, 8, 8, 8, . . .]; = 1.61803398874990 . . . , the golden ratio; r q √ = [1; 1, 1, 1, 1, . . .] = 1 + 1 + 1 + · · ·; = 1.23606797749979 . . . = [1; 4, 4, 4, 4, . . .]; 1 1 1 1 ... ...; = m+ 2m + 2m + 2m + 2m + = [m; m, 2m, . . .] ; = [m; 2, 2m, . . .] ; = [m; 1, 2m, . . .] ;

m2 + 2m − 1 = [m; 1, m − 1, 1, 2m, . . .] .

737

APPENDIX

738

Appendix K: Occupancy Enumeration

Surjections

Weak compositions (compositions without 0)

Set partitions

Integer partitions

parameters n distinct objects k distinct boxes n distinct objects any number of distinct boxes n identical objects k distinct boxes n identical objects any number of distinct boxes n distinct objects k identical boxes n distinct objects any number of identical boxes n identical objects k identical boxes n identical objects any number of identical boxes

formula k! nk ∑ki=1

n i

i!

n−1 k−1

2n−1 n k

Bn

pk (n) p(n)

Table 738: Enumeration formulas if no boxes are empty. Here

n k

is the Stirling number of the second kind, Bn is the Bell number, Eq. (C.4).

The Diophantine equation is a1 + a2 + · · · + ak = n,

(K.1)

where ai and n are integers. Nonnegative solution (a1 , a2 , . . . , ak ) of Eq. (K.1) is called the composition of n; when all ai > 0, the solution is called the weak composition of n. The number of distributions of n distinguishable balls into k distinct boxes of unlimited capacity, or the set of k-permutations of n with (unrestricted) repetitions is kn

with the (ordinary) enumerator

1 . 1 − kz

(K.2)

The number of distributions of n distinguishable balls into k distinct boxes, ni objects in the ith box is n . n1 , n1 , . . . , nk The number of distributions of n distinguishable balls into k distinct boxes and boxes may not be empty is k! nk .

Appendix K: Occupancy Enumeration

739

The number of distributions of n distinguishable balls into k distinct boxes and boxes may be empty is n+k−1 . n

The number of distributions of n identical balls into k distinguishable urns, each with capacity 6 1, or the number of solutions of Eq. (K.1) with ai = 0 or 1, i = 1, 2, . . . , k, is k with the enumerator (1 + z)k . (K.3) n The number of distributions of n identical balls into k distinguishable urns of unlimited capacity, or the number of nonnegative integer solutions of Eq. (K.1), or the number of ncombinations of k with repetitions is n+k−1 with the enumerator (1 − z)−k . (K.4) n The number of different monomials (xr11 xr22 . . . xrkk ) of degree n = r1 + r2 + · · · + rk in the vari ables x1 , x2 , . . . , xk is n+k−1 . n Objects identical (n), boxes indistinguishable (k), boxes may not be empty – n−1 k−1 . Objects distinct (n), boxes identical (k), boxes may be empty – ∑ki=1 ni . Objects distinct (n), boxes identical (k), boxes may not be empty – nk . Functions Compositions Set partitions Integer partitions

parameters n distinct objects k distinct boxes n identical objects k distinct boxes n distinct objects k identical boxes n identical objects k identical boxes

formula kn n+k−1 k−1

∑ki=1

= n i

n+k−1 n

∑ki=1 pi (n)

Table 739: Enumeration formulas if empty boxes are allowed. An asymptotic expression for p(n), the number of partitions of n, is given by the Rademacher formula (1937): √ 1 (K.5) p(n) ∼ √ eπ 2n/3 as n → ∞. 4n 3

APPENDIX

740

Appendix L: Generating Functions There is one-to-one correspondence between sequences of real (or complex) numbers and power series (called the generating function for the sequence or z-transformation). For a given sequence {an }n>0 , we associate two generating functions: ordinary, a(z), and exponential, a(z): ˆ zn and a(z) ˆ = ∑ an . a(z) = ∑ an zn n! n>0 n>0 These functions are related via the Laplace-Borel transformation: a(z) =

Z ∞ 0

1 a(zt) ˆ e dt = z −t

Z ∞ 0

a(t) ˆ e−t/z dt.

The inverse operation, called the coefficient operator, extracts the general term from the power series: [zn ]a(z) = an and n![zn ]a(z) ˆ = an . The basic rules related to generating functions and their coefficients are presented on page 288. Some others can be added: [zn ]

a(z) = ∑ [zk ]a(z) = a0 + a1 + · · · + an ; 1 − z 06k6n

(L.1)

Euler transformation: z n n k 1 n a . ak = ∑ [z ]a(z) = [z ] ∑ 1−z 1−z 06k6n k 06k6n k For an arbitrary infinitely differentiable function f , we have n n k dk f d n z f (z) = ∑ . z dz dzk k=1 k

(L.2)

(L.3)

Theorem L.86 (Abel) If the generating function F(z) = ∑n>0 an zn converges for |z| < 1, then lim (1 − z) F(z) = lim an ,

z→1−0

n→∞

Theorem L.87 (Gauss) a, b Γ(a) Γ(c − a − b) , 1 ≡ 2 F1 (a, b; c; 1) = F Γ(c − a) Γ(c − b) c (1 − z)c−a−b 2 F1 (c − a, c − b; c; z) = 2 F1 (a, b; c; z),

(an > 0) .

ℜ(c − a − b) > 0.

(L.4)

ℜ(c − a − b) > 0.

(L.5)

Appendix L: Generating Functions

741

k+1 a converges, where {a } is a positive, deTheorem L.88 (Euler) If the series ∑∞ k k 1 (−1) creasing sequence with limit zero, then ∞

∑(−1)k+1 ak = 1

∞ a1 1 ∞ + ∑(−1)k+1 ∆ak = ∑ ∆k a1 /2k+1 . 2 2 1 0

Generating functions: ex = 1 + x +

xn x2 x3 + + ··· = ∑ . 2! 3! n>0 n!

(L.6)

sin x =

∞ x3 x5 x7 x2k+1 1 ix 1 −ix e − e = x − + − + · · · = ∑ (−1)k . 2i 2i 3! 5! 7! (2k + 1)! k=0

(L.7)

cos x =

∞ x2 x4 x6 x2k 1 ix 1 −ix e + e = 1 − + − + · · · = ∑ (−1)k . 2 2 2! 4! 6! (2k)! k=0

(L.8)

∞ x3 x5 x7 x2k+1 1 x 1 −x e − e = x+ + + + ··· = ∑ . 2 2 3! 5! 7! k=0 (2k + 1)!

(L.9)

sinh x =

∞ x2 x4 x6 x2k 1 x 1 −x e + e = 1+ + + + ··· = ∑ . 2 2 2! 4! 6! k=0 (2k)! ∞ 2n+1 (−1)n+k+1 22n−k+1 k! 2n + 1 2n+1 2i −i = ∑ ∑ x tan x = 2ix e +1 (2n + 1)! k n=0 k=1

cosh x =

= x+

(L.10) (L.11)

4n (4n − 1) x3 2x5 17x7 62x9 + + + + · · · = ∑ (−1)n−1 B2n x2n−1 . 3 15 315 2835 (2n)! n>1

1 2 17 7 62 9 xn = x − x3 + x5 − x + x − ··· . (L.12) n! 3 15 315 2835 n>1 −α , b α α (L.13) x = (1 + x) = F ∑ k xk , Newton’s identity. b k r − tk k r − tk r k xr+1 =∑ z = xr , where z = xt+1 − xt , (L.14) z, ∑ (t + 1)x − t k>0 r − tk k k k>0 tanh(x) =

∑ En(−1)

and the solution is chosen be regular in z ∈ {|z| < |t t (t + 1)−(t+1) |}. This is also the region of convergence. n−k k n−k k zn+1 n+1 z(n − r) − n − 1 =∑ = ∑k z. (L.15) z, z r (z − 1)r+1 k>0 (z − 1)r+2 r k>1 k−n k z2n =∑ z. (L.16) (1 − z)n+1 k>0 n k k zn =∑ z. (L.17) n+1 (1 − z) k>0 n

APPENDIX

742 k+m k 1 =∑ z. (1 − z)m+1 k>0 k

(L.18)

Its finite version does not have a closed form, but the following may be preferred when m ≪ n: i+n i n n+1 1 − zn+1 ∑m k+m k 1 dm i=0 i (i − z) m 1−z , n ∈ N. (L.19) ∑ k z = m! dzm z 1 − z = (1 − z)m+1 k=0 m−r−1 r + k k−(m−r) k+m k zr−m z , m, r ∈ N; (L.20) − [r − m > 0] z = ∑ ∑ r r (1 − z)r+1 k=0 k>0 where [r − m > 0] is the Iverson bracket, which is 1 if r > m and 0 otherwise. k ⌊ p+q ⌋ k − ip u(k; p, q) = ∑ , iq i=0

(1 − z)q−1 = ∑ u(k; p, q)zk , (1 − z)q − z p+q k>0

xk 1 =∑ zk . 1 − x − xz k>0 (1 − x)k+1 kα + β − 1 zk yβ − 1 , =∑ β k k−1 k>1 kα zk , y = 1+ ∑ k>1 k − 1 k

p

1 − x2 + x arcsin x

= 2 F1

1 1 1 − , − ; ; x2 2 2 2

(L.22) y = 1 + zyα .

where where

y = 1 + zyα .

= (1 − x2 )3/2

x2k 1k 1k

∑ (1/2)k k! .

(L.23) (L.24)

(L.25)

k>0

! a, a + 21 x . 1 2 " √ √ n+1 n+1 # 1 − 1 + 4z 1 + 1 + 4z n− j j 1 √ − =∑ z. 2 2 j 1 + 4z j>0 √ √ n n 1 − 1 + 4x n−k k n 1 + 1 + 4x + =∑ x. 2 2 k k n−k √ 2n + k n (1 − 1 − 4z)k √ =∑ z , k = 0, 1, . . . . n 2k zk 1 − 4z n>0 √ √ 1 1 (1 + x)−2a + (1 − x)−2a = F 2 2

The logarithmic functions: 1 1 ln = ∑ Hn zn , 1−z 1−z n>1

(L.21)

(L.26)

(L.27)

(L.28)

(L.29)

1 1 ln = ∑ (n + 1)(Hn+1 − 1) zn , (L.30) (1 − z)2 1−z n>1


743

where Hn = ∑nk=1 k−1 is the n-th harmonic number. Also Hn ∑ n + p zn+p = − n>1

Z z p−1 t ln(1 − t) 0

1−t

1 n z . n n>1 nq

ln[(q − z)−1 ] = − ln q + ∑

dt,

(L.31)

The EGFs for Bn (x), the Bernoulli polynomials, and for Bn = Bn (0), the Bernoulli numbers:

ϕˆ B (x, z) =

z exz zn B (x) = , n ∑ ez − 1 n>0 n!

The reciprocal:

ϕˆ B (z) =

z zn B = . n ∑ ez − 1 n>0 n!

ez − 1 1 zn 1 = =∑ . ϕˆ B (z) z n>0 n + 1 n!

(L.32)

(L.33)

Other EGFs involving the Bernoulli numbers: zn z n (1 − 2 )B = , n ∑ ez + 1 n>0 n!

zn z z coth = 1 + ∑ Bn , 2 2 n! n>2

B2k (2z)2k . ∑ k>0 2k (2k)! √ 2n n 1 1 − 1 − 4z 1 2n =∑ z . The OGF for n+1 n , the Catalan numbers: 2z n>0 n + 1 n √ 2n + 1 n 4z + 1 − 4z − 1 =∑ z . The central binomial coefficients: 2z(1 − 4z) n n>0 sinh z ln z

=

(L.34)

(L.35) (L.36) (L.37)

The EGF for E2n , the Euler numbers: sec z =

∑ (−1)n

n>0

E2n 2n z , (2n)!

z2n 2 1 E = = ∑ 2n (2n)! . ez + e−z cosh z n>0

(L.38)

zn 2 ezx E (x) = . n ∑ ez + 1 n>0 n!

(L.39)

The EGF for En (x), the Euler polynomials: The EGF for tn , the tangent numbers:

tan z =

z2k+1

∑ t2k+1 (2k + 1)! .

(L.40)

k>0

z = ∑ Fn zn . 1 − z − z2 n>0 zn 1 ˆ √ eφ z − eφ z = ∑ Fn , n! 5 n>0

The OGF for Fn , the Fibonacci numbers: The EGF for Fn , the Fibonacci numbers:

(L.41) (L.42)

where φ and φˆ are defined in Eq. (D.4). Another generating function: n+1 1 = ∑ ∑ Fk2 zn . 1 − 2z − 2z2 + z3 n>0 k=0

(L.43)

APPENDIX

744

The EGF for Dn , the derangement numbers: zn e−z = ∑ Dn , 1 − z n>0 n!

n

Dn = n!

(−1)k . k=0 k!

∑

(L.44)

The EGF for Bn , the Bell numbers: z ˆ B(z) = ee −1 =

zn B ∑ n n! , n>0

1 Bn = e

n−1 kn ∑ = ∑ i Bi. i k>0 k! n

(L.45)

The Chebyshev polynomials of the first and second kinds, respectively: n ⌊n/2⌋ (−1)k n − k −n, n 1 − x = Tn (x) = F ∑ n − k k (2x)n−2k , 1 2 2 2 k=0 ⌊n/2⌋ n n n−2k 2 = ∑ x (x − 1)k ; Un (x) = ∑ xk Tn−k (x). k=0 k=0 2k

n > 1.

∞ 1 − t2 Tn (x)t n , = T (x) + 2 0 ∑ 1 − 2xt + t 2 n=1

1 = ∑ Un (x)t n , 1 − 2xt + t 2 n>0 The EGF for |sn,m | =

∞ 1 − tx Tn (x)t n . = ∑ 1 − 2xt + t 2 n=1 ⌊n/2⌋ k n−k Un (x) = ∑ (−1) (2x)n−2k . k k=0

n m , the signless Stirling numbers of first kind: n n z [− ln(1 − z)]m = ∑ . m! n>m m n!

The OGF and EGF for

n

, the Stirling numbers of second kind: n n n n z (ez − 1)m zm =∑ z and = ∑ . m m! ∏ j=1 (1 − jz) n>0 m n>m m n!

(L.46) (L.47) (L.48) (L.49)

(L.50)

m

The Motzkin number generating function: √ 1 − z − 1 − 2z − 3z2 = ∑ Mn zn . M(z) = 2z2 n>0

(L.51)

(L.52)

Power functions: # n + 1 ∑ knzk = (1 − z)−n−1 ∑ ∑ (−1)m+1 m − 1 (r − m + 1)n zr . r=1 m=1 k=1 n n 1 n k d = ∑ k! z (1 − z)−k−1 . = z dz 1−z k k=1 ∞

n

"

r

(L.53)


745

n n k zk z. = ez ∑ k! k=1 k

(L.54)

∞

∑ kn k=1

In particular,

zn

z

∑ n n! = z ez ;

∑ n zn = (1 − z)2 ; 1+z

∑ n2 zn = z (1 − z)3 ;

n>0

∑ n3 zn = z

n>0 ∞

∑ k4zk = k=1

1 + 4z + z2 ; (1 − z)4

z(1 + 11z + 11z2 + z3 ) ; (1 − z)5

(L.55)

n>0

n>0

zn

∑ n2 n! = (z + z2) ez ;

(L.56)

n>0

zn

∑ n3 n! = z (1 + 3z + z2) ez ;

(L.57)

n>0 ∞

∑ k5zk = k=1

z(1 + 26z + 66z2 + 26z3 + z4 ) . (1 − z)6

(L.58)

Legendre polynomials: (1 − 2xt + t 2 )−1/2 =

∞

∑ Pn(x)t n ,

(L.59)

n=0

n (n + k)! x−1 k −n, n + 1 1 − x , =∑ Pn (x) = F 2 2 2 1 k=0 (k!) (n − k)! n + 1, n + 1 (1 − t)n+1 ∞ m 2 m 1+t n+1 t . = (1 − t) F Pn ∑ t = 1−t tn 1 m=n n

2 n 2n − k n x−1 k n x − 1 k x + 1 n−k =∑ , Pn (x) = ∑ n k 2 2 2 k=0 k=0 k

(L.60) (L.61)

n

2 n xk . =∑ (1 − x) Pn k k xn 1, 1 xn = − x = − ln(1 − x) = −x F ∑ , ∑ 2 n>1 n n>1 n ! 1 , 1 x2k+1 1+x = 2x F 2 3 x2 = 2 ∑ , ln 1−x k>0 2k + 1 2 ! 1 1 ∞ (2n)! x2n+1 2, 2 2 = ∑ 2n , arcsin x = x F 3 x 2 n=0 2 (n!) (2n + 1) 2 ! 1 2n+1 n x 2, 1 2 (−1) = x , arctan x = x F ∑ 3 2n + 1 n>0 2 2a, a + 1 1+x =F x ≡ 2 F1 (2a, a + 1; a; x), (1 − x)2a+1 a n

1+x 1−x

(L.62)

(L.63) (L.64) (L.65) (L.66) (L.67) (L.68)

APPENDIX

746

√ 1−2a a − 21 , 1+ 1−x =F 2 2a

a 1 x ≡ 2 F1 a − 2 , a; 2a; x ,

(L.69)

where F is the hypergeometric function (see §2.6).

sin nx n z . n>0 n! −1 2n z arcsin z z2 = + ∑ n (2z)2n . 1 − z2 (1 − z2 )3/2 n>1 ez cos x sin(x sin x) =

∑

n m(n − j) + i n (1 + x)i n mn+i =∑∑ = ∑ x (1 + x) x . 1 − x(1 + x)m n>0 j n>0 j=0 x−1 βk x ∑ k zk = (1 − β )x + β , z = xβ . k>0 x−1 3k x ∑ k zk = 3 − 2x , z = x3 . k>0 2k 2k k 1 ∑ k kzk = 2z(1 − 4z)−3/2 . ∑ k z = √1 − 4z , k>0 k>0 √ √ k k 1 − 1 − 4z 1 − 1 − 4z 2k z 2k z √ , ∑ = ln , |z| < 41 . ∑ k k+1 = 2z k k z(1 + 1 − 4z) k>1 k>0 √ 2n z 1 2n n 1 − 1 − 4z z 2 =∑ =∑ z . 2 ln 2 arcsin , 2 2n 2 2z n>1 n n n>1 n n 2n n 1 2z arcsin z 1 2n −1 1 √ (2z)2n . arcsin(2z) = ∑ z , =∑ 2 2z 2n + 1 n n n 1 − z n>1 n>1 2n 2z(1 − 4z)−3/2 = ∑ n zn . n n>1 n n n+k 1 k √ where bn = ∑ = ∑ bk z , . k 1 − 6z + z2 k>0 k=0 k √ √ 2n n 1 1 − 1 − 4z 1 − 4z 1 + + z(ln 4 − 1) − . z = 2z ln z∑ z 2 2 n n>1 n(n + 1)

∑ zn

n>0

⌊n/2⌋

∑

r=0

e(n+1/2)z − e−z/2 =∑ ez/2 − e−z/2 k>0

1/ 1 − 2xz n r 2 1+ √ . x = 1 − z(1 + x) r 1 − 4xz2 n

∑ ik

i=0

!

zk = 1 + ez + e2z + · · · + enz . k!

(L.70) (L.71)

(L.72)

(L.73) (L.74) (L.75) (L.76) (L.77) (L.78) (L.79) (L.80)

(L.81)

(L.82)

(L.83)


747

zn emz = ∑ Q0 (m, n) mn , 1 − z n>0 n!

Q0 (m, n) = ∑ ni /mi

where

(L.84)

i>0

is the Ramanujan Q-function. The error function: 2 erf(z) = √ π

Z z 0

(−1)n 2 2 z2n+1 . e−t dt = √ ∑ π n>0 (2n + 1)n!

(L.85)

The exponential enumerator for the number of permutations: 1 . 1−z

(L.86)

The exponential enumerator for the number of circular permutations: − ln(1 − z) = ln

1 . 1−z

(L.87)

The Euler pentagonal theorem:

∏ (1 − z j ) = ∑ (−1)k zk(3k+1)/2 = 1 + ∑ (−1)k j>1

k∈Z

k>1

h

i zk(3k+1)/2 + zk(3k−1)/2 .

(L.88)

The OGF for the number of (unrestricted) partitions, p(n), of an integer n: 1 . j j>1 1 − z

1 + ∑ p(n)zn = ∏ n>1

(L.89)

For positive integers n, let σ (n) be the divisor function (the sum of the positive divisors of n). Then ! ! nzn n n n n (L.90) ∑ n · p(n)z = ∑ σ (n)z ∑ p(n)z . ∑ σ (n)z = ∑ 1 − zn , n>1 n>1 n>0 n>1 n>1

Let an be the number of alternating permutations in the set of permutations on n distinct integers. A permutation σ = σ1 σ2 · · · σn is said to be alternating if σ1 < σ2 > σ3 < σ4 · · · . The EGF for the sequence {an } is

∑ an

n>0

zn = sec z + tan z, n!

a0 = 1.

(L.91)

Bivariate generating functions 2n n n (1 + 4yx) = ∑ (−1) y x . n n=0 ∞ ∞ m + n 2 2m 2n (1 − 2y − 2x + y2 − 2yx + x2 )−1/2 = ∑ ∑ y x . m m=0 n=0 −1/2

∞

n

(L.92) (L.93)

APPENDIX

748

∑

(k) bn xk t n

n>0,k>1

√ x k k √ x − x 1 − 4t √ 1 − 1 − 4t = = ∑ Bk (t) x = ∑ , 2t − x + x 1 − 4t k>1 k>1 2t k

is the bivariate OGF of the ballot numbers, given by k 2n + k − 1 (k) , bn = ψn+k−1,n = n+k n

n ∈ N, k > 1.

(L.94)

(L.95)

The bivariate generating function of the coincidence numbers: ∞ n un eu(z−1) = ∑ ∑ dn,k zk , 1−u n! n=0 k=0

dn,k =

n! n−k (−1) j ∑ j! . k! j=0

(L.96)

Let Pk (n) be the number of permutations of n into k integral squares > 0. Then its ordinary generating function is 1 ∂k n . (L.97) k! ∑ Pk (n)z = k ∂ x (1 − x)(1 − xz)(1 − xz4 )(1 − xz9 ) · · · x=0 n>0 The Eulerian numbers:

n n w−1 kz = w . ∑ ∑ n! w − ez(w−1) k>0 n>0 k

(L.98)

The Jacobi identity

∑ t k zk k∈Z

2

= ∏ 1 − z2 j+2 j>0

1 + tz2 j+1

1 + t −1 z2 j+1 .

(L.99)

The OGF for the number of partitions, pm (n), of an integer n into m parts: 1+

∑

16m6n

(tz)m 1 = 1 + . ∑ j 2 m j>1 1 − tz m>1 (1 − z)(1 − z ) · · · (1 − z )

pm (n)t m zn = ∏

(L.100)

There are known the following three famous approximations: n

Taylor:

f (x) =

∑ k=0 n

f (k) (a) (x − a)k + remainder, k!

(L.101)

Euler:

f (x) =

I f (k) (a) ∑ k! Ek (x − a) + remainder, k=0

(L.102)

Boole:

f (x) =

A f (k) (a) Bk (x − a) + remainder, k! k=0

(L.103)

n

∑

where I f (x) is the averageRvalue of the function f on an interval including a (for instance, on the unit interval, I f (x) = 01 f (x + t) dt), A f (x) is the arithmetic average of the function (on the unit interval, A f (x) = 12 [ f (x) + f (x + 1)]) and Bn (x) is the nth Bernoulli polynomial and En (x) is the nth Euler polynomial.

Appendix M: Recurrences

749

Appendix M: Recurrences For any two distinct numbers a and b, the sequence un = an −bn satisfies the binary recurrence un+2 = (a + b)un+1 − abun ,

n = 1, 2, . . . . √

1+ 5 2

For instance, the Fibonacci sequence Fn corresponds to a =

and b =

(M.1) √ 1− 5 2 .

Let yn satisfies the recurrence (p2 + 4q 6= 0) yn = p yn−1 + q yn−2 , then its solution is yn =

y−1 = 0, y0 = 1,

⌊n/2⌋

∑

k=0

n = 1, 2, . . . ,

(M.2)

n − k k n−2k q p , k

and ym+n = ym yn + q ym−1 yn−1 . In general, the solution of the recurrence (M.2) is given by m−1 m−1 2m − 1 − k 2m−1−2k k 2m − 2 − k 2m−2−2k k+1 y2m = y0 ∑ p q + y1 ∑ p q , k k k=0 k=0 m−1 m−1 2m − k 2m−2k k 2m − 1 − k 2m−1−2k k+1 y2m+1 = y0 ∑ p q + y1 ∑ p q . k k k=0 k=0 It can be expressed more concisely as yn = Aλ1n + Bλ2n ,

A=

y0 λ2 − y1 , λ2 − λ1

B=

y1 − y0 λ1 , λ2 − λ1

(M.3)

where λ1 and λ2 are two distinct roots of the characteristic equation λ 2 = pλ + q. If the roots are the same (when p2 + 4q = 0), it has the general solution yn = y0 (1 − n)λ n + ny1 λ n ,

λ=

p , 2

p2 + 4q = 0.

(M.4)

Let x = cos θ , the n-th Chebyshev polynomials of the first and second kinds are denoted θ] , respectively. These polynomials are by Tn (x) = cos(n arccos x) and Un (cos θ ) = sin[(n+1) sin θ solutions of the following difference equation:

yn (x) − 2x yn−1 (x) + yn−2 (t) = 0 ,

n > 2,

(M.5)

subject to the respective initial conditions T−1 (x) = x,

T0 (x) = 1,

T1 (x) = x;

U−1 (x) = 0,

U0 (x) = 1,

U1 (x) = 2x .

The Chebyshev polynomials generate many fundamental sequences. In particular, the values of the Chebyshev polynomials of the second kind at i/2 are related to the Fibonacci numbers: Un (i/2) = in Fn+1 ,

n = 0, 1, 2, . . .

(i2 = −1).

APPENDIX

750

The n-th Legendre polynomials, Pn (x) =

1 dn 2 (x − 1)n , satisfy the recurrence: 2n n! dxn

(n + 1) Pn+1 (x) − (2n + 1)x Pn (x) + n Pn−1 (x) = 0, 2

The n-th Hermite polynomials, Hn (x) = (−1)n ex

P0 = 1, P1 = x.

(M.6)

d n −x2 e , satisfy the recurrence: dxn

Hn+1 (x) = 2xHn (x) − 2nHn−1 (x),

H0 (x) = 1,

H1 (x) = 2x.

(M.7)

The gamma function, Γ(z), z ∈ C, but z 6= 0, −1, −2, . . . .

Γ(z + 1) = z Γ(z),

(M.8)

For integer values, the gamma function is factorial, Γ(n + 1) = n!, with 0! = 1! = 1. k “Subfactorial” or derangement, Dn = n! ∑nk=0 (−1) k! : Dn = nDn−1 + (−1)n ,

an = nan−1 + 1,

n > 1,

D0 = a0 = 1.

(M.9)

“Superfactorial” an = n! ∑k>0 nk : an = nan−1 + 1,

n > 1,

a0 = 1.

(M.10)

The Bell numbers, Bn , satisfy the recurrence: Bn =

n−1

∑

k=0

n−1 Bk , k

The Bernoulli numbers: n n+1 ∑ k Bk = 0, k=0

n = 1, 2, 3, . . . ;

B0 = 1,

1 B1 = − , 2

B0 = B1 = 1.

(M.11)

B2 j+1 = 0, j > 0 .

(M.12)

The inverse relations: n 1 bk , an = ∑ n − k+1 k k=0 n

n bn = ∑ Bn−k ak . k=0 k n

(M.13)

The Bernoulli polynomials: Bn (x + 1) − Bn (x) = n xn−1 ,

n > 1.

(M.14)

The Euler numbers: n

∑ k=0

2n E2k = 0, 2k

E0 = 1,

E2n+1 = 0,

n > 0.

(M.15)

Appendix M: Recurrences

751

The inverse relations: n an = ∑ bn−2k , k>0 2k The Eulerian numbers: n n−1 n−1 = (k + 1) + (n − k) , k k k−1 The Catalan numbers, Cn =

1 2n n+1 n

n n−2k Cn+1 = ∑ 2 Ck , 2k k

n bn = ∑ E2k an−2k . k>0 2k n = 1, 0

n, k > 0,

(M.16)

n = 0, 0 < n < k. k (M.17)

, are related via Touchard’s identity

2n 2n 1 = Cn = n+1 n (n + 1)!

or via first-order difference equation 2n 2n 4n − 2 Cn−1 , Cn = Cn−1 + − = n+1 n−2 n−3

n

∏ (2k − 1),

n = 0, 1, 2, . . . ;

k=1

(M.18)

n > 1,

C0 = 1;

(M.19)

or via full history recurrence n

Cn+1 =

∑ Ck Cn−k ,

C0 = 1.

(M.20)

k=0

The Motzkin numbers, Mn , satisfy either a convolutional recurrence n

Mn+2 = Mn+1 + ∑ Mk Mn−k ,

M0 = M1 = 1;

(M.21)

k=0

or the second order difference equation (n + 2) Mn = (2n + 1)Mn−1 + 3(n − 1)Mn−2 ,

M0 = M1 = 1.

(M.22)

These two sequences, {Mn } and {Cn }, are connected by the relation n Mn = ∑ Ck , k=0 2k n

Cn+1 =

⌊n/2⌋

∑

k=0

n Mk . k

(M.23)

The coefficients of tangent and secant exponential generating functions ∞ zn E2n 2n 5 61 6 277 8 1 z + z + · · · = ∑ sn = ∑ (−1)n z , sec z = 1 + z3 + z4 + 2 24 720 8064 n! n>0 (2n)! n=0 ∞ zn 2 17 7 62 9 1382 11 1 z + z + z + · · · = ∑ tn tan z = z + z3 + z5 + 3 15 315 2835 155925 n=0 n!

APPENDIX

752

are connected via the following full history recurrence: n n sn = sn−2 − sn−4 + · · · for even n > 0, s0 = 1; 2 4 n n tn = sn−1 − sn−3 + · · · for odd n > 0, t1 = 1. 1 3

(M.24) (M.25)

The numbers tn are called the tangent numbers and they are non-zero only for odd values. For example, t1 = 1, t3 = 2, t5 = 16, t7 = 272, t9 = 7936, and so forth. Let un be the solution to the ménages problem, with u3 = 1, u4 = 2, u5 = 13, then (n − 2) un = (n2 − 2n) un−1 + n un−2 + 4 (−1)n−1 , un = n un−1 + 2 un−2 − (n − 4) un−3 − un−4 ,

n > 3, n > 4.

(M.26)

Let p(n) be the partition function (the number of representation of a positive integer n as a sum of positive integers), then p(n) =

∑ (−1)k+1 [p(n − f (k)) + p(n − f (−k))] ,

(M.27)

k>1

where f (k) = k(3k − 1)/2 are known as the pentagonal numbers. The divisor numbers, σ (n) (the sum of divisors of a positive integer n), satisfy the same recursive relation (M.27) as the partition function:

σ (n) = σ (n − 1) + σ (n − 2) − σ (n − 5) − σ (n − 7) + σ (n − 12) +σ (n − 15) − σ (n − 22) − σ (n − 26) + σ (n − 35) + · · · .

(M.28)

In Eq. (M.27), if p(0) occurs in the calculation, it should be replaced by 1; but in Eq. (M.28), if σ (0) occurs, it must be replaced by n. Obviously, summations in Eqs. (M.28) and (M.28) are finite because they are terminated once the argument of the functions p and σ becomes negative.

Answers/Hints to Selected Problems

Chapter 1 1.3 m(n + 1)/2 ≈ mn/2. 1.4 The minimum occurs when all the m elements of b are conveniently located in the first m positions of A. 1.8 Assume a > b, then there are 2⌊b⌋ + 1 horizontal rows of points (x, y) in the ellipse. Since the ellipse is convex, determining the end points of each row determines the size of the entire row. By handling separately the cases x = 0 and y = 0, we then find that the number of such points can be written as 2⌊a⌋ + 2⌊b⌋ + 1 + √ ⌊b⌋ a b2 − j2 . Hence the entire computational effort is linear in the smaller dimension of the ellipse. 4 ∑ j=1 b 1.9 The costs “crosses” between n = 32 and n = 64, and quite closer to the latter. 1.18 (a) n + 1;

(b) 15;

(c) 50;

(d) 13.

1.23 r3 = 6. 1.30 Induction does not work because the equation does not hold for initial values: m = n = 1. n

1.45 A possible tighter lower bound is Ln = 22

+n .

2 −F F n+1 as a quadratic equation for F . 1.49 View the formula Fn2 − Fn−1 n n−1 = (−1) n

1.50 In Power1 we have n + 1 operations. floor of the binary logarithm of n.

In Power2 we have at most 2 + 2λ (n), where λ (n) ≡ ⌊lg n⌋, the

Chapter 2 2.3 (a) All values. (b) All values. (c) All values. (d) When x is an integer, the equation is true. If not, let x = n + s. The equation holds for either 0 < s 6 13 or 32 < s < 1. √ 2.4 Let m = ⌊ x⌋, then B(x) = A(x) − 1 unless x ∈ {m2 , m(m + 1)} or (m + 1)2 − 1 6 x < (m + 1)2 in which case A(x) = B(x). 2.9 (a) ⌈n/2⌉2 .

(b) Let N =

n−2 3

, then the sum becomes

N+1 [8 + 15N + 6N 2 ]. 2

2.10 (a) n(n + 1)(2n + 1)/6. If m > n, the number of squares is n(n + 1)(3m + 1 − n)/6. m > n, the number of rectangles is mn(m + 1)(n + 1)/4. 2.12 (a) 1 x2

n (6n2 2

− 3n − 1).

(− ln(1 − x) − x).

(b)

Φ4 (n) = Φ2 (n)(3n2 + 3n − 1)/5.

(c)

n(3n+5) . 2(n+1)(n+2)

(b) n2 (n + 1)2 /4. If

(d) 1 −

1 n! .

(e)

(c) −4m3 − 9m2 − 6m − 1 = 2.13 (a) H⌊n/2⌋ . (b) 21 + 3H⌊(n−1)/2⌋ − 3Hn−1 − H⌊(n+1)/2⌋ + 3Hn+1 . 5 3 . (f) a−1 ea + 1 − 1 . 1 1 1 3 2 −(4m + 1)(m + 1) . (d) Hn−1 + 4 − 2n − 2(n+1) . No limit. (e) 4 + 2(n+1) − 2n 2 a2 a2

753


754 d d x dx 2.14 ∑nk=1 k2 xk = x dx ∑nk=1 xk . The limit is

2.18 (a)

∑

∑ ani . (b) ∑

i>1 n>i+1

2.19 (a)

∑ ani .

x(1+x) . (1−x)3

(c) ∑

i>1 n>i+2

ab 1−(ab)n 1−b 1−ab

n+1

1−an 1−a .

b − a1−b

(b)

ani .

∑

n−1 n−k

i>1 n>max{1,i−1} ab b−1

h

1−an 1−a

− b1n

1−(ab)n 1−ab

i

∑ ∑ ak j .

(d)

k=0 j=1

.

2.20 (a2 b)n(n+1)(n+2)/6 . 2.26 ln 2. 2.34 (a)

∆ (x + 3)(4x − 1) = 8x + 15.

2.35 ∇−1 1 = x +C,

(b)

∇−1 x = 2x (x + 1) +C,

5 . (3x+2)(3x+5)

∇−1 x2 = 6x (x + 1)(2x + 1) +C.

2.40 In all problems we use the equation ∆qk = qk (q − 1). R b (c) 1b 01 1−u 1−u u du. n−1 n (b) ∑k=1 ∑m=k+1 akm .

n 2.41 (a) ∑n−1 k=1 ∑m=1 akm .

2.46

(a) (2n − 1)2n+1 + 3;

(b) 3n 4n+2 − 4n+3 + 43 ;

(c) ∑nk=1 ∑nm=k akm .

a sin b−an+1 sin b(n+1)+an+2 sin bn . 1−2a cos b+a2

2.51 336, so no. 2.52 74. 2.53 (a) nm ;

(b) nm .

2.54 (a) n!;

(b) n(n2 + 1)/2.

2

2.56 (a) nn ;

2

(b) n1+(n−1) ;

2.63 (a) m(x + 1)m−1 ; 2.64 (a)

n 2(n+2) ;

(b)

(c) n1+n(n−1)/2 .

(b) mxm−1 ;

(c) m xm−1 ;

(d) m (x − 1)m−1 ;

5 12 .

2.65 (a) k2 = k(k − 1) = k2 − k

k3 = k3 − 3k2 + 2k;

(b) k2 = k2 + k

2.69 17850. 2.71 194,594,400 distinct anagrams. 2 +···+an 2.79 a1a+a . 1 ,a2 ,...,an 2.87 n = 14. 2.88

4 4 4 2 2 2 44 = 123, 552

2.91 480.

2.92 (a) 120;

(b) 12.

(e) m (x + a + 1)m−1 .

and

13 4 12 4 4 4 1 2 3 1 1 1 = 1, 098, 240.

k3 = k(k + 1)(k + 2) = k3 + 3k2 + 2k.

Answers/Hints to Selected Problems 9 20 2 5 = 558, 144.

2.93 (a)

755

(b) 2 × 558, 144 = 1, 116, 288.

2.94 168 = 518, 918, 400 choices. 2.95 1350. 2.97 3,960,000. 2.99 (a) 220; 2.101 (a) 0; 2.103 (a)

(b) 49 = 262, 144; 1 ; (b) − 16

(−4)−n 2n 1−2n n ;

(c) 220 × 9! = 79, 833, 600.

(c) −4; 1 2n 2 n ;

(b)

(d) 1; (c)

(e) 0;

(f) 232 63!!.

2n−1 n .

2.109 The total number of inversions in a permutation and its inverse must be equal to 2.116 (−1)r δt,r , where Kronecker’s delta is defined in Eq. (2.5).

2.117 Calculate the sum of the sequence, using the symmetry relation (n−1)/2

∑

k=1

n 2 .

n n k = n−k

1 n−1 n n 1 1 (n−1)/2 n n = ∑ + = (2n − 2) = 2n−1 − 1. = ∑ n−k k 2 k=1 2 k=1 k 2 k

This is an odd number, which implies the claim. 2.121 Substitute an (−1)n in Eq. (2.69) instead of an . i h 2 − H (2) − 2nH 2.126 (H ∗ H)n = (n + 1) Hn−1 n−1 + 2n − 2. n−1 2.130 ∑m k=0

k+(−n−1) = −n−1+m+1 = m−n = (−1)m n−1 m m m . k

k n+(m−n+k) m+k k n−m 2.132 ∑k>0 m+k ∑k>0 m−n+k xm−n+k n x = ∑k>0 m−n+k x = x xn−m = xn−m ∑ j>m−n n+j j x j = xn−m ∑ j>m−n −n−1 (−x) j = (1−x) n+1 . j n i+ j+m+n n−1 m−1 i+ j 2.136 ∑m i=1 ∑ j=1 (−1) j−1 i−1 i n n+ j n−1 i+ j = m (−1)m+i m+i m−1 = ∑m ∑i=1 i=1 (−1) j−1 i i−1 ∑ j=1 (−1) 1+1 2 = (−1)m+m−1+1 n−m+1 = n−m+1 . 2.144 (a) (1 + z)a ; 2.147 F

2.150 (a)

1 z

(b)

t+1, t−m−q, −p t+1−p−q,t−m+1 1 (2n)! 2n n = n!n!

≈

sin z;

(c)

1 z

m−1 i+1 i−1 n

sin[−1] z.

.

√ (2n)2n 2π 2n e−2n √ 2 ; n (n 2π n e−n )

(b)

(4n)! 4n n = n!(3n)!

≈

nn

√ 4n 2π 4n√e−4n √ (4n) . −n 2π n e (3n)3n 2π 3n e−3n

Chapter 3 n

3.20 Exactly the same calculation as above, with Pr[A] = 1 − Pr[A′ ] = 1 − 12 12n , giving the answer 5. The following table gives the probability of A for some values of n:


756

n 1 2 3 4 5 6 Pr[A] 0 0.08333 0.23611 0.42710 0.61805 0.77720

3.21

364 365

n−2

3.24 ∑m−1 k=0 3.25

; (b) n > 2 +

ln 2 365 ) ln( 364

≈ 254.7.

−n+1 (−p)k . k

pn−1 (1−qm ) , pn−1 +qm−1 −pn−1 qm−1

where p is the probability to see heads and q = 1 − p. (c) 134 · 8 · 39/

3.27 (a) 2197/8330; (b) 94/54145 ≈ 0.001736; provided about other hands.

52 6 ≈ 0.426482;

(d) no information is

3.28 1/2. 3.29 4/9. 3.30 (a) One half, by symmetry.

(b) 2/5,

(c) 1/3.

3.31 (a) 5/13, (b) 55/143. n 3.32 n−a r−a / r .

3.34 The probability that the value is even is 3/7.

3.42 T-to-H: Longest run:

PMF PMF

= ( 11, 165, 462, 330, 55, 1 )/1024; Separated by 2 flips: = ( 2, 176, 370, 254, 126, 56, 24, 10, 4, 2 ) / 1024.

3.43 Pr[Z = −1] = 72 , Pr[Z = 1] =

2 21

4 6 + 21 + 21 =

12 21

= 74 , Pr[Z = 1.6] =

3 21

PMF

= ( 731, 267, 25, 1 )/1024;

= 71 .

3.45 n = 4. 3.46 (a) Pr[R = r] = 2 × 2n−r−1 × ( 12 )2n−r . n−1 1 2n−r+1 = 2n−r × ( 1 )2n−r , 0 6 r 6 n. 2 2 n 3.47

5 4

(b) ∑nr=1

2n−r−1 −(2n−r−1) . n−1 2

(c) Pr[R′ = r] = 2 ×

2n−r × n

− π4 ≈ 0.4646018366.

3.59 Die C with either B or D provides the best advantage. 3.61 The PMF is Pr[D = k] = 1/n, 1 6 k 6 n, and E[D] = (n + 1)/2; and E[N] = ∑b=0 Pr[N > b] = m/(m + 2). 3.65 (a) (r − 1)(r − 2) 6 2n; 3.66

(b) 4n 6 r2 − r + 2.

n+2 4 .

3.68 For m = 365, there should be least 94 people. 3.72 (a) The expected value does not exist. 3.75 Hn . 3.76 107/29.

(b) E[X] =

w+b−1 . b−1

If b = 1, then E[X] is infinite.


757

3.77 The annual premium is $349.998/0.4 = $874.995. 3.78 29/3. 3.79 ∑n+1 k=1

n!k(k+1) . (n−k+1)!nk

3.80 (a) m/(m − n);

(b)

m+1 m+1−n .

n 3.82 (a) Tn = 2(2) ; (c) n−1 n 1 t(n−t) . ∑t=1 t P(t) 2

3.83

2−(n−1) ;

(d)

(n − 1)!/2n−1 ;

The full-history recurrence: 1 − P(n) =

(e)

(n+1)2 2n−1 .

3.84 E[R] = 2n

2n −n n ×4 ;

E[R′ ] = (2n + 1)

3.100 E[X] = Hn , V [X] = 2n − Hn − HN2 .

2n −n − 1. n ×4

3.101 (n + 1)/3. 3.103 β1 = x7/8 = 3.

17 √ 2 2

≈ 6.010407638, β2 = − 157 8 = −19.625. The lower octile is x1/8 = 1 and the upper octile is

3.104 µ = 211, σ 2 = 73.79.

3.110 1/4. 3.113 (a) 3/7.

(b)

2 m(m+1) .

3.114 1 > Pr[A ∪ B] = a + b − c. 3.116

w b+w

w+k b · b+w+k · b+w+2k .

3.126 No, these events are dependent. 3.127 The events Ai , i = 1, 2, 3 are jointly dependent because A1 ∩ A2 ∩ A3 = ∅. 3.130 E[X] = n2 /(2n − 1); V [X] =

2n2 (n−1)2 . (2n−1)2 (2n−3)

3.146 α = σY2 /(σX2 + σY2 ).

3.147 Cov(R, T ) = σ 2

n2 (n+1)2 4

3.154 (a) 394.4444; 3.156 (a) E[g(X,Y )] =

(b) 430;

and

ρ (R, T ) =

√ n(n+1) 3 5 √ . (2n+1) 3n2 +3n−1 2

h (c) Pr[T500 = 350] = Pr T1 =

77 ≈ 2.85185. 27

(b) E[X] = 34 ; E[Y ] =

3.157 20/216 = 5/54 ≈ 0.0926.

Chapter 4

350 500

11 . 9

i = 0.7 = 0.2, Pr[T500 = 450] = 0.


758 4.2

11 32

≈ 0.34375.

4.7 5/12. 4.8 ≈ 0.9951538592. 4.9 N 6 25. 4.11 p < benefit.

w−1 , (c−1)(w+c−1)

4.13 Pr[Xn = k] =

which for w = 128 and c = 12 suggests that for all p < 0.08306 this code delivers a

n k n−k n−k (−1)r , k p ∑r=0 r 1−qk+r

E[Xn ] Pr[Xn = 1]/q.

4.16 Pr[X1 + X2 = i, X3 = j, X3 + X4 = k] = √ √ pi p j √ . 1−pi 1−p j

4.17 ρ (Xi , X j ) = − √

n i j k− j i, j,k− j (p1 + p2 ) p3 p4 .

4.18 ≈ 1.61252 × 10−5 . 4.26 (a) Hn+1 − 1. 4.28 (a)

(b) e − 1/2.

Pr[X = n + k | X > n] = qk−1 p.

4.31 (a) E[Xn ] = n!. en (1) + 1/(n − 1)!.

(b) Pr[X = n | X > 1] = Pr[X = n]/ Pr[X > 1] = Pr[X = n − 1].

(b) The smallest possible Cn is n − 1, but Cn is unbounded. E[Cn ] = n!(en (1) − 1) −

4.32 (a) Minimum: — n − 1 times, Maximum — unbounded; Average — (n − 1)(n + 2)/2. n(n+3) 4 .

(b)

n(n+1) 2 Hn −

4.35 Let L be the number of additional serves in a game of tennis that has reached deuce, then E[L] = 4.36 E[M] =

2 1−2pq .

2+2q(p−s) 1−qs(2−qs+pr) .

4.41 E[G(r, p)] = r/p. 4.42 E[G(r, p)] = pr , E[G(r, p)4 ] =

r p4

E[G(r, p)2 ] =

(r+1)r p2

E[G(r, p)3 ] = 3 r + 6qr2 + rq(5q + 4) + q + 4q2 + q3 .

4.43 (a) Pr[X = k] = 2−(k−1) ;

− pr ,

r p3

2 r + 3rq + q2 + q ,

(b) Pr[X = k] = (k − 1)2−k .

(2)

4.45 V [X] = n2 Hn − n Hn . 4.48 h(57, 100; 500, 250) = 0.026324, h(57, 100; 5000, 2500) = 0.029773. For infinite population, the probability is 0.0300686. 20 20 20 4.54 5,2,9,4 / 7 14 ≈ 0.3873839 . . .. 4.56 Cov(Xi , X j ) = −Mi M j (N − n)/N 2 (N − 1). 4.64 (a)

Pr[X = 1] =

0.01×e−0.01 1!

= 0.0099005;

(b) Pr[X > 1] = 1 − 0.9900498 = 0.0099501663.


759

4.76 The sequence Xn converges almost surely, in probability, and mean. However, the sequence YN converges only in probability, and diverges in two others. 4.77 (a) Converges to the uniform distribution on (0, 1). (b) The limiting distribution is degenerate. (c) There is no limiting distribution. (d) The limit is not a distribution function. 4.83 17/30. 4.85 1/(1 + q). 4.88 Pr[ww] = (rm + q2m )(r f + 4.89

17 40

qf 2

).

= 0.425.

I − J when I > J So a given distance J − I when J > I k occurs at (1 6 I 6 n − k, J = I + k) and (1 6 J 6 n − k, I = J + k). For k = 0 the two events coincide, but otherwise they distinct, hence aren−k for 1 6 k < n 2 n2 The sum is indeed 1. Pr[D = k] = 1/n for k = 0 (b) Direct computation: 4.91 (a) The distribution of D is derived from the definiton: D =

E[D] =

2 2 n−1 ∑ k(n − k) = n2 n2 k=1

n

n(n − 1) (n − 1)n(2n − 1) − 2 6

Similarly we find for the second moment E[D2 ] = (n2 − 1)/6, and V [D] = 4.92 E[Xk ] =

n+2k −1 . 2k

V [Xk ] = (n − 1)

1 2k

=

n2 − 1 n ≈ 3n 3

(n2 −1)(n2 +2) . 18n2

+ n−2 − n−1 . 3k 4k

4.93 $442.70. 4.95 (a) 0.2451. 4.96

(b) 0.2139.

6 11 .

4.97 k(n + 1)/2. 4.98 1/3. 4.99

1−qn qn p .

4.100 (a) E[K] = (n − 1)/2; V [K] = (n + 1)/12. (d) V [K] = n4 . 4.105 (a) ≈ 0.308.

(b) ≈ 0.973.

4.106 The jailer is wrong. 4.107 ≈ 0.4899. 4.108

2 29

≈ 0.068966.

4.109 Pr[R = r] =

1 nr+1

r−1 − kr . ∑n−1 k=0 nk

(b) E[K] = 2n ; V [K] =

n−1 4 .

(c) E[K] =

3n 3n 4 , V [K] = 16 .


760 4.110 Pr[C | T ] =

n n+2m ;

Pr[C | T k ] = n/(n + 2k m).

4.114 A quick answer is 2/7 while the law of succession gives 3/9. (n)

4.115 limn→∞ pi

= 0 if i 6= i1 , i2 , · · · , ir .

4.118 Pr[W = k] =

1 n2

min{n,k−1}

E[W ] = n + 1.

∑i=max{1,k−n} 1;

4.121 E[N] = k + ek − ek (k). 4.122 Pr[X +Y = n] = (n − 1)p2 qn−2 . 1 10 2

4.123 a =

=

1 1024 .

The expected value is 509/128.

4.126 Pr[X +Y = k] = (1 + ρ )−m−n ρ k

4.128 Pr

h

Wn +n 2

m+n k .

i = k = Pr [Wn = 2k − n] =

4.131 E[Sn ] = np and V [Sn ] = npq. 4.135 Pr[D = r] =

4 N4

n

n+2k−n 2

p

n+2k−n 2

q

n+2k−n 2

=

n k k k p q .

[N(N − r)r + r3 /6].

4.140 For n = 5 we have Pr[Y1 6 2] ≈ 0.868313 but for n = 6 we have Pr[Y1 6 2] ≈ 0.912209. 4.141 (a) (n − 1)(n − 2)/6n2 . 4.146 E[Xmin ] =

4381705 2330636

(b) 1/6.

= 1.88004 . . ., V [Xmin ] =

6842931587015 5431864164496

= 1.25977 . . . .

4.147 Pr[xN 6 x] = 1 − p + pm . 4.160 |X − µ | 6 0.3.

4.164 (a) The Chebyshev inequality provides no bound (or any information) on the probabilities of deviations smaller than the standard deviation. (b) The Chebyshev inequality (4.44) gives better result. (c) Pr[X ∈ (10, 16)] = Pr[X > 10] − Pr[X > 16] < Pr[X > 10] 6 9/13. 4.168 Pr[X = k] = Pr[L = k − 1] = (1 − pa − pb )(pa + pb )k−1 . 4.172

n+1 k+1 .

4.173 The expected saving while sorting (n + 1)-array is ∑nj=1 4.177

2 j+1

= 2 (Hn+1 − 1).

n−k+1 . n−k+2

Chapter 5 (2)

5.6 V [D] = 2Hn + 4Hn − 80 9 . 5.15 (a) an = an−1 + an−2 .

(b) bn = bn−1 + bn−2 .

(c) cn = dn−m−1 , where dn = ∑m−1 k=0 dn−k−1 .


761 (b) rn = rn−1 + n = 1 + ∑nj=1 j = 1 + n(n + 1)/2.

5.18 (a) an = an−1 + n − 1 = ∑nj=1 ( j − 1) = n(n − 1)/2. (c) ak = 0, and rk = k + 1. 5.19 Gn+2 = Gn − Gn+1 . 5.22 (a) a1 = 1, a2 = 2, a3 = 3, a4 = 5.

(b) an = an−1 + an−2 .

5.31 Putting B(n) = A(n) − A(n − 1), we reduce to B(n) = [B(n − 2) − B(n − 1)]/n. 5.34 yn = xn−m+1 , n > m. 5.36 (a) xn = aFn−1 + bFn .

(b) xn = (a + r)Fn−1 + (b + r)Fn − r,

n > 0.

5.43 (b) S(h) = Fh−1 + 2Fh − 1 = Fh + Fh+1 − 1 = Fh+2 − 1, h > 0. (c) S(h) ≈ √ √ log( 5(n+1)) ⌋ − 2 ≈ 1.440 lg n − 0.3278. (d) S(h) = h, H(n) = n. (1 + 5)/2; H(n) = ⌊ log φ

φ√h+2 5

− 1, where φ =

5.44 Setting rn = un+1 /rn + a, we obtain from the given recurrence that un+1 + (a − b) un + (c − ab)un−1 = 0. 5.48 {un = un−1 + vn−1 , 5.50

vn = un−1 + vn−1 }.

n . 2n + 1

5.51 E[R] =

r b(b+1) .

√ r < b(b + 1) or b > 21 ( 4r + 1 − 1).

5.52 xn+1 = xn (1 − a)2 + a; xn =

1+(1−a)2n+1 . 2−a

5.54 xn = 1 for n > 2; x1 = 2. 5.57 xn = n! Hn . 5.58 pn+1 = (n + 1) pn + n!. 5.60 (a) an+1 = n!en (2).

(b) bn = n(n + 1)Hn −

n(n+3) 2 .

(m+1)/3

5.62 wn+1 = 106 × 4 41/3 −1−1 + 107.3 × 4(m+1)/3 ; d2025 ≈ 2.0757 × 1012 . 5.64 For n > 2, Tn = Tn−1 + 2n + 1 = n(n + 2). 1 2n 5.65 nun = 2(2n − 1)un−1 + 2n−1 n , un = 2 n Hn . A(n) = 2 A 5.70 T (n) = 3 T n2 + 2A 2n + 4A(n), A(n) = n A(1).

n 2

since the number of additions is essentially linear:

def

5.74 With br = a2r +1 , the recurrence becomes (2r + 1)br = 2r + 1 + 2r−1 (r − 1) + (2r−1 + 1)br−1 . 5.76 Setting W (n) = nE(n), W (n) = (k − 1)2k + 1 + (k + 1)(n − 2k + 1). 5.77 E(n) =

(n+1)⌊log2 n⌋+n−2⌊log2 n⌋+1 +2 . n

5.79 b = nr + 2s when n = 2r + s, 0 6 s < 2r .


762 5.80 J(n) = 2(n − 2⌊lg n⌋ ) + 1. 5.81 J(n + 1, q) = J(n, q) + q (mod n + 1).

5.85 (a) Partition1: there are n + 1 comparisons. Partition2: Qn = n − 1 for an array of size n. Cn = 2(n + 1)Hn − 4n. (b) Let the number of exchanges be Fn . For Partition1: E[En ] = n+1 3 Hn + 5n/9 − 11/18.For H + 5n/9 − 11/18. Partition2: E[En ] = n+1 n 3 5.88 (a) c2 = 1 + 12 × 0 + 21 × c2 .

(b) cn = 2Hn − 21 − n1 .

5.91 The height: hn = 2n + 1; dn = 4(n + 1)Hn − 4n + 1. A2n+1 −A , A2n+2 −1

5.92 xn =

√ where A = 2 + 3.

5.93 71/5 ≈ 1.47577316159. The Bisection algorithm requires 20 iterations, whereas the False position algorithm requires 1 iteration because it arrives at m = 71/5 right away. √ 1 e = √ . 5.97 For the error ek = xk − R, we get ek+1 = 21 xk + xRk − (xk − ek , so lim k+1 k→∞ e2 2 R k 5.98 To get 0.000001 accuracy: Newton’s method – 5, Bisection – 20, False position – 14, Chebyshev – 4, and Secant – 6 iterations. 5.100 Except the first and the last function, the fixed point iteration xn+1 = f (xn ) does not work. Therefore for these functions we need to use another method, for example, the Newton’s method. √ 5.107 (a) Gn+1 = G1n + 1. (b) The limit is φ = 21 [1 + 5]. 5.108 φ = 1 + 11 +

1 1 1+ 1+

1 φ

··· ;

= φ − 1.

5.117 6 iterations are required. 5.129 The coefficients {bn,k } satisfy the recurrence bn+1,k = bn,k + bn−1,k−1 , b1,0 = 1, b2,0 = 1, b2,1 = 2. 5.130 (a) G p (n, k) = 2k−n n ∑k−1 i=0 nG f (n − 1, k − 1).

k−1 n−1 i n−1 i ∑ j=0 j j

−i = 2k−1 n ∑k−1 i=0 2

n = 1, 2, . . . ,

k−1 i G p (n − 1, i).

(b) G f (n, k) =

Chapter 6 n

6.3 Any finite sum Sn = ∑ ak can be considered as a convolution of two sequences, {ak } and {1}. k=0

6.5

R 1 √u (−1)n ∑n>1 n+1/2 = − 0 1+u

6.7 S(z) =

cz 1 1−bz A 1−bz

6.9 (a) P(t; z) = etz . 6.11 (a)

1 1−z .

du =

π 2

− 2.

.

(b) P(t; z) = (1 − z)−t .

(b) (2z + 1) ez .

(c) (z2 + 2az + z + 1) ez .

(d) (1 + z)s .

(e) (1 − z)−s .

Answers/Hints to Selected Problems 6.12 (a) ez cost sin(z sint). 6.14 F(z) = e−z

dk 1 dzk 1−x

6.15 F(x; z) = ∑n>0

zn n!

763

(b) ez cost cos(z sint). k

d and G(z) = e−z dz k

zk 1−x .

−a/b ]x , which is manifestly multiplicative in x. ∏n−1 i=0 (ax + bi) = [(1 − bz)

6.16 "

∑

n>0

# n n a zk zn−k a zk n z k =∑ k ∑ k ak n! = ∑ ∑ k! (n − k)! k>0 k! n>0 k=0 k

6.19 ∑n is divisible by 2k

6.25

1 . 1−z(x1 +x2 +...+xn )

6.26

ez(u−1) 1−z .

6.27 N(x, y) = y

1−C(xy) 1−x−y ,

1 nz

= ∑n>1

1 (2k n)z

∑ n>k

zn−k a zk = ez ∑ k = ez a(z). ˆ (n − k)! k>0 k!

= ζ (z)/2kz .

√ where C(z) = (1 − 1 − 4z)/2z.

6.28 A(x, y) = [(1 − x)2− 2xy(1 + x) + x2 y2 ]−1/2 ; C(x, y) = (1 + y)/ 1 − x(1 + y)2 .

B(x, y) = xy [xy − 1 − x] /A(x, y);

6.29 (1 − z)−u . h √ i 6.31 13 e + √2e cos 23 6.32

2n n+1 .

6.34 (a)

1−x 1−2x ;

(b)

(1−x)2 . (1−x)3 −x3

6.35 2n/2 cos n4π . 6.36

n 2n+1 .

6.38 ∑n>0 F3n 2−3n =

16 31 .

6.44 gk = [zk ]g(z) × [zn−k ](u − vx)−1 . 6.49 2n/2 sin(nπ /4).

6.63 bn, j == ∑rk=d+1 ek bn+d−k, j−1 , 1 < j 6 n. 6.70 (a) ∑10 r=0

1 2(n−r−1) 1 2r r+1 r × n−r n−r−1 .

6.71 (a) rn = (n − 2)/n; 6.73 T (z) =

1−tz 1−2tz+z2

(b)

min(n−1,10)

(b) tn = ∑r=0

tr tn−r−1 .

n−2 2n−1 .

and U(z) =

1 . 1−2tz+z2

z 2 6.77 (a) ∑nk=0 Fk Fn−k = 15 [zn ]( 1−z−z 2) ;

(b) ∑nk=0

n k Fk Fn−k

n

z ] 1 (eφ1 z − eφ2 z )2 . = [ n! 5


764 1 1−z

6.78 (a)

z × 1−z−z 2;

6.80 (a) T j (t) = ∑r

z(1−z) . (1+z)(1−3z+z2 )

(b)

j 2 r j−2r , U (t) = ⌊n/2⌋ (−1)k n−k (2x)n−2k ; ∑k=0 n 2r (t − 1) t k

(b) Fn+1 = in Un (−i/2).

6.162 (px)2 /(1 − px − qpx2 ).

Chapter 7 7.8 If two sets are not disjoint then the property 7.7 does not hold. 7.16 Yes, unique decomposition holds. 7.21 0.04733 . . .. 7.22 40 for $100 and 21326 for the $1000. 7.23 sn = Fn+1 − Fn = Fn−1 , the Fibonacci number. 7.24 Fn+1 , the Fibonacci number. ⌊n/6⌋ 2r+k−1 2r+1 .

7.26 ∑r=0

1 7.28 [z95 ] 1−z 2

1 1 1 1−z5 1−z10 1−z39

7.34 (a) 1,353,275;

= 79.

(b) 19,720,001.

7.35 250. 7.36 880. 7.37 ∏ki=1 z pi 7.39

1 6n

1−zqi −pi +1 . 1−z

⌊ 6n ⌋ (−1) j nj ∑ j=0

7.40 2⌊n/2⌋ .

2n−6 j−1 . n−1

7.43 (a) ∑k>0 un,k = ∑k>0 7.49

⌋ ⌊ n+(p−1)k p . k

n−k+1 . k

(b)

1 (1−z)k+1

k+1 si z − zti +1 . ∏i=1

7.59 V [C] = β 2 n/4.

7.70

zk 1−zk

∏k−1 j=1

1 1−z j .

7.73 (a) ppalindrome (n) = ∑ j>0 pe (n − j), but a closed form is unlikely. 7.82 Em (z) =

zm 1+z

∏m−1 k=1

(n−1)! = k! 7.83 k k(n−k)!

n−1 k−1 .

1 1−kz

=

Γ( 1z −m+1) zΓ( 1z +2)

and Fm (z) = z ∏m−1 k=1

z 1−kz .

(b) 2⌊n/2⌋ .


765

7.97 Choosing a permutation of length k (1 6 k 6 n) is equivalent to breaking the set [1..n] into two disjoint parts, one containing k elements, from which we build cycles of length k. 7.98 (a) ez+z

2

/2 ;

(b)

√1 . 1−z2

7.99 O(z) = ∑k>0

k ϕˆ U (z) k!

7.100 (a) 504;

(b) 1680.

= exp(sinh(z)).

7.101 (a) er (z) es (z) et (z);

s

(b) ∑

q=r

3q z3q q,q,q (3q)! ;

(c) ∑tn=0 (zn /n!)(z0 + z1 + z2 )n = et (z(z0 + z1 + z2 )).

7.105 ϕˆ L (z) = ϕˆ R (z)ϕˆ E (z)ϕˆ P (z) − 1, where ϕˆ R (z) = 1 + z, ϕˆ E (z) = 1 + z + z2 /2, ϕˆ P (z) = 1 + z + z2 /2 + z3 /6.

7.106 (a) ϕa (z) = ∑7k=0 7k zk = ∑7k=0 6 ∑nk=0 7k zk = k−2 zk . 3281 40320

7.107

7 k k k! z .

(b)

ϕb (z) = ϕa (z) + ∑k>2 k!2

6 k k−2 z .

(c)

ϕc (z) =

≈ 0.081374.

7.108 (a) 2n − n.

(b)

7.109 5661.

n+1 3 + 1.

Chapter 8 8.1 The enumerator for the mixed trees: tm (z) = 1 + t3k (z). √ 1−2z−3z2 2z

8.5 pn,2 ≡ [zn ] 1−z−

= − 21 [zn+1 ](1 + z)1/2 (1 − 3z)1/2 .

8.7 Labels can be distributed to n nodes in n! ways. 8.9 (n + 1)/3. 8.13

1 n

1 + n1

k−1

.

8.19 The enumerator ϕ (m,t, z) =

m−r m tz . r (e − 1)

8.20 As shown in Eq. (8.30), (ez − 1)4 . 8.21 The PGF of X is pX (u) =

v! v t d v [t ](e

− t + ut)d , and its mean is v

r in−r+i r+i+1 . (−1) 8.22 (a) n! (−1)r+1 δn,r + ∑r−1 i=1 (n−r+i)! i n r∗

d−1 d

v−1

.

(b) The optimal values are as follows:

6 8 10 15 25 50 100 200 500 1000 5000 4 5 6 7 10 17 28 48 101 178 702

8.23 The single-bin enumerator is ϕ1 (t, u) = 1 + t + u (et − t − 1), and the enumerator of the entire distribution is ϕ1m (t, u).

8.28 e0 = 0, e1 = 5, e2 = 12, e3 = 4, e4 = 1, e5 = 0.

n0 = 22, n1 = 45, n2 = 30, n3 = 8, n4 = 1, n5 = 0.


766 8.32 e0 = 257 × 7! = 1295280. 8.33 From 26 through 35. r n−r n i s−i / s .

8.36 (a) q(n, r, s; i) =

(n) (b) n2ri ∑u>i u!

8.38 There are 46 prime numbers less than 200.

n−i r n−us u−i u ∑v>i v! v−i v .

8.39 144,000. 8.40 998910. 8.47 (n − 2)! [n2 − 3n + 3]. 8.51 (a) e1 = 15.

(b) n0 = 13.

8.52 Yes. 8.56 (a) A particular case of the next part. i 1 (b) ∑n−1 i=0 (−1) i! .

8.66 (a) ∑ni=0 (−1)i i!1 . 8.67 (a) 8.68

1 n!

∑ni=0

51 −1 k−1

(−1)i (n−i)! . i!

⌊n/4⌋

∑i=0 (−1)i

pk k!

∑n−k j=0

(−1) j j!

⌊n/2⌋ (−1)k 2k k!

8.81 n! ∑k=0

(c)

12 48−4i i k−4−4i .

m (2m− j)! j 2m− j .

1 k!

i ∑n−k i=0 (−1)

1 k!

− ∑nk=1

(b) 1 + 2 ∑ nk=1 (−1)k

8.69 The mode: k = 14; the mean: 1353098944755095 ≈ 27.678. 48886777480464 8.70

(b) n j =

1 i!

=

1 k! en−k (−1).

(−1)k (n−k)! . k!n!

E[Xmax ] = ∑44 j=1 Pr[Xmax > k] =

109378345 ; 6991908

the variance: V [Xmax ] =

p j. ∑n−k r=0

(−1)r r! .

1 −z− 8.82 The needed enumerator is exp(ln 1−z derangements.

z2 ) 2

= D(z)e−z

2

/2 ,

where D(z) is the

EGF

of the number of

8.87 On average, it will take n rounds. 1 m−i j (b) The difference is that each letter in a string has two neighbors (except 8.88 (a) ∑m i=k i! ∑ j=0 (−1) / j!. the end characters), whereas in the dance, each person has a single partner. 1 m−r 8.89 E[X k ] = ∑m r=0 r! ∑l=0

(−1)l l!

d = ∑m d=0 ∑l=0

(−1)l l!(d−l)!

8.94 (a) Since N(−1, z) = E(0, z), this is the (n − 1)/2, Vk = (n + 1)/12.

8.98 ∑nk=1 (−1)k−1

n k k · 2 · (2n − k)!

EGF

= ∑m d=0

1 d!

∑dl=0

d l l (−1) .

of all permutations with exactly zero rises.

8.101 The exponential generating function for the sequence {an } is

e2x 2−ex .

(b) µk =


767

Chapter 9 9.3 (a) Zero.

(b) With r-way decision made k times in a row: rk .

9.16 (a) {11}∗ = (11)∗ is the set of all even length strings of ones, and the empty string, ε . (b) {0, 1}∗ is the set of all binary strings with 0 and 1. (c) {1, 11}∗ is the set of all strings of the digit 1. (d) {a, b, c, . . . , z}∗ is the set of all strings formed from the Latin alphabet. 9.17 Regardless of A, the closure A∗ has the empty string. 9.20 The unique decomposition property says none of the pairs (a, b) ∈ A × B is repeated. 9.21 (a) Not. (b) Let a ∈ A and a 6= ε , then a = ε .a = a.ε . that all ni be equal to one particular value. 9.22 (a) AB = A. (b) AB = B = ∅. AΣ∗ = Σ∗ .

/ A]ε . (c) AB = B − [ε ∈

(c) The necessary and sufficient condition is

(d) No condition is needed for A because

9.30 (a) The language consists of those words in which every single-letter run is of even length. (b) If appears, 0 is not single. (c) This language consists of even length words with appending a corresponding letter. (d) L = {then, so, if.then, if.so, when.then, when.so}. 9.33 (a) ra = (b∗ ab∗ a)∗ b∗ . (b) rb = (a + b)∗ baba(a + b)∗ . (c) rc = a∗ ba∗ ba∗ . (d) rd = (a + b)(a + def b)∗ = (a + b)+ . (e) re = a∗ b∗ . (f) r f = a∗ (ε + b)a∗ (ε + b)a∗ b∗ or a∗ (ε + ba∗ b)a∗ b∗ . (g) rg = (a + ba)∗ bb((ab)∗ a + aa∗ )bb(a + ab)∗ or (a + ba)∗ bba(a + ba)∗ bb(a + ab)∗ . 9.39 (a) The ε does not move M from v0 to v3 . (d) This string is accepted.

(b) This string is not accepted.

(c) This string is accepted.

9.41 The initial state is an accepting one. 9.45 w13 = 0.w23 + 1.w13 ,

w23 = 0.w33 + 1.w13 ,

w33 = ε + 0.w33 + 1.w33 .

9.47 Upon substitution of x = r∗t into the equation, both regular expressions, r∗t and rr∗t + t, generate the same language. 9.49 Any x is a solution, but the Arden theorem gives only one of the solutions: a. 9.56 The language L is equal to w11 + w13 , where w11 = ε + 0.0+ .1w11 + 1.w11 , w13 = (1 + 0.0+ .1)w13 + 0.0+ . 9.58 The generating function: g(z) =

z2 /4 . 1−z/2−z2 /4

Extracting the coefficient, we obtain Fn /2n .

9.60 The enumerator (generating function) is ϕ (a, b) =

(1+a+b)((a−b)2 +1) . 1−4ab+2a2 b2

9.61 The enumerator (generating function) is ϕ (a, b, c) = s(I − T )−1 · u =

9.63 The generating function 9.64 (a) 2;

(b) 3 × 2n−1 ;

1 1 2 1−z 1− 1 z 1−z

=

(1+a+b+c)((a−b−c)2 +1) . 1−4ab−4ac+2a2 (b+c)2

z2 1−z−z2

(c) r(r − 1)n−1 .

9.65 (a) The enumerator is ϕ (1) (z0 , z1 ) =

1 . 1−z0 −z21

(b) The numerator: ϕ2 (z) =

1−z . z5 −2z3 −2z+1


768

∗ 9.67 Each word in L has a prefix and a suffix, generated by (ε + 00+ )1 and (ε + 00+ ), respectively.

9.72 The enumerator is G(z) =

pr zr (1−pz) . 1−z+qpr zr+1

9.80 For fair coin, the probability to see m heads or n tails is p(m, n) = is 14.

2m −1 2m +2n −2 .

The expected number of flips

9.81 The expected number of flips until either of these two runs happens for the first time is given by the following m n −pm −qn +1 formula: ppqnq+qp m −pm qn . 9.82 (a) 62, the enumerator is G(z) = p3 q2 z5 . (1−pz)(1−qz)−pqz2 −p2 q3 z5

enumerator is G(z) =

p5 z5 . 1−qz−pqz2 −p2 qz3 −p3 qz4 −p4 qz5

(c) 32, the enumerator is G(z) =

p4 q z5 . (1−pz)(1−qz−pqz2 −p2 qz3 −p3 qz4 )

9.83 (a) The expected value is

(1+p)(1+p2 ) 1−9p−9p2 −9p3

= 1111,

(b) 42, the enumerator is G(z) =

p4 q z5 . (1−pz)(1−qz−pqz2 −p2 qz3 )−p3 q2 z5

when p = 0.1.

(b)

1−pr 10qpr

=

(d) 34, the

10r −1 9 .

9.84 Two values: p = 1 and p ≈ 0.4367941684. 9.85 1/2 for fair coins, and 9.108 µ =

1+q pq ,

σ2 =

Pr[THH] =

in general case.

4+6p2 −p3 −8p . p2 q2

pq , q2 +p2 q pq ; p2 +pq2

9.113 (a) Pr[HHT] = p2 1−pq ,

q 1+q

q2 (b) 1−pq ; p2 q Pr[THH] = q2 +p2 q ,

Pr[HTT] = (d)

pq(1+q) q3 1+p , Pr[TTH] = p ; p2 q . Pr[HHT] = (1−pq)(1−q−pq)

Pr[HTT] =

(c) Pr[TTH] =

Chapter 10 10.1 By choosing f = g, we have f (n) ∈ / o(g(n)). 10.2 (a) f (n)/g(n) = [g(n) + o(g(n))]/g(n) = 1 + o(g(n))/g(n). O(1)) is identical with that of f (n) = O(g(n)).

(b) Since the meaning of f (n) = g(n)(1 +

10.3 By definition, f (n) ∈ Θ(g(n)) implies both f (n) ∈ O(g(n)) and f (n) ∈ Ω(g(n)). −2/3

−4/3

10.10 ln(n + n2/3 ) = ln n + ln(1 + n−1/3 ) = ln n + n−1/3 − n 2 + n1 − n 4 ± · · · . 2 10.12 ln(1 + ax + bx2 ) = ax + 2b−a 2 x +

2 4 2 a(a2 −3b) 3 x + 4a b−a4 −2b x4 + O(x5 ). 3

10.18 (a) At s = n/2, the binomial coefficients change relatively slowly. 10.20 c = 1/ ln 16. 2d −d 2 10.27 (a) T = an0 ed1 1 + 22n 1 + O(n−2 ) .

(b)

n n s t n−s . s−t ∼ s

(b) ab00 1 + b0 d1 +bn 1 ln a0 + O(n−2 ) .

10.28 Consider the minimal asymptotic approximation for Hn , ln n + γ , since the next term is 1/(2n), and when 9 . n is replaced by Cn . This yields HCn ∼ 2n ln 2 − 32 ln n + γ − 1/2 ln π − 8n


769 (b) By the induction, prove the recurrence βn,t+1 = βn,t dtn .

10.31 (a) Tt (x) = 1/[(1 − x)(1 − xd2 ) · · · (1 − xdt )]. 10.35 It follows from Exercise 10.34.

10.41 (a) ∆ f (n) = (2n + 1)2 , ∆g(n) = 4(n + 1)2 , hence the limit is 1. (b) ∆ f (n) = (n + 1) p and ∆g(n) = p p+1−1 p+1 . (c) ∆ f (n) = (2n+1) p = 2 p n p +O(n p−1 ) and ∆g(n) = (n+1) p+1 − ∑ j=0 j n j , hence the limit is 1 p+1 n p+1 = p n p + O(n p−1 ), hence the limit is 2 p /(p + 1). 10.48 (a) P(n, m) → 0.

(b) 1/2C.

(c) limn→∞ P(n,Cnα ) =

0 1

if 1 < α < 2, if α > 2.

k

+ 21 + . . . , where a−1 is the smallest root of 1 − z + qpk zk+1 = 10.54 E [max {X1 , . . . , Xn }] ∼ − loga n − γ +1+kqp ln a 0. (−1)n n! (−1)k Hn 1 1 1 R −ε +i∞ (−1)n n!dz 1 n + · · · . (b) − − ∼ + , 10.56 (a) Sn ∼ n2 − 21 + ln ∑ ∑ a n+1 n+1 ak az k ε −i∞ − k=2 3 2π i 2 e +1 a e +1 z (e +1) z z=zk

where zk = π (1 + 2k)i/a.

10.64 c = p/12. For the sum ∑nk=m k p , the asymptotic formula is the same. √ 1 − 1 + 1 ±···. 10.75 ln n! = (n + 21 ) ln n − n + ln 2π + 12n 360n3 1260n5 1 . 10.77 1/2 + n + 12n

2 (b) n! nt + t 3 /6n2 et /2n .

10.78 (a) t 3 ∈ o(n2 ).

1 5/2 + 5 (2n)3/2 + 5 (2n)1/2 + 7517 − 5 (2n)−1/2 + O(n−3/2 ). 2 (2n) 8 8 7168 128 √ 23 2 25 + 672 + 384 (2n)−1/2 + O(n−3/2 ). (c) Does not work.

10.80 (a) S2n = Tn −Un = 1 2

(2n)5/2 + 85

13 (2n)3/2 + 336

10.81 S2n (t) =

1 2

√ −1/2 1 + 4n + O ln 1 + 1 + t 2 − 21 lnt − 21 1 + t 2

10.82 S2n = ∑k>1 (−1)k−1

1 n2

(b) S2n =

.

p p ln (2n) + ln4n2n + 485n2 p ln p−1 (2n) − ln p (2n) + O . 3 n

ln p k k

√ ′ 10.84 Sn = n( 12 + 31 + 51 . . .); the rate of convergence is at least of the order 1/ n ln n. 10.85 c is in the range 2.5–2.6. 10.99 T (n) ∈ O(n lg n). 10.100 (a) ∑nk=2

1 k p (ln k)q

=Θ

n1−p (ln n)q

21−p − (ln 2)q .

(b) ∑nk=1 k p qk ∈ Θ (n p qn − q).

10.103 T (n) ∈ Θ (n log n). i 3 h 2 x(x+1) 1 10.109 yn = n−x c0 1 − 2n + nk22 + · · · , where k2 = 4x x2 + 5x3 + 3x 2 +3 . 10.110

4n 1 2n √ n+1 n = n nπ

i h 295911 1 1155 36939 9 + 2145 . 1 − 8n 7 n2 − 210 n3 + 215 n4 − 218 n5 + O n6

10.111 Dn = nn e−n n3/2 c0 1 + 43 + · · · . 10.112 (a) yn =

n 2n c0 e n12

1 + cn1 + nc22 + nc33 + · · · , where c1 = 401/42, and so on.

(b) yn = c0 n, obviously.


770 10.113 yn = 16n n−3/2 c0 + cn1 + · · · .

10.114 2 6 yn 6 2 + 4n

=⇒

limn→∞ yn = 2.

1 +O 10.115 Nn = nn e−n n1/2 c0 1 + 12n

√ 10.116 (a) yn = 2n+1 n! e − 2n − 1 −

1 n2

1 n+1

.

1 1 − 2(n+1)(n+2) − 4(n+1)(n+2)(n+3) −···.

(b)

√ 2.

10.117 an ∼ b0 1 − 1n + 2n12 + 3n13 + · · · .

10.133 (a) The weak law of large numbers holds for 0 < a 6 1 and the strong law is valid for 0 < a < 1. (b) The weak law of large numbers holds for 0 < a < 1/2 and the strong law is valid for a < −1/2.

10.134 X/m → e−λ

in probability.

def

10.135 Let Sn = ∑nk=1 Xk , then Sn /(n ln n) → 1 in probability. 10.138 Since E[B(n, p)] = np, V [B(n, p)] = npq, and E|Xk − E[Xk ]|3 = pq(p2 + q2 ), all conditions of the Lyapunov theorem hold. 10.144 ≈ 0.2358;

≈ 0.9973.

10.145 n = ⌈27055.4336p(1 − p)⌉. 10.147 (a) No such values. p 10.148 E(n) ∼ nπ /2. (n)

10.149 p0 ≈

q

(b) Any value of a.

3 4π n .

10.154 Sum E[R] = ∑k>0 2k(−1)k−1

1/2 k .

Chapter 11 11.19 a−k−n

nk−2 (k−2)!

11.26 fn (x) ∼ 11.31 f2n ∼

(au1 + ku0 /2)

1 , (x+1) z0n+1

4n √ , 2 nπ

z0 = ln

x+1 x

as n → ∞.

,

and when index is odd, the coefficient is zero.

11.34 hn ∼ g(2) 2−n−1 (Hn − ln 2). 5 2 n z−1 − 3z−1 , we have an ∼ 5 × 3 . 2 2190 38 6z −8z +32z−10 = + 539(3z+4) 11(2z−1) 6z4 −7z3 −8z2 +13z−4

11.38 (a) Since the generating function is (b)

Since the generating function is

n an ∼ − 38 11 2 .

3

−

102 49(z−1)

+

20 , 7(z−1)2

we have

1 1 − 2(2z+1) , we have an ∼ − 21 [2n + (−2)n ] or an ∼ −22k if (c) Since the generating function is 4z21−1 = 2(2z−1) n = 2k and zero otherwise. n 1 1 1 5 . (d) Since the generating function is 15z4−2z 2 +4z−4 = 5z−2 − 3z+2 , we have an ∼ − 2 2

Answers/Hints to Selected Problems (e) Since the generating function is (f) Since the generating function is 11.43 fn = 11.47

B2n (2n)!

q

5 π

771

1 1 1+5z = 2z+3 + 3z−2 , we have an 6z2 +5z−6 2z = 1 + 1 , we have a ∼ − 1 n z+2 z−2 2 z2 −4

21 + O(n−2 ) . 4n−1 n−3/2 1 + 40 n

n ∼ − 21 23 . n n − 21 − 21 21 .

= − (−42π 2 )n − (−162π 2 )n + o (4π )−2n .

11.54 E(m) =

q

nπ 2

1 + 32 + 12

q

π 2

n−1/2 + · · · .

11.58 cn ∼ c An n2(α2 −β2 ) . m l√ n ln 2 6 1. 11.67 T (n) −

p 11.70 β (p) = 1/ 2π p(1 − p). p+1 p .

11.72 ln

Chapter 12 12.26 Fn+1 , the (n + 1)-th Fibonacci number. ⌊n/2⌋ 4n−2 j 1 . (n−2 j)! j!

2

12.30 ex = ∑n>0 cn (x − 2)n , where cn = e4 ∑ j=0 12.33 (1 − x)−2 =

d dx

(1 − x)−1 = ∑n>0 (n + 1) xn ,

1 (1−x)3

=

1 2

∑n>0 (n + 1)(n + 2) xn .

12.37 ℜ f (z) = e−2xy cos(x2 − y2 ) and ℑ f (z) = e−2xy sin(x2 − y2 ). 12.62 S(z) =

√1 √ . 1−z 1−z−4cz

12.67 E2 (n, n) = [zn ](1 − 2z − 3z2 )−1/2 . 12.68 dn = [xn ](1 + x)n (2 + x)n = ∑k 12.73 A1 f (x) = 2x f (x) − f ′ (x),

n2 k k 2 .

A2 f (x) = 2x f (x) − f ′ (x).

12.76 (a) A1 − A2 = −2xI, where I is the identity operator. (2 − x2 ) f ′ (x) + x f ′′ (x). (d) B1 − B2 = − x12 I.

(b) (A1 − A2 ) f (x) ≡ 0.

(c) (A1 − A2 ) f (x) =

Bibliography [1] Abramowitz, Milton and Stegun, Irene A., Handbook of Mathematical Functions. Dover, New York, 1974. [2] Adel’son-Vel’skii, G. M. and Landis, E. M., An algorithm for the organization of information, Soviet Mathematics Doklady, 3, 1259 – 1263, 1962. [3] Aldous, David J. Probability Approximations via the Poisson Clumping Heuristic. Springer-Verlag, New York, 1989. [4] Aldous, D. and Diaconis, P., Shuffling cards and stopping times, The American Mathematical Monthly, 93, No. 5, 333 – 348, 1986.

[5] Aldous,, D., Flannery,, B., and Palacios, J. L., Two applications of urn processes — the fringe analysis of search trees and the simulation of quasi-stationary distributions of Markov chains. Probability in Engineering and and Informational Sciences, 2, 293 – 307, 1988. [6] Andrews, George E., The Theory of Partitions. Encyclopedia of Mathematics and its Applications, Vol. 2. Addison-Wesley, Reading, MA, 1976. [7] Apostol, Tom M., An elementary view of Euler’s summation formula, The American Mathematical Monthly, 106, No. 5, 409 – 418, 1999. [8] Apostol, Tom M. and Mnatsakanian, Mamikon A., Surprisingly accurate rational approximations, Mathematics Magazine, 75, No. 4, 307–310, 2002. [9] Arden, D. N., Delayed Logic and Finite State Machine, In Theory of Computing Machine Design, 1 – 33, University of Michigan Press, Ann Arbor, 1960. [10] Atallah, Mikhail J., (Editor), Algorithms and Theory of Computation Handbook, CRC Press, Boca Raton, FL, 1998. [11] Avidon, Michael, Mabry, Richard, and Sisson, Paul, Enumerating row arrangements of three species, Mathematics Magazine, 74, No. 2, 130 – 134, 2001. 773

774

Bibliography

[12] Bailey, David H., Borwein, Jonathan M., and Girgensohn, Roland, Experimental evaluation of Euler sums, Experimental Mathematics, 3, No. 1 17 – 30 (1994) [13] Bailey, Herb, Monty Hall uses a mixed strategy, Mathematics Magazine, 73, No. 2, 135 – 141 (2000). [14] Bender, Edward A., Asymptotic methods in enumeration, SIAM Review, 16, No. 4, 485 – 515, 1974. [15] Bender, Edward A. and Williamson, S. Gill, Foundations of Applied Combinatorics, Dover, New York, 2007. [16] Bentley, J. L., The whys and wherefores of algorithm design. In Proceedings of the 11th International Conference on Computer Science and Statistics: The Interface, 1978. [17] Berry, Donald A., Statistics, a Bayesian Perspective. Duxbury Press, Pacific Grove, CA, 1995. [18] Birkhoff, G. D. and Trjitzinsky, W. J., Analytic theory of singular difference equations, Acta Mathematica, 60, No. 1-2, 1 – 89, 1932. [19] Blom, Gunnar and Thorburn, Daniel, How many random digits are required until given sequences are obtained?, Journal of Applied Probability, 19, 518 – 531, 1982. [20] Boneh, Arnon and Hofri, M., The coupon-collector problem revisited—a survey of engineering problems and computational methods, Stochastic Models, 13, No. 1, 39–66, 1997. [21] Brawner, J. N., Dinner, dancing, and tennis, anyone? Mathematics Magazine, 74, No. 1, 29 – 36, 2000. [22] Borwein, Jonathan M., Borwein, Peter B. and Dilcher, K., Pi, Euler numbers, and asymptotic expansions, The American Mathematical Monthly, 96, No. 8 681 – 687 (1989). [23] Borwein, Jonathan M., Calkin, Neil J. and Manna, Dante, Euler-Boole summation revisited, The American Mathematical Monthly, 116 No. 5, 387 – 412 (2009) [24] de Bruijn, N. G., Asymptotic Methods in Analysis. Dover, New York, 1981. [25] Callan, David, Pair them up! A visual approach to the Chung-Feller theorem, The College Mathematics Journal, 26, No. 3, 196 – 198, 1995. [26] Charalambides, C. A., Enumerative Combinatorics. CRC Press, Boca Raton, FL, 2002. [27] Chebyshev, P. L., Sur deux théorèmes relatifs aux probabilités, Acta Mathematica, 14, 305 – 315, 1890.

Bibliography

775

[28] Chung, Kai L. and Feller, W., On fluctuations in coin tossing. Proceedings of the National Acadeny of Sciences of the United States of America, 35, 605 – 608, 1949. [29] Cohn, R. M., Difference Algebra. Interscience, New York, 1965. [30] Comtet, L., Advanced Combinatorics. Kluwer, Dordrecht, Boston, 1974. [31] Coppersmith, Don and Winograd, Shmuel, Matrix multiplication via arithmetic prograssions, Journal of Symbolic Computation, 9, Issue 3, 251 – 280, 1990. [32] Cormen, T. H., Leiserson, C. E., Rivest, R. L., and Stein, C., Introduction to Algorithms. 2nd Ed., MIT Press, Cambridge, MA, 2001. [33] Darling, Donald A., Accumulating success, The American Mathematical Monthly, 105, No. 6, 561 – 562, 1998. [34] Donaghey, R. and Shapiro, L. W., Motzkin numbers, Jornal of Combinatorial Theory, Series A, 23, Issue 3, 291 – 301, 1977. [35] Dwass, Meyer, Probability and statistics: an undergraduate course. New York, W. A. Benjamin, 1970 [36] Eggleton, R. and Guy, R., Catalan strikes again! How likely is a function to be convex? Mathematics Magazine, 61, No. 4, 211 – 219, 1988. [37] Egorychev, G. P., Integral Representation and the Computation of Combinatorial Sums. American Mathematical Society, Providence, RI, 1984. [38] Erös, Paul and Rényi, Alphred, On a new law of large numbers, Journal d’Analyse mathematique, 23, No. 1, 103 – 111, 1970. [39] Farmer, Jeff and Leth, Steven, An asymptotic formula for powers of binomial coefficients, The Mathematical Gazette, 385 – 391, 2007. ¨ [40] Feller, William, Uber den zentralen Grenzwertsatz der Wahrscheinlichkeitsrechnung, Mathematische Zeitschrift, 40, 521 – 559, 1935. [41] Feller, William, An Introduction to Probability Theory and its Applications. 3rd Ed., John Wiley & Sons, New York, 1968. [42] Finch, Steven R., Mathematical Constants (Encyclopedia of Mathematics and its Applications), Cambridge University Press, Cambridge, UK, 2003. [43] Flajolet, P. and Golin, M., Mellin transforms and asymptotics, Acta Informatica, 31, 673 – 696, 1994. [44] Flajolet, P. and Odlyzko, A., Singularity analysis of generating functions, SIAM Journal on Discrete Mathematics, 3, No. 2, 216 – 249, 1990.

776

Bibliography

[45] Flajolet, P. and Sedgewick, R., Mellin transforms and asymptotics: Finite differences and Rice’s integrals, Theoretical Computer Science, 144, Issue 1-2, 101 – 124, 1995. [46] Flajolet, P. and Sedgewick, R., Analytic Combinatorics, Cambridge University Press, Cambridge, UK, 2009. [47] Flajolet, P., Szpankowski, W., and Vallee, B., Hidden word statistics, Journal of the ACM (Association for Computing Machinery), 53, No. 1, 147 – 183, 2006. [48] Gelenbe, Erol and Mitrani, Israel, Analysis and Synthesis of Computer Systems, Academic Press, London, 1980. [49] Gonnet, Gaston H., Notes on the derivation of asymptotic expressions from summations, Information Processing Letters, 7, No. 4, 165 – 169, 1978. [50] Gordon, H., Discrete Probability, Springer, New York, 1997. [51] Gosper, R. William Jr., Decision procedure for indefinite hypergeometric summation, Proceedings of the National Academy of Sciences, USA, 75, No. 1, 40 – 42, 1978. [52] Gould, H. W., Some generalizations of Vandermonde’s convolution, The American Mathematical Monthly, 86, No. 2, 84 – 91, 1956. [53] Gould, H. W., Combinatorial Identities. A Standardized Set of Tables Listing 500 Binomial Coefficient Summations. 2nd Ed., Self-published, Morgantown, WV, 1972. [54] Gould, H. W. and Squire, William, Maclaurin’s second formula and its generalization, The American Mathematical Monthly, 70, No. 1, 44 – 62, 1963. [55] Goulden, I. P. and Jackson, D. M., Combinatorial Enumeration, John Wiley & Sons, New York, 1983. [56] Graham, D. L., Knuth, D. E., and Patashnik, O., Concrete Mathematics, A Foundation for Computer Science, 2nd Ed., Addison-Wesley, Menlo Park, CA, 1994. [57] Grabner, Peter and Prodinger, Helmut, The Fibonacci killer, Fibonacci Quarterly, 32, No. 5, 389 - 394, 1994. [58] Greene, D. H. and Knuth, D. E., Mathematics for the Analysis of Algorithms, 2nd Ed., Birkhäuser, Boston, 1982. [59] Gross, Donald and Harris, Carl M., Fundamentals of Queuing Theory, 3rd Ed., John Wiley & Sons, New York, 1998. [60] Guy, Richard K., The second strong law of small numbers, Mathematics Magazine, 63, No. 1, 3 – 20, 1990. [61] Harary, Frank, Graph Theory. Addison-Wesley, Reading, MA, 1969.

Bibliography

777

[62] Henrici, Peter, Applied and Computational Complex Analysis. John Wiley & Sons, New York, 1974. [63] Henze, Norbert, The distribution of spaces on lottery tickets, The Fibonacci Quarterly, 33, 426 – 431, 1995. [64] Hofri, Micha, Analysis of Algorithms: Computational Methods & Mathematical Tools. Oxford University Press, New York, 1995. [65] Hopcroft, John E., Motwasni, Rajeev, and Ullman, Jeffrey D., Introduction to Automata Theory, Languages and Computation. 3rd Ed., Addison Wesley, Boston, MA, 2007. [66] Howard, F. T., The number of multinomial coefficients divisible by a fixed power of a prime, Pacific Journal of Mathematics, 50, 99 – 108, 1974. [67] Huang, Xun-Cheng, A discrete l’Hôpital’s rule, The College Mathematics Journal, 19, No. 4, 321 – 329, 1988. [68] Jewett, Robert I. and Ross, Kenneth A., Random walks on Z, The College Mathematics Journal, 19, No. 4, 330 – 342, 1988. [69] Johnson, Norman L., Kotz, Samuel, and Kemp, Adrienne W., Univariate Discrete Distributions. 2nd Ed., John Wiley & Sons, New York, 1992. [70] Jones, W. B. and Thron, W. J., Continued Fractions: Analytic Theory and Applications, Encyclopedia of Mathematics and its Applications, 11, Addison-Wesley, Reading, MA, 1980. [71] Karr, M., Summation in finite terms, Journal of the ACM (Association for Computing Machinery), 28, 305 – 350, 1981. [72] Kaucky, Josef, Combinatorial Identities. Veda, Bratislava, 1975. [73] Khintchine, A. Y., Limit Theorems for Sums of Independent Random Variables, Gonti, Moscow, Leningrad, 1938. [74] Kirchenhofer, P., Prodinger, H., and Martinez, C., Analysis of Hoare’s F IND algorithm with median-of-three partition, Random Structures and Algorithms, 10, 143–156, 1997. [75] Knuth, Donald E., Euler’s constant to 1271 places, Mathematics of Computation, 16, 275 – 281, 1962, [76] Knuth, Donald E., Mathematical analysis of algorithms, Proceedings of IFIP Congress (International Federation of Information Processing), 19 – 27, 1971. [77] Knuth, Donald E., The Art of Computer Programming, Vol. I: Fundamental Algorithms. 3rd Ed., Addison-Wesley, Reading, MA, 1997.

778

Bibliography

[78] Knuth, Donald E., The Art of Computer Programming, Vol. II: Seminumerical Algorithms. 3rd Ed., Addison-Wesley, Reading, MA, 1998. [79] Knuth, Donald E., The Art of Computer Programming, Vol. III: Sorting and Searching. 2nd Ed., Addison-Wesley, Reading, MA, 1998. [80] Knuth, Donald E., Two notes on notation, The American Mathematical Monthly, 99, No. 5, 403 – 422, 1992. [81] Koshy, Thomas, Fibonacci and Lucas Numbers with Applications. John Wiley & Sons, New York, 2001. ¨ [82] Kummer, E. E., Uber die Ergänzungssätze zu den allgemeinen Reciprocitätsgesetzen, Journal für die reine und angewandte Mathematik, 44, 93 – 146, 1852. [83] Lalley, Steven P., Rifle shuffles and their associated dynamical systems, Journal of Theoretical Probability, 12, No. 4, 903 – 932, 1999. [84] Lampret, Vito, The Euler-Maclaurin and Taylor formulas: twin, elementary derivations, Mathematics Magazine, 74, No. 2, 109 – 122, 2001. [85] Laplace, P. S., Thèorie Analytique des Probabilitès. Paris, 1812. [86] Lawler, Gregory F. and Coyle, Lester, N., Lectures on Contemporary Probability, The American Mathematical Society, Providence, RI, 1999. [87] Lehmer, D. H., On the maxima and minima of Bernoulli polynomials, The American Mathematical Monthly, 47, No. 8, 533 – 538, 1940. [88] Lesigne, Emmanuel, Heads or Tails. An Introduction to Limit Theorems in Probability, The American Mathematical Society, Providence, RI, 2005. [89] Lewis, Harry L. and Papadimitriou, Christos H., Elements of the Theory of Computation. 2nd Ed., Prentice Hall, Upper Saddle River, NJ, 1998. ¨ [90] Lindeberg, J. W. Uber das Exponentialgesetz in der Wahrscheinlichkeitsrechnung, Mathematische Zeitschrift, 15, 211 – 225, 1922. [91] Lindley, Dennis V., Introduction to Probability and Statistics from a Bayesian Viewpoint. Cambridge University Press, Cambridge, UK, 1969–1970. [92] Loève, M., Probability Theory, 4th Ed., Springer-Verlag, New York, 1977. [93] Lothaire, M., Combinatorics on Words. Encyclopedia of Mathematics and its Applications, Vol. 17. Addison-Wesley, Reading, MA, 1983. [94] Lovász, Lászlo, Combinatorial Problems and Exercises. North-Holland, Amsterdam, 1979.

Bibliography

779

[95] Lyapunov, A. M., Sur une proposition de la théorie des probabilités, Bull. de l’Acad Imp. des Sci St. Petersbourg (5), 13, No. 4, 359 - 386, 1900. [96] Mahmoud, Hosam M., Pólya Urn Models. CRC Press, Boca Raton, FL, 2009. [97] Markov, A., The law of large numbers and the method of least squares, Izvestiya Fiz.Mat. Obschestva Kazan Univer. (2), 8 110 – 129, 1899. [98] Merlini, D, Sprugnoli, R., and Verri, M. C., The method of coefficients, The American Mathematical Monthly, 114, No. 1, 40 – 57, 2007. [99] Mitzenmacher, Michael and Upfal, Eli, Probability and Computing, Randomized Algorithms and Probabilistic Analysis. Cambridge University Press, Cambridge, UK, 2005. [100] Moenck, R., On computing closed forms for summation. Proc. MACSYMA User’s Conf. 225 – 236, 1977. [101] Moon, John W., Topics on Tournaments. Holt, Rinehart, and Winston, New York, 1968. [102] Niven, Ivan, Formal Power Series, The American Mathematical Monthly, 76, No. 8, 871 – 889, 1969. [103] Odlyzko, Andrew M., Asymptotic enumeration methods, Chapter 22 in Handbook of Combinatorics (Editors: R. L. Graham, M. Grötschel, L. Lovász), Elsevier, Amsterdam, 1995. [104] Panholzer, Alois and Prodinger, Helmu, A generating functions approach for the analysis of grand averages for multiple QUICKSELECT, Random Structures and Algorithms, 13, No. 3–4, 189–209, 1998. [105] Pemantle, Robin and Wilson, Mark C., Twenty combinatorial problems as asymtotics derived from multivariate generating functions, SIAM Review, 50, No. 2, 199 – 272, 2008. [106] Pergola, E., Pinzani, R., Rinaldi, S., and Sulanke, R. A., A bijective approach to the area of generalized Motzkin paths, Advances in Applied Mathematics, 28, Issues 3-4, 580 – 591, 2002. [107] Petkovsek, Marko, Wilf, Herbert S., and Zeilberger, Doron, A = B. A.K. Peters, Wellesley, MA, 1996. [108] Poisson, Siméon-Denis, Mémorie sur le calcul numérique des intégrales défines, Mémoires de l’Académie Royal des Sciences de l’Institut de France, séries 2, 6, 571 – 602, 1823. [109] Pólia, G., Berechnung eines bestimmten Integrals, Mathematische Annalen, 74, 204 – 212, 1913.

780

Bibliography

[110] Pólya, G. and Read, Ronald C., Combinatorial Enumeration of Groups, Graphs, and Chemical Compounds. Springer-Verlag, New York, 1987. [111] Pozdnyakov, Vladimir and Kulldorf, Martin, Waiting times for patterns and a method of gambling teams, The American Mathematical Monthly, 113, No. 2, 134 – 143, 2006. [112] Renault, Marc, Four proofs of the ballot theorem, Mathematics Magazine, 80, No. 5, 345 – 352, 2007. [113] Rice, S. O. Uniform asymptotic expansions for saddle point integrals – Application to a probability distribution occurring in noise theory (Uniform asymptotic expansion series for saddle point integrals applied to probability distribution in noise interference problems), Bell System Technical Journal, 47, 1971 – 2013, 1968. [114] Riordan, John, An Introduction to Combinatorial Analysis. Princeton University Press, Princeton, NJ, 1980. [115] Riordan, John, Combinatorial Identities. John Wiley & Sons, New York, 1968. [116] Ross, Sheldon M., Introduction to Probability Models. Academic Press, New York, 1989. [117] Roura, Salvador, Improved master theorems for divide-and-conquer recurrences, Journal of the ACM (Association for Computing Machinery), 48, No. 2, 170 – 205, 2001. [118] Rudin, Walter, Principles of Mathematical Analysis. 2nd Ed., McGraw-Hill, New York, 1964. [119] Sanchis, Gabriela R., Swapping hats: a generalization of Montmort’s problem, Mathematics Magazine, 71, No. 1, 53 – 57, 1998. [120] Sedgewick, Robert, Quicksort, Garland Publishing, New York, London, 1980. [121] Sedgewick, Robert, Algorithms in C, Addison-Wesley, Reading, MA, 1990. [122] Sedgewick, Robert and Flajolet, Philippe, An Introduction to the Analysis of Algorithms, Addison-Wesley, Reading, MA, 1996. [123] Shapiro, Lou and Hamilton, Wallace, The Catalan numbers visit the world series, Mathematics Magazine, 66, No. 1, 20 – 22, 1993. [124] Shiryayev, Albert N. Probability. Springer-Verlag, New York, 1996. [125] Stanley, Richard P., Enumerative Combinatorics, Vol. 1. Cambridge University Press, Cambridge, UK, 1997. [126] Steele, J. Michael, The Cauchy-Schwarz Master Class: An Introduction to the Art of Mathematical Inequalities, Cambridge University Press and the Mathematical Association of America, Cambridge, UK and Washington DC, 2004.

Bibliography

781

[127] Strassen V., Gaussian elimination is not optimal, Numerische Mathematik, 13, 354– 356, 1969. [128] Sudkamp, Thomas A., Languages and Machines, An Introduction to the Theory of Computer Science. 3rd Ed., Addison-Wesley, Reading, MA, 2006. [129] Szpankowski, Wojciech, Average Case Analysis of Algorithms on Sequences. John Wiley & Sons, New York, 2001. [130] Takács, Lajos, On the method of inclusion and exclusion, Journal of The American Statistical Association, 62, 102–113, 1967. [131] Tucker, Alan, Applied Combinatorics. 4th Ed., John Wiley & Sons, New York, 2002. [132] Tweddle, Ian, Approximating n!. Historical origins and error analysis, The American Journal of Physics, 52, Issue 6, 487 – 488, 1984. [133] Usiskin, Z., Max-Min probabilities in the voting paradox, Annals of Mathematical Statistics, 54, 857 – 862, 1964. [134] Verma, Rakesh M., A general method and a master theorem for divide-and-conquer recurrences with applications, Journal of Algorithms, 16, 67 – 79, 1994. [135] Whittaker, Edmund T. and Watson, George N., Course of Modern Analysis, 4th Ed., Cambridge University Press, Cambridge, UK, 1927. [136] Widder, David Vernon, The Laplace Transform, Princeton: Princeton University Press; London: H. Milford, Oxford University Press: Oxford, UK, 1946. [137] Wilf, Herbert, Generatingfunctionology. 2nd Ed., A.K. Peters, Wellesley, MA, 2006. [138] Wimmer, Gejza and Altman, Gabriel, Tresaurus of Univariate Discrete Probability Distributions. STAMM Verlag GmbH, 1999. [139] Wimp, Jet and Zeilberger, Doron, Resurrecting the Asymptotics of Linear Recurrences, Journal of Mathematical Analysis and Applications, 111, Issue 1, 162 – 176, 1985. [140] Winograd, Shmuel, On the parallel evaluation of certain arithmetic expressions, Journal of the ACM (Association for Computing Machinery), 22, No. 4, 477 – 492, 1975. [141] Wong, R., Asymptotic Approximations of Integrals, Academic Press, Boston, MA, 1989. [142] Worpitzky, J., Studien u¨ ber die Bernoullischen und Eulerschen Zahlen, Journal für die reine und angewandte Mathematik, 94, 202 – 232, 1883. [143] Yaglom, A. M. and Yaglom, I. M., Challenging Mathematical Problems with Elementary Solutions. Vol. 1, Dover, New York, 1987.

Index !!, see alternating factorial n , see E(n, k) k |=, 365 Abel convergence theorem, 668 Abel identity, 75, 350, 702 Abel theorem, 740 Abel’s summation formula, 721 Abel, Niels, 75, 721 Abelian labeled product, 405 Abramowitz, Milton, 581 Absolute convergence, 26 Absolute error, 548 Absorbing state, 528 Absorption time (of a Markov chain), 530 Accessible state, 528 Accuracy, 247 Ackerman function, 215 Addition formula, 59, 699 Adel’son-Vel’skii, G. M., 214 Adjacency in a sequence, 466 Admissible operations EGF, 279, 404, 431 OGF, 274, 381 Admissible set, 356 Algebra of events, 84 Algebraic singularity, 646 Algorithm Archimedes, 223 binary search, 5, 224 bisection, 237 divide-and-conquer, 223, 225, 227, 229 false position, 237 fast Fourier transform, 603 find, 254

interpolation search, 238 Karatsuba, 223 mergesort, 227 Newton-Raphson, 238 quickselect, 254 quicksort, 229 random permutation efficient, 204 inefficient, 203 regula falsi, 237 Steffensen, 239 tri-diagonal marching, 540 Alphabet, 438, 489, 490 Alternating factorial, 43 Alternating permutation, 431, 747 Alternating series, 549, 589 Analytic function, 671 Andre, Desire, 52 Andrews, George, 205, 393 Aperiodic state, 529 Apostol, Tom, 248, 580 Appell polynomials, 281, 706 Approximant, 733 Approximation Padé, 242, 732 rational, 242, 732 Arc, see contour Archimedes algorithm, 223 Archimedes’s estimate of π , 247 Arden lemma, 504 Arithmetic mean, 556, 558 Arithmetic series, 15 Ascent in permutations, 479 Asymptotic expansion, 550, 551

783

784

generalized, 551 Asymptotic notation, 546 O, 546, 547 o, 546 Ω, 547 ∼, 546 Θ, 547 Asymptotic sequence, 551 Asymptotic series, 550 Asymptotics, 545 Catalan numbers, 631 entire GF, 649–658 Motzkin numbers, 631 scale, 551 Atom, 359, 375 Average, see expectation Average, mean, 18, 132 AVL tree, 214 Backward difference, 705 Ballot numbers, 60, 257, 322, 748 Ballot paths, 60 Ballot problem, 60, 623 Banach matchbox problem, 98, 112 Banach, Stefan, 98 Bayes little theorem, 170 Bayes theorem, 170 Bayes, Thomas, 170 Beckenbach, Edwin, 18 Bell numbers, 314, 443, 708, 744, 750 Bell polynomials, 722 Bell, Eric, 314 Bellman, Richard, 18 Bender, Edward, 401, 514 Bentley, Jon L., 3 Bernoulli Daniel, 109 Jakob, 136 Bernoulli distribution, 136 Bernoulli numbers, 580, 581, 647, 706, 710, 743, 750 asymptotics, 648

Index Bernoulli polynomials, 580, 581, 706, 710, 743, 750 Bernoulli random variable, 136, 337 Bernstein polynomials, 136, 659 Berry, Donald, 170 Bertrand, Joseph, 60 Bessel function, 736 Beta function, 570, 694, 726 Bienaymé, Irénée-Jules, 188 Bijection, 42, 53, 426 tree-permutation, 433 Bin enumerators, 442 Binary search, 5, 224, 300 Binary search tree, 234, 235, 346 balanced, 559 internal path length, 234 leaf count, 234, 341, 343, 559 parameters, 234 statistics, 234, 235 Binary tree, 17, 214, 235, 300, 304, 424 BST statistics, 235, 341, 343, 346 Catalan statistics, 235, 341, 343, 346 counting, 299 depth, 17 diameter, 235 extrema of parameters, 346 full nodes count, 346 height, 17, 235 internal path length, 346 leaf count, 346 search tree statistics, 235 Binomial coefficients, 48, 56–70, 699 addition formula, 59, 699 asymptotics, 77, 553 central, 743 inverse relations, 64, 683, 704 negating the upper argument, 61, 699 summations, 75, 702 symmetry, 58 Binomial convolution, 279, 309 Binomial distribution, 136, 337 asymptotics, 556

Index Binomial generating function, 280, 283 Binomial moment, 468 Binomial parameters, 137 Binomial random variable, 136, 337 Binomial sequence, 279 Binomial theorem, 48, 75 Birkhoff, George, 604, 608 Birthday problem, 90, 101, 108, 402, 416, 554 Bisection method, 237, 238 Bivariate ordinary generating function, 283 Bivariate recurrence, 251, 319 Bonferroni inequalities, 460, 729 Boole inequality, 729 Boole summation formula, 590 Boolean function, 460 Boolean variable, 460 Borel-Cantelli theorem, 159 Bose-Einstein statistics, 420, 441 Boundary point, 678 Box product, 431, 432 Branch, 663, 665 Brouncker, William, 246 BSF, see Boole summation formula BST, see binary search tree Bunyakovskii inequality, 101 Canonical denominator, 245, 733 numerator, 245, 733 Canonical form of the sum, 72 Cardinality, 12, 80 Cardinality formula, 81 Cartesian product, 360 Cassini identity, 16, 716 Catalan binary tree, 346 Catalan binary tree statistics, 235 Catalan constant, 736 Catalan generating function, 322 Catalan numbers, 60, 284, 300, 320, 401, 426, 608, 743, 751 Catalan statistics, 235 Catalan tree, 426

785 Catalan, Eugene, 60 Catenation, 361, 490 Cauchy binomial formula, 76 Cauchy convergence condition, 679 Cauchy formula, 649 Cauchy principal value, 282, 600 Cauchy product, 673 Cauchy residue theorem, 682 Cauchy theorem, 649, 679 Cauchy, Augustin Louis, 600, 649 Cauchy-Schwarz inequality, 101 Cayley, Arthur, 430 Ceiling function, 28 Central binomial coefficients, 743 Central Delannoy numbers, 686 Central limit theorem, 159, 338, 616, 617 Central moment, 113 Central trinomial numbers, 320, 352 Certain event, 84 Cesàro sum, 668 Chain length, 485 Chapman-Kolmogorov equations, 528 Characteristic equation, 268, 607 Charalambides, Charalambos, 261 Chebyshev inequality, 188, 592 Chebyshev polynomial, 269, 306, 309, 734, 744, 749 Chebyshev, Pafnuty L., 188, 592, 610, 611 Chebyshev-Bienaymè inequality, 188 Chebyshev-Cantelli inequality, 188, 190 Chinese estimate of π , 247 Chu-Shih-Chieh, 56 Chung-Feller theorem, 627 Circular permutation, 54, 404, 614 Clark, Dean, 458 Closed set, 679 Closure, 362, 438, 492 enumerator, 493 Cluster, of rods, 380 Coefficient binomial, 56–70 asymptotics, 77

786

summation, 75 extraction, 287 kurtosis, 114 multinomial, 49 skewness, 114, 137, 153 trinomial, 49 Coin flips waiting times, 522 Coin patterns, waiting times, 522 Coincidence number, 258, 284, 730, 748 Coincidences (in permutation), 258 Collision resolution interval, 407 Combinations, 48 Combinations with repetition, 257 Combinatorics, 25, 355, 428 Communicating class, 528 Commutator, 688 Complement, 81 Complex numbers, 661 argument, 663 arithmetic, 661 Cartesian form, 662 conjugate, 662 modulus, 663 polar form, 662, 664 pure imaginary, 662 pure real, 662 trigonometric form, 663 Complexity of a problem, 2 Component of composition, 364 Compositional inverse, 289 Compositions, 204, 738 component, 204 homogeneous, 364–374 inhomogeneous, 367–369 palindromic, 374 weak, 364, 392, 738 Compositions of integers, 364–381 Compound event, 84 Comtet, Louis, 463 Concatenation, 361, 490 Conditional PMF, 133 Conditional distribution, 129

Index Conditional expected value, 133 Conditional probability, 120, 133, 163 Conditional variance, 133 Conditionally dependent events, 123 Conditionally independent events, 123 Configuration enumeration, 444 Confluent hypergeometric series, 73 Constants Catalan, 736 Euler, 27, 714 Khinchin, 736 Pythagoras’, 736 Stieltjes, 696, 714 Continued fraction, 244, 732 approximant, 245 convergent, 245 improper, 244, 732 ordinary, 247 periodic, 736 proper, 244, 732 regular, 247 Contour, 678 Contour integral, 649, 679 Convergence of iterates linear, 238 order p, 238 Convergence of random variables almost everywhere, 158 almost sure, 158 in distribution, 159 in mean, 160 in probability, 158 mean square, 160 strong, 158 Convergence of series, 26 absolute, 26 Convergence test Abel, 668 Cauchy, 630 D’Alembert, 630 ratio, 630 root, 630

Index Convergent, 248, 733 Convolution, 66, 175, 275, 320, 322, 337 binomial, 279 density functions, 175 distribution functions, 175 mass functions, 175 nested, 176 power series, 673 Vandermonde, 67, 68, 70, 152 Convolution of uniform variates, 370 Cormen, Thomas, 424, 602 Correlation, 130 Cost (of algorithm), 2, 3 Cost model, 2 Coupled recurrence, 315, 318 Coupon-collector problem, 150, 471, 477– 479, 579 Covariance, 130 Critical range method, 563, 564 Cumulant generating function, 333 Cumulative distribution function, 95 Currency, 368 Cycle class, 412 Cycle numbers, see Stirling numbers, first kind, 261, 333, 409, 708 Cycles in permutations, 262, 408, 614 Darboux theorem, 646 Darling, Donald, 180 De Moivre formula, 664 De Moivre, Abraham, 77, 272, 423, 616, 664 De Moivre-Laplace theorem, 617 De Morgan laws, 81, 446, 452 De Morgan, Augustin, 81 Deck of cards, 93 Decreasing failure rate, 625 Delannoy numbers, 326, 686 Dependent events, 123 Derangement, 45, 65, 258, 469, 608, 708, 730, 744, 750 Descending tree, 431 Descent in permutations, 479

787 Deterministic finite automaton, 498 Deutsch, Emeric, 118, 310 DFA , see deterministic finite automaton Diameter (of a graph), 235 Difference equation, see recurrence, 208 first order, linear, 215 general solution, 210 homogeneous, 604 hypergeometric, 607 partial, 251, 319 particular solution, 210 vector, 217, 315 Difference operator, 38, 266, 292 Differential operator, 686 Digamma function, 36, 727 Digraph, 423 Diophantine equation, 364, 738 Dirichlet generating function, 281 Dirichlet, Lejeune, 587, 695 Discrete Cauchy distribution, 110 Discrete l’Hôpital rule, 561 Discrete random variable, 85 Dispersion, 113 Distribution Bernoulli, 136 binomial, 136, 556 Cauchy, discrete, 110 geometric, 141, 335 geometric, shifted, 144 hypergeometric, 151 limit, 557 linear, 122 multinomial, 140 negative-binomial, 148, 179, 192 negative-binomial, shifted, 149 normal, 157 Poisson, 154, 335, 556 as a limit, 557 standard normal, 157 triangular, 176 uniform, 97, 102, 336 Distribution function, see Probability

788

Complementary, 95 Divide-and-conquer algorithm, 223, 225, 227, 229 Divide-and-conquer recurrence, 221, 223, 225, 227, 229 Divisor number, 394, 747, 752 Domain, 679, 686 of a function, 678 Domain transformation, 222 Double factorial, 43 Drop in permutations, 479 Dyck path, 179, 627

ε , 363, 490 E(n, k), 257 Eccentricity of orbit, 353 Efron dice, 105, 106 Efron, Bradley, 105 EGF, 271, 277 Egorychev, Georgii Petrovich, 271, 327 Egyptian fraction representation, 18 Ehrenfest, Paul and Tatyana, 196 Empty string, 363, 490 Empty word, 358, 490 Engset random variable, 178 Entire function, 282, 630, 669 Enumeration Principle, 359 Enumerator, 355, 357 as GF, 357 definition, 356 exponential, 356 in PIE, 464, 465, 467 Equation Chapman-Kolmogorov, 528 characteristic, 268 Diophantine, 364, 738 Kepler, 352 word, 503 Eratosthenes, 591 Erdös, Paul, 13, 14, 613 Ergodic state, 529, 530 Error absolute, 548

Index relative, 548 Error function, 677, 729, 747 ESF, see Euler summation formula Estimate, consistent, 132 Estimate, unbiased, 132 Euclid, 56 Euclid algorithm, 209 Euclidean metric, 181 Euler constant, 27, 181, 587, 696, 714, 720, 727, 736 Euler formula, 663 Euler numbers, 714, 743, 750 Euler pentagonal number theorem, 398, 747 Euler polynomials, 590, 714, 743 Euler summation formula, 67, 256, 579– 589, 690, 691 Euler summation formula variant, 582 Euler theorem, 740 Euler transformation, 569, 591, 740 Euler’s identity, 75 Euler’s method (DEs), 255 Euler, Leonhard, 27, 67, 244, 245, 258, 272, 355, 396, 580, 661, 694, 707 Eulerian numbers, 54, 132, 482, 488, 712, 713, 748, 751 Eulerian polynomials, 713 Event, 84 Events algebra, 84 associated conditionally, 120 Example PIE via enumerators, 464 accumulating success, 180 alternating series, 589 amortization loan, 200 ballot problem, 60, 623 balls and urns, 123, 163 biased-to-fair coins, 147 binary search, 300 binary sequences, 386 binary summation tree, 301 binary tree, 300

Index binomial series, 673 birthday problem, 90, 108 boolean functions, 460 Borel transform, 293 bridge couple problem, 463 cards, 122 Cassini identity, 16 Catalan numbers, 320 chains in a power set, 485 comparing polynomials, 92 compositions, 204, 458 convolution, 338, 441 convolution of fair dice, 176 convolution of uniform distributions, 177 counting with substitution, 386 coupon-collector problem, 150, 477 cumulative die score, 639 derangement, 258 enumerator for Kleene closure, 493 Euclidean distance, 181 Fibonacci numbers EGF, 308 Fibonacci numbers OGF, 307 finite differences, 292 finite word generator, 192 flipping coins, 92, 97, 121, 138, 140, 149, 185, 206, 217, 517, 534 floating-point summation, 301 full occupancy, 464 gaseous diffusion, 196 grid paths, 51, 178, 323 harmonic numbers asymptotics, 586 hashing, 142, 179 hidden patterns, 381 induction proof, 21 insertion sort, 102 involutions, 411 Kepler equation, 352 Kummer, 51 limited selection, 375, 378 limited Smirnov problem, 386 linear search, 193

789 loaded dice, 338 log piles, 206 longest run, 613 lottery, 91, 121, 127 Markov chain, 533, 534, 539 matching problem, 65 max search, 190 merge sort, 600 money changing, 368 Monty Hall, 165 Newton’s factorial series, 282 non-transitive dice, 105 onvolution of uniform variates, 370 order statistics of die rolls, 185 partitions, 205 pattern matching, 197 permutations with singletons, 409 Poisson random variable, 156 power set, 357 probability generating function, 338 quicksort, 340 radius of convergence, 670 random integers, 166 random numbers, 185 random permutation, 203 random search, 149 random walk, 178, 302, 623 random walk on a graph, 533 reflecting/absorbing random walk, 539 regular expression, 497 regular expression identities, 497 regular language, 495 rolling dice, 82, 87, 114, 120, 124, 126, 441, 522, 523 sampling with replacement, 143 separated samples, 375, 466 Smirnov problem, 386 Smirnov sequences, 437, 438 square root, 57 St. Petersburg paradox, 109 sum of random numbers, 177, 180 taboo subwords, 483

790

Terquem problem, 378 tree parameters, 341 tri-diagonal matrix, 214 TV game, 165 two-men competition, 147, 334 uniform distribution, 115, 336 waiting time, 108, 212, 517, 521–523, 533, 534, 539 Exercise AVL tree, 214 backward difference, 39 ballot numbers, 257 balls and urns, 93, 111, 112, 123, 126, 144, 167–169, 197, 218, 220, 339, 457 Banach matchbox, 98, 112 Bell numbers, 314 binary insertion sort, 195 binary search, 225, 226, 323, 541 binary search tree, 234, 559 binary tree, 17, 304 binomial random variable, 137, 139, 338 birthday problem, 91, 108, 454, 473 bridge problem, 207 cards, 54, 93, 122, 192 Catalan numbers, 305 Chebyshev inequality, 188–192 coincidence number, 479 compositions, 374 convolution, 177, 178, 339 couples, 127 coupon-collector problem, 478, 579 cycles, 412 dancing problem, 260, 479 derangement, 279 dinner problem, 259 divide-and-conquer, 223 Egyptian fraction representation, 18 error-correction, 139 Euclidean distance, 182 expected value, 106, 111, 112, 114

Index Fermat’s little theorem, 53 Fibonacci numbers, 207, 212, 322 file system, 195 finite difference, 39, 41 flipping coins, 92, 98, 108, 123, 134, 137, 139, 145, 148, 149, 168, 169, 172, 206, 208, 213, 218, 380, 514, 519, 522, 526 flipping pairs of coins, 380 forward difference, 39 full-history recurrence, 309, 310, 314 Galileo problem, 88 Gates-Ballmer, 227 geometric random variable, 141, 144, 145, 186, 194 grid paths, 224, 325, 326 harmonic numbers, 41, 291 hashing, 143, 444 hypergeometric random variable, 152, 153 insertion sort, 126, 194 jailer’s paradox, 172 Josephus, 228 Karatsuba algorithm, 223 lie detector, 171 limited selection, 377, 379, 390 limits of distributions, 557 loaded dice, 176 local maximum, 434 log piles, 252, 313, 323 longest run, 208, 371 lottery, 128, 133, 187, 470, 659 matching problem, 469 matrix multiplication, 217 maxima in permutation, 118 merge sort, 228 min-cut problem, 169 moments, 114 Moore’s law, 220 multinomial random variable, 140 negative-binomial random variable, 149, 177, 192

Index non-transitive dice, 106 numbers multiplication, 223 occupancy problem, 459, 615 oriented trees, 427, 428 Padé approximation, 244, 246 pair-wise dependence, 126 partitions, 205 set, 413 pattern matching, 197 permutation, 93, 118, 192, 279 plane trees, 426 Poisson random variable, 155, 156 Pólya urn model, 110, 457 probability generating function, 337, 339 quickselect analysis, 254, 346 quicksort, 231 random graph, 126 random integers, 111, 126, 168, 169, 172, 186 random number generator, 145, 187, 526 random numbers, 99, 169, 187 random permutation, 112 random walk, 179, 628 reciprocal approximation, 241 red and black cards, 326 rolling dice, 122, 129, 134, 139, 140, 176, 186, 192, 337, 339 rolling dice to n-run, 542 rolling dice to 7, 168 root approximation, 238, 240–242, 250 roulette, 100 rounding errors in addition, 622 search, 194 separated samples, 377 on circle, 390 set partitions, 413 Shapiro, L. W., 305 singles, 108, 254 Skolem sets, 379 Smirnov’s sequences, 438

791 St. Petersburg paradox, 615 Stirling cycle numbers, 333 string generator, 193 sum of random numbers, 169, 177, 233, 437 sum of random variables, 177, 178 tournament, 20, 112 trees, 235, 425 tri-diagonal matrix, 214 truncated factorials, 206 two-men competition, 147 variegated strings, 464 volleyball, 147 waiting time, 169, 340, 526 word generator, 193 Worpitzky’s identity, 54 Expectation operator, 100 properties, 100 Expected value, 100 conditional, 133 Experiment, 82 Exponential generating function Bernoulli numbers, 690, 743 Bernoulli polynomials, 743 Exponential generating function (EGF), 277, 283 Exponential integral, 695 Extended transition function, 498 Exterior point, 678 Faà-di-Bruno formula, 723 Factorial, 43, 750 alternating, 43 double, 43 falling, 46, 206, 699 rising, 46, 72, 206, 699 Factorial generating function, 280, 283 Factorial moment, 113, 332 Failure rate, 625 Falling factorial, 46, 206, 699 Farey mean, 239 Farmer, Jeff, 621

792

Fast Fourier transform, 603 Feller, William, 339, 457, 616, 627 Fermat’s little theorem, 53 Fermat, Pierre de, 62 Fermi-Dirac statistics, 441 Fibonacci numbers, 15, 17, 246, 307–309, 313, 322, 328, 330, 715, 743 Fibonacci polynomial, 715 Fibonacci recurrence, 212 Fibonacci, Leonardo, 319, 715 Field of complex numbers, 661 Finite difference, 705, 724 Finite state automaton, 497 Finite state machine, 497 Finite word generator, 192 First passage time, 531 First-order differential equation, 312 Fixed point of permutation, 258, 469 Flajolet, Philippe, 355, 423, 431, 464, 599, 648 Floor function, 28 Formula Cauchy, 649 Chebyshev iteration scheme, 239 De Moivre, 290, 664 Euler, 663 Euler summation, 256 Jensen, 723 Newton-Raphson extended, 239 Rademacher, 739 Forward difference, 37, 705 Fractional part, 247 Free tree, 424 Fresnel integral, 651 FSA , see finite state automaton Full occupancy, 464 Full-history recurrence, 207, 208, 230, 233, 252, 301, 313, 314, 370 Function Ackerman, 215 analytic, 671, 679 arctangent, 734, 745

Index Bessel, 736 beta, 570, 694, 726 bijection, 53 boolean, 460 branch, 665 ceiling, 28 characteristic, 356 complex, 677–683 digamma, 36, 727 entire, 282, 630, 669 error, 677, 729, 747 exponential integral, 695 fixed point, 237 floor, 28 gamma, 570, 694, 725, 726 asymptotics, 726 recurrence, 201 holomorphic, 671, 679 hypergeometric, 72, 458, 705 incomplete beta, 695 incomplete exponential, 154, 155, 290, 419 incomplete gamma, 154, 695, 726 incomplete zeta, 41, 587, 719 indicator, 29 Lambert, 349, 430, 694 logarithm, 665 logistic, 237 meromorphic, 681 omega, 349, 430, 694 partition, 392, 752 polygamma, 727 probability density, 96, 183 Ramanujan Q, 36, 556, 747 regular, 630 scaled gamma, 726, 731 super-exponentially small, 645 surjection, 465 toll, 235, 599, 601 zeta, 282, 581, 587, 695, 707 Fundamental matrix, 529, 538 Fundamental Principle of Counting, 43

Index Galileo problem, 88 Gambler’s ruin problem, 304, 541 Gamma function, 201, 570, 694, 725, 726, 750 asymptotics, 726 incomplete, 726 poles, 726 recurrence, 201 scaled, 726, 731 Gaseous diffusion, 196 Gauss identity, 75 Gauss theorem, 727, 740 Gauss, Carl Friedrich, 62, 151, 157, 591, 616, 661, 705 Gaussian hypergeometric series, 73 Gaussian integral, 729 GCD, see greatest common divisor Generalized asymptotic expansion, 551 Generalized harmonic number, 587 Generating function, 280, 740 binomial, 280, 283 bivariate, 283, 341 Catalan, 322 convergence, 668 cumulant, 333 Dirichlet, 281, 599 exponential (EGF), 277, 283 factorial, 280, 283 moment (MGF), 333 Motzkin numbers, 734, 744 multiplicative, 277, 280 multivariate, 283, 294, 295 Newtonian, 280 ordinary (OGF), 272 Poisson, 407 probability (PGF), 331 uniqueness, 273 Generating functions convergence, 668 limit theorems, 668 Geometric distribution, 141, 335 Geometric mean, 556, 558

793 Geometric series, 32 Gessel, Ira, 685 GF, 271 Golden ratio, 23, 238, 308, 715, 737 Golin, Mordecai, 599 Goulden, Ian P., 347 Grabner, Peter, 575 Graham, Ronald, 18, 72, 221 Grammar, 491 Grant’s approximation, 240 Graph, 13, 423 cycle, 13 diameter, 235 directed, 13, 424 edge, 13 node, 13 path, 13 size, 13 tournament, 20, 112 undirected, 13 vertex, 13 Greatest common divisor, 54, 209 Gregory, James, 56 Grid path, 51, 179, 323, 325, 326, 401 Halley’s method, 241 Halley, Edmund, 241 Hamiltonian cycle, 20, 112 Hamiltonian path, 20 Hamming code, 139 Hamming distance, 459 Harary, Frank, 424 Harmonic mean, 27, 558 Harmonic numbers, 27, 41, 63, 192, 201, 341, 659, 702, 719, 743 GF, 276, 743 asymptotics, 586 second order, 291 summation, 41, 69, 291, 719 Hash function, 142 Hashing, 7, 142, 179, 444 closed, 143 open addressing, 143

794

Height function (of a tree), 235 Henrici, Peter, 75, 271, 347, 661 Henze, Norbert, 127 Hermite polynomial, 750 Hermite, Charles, 550 Hilbert, David, 80 Hill-Huntington rounding, 18 Hoare’s Find analysis, 346 Hoare, C. A. R., 229, 254 Hofri, Micha, 169, 604 Hoggatt identity, 17, 716 Holomorphic, see analytic Holomorphic function, 671 Homogenization of recurrence, 604 Horizontal generating function, 284 Horizontal summation, 69 Hypergeometric difference equation, 607 Hypergeometric distribution, 151 dual form, 153 limit, 557 symmetric form, 153 Hypergeometric function, 72, 458, 705 Hypergeometric series, 72, 151, 705 Hyperharmonic numbers, 719 I, xvii, 12 I.i.d., 131 Identity Abel, 75, 350, 702 Cassini, 16, 716 Gauss, 75 Hoggatt, 17, 716 Lagrange, 42 Saalschütz, 75 Touchard, 751 Worpitzky, 713 Identity matrix, 65 Identity operator, 38, 688 Imaginary axis, 662 Implicit function theorem, 694 Impossible event, 84 Improper continued fraction, 244, 732 Inadmissible set, 357

Index Inclusion and exclusion (PIE), 446–479 Incomplete beta function, 695 Incomplete exponential function, 154, 155, 290, 419 Incomplete gamma function, 154, 695, 726 Incomplete zeta function, 41, 587, 719 Increasing tree, 431 Independent events, 123, 125 Independent random variables, 125, 129 Indeterminate limit, 559 Indicator, 440 Indicator function, 29 Indicator random variable, 106–108, 161, 181, 190 Induction, 14–22 Inequality Bonferroni, 460, 729 Boole, 729 Bunyakovskii, 101 Cauchy-Schwarz, 101 Chebyshev, 188 Chebyshev-Bienaymè, 188 Chebyshev-Cantelli, 188, 190 Jensen, 558, 723 Markov, 189 Insertion sort, 102 Integer functions, 221 Integer sets, xvii, 12 Integral contour, 679 principal value, 282, 600 properties, 697, 698 Stieltjes, 696, 698 Integral-sum comparison, 37 Integrand, 697 Interior point, 678 Internal path length, 234 Interpolation search, 238 Intersection, 80 Inverse of permutation, 45 Inversion, 45 for binomial coefficients, 64, 683, 704

Index formula, 709 table, 45, 104 Involution, 45, 411 Iteration, 22–24, 201 Iverson bracket, xviii, 29, 451 Iverson notation, xviii, 29, 342 Iverson, Kenneth E., 29 Jackson, David M., 347 Jacobi identity, 748 Jensen’s inequality, 558, 723 Johnson, Norman, 149 Joint probability mass function, 127 Jones, William, 249 Josephus problem, 228 Karatsuba algorithm, 223 Karatsuba, Anatolii Alexeevich, 223 Kepler’s equation, 352 Khayyam, Omar, 56, 61 Khinchin constant, 736 Khinchin, Aleksandr, 610, 613, 617 Kleene closure, see closure, 362, 381, 492 Knuth, Donald, 18, 36, 46, 72, 209, 221, 254, 261, 400, 424, 556 Kolmogorov’s condition, 611 Kolmogorov, Andrei, 80, 88, 611 Koshy, Thomas, 15 Kronecker delta, 29, 48, 64, 261, 684, 721 Kronecker, Leopold, 29 Kummer’s theorem, 50 Kummer, Ernst E., 50 Kurtosis, 114 Poisson, 155 L’Hôpital rule, 286 Labeled product, 404, 405 Labeled sum, 404 Lagrange identity, 42, 721 Lagrange inversion theorem, 347, 351 Lagrange series, 351 Lagrange, Joseph-Louis, 348 Lagrange-Bürmann inversion theorem, 354

795 Lambert function, 349 Lambert, Johann, 694 Lampret, Vito, 580 Landis, E. M., 214 Language, 489, 491 regular, 495 counting, 505 Laplace method, 656–658 one-sided, 657 Laplace transform, 181, 315, 650 Laplace, Pierre, 170, 173, 355, 423, 487, 616, 656 Laplace-Borel transform, 293, 740 Lattice path, see grid path Lattice point, 51, 323, 325, 326 Lattice walk, see grid path Laurent series, 680 principal part, 680 regular part, 680 Law of large numbers, 88, 158 strong, Kolmogorov, 610 weak, Chebyshev, 610 Law of small numbers, 556 Law of succession, 173 Leading numbers, 521 Left-to-right maxima, 190 Legendre polynomial, 309, 353, 745, 750 Leibniz number, 71, 261 Leibniz, Gottfried, 549 Lemma reflection principle, 52 Watson, 650 Leth, Steven, 621 L’Hôpital rule, 559 Liber abbaci, 715 Limited selection, 375–381 Lindeberg, Jarl Waldemar, 616 Lindley, Dennis, 170 Linear convergence, 238 Linear distribution, 122 Linear operator, 36, 101 Linear programming, 2

796

Linear search, 193 Linearly independent, 280 Linearly independent solutions, 608 Lipschitz condition, 237, 255 LIT, see Lagrange inversion theorem Loaded dice, 176 Local limit theorem, 618 Logarithm function, 665, 743 principal value, 665 Logarithmic integral, 592, 594 Logistic function, 237 Longest run, 208 Lucas numbers, 715 Lucas polynomials, 715 Lucas, Edouard, 221, 462, 715 Lyapunov condition, 617 Lyapunov, Alexander, 616 |=, 377 Maclaurin series, 267, 668 Maclaurin, Colin, 580 Mahmoud, Hosam, 457 Manhattan metric, 179, 182 Maple, 9, 28, 78, 111, 149, 288, 289, 304, 306, 313, 317, 374, 399, 662, 676 choose, 111 coeftayl, 374 mtaylor, 399 Marginal probability mass function, 129 Marking operation, 388 Markov chain, 527–544 ergodic, 529, 530 higher-order, 532 irreducible, 528 periodic, 529 recurrent, 528 Markov inequality, 189 Markov’s condition, 611 Markov, Andrei, 189, 527, 611, 616 Markovian property, 527 Master theorem, 601 Mathematica, 288 Mathematical induction, see induction

Index Matrix fundamental, 529 substochastic, 529 Max search, 190 Maxima left-to-right, 190 Maxwell-Boltzmann statistics, 420, 441 Mean, 18, 100, 558 Mean, weighted, 558 Median, 101, 142, 153, 186 Mediant (Farey), 239 Mellin-Perron formula, 41 Ménages problem, 462, 463 Mendel, Gregor, 88 Merge sort, 227, 228, 600 Meromorphic function, 681 Method Birkhoff-Trjitzinsky, 604, 608 FEBS, 540 forward elimination and back substitution, 540 Halley, 241 Newton, 238 Newton-Raphson, 238 Rice, 569, 570 secant, 238 Steffensen, 239 undetermined coefficients, 30 Min cut-set problem, 169 Mixing, of distributions, 164 Mnatsakanian, Mamikon, 248 Mode, 101, 153 Moment about the mean, 113 about the origin, 112 and PGF, 341 central, 113 factorial, 113, 332 generating function (MGF), 333 Money changing problem, 368 Monotonicity of weighted means, 558 Montmort, Pierre Remond, 65, 469

Index Moon, John, 112 Motzkin numbers, 428, 751 generating function, 734, 744 Move-constrained permutations, 475, 476, 514 Multinomial coefficients, 49 Multinomial distribution, 140 Multinomial theorem, 49 Multiple series, 34 Multiplication rule (for counting), 43 Multiplicative OGF, 277, 280 Multisection, 285, 286 Multiset operation, 391, 427 Multivariate generating function, 294, 295 Multivariate hypergeometric distribution, 153 Multivariate ordinary generating function, 283 N, xvii, 12 [1..n], xvii, 12 n-fold convolution, see nested convolution Natural random variable, 85 Negating the upper argument, 61, 699 Negative-binomial distribution, 148, 179, 192 Nested convolution, 176 Nested recurrence, 215 Newton’s backward-difference formula, 282 Newton’s factorial series, 282 Newton’s forward-difference formula, 282 Newton’s identity, 741 Newton’s method, 238 Newton’s series, 56 Newtonian generating function, 280 Newtonian sequence, 277 NFA , see nondeterministic finite automaton Niven, Ivan, 272 Non-transitive dice, 105 Nondeterministic finite automaton, 498 Norm of sequence, 558 Normal approximation, 620 Normal distribution, 157

797 standard, 114 Normal Padé table, 243 Northeastern lattice path, 51, 60, 68, 326 Notation asymptotic, 546 {·}, 580 integer sets, xvii, 12 Null word, 363 Numbers ballot, 60, 257, 322, 748 Bell, 314, 443, 708, 744, 750 Bernoulli, 580, 581, 647, 648, 706, 710, 743, 750 Catalan, 60, 284, 320, 401, 608, 743, 751 central Delannoy, 686 central trinomial, 320, 352 coincidence, 258, 284, 730 cycle, 261, 333, 409, 708 Delannoy, 326, 686 derangement, 258, 708, 730, 744 divisor, 394, 747, 752 Euler, 714, 743, 750 Eulerian, 132, 482, 488, 712, 713, 748, 751 Fibonacci, 15, 17, 246, 307–309, 322, 328, 330, 715, 743 generalized harmonic, 587 harmonic, 27, 63, 201, 659, 702 GF, 276, 743 asymptotics, 586 summation, 41, 69, 291 leading, 521 Leibniz, 71 Lucas, 715 Motzkin, 428, 751 natural, 29 Pell, 310 pentagonal, 752 prime, 591 rank, 260, 731 Stirling, 708

Index

798

Stirling, first kind, 285, 333, 409, 708 Stirling, second kind, 314, 315, 406, 443, 702, 708, 710, 744 subset, 63, 261, 314 tangent, 743, 752 tetrahedral, 62 triangular, 62, 703 tribonacci, 340 Occupancy, 464 Occupancy configurations, 444 Occupancy enumeration, 434–446 Occupancy vector, 435 Octile, 101 Odlyzko, Andrew, 484, 648 OGF, 271, 272 Omega function, 349, 430 Open set, 679 Operator, 36 backward difference, 38, 705 difference, 38, 266, 292, 724 forward difference, 37, 38, 705 identity, 38, 688, 724 linear, 36 shift, 38, 266, 690, 724 Order of difference equation, 208, 251 Order statistics, 183 Ordered tree, 425 Ordinary continued fraction, 247 Ordinary generating function (OGF), 272 Outcome, 82 Overlap, 381, 519 Padé approximants, 242, 732 Padé index, 243 Padé table, 242 normal, 243 Padé, Henri, 242 Palindromic composition, 374 Paradox jailer, 172 Parrondo, 542 prisoner, 172

St. Petersburg, 109, 615 Parrondo paradox, 542 Parrondo, Juan, 542 Partial difference equation, 319 Partial fraction decomposition, 26, 307, 692 Partial quotient, 244, 732 Partition function, 392, 752 Partitional product, 405 Partitions, 205, 392–400 canonical representation, 401 odd parts, 395 Pascal distribution, see negative binomial distribution, 149 Pascal triangle, 59, 60 Pascal, Blaise, 149 Patashnik, Oren, 18, 72, 221 Path enumeration, see grid path Pattern, 483 Pattern matching, 197 Pell numbers, 310 Pentagonal number, 752 Periodic Bernoulli polynomials, 580 Periodic continued fraction, 736 Periodic Euler polynomials, 590 Periodic state, 529 Permutation, 45, 404, 469 alternating, 431, 747 ascents, rises, 479 canonical representation, 262, 408, 614 circular, 54, 404, 614 cycle representation, 408, 614 cycles, 408, 412 down-up, 431 fixed point, 258, 469 inverse, 45 involution, 411 left-to-right maxima, 190 record values, 190, 191 up-down, 431 zig-zag, 431 Persistent state, 528 Perturbation (of sum), 30

Index Petkovsek, Marko, 58, 72 PFD , see partial fraction decomposition PGF, 331 PIE, see principle of inclusion and exclusion Pigeonhole principle, 12, 13 Pile, 401 Plane tree, 425 Plane walk, see grid path Pochhammer symbol, 46, 699 Poincaré, Jules, 550 Poisson distribution, 154, 335, 556 as a limit, 557 Poisson generating function, 407 Poisson parameter, 155 Poisson, Siméon, 154, 581 Pole, 634, 681 order, 681 simple, 681 Pólya urn model, 110, 457 Pólya, George, 110, 368, 457, 487, 621 Polygamma function, 727 Polynomials Appell, 281, 706 Bell, 722 Bernoulli, 581, 706, 743, 750 table, 581 Bernstein, 136, 659 Chebyshev, 269, 306, 309, 734, 744, 749 Euler, 590, 714, 743 Eulerian, 713 Fibonacci, 715 Hermite, 750 Legendre, 309, 353, 745, 750 Lucas, 715 periodic Euler, 590 Power series, 667–677 convergence, 668 divergence, 668 logarithm, 743 radius of convergence, 669

799 Power set, 357, 390, 485 Prefix, 492 Prime number distribution function, 593 Prime Number Theorem, 592 Prime numbers, 591 Principal branch, 663 Principal part, 674 Principal value, 282, 600, 695 Principle of inclusion and exclusion, 446– 479 symmetric, 456 Principle of indifference, 87 Probability conditional, 120 cumulative distribution function, 95 distribution function, 95 of union, 468 Probability density function, 96, 183 Probability distribution, 95 Probability generating function (PGF), 331 Probability mass function (PMF), 95 Probability measure, 86 Probability of absorption, 530, 531 Probability space, 86 Problem ballot, 623 Banach matchbox, 98, 112 birthday, 90, 101, 108, 402, 416, 554 coupon-collector, 471, 477 Galileo, 88 gambler’s ruin, 304, 541 Josephus, 228 matching, 469 matrix multiplication, 8 ménages, 462, 463 money changing, 368 Montmort, 65, 469 path counting, 401 piles, 401 Smirnov sequences, 437 subset testing, 3 tennis tactics, 94

800

towers of Hanoi, 221 Problem complexity, 2 Prodinger, Helmut, 575 Product Cartesian, 360 Cauchy, 673 labeled, 404, 405 labeled, Abelian, 405 partitional, 405 rule, 360 Product rule (of ∆), 38 Proofs, 10–22 Proper continued fraction, 244, 732 Punnett square, 89 Punnett, Reginald Crundall, 89 Pythagoras’ constant, 736 Pythagorean theorem, 279 Quadratic mean, 558 Quantile, 101 Quartile, 101 Quickselect, 254 Quickselect analysis, 346 Quicksort, 229, 232, 340 analysis, 340, 345 partition, 232 recursive calls, 345 Quicksort recurrence, 229, 341 Quotient rule (of ∆), 38 Raabe’s multiplication formula, 707 Rademacher, Hans, 739 Radius of convergence, 630 Ramanujan Q-function, 36, 556, 747 Random search, 149 Random variable, 85 absolutely continuous, 96 Bernoulli, 136, 337 binomial, 136, 337 continuous, 96 degenerate, 335 discrete, 85 Engset, 178

Index geometric, 141, 335 geometric, shifted, 144 independent, 129 indicator, 106–108, 161, 181, 190 natural, 85 negative-binomial, 148 negatively correlated, 130 Poisson, 154, 335, 339 positively correlated, 130 symmetric, 102 uncorrelated, 130 uniform, 132, 335, 338 Random walk, 138, 178, 179, 197, 302, 533, 539, 623, 628 symmetric, 624 Randomization, 163, 164, 537 Markov chain, 531 Rank numbers, 260, 731 Rank of permutation, 260 Raphson, Joseph, 238 Rate of occurrence, 154 Rational approximation, 242, 247, 732 Real axis, 662 Recurrence, 199–270 binary search, 224 binomial coefficients, 256 bivariate, 319 constant coefficients, 268 coupled, 315 divide-and-conquer, 221 Fibonacci, 212 first order, linear, 215 full-history, 207, 208, 230, 233, 252, 301, 313, 314, 370, 752 homogeneous, 210 merge sort, 227 nested, 215 non-homogeneous, 210 order, 208 quicksort, 229, 341 vector, 217, 223, 315 vertical, 62

Index Recurrent state, 528 Recursion, 22–24, 203 Reflection principle, 52 Regular continued fraction, 247 Regular expression, 496 Regular expression identities, 497 Regular function, see analytic function, 630 Regular languages, 495 counting, 505 Regularity condition, 358 Relative error, 548 Rényi, Alphred, 613 Replacement, selection with, 44, 391 Replacement, selection without, 44, 45, 48, 94, 151 Residue, 680 Resolvent, 317, 510, 529 Rhind papyrus, 18 Rice method, 569, 570 Riemann condition, 679 Riemann hypothesis, 592 Riemann zeta function, 282 Riemann, Bernhard, 592, 695 Riordan, John, 58, 75, 327, 331, 462, 722 Rise in permutations, 479 Rising factorial, 46, 206, 699 Romberg algorithm, 256 Root of an equation, 236 Rooted tree, 424 Rouché, Eugène, 682 Rouché theorem, 351, 682 Roulette American, 100, 623 European, 100 Russian, 251 Rounding (to integer), 18 Roura, Salvador, 603 Rule multiplication (for counting), 43 product (of ∆), 38 quotient (of ∆), 38 sum (for counting), 43

801 Runs ascents in permutations, 479, 480 length, 480 s-harmonic number, 41 Saalschütz formula, 75 Sample average, 158 Sample error, 131 Sample maximum, 183 Sample mean, 131 Sample median, 183 Sample minimum, 183 Sample space, 82–84 countable, 83 uncountable, 84 Sampling, 131, 183 ordered, 84 unordered, 84 with replacement, 84, 111, 257 without replacement, 84, 112 Sanchis, Gabriela, 476 Scaled gamma function, 726, 731 Secant method, 238 Sedgewick, Robert, 46, 229 Seidel average, 132 Selection with replacement, 44, 391 Selection without replacement, 44, 45, 48, 94, 151 Selkow, Stanley, 201 Sequence binomial, 279 Newtonian, 277 Sequence operation, 362 Series alternating, 549 arithmetic, 15 asymptotic, 550 division, 673 geometric, 32 hypergeometric, 72, 705 Laurent, 680 Maclaurin, 668 multiple, 34

802

operations, 674 power, 667–677 Taylor, 667, 668 Set, xvii, 12, 80 Set characteristic function, 356 Shift operator, 38 Shifted geometric distribution, 144 Shifted negative-binomial, 149 Shuffle operation, 413 Sieve formula, 452 Signature, 73 Signless Stirling numbers, first kind, 261, 744 Simple event, 84 Simple pole, 634 Singleton cycle, 108, 254, 262 Singular point, 669 Singularity, 679 branch, 680 essential, 681 isolated, 679 pole, 681 removable, 680 Skewness, 114 Poisson, 155 Skolem sets, 379 Smirnov problem, 377, 386, 437 Smirnov sequences, 437 Smirnov set, 437, 438 Smirnov, Vladimir, 437 Snake oil summation, 327 Sorting insertion, 102 mergesort, 227, 228 quicksort, 229 St. Petersburg paradox, 109, 615 Standard deviation, 113, 157 Standard normal distribution, 157 Standardization of N(µ , σ ), 158 Stanley, Richard, 291 Steffensen’s method, 239 Stegun, Irene, 581

Index Stieltjes constants, 696, 714 Stieltjes integral, 696, 698 integrand, 697 integrator, 697 Stirling approximation, 77, 78, 700, 726 Stirling formula, 553, 589 Stirling numbers, 708 first kind, 261–263, 285, 333, 409, 708 signless, 261, 744 generating function, 285, 333 second kind, 63, 261–263, 314, 315, 406, 443, 702, 708, 710, 744 Stirling, James, 77 Stochastic convergence, 158 Stochastic property, 527 Strassen algorithm, 8 Strassen, Volker, 8 Strings, 489 empty, 490 left-to-right-minima, 516 Subfactorial, 750 Subset numbers, see Stirling numbers, second kind, 261 Substitution operation, 383 Substochastic matrix, 529 Succession in a sequence, 466 Sudkamp, Thomas A., 497 Suffix, 492 Sum, see series, summation labeled, 404 Sum function, 669 Sum of the series, 668 Sum rule, 359 Sum rule (for counting), 43 Sum-integral comparison, 37 Summation, 26–42 alternating, 589 by parts, 36, 39, 40, 62, 724 powers, 584 snake oil, 327 telescopic, 26, 109 Super-exponentially small function, 645

Index Support, 106, 558 Supremum, 158 Sure event, 84 Surjection, 406, 465 Sylvester, James Joseph, 452 Symbolic method, 423 Symmetric inclusion-exclusion pair, 456 Symmetric random variable, 102 Symmetric random walk, 624 Szegö, Gabor, 621 Szekeres, George, 13, 14 Taboo subwords, 483 Tangent numbers, 743, 752 Tauberian theorem, 181, 652 Taylor series, 667, 668 Telescoping sum, 26, 109 Ternary trees, 304 Terquem problem, 378 Theon’s ladder, 240 Theorem PIE, 451, 468 error estimate for alternating series, 549 Abel, 740 Abel convergence, 668 arcsine law, 627 Arden, 504 Bayes, 170 Borel-Cantelli, 159 Cauchy, 649, 679 Cauchy residue, 682 central limit, 338, 616, 617 Chebyshev, 611 Chung-Feller, 627 Darboux, 646 de Moivre-Laplace, 617 Erdos and Renyi, 613 Euler, 740 Euler pentagonal number, 398, 747 Fermat’s little, 53 Flajolet and Odlyzko, 648 Gauss, 727, 740 implicit function, 694

803 iterated logarithm, 613 Khinchin, 610 Kolmogorov, 611 Kummer, 50 Lagrange, 736 Lagrange inversion, 347, 351 Lagrange-Bürmann inversion, 354 law of succession, 173 little Bayes, 170 local limit, 618 Markov, 611 master, 601 Prime Number, 592 Pythagorean, 279 radius of convergence, 630 Rouché, 682 Tauber, 181, 652 total probability, 162 transfer, 645 Thron, Wolfgang, 249 Toll function, 235, 599, 601 Total expectation, 162 Total probability, theorem, 162 Touchard’s identity, 751 Tournament graph, 20, 112 Towers of Hanoi problem, 221 Transfer theorem, 645 Transformation between GFs, 293 Transient state, 528 Transition matrix, 527 canonical form, 529 Transition probability matrix, 527 Transposition, 411 Tree, 423 ancestor, 424 AVL, 214 binary, 17, 214, 299, 424 counting, 299 depth, 17 height, 17 binary search, 234, 235 Catalan, 426

Index

804

Cayley, 430 children, 424 degree-restricted, 427 depth, 428 descendant, 424 descending, 431 enumeration, 18, 423–430 free, 424, 426 labeled, 18 height, 428 increasing, 431 k-ary, 424 labeled, 429 leaf, 424 ordered, 425 labeled, 429 oriented, 426 labeled, 430 plane, 425 labeled, 429 rooted, 424, 426 sibling, 424 terminology, 424 ternary, 304 counting, 304 unlabeled, 424 unordered, 426 Tree function, 430 Tree parameters, 341 Tri-diagonal marching algorithm, 540 Triangular distribution, 176 Triangular numbers, 62, 703 Tribonacci numbers, 340 Trinomial coefficients, 49, 50 Trjitzinsky, W. J., 604, 608 Tschebyscheff, Pafnuty L., 188 Undetermined coefficients, method, 30 Uniform distribution, 115 Uniform random variable, 132, 336, 338 Uniformity, 97, 102 Union, 81 Unique decomposition, 360, 493

Unique factorization, 360 Universal set, 80 Vandermonde convolution, 67, 68, 70, 75, 152 Vandermonde’s formula, 69 Variance, 113 conditional, 133 Poisson, 155 Variegated strings, 464 Vector difference equation, 315 Vector recurrence, 223, 315 Venn diagram, 81, 83, 448 Venn, John, 80 Verma, Rakesh, 602 Vertical generating function, 284 von Neumann, John, 148 Waiting time, 487, 519, 522 Watson’s lemma, 650 Watson, George, 347, 580 Weak composition, 364, 392, 738 Weierstrass condition, 679 Weight, 356 Weighted mean, 558 Whittaker, Edmund, 347, 580 Wilf, Herbert, 327 Williamson, Gill, 401, 514 Winograd, Shmuel, 10 Word, 489, 490 equation, 503 length, 490 Worpitzky’s identity, 54, 713 Wronskian, 608 Z, xvii, 12 Z+ , xvii, 12 z-transform, 272 Zadeh, Lotfi, 356 Zero of a function, 236 Zeta function, 282, 292, 581, 587, 695, 707 Zig-zag permutation, 431

Methods in Algorithmic Analysis presents numerous techniques and methods used for analyzing algorithms. It highlights mathematical techniques and methods that are practical and relevant to theoretical aspects of computer science. After introducing basic mathematical and combinatorial methods, the text focuses on various aspects of probability. It explores the role of recurrences in computer science, numerical analysis, engineering, and discrete mathematics applications. The author then describes the powerful tool of generating functions and discusses the symbolic method, the principle of inclusion and exclusion, and its applications. The book goes on to show how strings can be manipulated and counted, how the finite state machine and Markov chains can help solve probabilistic and combinatorial problems, how to derive asymptotic results, and how convergence and singularities play leading roles in deducing asymptotic information from generating functions. Features • Provides a solid theoretical and mathematical background on the analysis of algorithms • Includes basic material on combinatorics and probability • Presents information on asymptotics not usually found in similar books, including the critical range method, Rice’s method, the Euler and Boole summation formulas, and the Birkhoff–Trjitzinsky method • Offers tutorials throughout the text on topics as diverse as probabilistic methods, enumeration with generating functions, occupancy enumeration, and combinatorics of strings • Contains examples drawn from the field of computer science • Supplies a large number of useful formulas in an appendix

Dobrushkin

Accompanied by more than 1,000 examples and exercises, this comprehensive book develops an understanding of the mathematical methodology behind the analysis of algorithms. It emphasizes the important relation between continuous (classical) mathematics and discrete mathematics, which is the basis of computer science.


Computer Science/Computer Engineering/Computing


Vladimir A. Dobrushkin

C6829

C6829_Cover.indd 1

10/1/09 10:24 AM

Methods in Algorithmic Analysis

Probabilistic methods in algorithmic discrete mathematics

Probabilistic methods in algorithmic discrete mathematics

Probabilistic methods in algorithmic discrete mathematics

Algorithmic and Analysis Techniques in Property Testing

Methods in Nonlinear Analysis

Ramsey methods in analysis

Asymptotic Methods in Analysis

Asymptotic methods in analysis

Methods in nonlinear analysis

Ramsey Methods in Analysis

Matrix Methods in Analysis

Algorithmic methods for artificial intelligence

Elementary numerical analysis: algorithmic approach

Elementary numerical analysis: algorithmic approach

Functional Analysis Methods in Numerical Analysis

Probabilistic methods in combinatorial analysis

Fractal-Based Methods in Analysis

Thermal methods in petroleum analysis

Asymptotic methods in analysis MCap

Nonlinear methods in numerical analysis

Methods in Agricultural Chemical Analysis

Asymptotic methods in analysis MCap

Probabilistic Methods for Algorithmic Discrete Mathematics

Elementary Numerical Analysis: An Algorithmic Approach

Elementary Numerical Analysis: An Algorithmic Approach

Elementary numerical analysis - an algorithmic approach

Elementary Numerical Analysis An Algorithmic Approach

Elementary Numerical Analysis: An Algorithmic Approach

Methods of nonlinear analysis,

Methods in Algorithmic Analysis

Probabilistic methods in algorithmic discrete mathematics

Probabilistic methods in algorithmic discrete mathematics

Probabilistic methods in algorithmic discrete mathematics

Algorithmic and Analysis Techniques in Property Testing

Methods in Nonlinear Analysis

Ramsey methods in analysis

Asymptotic Methods in Analysis

Asymptotic methods in analysis

Methods in nonlinear analysis

Ramsey Methods in Analysis

Matrix Methods in Analysis

Algorithmic methods for artificial intelligence

Elementary numerical analysis: algorithmic approach

Elementary numerical analysis: algorithmic approach

Functional Analysis Methods in Numerical Analysis

Probabilistic methods in combinatorial analysis

Fractal-Based Methods in Analysis

Thermal methods in petroleum analysis

Asymptotic methods in analysis MCap

Nonlinear methods in numerical analysis

Methods in Agricultural Chemical Analysis

Asymptotic methods in analysis MCap

Probabilistic Methods for Algorithmic Discrete Mathematics

Elementary Numerical Analysis: An Algorithmic Approach

Elementary Numerical Analysis: An Algorithmic Approach

Elementary numerical analysis - an algorithmic approach

Elementary Numerical Analysis An Algorithmic Approach

Elementary Numerical Analysis: An Algorithmic Approach

Methods of nonlinear analysis,

Recommend Documents