Information, Physics, and Computation (Oxford Graduate Texts)

INFORMATION, PHYSICS, AND COMPUTATION This page intentionally left blank Information, Physics, and Computation Marc...

Author: Marc Mezard | Andrea Montanari

323 downloads 1889 Views 8MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

INFORMATION, PHYSICS, AND COMPUTATION

This page intentionally left blank

Information, Physics, and Computation Marc Mézard Laboratoire de Physique Théorique et Modèles Statistiques, CNRS, and Université Paris Sud

Andrea Montanari Department of Electrical Engineering and Department of Statistics, Stanford University, and Laboratoire de Physique Théorique de l’ENS, Paris

1

3

Great Clarendon Street, Oxford OX2 6DP Oxford University Press is a department of the University of Oxford. It furthers the University’s objective of excellence in research, scholarship, and education by publishing worldwide in Oxford New York Auckland Cape Town Dar es Salaam Hong Kong Karachi Kuala Lumpur Madrid Melbourne Mexico City Nairobi New Delhi Shanghai Taipei Toronto With offices in Argentina Austria Brazil Chile Czech Republic France Greece Guatemala Hungary Italy Japan Poland Portugal Singapore South Korea Switzerland Thailand Turkey Ukraine Vietnam Oxford is a registered trade mark of Oxford University Press in the UK and in certain other countries Published in the United States by Oxford University Press Inc., New York c Oxford University Press 2009 The moral rights of the authors have been asserted Database right Oxford University Press (maker) First Published 2009 All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, without the prior permission in writing of Oxford University Press, or as expressly permitted by law, or under terms agreed with the appropriate reprographics rights organization. Enquiries concerning reproduction outside the scope of the above should be sent to the Rights Department, Oxford University Press, at the address above You must not circulate this book in any other binding or cover and you must impose the same condition on any acquirer British Library Cataloguing in Publication Data Data available Library of Congress Cataloging in Publication Data Data available Printed in Great Britain on acid-free paper by CPI, Antony Rowe, Chippenham, Wiltshire ISBN 978–0–19–857083–7 (Hbk.) 1 3 5 7 9 10 8 6 4 2

Preface Over the last few years, several research areas have witnessed important progress through the unexpected collaboration of statistical physicists, computer scientists, and information theorists. This dialogue between scientific disciplines has not been without difficulties, as each field has its own objectives and rules of behaviour. Nonetheless, there is increasing consensus that a common ground exists and that it can be fruitful. This book aims at making this common ground more widely accessible, through a unified approach to a selection of important research problems that have benefited from this convergence. Historically, information theory and statistical physics have been deeply linked since Shannon, sixty years ago, used entropy to quantify the information content of a message. A few decades before, entropy had been the cornerstone of Boltzmann’s statistical mechanics. However, the two topics separated and developed in different directions. Nowadays most statistical physicists know very little of information theory, and most information theosists know very little of statistical physics. This is particularly unfortunate, as recent progress on core problems in both fields has been bringing these two roads closer over the last two decades. In parallel, there has been growing interest in applying probabilistic concepts in computer science, both in devising and in analysing new algorithms. Statistical physicists have started to apply to this field the non-rigorous techniques they had developed to study disordered systems. Conversely, they have become progressively aware of the powerful computational techniques invented by computer scientists and applied them in large scale simulations. In statistical physics, the last quarter of the twentieth century has seen the emergence of a new topic. The main focus until then had been on ‘ordered’ materials: crystals in which atoms vibrate around equilibrium positions arranged in a periodic lattice, or liquids and gases in which the density of particles is uniform. In the 1970s, the interest in strongly disordered systems started to grow, through studies of spin glasses, structural glasses, polymer networks, etc. The reasons for this development were the incredible richness of behaviour in these systems and their many applications in materials science, and also the variety of conceptual problems which are involved in the understanding of these behaviours. Statistical physics deals with the collective behavior of many interacting components. With disordered systems, it started to study collective behaviour of systems in which all of the components are heterogeneous. This opened the way to the study of a wealth of problems outside of physics, where heterogeneity is common currency. Some of the most spectacular recent progress in information theory concerns errorcorrecting codes. More than fifty years after Shannon’s theorems, efficient codes have now been found which approach Shannon’s theoretical limit. Turbo codes and lowdensity parity-check (LDPC) codes have allowed large improvements in error cor-

Preface

rection. One of the main ingredients of these schemes is message-passing decoding strategies, such as the celebrated ‘belief propagation’ algorithm. These approaches are intimately related to the mean-field theories of disordered systems developed in statistical physics. Probability plays an important role in theoretical computer science, from randomized algorithms to probabilistic combinatorics. Random ensembles of computational problems are studied as a way to model real-world situations, to test existing algorithms, or to develop new ones. In such studies one generally defines a family of instances and endows it with a probability measure, in the same way as one defines a family of samples in the case of spin glasses or LDPC codes. The discovery that the hardest-to-solve instances, with all existing algorithms, lie close to a phase transition boundary spurred a lot of interest. Phase transitions, or threshold phenomena, are actually found in all of these three fields, and play a central role in each of them. Predicting and understanding them analytically is a major challenge. It can also impact the design of efficient algorithms. Statistical physics suggests that the reason for the hardness of random constraint satisfaction problems close to phase transitions is a structural one: it hinges on the existence of a glass transition, a structural change in the geometry of the set of solutions. This understanding has opened up new algorithmic perspectives. In order to emphasize the real convergence of interest and methods in all of these fields, we have adopted a unified approach. This book is structured in five large parts, focusing on topics of increasing complexity. Each part typically contains three chapters that present some core topics in each of the disciplines of information theory, statistical physics, and combinatorial optimization. The topics in each part have a common mathematical structure, which is developed in additional chapters serving as bridges. • Part I (Chapters 1–4) contains introductory chapters to each of the three disciplines and some common probabilistic tools. • Part II (Chapters 5–8) deals with problems in which independence plays an important role: the random energy model, the random code ensemble, and number partitioning. Thanks to the independence of random variables, classical techniques can be applied successfully to these problems. The part ends with a description of the replica method. • Part III (Chapters 9–13) describes ensembles of problems on graphs: satisfiability, low-density parity-check codes, and spin glasses. Factor graphs and statistical inference provide a common language. • Part IV (chapters 14–17) explains belief propagation and the related ‘replicasymmetric’ cavity method. These can be thought of as approaches to studying systems of correlated random variables on large graphs, when the correlations decay fast enough with distance. The part shows the success of this approach with three problems: decoding, assignment, and ferromagnets. • Part V (Chapters 18–22) is dedicated to an important consequence of long-range correlations, namely the proliferation of pure states and ‘replica symmetry breaking’. It starts with the simpler problem of random linear equations with Boolean variables, and then develops the general approach and applies it to satisfiability and coding. The final chapter reviews some open problems.

Preface

At the end of each chapter, a section of notes provides pointers to the literature. The notation and symbols are summarized in Appendix A. The definitions of new concepts are signalled by boldfaced fonts, both in the text and in the index. The book contains many examples and exercises of various difficulty, which are signalled by a light grey background. They are an important part of the book. As the book develops, we venture into progressively less well-understood topics. In particular, the number of mathematically proved statements decreases and we rely on heuristic or intuitive explanations in some places. We have put special effort into distinguishing what has been proved from what has not, and into presenting the latter as clearly and as sharply as we could. We hope that this will stimulate the interest and contributions of mathematically minded readers, rather than alienate them. This is a graduate-level book, intended to be useful to any student or researcher who wants to study and understand the main concepts and methods in this common research domain. The introductory chapters help to set up the common language, and the book should thus be understandable by any graduate student in science with some standard background in mathematics (probability, linear algebra, and calculus). Our choice of presenting a selection of problems in some detail has left aside a number of other interesting topics and applications. Some of them are of direct common interest in information, physics, and computation, for instance source coding, multiple-input multiple-output communication, and learning and inference in neural networks. But the concepts and techniques studied in this book also have applications in a broader range of ‘complex systems’ studies, ranging from neurobiology, or systems biology, to economics and the social sciences. A few introductory pointers to the literature are provided in the Notes of Chapter 22. The critical reading and the many comments of Heiko Bauke, Alfredo Braunstein, John Eduardo Realpe Gomez, Florent Krzakala, Frauke Liers, Stephan Mertens, Elchanan Mossel, Sewoong Oh, Lenka Zdeborov´ a, have been very useful. We are grateful to them for their feedback. We have also been stimulated by the kind encouragements of Amir Dembo, Persi Diaconis, James Martin, Balaji Prabhakar, Federico Ricci-Tersenghi, Bart Selman and R¨ udiger Urbanke, and the discreet attention and steady support of Sonke Adlung, from Oxford University Press.

This book is dedicated to Fanny, Mathias, Isabelle, Claudia, and Ivana.

Marc Mézard and Andrea Montanari, December 2008


Contents PART I

BACKGROUND

1

Introduction to information theory 1.1 Random variables 1.2 Entropy 1.3 Sequences of random variables and their entropy rate 1.4 Correlated variables and mutual information 1.5 Data compression 1.6 Data transmission Notes

3 3 5 8 10 12 16 21

2

Statistical physics and probability theory 2.1 The Boltzmann distribution 2.2 Thermodynamic potentials 2.3 The fluctuation–dissipation relations 2.4 The thermodynamic limit 2.5 Ferromagnets and Ising models 2.6 The Ising spin glass Notes

23 24 28 32 33 35 44 46

3

Introduction to combinatorial optimization 3.1 A first example: The minimum spanning tree 3.2 General definitions 3.3 More examples 3.4 Elements of the theory of computational complexity 3.5 Optimization and statistical physics 3.6 Optimization and coding Notes

47 48 51 51 54 60 61 62

4

A probabilistic toolbox 4.1 Many random variables: A qualitative preview 4.2 Large deviations for independent variables 4.3 Correlated variables 4.4 The Gibbs free energy 4.5 The Monte Carlo method 4.6 Simulated annealing 4.7 Appendix: A physicist’s approach to Sanov’s theorem Notes

65 65 66 72 77 80 86 87 89

Ü

Contents

PART II INDEPENDENCE 5

The random energy model 5.1 Definition of the model 5.2 Thermodynamics of the REM 5.3 The condensation phenomenon 5.4 A comment on quenched and annealed averages 5.5 The random subcube model Notes

93 93 94 100 101 103 105

6

The random code ensemble 6.1 Code ensembles 6.2 The geometry of the random code ensemble 6.3 Communicating over a binary symmetric channel 6.4 Error-free communication with random codes 6.5 Geometry again: Sphere packing 6.6 Other random codes 6.7 A remark on coding theory and disordered systems 6.8 Appendix: Proof of Lemma 6.2 Notes

107 107 110 112 120 123 126 127 128 128

7

Number partitioning 7.1 A fair distribution into two groups? 7.2 Algorithmic issues 7.3 Partition of a random list: Experiments 7.4 The random cost model 7.5 Partition of a random list: Rigorous results Notes

131 131 132 133 136 140 143

8

Introduction to replica theory 8.1 Replica solution of the random energy model 8.2 The fully connected p-spin glass model 8.3 Extreme value statistics and the REM 8.4 Appendix: Stability of the RS saddle point Notes

145 145 155 163 166 169

PART III 9

MODELS ON GRAPHS

Factor graphs and graph ensembles 9.1 Factor graphs 9.2 Ensembles of factor graphs: Definitions 9.3 Random factor graphs: Basic properties 9.4 Random factor graphs: The giant component 9.5 The locally tree-like structure of random graphs Notes

10 Satisfiability 10.1 The satisfiability problem

173 173 180 182 187 191 194 197 197

Contents

10.2 Algorithms 10.3 Random K -satisfiability ensembles 10.4 Random 2-SAT 10.5 The phase transition in random K (≥ 3)-SAT Notes

Ü

199 206 209 209 217

11 Low-density parity-check codes 11.1 Definitions 11.2 The geometry of the codebook 11.3 LDPC codes for the binary symmetric channel 11.4 A simple decoder: Bit flipping Notes

219 220 222 231 236 239

12 Spin glasses 12.1 Spin glasses and factor graphs 12.2 Spin glasses: Constraints and frustration 12.3 What is a glass phase? 12.4 An example: The phase diagram of the SK model Notes

241 241 245 250 262 265

13 Bridges: Inference and the Monte Carlo method 13.1 Statistical inference 13.2 The Monte Carlo method: Inference via sampling 13.3 Free-energy barriers Notes

267 268 272 281 287

PART IV

SHORT-RANGE CORRELATIONS

14 Belief propagation 14.1 Two examples 14.2 Belief propagation on tree graphs 14.3 Optimization: Max-product and min-sum 14.4 Loopy BP 14.5 General message-passing algorithms 14.6 Probabilistic analysis Notes

291 292 296 305 310 316 317 325

15 Decoding with belief propagation 15.1 BP decoding: The algorithm 15.2 Analysis: Density evolution 15.3 BP decoding for an erasure channel 15.4 The Bethe free energy and MAP decoding Notes

327 327 329 342 347 352

16 The 16.1 16.2 16.3

355 356 357 366

assignment problem The assignment problem and random assignment ensembles Message passing and its probabilistic analysis A polynomial message-passing algorithm

Ü

Contents

16.4 Combinatorial results 16.5 An exercise: Multi-index assignment Notes 17 Ising models on random graphs 17.1 The BP equations for Ising spins 17.2 RS cavity analysis 17.3 Ferromagnetic model 17.4 Spin glass models Notes PART V

371 376 378 381 381 384 386 391 399

LONG-RANGE CORRELATIONS

18 Linear equations with Boolean variables 18.1 Definitions and general remarks 18.2 Belief propagation 18.3 Core percolation and BP 18.4 The SAT–UNSAT threshold in random XORSAT 18.5 The Hard-SAT phase: Clusters of solutions 18.6 An alternative approach: The cavity method Notes

403 404 409 412 415 421 422 427

19 The 1RSB cavity method 19.1 Beyond BP: Many states 19.2 The 1RSB cavity equations 19.3 A first application: XORSAT 19.4 The special value x = 1 19.5 Survey propagation 19.6 The nature of 1RSB phases 19.7 Appendix: The SP(y) equations for XORSAT Notes

429 430 434 444 449 453 459 463 465

20 Random K-satisfiability 20.1 Belief propagation and the replica-symmetric analysis 20.2 Survey propagation and the 1RSB phase 20.3 Some ideas about the full phase diagram 20.4 An exercise: Colouring random graphs Notes

467 468 474 485 488 491

21 Glassy states in coding theory 21.1 Local search algorithms and metastable states 21.2 The binary erasure channel 21.3 General binary memoryless symmetric channels 21.4 Metastable states and near-codewords Notes

493 493 500 506 513 515

22 An ongoing story 22.1 Gibbs measures and long-range correlations

517 518

Contents

22.2 Higher levels of replica symmetry breaking 22.3 Phase structure and the behaviour of algorithms Notes

Ü

524 535 538

Appendix A Symbols and notation A.1 Equivalence relations A.2 Orders of growth A.3 Combinatorics and probability A.4 Summary of mathematical notation A.5 Information theory A.6 Factor graphs A.7 Cavity and message-passing methods

541 541 542 543 544 545 545 545

References

547

Index

565


Part I Background


1 Introduction to information theory This chapter introduces some of the basic concepts of information theory, as well as the definitions and notation of probability theory that will be used throughout the book. The notion of entropy, which is fundamental to the whole topic of this book, is introduced here. We also present two major questions of information theory, those of data compression and error correction, and state Shannon’s theorems. Sec. 1.1 introduces the basic notations in probability. The notion of entropy, and the entropy rate of a sequence are discussed in Sections 1.2 and 1.3. A very important concept in information theory is the mutual information of two random variables, which is introduced in Section 1.4. Then we move to the two main aspects of the theory, the compression of data, in Sec. 1.5, and the transmission of data in, Sec. 1.6.

1.1

Random variables

The main object of this book will be the behaviour of large sets of discrete random variables. A discrete random variable X is completely defined1 by the set of values it can take, X , which we assume to be a finite set, and its probability distribution {pX (x)}x∈X . The value pX (x) is the probability that the random variable X takes the value x. The probability distribution pX : X → [0, 1] is a non-negative function that satisfies the normalization condition pX (x) = 1 . (1.1) x∈X

We shall denote by P(A) the probability of an event A ⊆ X , so that pX (x) = P(X = x). To lighten the notation, when there is no ambiguity, we shall use p(x) to denote pX (x). If f (X) is a real-valued function of the random variable X, the expectation value of f (X), which we shall also call the average of f , is denoted by pX (x)f (x) . (1.2) Ef = x∈X

While our main focus will be on random variables taking values in finite spaces, we shall sometimes make use of continuous random variables taking values in Rd or in some smooth finite-dimensional manifold. The probability measure for an 1 In probabilistic jargon (which we shall avoid hereafter), we take the probability space P (X , P(X ), pX ), where P(X ) is the σ-field of the parts of X and pX = x∈X pX (x) δx .

Introduction to information theory

‘infinitesimal element’ dx will be denoted by dpX (x). Each time when pX admits a density (with respect to the Lebesgue measure), we shall use the notation pX (x) for the value of this density at the point x. The total probability P(X ∈ A) that the variable X takes a value in some (measurable) set A ⊆ X is given by the integral dpX (x) = I(x ∈ A) dpX (x) , (1.3) P(X ∈ A) = x∈A

where the second form uses the indicator function I(s) of a logical statement s, which is defined to be equal to 1 if the statement s is true, and equal to 0 if the statement is false. The expectation value E f (X) and the variance Var f (X) of a real-valued function f (x) are given by 2 Var f (X) = E{f (X)2 } − {E f (X)} . (1.4) E f (X) = f (x) dpX (x) , Sometimes, we may write EX f (X) to specify the variable to be integrated over. We shall often use the shorthand pdf for the probability density function pX (x). Example 1.1 A fair die with M faces has X = {1, 2, . . . , M } and p(i) = 1/M for all i ∈ {1, . . . , M }. The average of x is E X = (1 + · · · + M )/M = (M + 1)/2. Example 1.2 Gaussian variable. A continuous variable X ∈ R has a Gaussian distribution of mean m and variance σ 2 if its probability density is [x − m]2 1 exp − . (1.5) p(x) = √ 2σ 2 2πσ We have EX = m and E(X − m)2 = σ 2 . Appendix A contains some definitions and notation for the random variables that we shall encounter most frequently The notation of this chapter refers mainly to discrete variables. Most of the expressions can be transposed to the case of continuous variables by replacing sums x by integrals and interpreting p(x) as a probability density.

Exercise 1.1 Jensen’s inequality. Let X be a random variable taking values in a set X ⊆ R and let f be a convex function (i.e. a function such that ∀x, y and ∀α ∈ [0, 1]: f (αx + (1 − α)y) ≤ αf (x) + (1 − α)f (y)). Then Ef (X) ≥ f (EX) .

(1.6)

Supposing for simplicity that X is a finite set with |X | = n, prove this equality by recursion on n.

Entropy

1.2

Entropy

The entropy HX of a discrete random variable X with probability distribution p(x) is defined as 1 , (1.7) p(x) log2 p(x) = E log2 HX ≡ − p(X) x∈X

where we define, by continuity, 0 log2 0 = 0. We shall also use the notation H(p) whenever we want to stress the dependence of the entropy upon the probability distribution of X. In this chapter, we use the logarithm to base 2, which is well adapted to digital communication, and the entropy is then expressed in bits. In other contexts, and in particular in statistical physics, one uses the natural logarithm (with base e ≈ 2.7182818) instead. It is sometimes said that, in this case, entropy is measured in nats. In fact, the two definitions differ by a global multiplicative constant, which amounts to a change of units. When there is no ambiguity, we shall use H instead of HX . Intuitively, the entropy HX is a measure of the uncertainty of the random variable X. One can think of it as the missing information: the larger the entropy, the less a priori information one has on the value of the random variable. It roughly coincides with the logarithm of the number of typical values that the variable can take, as the following examples show. Example 1.3 A fair coin has two values with equal probability. Its entropy is 1 bit.

Example 1.4 Imagine throwing M fair coins: the number of all possible outcomes is 2M . The entropy equals M bits.

Example 1.5 A fair die with M faces has entropy log2 M .

Example 1.6 Bernoulli process. A Bernoulli random variable X can take values 0, 1 with probabilities p(0) = q, p(1) = 1 − q. Its entropy is HX = −q log2 q − (1 − q) log2 (1 − q) ,

(1.8)

which is plotted as a function of q in Fig. 1.1. This entropy vanishes when q = 0 or q = 1 because the outcome is certain; it is maximal at q = 1/2, when the uncertainty of the outcome is maximal. Since Bernoulli variables are ubiquitous, it is convenient to introduce the function H(q) ≡ −q log q − (1 − q) log(1 − q) for their entropy.


1 0.9 0.8 0.7

H(q)

0.6 0.5 0.4 0.3 0.2 0.1 0 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

q Fig. 1.1 The entropy H(q) of a binary variable with p(X = 0) = q and p(X = 1) = 1 − q, plotted versus q.

Exercise 1.2 An unfair die with four faces and p(1) = 1/2, p(2) = 1/4, p(3) = p(4) = 1/8 has entropy H = 7/4, smaller than that of the corresponding fair die.

Exercise 1.3 DNA is built from a sequence of bases which are of four types, A, T, G, C. In the natural DNA of primates, the four bases have nearly the same frequency, and the entropy per base, if one makes the simplifying assumption of independence of the various bases, is H = − log2 (1/4) = 2. In some genuses of bacteria, one can have big differences in concentrations: for example, p(G) = p(C) = 0.38, p(A) = p(T ) = 0.12, giving a smaller entropy H ≈ 1.79.

Exercise 1.4 In some intuitive way, the entropy of a random variable is related to the ‘risk’ or ‘surprise’ which is associated with it. Let us see how these notions can be made more precise. Consider a gambler who bets on a sequence of Bernoulli random variables Xt ∈ {0, 1}, t ∈ {0, 1, 2, . . . }, with mean EXt = p. Imagine he knows the distribution of the Xt ’s and, at time t, he bets a fraction w(1) = p of his money on 1 and a fraction w(0) = (1 − p) on 0. He loses whatever is put on the wrong number, while he doubles whatever has been put on the right one. Define the average doubling rate of his wealth at time t as 1 Wt = E log2 t

(

t Y

) 2w(Xt )

.

(1.9)

t =1

It is easy to prove that the expected doubling rate EWt is related to the entropy of Xt : EWt = 1 − H(p). In other words, it is easier to make money out of predictable events.

Entropy

Another notion that is directly related to entropy is the Kullback–Leibler (KL) divergence between two probability distributions p(x) and q(x) over the same finite space X . This is defined as D(q||p) ≡

x∈X

q(x) log

q(x) , p(x)

(1.10)

where we adopt the conventions 0 log 0 = 0 and 0 log(0/0) = 0. It is easy to show that (i) D(q||p) is convex in q(x); (ii) D(q||p) ≥ 0; and (iii) D(q||p) > 0 unless q(x) ≡ p(x). The last two properties derive from the concavity of the logarithm (i.e. the fact that the function − log x is convex) and Jensen’s inequality (eqn (1.6)): if E denotes the expectation with respect to the distribution q(x), then −D(q||p) = E log[p(x)/q(x)] ≤ log E[p(x)/q(x)] = 0. The KL divergence D(q||p) thus looks like a distance between the probability distributions q and p, although it is not symmetric. The importance of the entropy, and its use as a measure of information, derives from the following properties: 1. HX ≥ 0. 2. HX = 0 if and only if the random variable X is certain, which means that X takes one value with probability one. 3. Among all probability distributions on a set X with M elements, H is maximum when all events x are equiprobable, with p(x) = 1/M . The entropy is then HX = log2 M . To prove this statement, note that if X has M elements, then the KL divergence D(p||p) between p(x) and the uniform distribution p(x) = 1/M is D(p||p) = log2 M − H(p). The statement is a direct consequence of the properties of the KL divergence. 4. If X and Y are two independent random variables, meaning that pX,Y (x, y) = pX (x)pY (y), the total entropy of the pair X, Y is equal to HX + HY : HX,Y = −

pX,Y (x, y) log2 pX,Y (x, y)

x,y

=−

pX,Y (x, y) (log2 pX (x) + log2 pY (y)) = HX + HY .

(1.11)

x,y

5. For any pair of random variables, one has in general HX,Y ≤ HX + HY , and this result is immediately generalizable to n variables. (The proof can be obtained by using the positivity of the KL divergence D(p1 ||p2 ), where p1 = pX,Y and p2 = pX pY .) 6. Additivity for composite events. Take a finite set of events X , and decompose it into X = X1 ∪ X2 , where X1 ∩ X2 = ∅. Denote by q1 = x∈X1 p(x) the probability of X1 , and denote by q2 the probability of X2 . For each x ∈ X1 , define as usual the conditional probability of x, given that x ∈ X1 , by r1 (x) = p(x)/q1 and define r2 (x) similarly as the conditional probability of x, given that x ∈ X2 . The as the sum of two contributions HX = total entropy can then be written r), where − x∈X p(x) log2 p(x) = H(q) + H(q,


H(q) = −q1 log2 q1 − q2 log2 q2 H(q, r) = −q1 r1 (x) log2 r1 (x) − q2 r2 (x) log2 r2 (x) . x∈X1

(1.12) (1.13)

x∈X1

The proof is straightforward and is done by substituting the laws r1 and r2 by their definitions. This property can be interpreted as the fact that the average information associated with the choice of an event x is additive, being the sum of the information H(q) associated to a choice of subset, and the information r) associated with the choice of the event inside the subset (weighted by the H(q, probability of the subset). This is the main property of the entropy, which justifies its use as a measure of information. In fact, this is a simple example of the chain rule for conditional entropy, which will be illustrated further in Sec. 1.4. Conversely, these properties, together with appropriate hypotheses of continuity and monotonicity, can be used to define the entropy axiomatically.

1.3

Sequences of random variables and their entropy rate

In many situations of interest, one deals with a random process which generates sequences of random variables {Xt }t∈N , each of them taking values in the same finite space X . We denote by PN (x1 , . . . , xN ) the joint probability distribution of the first N variables. If A ⊂ {1, . . . , N } is a subset of indices, we denote by A its complement A = {1, . . . , N } \ A and use the notation xA = {xi , i ∈ A} and xA = {xi , i ∈ A} (the set subscript will be dropped whenever it is clear from the context). The marginal distribution of the variables in A is obtained by summing PN over the variables in A:

PA (xA ) =

PN (x1 , . . . , xN ) .

(1.14)

xA

Example 1.7 The simplest case is when the Xt ’s are independent. This means that PN (x1 , . . . , xN ) = p1 (x1 )p2 (x2 ) . . . pN (xN ). If all the distributions pi are identical, equal to p, the variables are independent identically distributed, and abbreviated as i.i.d. The joint distribution is PN (x1 , . . . , xN ) =

N t=1

p(xi ) .

(1.15)

Sequences of random variables and their entropy rate

Example 1.8 The sequence {Xt }t∈N is said to be a Markov chain if PN (x1 , . . . , xN ) = p1 (x1 )

N −1

w(xt → xt+1 ) .

(1.16)

t=1

Here {p1 (x)}x∈X is called the initial state, and the {w(x → y)}x,y∈X are the transition probabilities of the chain. The transition probabilities must be nonnegative and normalized: w(x → y) = 1 , for any x ∈ X . (1.17) y∈X

When we have a sequence of random variables generated by a process, it is intuitively clear that the entropy grows with the number N of variables. This intuition suggests that we should define the entropy rate of a sequence xN ≡ {Xt }t∈N as hX = lim HX N /N , N →∞

(1.18)

if the limit exists. The following examples should convince the reader that the above definition is meaningful. Example 1.9 If the Xt ’s are i.i.d. random variables with distribution {p(x)}x∈X , the additivity of entropy implies p(x) log p(x) . (1.19) hX = H(p) = − x∈X

Example 1.10 Let {Xt }t∈N be a Markov chain with initial state {p1 (x)}x∈X and transition probabilities {w(x → y)}x,y∈X . Call {pt (x)}x∈X the marginal distribution of Xt and assume the following limit to exist independently of the initial condition: p∗ (x) = lim pt (x) . t→∞

(1.20)

As we shall see in Chapter 4, this indeed turns out to be true under quite mild hypotheses on the transition probabilities {w(x → y)}x,y∈X . It is then easy to show that p∗ (x) w(x → y) log w(x → y) . (1.21) hX = − x,y∈X

½¼


If you imagine, for instance, that a text in English is generated by picking letters randomly from the alphabet X , with empirically determined transition probabilities w(x → y), then eqn (1.21) gives a rough estimate of the entropy of English. A more realistic model can be obtained using a Markov chain with memory. This means that each new letter xt+1 depends on the past through the values of the k previous letters xt , xt−1 , . . . , xt−k+1 . Its conditional distribution is given by the transition probabilities w(xt , xt−1 , . . . , xt−k+1 → xt+1 ). Computing the corresponding entropy rate is easy. For k = 4, one obtains an entropy of 2.8 bits per letter, much smaller than the trivial upper bound log2 27 (there are 26 letters, plus the space symbol), but many words so generated are still not correct English words. Better estimates of the entropy of English, obtained through guessing experiments, give a number around 1.3.

1.4

Correlated variables and mutual information

Given two random variables X and Y taking values in X and Y, we denote their joint probability distribution as pX,Y (x, y), which is abbreviated as p(x, y), and we denote the conditional probability distribution for the variable y, given x, as pY |X (y|x), abbreviated as p(y|x). The reader should be familiar with the classical Bayes’ theorem p(y|x) = p(x, y)/p(x) .

(1.22)

When the random variables X and Y are independent, p(y|x) is independent of x. When the variables are dependent, it is interesting to have a measure of their degree of dependence: how much information does one obtain about the value of y if one knows x? The notions of conditional entropy and mutual information will answer this question. We define the conditional entropy HY |X as the entropy of the law p(y|x), averaged over x: p(x) p(y|x) log2 p(y|x) . (1.23) HY |X ≡ −

x∈X

y∈Y

The joint entropy HX,Y ≡ − x∈X ,y∈Y p(x, y) log2 p(x, y) of the pair of variables x, y can be written as the entropy of x plus the conditional entropy of y given x, an identity known as the chain rule: (1.24) HX,Y = HX + HY |X . In the simple case, where the two variables are independent, HY |X = HY , and HX,Y = HX + HY . One way to measure the correlation of the two variables is to use the mutual information IX,Y , which is defined as p(x, y) IX,Y ≡ . (1.25) p(x, y) log2 p(x)p(y) x∈X ,y∈Y

It is related to the conditional entropies by IX,Y = HY − HY |X = HX − HX|Y .

(1.26)

This shows that the mutual information IX,Y measures the reduction in the uncertainty of x due to the knowledge of y, and is symmetric in x, y.

Correlated variables and mutual information

½½

Proposition 1.11 IX,Y ≥ 0. Moreover, IX,Y = 0 if and only if X and Y are independent variables. Proof Write IX,Y = Ex,y − log2 {p(x)p(y)/p(x, y)}. Consider the random variable u = (x, y) with probability distribution p(x, y). As the function − log( · ) is convex, one can apply Jensen’s inequality (eqn (1.6)). This gives the result IX,Y ≥ 0

Exercise 1.5 A large group of friends plays the following game (‘telephone without cables’). The person number zero chooses a number X0 ∈ {0, 1} with equal probability and communicates it to the person number one without letting the others hear, and so on. The first person communicates the number to the second person, without letting anyone else hear. Call Xn the number communicated from the n-th to the (n + 1)-th person. Assume that, at each step a person may become confused and communicate the wrong number with probability p. How much information does the n-th person have about the choice of the first person? We can quantify this information through IX0 ,Xn ≡ In . Show that In = 1 − H(pn ) with pn given by 1 − 2pn = (1 − 2p)n . In particular, as n → ∞, In =

˜ (1 − 2p)2n ˆ 1 + O((1 − 2p)2n ) . 2 log 2

(1.27)

The ‘knowledge’ about the original choice decreases exponentially along the chain.

Mutual information is degraded when data is transmitted or processed. This is quantified as follows Proposition 1.12 Data-processing inequality. Consider a Markov chain X → Y → Z (so that the joint probability of the three variables can be written as p1 (x)w2 (x → y)w3 (y → z)). Then IX,Z ≤ IX,Y . In particular, if we apply this result to the case where Z is a function of Y , Z = f (Y ), we find that applying f degrades the information: IX,f (Y ) ≤ IX,Y . Proof We introduce the mutual information of two variables conditioned on a third one: IX,Y |Z = HX|Z − HX|(Y Z) . The mutual information between a variable X and a pair of variables (Y Z) can be decomposed using the following chain rule: IX,(Y Z) = IX,Z + IX,Y |Z = IX,Y + IX,Z|Y . If we have a Markov chain X → Y → Z, X and Z are independent when we condition on the value of Y , and therefore IX,Z|Y = 0. The result follows from the fact that IX,Y |Z ≥ 0. The conditional entropy also provides a lower bound on the probability of guessing a random variable. Suppose you want to guess the value of the random variable X, but you observe only the random variable Y (which can be thought of as a noisy version of

= g(Y ), which is your estimate for X. What X). From Y , you compute a function X is the probability Pe that you guessed incorrectly? Intuitively, if X and Y are strongly correlated, one can expect that Pe is small, whereas it increases for less well-correlated variables. This is quantified as follows. Proposition 1.13 Fano’s inequality. Consider a random variable X taking values

where X

= g(Y ) is an estimate in the alphabet X , and the Markov chain X → Y → X,

½¾


= X). This for the value of X. Define the probability of making an error as Pe = P(X is bounded from below as follows: H(Pe ) + Pe log2 (|X | − 1) ≥ H(X|Y ) .

(1.28)

= X), equal to 0 if X

= X and to 1 Proof Define a random variable E = I(X otherwise, and decompose the conditional entropy HX,E|Y using the chain rule in two ways: HX,E|Y = HX|Y + HE|X,Y = HE|Y + HX|E,Y . Then notice that (i) HE|X,Y = 0 (because E is a function of X and Y ); (ii) HE|Y ≤ HE = H(Pe ) and (iii) HX|E,Y = (1 − Pe )HX|E=0,Y + Pe HX|E=1,Y = Pe HX|E=1,Y ≤ Pe log2 (|X | − 1).

Exercise 1.6 Suppose that X can take k values, and that its distribution is p(1) = 1 − p, p(x) = p/(k − 1) for x ≥ 2. If X and Y are independent, what is the value of the right-hand p side of Fano’s inequality? Assuming that 1 − p > k−1 , what is the best guess one can make about the value of X? What is the probability of error? Show that Fano’s inequality holds as an equality in this case.

1.5

Data compression

Imagine an information source which generates a sequence of symbols X = {X1 , . . . , XN } taking values in a finite alphabet X . We assume a probabilistic model for the source, meaning that the Xi are random variables. We want to store the information contained in a given realization x = {x1 . . . xN } of the source in the most compact way. This is the basic problem of source coding. Apart from being an issue of the utmost practical interest, it is a very instructive subject. It in fact allows us to formalize in a concrete fashion the intuitions of ‘information’ and ‘uncertainty’ which are associated with the definition of entropy. Since entropy will play a crucial role throughout the book, we present here a little detour into source coding. 1.5.1

Codewords

We first need to formalize what is meant by ‘storing the information’. We define a source code for the random variable X to be a mapping w which associates with any possible information sequence in X N a string in a reference alphabet, which we shall assume to be {0, 1}: w : X N→ {0, 1}∗ x → w(x) .

(1.29)

Here we have used the convention of denoting by {0, 1}∗ the set of binary strings of arbitrary length. Any binary string which is in the image of w is called a codeword. Often, the sequence of symbols X1 . . . XN is a part of a longer stream. The compression of this stream is realized in three steps. First, the stream is broken into blocks

Data compression

½¿

of length N . Then, each block is encoded separately using w. Finally, the codewords are glued together to form a new (hopefully more compact) stream. If the original stream consists of the blocks x(1) , x(2) , . . . , x(r) , the output of the encoding process will be the concatenation of w(x(1) ), . . . , w(x(r) ). In general, there is more than one way of parsing this concatenation into codewords, which may cause troubles when one wants to recover the compressed data. We shall therefore require the code w to be such that any concatenation of codewords can be parsed unambiguously. The mappings w satisfying this property are called uniquely decodable codes. Unique decodability is certainly satisfied if, for any x, x ∈ X N , w(x) is not a prefix of w(x ) (see Fig. 1.2). In such a case the code is said to be instantaneous. Hereafter, we shall focus on instantaneous codes, since they are both practical and slightly simpler to analyse. Now that we have specified how to store information, namely using a source code, it is useful to introduce a figure of merit for source codes. If lw (x) is the length of the string w(x), the average length of the code is

L(w) =

p(x) lw (x) .

(1.30)

x∈X N

Example 1.14 Take N = 1, and consider a random variable X which takes values in X = {1, 2, . . . , 8} with probabilities p(1) = 1/2, p(2) = 1/4, p(3) = 1/8, p(4) = 1/16, p(5) = 1/32, p(6) = 1/64, p(7) = 1/128, and p(8) = 1/128. Consider the two codes w1 and w2 defined by the table below: x 1 2 3 4 5 6 7 8

p(x) 1/2 1/4 1/8 1/16 1/32 1/64 1/128 1/128

w1 (x) w2 (x) 000 0 001 10 010 110 011 1110 100 11110 101 111110 110 1111110 111 11111110

(1.31)

These two codes are instantaneous. For instance, looking at the code w2 , the encoded string 10001101110010 can be parsed in only one way, since each symbol 0 ends a codeword. It thus corresponds to the sequence x1 = 2, x2 = 1, x3 = 1, x4 = 3, x5 = 4, x6 = 1, x7 = 2. The average length of code w1 is L(w1 ) = 3, and the average length of code w2 is L(w2 ) = 247/128. Notice that w2 achieves a shorter average length because it assigns the shortest codeword (namely 0) to the most probable symbol (i.e. x = 1).

½

Introduction to information theory 1

0

0

1

0

1 11

0 000

1

1

0

0

001

1

0

1

101

Fig. 1.2 An instantaneous source code: each codeword is assigned to a node in a binary tree in such a way that none of them is the ancestor of another one. Here, the four codewords are framed.

Example 1.15 A useful graphical representation of a source code can be obtained by drawing a binary tree and associating each codeword with the corresponding node in the tree. In Fig. 1.2, we represent a source code with |X N | = 4 in this way. It is quite easy to recognize that the code is indeed instantaneous. The codewords, which are framed, are such that no codeword is the ancestor of any other codeword in the tree. Given a sequence of codewords, parsing is immediate. For instance, the sequence 00111000101001 can be parsed only into 001, 11, 000, 101, 001. 1.5.2

Optimal compression and entropy

Suppose that you have a ‘complete probabilistic characterization’ of the source you want to compress. What is the ‘best code’ w for this source? This problem was solved (to a large extent) by Shannon in his celebrated 1948 paper, by connecting the best achievable average length to the entropy of the source. Following Shannon, we assume that we know the probability distribution of the source p(x). Moreover, we interpret ‘best code’ as ‘code with the shortest average length’. Theorem 1.16 Let L∗N be the shortest average length achievable by an instantaneous code for the variable X = {X1 , . . . , XN }, which has entropy HX . Then: 1. For any N ≥ 1 HX ≤ L∗N ≤ HX + 1 . (1.32) 2. If the source has a finite entropy rate h = limN →∞ HX /N , then lim

N →∞

1 ∗ L = h. N N

(1.33)

Proof The basic idea of the proof of eqn (1.32) is that if the codewords were too short, the code would not be instantaneous. Kraft’s inequality makes this simple remark more precise. For any instantaneous code w, the lengths lw (x) satisfy 2−lw (x) ≤ 1 . (1.34) x∈X N

Data compression

½

This fact is easily proved by representing the set of codewords as a set of leaves on a binary tree (see Fig. 1.2). Let LM be the length of the longest codeword . Consider the set of all the 2LM possible vertices in the binary tree at the generation LM ; let us call them the ‘descendants’. If the information x is associated with a codeword at generation l (i.e. lw (x) = l), there can be no other codewords in the branch of the tree rooted at this codeword, because the code is instantaneous. We ‘erase’ the corresponding 2LM −l descendants, which cannot be codewords. The subsets of erased descendants associated with each codeword are not overlapping. Therefore the total number of erased descendants, x 2LM −lw (x) , must be less than or equal to the total number of descendants, 2LM . This establishes Kraft’s inequality. Conversely, for any set of lengths {l(x)}x∈X N which satisfy Kraft’s inequality (1.34), there exists at least one code whose codewords have lengths {l(x)}x∈X N . A possible construction is obtained as follows. Consider the smallest length l(x) and take the first allowed binary sequence of length l(x) to be the codeword for x. Repeat this operation with the next shortest length and so on, until all the codewords have been exhausted. It is easy to show that this procedure is successful if eqn (1.34) is satisfied. ∗ The problem is therefore reduced to finding the set of codeword lengths l(x) = l (x) which minimize the average length L = x p(x)l(x) subject to Kraft’s inequality (1.34). Supposing first that l(x) can take arbitrary non-negative real values, this is easily done with Lagrange multipliers, and leads to l(x) = − log2 p(x). This set of optimal lengths, which in general cannot be realized because some of the l(x) are not integers, gives an average length equal to the entropy HX . It implies the lower bound in eqn (1.32). In order to build a real code with integer lengths, we use l∗ (x) = − log2 p(x) .

(1.35)

Such a code satisfies Kraft’s inequality, and its average length is less than or equal than HX + 1, proving the upper bound in eqn (1.32). The second part of the theorem is a straightforward consequence of the first part. The code that we have constructed in the proof is often called a Shannon code. For long strings (N 1), it is close to optimal. However, it has no reason to be optimal in general. For instance, if only one p(x) is very small, it will assign x to a very long codeword, while shorter codewords are available. It is interesting to know that, for a given source {X1 , . . . , XN }, there exists an explicit construction of the optimal code, called Huffman’s code. At first sight, it may appear that Theorem 1.16, together with the construction of Shannon codes, completely solves the source coding problem. Unhappily, this is far from true, as the following arguments show. From a computational point of view, the encoding procedure described above is unpractical when N is large. One can build the code once for all, and store it somewhere, but this requires Θ(|X |N ) memory. On the other hand, one could reconstruct the code every time a string required to be encoded, but this takes Θ(|X |N ) operations. One can use the same code and be a little smarter in the encoding procedure, but this

½


does not yield a big improvement. (The symbol Θ means ‘of the order of’; the precise definition is given in Appendix A.) From a practical point of view, the construction of a Shannon code requires an accurate knowledge of the probabilistic law of the source. Suppose now that you want to compress the complete works of Shakespeare. It is exceedingly difficult to construct a good model for the source ‘Shakespeare’. Even worse, when you will finally have such a model, it will be of little use for compressing Dante or Racine. Happily, source coding has made tremendous progresses in both directions in the last half-century. However, in this book, we shall focus on another crucial aspect of information theory, the transmission of information.

1.6

Data transmission

We have just seen how to encode information in a string of symbols (we used bits, but any finite alphabet is equally good). Suppose now that we want to communicate this string. When the string is transmitted, it may be corrupted by noise, which depends on the physical device used for the transmission. One can reduce this problem by adding redundancy to the string. This redundancy is to be used to correct some of the transmission errors, in the same way as redundancy in the English language could be used to correct some of the typos in this book. This is the domain of channel coding. A central result in information theory, again due to Shannon’s pioneering work of 1948, relates the level of redundancy to the maximal level of noise that can be tolerated for error-free transmission. As in source coding, entropy again plays a key role in this result. This is not surprising, in view of the duality between the two problems. In data compression, one wants to reduce the redundancy of the data, and the entropy gives a measure of the ultimate possible reduction. In data transmission, one wants to add some well-tailored redundancy to the data.

1.6.1

Communication channels

A typical flowchart of a communication system is shown in Fig. 1.3. It applies to situations as diverse as communication between the earth and a satellite, cellular phones, and storage within the hard disk of a computer. Alice wants to send a message m to Bob. Let us assume that m is an M -bit sequence. This message is first encoded into a longer one, an N -bit message denoted by x, with N > M , where the added bits will provide the redundancy used to correct transmission errors. The encoder is a map from {0, 1}M to {0, 1}N . The encoded message is sent through a communication channel. The output of the channel is a message y. In the case of a noiseless channel, one would simply have y = x. In the case of a realistic channel, y is in general a string of symbols different from x. Note that y is not necessarily a string of bits. The channel is described by a transition probability Q(y|x). This is the probability that the received signal is y, conditional on the transmitted signal being x. Different physical channels are described by different functions Q(y|x). The decoder takes the message y and deduces from it an estimate m of the sent message.

Data transmission

Encoder

m original message M bits

x

Transmission Channel

encoded message N bits

½

Decoder

y received message

m' estimate of the original message M bits

Fig. 1.3 Typical flowchart of a communication device.

Exercise 1.7 Consider the following example of a channel with insertions. When a bit x is fed into the channel, either x or x0 is received with equal probability 1/2. Suppose that you send the string 111110. The string 1111100 will be received with probability 2 · 1/64 (the same output can be produced by an error in either the fifth or the sixth digit). Notice that the output of this channel is a bit string which is always longer than or equal in length to the transmitted string. A simple code for this channel is easily constructed: use the string 100 for each 0 in the original message and 1100 for each 1. Then, for instance, we have the encoding 01101 → 100110011001001100 .

(1.36)

The reader is invited to define a decoding algorithm and verify its effectiveness.

Hereafter, we shall consider memoryless channels. In this case, for any input x = (x1 , ..., xN ), the output message is a string of N letters y = (y1 , ..., yN ) from an alphabet Y yi (not necessarily binary). In a memoryless channel, the noise acts independently on each bit of the input. This means that the conditional probability Q(y|x) factorizes, i.e. N Q(yi |xi ) , (1.37) Q(y|x) = i=1

and the transition probability Q(yi |xi ) is independent og i. Example 1.17 Binary symmetric channel (BSC). The input xi and the output yi are both in {0, 1}. The channel is characterized by one number, the probability p that the channel output is different from the input, called the crossover (or flip) probability. It is customary to represent this type of channel by the diagram on the left of Fig. 1.4.

Example 1.18 Binary erasure channel (BEC). In this case some of the input bits are erased instead of being corrupted: xi is still in {0, 1}, but yi now belongs to {0, 1, ∗}, where ∗ means that the symbol has been erased. In the symmetric case, this channel is described by a single number, the probability that a bit is erased, see Fig. 1.4, middle.

½


1−p 0

1− 0

0

p

1−p

0

p 1

1 0

1

1

1−

p

*

1

0

1

1−p

1

Fig. 1.4 Three communication channels. Left: the binary symmetric channel. An error in the transmission, in which the output bit is the opposite of the input one, occurs with probability p. Middle: the binary erasure channel. An error in the transmission, signaled by the output ∗, occurs with probability . Right: the Z channel. An error occurs with probability p whenever a 1 is transmitted.

Example 1.19 Z channel. In this case the output alphabet is again {0, 1}. Now, however, a 0 is always transmitted correctly, whereas a 1 becomes a 0 with probability p. The name of this channel comes from its graphical representation: see Fig. 1.4, right. A very important characteristic of a channel is the channel capacity C. This is defined in terms of the mutual information IX,Y of the variables X (the bit which was sent) and Y (the signal which was received), through p(x, y) . (1.38) p(x, y) log2 C = max IX,Y = max p(x)p(y) p(x) p(x) x∈X ,y∈Y

We recall that in our case p(x, y) = p(x)Q(y|x), and IX,Y measures the reduction in the uncertainty of x due to the knowledge of y. The capacity C gives a measure of how faithful a channel can be. If the output of the channel is pure noise, x and y are uncorrelated and C = 0. At the other extreme if y = f (x) is known for sure, given x, then C = max{p(x)} H(p) = 1 bit (for binary inputs). The reason for our interest in the capacity will become clear in Section 1.6.3 with Shannon’s coding theorem, which shows that C characterizes the amount of information which can be transmitted faithfully through a channel. Example 1.20 Consider a binary symmetric channel with flip probability p. Let us call the probability that the source sends x = 0 q, and the probability of x = 1 1 − q. It is easy to show that the mutual information in eqn (1.38) is maximized when zeros and ones are transmitted with equal probability (i.e. when q = 1/2). Using eqn (1.38), we get C = 1 − H(p) bits, where H(p) is the entropy of a Bernoulli process with parameter p (plotted in Fig. 1.1). Example 1.21 Consider now a binary erasure channel with error probability . The same argument as above applies. It is therefore easy to obtain C = 1 − .

Data transmission

½

Exercise 1.8 Compute the capacity of a Z channel.

1.6.2

Error-correcting codes

We need one last ingredient in order to have a complete definition of the channel coding problem: the behaviour of the information source. We shall assume that the source produces a sequence of uncorrelated, unbiased bits. This may seem at first a very crude model for any real information source. Surprisingly, Shannon’s source–channel separation theorem ensures that there is indeed no loss of generality in treating this case. The sequence of bits produced by the source is divided into blocks m1 , m2 , m3 , . . . of length M . The encoding is a mapping from {0, 1}M m to {0, 1}N , with N ≥ M . Each possible M -bit message m is mapped to a codeword x(m), which can be seen as a point in the N -dimensional unit hypercube. The codeword length N is also called the block length. There are 2M codewords, and the set of all possible codewords is called the codebook. When a message is transmitted, N the corresponding codeword x is corrupted to y ∈ Y N with probability Q(y|x) = i=1 Q(yi |xi ). The output alphabet Y depends on the channel. The decoder is a mapping from Y N to {0, 1}M which takes the received message y ∈ Y N and maps it to one of the possible original messages m = d(y) ∈ {0, 1}M . An error-correcting code is defined by a pair of functions, the encoding x(m) and the decoding d(y). The ratio M (1.39) R= N of the original number of bits to the transmitted number of bits is called the rate of the code. The rate is a measure of the redundancy of the code. The smaller the rate, the more redundancy is added to the code, and the more errors one should be able to correct. The block error probability of a code on an input message m, denoted by PB (m), is the probability that the decoded message differs from the message which was sent: Q(y|x(m)) I(d(y) = m) . (1.40) PB (m) = y

Knowing the error probability for each possible transmitted message amounts to an exceedingly detailed characterization of the performance of the code. One can therefore introduce a maximal block error probability as ≡ Pmax B

max

m∈{0,1}M

PB (m) .

(1.41)

This corresponds to characterizing the code by its ‘worst case’ performances. A more optimistic point of view corresponds to averaging over the input messages. Since we

¾¼


have assumed all of them to be equiprobable, we introduce the average block error probability, defined as Pav B ≡

1 2M

PB (m) .

(1.42)

m∈{0,1}M

Since this is a very common figure of merit for error-correcting codes, we shall call it simply the block error probability, and use the symbol PB without further specification hereafter. Example 1.22 Repetition code. Consider a BSC which transmits a wrong bit with probability p. A simple code consists in repeating each bit k times, with k odd. Formally, we have M = 1, N = k, and . . . 00 , x(0) = 000

(1.43)

k

x(1) = 111 . . . 11 .

(1.44)

k

This code has rate R = M/N = 1/k. For instance, with k = 3, the original stream 0110001 is encoded as 000111111000000000111. A possible decoder consists in parsing the received sequence into groups of k bits, and finding the message m using a majority rule among the k bits. In our example with k = 3, if the received group of three bits is 111 or 110 or any permutation, the corresponding input bit is assigned to 1, otherwise it is assigned to 0. For instance, if the channel output is 000101111011000010111, this decoder returns 0111001.

Exercise 1.9 The k-repetition code corrects up to k/2 errors per group of k bits. Show that the block error probability for general k is PB =

„ « k X k (1 − p)k−r pr . r

(1.45)

r=k/2

Note that, for any finite k and p > 0, PB is strictly positive. In order to have PB → 0, we must consider k → ∞. Since the rate is 1/k, the price to pay for a vanishing block error probability is a vanishing communication rate! Happily, however, we shall see that much better codes exist.

1.6.3

The channel coding theorem

Consider a communication channel whose capacity (eqn (1.38)) is C. In his seminal 1948 paper, Shannon proved the following theorem. Theorem 1.23 For every rate R < C, there exists a sequence of codes {CN }, of block length N , rate RN , and block error probability PB,N , such that RN → R and

Notes

¾½

PB,N → 0 as N → ∞. Conversely, if, for a sequence of codes {CN }, one has RN → R and PB,N → 0 as N → ∞, then R < C. In practice, for long messages (i.e. large N ), reliable communication is possible if and only if the communication rate remains below the channel capacity. The direct part of the proof will be given in Sec. 6.4 using the random code ensemble. We shall not give a full proof of the converse part in general, but only in the case of a BSC, in Sec. 6.5.2. Here we confine ourselves to some qualitative comments and provide the intuitive idea underlying this theorem. First of all, the result is rather surprising when one meets it for the first time. As we saw in the example of repetition codes above, simple-minded codes typically have a positive error probability for any non-vanishing noise level. Shannon’s theorem establishes that it is possible to achieve a vanishing error probability while keeping the communication rate bounded away from zero. One can get an intuitive understanding of the role of the capacity through a qualitative argument, which uses the fact that a random variable with entropy H ‘typically’ takes 2H values. For a given codeword x(m) ∈ {0, 1}N , the channel output y is a random variable with an entropy Hy|x = N Hy|x . There exist about 2N Hy|x such outputs. For perfect decoding, one needs a decoding function d(y) that maps each of them to the original message m. Globally, the typical number of possible outputs is 2N Hy , and therefore one can distinguish at most 2N (Hy −Hy|x ) codewords. In order to have a vanishing maximal error probability, one needs to be able to send all of the 2M = 2N R codewords. This is possible only if R < Hy − Hy|x ≤ C.

Notes There are many textbooks that provide introductions to probability and to information theory. A classic probability textbook is Feller (1968). For a more recent reference see Durrett (1995). The original Shannon paper (Shannon, 1948) is universally recognized as the foundation of information theory. A very nice modern introduction to the subject is the book by Cover and Thomas (1991). The reader may find in there a description of Huffman codes, which we did not treat in the present Chapter, as well as more advanced topics in source coding. We did not show that the six properties listed in Section 1.2 in fact provide an alternative (axiomatic) definition of entropy. The interested reader is referred to Csisz´ ar abd K¨ orner (1981). An advanced book on information theory with much space devoted to coding theory is Gallager (1968). The recent and very rich book by MacKay (2002) discusses the relations with statistical inference and machine learning. The information-theoretic definition of entropy has been used in many contexts. It can be taken as a founding concept in statistical mechanics. This approach, pioneered by Jaynes (1957), is discussed by Balian (1992).


2 Statistical physics and probability theory One of the greatest achievements of science has been to realize that matter is made out of a small number of simple elementary components. This result seems to be in striking contrast to our experience. Both at a simply perceptual level and with more refined scientific experience, we come into touch with an ever-growing variety of states of matter with disparate properties. The ambitious purpose of statistical physics (and, more generally, of a large branch of condensed matter physics) is to understand this variety. It aims at explaining how complex behaviours can emerge when large numbers of identical elementary components are allowed to interact. We have, for instance, experience of water in three different states (solid, liquid and gaseous). Water molecules and their interactions do not change when passing from one state to the other. Understanding how the same interactions can result in qualitatively different macroscopic states, and what governs the change of state, is a central topic of statistical physics. The foundations of statistical physics rely on two important steps. The first one consists in passing from the deterministic laws of physics, such as Newton’s laws, to a probabilistic description. The idea is that a precise knowledge of the motion of each molecule in a macroscopic system is inessential to an understanding of the system as a whole: instead, one can postulate that the microscopic dynamics, because of its chaoticity, allows a purely probabilistic description. The detailed justification of this basic step has been achieved in only a small number of concrete cases. Here we shall bypass any attempt at such a justification: we directly adopt a purely probabilistic point of view, as a basic postulate of statistical physics. The second step starts from the probabilistic description and recovers determinism at a macroscopic level by some sort of law of large numbers. We all know that water boils at 100o C (at atmospheric pressure) and that its density (at 25o C and atmospheric pressure) is 1 g/cm3 . The regularity of these phenomena is not related to the deterministic laws which rule the motions of water molecules. It is instead a consequence of the fact that, because of the large number of particles involved in any macroscopic system, fluctuations are ‘averaged out’. We shall discuss this kind of phenomenon in Section 2.4 and, more mathematically, in Chapter 4. The purpose of this chapter is to introduce the most basic concepts of this discipline for an audience of non-physicists with a mathematical background. We adopt a somewhat restrictive point of view, which keeps to classical (as opposed to quantum) statistical physics, and basically describes it as a branch of probability the-

Statistical physics and probability theory

ory (Sections 2.1–2.3). In Section 2.4 we focus on large systems, and stress that the statistical-physics approach becomes particularly meaningful in this regime. Theoretical statistical physics often deals with highly idealized mathematical models of real materials. The most interesting (and challenging) task is in fact to understand the qualitative behaviour of such systems. With this aim, one can discard any ‘irrelevant’ microscopic detail from the mathematical description of the model. In Section 2.5, the study of ferromagnetism through the introduction of the Ising model gives an example of this modelling procedure. Compared with the case of Ising ferromagnets, the theoretical understanding of spin glasses is much less developed. Section 2.6 presents a rapid preview of this fascinating subject.

2.1

The Boltzmann distribution

The basic ingredients for a probabilistic description of a physical system are: • A space of configurations X . One should think of x ∈ X as giving a complete microscopic determination of the state of the system under consideration. We are not interested in defining the most general mathematical structure for X such that a statistical-physics formalism can be constructed. Throughout this book we shall in fact consider only two very simple types of configuration spaces: (i) finite sets, and (ii) smooth, compact, finite-dimensional manifolds. If the system contains N ‘particles’, the configuration space is a product space: XN = X × · · · × X .

(2.1)

N

The configuration of the system has the form x = (x1 , . . . , xN ). Each coordinate xi ∈ X is meant to represent the state (position, orientation, etc.) of one of the particles. Except for a few examples, we shall focus on configuration spaces of type (i). We shall therefore adopt a discrete-space notation for X . The generalization to continuous configuration spaces is in most cases intuitively clear (although it may present some technical difficulties). • A set of observables, which are real-valued functions on the configuration space O : x → O(x). If X is a manifold, we shall limit ourselves to observables which are smooth functions of the configuration x. Observables are physical quantities which can be measured through an experiment (at least in principle). • Out of all the observables, a special role is played by the energy function E(x). When the system is an N -particle system, the energy function generally takes the form of sums of terms involving few particles. An energy function of the form E(x) =

N

Ei (xi )

(2.2)

i=1

corresponds to a non-interacting system. An energy of the form E(x) = Ei1 ,...,ik (xi1 , . . . , xik )

(2.3)

i1 ,...,ik

is called a k-body interaction. In general, the energy will contain some pieces involving k-body interactions, with k ∈ {1, 2, . . . , K}. An important feature of


real physical systems is that K is never a large number (usually K = 2 or 3), even when the number of particles N is very large. The same property holds for all measurable observables. However, for the general mathematical formulation which we shall use here, the energy can be any real-valued function on X . Once the configuration space X and the energy function are fixed, the probability µβ (x) for the system to be found in the configuration x is given by the Boltzmann distribution: 1 e−βE(x) , Z(β) = e−βE(x) . (2.4) µβ (x) = Z(β) x∈X

The real parameter T = 1/β is the temperature (and β is referred to as the inverse temperature). Note that the temperature is usually defined as T = 1/(kB β), where the value of kB , Boltzmann’s constant, depends on the unit of measure for the temperature. Here we adopt the simple choice kB = 1. The normalization constant Z(β) is called the partition function. Notice that eqn (2.4) indeed defines the density of the Boltzmann distribution with respect to some reference measure. The reference measure is usually the counting measure if X is discrete or the Lebesgue measure if X is continuous. It is customary to denote the expectation value with respect to the Boltzmann measure by angle brackets: the expectation value O(x) of an observable O(x), also called its Boltzmann average, is given by O =

µβ (x)O(x) =

x∈X

1 −βE(x) e O(x) . Z(β)

(2.5)

x∈X

Example 2.1 One intrinsic property of elementary particles is their spin. For ‘spin1/2’ particles, the spin σ takes only two values: σ = ±1. A localized spin-1/2 particle, whose only degree of freedom is the spin, is described by X = {+1, −1}, and is called an Ising spin. The energy of the spin in a state σ ∈ X in a magnetic field B is E(σ) = −B σ .

(2.6)

The Boltzmann probability of finding the spin in the state σ is µβ (σ) =

1 e−βE(σ) Z(β)

Z(β) = e−βB + eβB = 2 cosh(βB) .

The average value of the spin, called the magnetization, is σ = µβ (σ) σ = tanh(βB) .

(2.7)

(2.8)

σ∈{1,−1}

At high temperatures, T |B|, the magnetization is small. At low temperatures, the magnetization its close to its maximal value: σ = 1 if B > 0. Section 2.5 will discuss the behaviour of many Ising spins, with some more complicated energy functions.


Example 2.2 Some spin variables can have a larger space of possible values. For instance, a Potts spin with q states takes values in X = {1, 2, . . . , q}. In the presence of a magnetic field of intensity h pointing in the direction r ∈ {1, . . . , q}, the energy of the Potts spin is E(σ) = −B I(σ = r) . (2.9) In this case, the average value of the spin in the direction of the field is I(σ = r) =

exp(βB) . exp(βB) + (q − 1)

(2.10)

Example 2.3 Let us consider a single water molecule inside a closed container: for instance, inside a bottle. A water molecule H2 O is already a complicated object, but in a first approximation, we can neglect its structure and model the molecule as a point inside the bottle. The space of configurations then reduces to X = BOTTLE ⊂ R3 ,

(2.11)

where we have denoted by BOTTLE the region of R3 delimited by the container. Note that this description is not very accurate at a microscopic level. The description of the precise form of the bottle can be quite complex. On the other hand, it is a good approximation to assume that all positions of the molecule are equiprobable: the energy is independent of the particle’s position x ∈ BOTTLE. One then has: µ(x) =

1 , Z

Z = |X | ,

(2.12)

and the Boltzmann average of the particle’s position, x, is the barycentre of the bottle.

Example 2.4 In assuming that all the configurations in the previous example are equiprobable, we neglected the effect of gravity on the water molecule. In the presence of gravity our water molecule at position x has an energy E(x) = w h(x) ,

(2.13)


where h(x) is the height corresponding to the position x and w is a positive constant, determined by terrestrial attraction, which is proportional to the mass of the molecule. Given two positions x and y in the bottle, the ratio of the probabilities to find the particle at these positions is µβ (x) = exp{−βw[h(x) − h(y)]} . µβ (y)

(2.14)

For a water molecule at a room temperature of 20o C (T = 293 K), one has βw ≈ 7× 10−5 m−1 . Given a point x at the bottom of the bottle and y at a height of 20 cm, the probability to find a water molecule ‘near’ x is approximately 1.000014 times larger than the probability to find it ‘near’ y. For a tobacco-mosaic virus, which is about 2 × 106 times heavier than a water molecule, the ratio is µβ (x)/µβ (y) ≈ 1.4 × 1012 , which is very large. For a grain of sand, the ratio is so large that one never observes the grain floating at around y. Note that, while these ratios of probability densities are easy to compute, the partition function and therefore the absolute values of the probability densities can be much more complicated to estimate, and depend on the shape of the bottle. Example 2.5 In many important cases, we are given the space of configurations X and a stochastic dynamics defined on it. The most interesting probability distribution for such a system is the stationary state µst (x) (we assume that it is unique). For the sake of simplicity, we can consider a finite space X and a discrete-time Markov chain with transition probabilities {w(x → y)} (in Chapter 4 we shall recall some basic definitions concerning Markov chains). It happens sometimes that the transition rates satisfy, for any pair of configurations x, y ∈ X , the relation f (x)w(x → y) = f (y)w(y → x) ,

(2.15)

for some positive function f (x). As we shall see in Chapter 4, when this condition, called detailed balance, is satisfied (together with a few other technical conditions), the stationary state has the Boltzmann form (2.4) with e−βE(x) = f (x).

Exercise 2.1 As a particular realization of the above example, consider an 8×8 chessboard and a special piece sitting on it. At any time step, the piece will stay still (with probability 1/2) or move randomly to one of the neighbouring positions (with probability 1/2). Does this process satisfy the condition (2.15)? Which positions on the chessboard have lower and higher ‘energy’ ? Compute the partition function.

From a purely probabilistic point of view, one can wonder why one bothers to decompose the distribution µβ (x) into the two factors e−βE(x) and 1/Z(β). Of course the motivations for writing the Boltzmann factor e−βE(x) in exponential form come essentially from physics, where one knows (either exactly or to within some level


of approximation) the form of the energy. This also justifies the use of the inverse temperature β (after all, one could always redefine the energy function in such a way as to set β = 1). However, even if we adopt a mathematical viewpoint, and if we are interested in only a particular distribution µ(x) which corresponds to a particular value of the temperature, it is often illuminating to embed it into a one-parameter family as is done in the Boltzmann expression (2.4). Indeed, eqn (2.4) interpolates smoothly between several interesting situations. As β → 0 (high-temperature limit), one recovers the uniform probability distribution lim µβ (x) =

β→0

1 . |X |

(2.16)

Both the probabilities µβ (x) and the expectation values O(x) of the observables can be expressed as convergent Taylor expansions around β = 0. For small β the Boltzmann distribution can be seen as a ‘softening’ of the original distribution. In the limit β → ∞ (low-temperature limit), the Boltzmann distribution concentrates on the global maxima of the original distribution. More precisely, a configuration x0 ∈ X such that E(x) ≥ E(x0 ) for any x ∈ X is called a ground state. The minimum value of the energy E0 = E(x0 ) is called the ground state energy. We shall denote the set of ground states by X0 . It is elementary to show that, for a discrete configuration space, lim µβ (x) =

β→∞

1 I(x ∈ X0 ) , |X0 |

(2.17)

where I(x ∈ X0 ) = 1 if x ∈ X0 and I(x ∈ X0 ) = 0 otherwise. The above behaviour is summarized in physicists’ jargon by saying that, at low temperature, ‘low energy configurations dominate’ the behaviour of the system.

2.2

Thermodynamic potentials

Several properties of the Boltzmann distribution (eqn (2.4)) are conveniently summarized through the thermodynamic potentials. These are functions of the temperature 1/β and of the various parameters defining the energy E(x). The most important thermodynamic potential is the free energy F (β) = −

1 log Z(β) , β

(2.18)

where Z(β) is the partition function already defined in eqn (2.4). The factor −1/β in eqn (2.18) is due essentially to historical reasons. In calculations, it is often more convenient to use the free entropy1 Φ(β) = −βF (β) = log Z(β). 1 Unlike the other potentials, there is no universally accepted name for Φ(β); however, because this potential is very useful, we have adopted the name ‘free entropy’ for it.


Two more thermodynamic potentials are derived from the free energy: the internal energy U (β) and the canonical entropy S(β): ∂ ∂F (β) (βF (β)) , S(β) = β 2 . (2.19) ∂β ∂β By direct computation, one obtains the following identities concerning the potentials defined so far: 1 1 (2.20) F (β) = U (β) − S(β) = − Φ(β) , β β U (β) = E(x) , (2.21) S(β) = − µβ (x) log µβ (x) , (2.22) U (β) =

x

∂2 − 2 (βF (β)) = E(x)2 − E(x)2 . ∂β

(2.23)

For discrete X , eqn (2.22) can be rephrased by saying that the canonical entropy is the Shannon entropy of the Boltzmann distribution, as we defined it in Chapter 1. This implies that S(β) ≥ 0. Equation (2.23) implies that the free entropy is a convex function of the temperature. Finally, eqn (2.21) justifies the name ‘internal energy’ for U (β). In order to have some intuition of the content of these definitions, let us reconsider the high- and low-temperature limits already treated in the previous section. In the high-temperature limit, β → 0, one finds 1 (2.24) F (β) = − log |X | + E(x)0 + Θ(β) , β (2.25) U (β) = E(x)0 + Θ(β) , S(β) = log |X | + Θ(β) . (2.26) (Recall that Θ stands for ‘of the order of-’; see Appendix A.) The interpretation of these formulae is straightforward. At high temperature, the system can be found in any possible configuration with similar probabilities (the probabilities being exactly equal when β = 0). The entropy counts the number of possible configurations. The internal energy is just the average value of the energy over the configurations with uniform probability. While the high-temperature expansions (2.24)–(2.26) have the same form for both a discrete and a continuous configuration space X , in the low-temperature case we must be more careful. If X is finite, we can meaningfully define the energy gap ∆E > 0 as follows (recall that we have denoted by E0 the ground-state energy): ∆E = min{E(y) − E0 : y ∈ X \X0 } .

(2.27)

With this definition, we get 1 log |X0 | + Θ(e−β∆E ) , β E(β) = E0 + Θ(e−β∆E ) , S(β) = log |X0 | + Θ(e−β∆E ) . F (β) = E0 −

(2.28) (2.29) (2.30)

¿¼


1.0 0.5 0.0 -0.5 -1.0 -1.5 -2.0

F (β)

-2.5

U (β) S(β)

-3.0 0.0

0.5

1.0

1.5

2.0

2.5

3.0

T

Fig. 2.1 Thermodynamic potentials for a two-level system with 1 = −1 and 2 = +1 as a function of the temperature T = 1/β.

The interpretation is that, at low temperature, the system is found with equal probability in any of the ground states, and nowhere else. Once again, the entropy counts the number of available configurations, and the internal energy is the average of their energies (which coincide with that of the ground state).

Exercise 2.2 A two-level system. This is the simplest non-trivial example: X = {1, 2}, E(1) = 1 , E(2) = 2 . Without loss of generality, we assume 1 < 2 . This example can be used as a mathematical model for many physical systems, such as the spin-1/2 particle discussed above. Derive the following results for the thermodynamic potentials (where ∆ = 2 − 1 is the energy gap): F (β) = 1 −

1 log(1 + e−β∆ ) , β

(2.31)

U (β) = 1 +

e−β∆ ∆, 1 + e−β∆

(2.32)

−β∆

S(β) =

e β∆ + log(1 + e−β∆ ) . 1 + e−β∆

(2.33)

The behaviour of these functions is presented in Fig. 2.1. The reader can work out the asymptotics, and check the general high- and low-temperature behaviour given above.

Exercise 2.3 We return to the example of the previous section: one water molecule, modelled as a point, in a bottle. We consider the case of a cylindrical bottle of base B ⊂ R2 (surface area |B|) and height d.


¿½

Using the energy function in eqn (2.13), derive the following explicit expressions for the thermodynamic potentials: 1 1 1 − e−βwd log |B| − log , β β βw wd 1 , U (β) = − βwd β e −1 « „ βwd βwd . S(β) = log |Bd| + 1 − βwd − log e −1 1 − e−βwd F (β) = −

(2.34) (2.35) (2.36)

Note that the internal-energy formula can be used to compute the average height of the molecule h(x) = U (β)/w. This is a consequence of the definition of the energy (see eqn (2.13)) and of eqn (2.21). If we plug in the correct constant w, we can find that the average height falls below 49.99% of the height of the bottle d = 20 cm only when the temperature is below 3.2 K.

Exercise 2.4 Using eqns (2.34)–(2.36), derive the low-temperature expansions F (β) = −

1 log β

„

|B| βw

«

+ Θ(e−βwd ) ,

1 + Θ(e−βwd ) , β « „ |B|e + Θ(e−βwd ) . S(β) = log βw

U (β) =

(2.37) (2.38) (2.39)

In this case X is continuous, and the energy has no gap. Nevertheless, these results can be understood as follows: at low temperature, the molecule is confined to a layer of height of order 1/(βw) above the bottom of the bottle. It therefore occupies a volume of size |B|/(βw). Its entropy is approximately given by the logarithm of such a volume.

Exercise 2.5 Let us reconsider the above example and assume the bottle to have a different shape, for instance a sphere of radius R. In this case it is difficult to compute explicit expressions for the thermodynamic potentials, but one can easily compute the low-temperature expansions. For the entropy, one gets at large β „ S(β) = log

2πe2 R β 2 w2

« + Θ(1/β) .

(2.40)

The reader should try to understand the difference between this result and eqn (2.39) and provide an intuitive explanation, as in the previous example. Physicists say that the low-temperature thermodynamic potentials reveal the ‘low-energy structure’ of the system.

¿¾

2.3


The fluctuation–dissipation relations

It often happens that the energy function depends smoothly upon some real parameters. These can be related to the experimental conditions under which a physical system is studied, or to some fundamental physical quantity. For instance, the energy of a water molecule in a gravitational field (see eqn (2.13)) depends upon the weight w of the molecule itself. Although this is a constant number in the physical world, it is useful, in the theoretical treatment, to consider it as an adjustable parameter. It is therefore interesting to consider an energy function Eλ (x) which depends smoothly upon some parameter λ and admits the following Taylor expansion in the neighbourhood of λ = λ0 : ∂E (x) + O((λ − λ0 )2 ) . (2.41) Eλ (x) = Eλ0 (x) + (λ − λ0 ) ∂λ λ0 The dependence of the free energy and of other thermodynamic potentials upon λ in the neighbourhood of λ0 is easily related to the explicit dependence of the energy function itself. Let us consider the partition function, and expand it to first order in λ − λ0 : ∂E 2 Z(λ) = (x) + O((λ − λ0 ) ) exp −β Eλ0 (x) + (λ − λ0 ) ∂λ λ0 x ∂E 2 = Z(λ0 ) 1 − β(λ − λ0 ) + O((λ − λ0 ) ) , (2.42) ∂λ λ0 0

where we have denoted by ·0 the expectation with respect to the Boltzmann distribution at λ = λ0 . This shows that the free entropy behaves as ∂Φ ∂E = −β , (2.43) ∂λ λ0 ∂λ λ0 0

One can also consider the λ dependence of the expectation value of a generic observable A(x). Using again the Taylor expansion, one finds that ∂E ∂Aλ = −β A ; . (2.44) ∂λ λ0 ∂λ λ0 0

where we have denoted by A; B the connected correlation function: A; B = AB − AB. A particular example of this relation was given in eqn (2.23). The result (2.44) has important practical consequences and many generalizations. Imagine you have an experimental apparatus that allows you to tune some parameter λ (for instance the pressure of a gas, or the magnetic or electric field acting on some material) and to monitor the value of an observable A(x) (the volume of the gas, or the polarization or magnetization of the material). The quantity on the left-hand

The thermodynamic limit

¿¿

side of eqn (2.44) is the response of the system to an infinitesimal variation of the tunable parameter. On the right-hand side, we find some correlation function within the ‘unperturbed’ system. One possible application is to measure correlations within a system by monitoring its response to an external perturbation. The relation (2.44) between a correlation and a response is called the fluctuation–dissipation theorem.

2.4

The thermodynamic limit

The main purpose of statistical physics is to understand the macroscopic behaviour of a large number, N 1, of simple components (atoms, molecules, etc.) when they are brought together. To be concrete, let us consider a few drops of water in a bottle. A configuration of the system is given by the positions and orientations of all the H2 O molecules inside the bottle. In this case X is the set of positions and orientations of a single molecule, and N is typically of order 1023 (more precisely, 18 g of water contains approximately 6 × 1023 molecules). The sheer magnitude of such a number leads physicists to focus on the N → ∞ limit, also called the thermodynamic limit. As shown by the examples below, for large N , the thermodynamic potentials are often proportional to N . One is thus led to introduce the intensive thermodynamic potentials as follows. Let us denote by FN (β), UN (β), and SN (β) the free energy, internal energy, and canonical entropy, respectively. for a system with N ‘particles’. The free energy density is defined by f (β) = lim FN (β)/N , N →∞

(2.45)

if the limit exists, which is usually the case (at least if the forces between particles decrease fast enough at large distance). One defines analogously the energy density u(β) and the entropy density s(β). The free energy FN (β), is, quite generally, an analytic function of β in a neighbourhood of the real β axis. This is a consequence of the fact that Z(β) is analytic throughout the entire β plane, and strictly positive for real β’s. A question of great interest is whether analyticity is preserved in the thermodynamic limit (2.45), under the assumption that the limit exists. Whenever the free energy density f (β) is nonanalytic, one says that a phase transition occurs. Since the free entropy density φ(β) = −βf (β) is convex, the free energy density is necessarily continuous whenever it exists. In the simplest cases, the non-analyticities occur at isolated points. Let βc be such a point. Two particular types of singularities occur frequently: • The free energy density is continuous, but its derivative with respect to β is discontinuous at βc . This singularity is called a first-order phase transition. • The free energy and its first derivative are continuous, but the second derivative is discontinuous at βc . This is called a second-order phase transition. Higher-order phase transitions can be defined as well, along the same lines. Apart from being interesting mathematical phenomena, phase transitions correspond to qualitative changes in the underlying physical system. For instance, the

¿


transition from water to vapor at 100 o C at normal atmospheric pressure is modelled mathematically as a first-order phase transition in the above sense. A great part of this book will be devoted to the study of phase transitions in many different systems, where the interacting ‘particles’ can be very diverse objects such as bits of information or occupation numbers on the vertices of a graph. When N grows, the volume of the configuration space increases exponentially: |XN | = |X |N . Of course, not all of the configurations are equally important under the Boltzmann distribution: the lowest-energy configurations have greater probability. What is important is therefore the number of configurations at a given energy. This information is encoded in the energy spectrum of the system, N∆ (E) = |Ω∆ (E)| ,

Ω∆ (E) ≡ {x ∈ XN : E ≤ E(x) < E + ∆} .

(2.46)

In many systems of interest, the energy spectrum diverges exponentially as N → ∞, if the energy is scaled linearly with N . More precisely, there exists a function s(e) such that, given two numbers e and δ > 0, 1 log NN δ (N e) = sup s(e ) . N →∞ N e ∈[e,e+δ] lim

(2.47)

The function s(e) is called the microcanonical entropy density. The statement (2.47) is often rewritten in the more compact form E . . (2.48) N∆ (E) =N exp N s N . The notation AN =N BN will be used throughout the book to denote that two quantities AN and BN (which behave exponentially in N ) are equal to leading exponential . order, meaning limN →∞ (1/N ) log(AN /BN ) = 0. We shall often use = without an index when there is no ambiguity about the large variable N . The microcanonical entropy density s(e) conveys a great amount of information about the system. Furthermore, it is directly related to the intensive thermodynamic potentials through a fundamental relation Proposition 2.6 If the microcanonical entropy density (2.47) exists for any e and if the limit in eqn (2.47) is uniform in e, then the free entropy density (2.45) exists and is given by φ(β) = max[s(e) − βe] . e

(2.49)

If the maximum of s(e)−βe is unique, then the internal-energy density equals arg max[s(e)− βe]. Proof The basic idea is to write the partition function as ∞ . . N∆ (k∆) e−β∆ = exp{N s(e) − N βe} de , ZN (β) =

(2.50)

k=−∞

and to evaluate the last integral by the saddle point method. The reader will find references in the Notes section at the end of this chapter.

Ferromagnets and Ising models

¿

Example 2.7 Let us consider N identical two-level systems, i.e. XN = X × · · · × X , with X = {1, 2}. We take the energy to be the sum of the single-system energies: E(x) = Esingle (x1 ) + · · · + Esingle (xN ), with xi ∈ X . As in the previous section, we set Esingle (1) = 1 , Esingle (2) = 2 > 1 , and ∆ = 2 − 1 . The energy spectrum of this model is quite simple. For any energy E = N 1 +n∆, there are N n configurations x with E(x) = E. Therefore, using the definition (2.47), we get e − 1 s(e) = H . (2.51) ∆ Equation (2.49) can now be used to get f (β) = 1 −

1 log(1 + e−β∆ ) , β

(2.52)

which agrees with the result obtained directly from the definition (2.18). The great attention paid by physicists to the thermodynamic limit is extremely well justified by the huge number of degrees of freedom involved in a macroscopic piece of matter. Let us stress that the interest of the thermodynamic limit is more general than these huge numbers might suggest. First of all, it often happens that fairly small systems are well approximated by the thermodynamic limit. This is extremely important for numerical simulations of physical systems: one cannot, of course, simulate 1023 molecules on a computer! Even the cases in which the thermodynamic limit is not a good approximation are often fruitfully analysed as violations of this limit. Finally, the insight gained from analysing the N → ∞ limit is always crucial in understanding moderate-size systems.

2.5


Magnetic materials contain molecules with a magnetic moment, a three-dimensional vector which tends to align with the magnetic field felt by the molecule. Moreover, the magnetic moments of two different molecules interact with each other. Quantum mechanics plays an important role in magnetism. Because of quantum effects, the space of possible configurations of a magnetic moment becomes discrete. Quantum effects are also the origin of the ‘exchange interaction’ between magnetic moments. In many materials, the effect of the exchange interaction is such that the energy is lower when two moments align. While the behaviour of a single magnetic moment in an external field is qualitatively simple, when we consider a bunch of interacting moments, the problem is much richer, and exhibits remarkable collective phenomena. A simple mathematical model for such materials is the Ising model. This describes the magnetic moments by Ising spins localized at the vertices of a certain region of a d-dimensional cubic lattice. To keep things simple, let us consider a region L which is a cube of side L: L = {1, . . . , L}d . On each site i ∈ L, there is an Ising spin σi ∈ {+1, −1} (see Fig. 2.2).

¿


Fig. 2.2 A configuration of a two-dimensional Ising model with L = 5. There is an Ising spin σi on each vertex i, shown by an arrow pointing up if σi = +1 and pointing down if σi = −1. The energy (2.53) is given by the sum of two types of contributions: (i) a term −σi σj for each edge (ij) of the graph, such that the energy is minimized when the two neighbouring spins σi and σj point in the same direction; and (ii) a term −Bσi for each site i, due to the coupling to an external magnetic field. The configuration depicted here has an energy −8 + 9B.

A configuration σ = (σ1 . . . σN ) of the system is given by assigning the values of all the spins in the system. Therefore, the space of configurations XN = {+1, −1}L has the form (2.1), with X = {+1, −1} and N = Ld . The definition of a ferromagnetic Ising model is completed by the definition of the energy function. A configuration σ has an energy E(σ) = − σi σj − B σi , (2.53) (ij)

i∈L

where the sum over (ij) runs over all the (unordered) pairs of sites i, j ∈ L which are nearest neighbours. The real number B measures the applied external magnetic field. Determining the free energy density f (β) in the thermodynamic limit for this model is a non-trivial task. The model was invented by Wilhem Lenz in the early 1920s, who assigned the task of analysing it to his student Ernst Ising. In his thesis of 1924, Ising solved the d = 1 case and showed the absence of phase transitions. In 1948, Lars Onsager brilliantly solved the d = 2 case, exhibiting the first soluble ‘finite-dimensional’ model with a second-order phase transition. In higher dimensions, the problem is unsolved, although many important features of the solution are well understood. Before embarking on any calculations, let us discuss some qualitative properties of this model. Two limiting cases are easily understood. At infinite temperature, β = 0, the energy (2.53) no longer matters and the Boltzmann distribution weights all the configurations with the same factor 2−N . We have therefore an assembly of completely independent spins. At zero temperature, β → ∞, the Boltzmann distribution concen-


¿

trates onto the ground state(s). If there is no magnetic field, i.e. B = 0, there are two degenerate ground states: the configuration σ (+) with all the spins pointing up, σi = +1, and the configuration σ (−) with all the spins pointing down, σi = −1. If the magnetic field is set to some non-zero value, one of the two configuration dominates: σ (+) if B > 0 and σ (−) if B < 0. Notice that the reaction of the system to the external magnetic field B is quite different in the two cases. To see this fact, define a ‘rescaled’ magnetic field x = βB and take the limits β → 0 and β → ∞ keeping x fixed. The expected value of any spin in L, in the two limits, is tanh(x) for β → , (2.54) σi = tanh(N x) for β → ∞ . Each spin reacts independently for β → 0. In contrast, they react as a whole as β → ∞: one says that the response is cooperative. A useful quantity for describing the response of the system to the external field is the average magnetization, MN (β, B) =

1 σi . N

(2.55)

i∈L

Because of the symmetry between the up and down directions, MN (β, B) is an odd function of B. In particular, MN (β, 0) = 0. A cooperative response can be emphasized by considering the spontaneous magnetization M+ (β) = lim

lim MN (β, B) .

B→0+ N →∞

(2.56)

It is important to understand that a non-zero spontaneous magnetization can appear only in an infinite system: the order of the limits in eqn (2.56) is crucial. Our analysis so far has shown that a spontaneous magnetization exists at β = ∞: M+ (∞) = 1. On the other hand, M+ (0) = 0. It can be shown that the spontaneous magnetization M+ (β) is always zero in a high temperature phase defined by β < βc (d) (such a phase is called paramagnetic). In one dimension (d = 1), we shall show below that βc (1) = ∞. The spontaneous magnetization is always zero, except at zero temperature (β = ∞): one speaks of a zero-temperature phase transition. In dimensions d ≥ 2, βc (d) is finite, and M+ (β) becomes non-zero in the ferromagnetic phase , i.e. for β > βc : a phase transition takes place at β = βc . The temperature Tc = 1/βc is called the critical temperature. In the following, we shall discuss the d = 1 case, and a variant of the model, called the Curie–Weiss model, where each spin interacts with all the other ones: this is the simplest model which exhibits a finite-temperature phase transition. 2.5.1

The one-dimensional case

The d = 1 case has the advantage of being simple to solve. We want to compute the N −1 partition function (2.4) for a system of N spins with energy E(σ) = − i=1 σi σi+1 − N B i=1 σi . We shall use the transfer matrix method, which belongs to the general dynamic programming strategy familiar to computer scientists.

¿


We introduce a partial partition function, where the configurations of all spins σ1 ,. . . , σp have been summed over, at fixed σp+1 : p p (2.57) exp β σi σi+1 + βB σi . zp (β, B, σp+1 ) ≡ σ1 ,...,σp

i=1

i=1

The partition function (2.4) is given by ZN (β, B) = σN zN −1 (β, B, σN ) exp(βBσN ). Obviously, zp satisfies the recursion relation T (σp+1 , σp )zp−1 (β, B, σp ) (2.58) zp (β, B, σp+1 ) = σp =±1

where we have defined the transfer matrix T (σ, σ ) = exp [βσσ + βBσ ]. This is the 2 × 2 matrix β+βB −β−βB e e (2.59) T = e−β+βB eβ−βB exp(βB) 1 and ψR = , and the Introducing the two-component vectors ψL = exp(−βB) 1 standard scalar product between vectors (a, b) = a1 b1 + a2 b2 , the partition function can be written in matrix form: ZN (β, B) = (ψL , T N −1 ψR ) .

(2.60)

Let us call the eigenvalues of T λ1 , λ2 , and the corresponding eigenvectors ψ1 , ψ2 . It is easy to realize that ψ1 , ψ2 can be chosen to be linearly independent, and hence ψR can be decomposed as ψR = u1 ψ1 + u2 ψ2 . The partition function is then expressed as −1 −1 + u2 (ψL , ψ2 ) λN . ZN (β, B) = u1 (ψL , ψ1 ) λN 1 2

The diagonalization of the matrix T gives λ1,2 = eβ cosh(βB) ± e2β sinh2 βB + e−2β .

(2.61)

(2.62)

For β finite, in the large-N limit, the partition function is dominated by the largest eigenvalue λ1 , and the free-entropy density is given by φ = log λ1 : φ(β, B) = log eβ cosh(βB) + e2β sinh2 βB + e−2β . (2.63) Using the same transfer matrix technique, we can compute expectation values of observables. For instance, the expected value of a given spin is σi =

1 (ψL , T i−1 σ ˆ T N −i ψR ) , ZN (β, B)

where σ ˆ is the following matrix:

(2.64)


¿

1 0.8

β = 1.5

β=1

0.6 0.4

β = 0.5

M

0.2 0 -0.2 -0.4 -0.6 -0.8 -1 -1

-0.5

0 B

0.5

1

Fig. 2.3 The average magnetization of the one-dimensional Ising model, as a function of the magnetic field B, at inverse temperatures β = 0.5, 1, 1.5.

σ ˆ=

1 0 0 −1

.

(2.65)

Averaging over the position i, one can compute the average magnetization MN (β, B). In the thermodynamic limit, we get lim MN (β, B) =

N →∞

sinh βB 2

sinh βh +

e−4β

=

1 ∂φ (β, B) . β ∂B

(2.66)

Both the free energy and the average magnetization turn out to be analytic functions of β and B for β < ∞. In particular the spontaneous magnetization vanishes at any non-zero temperature: M+ (β) = 0 ,

∀β < ∞ .

(2.67)

In Fig. 2.3, we plot the average magnetization M (β, B) ≡ limN →∞ MN (β, B) as a function of the applied magnetic field B for various values of the temperature β. The curves become steeper and steeper as β increases. This statement can be made more quantitative by computing the susceptibility associated with the average magnetization, ∂M (β, 0) = β e2β . (2.68) ∂h This result can be interpreted as follows. A single spin in a field has a susceptibility χ(β) = β. If we consider N spins constrained to take the same value, the corresponding susceptibility will be N β, as in eqn (2.54). In the present case, the system behaves as if the spins were grouped into blocks of χM (β)/β spins each. The spins in each group are constrained to take the same value, while spins belonging to different blocks are independent. This qualitative interpretation can be given further support by computing a correlation function. χM (β) =

¼


Exercise 2.6 Consider the one-dimensional Ising model in zero field, B = 0. Show that when δN < i < j < (1 − δ)N , the correlation function σi σj is, in the large-N limit, σi σj = e−|i−j|/ξ(β) + Θ(e−αN ) ,

(2.69)

where ξ(β) = −1/ log tanh β. [Hint: You can either use the general transfer matrix formalism or, more simply, use the identity eβσi σi+1 = cosh β(1 + σi σi+1 tanh β).]

Note that, in eqn (2.69), ξ(β) gives the typical distance below which two spins in the system are well correlated. For this reason, it is usually called the correlation length of the model. This correlation length increases as the temperature decreases: spins become correlated at larger and larger distances. The result (2.69) is clearly consistent with our interpretation of the susceptibility. In particular, as β → ∞, ξ(β) ≈ e2β /2 and χM (β) ≈ 2βξ(β). The connection between correlation length and susceptibility is very general and can be understood as a consequence of the fluctuation–dissipation theorem (2.44): χM (β) = βN

=

N 1 σi N i=1

;

N 1 σi N i=1

N N β β σi ; σj = σi σj , N i,j=1 N i,j=1

(2.70)

where the last equality comes from the fact that σi = 0 when B = 0. Using eqn (2.69), we get χM (β) = β

+∞

e−|i|/ξ(β) + Θ(e−αN ) .

(2.71)

i=−∞

It is therefore evident that a large susceptibility must correspond to a large correlation length. 2.5.2

The Curie–Weiss model

The exact solution of the one-dimensional model led Ising to think that there could not be a phase transition for any dimension. Some thirty years earlier, a qualitative theory of ferromagnetism had been put forward by Pierre Curie. Such a theory assumed the existence of a phase transition at a non-zero temperature Tc (the ‘Curie point’) and a non-vanishing spontaneous magnetization for T < Tc . The dilemma was eventually solved by Onsager’s solution of the two-dimensional model. Curie’s theory is realized exactly within a rather abstract model: the Curie–Weiss model. We shall present it here as one of the simplest solvable models with a finitetemperature phase transition. Once again, we have N Ising spins σi ∈ {±1}, and a


½

configuration is given by σ = (σ1 , . . . , σN ). However, the spins no longer sit on a ddimensional lattice: they all interact in pairs. The energy function, in the presence of a magnetic field B, is given by E(σ) = −

N 1 σi σj − B σi , N i=1

(2.72)

(ij)

where the sum over (ij) runs over all of the N (N − 1)/2 couples of spins. Notice the peculiar 1/N scaling in front of the exchange term. The exact solution presented below shows that this is the only choice which yields a non-trivial free energy density in the thermodynamic limit. This can be easily understood intuitively as follows. The sum over (ij) involves Θ(N 2 ) terms of order Θ(1). In order to get an energy function that scales as N , we need to put a coefficient 1/N in front. In adopting the energy function (2.72), we gave up an attempt to describe any finite-dimensional geometrical structure. This is a severe simplification, but has the advantage of making the model exactly soluble. The Curie–Weiss model is the first example of a large family: the mean-field models. We shall explore many instances of this family throughout the book. A possible approach to the computation of the partition function consists in observing that the energy function can be written in terms of a simple observable, the instantaneous (or empirical) magnetization, m(σ) =

N 1 σi . N i=1

(2.73)

Notice that this is a function of the configuration σ, and should not be confused with its expected value, the average magnetization (see eqn (2.55)). It is a ‘simple’ observable because it is equal to a sum of observables depending upon a single spin. We can write the energy of a configuration in terms of its instantaneous magnetization: E(σ) =

1 1 N − N m(σ)2 − N B m(σ) . 2 2

This implies the following formula for the partition function: Nβ 2 −N β/2 ZN (β, B) = e m + N βBm , NN (m) exp 2 m

(2.74)

(2.75)

where the sum over m runs over all of the possible instantaneous magnetizations of N Ising spins: m = −1 + 2k/N with 0 ≤ k ≤ N , where k is an integer number, and NN (m) is the number of configurations that have a given instantaneous magnetization m. This is a binomial coefficient whose large-N behaviour can be expressed in terms of the entropy function of Bernoulli variables: 1+m N . . (2.76) = exp N H NN (m) = N (1 + m)/2 2

¾


To leading exponential order in N , the partition function can thus be written as +1 . ZN (β, B) = eN φmf (m;β,B) dm , (2.77) −1

where we have defined β φmf (m; β, B) = − (1 − m2 ) + βBm + H 2

1+m 2

.

(2.78)

The integral in eqn (2.77) is easily evaluated by the Laplace method, to get the final result for the free energy density φ(β, B) =

max m∈[−1,+1]

φmf (m; β, B) .

(2.79)

One can see that the maximum is obtained away from the boundary points, so that the corresponding m must be a stationary point of φmf (m; β, B), which satisfies the saddle point equation ∂φmf (m; β, B)/∂m = 0: m∗ = tanh(βm∗ + βB) .

(2.80)

In the above derivation, we were slightly sloppy in two steps: substituting the binomial coefficient by its asymptotic form, and changing the sum over m into an integral. The mathematically minded reader is invited to show that these passages are indeed correct. With a little more work, the above method can be extended to expectation values of observables. Let us consider, for instance, the average magnetization M (β, B). It can be easily shown that, whenever the maximum of φmf (m; β, B) over m is nondegenerate, M (β, B) ≡ lim m(σ) = m∗ (β, B) ≡ arg max φmf (m; β, B) . N →∞

m

(2.81)

We can now examine the implications that can be drawn from Eqs. (2.79) and (2.80). Let us consider first the B = 0 case (see Fig. 2.4). The function φmf (m; β, 0) is symmetric in m. For 0 ≤ β ≤ 1 ≡ βc , it is also concave and achieves its unique maximum at m∗ (β) = 0. For β > 1, m = 0 remains a stationary point but becomes a local minimum, and the function develops two degenerate global maxima at m± (β), with m+ (β) = −m− (β) > 0. These two maxima bifurcate continuously from m = 0 at β = βc . A phase transition takes place at βc . Its meaning can be understood by computing the expectation value of the spins. Notice that the energy function (2.72) is symmetric under a spin-flip transformation which maps σi → −σi for all i. Therefore σi = (−σi ) = 0, and the average magnetization vanishes, i.e. M (β, 0) = 0. On the other hand, the spontaneous magnetization, defined in eqn (2.56), is zero in the paramagnetic phase, i.e. for β < βc , and equal to m+ (β) in the ferromagnetic phase for β > βc . The physical interpretation of this phase is the following: for any finite N , the pdf of the instantaneous magnetization m(σ) has two symmetric peaks, at m± (β),

¿

Ferromagnets and Ising models 0.4

1

0.3

0.5

φ

m

0.2

0

0.1

− 0.5

0 −1

− 0.5

0

1

0.5

m

−1 0

1

2

3

β

Fig. 2.4 Left: the function φmf (m; β, B = 0) plotted versus m, for β = 0.7, 0.9, 1.1, 1.3 (from top to bottom). For β < βc = 1, there is a unique maximum at m = 0; for β < βc = 1, there are two degenerate maxima at two symmetric values ±m+ (β). Right: the values of m which maximize φmf (m; β, B = 0), plotted versus β. The phase transition at βc = 1 is signalled by a bifurcation.

which become sharper and sharper as N increases. Any external perturbation which breaks the symmetry between the peaks, for instance a small positive magnetic field B, favours one peak with respect to the other one, and therefore the system develops a spontaneous magnetization. Let us stress that the occurrence of a phase transition is a property of systems in the thermodynamic limit N → ∞. In physical magnets, symmetry breaking can arise, for instance, from impurities, subtle effects of dipolar interactions together with the shape of the magnet, or an external magnetic field. The result is that at low enough temperatures some systems, the ferromagnets, develop a spontaneous magnetization. If a magnet made of iron is heated, its magnetization disappears at a critical temperature Tc = 1/βc ≈ 770 o C. The Curie Weiss model is a simple solvable case exhibiting this phase transition.

Exercise 2.7 Compute the expansions of m+ (β) and of φ(β, B = 0) near β = βc , and show that the transition is of second order. Compute the low-temperature behaviour of the spontaneous magnetization.

Exercise 2.8 Inhomogeneous Ising chain. The one-dimensional Ising problem does not have a finite-temperature phase transition, as long as the interactions are short-range and translationally invariant. On the other hand, if the couplings in the Ising chain grow fast enough at large distance, one can have a phase transition. This is not a very realistic model from the point of view of physics, but it is useful as a solvable example of a phase transition.

Statistical physics and probability theory P −1 Consider a chain of Ising spins σ0 , σ1 , . . . , σN with energy E(σ) = − N n=0 Jn σn σn+1 . Suppose that the coupling constants Jn form a positive, monotonically increasing sequence, growing logarithmically. More precisely, we assume that limn→∞ Jn / log n = 1. Denote by · + and · − the expectation value with respect to the Boltzmann probability distribution when the spin σN is fixed at σN = +1 and −1, respectively. Q −1 (a) Show that, for any n ∈ {0, . . . , N − 1}, the magnetization is σn ± = N p=n tanh(βJp ) (b) Show that the critical inverse temperature βc = 1/2 separates two regimes, such that for β < βc , one has limN →∞ σn + = limN →∞ σn − = 0, and for β > βc , one has limN →∞ σn ± = ±M (β), with M (β) > 0. Notice that in this case, the role of the symmetry-breaking field is played by the choice of boundary condition.

2.6

The Ising spin glass

In real magnetic materials, localized magnetic moments are subject to several sources of interaction. Apart from the exchange interaction mentioned in the previous section, they may interact through intermediate conduction electrons, for instance. As a result, depending on the material considered, their interaction can be either ferromagnetic (their energy is minimized when they are parallel) or antiferromagnetic (their energy is minimized when they point opposite to each other). Spin glasses are a family of materials whose magnetic properties are particularly complex. They can be produced by diluting a small fraction of a magnetic transition metal such as manganese into a noble metal such as copper in a ratio, say, of 1 : 100. In such an alloy, magnetic moments are localized at manganese atoms, which are placed at random positions in a copper background. Depending on the distance between two manganese atoms, the net interaction between their magnetic moments can be either ferromagnetic or antiferromagnetic. The Edwards–Anderson model is a widely accepted mathematical abstraction of these physical systems. Once again, the basic degrees of freedom are Ising spins σi ∈ {−1, +1} sitting on the vertices of a d-dimensional cubic lattice L = {1, . . . , L}d , i ∈ L. The configuration space is therefore {−1, +1}L . As in the ferromagnetic Ising model, the energy function reads E(σ) = −

(ij)

Jij σi σj − B

σi ,

(2.82)

i∈L

where (ij) runs over each edge of the lattice. Unlike the case of the Ising ferromagnet, however, a different coupling constant Jij is now associated with each edge (ij), and its sign can be positive or negative. The interaction between spins σi and σj is ferromagnetic if Jij > 0 and antiferromagnetic if Jij < 0. A pictorial representation of this energy function is given in Fig. 2.5. The Boltzmann distribution is given by

The Ising spin glass

Fig. 2.5 A configuration of a two-dimensional Edwards–Anderson model with L = 5. Spins are coupled by two types of interactions: ferromagnetic (Jij = +1), indicated by a continuous line, and antiferromagnetic (Jij = −1), indicated by a dashed line. The energy of the configuration shown here is −14 − 7h.

⎧ ⎫ ⎨ ⎬ 1 exp β µβ (σ) = Jij σi σj + βB σi , ⎩ ⎭ Z(β) i∈L (ij) ⎧ ⎫ ⎨ ⎬ exp β Jij σi σj + βB σi . Z(β) = ⎩ ⎭ σ

(ij)

(2.83)

(2.84)

i∈L

It is important to notice that the couplings {Jij } play a completely different role from the spins {σi }. The couplings are just parameters involved in the definition of the energy function, like the magnetic field B, and they are not summed over when computing the partition function. In principle, for any particular sample of a magnetic material, one should estimate experimentally the values of the Jij ’s and then compute the partition function. We could have made explicit the dependence of the partition function and of Boltzmann distribution on the couplings by using notation such as Z(β, B; {Jij }), µβ,B;{Jij } (σ). However, when these explicit mentions are not necessary, we prefer to keep to lighter notation. The present understanding of the Edwards–Anderson model is much poorer than for the ferromagnetic models introduced in the previous section. The basic reason for this difference is frustration, which is illustrated in Fig. 2.6 for an L = 2, d = 2 model (a model consisting of just four spins). A spin glass is frustrated whenever there exist local constraints that are in conflict, meaning that it is not possible to satisfy all of them simultaneously. In the Edwards– Anderson model, a plaquette is the name given to a group of four neighbouring spins forming a square (i.e. a cycle of length four). A plaquette is frustrated if and only if the product of the Jij ’s along all four edges of the plaquette is negative. As shown in Fig. 2.6, it is then impossible to minimize simultaneously all of the four local energy


Fig. 2.6 Four configurations of a small Edwards–Anderson model: continuous lines indicate ferromagnetic interactions (Jij = +1), and dashed lines indicate antiferromagnetic interactions (Jij = −1). In zero magnetic field (B = 0), the four configurations are degenerate and have energy E = −2. The double bar indicates an unsatisfied interaction. Notice that there is no configuration with a lower energy. This system is frustrated since it is impossible to satisfy simultaneously all constraints.

terms associated with each edge. In a spin glass, the presence of a finite density of frustrated plaquettes generates a very complicated energy landscape. The resulting effect of all the interactions is not obtained by ‘summing’ the effects of each of them separately, but is the outcome of a complex interplay. The ground state spin configuration (the one satisfying the largest possible number of interactions) is difficult to find: it cannot be guessed on symmetry grounds. It is also frequent to find in a spin glass a configuration which is very different from the ground state but has an energy very close to the ground state energy. We shall explore these and related issues throughout the book.

Notes There are many good introductory textbooks on statistical physics and thermodynamics, for instance Reif (1965) and Huang (1987). Going towards more advanced texts, we can suggest the books by Ma (1985) and Parisi (1988). A more mathematically minded presentation can be found in the books by Gallavotti (1999) and Ruelle (1999). The reader will find there a proof of Proposition 2.6. The two-dimensional Ising model in a vanishing external field can also be solved by a transfer matrix technique: see for instance Baxter (1982). The transfer matrix, which links one column of the lattice to the next, is a 2L ×2L matrix, and its dimension diverges exponentially with the lattice size L. Finding its largest eigenvalue is therefore a complicated task. No one has found the solution so far for B = 0. Spin glasses will be a recurring theme in this book, and more will be said about them in the following chapters. An introduction to this subject from a physicist’s point of view is provided by the book by Fischer and Hertz (1993) and the review by Binder and Young (1986). The concept of frustration was introduced by Toulouse (1977).

3 Introduction to combinatorial optimization This chapter provides an elementary introduction to some basic concepts in theoretical computer science. Which computational tasks can/cannot be accomplished efficiently by a computer? How much resources (time, memory, etc.) are needed for solving a specific problem? What is the performance of a specific solution method (an algorithm), and, whenever more than one method is available, which one is preferable? Are some problems intrinsically harder than others? These are some of the questions one would like to answer. One large family of computational problems is formed by combinatorial optimization problems. These consist in finding an element of a finite set which maximizes (or minimizes) an easy-to-evaluate objective function. Several features make such problems particularly interesting. First of all, most of the time they are equivalent to decision problems (questions which require a yes/no answer), which is the most fundamental class of problems within computational-complexity theory. Second, optimization problems are ubiquitous both in applications and in the pure sciences. In particular, there exist some evident connections both with statistical mechanics and with coding theory. Finally, they form a very large and well-studied family, and therefore an ideal context for understanding some advanced issues. One should, however, keep in mind that computation is more than just combinatorial optimization. A larger family, which we shall also discuss later on, contains the counting problems: one wants to count how many elements of a finite set have some easy-to-check property. There are also other important families of computational problems that we shall not address at all, such as continuous optimization problems. The study of combinatorial optimization is introduced in Section 3.1 through the simple example of the minimum spanning tree. This section also contains the basic definitions of graph theory that we shall use throughout the book. General definitions and terminology are given in Section 3.2. These definitions are illustrated further in Section 3.3 with several additional examples. Section 3.4 provides an informal introduction to some basic concepts in computational complexity: we define the classes P and NP, and the notion of NP-completeness. As mentioned above, combinatorial optimization problems often appear bothe in the pure sciences and in applications. The examples of statistical physics and coding are briefly discussed in Sections 3.5 and 3.6.

Introduction to combinatorial optimization

g

1

f

e

6

4

a

5

3

7

5

2

b

2

c

5

d

Fig. 3.1 This graph has seven vertices (labelled a to g) and 10 edges. The ‘cost’ of each edge is indicated next to it. In the minimum-spanning-tree problem, one seeks a loop-free subgraph of minimum cost connecting all vertices.

3.1

A first example: The minimum spanning tree

The minimum-spanning-tree problem is easily stated and may appear in many practical applications. Suppose, for instance, you have a collection of computers in a building. You may want to connect them pairwise in such a way that the resulting network is connected and the amount of cable used is a minimum. 3.1.1

Definition and basics of graph theory

A mathematical abstraction of the above practical problem requires a few basic definitions from graph theory. A graph is a set V of vertices, labelled by {1, 2, . . . , |V|}, together with a set E of edges connecting them: G = (V, E). The vertex set can be any finite set but one often takes the set of the first |V| integers V = {1, 2, . . . , |V|}. The edges are simply unordered pairs of distinct vertices E ⊆ V × V. For instance, an edge joining vertices i and j is identified by e = (i, j). A weighted graph is a graph where a cost (a real number) is associated with every edge. The degree of a vertex is the number of edges connected to it. A path between two vertices i and j is a set of edges {(j, i2 ); (i2 , i3 ); (i3 , i4 ); . . . ; (ir−1 , ir ); (ir , j)} ⊆ E. A graph is connected if, for every pair of vertices, there is a path which connects them. A completely connected graph, or complete graph, also called a clique, is a graph where all of the |V|(|V| − 1)/2 edges are present. A cycle is a path starting and ending on the same vertex. A tree is a connected graph without cycles. Consider the graph in Fig. 3.1. You are asked to find a tree (a subset of the edges forming a cycle-free subgraph) such that any two vertices are connected by exactly one path (in this case the tree is said to be ‘spanning’). To find such a subgraph is an easy task. The edges {(a, b); (b, c); (c, d); (b, g); (d, e)}, for instance, do the job. However in our problem a cost is associated with each edge. The cost of a subgraph is assumed to be equal to the sum of the costs of its edges, and you want to minimize it. This is a non-trivial problem. In general, an instance of the minimum-spanning-tree (MST) problem is given by a connected weighted graph (where each edge e has a cost w(e) ∈ R). The optimization problem consists in finding a spanning tree with minimum cost. What one seeks

A first example: The minimum spanning tree

is an algorithm which, given an instance of the MST problem, outputs the spanning tree with lowest cost. 3.1.2

An efficient algorithm

The simple-minded approach would consist in enumerating all the spanning trees for the given graph, and comparing their weights. However, the number of spanning trees grows very rapidly with the size of the graph. Consider, as an example, the complete graph on N vertices. The number of spanning trees of such a graph is, according to the Cayley formula, N N −2 . Even if the cost of any such tree were evaluated in 10−3 s, it would take two years to find the MST of an N = 12 graph, and half a century for N = 13. At the other extreme, if the graph is very simple, it may contain only a small number of spanning trees, a single one in the extreme case where the graph is itself a tree. Nevertheless, in most of the interesting examples, the situation is nearly as dramatic as in the complete-graph case. A much better algorithm can be obtained from the following theorem Theorem 3.1 Let U ⊂ V be a proper subset of the vertex set V (such that neither U nor V\U is empty). Consider the subset F of edges which connect a vertex in U to a vertex in V\U, and let e ∈ F be an edge of lowest cost in this subset: w(e) ≤ w(e ) for any e ∈ F. If there are several such edges, e can be any one of them. There exists then a minimum spanning tree which contains e. Proof Consider an MST T , and suppose that it does not contain the edge e mentioned in the statement of the theorem. This edge is such that e = (i, j), with i ∈ U and j ∈ V\U. The spanning tree T must contain a path between i and j. This path contains at least one edge f connecting a vertex in U to a vertex in V\U, and f is distinct from e. Now consider the subgraph T built from T by removing the edge f and adding the edge e. We leave to the reader the exercise of showing that T is a spanning tree. If we denote by E(T ) the cost of tree T , E(T ) = E(T ) + w(e) − w(f ). Since T is an MST, E(T ) ≥ E(T ). On the other hand, e has the minimum cost within F, and hence w(e) ≤ w(f ). Therefore w(e) = w(f ) and T is an MST containing e. This result allows one to construct a minimum spanning tree of G incrementally. One starts from a single vertex. At each step a new edge is added to the tree, where the cost of this edge is minimum among all the edges connecting the already existing tree with the remaining vertices. After N − 1 iterations, the tree will be spanning. The pseudo-code for this algorithm is as follows. MST algorithm (graph G = (V, E), weight function w : E → R+ ) 1: Set U := {1}, T := ∅ and E = 0; 2: while V\U is not empty: 3: Let F := {e = (i, j) ∈ E such that i ∈ U, j ∈ V\U}; 4: Find e∗ = (i∗ , j∗ ) := arg mine∈F {w(e)}; 5: Set U := U ∪ j∗ , T := T ∪ e∗ , and E := E + w(e∗ ); 6: end 7: return the spanning tree T and its cost E.

¼


h

1

f

g

4

e

5

a

3 2

7

b

2

c

d

Fig. 3.2 A minimum spanning tree for the graph defined in Fig. 3.1. The cost of this tree is E = 17.

Exercise 3.1 Write a code for this algorithm, and find an MST for the problem described in Fig. 3.1. A solution is given in Fig. 3.2.

Exercise 3.2 Show explicitly that the algorithm MST always outputs a minimum spanning tree. Theorem 3.1 establishes that, for any U ⊂ V and any lowest-cost edge e among the ones connecting U to V\U , there exists an MST containing e. This does not guarantee, however, that when two different sets U1 and U2 and the corresponding lowest-cost edges e1 and e2 are considered, there exists an MST containing both e1 and e2 . The above algorithm works by constructing a sequence of such U ’s and adding to the tree the corresponding lowestweight edges. It is therefore not obvious a priori that it will output an MST (unless this is unique).

Let us analyse the number of elementary operations required by the algorithm to construct a spanning tree on a graph with N vertices. By ‘elementary operation’ we mean comparisons, sums, multiplications, etc., all of them counting as one operation. Of course, the number of such operations depends on the graph, but we can find a simple upper bound by considering the completely connected graph. Most of the operations in the above algorithm are comparisons of edge weights for finding e∗ in step 4. In order to identify e∗ , one has to scan at most |U| × |V\U| = |U| × (N − |U|) edges connecting U to V\U. Since |U| = 1 at the beginning and is augmented by one element at each iteration of the cycle 2–6, the number of comparisons is bounded from N above by U =0 U (N − U ) ≤ N 3 /6.1 This is an example of a polynomial algorithm, whose computing time grows like a power of the number of vertices. The insight gained from the above theorem provides an algorithm which is much better than the naive one, at least when N is large.

1 The

algorithm can easily be improved by keeping an ordered list of the edges already encountered

General definitions

½

Exercise 3.3 Suppose you are given a weighted graph (V, E) in which the weights are all different, and the edges are ordered in such a way that their weights form an increasing sequence w(e1 ) < w(e2 ) < w(e3 ) < · · · . Another graph with the same (V, E) has different weights w (e), but they are also increasing in the same sequence, i.e. w (e1 ) < w (e2 ) < w (e3 ) < · · · . Show that the MST is the same in these two graphs.

3.2

General definitions

The MST problem is an example of a combinatorial optimization problem. This is defined by a set of possible instances. An instance of the MST problem is defined by a connected weighted graph. In general, an instance of a combinatorial optimization problem is described by a finite set X of allowed configurations, and a cost function E defined on this set and taking values in R. The optimization problem consists in finding the optimal configuration C ∈ X , namely the configuration with the smallest cost E(C). Any set of such instances defines a combinatorial optimization problem. For a particular instance of the MST problem, the space of configurations X is simply the set of spanning trees on the given graph, and the cost function associated with each spanning tree is the sum of the costs of its edges. We shall say that an algorithm solves an optimization problem if, for every instance of the optimization problem, it gives the optimal configuration or if it computes its cost. In all the problems which we shall discuss, there is a ‘natural’ measure of the size of the problem N (typically the number of variables used to define a configuration, such as the number of edges of the graph in the MST problem), and the number of configurations scales, at large N like cN or, in some cases, even faster, for example like N ! or N N . Notice that, quite generally, evaluating the cost function of a particular configuration is an easy task. The difficulty of solving the combinatorial optimization problem therefore comes essentially from the size of the configuration space. It is generally accepted practice to estimate the complexity of an algorithm by the number of ‘elementary operations’ required to solve the problem. Usually one focuses on the asymptotic behaviour of this quantity as N → ∞. It is obviously of great practical interest to construct algorithms whose complexity is as small as possible. One can solve a combinatorial optimization problem at several levels of refinement. Usually, one distinguishes three types of problems: • The optimization problem: Find an optimal configuration C ∗ . • The evaluation problem: Determine the cost E(C ∗ ) of an optimal configuration. • The decision problem: Answer the question ‘Is there a configuration of cost less than a given value E0 ?’

3.3

More examples

The general setting described in the previous section includes a large variety of problems of both practical and theoretical interest. In the following, we shall provide a few selected examples.

¾


3.3.1

Eulerian circuit

One of the oldest documented examples goes back to the eighteenth century. The old city of K¨ onigsberg had seven bridges (see Fig. 3.3), and its inhabitants were wondering whether it was possible to cross each of these bridges once and get back home. This can be generalized and translated into graph-theoretic language as the following decision problem. We define a multigraph exactly like a graph except for the fact that two given vertices can be connected by several edges. The problem consists in finding whether there is there a circuit which goes through all edges of the graph only once, and returns to its starting point. Such a circuit is now called an Eulerian circuit, because this problem was solved by Euler in 1736, when he proved the following nice theorem. As for ordinary graphs, we define the degree of a vertex as the number of edges which have the vertex as an end-point. Theorem 3.2 Given a connected multigraph, there exists an Eulerian circuit if and only if every vertex has even degree. This theorem directly provides an algorithm for the decision problem whose complexity grows linearly with the number of vertices of the graph: just go through all the vertices of the graph and check their degree.

Exercise 3.4 Show that if an Eulerian circuit exists, the degrees are necessarily even. Proving the inverse implication is more difficult. A possible approach consists in showing the following slightly stronger result. If all the vertices of a connected graph G have even degree except for i and j, then there exists a path from i to j that visits each edge in G once. This can be proved by induction on the number of vertices. [Hint: Start from i and take a step along the edge (i, i ). Show that it is possible to choose i in such a way that the residual graph G\(i, i ) is connected.]

3.3.2

Hamiltonian cycle

More than a century after Euler’s theorem, the great scientist Sir William Hamilton introduced in 1859 a game called the icosian game. In its generalized form, it basically asks whether there exists, in a graph, a Hamiltonian cycle, that is a path that goes once through every vertex of the graph, and gets back to its starting point. This is another decision problem, and, at first sight, it seems very similar to the Eulerian circuit. However, it turns out to be much more complicated. The best existing algorithms for determining the existence of a Hamiltonian cycle in a given graph run in a time which grows exponentially with the number of vertices N . Moreover, the theory of computational complexity, which we shall describe in Section 3.4, strongly suggests that this problem is in fact intrinsically difficult. 3.3.3

Travelling salesman problem

Given a complete graph with N points, and the distances dij between all pairs of points 1 ≤ i < j ≤ N , the famous travelling salesman problem (TSP) is an optimization problem: find a Hamiltonian cycle of minimum total length. One can consider the case

More examples

¿

C

D

A

B Fig. 3.3 Left: a map of the old city of K¨ onigsberg, with its seven bridges, as drawn in Euler’s paper of 1736. The problem is whether one can walk through the city, crossing each bridge exactly once, and get back home. Right: a graph summarizing the problem. The vertices A, B, C, D are the various areas of land separated by a river; an edge exists between two vertices whenever there is a bridge. The problem is to make a closed circuit on this graph, going exactly once through every edge.

where the points are in a portion of the plane, and the distances are Euclidean distances (we then speak of a Euclidean TSP), but of course the problem can be stated more generally, with dij representing general costs, which are not necessarily distances. As for the Hamiltonian cycle problem, the best algorithms known so far for the TSP have a running time which grows exponentially with N at large N . Nevertheless, Euclidean problems with thousands of points can be solved. 3.3.4

Assignment

Given N persons and N jobs, and a matrix Cij giving the affinity of person i for job j, the assignment problem consists in finding the assignment of the jobs to the persons (an exact one-to-one correspondence between jobs and persons) which maximizes the total affinity. A configuration is characterized by a permutation of the N indices (there are thus N ! configurations), and the cost of a permutation π is i Ciπ(i) . This is an example of a polynomial problem: there exist algorithms that solve it in a time growing like N 3 . 3.3.5

Satisfiability

In the satisfiability problem, one has to find the values of N Boolean variables xi ∈ {T, F } which satisfy a set of logical constraints. Since each variable can be either true or false, the space of configurations has a size |X | = 2N . Each logical constraint, called in this context a clause, takes a special form: it is a logical OR (for which we use the symbol ∨) of some variables or their negations. For instance x1 ∨ x2 is a 2-clause (a ‘2-clause’ is a clause of length 2, i.e. one which involves exactly two variables), which is satisfied if either x1 = T or x2 = F , or both. Analogously, x1 ∨ x2 ∨ x3 is a 3-clause, which is satisfied by all configurations of the three variables except x1 = x2 = T , x3 = F . The problem is to determine whether there exists a configuration


which satisfies all constraints (a decision problem), or to find the configuration which minimizes the number of violated constraints (an optimization problem). The Ksatisfiability (or ‘K-SAT’) problem is the restriction of the satisfiability problem to the case where all clauses have length K. In 2-satisfiability, the decision problem is easy: there exists an algorithm that runs in a time growing linearly with N . For K-satisfiability, and therefore also for the general satisfiability problem, all known algorithms that solve the decision problem run in a time which grows exponentially with N . 3.3.6

Colouring and vertex covering

Given a graph and an integer q, the well-known q-colouring problem asks if it is possible to colour the vertices of the graph using q colours, in such a way that two vertices connected by an edge have different colours. In the same spirit, the vertexcovering problem asks if it is possible to cover the vertices with ‘pebbles’, using the smallest possible number of pebbles, in such a way that every edge of the graph has at least one of its two end-points covered by a pebble. 3.3.7

Number partitioning

Number partitioning is an example which does not come from graph theory. An instance of the problem consists in a set S of N integers S = {x1 , . . . , xN }. A configuration is a partition numbers into two groups A and S \ A . Is there a of these partition such that i∈A xi = i∈S\A xi ?

3.4

Elements of the theory of computational complexity

One of the main branches of theoretical computer science aims at constructing an intrinsic theory of computational complexity. One would like, for instance, to establish which problems are harder than others. By ‘harder problem’, we mean a problem that takes a longer running time to be solved. In order to discuss rigorously the computational complexity of a problem, however, we would need to define a precise model of computation (introducing, for instance, Turing machines). This would take us too far. We shall instead evaluate the running time of an algorithm in terms of ‘elementary operations’: comparisons, sums, multiplications, etc. This informal approach is essentially correct as long as the size of the operands remains uniformly bounded. 3.4.1

The worst-case scenario

As we already mentioned in Section 3.2, a combinatorial optimization problem is defined by the set of its possible instances. Given an algorithm that solves the problem, its running time will vary from instance to instance, even if the ‘size’ of the instance is fixed. How should we quantify the overall hardness of the problem? A crucial choice in computational-complexity theory consists in considering the ‘worst’ instance (i.e. the one which takes the longest time to be solved) out of all instances of the same size. This choice has two advantages: (i) it allows one to construct a ‘universal’ theory; and (ii) once the worst-case running time of a given algorithm has been estimated, this provides a performance guarantee for any instance of the problem.


3.4.2

Polynomial or not?

A second crucial choice consists in classifying algorithms into two classes. (i) Polynomial, if the running time is bounded from above by a fixed polynomial in the size of the instance. In mathematical terms, let TN be the number of operations required for solving an instance of size N in the worst case. The algorithm is polynomial when there exists a constant k such that TN = O(N k ). (ii) Super-polynomial, if no such upper bound exists. This is, for instance, the case if the time grows exponentially with the size of the instance (we shall call algorithms of this type exponential), i.e. TN = Θ(k N ) for some constant k > 1. Example 3.3 In Section 3.1.2, we were able to show that the running time of the MST algorithm is bounded from above by N 3 , where N is the number of vertices in the graph. This implies that such an algorithm is polynomial. Notice that we have not given a precise definition of the ‘size’ of a problem. One may wonder whether, by changing the definition, a particular problem could be classified both as polynomial and as super-polynomial. Consider, for instance, the assignment problem with 2N points. One can define the size as being N , or 2N , or even N 2 which is the number of possible person–job pairs. The last definition would be relevant if one were to count, for instance, the number of entries in the person–job cost matrix. However, all of these ‘natural’ definitions of size are polynomial functions of the others. Therefore the choice of one or another of these definitions does not affect the classification of an algorithm as polynomial or super-polynomial. We shall discard other definitions (such as eN or N !) as ‘unnatural’, without further ado. The reader can convince him/herself of this for each of the examples of the previous Section. 3.4.3

Optimization, evaluation, and decision

In order to get a feeling for their relative levels of difficulty, let us come back for a while to the three types of optimization problem defined in Section 3.2, and study which one is the hardest. Clearly, if the cost of any configuration can be computed in polynomial time, the evaluation problem is not harder than the optimization problem: if one can find the optimal configuration in polynomial time, one can also compute its cost in polynomial time. The decision problem (deciding whether there exists a configuration with a cost smaller than a given E0 ) is not harder than the evaluation problem. So the order of increasing difficulty is decision, evaluation, optimization. However, in many cases where the costs take discrete values, the evaluation problem is not harder than the decision problem, in the following sense. Suppose that we have a polynomial algorithm that solves the decision problem, and that the costs of all configurations can be scaled so as to be integers in an interval [0, Emax ] of length Emax = exp{O(N k )} for some k > 0. An algorithm that solves the decision problem can be used to solve the evaluation problem by dichotomy. One first takes E0 = Emax /2. If there exists a configuration of energy smaller than E0 , one iterates taking E0 to be the centre of the interval [0, Emax /2]. In the opposite case, one iterates


taking E0 to be the centre of the interval [Emax /2, Emax ]. Clearly, this procedure finds the cost of the optimal configuration(s) in a time which is also polynomial.

3.4.4

Polynomial reduction

One would like to compare the levels of difficulty of various decision problems. The notion of polynomial reduction formalizes the sentence ‘not harder than’ which we have used so far, and helps us to obtain a classification of decision problems. Roughly speaking, we say that a problem B is not harder than A if any efficient algorithm for A (if such an algorithm exists) could be used as a subroutine of an algorithm that solves B efficiently. More precisely, given two decision problems A and B, one says that B is polynomially reducible to A if the following conditions hold: 1. There exists a mapping R which transforms any instance I of problem B into an instance R(I) of problem A, such that the solution (yes/no) of the instance R(I) of A gives the solution (yes/no) of the instance I of B. 2. The mapping I → R(I) can be computed in a time which is polynomial in the size of I. 3. The size of R(I) is polynomial in the size of I. This is in fact a consequence of the previous assumptions, but there is no harm in stating it explicitly. A mapping R that satisfies the above requirements is called a polynomial reduction. Constructing a polynomial reduction between two problems is an important achievement, since it effectively reduces their study to the study of just one of them. Suppose for instance that we have a polynomial algorithm AlgA for solving A. A polynomial reduction of B to A can then be used for to construct a polynomial algorithm for solving B. Given an instance I of B, the algorithm just computes R(I), feeds it into AlgA , and outputs the output of AlgA . Since the size of R(I) is polynomial in the size of I, the resulting algorithm for B is still polynomial. Let us work out an explicit example. We shall show that the problem of the existence of a Hamiltonian cycle in a graph is polynomially reducible to the satisfiability problem.

Example 3.4 An instance of the Hamiltonian cycle problem is a graph with N vertices, labelled by i ∈ {1, . . . , N }. If there exists a Hamiltonian cycle in the graph, it can be characterized by N 2 Boolean variables xri ∈ {0, 1}, where xri = 1 if vertex number i is the r-th vertex in the cycle, and xri = 0 otherwise (one can take, for instance, x11 = 1). We shall now write down a number of constraints that the variables xri must satisfy in order for a Hamiltonian cycle to exist, and ensure that these constraints take the form of the clauses used in the satisfiability problem (identifying x = 1 as true and x = 0 as false):


• Each vertex i ∈ {1, . . . , N } must belong to the cycle: this can be written as the clause x1i ∨ x2i ∨ · · · ∨ xN i , which is satisfied only if at least one of the numbers x1i , x2i , . . . , xN i equals one. • For every r ∈ {1, . . . , N }, one vertex must be the r-th visited vertex in the cycle: xr1 ∨ xr2 ∨ · · · ∨ xrN . • Each vertex i ∈ {1, . . . , N } must be visited only once. This can be implemented ¯sj , for 1 ≤ r < s ≤ N . through the N (N − 1)/2 clauses x ¯rj ∨ x • For every r ∈ {1, . . . , N }, there must be only one r-th visited vertex in the cycle. This can be implemented through the N (N − 1)/2 clauses xri ∨ xrj , for 1 ≤ i < j ≤ N. • If two vertices i < j are not connected by an edge of the graph, these vertices should not appear consecutively in the list of vertices belonging to the cycle. Therefore we add, for every such pair and for every r ∈ {1, . . . , N }, the clauses xri ∨ x(r+1)j and xrj ∨ x(r+1)i (with the ‘cyclic’ convention N + 1 = 1). It is straightforward to show that the size of the satisfiability problem constructed in this way is polynomial in the size of the Hamiltonian cycle problem. We leave it as an exercise to show that the set of all the above clauses is a sufficient set: if the N 2 variables satisfy all the above constraints, they describe a Hamiltonian cycle. 3.4.5

Complexity classes

Let us continue to focus on decision problems. The classification of these problems with respect to polynomiality is as follows: • Class P. These are the polynomial problems, which can be solved by an algorithm running in polynomial time. An example (see Section 3.1) is the decision version of the minimum-spanning-tree problem (which asks for a yes/no answer to the following question: given a graph with costs on the edges, and a number E0 , is there a spanning tree with total cost less than E0 ?). • Class NP. This is the class of non-deterministic polynomial problems, which can be solved in polynomial time by a ‘non-deterministic’ algorithm. Roughly speaking, such an algorithm can run in parallel on an arbitrarily large number of processors. We shall not explain this notion in detail here, but rather use an alternative, equivalent characterization. We say that a problem is in the class NP if there exists a ‘short’ certificate which allows one to check a ‘yes’ answer to the problem. A short certificate means a certificate that can be checked in polynomial time. A polynomial problem such as the MST problem described above is automatically in NP, and so P ⊆ NP. The decision version of the TSP is also in NP: if there is a TSP tour with cost smaller than E0 , the short certificate is simple: just give the tour, and its cost can be computed in linear time, allowing one to check that it is smaller than E0 . The satisfiability problem also belongs to NP: a certificate is obtained from the assignment of variables satisfying all clauses. Checking that all clauses are satisfied is linear in the number of clauses, taken here as the size of the system. In fact, there are many important problems in the class NP, with a broad


spectrum of applications ranging from routing to scheduling, chip verification, and protein folding. • Class NP-complete. These are the hardest problems in the class NP. A problem is NP-complete if (i) it is in NP, and (ii) any other problem in NP can be polynomially reduced to it, using the notion of polynomial reduction defined in Section 3.4.4. If A is NP-complete, then for any other problem B in NP, there is a polynomial reduction that maps B to A. In other words, if we had a polynomial algorithm to solve A, then all the problems in the broad class NP could be solved in polynomial time. It is not a priori obvious whether there exist any NP-complete problems. A major achievement of the theory of computational complexity is the following theorem, obtained by Cook in 1971. Theorem 3.5 The satisfiability problem is NP-complete.

We shall not give the proof of the theorem here. Let us just mention that the satisfiability problem has a very universal structure (an example of which was shown above, in the polynomial reduction of the Hamiltonian cycle problem to satisfiability). A clause is built as a logical OR (denoted by ∨) of some variables or their negations. A set of several clauses to be satisfied simultaneously is the logical AND (denoted by ∧) of those clauses. Therefore a satisfiability problem is written in general in the form (a1 ∨ a2 ∨ . . . ) ∧ (b1 ∨ b2 ∨ . . . ) ∧ . . . , where the ai , bi are ‘literals’, i.e. any of the original variables or their negations. This form is called a conjunctive normal form (CNF), and it is easy to see that any logical statement between Boolean variables can be written as a CNF. This universal decomposition gives an idea of why the satisfiability problem plays a central role.

3.4.6

P = NP?

When an NP-complete problem A is known, one can find other NP-complete problems relatively easily: if there exists a polynomial reduction from A to another problem B ∈ NP, then B is also NP-complete. In fact, whenever RA←P is a polynomial reduction from a problem P to A and RB←A is a polynomial reduction from A to B, then RB←A ◦ RA←P is a polynomial reduction from P to B. Starting from the satisfiability problem, it has been possible to find, by this method, thousands of NP-complete problems. To quote a few of them, among the problems we have encountered so far, the Hamiltonian cycle problem, the TSP, and the 3-satisfiability problem are NPcomplete. Actually, most NP problems can be classified either as being in P or as NP-complete (see Fig. 3.4). The precise status of some NP problems, however, such as graph isomorphism, is still unknown.


NP-complete SAT 3SAT

3-Colouring

TSP (d) Hamiltonian cycle

NP

2SAT Eulerian circuit

P

2-colouring

Assignment

Fig. 3.4 Classification of some well known decision problems. If P = NP, the classes P and NP-complete are disjoint. But if it happened that P=NP, all the problems in NP, and in particular all those mentioned here, would be solvable in polynomial time.

Exercise 3.5 Show that the 3-satisfiability problem is NP-complete, by constructing a polynomial reduction from the satisfiability problem. The idea is to transform all possible clauses into sets of 3-clauses, using the following steps: • A 2-clause x1 ∨ x2 can be written as two 3-clauses (x1 ∨ x2 ∨ y) ∧ (x1 ∨ x2 ∨ y) with one extra variable y. • Write a 1-clause with four 3-clauses and two extra variables. • Show that a k-clause x1 ∨ x2 ∨ · · · ∨ xk with k ≥ 4 can be written with k − 3 auxiliary variables as (x1 ∨ x2 ∨ y1 ) ∧ (x3 ∨ y 1 ∨ y2 ) ∧ · · · ∧ (xk−2 ∨ y k−4 ∨ yk−3 ) ∧ (xk−1 ∨ xk ∨ y k−3 ).

Finally, those problems which, not being in NP, are at least as hard as NP-complete problems, are usually called NP-hard. These include both decision problems for which a short certificate does not exist, and non-decision problems. For instance, the optimization and evaluation versions of the TSP are NP-hard. However, in such cases, we shall choose between the expressions ‘the TSP is NP-complete’ and ‘the TSP is NP-hard’ rather freely. One major open problem in the theory of computational complexity is whether the classes P and NP are distinct or not. It might be that P = NP = NP-complete: this would be the case if someone found a polynomial algorithm for one NP-complete problem. This would imply that any problem in the broad class NP could be solved in polynomial time. It is a widespread conjecture that there exists no polynomial algorithm for NPcomplete problems. In this case the classes P and NP-complete would be disjoint. Moreover, it is known that if P = NP, then there are NP problems which are neither in P nor in NP-complete.

¼


3.4.7

Other complexity classes

Notice the fundamental asymmetry in the definition of the class NP: the existence of a short certificate is required only for the ‘yes’ answers. To understand the meaning of this asymmetry, consider the problem of unsatisfiability (which is the complement of the satisfiability problem), formulated as ‘given a set of clauses, is the problem unsatisfiable?’ It is not clear if there exists a short certificate that allows one to check a ‘yes’ answer: it is very difficult to prove that a problem cannot be satisfied without checking an exponentially large number of possible configurations. So it is not at all obvious that unsatisfiability is in NP. Problems which are complements of those in NP define the class of co-NP problems, and it is not known whether NP = co-NP or not, although it is widely believed that co-NP is different from NP. This consideration opens a Pandora’s box containing many other classes of complexity, but we shall immediately close it, since leaving it open would carry us too far.

3.5 3.5.1

Optimization and statistical physics General relation

There exists a natural mapping from optimization to statistical physics. Consider an optimization problem defined by a finite set X of allowed configurations, and a cost function E defined on this set with values in R. Although optimization consists in finding the configuration C ∈ X with the smallest cost, one can introduce a probability measure of the Boltzmann type on the space of configurations: for any β, each C is assigned a probability µβ (C) =

1 e−βE(C) , Z(β)

Z(β) =

e−βE(C) .

(3.1)

C∈X

The positive parameter β plays the role of an inverse temperature. In the limit β → ∞, the probability distribution µβ concentrates on the configurations of minimum energy (ground states in statistical-physics jargon). This is the relevant limit for optimization problems. Note that there exist many alternatives to the straightforward generalization (3.1). In some problems, it may be useful to use more than one inverse temperature parameter β. Some of these parameters can be used to ‘soften’ constraints. For instance in the TSP, one might like to relax the constraint that a configuration is a tour, by summing over all length-N paths of the salesman, with an extra cost each time the path does not make a full tour, associated with an inverse temperature β1 . The length of the path would be another cost, associated with an inverse temperature β2 . The original problem is recovered when both β1 and β2 go to infinity. In the statistical-physics approach, one generalizes the optimization problem to study properties of the distribution µβ at finite β. In many cases, it is useful to follow µβ as β increases (for instance by monitoring the thermodynamic properties, i.e. the internal energy, entropy, . . . ). This may be particularly useful, both for analytical and for algorithmic purposes, when the thermodynamic properties evolve smoothly. An example of a practical application is the simulated-annealing method, which samples the configuration space at larger and larger values of β until it finds a ground state.

Optimization and coding

½

This method will be described in Chapter 4. As we shall see, the occurrence of phase transitions poses major challenges to this kind of approach. 3.5.2

Spin glasses and maximum cuts

To give a concrete example, let us go back to the spin glass problem of Section 2.6. This involves N Ising spins σ1 , . . . , σN in {±1}, located on the vertices of a graph, and the energy function is Jij σi σj , (3.2) E(σ) = − (ij)

where the sum (ij) runs over all edges of the graph, and the variables Jij are exchange couplings, which can be either positive or negative. Given the graph and the exchange couplings, what is the ground state of the corresponding spin glass? This is a typical optimization problem. In fact, it very well known in computer science in a slightly different form. Each spin configuration partitions the set of vertices into two complementary subsets V± = {i | σi = ±1}. Let us call the set of edges with one end-point in V+ and the other in V− γ(V+ ). The energy of the configuration can be written as Jij , (3.3) E(σ) = −C + 2 (ij)∈γ(V+ )

where C = (ij) Jij . Finding the ground state of the spin glass is thus equivalent to finding a partition of the vertices V = V+ ∪ V− such that (ij)∈γ(V+ ) cij is maximum, where cij ≡ −Jij . This problem is known as the maximum cut (max-cut) problem: the set of edges γ(V+ ) is a cut, each cut is assigned a weight (ij)∈γ(V+ ) cij , and one seeks the cut with maximal weight. Standard results on the max-cut problem immediately apply. In general, this is an NP-hard problem, but there are some categories of graphs for which it is polynomially solvable. In particular, the max-cut of a planar graph can be found in polynomial time, providing an efficient method to obtain the ground state of a spin glass on a square lattice in two dimensions. The three-dimensional spin glass problem falls into the general NP-hard class, but efficient ‘branch and bound’ methods, based on its max-cut formulation, have been developed for this problem in recent years. Another well-known application of optimization to physics is the random-field Ising model, which is a system of Ising spins with ferromagnetic couplings (all Jij are positive), but with a magnetic field hi which varies from site to site, taking both positive and negative values. Its ground state can be found in polynomial time thanks to the equivalence with the problem of finding a maximal flow in a graph.

3.6

Optimization and coding

Computational-complexity issues are also crucial in all problems of information theory. We shall see this recurrently throughout the book, but let us just give here some small examples in order to fix our ideas.

¾


Consider the error-correcting-code problem of Chapter 1. We have a code which maps an original message to a codeword x, which is a point in the N -dimensional hypercube {0, 1}N . There are 2M codewords (with M < N ), which we assume to be a priori equiprobable. When the message is transmitted through a noisy channel, the codeword x is corrupted to –say– a vector y with probability Q(y|x). The decoder maps the received message y to one of the possible input codewords x = d(y). As we saw a measure of performance is the average block error probability Pav B ≡

1 Q(y|x) I(d(y) = x) . 2M x y

(3.4)

A simple decoding algorithm would be the following: for each received message y, consider all the 2M codewords, and determine the most likely one: d(y) = arg maxx∈Code Q(y|x). It is clear that this algorithm minimizes the average block error probability. For a general code, there is no better way to maximize Q(y|x) than to go through all codewords and compute their likelihood one by one. This takes a time of order 2M , which is definitely too large. Recall that in fact, to achieve reliable communication, M and N have to be large (in data transmission applications, one may use an N as large as 105 ). One might object that ‘decoding a general code’ is too general a problem. Just to specify a single instance, we would need to specify all the codewords, which takes N 2M bits. Therefore, the complexity of decoding could be a trivial consequence of the fact that even reading the input takes a huge time. However, it can be proved that, for codes that admit a concise specification (polynomial in the block length), also, decoding is NP-hard. We shall see some examples, namely linear codes, in the following chapters.

Notes We have left aside most of the algorithmic issues in this chapter. A general approach to design efficient algorithms consists in finding a good ‘convex relaxation’ of the problem. The idea is to enlarge the space of feasible solutions in such a way that the problem translates into minimizing a convex function, a task that can be performed efficiently. A general introduction to combinatorial optimization, including all these aspects, is provided by Papadimitriou and Steiglitz (1998). Convex optimization is the topic of many textbooks, for instance Boys and Vandenberghe (2004). The MST algorithm described in Section 3.1 was found by Prim (1957). A complete treatment of computational complexity theory can be found in Garey and Johnson (1979), or in the more recent book by Papadimitriou (1994). The seminal theorem by Cook (1971) was independently rediscovered by Levin in 1973. The reader can find a proof in the above books. Euler discussed the problem of the seven bridges of K¨ onisberg in Euler (1736). The TSP, which is simple to state and difficult to solve, and lends itself to nice representations in figures, has attracted much attention. The interested reader can find many references and also pictures of optimal tours with thousands of vertices, including tours of the main cities in various countries, applets, etc. on the Web, starting for instance from Applegate et al. (2008).

Notes

¿

The book by Hartmann and Rieger (2002) focuses on the use of optimization algorithms for solving some problems in statistical physics. In particular, it explains the determination of the ground state of a random-field Ising model with a maximum-flow algorithm. A recent volume edited by the same authors (Hartmann and Rieger, 2004) addresses several algorithmic issues connecting optimization and physics. Chapter 4 of that volume describes the branch-and-cut approach to the maximum cut problem used for spin glass studies. The book by Hartmann and Weigt (2005) contains an introduction to combinatorial optimization considered as a physics problem, with particular emphasis on the vertex cover problem. Standard computational problems in coding theory are reviewed in Barg (1998). Some more recent issues are addressed by Spielman (1997). Finally, the first proof of NP-completeness for a decoding problem was obtained by Berlekamp et al. (1978).


4 A probabilistic toolbox The three fields that form the subject of this book all deal with large sets of discrete variables. Often, a probability distribution can be defined naturally on such variables. Not surprisingly, the problems arising in these three fields possess common underlying structures, and some general probabilistic techniques can be applied to all three domains. This chapter describes some of them, concentrating on the mathematical structures, namely large deviations on the one hand, and Markov chains for Monte Carlo computations on the other hand. These tools will reappear several times in the following chapters. Since this chapter is more technical than the previous ones, we devote the whole of Section 4.1 to a qualitative introduction to the subject. In Section 4.2, we consider the large-deviation properties of simple functions of many independent random variables. In this case, many explicit results can be easily obtained. We present a few general tools for correlated random variables in Section 4.3 and the idea of the Gibbs free energy in Section 4.4. Section 4.5 provides a simple introduction to the Monte Carlo Markov chain method for sampling configurations from a given probability distribution. Finally, in Section 4.6, we show how simulated annealing exploits Monte Carlo techniques to solve optimization problems.

4.1

Many random variables: A qualitative preview

Consider a set of N random variables x = (x1 , x2 , . . . , xN ), with xi ∈ X and a probability distribution PN (x) = PN (x1 , . . . , xN ) .

(4.1)

This could be, for instance, the Boltzmann distribution for a physical system with N degrees of freedom. The entropy of this distribution is HN = −E log PN (x). It often happens that this entropy grows linearly with N at large N . This means that the entropy per variable hN = HN /N has a finite limit: limN →∞ hN = h. It is then natural to characterize any particular realization of the random variables (x1 , . . . , xN ) by computing the quantity 1 1 log , (4.2) r(x) = N PN (x) which measures how probable the event (x1 , . . . , xN ) is. The expectation of r is Er(x) = hN . One may wonder whether r(x) fluctuates a lot, or whether its distribution is

A probabilistic toolbox

strongly peaked around r = hN . The latter hypothesis turns out to be the correct one in many cases. It often happens that the probability distribution of r(x) behaves exponentially with N , i.e. . (4.3) P{r(x) ≈ ρ} = e−N I(ρ) . where I(ρ) has a non-degenerate minimum at ρ = limN →∞ hN = h, and I(h) = 0. This means that, with large probability, a randomly chosen configuration x has an empirical entropy r(x) ‘close to’ h, and, because of the definition (4.2), its probability is approximately exp(−N h). Since the total probability of realizations x such that f (x) ≈ . h is close to one, their number must behave as N = exp(N h). In other words, the whole probability is carried by a small fraction of all configurations (since their number, exp(N h), is in general exponentially smaller than |X |N ), and these configurations all have the same probability. This property, called asymptotic equipartition, holds in many of the problems we are interested in, and its consequences are very important. Suppose, for instance, that one is interested in compressing the information contained in the variables (x1 , . . . , xN ), which are a sequence of symbols produced by an information source. Clearly, one should focus on ‘typical’ sequences x such that f (x) is close to h, because all other sequences have a vanishing small probability. Since there are about exp(N h) such typical sequences, one must be able to encode them in N h/ log 2 bits by simply numbering them. Another general problem consists in sampling from the probability distribution with distribution PN (x), one PN (x). With n samples x1 , . . . , xn drawn independently r 1 k P (x)O(x) as O(x ). This can estimate expectation values E O(x) ≡ k=1 x N n √ N avoids summing over |X | terms, and the precision usually improves like 1/ n at large n. A naive sampling algorithm could be the following. First, ‘propose’ a configuration x from the uniform probability distribution PNunif (x) = 1/|X |N . This is simple to do using a pseudorandom generator: N log2 |X | unbiased random bits are sufficient to sample from PNunif (x). Then ‘accept’ the configuration with probability PN (x). Such an algorithm is very inefficient. It is clear that, to estimate expectation of ‘well-behaved’ observables, we need to generate typical configurations x, i.e. configurations such that r(x) is close to h. However, such configurations are exponentially rare, and the above algorithm will require a time of order exp[N (log |X | − h)] to generate one of them. The Monte Carlo Markov chain method provides a better alternative.

4.2

Large deviations for independent variables

A behaviour of the type (4.3) is an example of a large-deviation principle. We shall first study the simplest case where such a behaviour is found, namely the case of independent random variables. This case is instructive because all properties can be controlled in great detail. 4.2.1

How typical is a series of observations?

Suppose that you are given given the values s1 , . . . , sN of N i.i.d. random variables drawn from a finite space X according to a known probability distribution {p(s)}s∈X . The si could be produced, for instance, by an information source, or by repeated


measurements on a physical system. You would like to know if the sequence s = (s1 , . . . , sN ) is a typical one, or if you have found a rare event. If N is large, one can expect that the number of appearances of a given x ∈ X in a typical sequence will be close to N p(x). The method of types allows us to quantify this statement. The type qs (x) of a sequence s is the frequency of appearance of the symbol x in the sequence: qs (x) =

N 1 I(x = si ) . N i=1

(4.4)

The type qs (x), considered as a function of x, has the properties of a probability distribution over X : qs (x) ≥ 0 for any x ∈ X , and x qs (x) = 1. In the following, we shall denote by M(X ) the space of probability distributions over X : M(X ) ≡ {q ∈ RX such that q(x) ≥ 0 , x q(x) = 1}. In particular, qs ∈ M(X ). The expectation of the type qs (x) coincides with the original probability distribution: E qs (x) = p(x) .

(4.5)

Sanov’s theorem estimates the probability that the type of the sequence differs from p(x). Theorem 4.1. (Sanov) Let s1 , . . . , sN ∈ X be N i.i.d. random variables drawn from the probability distribution p(x), and let K ⊂ M(X ) be a compact set of probability distributions over X . If q is the type of (s1 , . . . , sN ), then . P [q ∈ K] = exp[−N D(q ∗ ||p)] ,

(4.6)

where q∗ = arg minq∈K D(q||p), and D(q||p) is the Kullback–Leibler divergence defined in eqn (1.10) . Informally, this theorem means that the probability of finding a sequence with type q behaves at large N like exp[−N D(q||p)]. For large N , typical sequences have a type q(x) ≈ p(x), and those with a different type are exponentially rare. The proof of the theorem is a straightforward application of Stirling’s formula and is left as an exercise for the reader. In Section 4.7 we give a derivation using a ‘field-theoretical’ manipulation as used in physics. This may be an instructive, simple example for the reader who wants to get used to this kind of technique, frequently used by physicists.

Example 4.2 Let the si be the outcome of a biased coin: X = {head, tail}, with p(head) = 1 − p(tail) = 0.8. What is the probability of getting 50 heads and 50 tails in 100 throws of the coin? Using eqns (4.6) and (1.10) with N = 100 and q(head) = q(tail) = 0.5, we get Prob[50 tails] ≈ 2.04 × 10−10 .


Example 4.3 Consider the reverse case: we take a fair coin (p(head) = p(tail) = 0.5) and ask what the probability of getting 80 heads and 20 tails is. Sanov’s theorem provides the estimate Prob[80 heads] ≈ 4.27 × 10−9 , which is much higher than the value computed in the previous example.

Example 4.4 A simple model of a column of the atmosphere is obtained by considering N particles in the earth’s gravitational field. The state of particle i ∈ {1, . . . , N } is given by a single coordinate zi ≥ 0 which measures its height with respect to ground level. For the sake of simplicity, we assume the zi to be integer numbers. We can, for instance, imagine that the heights are discretized in terms of some small unit length (e.g. millimetres). The N -particle energy function reads, in properly chosen units, E=

N

zi .

(4.7)

i=1

The type of a configuration {z1 , . . . , zN } can be interpreted as the density profile ρ(z) of the configuration: ρ(z) =

N 1 I(z = zi ) . N i=1

(4.8)

Using the Boltzmann probability distribution (2.4), it is simple to compute the expected density profile, which is usually called the ‘equilibrium’ profile: ρeq (z) ≡ ρ(z) = (1 − e−β ) e−βz .

(4.9)

If we take a snapshot of the N particles at a given instant, their density will show some deviations with respect to ρeq (z). The probability of seeing a density profile ρ(z) is given by eqn (4.6) with p(z) = ρeq (z) and q(z) = ρ(z). For instance, we can compute the probability of observing an exponential density profile like that in eqn (4.9) but with a different parameter λ: ρλ (z) = (1 − e−λ ) e−λz . Using eqn (1.10), we get: β−λ 1 − e−λ . (4.10) + λ D(ρλ ||ρeq ) = log 1 − e−β e −1 The function Iβ (λ) ≡ D(ρλ ||ρeq ) is depicted in Fig. 4.1.


2 1.8

β = 0.5

1.6

β = 1.0

Iβ (λ)

1.4

β = 2.0

1.2 1 0.8 0.6 0.4 0.2 0 0

0.5

1

1.5

2

2.5

3

3.5

4

λ

Fig. 4.1 In an atmosphere where the equilibrium density profile is ρeq (z) ∝ e−βz , the probability of observing an atypical profile ρ(z) ∝ e−λz is, for a large number of particles N , exp[−N Iβ (λ)]. The curves of Iβ (λ) plotted here show that small values of λ are very rare.

Exercise 4.1 The previous example is easily generalized to the density profile of N particles in an arbitrary potential V (x). Show that the Kullback–Leibler divergence takes the form X X D(ρ||ρeq ) = β V (x)ρ(x) + ρ(x) log ρ(x) + log z(β) . (4.11) x

4.2.2

x

How typical is an empirical average?

The result (4.6) contains detailed information concerning the large fluctuations of the random variables {si }. Often one is interested in measuring some property of the particles, that is described by a real number f (si ). The empirical average of the measurement is given by N 1 f≡ f (si ) . N i=1

(4.12)

Of course f , will be ‘close’ to E f (x) with high probability. The following result quantifies the probability of rare fluctuations. Corollary 4.5 Let s1 , . . . , sN be N i.i.d. random variables drawn from a probability distribution p(.). Let f : X → R be a real-valued function and let f be its empirical average. If A ⊂ R is a closed interval of the real axis, then . (4.13) P [f ∈ A] = exp[−N I(A)] , where

¼


1 heq = 1

0.6

heq = 2 heq = 3

I(h)

0.8

0.4 0.2 0 0

0.5

1

1.5

2

2.5

3 h

3.5

4

4.5

5

5.5

6

Fig. 4.2 Probability of an atypical average height for N particles with the energy function (4.7).

I(A) = min D(q||p) q(x)f (x) ∈ A . q

(4.14)

x∈X

Proof: f is related to the type of x1 , . . . , xN through f = x q(x)f (x). Therefore we can apply Theorem 4.1, using for K the set of distributions such that the expectation value of f is in A: # K=

q ∈ M(X ) |

$ q(x)f (x) ∈ A

.

(4.15)

x∈X

This implies eqn (4.13) and (4.14) directly.

Exercise 4.2 Let s1 , . . . , sN be N i.i.d. random variables drawn from a probability distri˘ bution p( · ) with bounded support. Show that, to leading exponential order, P s1 + · · · + ¯ . ˘ ¯N sN ≤ 0 = inf z≥0 E [e−zs1 ] .

Example 4.6 We look again at N particles in a gravitational field, and consider the average height of the particles z=

N 1 zi . N i=1

(4.16)


½

The expected value of this quantity is E(z) = zeq = (eβ − 1)−1 . The probability of a fluctuation in z is easily computed using the above corollary. For z > zeq , we obtain . P [z > z] = exp[−N I(z)], where 1 + zeq z I(z) = (1 + z) log + z log . (4.17) 1+z zeq . Analogously, for z < zeq , P [z < z] = exp[−N I(z)], with the same rate function I(z). The function I(z) is depicted in Fig. 4.2.

Exercise 4.3 One can construct a thermometer using a system of N particles with the energy function (4.7). Whenever the temperature is required, one takes a snapshot of the N particles, computes x, and estimates the inverse temperature βest using the formula (eβest − 1)−1 = x. What (for N 1) is the probability of getting a result βest = β?

4.2.3

Asymptotic equipartition

The above tools can also be used for counting the number of configurations s = (s1 , . . . , sN ) with either a given type q(x) or a given empirical average of some observable f . One finds, for instance, the following. Proposition 4.7 The number NK,N of sequences s which have a type belonging to the . compact set K ⊂ M(X ) behaves as NK,N = exp{N H(q∗ )}, where q∗ = arg max{H(q) | q ∈ K}. This result can be stated informally by saying that ‘there are approximately eN H(q) sequences with type q’. Proof The idea is to apply Sanov’s theorem, taking the ‘reference’ distribution p(x) to be the uniform probability distribution punif (x) = 1/|X |. Using eqn (4.6), we get . NK,N = |X |N Punif [q ∈ K] = exp{N log |X |−N D(q∗ ||punif )} = exp{N H(q∗ )} . (4.18) We now return to a generic sequence s = (s1 , . . . , sN ) of N i.i.d. variables with a probability distribution p(x). As a consequence of Sanov’s theorem, we know that the most probable type is p(x) itself, and that deviations are exponentially rare in N . We expect that almost all the probability will be concentrated into sequences that have a type close to p(x) in some sense. On the other hand, because of the above proposition, the number of such sequences is exponentially smaller than the total number of possible sequences |X |N . These remarks can be made more precise by defining what is meant by a sequence having a type ‘close to p(x)’. Given a sequence s, we introduce the quantity r(s) ≡ −

N 1 1 log PN (s) = − log p(xi ) . N N i=1

(4.19)

¾


Clearly, E r(s) = H(p). The sequence s is said to be ε-typical if and only if |r(s) − H(p)| ≤ ε. Theorem 4.8 Let TN,ε be the set of ε-typical sequences. Then we have the following properties. (i) limN →∞ P[s ∈ TN,ε ] = 1. (ii) For N large enough, eN [H(p)−ε] ≤ |TN,ε | ≤ eN [H(p)+ε] . (iii) For any s ∈ TN,ε , e−N [H(p)+ε] ≤ PN (s) ≤ e−N [H(p)−ε] . Proof Since r(s) is an empirical average, we can apply Corollary 4.5. This allows us to . / TN,ε ] = exp(−N I). estimate the probability of the sequence not being typical as P[ s ∈ being taken over the set K The exponent is given by I = minq D(q||p), theminimum of all probability distributions q(x) such that x∈X q(x) log[1/q(x)] − H(p) ≥ ε. But D(q||p) > 0 unless q = p, and p does not belong to K. Therefore I > 0 and / TN,ε ] = 0, which proves (i). limN →∞ P[ s ∈ The condition for q(x) to be the type of an ε-typical sequence can be rewritten as |D(q||p) + H(q) − H(p)| ≤ ε. Therefore, for any ε-typical sequence, |H(q) − H(p)| ≤ ε, and Proposition 4.7 leads to (ii). Finally, (iii) is a direct consequence of the definition of ε-typical sequences. The behaviour described in this theorem is usually referred to as the asymptotic equipartition property. Although we have proved it only for i.i.d. random variables, it holds in a much broader context. In fact, it will be found in many interesting systems throughout the book.

4.3

Correlated variables

In the case of independent random variables in finite spaces, the probability of a large fluctuation is easily computed by combinatorics. It would be nice to have some general result for large deviations of non-independent random variables. In this section, we shall describe the use of Legendre transforms and saddle point methods to study the general case. This method corresponds to a precise mathematical statement: the G¨ artner–Ellis theorem. We shall first describe the approach informally and apply it to a few examples. Then we shall state the theorem and discuss it. 4.3.1

Legendre transformation

We consider a set of N random variables x = (x1 , . . . , xN ), with xi ∈ X and a probability distribution PN (x) = PN (x1 , . . . , xN ) .

(4.20)

Let f : X → R be a real-valued function. We are interested in estimating, for large N , the probability distribution of the empirical average N 1 f (x) = f (xi ) . N i=1

(4.21)

In the previous section, we studied the particular case in which the xi are i.i.d. random variables. We proved that, quite generally, a finite fluctuation of f (x) is exponentially

Correlated variables

¿

unlikely. It is natural to expect that the same statement will hold true if the xi are ‘weakly correlated’. Whenever PN (x) is the Boltzmann distribution for some physical system, this expectation is supported by physical intuition. We can think of the xi as the microscopic degrees of freedom that make up the system, and of f (x) as a macroscopic observable (pressure, magnetization, etc.). It is a common observation that the relative fluctuations of macroscopic observables are very small. Let us thus assume that the distribution of f follows a large-deviation principle, meaning that the asymptotic behaviour of the distribution at large N is . PN (f ) = exp[−N I(f )] ,

(4.22)

with a rate function I(f ) ≥ 0. In order to determine I(f ), a useful method is to ‘tilt’ the measure PN (·) in such a way that the rare events responsible for O(1) fluctuations of f become likely. In practice, we define the (logarithmic) moment-generating function of f as follows

ψN (t) =

% & 1 log E eN tf (x) , N

t ∈ R.

(4.23)

When the large-deviation principle (4.22) holds, we can evaluate the large-N limit of ψN (t) using the saddle point method: 1 log lim ψN (t) = lim N →∞ N →∞ N

e

N tf −N I(f )

df

= ψ(t) ,

(4.24)

where ' ( ψ(t) = sup tf − I(f ) .

(4.25)

f ∈R

In other words, ψ(t) is the Legendre transform of I(f ), and it is a convex function of t by construction (this can be proved by differentiating eqn (4.23) twice). It is therefore natural to invert the Legendre transform (4.25) as follows: ' ( Iψ (f ) = sup tf − ψ(t) ,

(4.26)

t∈R

and we expect Iψ (f ) to coincide with the convex envelope of I(f ). This procedure is useful whenever computing ψ(t) is easier than a direct estimate of the probability distribution PN (f ). It is useful to gain some insight by considering a few examples.


0.3 β = 0.25

0.25

β = 0.5 β=1

I(m)

0.2

β=2

0.15 0.1 0.05 0 -1

-0.8 -0.6 -0.4 -0.2

0

0.2

0.4

0.6

0.8

1

m

Fig. 4.3 Rate function for the magnetization of the one-dimensional Ising model. Notice that, as the temperature is lowered (β is increased), the probability of large fluctuations increases.

Exercise 4.4 Consider the one-dimensional Ising model without an external magnetic field (see Section 2.5.1). The variables are N Ising spins σi ∈ {+1, −1}, and their distribution is the Boltzmann distribution PN (σ) = exp[−βE(σ)]/Z with an energy function E(σ) = P −1 − N i=1 σi σi+1 . We want to compute the large-deviation properties of the magnetization, m(σ) =

N 1 X σi , N i=1

(4.27)

using the moment-generating function. (a) Show that ψ(t) = φ(β, t/β) − φ(β, 0), where φ(β, B) is the free-energy density of the model in an external magnetic field B found in eqn (2.63). Explicitly, this gives ψ(t) = log

cosh t +

! p sinh2 t + e−4β . 1 + e−2β

(4.28)

(b) The function ψ(t) is convex and analytic for any β < ∞. By applying eqn (4.26), obtain numerically the rate function Iψ (m). The result is shown in Fig. 4.3 for several inverse temperatures β.

Example 4.9 Consider a Markov chain X0 , X1 , . . . , Xi , . . . taking values in a finite state space X , and assume all the elements of the transition matrix w(x → y) to be strictlypositive. Let us study the large-deviation properties of the empirical average (1/N ) i f (Xi ).

Correlated variables 0.2

0.2 β = 0.5

β = 0.5 β=1 β = 1.25 β = 1.5

β=1

0.15

0.15

β = 1.25

0.1

0.1

ψ(t)

I(m)

β = 1.5

0.05

0.05

0

0 -1

-0.8 -0.6 -0.4 -0.2

0

0.2

0.4

0.6

0.8

1

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

t

m

Fig. 4.4 The rate function for large fluctuations of the magnetization in the Curie–Weiss model (left), and the corresponding generating function (right).

One can show that the limit moment-generating function ψ(t) (see eqn (4.23)) exists, and can be computed using the following recipe. Define the ‘tilted’ transition probabilities as wt (x → y) = w(x → y) exp[t f (y)]. Let λ(t) be the largest solution of the eigenvalue problem φlt (x) wt (x → y) = λ(t) φlt (y) (4.29) x∈X

(which is unique and positive because of the Perron–Frobenius theorem). The moment-generating function is given simply by ψ(t) = log λ(t). An interesting new phenomenon emerges if we consider now the Curie–Weiss model with a vanishing external field. We again study N Ising spins σi ∈ {+1, −1} with a distribution PN (σ) = exp[−βE(σ)]/Z, and the energy function is E(σ) = −

1 σi σj . N

(4.30)

(ij)

We are interested in the large fluctuations of the global magnetization (4.27). In this case one can compute the large-deviation function directly as explained in the following exercise.

Exercise 4.5 Using the arguments of Section 2.5.2, show that, for any m1 < m2 , . PN {m(σ) ∈ [m1 , m2 ]} =

1 ZN (β)

Z

m2

eN φmf (m;β) dm ,

m1

where φmf (m; β) = βm2 /2 − log[2 cosh(βm)].

This exercise shows that the large-deviation principle (4.22) holds, with

(4.31)


I(m) = φmf (m∗ ; β) − φmf (m; β) ,

(4.32)

and m∗ (β) is the largest solution of the Curie–Weiss equation m = tanh(βm). The function I(m) is represented in Fig. 4.4, left frame, for several values of the inverse temperature β. For β < βc = 1, I(m) is convex and has its unique minimum at m = 0. A new situation appears when β > βc . The function I(m) is non-convex, with two degenerate minima at m = ±m∗ (β). In words, the system can be found in either of two well-distinguished ‘states’: the states of positive and negative magnetization. There is no longer a unique typical value of the magnetization such that large fluctuations away from this value are exponentially rare. Let us now look at what happens if one uses the generating-function approach. It is easy to see that the limit (4.23) exists and is given by ψ(t) =

sup [mt − I(m)] .

(4.33)

m∈[−1,1]

While at high temperature, β < 1, ψ(t) is convex and analytic, for β > 1 ψ(t) develops a singularity at t = 0. In particular, we have ψ (0+) = m∗ (β) = −ψ (0−). We now compute Iψ (m) using eqn (4.26). A little thought shows that, for any m ∈ [−m∗ (β), m∗ (β)], the supremum is achieved for t = 0, which yields Iψ (m) = 0. Outside this interval, the supremum is achieved at the unique solution of ψ (t) = m, and Iψ (m). As anticipated, Iψ (m) is the convex envelope of I(m). In the range (−m∗ (β), m∗ (β)), an . estimate of the magnetization fluctuations through the function = exp(−N Iψ (m)) would overestimate the fluctuations. 4.3.2

The G¨ artner–Ellis theorem

The G¨ artner–Ellis theorem has several formulations, which usually require some technical definitions beforehand. Here we shall state it in a simplified (and somewhat weakened) form. We need only the definition of an exposed point: x ∈ R is an exposed point of the function F : R → R if there exists t ∈ R such that ty − F (y) > tx − F (x) for any y = x. If, for instance, F is convex, a sufficient condition for x to be an exposed point is that F is twice differentiable at x, with F (x) > 0. Theorem 4.10. (G¨ artner–Ellis) Consider a function f (x) (not necessarily of the form (4.21)) and assume that the moment-generating function ψN (t) defined in eqn (4.23) exists and has a finite limit ψ(t) = limN →∞ ψN (t) for any t ∈ R. Define Iψ (·) as the inverse Legendre transform of eqn (4.26), and let E be the set of exposed points of Iψ (·). 1. For any closed set F ∈ R, 1 log PN (f ∈ F ) ≤ − inf Iψ (f ) . f ∈F N

(4.34)

1 log PN (f ∈ G) ≥ − inf Iψ (f ) . f ∈G∩E N

(4.35)

lim sup N →∞

2. For any open set G ∈ R, lim sup N →∞

The Gibbs free energy

3. If, moreover, ψ(t) is differentiable for any t ∈ R, then the last statement holds true if the inf is taken over the whole set G (rather than over G ∩ E). Informally, the inverse Legendre transform (4.26) generically yields an upper bound on the probability of a large fluctuation of the macroscopic observable. This upper bound is tight unless a ‘first-order phase transition’ occurs, corresponding to a discontinuity in the first derivative of ψ(t), as we saw in the low-temperature phase of the Curie–Weiss model. It is worth mentioning that ψ(t) can be non-analytic at a point t∗ even though its first derivative is continuous at t∗ . This corresponds, in statistical-mechanics jargon, to a ‘higher-order’ phase transition. Such phenomena have interesting probabilistic interpretations too. 4.3.3

Typical sequences

Let us return to the concept of typical sequences introduced in Section 4.2. More precisely, we want to investigate the large deviations of the probability itself, measured by r(x) = −(1/N ) log P (x). For independent random variables, the study in Section 4.2.3 led to the concept of ε-typical sequences. What can one say about general sequences? Let us compute the corresponding moment-generating function (4.23): ⎧ ⎫ ⎨ ⎬ 1 log PN (x)1−t . ψN (t) = (4.36) ⎩ ⎭ N x Without loss of generality, we can assume PN (x) to have the Boltzmann form PN (x) =

1 exp{−βEN (x)} , ZN (β)

(4.37)

with energy function EN (x). Inserting this into eqn (4.36), we get ψN (t) = βfN (β) − βfN (β(1 − t)) ,

(4.38)

where fN (β) = −(1/N ) log ZN (β) is the free-energy density of the system with energy function EN (x) at inverse temperature β. Let us assume that the thermodynamic limit f (β) = limN →∞ fN (β) exists and is finite. It follows that the limiting generating function ψ(t) exists and we can apply the Gärtner–Ellis theorem to compute the probability of a large fluctuation of r(x). As long as f (β) is analytic, large fluctuations are exponentially depressed and the asymptotic equipartition property of independent random variables is essentially recovered. On the other hand, if there is a phase transition at β = βc , where the first derivative of f (β) is discontinuous, then the likelihood r(x) may take several distinct values with a non-vanishing probability. This is what happened in our Curie–Weiss example.

4.4 4.4.1

The Gibbs free energy Variational principle

In the introduction to statistical physics in Chapter 2, we assumed that the probability distribution of the configurations of a physical system was the Boltzmann distribution.


It turns out that this distribution can be obtained from a variational principle. This is interesting, both as a matter of principle and for the purpose of finding approximation schemes. Consider a system with a configuration space X , and a real-valued energy function E(x) defined on this space. The Boltzmann distribution is µβ (x) = exp[−β(E(x) − F (β))], where F (β), the ‘free energy’, is a function of the inverse temperature β defined by the fact that x∈X µβ (x) = 1. We define the Gibbs free energy G[P ] as the following real-valued functional over the space of probability distributions P (x) on X :

G[P ] =

P (x)E(x) +

x∈X

1 P (x) log P (x) . β

(4.39)

x∈X

(The Gibbs free energy should not be confused with F (β).) It is easy to rewrite the Gibbs free energy in terms of the KL divergence between P (x) and the Boltzmann distribution µβ (x):

G[P ] =

1 D(P ||µβ ) + F (β) . β

(4.40)

This representation implies straightforwardly the following proposition (the Gibbs variational principle). Proposition 4.11 The Gibbs free energy G[P ] is a convex functional of P (x), and it achieves its unique minimum on the Boltzmann distribution P (x) = µβ (x). Moreover, G[µβ ] = F (β), where F (β) is the free energy.

The relation between the Gibbs free energy and the Kullback–Leibler divergence in eqn (4.40) implies a simple probabilistic interpretation of the Gibbs variational principle. Imagine that a large number N of copies of the same physical system have been prepared. Each copy is described by the same energy function E(x). Now consider the empirical distribution P (x) of the N copies. Typically, P (x) will be close to the Boltzmann distribution µβ (x). Sanov’s theorem implies that the probability of an . ‘atypical’ distribution is exponentially small in N : P[P ] = exp[−N (G[P ] − F (β))] . When the partition function of a system cannot be computed exactly, the above result suggests a general line of approach for estimating the free energy: one can minimize the Gibbs free energy in some restricted subspace of ‘trial probability distributions’ P (x). These trial distributions should be simple enough that G[P ] can be computed, but the restricted subspace should also contain distributions which are able to give a good approximation to the true behaviour of the physical system. For each new physical system one will thus need to find a good restricted subspace.

The Gibbs free energy

2

2

0

0

-2

-2

-4

-4 F(t)

-6

F(t) -6

G(t) a(t)

-8

G(t) a(t)

-8

-10

-10 -6

-5

-4

-3

-2 t

-1

0

1

2

-6

-5

-4

-3

-2 t

-1

0

1

2

Fig. 4.5 Variational estimates of the free energy of the model (4.41), using the Gibbs free energy with the trial distribution (4.42) on the left and (4.44) on the right.

Exercise 4.6 Consider a system with configuration space X = R and energy E(x) =

1 2 1 4 tx + x , 2 4

(4.41)

with t“ ∈ R. Let us ” find an upper bound on its free energy at temperature β = 1, F (t) = R − log e−E(x) dx , using the Gibbs variational principle. We first use the following family of Gaussian trial probability distributions: Qa (x) = √

2 1 e−x /2a . 2πa

(4.42)

Show that the corresponding Gibbs free-energy at β = 1 is: G[Qa ] =

1 1 3 ta + a2 − (1 + log 2πa) ≡ G(a, t) . 2 4 2

(4.43)

The Gibbs principle implies that F (t) ≤ mina G(a, t). In Fig. 4.5, we have plotted the optimal value of a, aopt (t) = argmina G(a, t), and the corresponding estimate Gopt (t) = G(aopt (t), t).

Exercise 4.7 Consider the same problem with the family of trial distributions 2 1 Qa (x) = √ e−(x−a) /2 . 2π

(4.44)

Determine the optimal value of aopt , and the corresponding upper bound on F (t) (see Fig. 4.5). Notice the bifurcation at tcr = −3. For t > tcr , one finds that aopt (t) = 0, while G[Qa ] has two degenerate local minima a = ±aopt (t) for t ≤ tcr .

¼


4.4.2

Mean-field approximation

An important application of the variational approximation using the Gibbs free energy is the mean-field approximation. Usually, we do not know how to compute the free energy of a problem with many interacting variables. A natural approximation is to take the class of distributions over independent variables as trial family. In order to illustrate this idea, consider the Ising model on a d-dimensional cubic lattice L of linear size L (i.e. L = [L]d ) (see Section 2.5). The energy function is 1 E(σ) = − σi σj − B σi . (4.45) d i∈L

(ij)

Notice that we have chosen a particular scale of energy by deciding that the strength of the interaction between the spins is J = 1/d. This choice will allow us to make direct contact with the Curie–Weiss model. For the sake of simplicity, we assume periodic boundary conditions. This means that two sites i = (i1 , . . . , id ) and j = (j1 , . . . , jd ) are considered nearest neighbours if, for some l ∈ {1, . . . , d}, il − jl = ±1 (mod L) and il = jl for any l = l. The sum over (ij) in eqn (4.45) runs over all nearest-neighbour pairs in L. In order to obtain a variational estimate of the free energy F (β) at inverse temperature β, we evaluate the Gibbs free energy on the following trial distribution: qm (σi ) , (4.46) Qm (σ) = i∈L

where qm (+) = (1 + m)/2, qm (−) = (1 − m)/2, and m ∈ [−1, +1]. Note that, under Qm (σ), the σi are i.i.d. random variables with expectation m. It is easy to evaluate the density of the Gibbs free energy, g(m; β, B) ≡ G[Qm ]/Ld , on this distribution: 1+m 1 1 2 . (4.47) g(m; β, B) = − m − B m − H 2 β 2 The Gibbs variational principle implies an upper bound on the free-energy density fd (β, B) ≤ inf m g(m; β, B). Notice that, apart from an additive constant, this expression (4.47) has the same form as the solution of the Curie–Weiss model (see eqn (2.79)). This immediately implies that fd (β, B) ≤ fCW (β, h) − 21 . The minimization of g(m; β, B) has already been discussed in Section 2.5.2; we shall not repeat it here. Let us just mention that, in zero magnetic field (B = 0), at low enough temperature, the mean-field approximation always predicts that the Ising model has a phase transition to a ferromagnetic phase. While this prediction is qualitatively correct when d ≥ 2, it is wrong when d = 1, as we saw through the exact solution in Section 2.5.1. Indeed, the mean-field approximation becomes better the larger the dimension d, and it is asymptotically exact for d → ∞.

4.5

The Monte Carlo method

The Monte Carlo method is an important generic tool which is common to probability theory, statistical physics, and combinatorial optimization. In all of these fields, we


½

are often confronted with the problem of sampling a configuration x ∈ X N (here we assume X to be a finite space) from a given distribution P (x). This can be quite difficult when N is large, because there are too many configurations, because the typical configurations are exponentially rare, and/or because the distribution P (x) is specified by the Boltzmann formula with an unknown normalization (the partition function). A general approach is to construct a Markov chain which is guaranteed to converge to the desired P (x), and then simulate it on a computer. The computer is of course assumed to have access to some source of randomness: in practice, pseudo-random number generators are used. If the chain is simulated for a long enough time, the final configuration has a distribution ‘close’ to P (x). In practice, the Markov chain is defined by a set of transition rates w(x → y) with x, y ∈ X N which satisfy the following conditions: 1. The chain is irreducible, i.e. for any pair of configurations x and y there exists a path (x0 , x1 , . . . xn ) of length n, connecting x to y with non-zero probability. This means that x0 = x, xn = y and w(xi → xi+1 ) > 0 for i = 0, . . . , n − 1. 2. The chain is aperiodic: for any pair x and y, there exists a positive integer n(x, y) such that, for any n ≥ n(x, y), there exists a path of length n connecting x to y with non-zero probability. Note that, for an irreducible chain, aperiodicity is easily enforced by allowing the configuration to remain unchanged with non-zero probability: w(x → x) > 0. 3. The distribution P (x) is stationary with respect to the probabilities w(x → y): P (x) w(x → y) = P (y) . (4.48) x

Sometimes a stronger condition (implying stationarity) is satisfied by the transition probabilities. In this case, for each pair of configurations x, y such that either w(x → y) > 0 or w(y → x) > 0, one has P (x) w(x → y) = P (y) w(y → x) .

(4.49)

This condition is referred to as reversibility or detailed balance. The strategy of designing and simulating such a process in order to sample from P (x) goes under the name of the dynamic Monte Carlo method or the Markov chain Monte Carlo (MCMC) method (hereafter we shall refer to it simply as the Monte Carlo method). The theoretical basis for such an approach is provided by a classic theorem. Theorem 4.12 Assume the rates w(x → y) to satisfy the conditions 1–3 above. Let X 0 , X 1 , . . . , X t , . . . be random variables distributed according to the Markov chain with rates w(x → y) and initial condition X 0 = x0 . Let f : X N → R be any-real valued function. Then: 1. The probability distribution of Xt converges to the stationary distribution: lim P[X t = x] = P (x) .

t→∞

(4.50)

¾


2. Time averages converge to averages over the stationary distribution: 1 f (X s ) = P (x)f (x) t→∞ t x s=1 t

lim

almost surely.

(4.51)

The proof of this theorem can be found in any textbook on Markov processes. Here we shall illustrate it by considering some simple Monte Carlo algorithms which are frequently used in statistical mechanics (although they are by no means the most efficient ones). Consider a system of N Ising spins σ = (σ1 . . . σN ) with energy function E(σ) and inverse temperature β. We are interested in sampling the Boltzmann distribution µβ . The Metropolis algorithm is defined as follows. We denote by σ (i) the configuration (i) which coincides with σ except for the site i (σi = −σi ), and let ∆Ei (σ) ≡ E(σ (i) ) − E(σ). At each step, an integer i ∈ [N ] is chosen randomly with a uniform distribution, and the spin σi is flipped with probability wi (σ) = exp{−β max[∆Ei (σ), 0]} . More explicitly, the transition probabilities are given by N N 1 1 (i) w(σ → τ ) = wi (σ) I(τ = σ ) + 1 − wi (σ) I(τ = σ) . N i=1 N i=1

(4.52)

(4.53)

It is easy to check that this definition satisfies both the irreducibility and the stationarity conditions for any energy function E(σ) and inverse temperature β < 1. Furthermore, the chain satisfies the detailed balance condition µβ (σ) wi (σ) = µβ (σ (i) ) wi (σ (i) ) .

(4.54)

Whether the condition of aperiodicity is fulfilled depends on the energy function. It is easy to construct systems for which it does not hold. Take, for instance, a single spin, N = 1, and let E(σ) = 0: the spin is flipped at each step, and there is no way to have a transition from σ = +1 to σ = −1 in an even number of steps. (Luckily, this kind of pathology is easily cured by modifying the algorithm as follows. At each step, with probability 1 − ε, a site i is chosen and a spin flip is proposed as above. With probability ε, nothing is done, i.e. a null transition σ → σ is realized.)

Exercise 4.8 Variants of this chain can be obtained by changing the flipping probabilities given by eqn (4.52). A popular choice is the heat bath algorithm (also referred to as Glauber dynamics): wi (σ) =

» „ «– β∆Ei (σ) 1 1 − tanh . 2 2

Prove irreducibility, aperiodicity, and stationarity for these transition probabilities.

(4.55)


¿

One reason that makes the heat bath algorithm particularly interesting is that it can be easily generalized for the purpose of sampling from a distribution P (x) in any system whose configuration space has the form X N . In this algorithm, one chooses a variable index i, fixes all the other variables, and assigns a new value to the i-th one according to its conditional distribution. A more precise description is provided by the following pseudocode (Recall that, given a vector x ∈ X N , we denote by x∼i the N − 1-dimensional vector obtained by removing the i-th component of x.) Heat bath algorithm (P (x), number of iterations r) 1: Generate x(0) uniformly at random in X N ; 2: for t = 1 to t = r: 3: Draw a uniformly random integer i ∈ {1, . . . , N } 4: For each z ∈ X , compute (t−1)

P (Xi = z|X ∼i = x∼i 5: 6: 7:

(t) xj

(t−1) xj

(t−1)

)=

P

P (Xi =z,X ∼i =x∼i

)

(t−1)

P (Xi =z ,X ∼i =x∼i (t) and xi = z,

z ∈X

)

;

= for each j = i, Set (t−1) where z is drawn from the distribution P (Xi = z|X ∼i = x∼i ); end return the sequence x(t) , t = 1, · · · , r.

Let us stress that this algorithm only requires one to be able to compute the probability P (x) up to a multiplicative constant. If, for instance, P (x) is given by Boltzmann’s law (see Section 2.1) it is enough to be able to compute the energy E(x) of a configuration, and it is not necessary to compute the partition function Z(β). This is a very general method for defining a Markov chain with the desired properties. The proof is left as exercise.

Exercise 4.9 Assuming for simplicity that ∀x, P (x) > 0, prove irreducibility, aperiodicity and stationarity for the heat bath algorithm.

Theorem 4.12 confirms that the Monte Carlo method is indeed a viable approach for sampling from a given probability distribution. However, it does not provide any information concerning its computational efficiency. In order to discuss this issue, it is convenient to assume that simulating a single step X t → X t+1 of the Markov chain has a time cost equal to one. This assumption is a good one as long as sampling a new configuration requires a number of operations that depends only mildly (e.g. polynomially) on the system size. This is the case in the two examples provided above, and we shall stick here to this simple scenario. Computational efficiency reduces therefore to the following question: how many steps of the Markov chain should be simulated? Of course, there is no unique answer to such a generic question. We shall limit ourselves to introducing two important figures of merit. The first concerns the following problem: how many steps should be simulated in


order to produce a single configuration x which is distributed approximately according to P (x)? In order to specify what is meant by ‘approximately’ we have to introduce a notion of distance between two distributions P1 (·) and P2 (·) on X N . A widely used definition is given by the variation distance ||P1 − P2 || =

1 |P1 (x) − P2 (x)| . 2 N

(4.56)

x∈X

Consider now a Markov chain satisfying the conditions 1–3 above with respect to a stationary distribution P (x), and denote by Pt (x|x0 ) the distribution of X t conditional on the initial condition X 0 = x0 . Let dx0 (t) = ||Pt (·|x0 ) − P (·)|| be the distance from the stationary distribution. The mixing time (or variation threshold time) is defined as τeq (ε) = min{t > 0 : sup dx0 (t) ≤ ε} .

(4.57)

x0

In this book, we shall often refer informally to this quantity as the equilibration time. The number ε can be chosen arbitrarily, a change in ε usually implying a simple multiplicative change in τeq (ε). For of this reason, the convention ε = 1/e is sometimes adopted. Rather than producing a single configuration with the prescribed distribution, one is often interested in computing the expectation value of some observable O(x). In principle, this can be done by averaging over many steps of the Markov chain as suggested by eqn (4.51). It is therefore natural to ask the following question. Assume that the initial condition X 0 is distributed according to the stationary distribution P (x). This can be obtained (to a good approximation) by simulating τeq (ε) steps of the chain in a preliminary (equilibration) phase. We shall denote by · the expectation with respect to the Markov chain with this initial condition. How many steps should we average over in order to get expectation values within some prescribed accuracy? In other words, we estimate P (x)O(x) ≡ EP O by OT ≡ It is clear that OT =

Var(OT ) =

T −1 1 O(X t ) . T t=0

(4.58)

P (x)O(x). Let us compute the variance of this estimator: T −1 T −1 1 1 O ; O = (T − t)O0 ; Ot , s t T 2 s,t=0 T 2 t=0

(4.59)

where we have used the notation Ot ≡ O(X t ). Let us introduce the autocorrelation T −1 function CO (t−s) ≡ Os ; Ot /O0 ; O0 , so that Var(OT ) = (O0 ; O0 /T 2 ) t=0 (T − t) CO (t). General results for Markov chains on finite state spaces imply that CO (t) decreases exponentially as t → ∞. Therefore, for large T , we have Var(OT ) =

O τint [EP O2 − (EP O)2 ] + O(T −2 ) . T

(4.60)


1 0 -1

2.0⋅105

5.0⋅105

8.0⋅105

1.1⋅106

1.4⋅106

2.0⋅105

5.0⋅105

8.0⋅105

1.1⋅106

1.4⋅106

2.0⋅105

5.0⋅105

8.0⋅105

1.1⋅106

1.4⋅106

1 0 -1 1 0 -1

Fig. 4.6 Monte Carlo simulation of the Curie–Weiss model at three different temperatures: from top to bottom β = 0.8, 1.0, and 1.2. Here, we reproduce the global magnetization m(σ) as a function of the number of iterations. O The integrated autocorrelation time τint is given by

O τint ≡

∞

CO (t) ,

(4.61)

t=0

and provides a reference for estimating how long the Monte Carlo simulation should be run in order to achieve some prescribed accuracy. equation (4.60) can be interpreted O by saying that one statistically independent estimate of EP O is obtained every τint iterations.

Example 4.13 Consider the Curie–Weiss model (see Section 2.5.2), at inverse temperature β, and use the heat bath algorithm described above in order to sample from the Boltzmann distribution. In Fig. 4.6 we reproduce the evolution of the global magnetization m(σ) during three different simulations at inverse temperatures β = 0.8, 1.0, and 1.2 for a model of N = 150 spin. In all cases, we initialized the Markov chain by drawing a random configuration with uniform probability. A spectacular effect occurs at the lowest temperature, β = 1.2. Although the Boltzmann average of the global magnetization vanishes, i.e. m(σ) = 0, the sign of the magnetization remains unchanged over extremely long time scales. It is clear that the equilibration time is at least as large as these scales. An order-of-magnitude estimate would be τeq > 105 . Furthermore, this equilibration time diverges exponentially at large N . Sampling from the Boltzmann distribution using the present algorithm becomes exceedingly difficult at low temperature.


4.6

Simulated annealing

As we mentioned in Section 3.5, any optimization problem can be ‘embedded’ in a statistical-mechanics problem. The idea is to interpret the cost function E(x), x ∈ X N , as the energy of a statistical-mechanics system and consider the Boltzmann distribution µβ (x) = exp[−βE(x)]/Z. In the low-temperature limit β → ∞, the distribution becomes concentrated on the minima of E(x), and the original optimization setting is recovered. Since the Monte Carlo method provides a general technique for sampling from the Boltzmann distribution, one may wonder whether it can be used, in the β → ∞ limit, as an optimization technique. A simple-minded approach would be to take β = ∞ at the outset. Such a strategy is generally referred to as a quench in statistical physics and a greedy search in combinatorial optimization, and is often bound to fail. Consider the stationarity condition (4.48) and rewrite it using the Boltzmann formula:

e−β [E(x)−E(y)] w(x → y) = 1 .

(4.62)

x

All the terms in the sum on the left-hand side must be less than or equal to one. This implies 0 ≤ w(x → y) ≤ exp{−β [E(y) − E(x)]}. Therefore, for any pair of configurations x, y such that E(y) > E(x), we have w(x → y) → 0 in the β → ∞ limit. In other words, the energy is always non-increasing along the trajectories of a zero-temperature Monte Carlo algorithm. As a consequence, the corresponding Markov chain is not irreducible (although it is irreducible at any β < ∞) and is not guaranteed to converge to the equilibrium distribution, i.e. to find a global minimum of E(x). In fact, if the energy landscape E(x) has local minima, this algorithm can get trapped in such a local minimum forever. Another simple-minded approach would be to set β to some large but finite value. Although the Boltzmann distribution gives some weight to near-optimal configurations, the algorithm will also visit, from time to time, optimal configurations which are the most probable ones. How large should β be? How much time must we wait before an optimal configuration is visited? We can assume without loss of generality that the minimum of the cost function (the ground state energy) is zero: E0 = 0. A meaningful quantity to look at is the probability for E(x) = 0 under the Boltzmann distribution at an inverse temperature β. We can easily compute the logarithmic moment-generating function of the energy: ⎡ ⎤ −(β−t)E(x) 1 1 xe log ⎣ log . µβ (x) etE(x) ⎦ = ψN (t) = −βE(x) N N xe x

(4.63)

This is given by ψN (t) = φN (β − t) − φN (β), where φN (β) is the free-entropy density at inverse temperature β. Clearly µβ [E(x) = 0] = exp[N ψN (−∞)] = exp{N [φN (∞) − φN (β)]}, and the average time that we must wait before visiting the optimal configuration is 1/µβ [E(x) = 0] = exp[−N ψN (−∞)].

Appendix: A physicist’s approach to Sanov’s theorem

Exercise 4.10 Assume that the cost function takes integer values E = 0, 1, 2 . . . and denote XE the set of configurations x such that E(x) = E. You want the Monte Carlo trajectories to spend a fraction (1 − ε) of the time on optimal solutions. Show that the temperature must be chosen such that „ β = log

|X1 | ε|X0 |

« + Θ(ε) .

(4.64)

In Section 2.4 we argued that, for many statistical-mechanics models, the freeentropy density has a finite thermodynamic limit φ(β) = limN →∞ φN (β). In the following chapters, we shall show that this is also the case for several interesting optimization problems. This implies that µβ [E(x) = 0] vanishes in the N → ∞ limit. In order to have a non-negligibile probability of hitting a solution of the optimization problem, β must be scaled with N in such a way that β → ∞ as N → ∞. On the other hand, if we let β → ∞, we are going to face the reducibility problem mentioned above. Although the Markov chain is formally irreducible, its equilibration time will diverge as β → ∞. The idea of simulated annealing consists in letting β vary with time. More precisely, one decides on an annealing schedule {(β1 , n1 ); (β2 , n2 ); . . . (βL , nL )}, with inverse temperatures βi ∈ [0, ∞] and integers ni > 0. The algorithm is initialized on a configuration x0 and executes n1 Monte Carlo steps at temperature β1 , n2 steps at temperature β2 , . . . , and nL steps at temperature βL . The final configuration of each cycle i (with i = 1, . . . , L − 1) is used as the initial configuration of the next cycle. Mathematically, such a process is a time-dependent Markov chain. The common wisdom about the simulated-annealing algorithm is that varying the temperature with time should help one to avoid the two problems encountered above. Usually, one takes the βi to be an increasing sequence. In the first stages, a small β should help in equilibrating across the space of configurations X N . As the temperature is lowered, the probability distribution becomes concentrated on the lowest-energy regions of this space. Finally, in the late stages, a large β forces the system to fix the few wrong details, and to find a solution. Of course, this image is somewhat simplistic, but we shall see later with some examples, in particular in Chapter 15, how the method works in practice, and what its limitations are.

4.7

Appendix: A physicist’s approach to Sanov’s theorem

Let us show how the formulae of Sanov’s theorem can be obtained using the ‘fieldtheoretic’ type of approach used in statistical physics. The theorem is easy to prove by standard techniques. The aim of this section is not so much to give a proof, but rather to show, with a simple example, a type of calculation that is very common in physics, and which can be powerful. We shall not aim at a rigorous derivation. The probability that the type of the sequence x1 , · · · , xN is equal to q(x) can be written as


$ N 1 P[q(x)] = E I q(x) = δx,xi N i=1 x∈X N 1 = p(x1 ) · · · p(xN ) I q(x) = δx,xi , N i=1 x ···x #

1

(4.65)

N

where we have introduced the Kronecker symbol δx,y , equal to 1 if x = y and 0 otherwise. A typical approach in field theory is to introduce some auxiliary variables N in order to enforce the constraint that q(x) = (1/N ) i=1 δx,xi . For each x ∈ X , one introduces a variable λ(x), and uses an ‘integral representation’ of the constraint in the form N N 2π dλ(x) 1 exp iλ(x) N q(x) − . (4.66) δx,xi = δx,xi I q(x) = N i=1 2π 0 i=1 Dropping q-independent factors, we get P[q(x)] = C dλ(x) exp{N S[λ]} ,

(4.67)

x∈X

where C is a normalization constant, and the action S is given by −iλ(x) S[λ] = i . λ(x)q(x) + log p(x) e x

(4.68)

x

In the large-N limit, the integral in eqn (4.67) can be evaluated by the saddle point method. The saddle point λ(x) = λ∗ (x) is found by solving the stationarity equations ∂S/∂λ(x) = 0 for any x ∈ X . One obtains a family of solutions −iλ(x) = C + log(q(x)/p(x)), with C arbitrary. The freedom in the choice of C comes from the fact that x ( i δx,xi ) = N for any configuration x1 . . . xN , and therefore one of the constraints is in fact redundant. This choice can be made arbitrarily: regardless of the choice, the action at the saddle point is S[λ∗ ] = S0 −

x

q(x) log

q(x) , p(x)

(4.69)

. where S0 is a q-independent constant. One thus gets P[q(x)] = exp[−N D(q||p)]. The reader who has never encountered this type of reasoning before may wonder why use such an indirect approach. It turns out that it is a very common formalism in statistical physics, where similar methods are also applied, under the name ‘field theory’, to continuous spaces X (some implicit discretization is then usually assumed at intermediate steps, and the correct definition of a continuum limit is often not obvious). In particular, the reader interested in the statistical-physics approach to optimization problems or information theory will often find this type of calculation in research papers. One of the advantages of this approach is that it provides a formal solution to a large variety of problems. The quantity to be computed is expressed

Notes

in an integral form as in eqn (4.67). In problems that have a ‘mean-field’ structure, the dimension of the space over which the integration is performed does not depend upon N . Therefore its leading exponential behaviour at large N can be obtained by saddle point methods. The reader who wants to get some practice with this approach is invited to ‘derive’ the various theorems and corollaries of this chapter in this way.

Notes The theory of large deviations is set out in the book by Dembo and Zeitouni (1998), and its connections to statistical physics can be found in Ellis (1985). A discussion of statistical physics from a computational perspective, with a nice introduction to Monte Carlo methods, is provided by Krauth (2006). Markov chains on discrete state spaces are treated by Norris (1997), and a more formal presentation of the Monte Carlo method is given by Sokal (1996). Simulated annealing was introduced by Kirkpatrick et al. (1983). One of its main qualities is that it is very versatile. In some sense it is a completely ‘universal’ optimization algorithm: it can be defined without reference to any particular problem. Fo this reason, however, it often overlooks important structures that may help in solving a specific problem.


Part II Independence


5 The random energy model The random energy model (REM) is probably the simplest statistical-physics model of a disordered system which exhibits a phase transition. It is not supposed to give a realistic description of any physical system, but it provides an example with which various concepts and methods can be studied in full detail. Moreover, owing to its simplicity, the same mathematical structure appears in a large number of contexts. This is witnessed by the examples from information theory and combinatorial optimization presented in the next two chapters. The model is defined in Section 5.1 and its thermodynamic properties are studied in Section 5.2. The simple approach developed in these sections turns out to be useful in a large variety of problems. A more detailed, and more involved, study of the low-temperature phase is developed in Section 5.3. Section 5.4 provides an introduction to the ‘annealed approximation,’ which will be useful in more complicated models. Finally, in Section 5.5 we consider a variation of the REM that provides a ‘cartoon’ for the structure of the set of solutions of random constraint satisfaction problems.

5.1

Definition of the model

A statistical-mechanics model is defined by a set of configurations and an energy function defined on this space. In the REM, there are M = 2N configurations (as in a system of N Ising spins), denoted by indices i, j, . . . ∈ {1, . . . , 2N }. The REM is a disordered model: the energy is not a deterministic function, but rather a stochastic process. A particular realization of such a process is usually called a sample (or instance). In the REM, one makes the simplest possible choice for this process: the energies {Ei } are i.i.d. random variables (the energy of a configuration is also called an energy level). For definiteness, we shall keep here to the case where the energies have Gaussian distribution with zero mean and variance N/2, but other distributions could be studied as well. The scaling with N of the distribution should always be chosen in such a way that thermodynamic potentials are extensive. The pdf for the energy Ei of the state i is given by 2 1 e−E /N . (5.1) P (E) = √ πN Given an instance of the REM, defined by the 2N real numbers {E1 , E2 , . . . , E2N }, one assigns to each configuration j a Boltzmann probability µβ (j) in the usual way: µβ (j) =

1 exp (−βEj ) . Z

(5.2)

The random energy model

where β = 1/T is the inverse of the temperature, and the normalization factor Z, the partition function, is given by N

Z=

2

exp (−βEj ) .

(5.3)

j=1

Notice that Z depends upon the temperature β, the ‘sample size’ N , and the particular realization of the energy levels E1 , . . . , EM . We shall write Z = ZN (β) to emphasize the dependence of the partition function upon N and β. It is important not to be confused by the existence of two levels of probability in the REM, as in all disordered systems. We are interested in the properties of a probability distribution, the Boltzmann distribution (5.2), which is itself a random object because the energy levels are random variables. Physically, a particular realization of the energy function corresponds to a given sample of some substance whose microscopic features cannot be controlled experimentally. This is what happens, for instance, in a metallic alloy: only the proportions of the various components can be controlled. The precise positions of the atoms of each species can be described by random variables. The expectation value with respect to the sample realization will be denoted in the following by E(·). For a given sample, Boltzmann’s law (5.2) gives the probability of occupying the various possible configurations, according to their energies. The average with respect to the Boltzmann distribution will be denoted by · . In experiments one deals with a single (or a few) sample(s) of a given disordered material. One might therefore be interested in computing the various thermodynamic potentials (the free energy FN , internal energy UN , or entropy SN ) for one given sample. This is an extremely difficult task. However, in most cases, as N → ∞, the probability distributions of intensive thermodynamic potentials concentrate around their expected values. Formally, for any tolerance θ > 0, XN XN −E ≥ θ = 0, (5.4) lim P N →∞ N N where X is a thermodynamic potential (X = F, S, U, . . . ). In statistical physics, the quantity X is then said to be self-averaging (in probability theory, one says that it concentrates). This essential property can be summarized in plain language by saying that almost all large samples ‘behave’ in the same way. This is the reason why different samples of alloys with the same chemical composition have the same thermodynamic properties. Often the convergence is exponentially fast in N (this happens, for instance, in the REM): this means that the expected value E XN provides a good description of the system even at moderate sizes.

5.2

Thermodynamics of the REM

In this section we compute the thermodynamic potentials of the REM in the thermodynamic limit N → ∞. Our strategy consists in estimating the microcanonical entropy density, which was introduced in Section 2.4. This knowledge is then used for computing the partition function Z to leading exponential order for large N .


5.2.1

Direct evaluation of the entropy

Let us consider an interval of energy I = [N ε, N (ε + δ)], and denote by N (ε, ε + δ) the number of configurations i such that Ei ∈ I. Each energy level Ei belongs to I independently, with probability - ε+δ 2 N e−N x dx . (5.5) PI = π ε Therefore N (ε, ε + δ) is a binomial random variable, and its expectation and variance are given by E N (ε, ε + δ) = 2N PI ,

Var N (ε, ε + δ) = 2N PI [1 − PI ] .

(5.6)

Because of the appropriate scaling of the interval I with N , the probability PI depends exponentially upon N . To exponential accuracy, we thus have . (5.7) E N (ε, ε + δ) = exp N max sa (x) , x∈[ε,ε+δ] VarN (ε, ε + δ) . = exp −N max sa (x) , (5.8) [E N (ε, ε + δ)]2 x∈[ε,ε+δ] where√sa (x) ≡ log 2 − x2 . Note that sa (x) ≥ 0 if and only if x ∈ [−ε∗ , ε∗ ], where ε∗ = log 2. The intuitive content of these equalities is the following. When ε is outside the interval [−ε∗ , ε∗ ], the average density of energy levels is exponentially small in N : for a typical sample, there is no configuration at energy Ei ≈ N ε. In contrast, when ε ∈] − ε∗ , ε∗ [, there is an exponentially large density of levels, and the fluctuations of this density are very small. This result is illustrated by a small numerical experiment in Fig. 5.1. We now give a more formal version of this statement. Proposition 5.1 Define the entropy function sa (ε) = log 2 − ε2 if |ε| ≤ ε∗ , (5.9) s(ε) = −∞ if |ε| > ε∗ . Then, for any pair ε and δ, with probability one, lim

N →∞

1 log N (ε, ε + δ) = sup s(x) . N x∈[ε,ε+δ]

(5.10)

Proof The proof makes simple use of the two moments of the number of energy levels in I, given in eqns (5.7), and (5.8). . Assume first that the interval [ε, ε+δ] is disjoint from [−ε∗ , ε∗ ]. Then E N (ε, ε+δ) = −AN , where A = − supx∈[ε,ε+δ] sa (x) > 0. As N (ε, ε + δ) is an integer, we have the e simple inequality . (5.11) P[N (ε, ε + δ) > 0] ≤ E N (ε, ε + δ) = e−AN . In words, the probability of having an energy level in any fixed interval outside [−ε∗ , ε∗ ] is exponentially small in N . An inequality of the form (5.11) goes under the name of


-1

-0.5

0

ε

0.5

1

-1

-0.5

0

ε

0.5

1

-1

-0.5

0

0.5

1

ε

Fig. 5.1 Histogram of the energy levels for three samples of the random energy model with increasing size: from left to right N = 10, 15, and 20. Here we have plotted N −1 log N (ε, ε+δ) versus ε, with δ = 0.05. The dashed curve gives the N → ∞ analytical prediction (5.9).

Markov inequality, and the general strategy is sometimes called the first-moment method. Assume now that the intersection between [ε, ε + δ] and [−ε∗ , ε∗ ] is an interval of non-zero length. In this case N (ε, ε + δ) is tightly concentrated around its expectation E N (ε, ε + δ), as can be shown using the Chebyshev inequality. For any fixed C > 0, one has N (ε, ε + δ) Var N (ε, ε + δ)2 . −BN =e − 1 > C ≤ 2 , P E N (ε, ε + δ) C [E N (ε, ε + δ)]2

(5.12)

where B = supx∈[ε,ε+δ] sa (x) > 0. Finally, the statement (5.10) follows from the previous estimates by a straightfoward application of the Borel–Cantelli lemma.

Exercise 5.1 (Large deviations) Let Nout (δ) be the total number of configurations j such that |Ej | > N (ε∗ + δ), with δ > 0. Use Markov inequality to show that the fraction of samples in which there exist such configurations is exponentially small.

Besides being an interesting mathematical statement, Proposition 5.1 provides a good quantitative estimate. As shown in Fig. 5.1, even at N = 20, the outcome of a numerical experiment is quite close to the asymptotic prediction. Note that, for energies in the interval ] − ε∗ , ε∗ [, most of the discrepancy is due to the fact that we have dropped subexponential factors in E N (ε, ε+δ): this produces corrections of order Θ(log N/N ) to the asymptotic behaviour (5.10). The contribution due to fluctuations of N (ε, ε + δ) around its average is exponentially small in N .


5.2.2

Thermodynamics and phase transition

From the previous result on the microcanonical entropy density, we now compute the 2N partition function ZN (β) = i=1 exp(−βEi ). In particular, we are interested in intensive thermodynamic potentials such as the free-entropy density φ(β) = limN →∞ (1/N ) log ZN (β). We start with a quick (and loose) argument, using the general approach outlined in Section 2.4. This amounts to discretizing the energy axis using some step size δ, and counting the energy levels in each interval using eqn (5.10). Taking at the end the limit δ → 0 (after the limit N → ∞), one expects to get, to leading exponential order, ε∗ . exp [N (sa (ε) − βε)] d . (5.13) ZN (β) = −ε∗

A rigorous formulation of this result can be obtained by analogy1 with the general equivalence relation stated in Proposition 2.6. We find the following free-entropy density: φ(β) =

max [sa (ε) − βε] .

(5.14)

ε∈[−ε∗ ,ε∗ ]

Note that although every sample of the REM is a new statistical-physics system, with its own thermodynamic potentials, we have found that, with high probability, a random sample has a free-entropy (or free-energy) density arbitrarily close to (5.14). A little more work shows that the internal energy and entropy density concentrate as well. More precisely, for any fixed tolerance θ > 0, we have |(1/N ) log ZN (β) − φ(β)| < θ with probability approaching one as N → ∞. Let us now discuss the physical content of the result (5.14). The optimization problem on the right-hand side can be solved through the geometrical construction illustrated in Fig. 5.2. One has to find a tangent to the curve sa (ε) = log 2 − ε2 with slope β ≥ 0. We denote by εa (β) = −β/2 the abscissa of the tangent point. If εa (β) ∈ [−ε∗ , ε∗ ], then the ‘max’ in eqn (5.14) is realized in εa (β). In the other case εa (β) < −ε∗ (because β ≥ 0), and the ‘max’ is realized in −ε∗ . Therefore we have the following result Proposition 5.2 The free-energy density of the REM, f (β) = −φ(β)/β, is equal to 1 −√ 4 β − log 2/β if β ≤ βc , where βc = 2 log 2 . (5.15) f (β) = if β > βc , − log 2

This shows that a phase transition (i.e. a non-analyticity of √ the free-energy density) takes place at the inverse critical temperature βc = 1/Tc = 2 log 2. It is a second-order phase transition in the sense that the derivative of f (β) is continuous, but because of the condensation phenomenon which we shall discuss in Section 5.3, it is often called a ‘random first-order’ transition. The other thermodynamic potentials are obtained through the usual formulae (see Section 2.2). They are plotted in Fig. 5.3. 1 The task is, however, more difficult here, because the density of energy levels N (ε, ε + δ) is a random function whose fluctuations must be controlled.


0.8

slope β 0.6 0.4 0.2 0 -0.2

ε∗

-0.4 -1

ε1 -0.5

0

0.5

1

ε Fig. 5.2 The ‘annealed’ entropy density sa (ε) = log 2 − ε2 of the REM as a function of the energy density ε. The canonical entropy density s(β) is the ordinate of the point with slope dsa /dε = β when this point lies within the interval [−ε∗ , ε∗ ] (this is for instance the case at ε = ε1 in the plot), and s(β) = 0 otherwise. This gives rise to a phase transition at √ βc = 2 log 2.

The two temperature regimes or ‘phases’, β < βc and β > βc have distinct qualitative properties, which are most easily characterized through the thermodynamic potentials.

• In the high-temperature phase, i.e. for β ≤ βc , the energy and entropy densities are u(β) = −β/2 and s(β) = log 2 − β 2 /4, respectively. The Boltzmann measure is dominated by configurations with an energy Ei ≈ −N β/2. There is an exponentially large number of configurations having such an energy density (the microcanonical entropy density s(ε) is strictly positive at ε = −β/2), and the Boltzmann measure is roughly equidistributed among such configurations. In the infinite-temperature limit β → 0 the Boltzmann measure becomes uniform, and one finds, as expected, u(β) → 0 (because nearly all configurations have an energy Ei /N close to 0) and s → log 2.

• In the low-temperature phase, i.e. for β > βc , the thermodynamic potentials are constant: u(β) = −ε∗ and s(β) = 0. The relevant configurations are the ones with the lowest energy density, namely those with Ei /N ≈ −ε∗ . The Boltzmann measure is dominated by a relatively small set of configurations, which is not exponentially large in N (the entropy density vanishes).


1 0.5 0 -0.5 -1 -1.5 0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

T

Fig. 5.3 Thermodynamics of the REM: the free-energy density (full line), the energy density (dashed line) and the entropy density (dotted line), are plotted versus the temperature √ T = 1/β. A phase transition takes place at Tc = 1/(2 log 2) ≈ 0.6005612.

Exercise 5.2 The REM was originally motivated by the attempt to provide a simple model of a spin glass. One can generalize it by introducing the effect of a magnetic field B. The 2N configurations are divided into N + 1 groups. Each by its ‘magnetiza„ group is labelled « N tion’ M ∈ {−N, −N + 2, . . . , N − 2, N }, and includes configurations. Their (N + M )/2 p energies {Ej } are independent Gaussian variables with variance N/2 as in eqn (5.1) and mean E Ej = −M B which depends upon the group that j belongs to. Show that there exists a phase transition line βc (B) in the plane β, B such that 1 EM = N

j

tanh [βB] when tanh [βc (B)B] when

β ≤ βc (B) , β > βc (B) ,

(5.16)

and plot the magnetic susceptibility (∂M/∂B)β = 0 versus T = 1/β.

Exercise 5.3 Consider a generalization of the the pdf of the energies, instead ˆ REM where ˜ of being Gaussian, is given by P (E) ∝ exp −C|E|δ , where δ > 0. Show that in order b (i.e. the to have extensive thermodynamic potentials, one must scale C as C = N 1−δ C b Compute the critical temperature thermodynamic limit N → ∞ should be taken at fixed C). and the ground state energy density. What is the qualitative difference between the cases δ > 1 and δ < 1?

½¼¼


5.3

The condensation phenomenon

In the low-temperature phase, a smaller-than-exponential set of configurations dominates the Boltzmann measure: we say that a condensation of the measure onto these configurations takes place. This is a scenario which is typical of the appearance of a glass phase, and we shall encounter it in several other problems, including, for instance, the satisfiability and colouring problems. It usually leads to many difficulties in finding the relevant configurations. In order to characterize the condensation, one can compute a participation ratio YN (β), defined in terms of the Boltzmann weights (5.2), by ⎤⎡ ⎤−2 ⎡ 2N µβ (j)2 = ⎣ e−2βEj ⎦ ⎣ e−βEj ⎦ . (5.17) YN (β) ≡ j=1

j

j

One can think of 1/YN (β) as giving some estimate of the ‘effective’ number of configurations which contribute to the measure. If the measure were equidistributed on r levels, one would have YN (β) = 1/r. The participation ratio can be expressed as YN (β) = ZN (2β)/ZN (β)2 , where ZN (β) is the partition function at inverse temperature β. The analysis in the pre. 2 vious section showed that ZN (β) = exp[N √ (log 2 + β /4)] with very small fluctuations . when β < βc , while ZN (β) = exp[N β log 2] when β > βc . This indicates that YN (β) is exponentially small in N for almost all samples in the high-temperature phase, i.e. for β < βc , in agreement with the fact that the measure is not condensed at high temperatures. In the low-temperature phase, in contrast, we shall see that YN (β) is finite and fluctuates from sample to sample. The computation of E Y (we drop the arguments N and β from now on) for the low-temperature phase is slightly involved. It requires us to control those energy levels Ei such that Ei /N ≈ −ε∗ . We give here only a sketch of the computation, and leave 2 the . ∞ details to the reader as an exercise.N Using the integral representation 1/Z = t exp(−tZ) dt, one gets (using M = 2 ) 0 EY = M E

∞

t exp [−2βE1 ] exp −t

0 ∞

=M

M

e

−βEi

dt

(5.18)

i=1

t a(t) [1 − b(t)]M −1 dt ,

(5.19)

0

where a(t) ≡ b(t) ≡

( ' exp −2βE − te−βE dP (E) ,

(5.20)

[1 − exp(−te−βE )] dP (E) ,

(5.21)

and P (E) is the Gaussian distribution (5.1). For large N , the leading contributions to E Y come from the regions where E is close to −N ε0 and log t is close to −N βε0 , where

A comment on quenched and annealed averages

ε0 = ε ∗ −

√ 1 log πN 2ε∗

½¼½

(5.22)

is fixed by the condition 2N P (−N ε0 ) = 1. This can be thought of as a refined estimate for the energy density of the lowest-energy configuration. We thus change variables from E, t to u, θ using E = −N ε0 +u and t = θ exp(−N βε0 ), and we study the regime where u and θ are finite as N → ∞. In this regime, the function P (E) can be replaced by 2−N eβc u . One gets 1 2N βε0 +∞ e2N βε0 βc /β−2 βc βc u−2βu−ze−βu e z , (5.23) a(t) du e = Γ 2− M Mβ β −∞ +∞ 1 1 βc b(t) z βc /β Γ − , (5.24) du eβc u [1 − exp(−ze−βu )] = − M −∞ Mβ β where Γ(x) is Euler’s gamma function. Note that the substitution P (E) 2−N eβc u is harmless because the resulting integrals (5.23) and (5.24) converge at large u. For large N , the expression [1 − b(t)]M −1 in eqn (5.19) can be approximated by −M b(t) , and one finally obtains e ∞ EY = M dt t a(t) e−M b(t) (5.25) 0 ∞ βc βc 1 1 Γ − z βc /β = 1 − βc /β , dz z βc /β−1 exp = Γ 2− β β β β 0 where we have used the approximate expressions (5.23) and (5.24), and the equalities are understood to hold up to corrections which vanish as N → ∞. We therefore obtain the following. Proposition 5.3 In the REM, when N → ∞, the expectation value of the participation ratio is 0 when T > Tc , (5.26) EY = 1 − T /Tc when T ≤ Tc . This gives a quantitative measure of the degree of condensation of the Boltzmann measure: when T decreases, condensation starts at the phase transition temperature Tc . At lower temperatures, the participation ratio Y increases, meaning that the measure becomes concentrated onto fewer and fewer configurations, until at T = 0 only one configuration contributes and Y = 1. With the participation ratio, we have a first qualitative and quantitative characterization of the low-temperature phase. Actually, the energies of the relevant configurations in this phase have many interesting probabilistic properties, to which we shall return in Chapter 8.

5.4

A comment on quenched and annealed averages

In the previous section, we found that the self-averaging property holds in the REM, and this result allowed us to discuss the thermodynamics of a generic sample.

½¼¾


Self-averaging of the thermodynamic potentials is a very frequent property, but in more complicated systems it is often difficult to compute their expectation. We discuss here an approximation which is frequently used in such cases, the ‘annealed average’. When the free-energy density is self-averaging, the value of fN is roughly the same for almost all samples and can be estimated by its expectation, called the quenched average, fN,q : T fN,q = E fN = − E log ZN . (5.27) N This is the quantity that we computed in eqn (5.15). In general, it is hard to compute the expectation of the logarithm of the partition function. A much easier task is to compute the annealed average, fN,a = −

T log(E ZN ) . N

(5.28)

Let us compute this quantity for the REM. Starting from the partition function (5.3), we find: 2N 2 e−βEi = 2N E e−βE = 2N eN β /4 , (5.29) E ZN = E i=1

yielding fN,a (β) = −β/4 − log 2/β. Let us compare this with the correct free-energy density found in eqn (5.15). Jensen’s inequality (eqn (1.6)) shows that the annealed free-energy density fa (β) is always smaller than or equal to the quenched value (remember that the logarithm is a concave function). In the REM, and a few other particularly simple problems, the annealed average gives the correct result in the high-temperature phase, i.e. for T > Tc , but fails to identify the phase transition, and predicts wrongly a free-energy density in the low-temperature phase which is the analytic prolongation of the free-energy density at T > Tc . In particular, it yields a negative entropy density sa (β) = log 2 − β 2 /4 for T < Tc (see Fig. 5.2). A negative entropy is impossible in a system with a finite configuration space, as can be seen from the definition of entropy. It thus signals a failure, and the reason is easily understood. For a given sample with a free-energy density f , the partition function behaves as ZN = exp(−βN fN ). Self-averaging means that fN has small sample-to-sample fluctuations. However, these fluctuations, exist and are amplified in the partition function because of the factor N in the exponent. This implies that the annealed average of the partition function can be dominated by some very rare samples (those with an anomalously low value of fN ). Consider, for instance, the lowtemperature limit. We already know that in almost all samples the configuration with the lowest energy density is found at Ei ≈ −N ε∗ . However, there exist exceptional samples where one configuration has a smaller energy Ei = −N ε where ε > ε∗ . These 2 . samples are exponentially rare (they occur with probability = 2N e−N ε ), they are irrelevant as far as the quenched average is concerned, but they dominate the annealed average. Let us add a short semantic note. The terms ‘quenched’ and ‘annealed’ originate from the thermal processing of materials, for instance in the context of the metallurgy

The random subcube model

½¼¿

of alloys: a quench corresponds to preparing a sample by bringing it suddenly from a high to low a temperature. During a quench, atoms do not have time to change position (apart from some small vibrations). A given sample is formed by atoms in some random positions. In contrast in an annealing process, one gradually cools down the alloy, and the various atoms will find favourable positions. In the REM, the energy levels Ei are quenched: for each given sample, they take certain fixed values (like the positions of atoms in a quenched alloy). In the annealed approximation, one treats the configurations i and the energies Ei on the same footing. One says that the variables Ei are thermalized (like the positions of atoms in an annealed alloy). In general, the annealed average can be used to find a lower bound on the free energy for any system with a finite configuration space. Useful results can be obtained, for instance, using the following two simple relations, valid for all temperatures T = 1/β and sizes N : dfN,q (T ) ≤ 0. (5.30) fN,q (T ) ≥ fN,a (T ) , dT The first relation one follows from Jensen’s inequality as mentioned above, and the second can be obtained from the positivity of the canonical entropy (see eqn (2.22)), after averaging over the quenched disorder. In particular, if one is interested in optimization problems (i.e. in the limit of vanishing temperature), the annealed average provides the following general bound. Proposition 5.4 The ground state energy density 1 uN (T = 0) ≡ E min E(x) N x∈X N

(5.31)

satisfies the bound uN (0) ≥ maxT ∈[0,∞] fN,a (T ) . Proof Consider the annealed free-energy density fN,a (T ) as a function of the temperature T = 1/β. For any given sample, the free energy is a concave function of T because of the general relation (2.23). It is easy to show that the same property holds for the annealed average. Let T∗ be the temperature at which fN,a (T ) achieves its maximum, ∗ ∗ be its maximum value. If T∗ = 0, then uN (0) = fN,q (0) ≥ fN,a . If T∗ > 0, and let fN,a using the two inequalities (5.30), one gets uN (0) = fN,q (0) ≥ fN,q (T∗ ) ≥ fa (T∗ ) .

(5.32)

√In the REM, this result immediately implies that u(0) ≥ maxβ [−β/4 − log 2/β] = − log 2, which is actually a tight bound.

5.5

The random subcube model

In the spirit of the REM, it is possible to construct a toy model for the set of solutions of a random constraint satisfaction problem. The random subcube model is defined by three parameters N, α, p. It has 2N configurations: the vertices x = (x1 , · · · , xN ) of the unit hypercube {0, 1}N . An instance of the model is defined by a subset S of the

½


hypercube, the ‘set of solutions’. Given an instance, the analogue of the Boltzmann measure is defined as the uniform distribution µ(x) over S. The solution space S is the union of M = 2(1−α)N random subcubes which are i.i.d. subsets. Each subcube Cr , r ∈ {1, . . . , M }, is generated through the following procedure: 1. Generate the vector t(r) = (t1 (r), t2 (r), . . . , tN (r)), with independent entries ⎧ ⎨ 0 with probability (1 − p)/2 , (5.33) ti (r) = 1 with probability (1 − p)/2 , ⎩ ∗ with probability p. 2. Given the values of {ti (r)}, Cr is a subcube constructed as follows. For all i’s such that ti (r) is 0 or 1, fix xi = ti (r). Such variables are said to be ‘frozen’ for the subcube Cr . For all other i’s, xi can be either 0 or 1. These variables are said to be ‘free’. A configuration x may belong to several subcubes. Whenever it belongs to at least one subcube, it is in S. To summarize, α < 1 fixes the number of subcubes, and p ∈ [0, 1] fixes their size. The model can be studied using exactly the same methods as for the REM. Here we shall just describe the main results, omitting all proofs. It is a good exercise to work out the details and prove the various assertions. Let us denote by σr the entropy density of the r-th cluster in bits: σr = (1/N ) log2 |Cr |. It is clear that σr coincides with the fraction of ∗’s in the vector t(r). In the large-N limit, the number of clusters N (σ) with an entropy density σ obeys a large-deviation principle: . (5.34) N (σ) = 2N Σ(σ) . The function Σ(σ) is given as follows. Let D(σ||p) denote the Kullback–Leibler divergence between a Bernoulli distribution with mean σ and a Bernoulli distribution with mean p. As we saw in Section 1.2, this is given by D(σ||p) = σ log2

σ 1−σ + (1 − σ) log2 . p 1−p

We define [σ1 (p, α), σ2 (p, α)] as the interval in which D(σ||p) ≤ 1 − α. Then 1 − α − D(σ||p) when σ ∈ [σ1 (p, α), σ2 (p, α)] , Σ(σ) = −∞ otherwise.

(5.35)

(5.36)

We can now derive the phase diagram (see Fig. 5.4). We denote by s the total entropy density of the solution space, i.e. s = (1/N ) log2 |S|. Consider a configuration x. The expected number of clusters to which it belongs is 2N (1−α) ((1 + p)/2)N . Therefore, if α < αd ≡ log2 (1 + p), the solution space contains all but a vanishing fraction of the configurations, with high probability: s = log 2. On the other hand, if α > αd , the probability that a configuration in S belongs to at least two distinct clusters is very small. In this regime, s = maxσ (Σ(σ)+σ). As in the REM, there are two cases. (i) The maximum of Σ(σ) + σ is achieved for σ = σ∗ (p, α) ∈]s1 (p, α), s2 (p, α)[. This happens

Notes

½

0.25 1 0.2

slope = −1

s

0.8

0.15

Σ(σ)

0.6

0.1

0.4

0.05

Σmax

0

σ1

0.2

σ∗ σ2

Σ∗

-0.05

0 0.3

0.4

0.5

0.6

σ

0.7

0.8

0.9

0.5

0.6

0.7

0.8

0.9

1

α

Fig. 5.4 Left: the function Σ(σ) of the random subcube model, for p = 0.6 and α = 0.8 ∈]αd , αc [. The maximum of the curve gives the total number of clusters Σmax . A ‘typical’ random solution x ∈ S belongs to one of the eN Σ(σ∗ ) clusters with entropy density σ∗ , with Σ (σ∗ ) = −1. As α increases above αc , random solutions condense into a few clusters with entropy density s2 . Right: thermodynamic quantities plotted versus α for p = 0.6: the total entropy s, the total number of clusters Σmax , and the number of clusters where typical configurations are found, Σ∗ .

when α < αc (p) ≡ log2 (1+p)+(1−p)/(1+p). In this case s = (1−α) log 2+log(1+p). (ii) The maximum of Σ(σ) + σ is obtained for σ = σ2 (p, α). In this case s = σ2 (p, α). Altogether, we have found three phases: • For α < αd , subcubes overlap and one big cluster contains most of the configurations: stot = 1. • For αd < α < αc , the solution space S is split into 2N (1−α) non-overlapping clusters of configurations (every subcube is a cluster of solutions, separated from the others). Most configurations of S are in the eN Σ(s∗ ) clusters which have an entropy density close to s∗ (p, α). Note that the majority of clusters have entropy density 1 − p < s∗ . There is a tension between the number of clusters and their size (i.e. their internal entropy). The result is that the less numerous, but larger, clusters with entropy density s∗ dominate the uniform measure. • For α > αc , the solution space S is still partitioned into 2N (1−α) non-overlapping clusters of configurations. However, most solutions are in clusters with entropy density close to s2 (p, α). The number of such clusters is not exponentially large. In fact, the uniform measure over S shows a condensation phenomenon, which is completely analogous to that in the REM. One can define a participation ratio 2 µ(r) , where µ(r) is the probability that a configuration of S chosen Y = r uniformly at random belongs to cluster r; µ(r) = eN σr / r eN σr . This participation ratio is finite, and equal to 1 − m, where m is the slope m = −(dΣ/dσ), evaluated at s2 (p, α).

Notes The REM was invented by Derrida (1980), as an extreme case of family of spin glass models. Here we have followed his original analysis, which makes use of the microcanon-

½


ical entropy. More detailed computations can be found in Derrida (1981), including the solution to Exercise 5.2. The condensation formula (5.3) appeared first in Gross and Mézard (1984) as an application of replica computations which we shall discuss in Chapter 8. The direct estimate of the participation ratio presented here and the analysis of its fluctuations were developed by Mézard et al. (1985a) and Derrida and Toulouse (1985). We shall return to the properties of the fascinating condensed phase in Chapter 8. Exercise 5.3 shows a phase transition which goes from second-order when δ > 1, to first-order when δ < 1. Its solution can be found in Bouchaud and Mézard (1997). The random subcube model was introduced by Achlioptas (2007) and studied in detail by Mora and Zdeborov´ a (2007). We refer to that paper for the derivations omitted from Section 5.5. As a final remark, note that in most of the physics literature, authors do not explicitely write down all of the mathematical steps leading, for instance, to eqn (5.13), preferring a more synthetic presentation which focuses on the basic ideas. In more complicated problems, it may be very difficult to fill in the corresponding mathematical gaps. In many of the models studied in this book, this is still beyond the range of rigorous techniques. The recent book by Talagrand (2003) adopts a fully rigorous point of view, and it starts with a presentation of the REM which nicely complements the one given here and in Chapter 8.

6 The random code ensemble As already explained in Section 1.6, one of the basic problems of information theory consists in communicating reliably over a noisy communication channel. Errorcorrecting codes achieve this task by systematically introducing some form of redundancy into the message to be transmitted. One of the major breakthroughs accomplished by Claude Shannon was to understand the importance of code ensembles. He realized that it is much easier to construct ensembles of codes which have good properties with high probability than to exhibit explicit examples that achieve the same performance. In a nutshell, ‘stochastic’ design is much easier than ‘deterministic’ design. At the same time, he defined and analysed the simplest such ensembles, which has been named thereafter the random code ensemble (or, sometimes, the Shannon ensemble). Despite its great simplicity, the random code ensemble (RCE) has very interesting properties and, in particular, it achieves the optimal error-correcting performance. It provides therefore a proof of the ‘direct’ part of the channel coding theorem: it is possible to communicate with vanishing error probability as long as the communication rate is smaller than the channel capacity. Furthermore, it is the prototype of a code based on a random construction. In the following chapters we shall explore several examples of this approach, and the random code ensemble will serve as a reference. We introduce the idea of code ensembles and define the RCE in Section 6.1. Some properties of this ensemble are described in Section 6.2, and its performance over the BSC is estimated in Section 6.3. We generalize these results to a general discrete memoryless channel in Section 6.4. Finally, in Section 6.5, we show that the RCE is optimal by a simple sphere-packing argument.

6.1

Code ensembles

An error-correcting code is defined as a pair of an encoding and a decoding map. The encoding map is applied to the information sequence to get an encoded message which is transmitted through the channel. The decoding map is applied to the (noisy) channel output. For the sake of simplicity, we shall assume throughout this chapter that the message to be encoded is given as a sequence of M bits and that encoding produces a redundant sequence of N > M bits. The possible codewords (i.e. the 2M points in the space {0, 1}N which are all possible outputs of the encoding map) form the codebook CN . We denote by Y the output alphabet of the communication channel. We use the notation

½

The random code ensemble

x : {0, 1}M → {0, 1}N Y N → {0, 1}N xd :

encoding map , decoding map .

(6.1) (6.2)

Notice that the definition of the decoding map here is slightly different from the one given in Section 1.6. Here we consider only the difficult part of the decoding procedure, namely how to reconstruct from the received message the codeword which was sent. To complete the decoding as defined in Section 1.6, one needs to get back the original message by knowing the codeword, but this is assumed to be an easier task (encoding is assumed to be injective). The customary recipe for designing a code ensemble is the following. (i) Define a subset of the space of encoding maps (6.1). (ii) Endow this set with a probability distribution. (iii) Finally, for each encoding map in the ensemble, define the associated decoding map. In practice, this last step is accomplished by declaring that one of a few general ‘decoding strategies’ is adopted. We shall introduce two such strategies below. M Our first example is the random code ensemble. Note that there exist 2N 2 possible encoding maps of the type (6.1): one must specify N bits for each of the 2M codewords. In the RCE, any of these encoding maps is picked with uniform probability. The code is therefore constructed as follows. For each of the possible information messages (m) (m) (m) m ∈ {0, 1}M , we obtain the corresponding codeword x(m) = (x1 , x2 , . . . , xN ) by tossing an unbiased coin N times: the i-th outcome is assigned to the i-th coordinate (m) xi .

Exercise 6.1 Note that, with this definition, the code is not necessarily injective: there could be two information messages m1 = m2 with the same codeword: x(m1 ) = x(m2 ) . This is an annoying property for an error-correcting code: any time that we send either of the messages m1 or m2 , the receiver will not be able to distinguish between them, even in the absence of noise. Happily enough, though, these unfortunate coincidences occur rarely, i.e. their number is much smaller than the total number of codewords 2M . What is the expected number of pairs m1 , m2 such that x(m1 ) = x(m2 ) ? What is the probability that all the codewords are distinct?

Let us now turn to the definition of the decoding map. We shall introduce two of the most important decoding schemes here: word MAP (MAP stands for ‘maximum a posteriori probability’ here) and symbol MAP decoding. Both schemes can be applied to any code. In order to define them, it is useful to introduce the probability distribution P (x|y) for x to be the channel input conditional on the received message y. For a memoryless channel with a transition probability Q(y|x), this probability has an explicit expression as a consequence of the Bayes rule: 1 Q(yi |xi ) P(x) . Z(y) i=1 N

P(x|y) =

(6.3)

Here Z(y) is fixed by the normalization condition x P(x|y) = 1, and P(x) is the a priori probability for x to be the transmitted message. Throughout this book, we shall

Code ensembles

½

assume that the transmitter chooses the codeword to be transmitted with uniform probability. Therefore P(x) = 1/2M if x ∈ CN and P(x) = 0 otherwise. This can also be written formally as 1 I(x ∈ CN ) . (6.4) P(x) = |CN | Indeed, it is not hard to realize that Z(y) is the probability of observing y when a random codeword is transmitted. Hereafter, we shall use µy ( · ) to denote the a posteriori distribution (6.3) and µ0 ( · ) for the a priori one (6.4). We can thus rewrite eqn (6.3) as 1 Q(yi |xi ) µ0 (x) . Z(y) i=1 N

µy (x) =

(6.5)

(i)

It is also useful to define the marginal distribution µy (xi ) = P(xi |y) of the i-th bit of the transmitted message conditional on the output message. This is obtained from the distribution (6.5) by marginalizing over all the bits xj with j = i: µy (x) , (6.6) µ(i) y (xi ) = x\i

where we have introduced the shorthand x\i ≡ {xj : j = i}. Word MAP decoding outputs the most probable transmitted codeword, i.e. it maximizes the distribution (6.5): xw (y) = arg max µy (x) . x

(6.7)

We do not specify what to do in the case of a tie (i.e. if the maximum is degenerate), since this is irrelevant for all the coding problems that we shall consider. Scrupulous readers can chose their own convention in such cases. A strongly related decoding strategy is maximum-likelihood decoding. In this case, one maximizes Q(y|x) over x ∈ CN . This coincides with word MAP decoding whenever the a priori distribution over the transmitted codeword P(x) = µ0 (x) is taken to be uniform as in eqn (6.4). From now on, we shall therefore blur the distinction between these two strategies. Symbol (or bit) MAP decoding outputs the sequence of the most probable transmitted bits, i.e. it maximizes the marginal distribution (6.6): (N ) (x ) , . . . , arg max µ (x ) . (6.8) xb (y) = arg max µ(1) 1 N y y x1

xN

Exercise 6.2 Consider a code of block length N = 3 and codebook size |C| = 4, with codewords x(1) = 001, x(1) = 101, x(1) = 110, x(1) = 111. What is the code rate? This code is used to communicate over a binary symmetric channel with flip probability p < 0.5. Suppose that the channel output is y = 000. Show that the word MAP decoding outputs the codeword 001. Now apply symbol MAP decoding to decode the first bit x1 . Show that the result coincides with that of word MAP decoding only when p is small enough.

½½¼


It is important to note that each of the above decoding schemes is optimal with respect a different criterion. Word MAP decoding minimizes the average block error probability PB defined in Section 1.6.2. This is the probability, with respect to the channel distribution Q(y|x), that the decoded codeword xd (y) is different from the transmitted one, averaged over the transmitted codeword: PB ≡

1 P[xd (y) = x] . |C|

(6.9)

x∈C

Bit MAP decoding minimizes the bit error probability, or bit error rate (BER), Pb . This is the fraction of incorrect bits, averaged over the transmitted codeword: N 1 1 Pb ≡ P[xdi (y) = xi ] . |C| N i=1

(6.10)

x∈C

Exercise 6.3 Show that word MAP and symbol MAP decoding are indeed optimal with respect to the above criteria.

6.2

The geometry of the random code ensemble

We begin our study of the RCE by first working out some of its geometrical properties. A code from this ensemble is defined by a codebook, i.e. a set CN of 2M points (the codewords) in the Hamming space {0, 1}N . Each of these points is drawn with uniform probability from the Hamming space. The simplest question one may ask about CN is the following. Suppose you sit on one of the codewords and look around you. How many other codewords are there at a given distance? We shall use here the Hamming distance: the distance between two points x, y ∈ {0, 1}N is the number of coordinates in which they differ. This question is addressed through the distance enumerator Nx(0) (d) with respect to a codeword x(0) ∈ CN . This is defined as the number of codewords in x ∈ CN whose Hamming distance from x(0) is equal to d: d(x, x(0) ) = d. We shall now compute the typical properties of the distance enumerator for a random code. The simplest quantity to look at is the average distance enumerator E Nx(0) (d), the average being taken over the code ensemble. In general one should also specify which one of the codewords is x(0) . Since, in the, RCE all codewords are drawn independently, each one with uniform probability over the Hamming space, such a specification is irrelevant and we can in fact fix x(0) to be the all-zeros codeword, x(0) = 000 · · · 00. Therefore we are asking the following question: take 2M − 1 points at random with uniform probability in the Hamming space {0, 1}N ; what is the average number of points at distance d form the corner 00 · · · 0? This is simply the number of points (2M − 1), times the fraction of the Hamming space ‘volume’ at a distance d from 000 · · · 0 (2−N Nd ):

The geometry of the random code ensemble

½½½

0.8 R − 1 + H2 (δ)

0.6 0.4 0.2 0 -0.2 -0.4 -0.6 0

δGV ? 0.1 0.2

0.3

0.4

0.5 δ

0.6

0.7

0.8

0.9

1

Fig. 6.1 Growth rate of the distance enumerator for the random code ensemble, with rate R = 1/2, as a function of the Hamming distance d = N δ.

E Nx(0) (d) = (2

M

−N

− 1) 2

N d

. =

2N [R−1+H2 (δ)] .

(6.11)

In the second expression above, we have introduced the fractional distance δ ≡ d/N and the rate R ≡ M/N , and considered the N → ∞ asymptotics with these two quantities kept fixed. We plot the function R−1+H2 (δ) (which is sometimes called the growth rate of the distance enumerator) in Fig. 6.1. For δ small enough, i.e. δ < δGV , the growth rate is negative: the average number of codewords at a small distance from x(0) vanishes exponentially with N . By the Markov inequality, the probability of having any codeword at all at such a short distance vanishes as N → ∞. The distance δGV (R), called the Gilbert–Varshamov distance, is the smallest root of R − 1 + H2 (δ) = 0. For instance, we have δGV (1/2) ≈ 0.110278644. Above the Gilbert–Varshamov distance, i.e. δ > δGV , the average number of code. words is exponentially large, with the maximum occurring at δ = 1/2: E Nx(0) (N/2) = NR M 2 = 2 . It is easy to show that the distance enumerator Nx(0) (d) is sharply concentrated around its average in the whole regime δGV < δ < 1 − δGV . This can be done using arguments similar to those developed in Section 5.2 for the random energy model (REM configurations become codewords in the present context, the role of the energy is played by the Hamming distance, and the Gaussian distribution of the energy levels is replaced here by the binomial distribution). A pictorial interpretation of the above result is shown in Fig. 6.2 (but note that it is often misleading to interpret phenomena

½½¾


codewords

x(0) δ GV

Fig. 6.2 A pictorial view of a typical code from the random code ensemble. The codewords are random points in the Hamming space. If we pick a codeword at random from the code and consider a ball of radius N δ around it, the ball will not contain any other codeword as long as δ < δGV (R), it will contain exponentially many codewords when δ > δGV (R).

occurring in spaces with a large number of dimensions using finite-dimensional images: such images must be handled with care!).

Exercise 6.4 The random code ensemble can easily be generalized to other (nonbinary) alphabets. Consider, for instance, a q-ary alphabet, i.e. an alphabet with letters {0, 1, 2, . . . , q − 1} ≡ A. A code CN is constructed by taking 2M codewords with uniform probability in AN . We can define the distance between any two codewords dq (x, y) as the number of positions in which the sequences x, y differ. Show that the average distance enumerator is now . E Nx(0) (d) = 2N [R−log2 q+δ log2 (q−1)+H2 (δ)] ,

(6.12)

where δ ≡ d/N and R ≡ M/N . The maximum of the above function is no longer at δ = 1/2. How can we explain this phenomenon in simple terms?

6.3

Communicating over a binary symmetric channel

We shall now analyse the performance of the RCE when used for communicating over a binary symmetric channel (BSC) as defined in Fig. 1.4. We start by considering a word MAP (or, equivalently, maximum-likelihood) decoder, and then analyse the slightly more complicated symbol MAP decoder. Finally, we introduce another decoding strategy, inspired by the statistical-physics analogy, that generalizes word MAP and symbol MAP decoding.


codewords

x (0)

codewords

x (0)

y

½½¿

y x (1)

Fig. 6.3 A pictorial view of word MAP decoding for a BSC. A codeword x(0) is chosen and transmitted through a noisy channel. The channel output is y. If the distance between x(0) and y is small enough (left frame), the transmitted message can be safely reconstructed by looking for the codeword closest to y. In the opposite case (right frame), the closest codeword x1 does not coincide with the transmitted one.

6.3.1

Word MAP decoding

For a BSC, both the channel input x and the output y are sequences of bits of length N . The probability for the codeword x to be the channel input conditional on the output y, defined in eqns (6.5) and (6.4), depends uniquely on the Hamming distance d(x, y) between these two vectors. Denoting by p the channel flip probability, we have µy (x) =

1 pd(x,y) (1 − p)N −d(x,y) I(x ∈ CN ) , Z (y)

(6.13)

where Z (y) is a normalization constant which depends uniquely upon y (up to a factor, this coincides with the normalization Z(y) in eqn (6.5)). Without loss of generality, we can assume that p < 1/2. Therefore word MAP decoding, which prescribes that µy (x) is maximized with respect to x, outputs the codeword which is the closest to the channel output. We have obtained a purely geometrical formulation of the original communication problem. A random set of points CN is drawn in the Hamming space {0, 1}N , and one of them (let us call it x(0) ) is chosen for communicating. Noise perturbs this vector, yielding a new point y. Decoding consists in finding the closest point to y among all the points in CN , and fails every time this is not x(0) . The block error probability is simply the probability for such an event to occur. This formulation is illustrated in Fig. 6.3. This description should immediately make it clear that the block error probability vanishes, (in the N → ∞ limit) as soon as p is below some finite threshold. In the previous Section we saw that, with high probability, the closest codeword x ∈ CN \x(0) to x(0) lies at a distance d(x , x(0) ) N δGV (R). On the other hand, y is obtained from x(0) by flipping each bit independently with probability p, and therefore d(y, x(0) ) N p with high probability. By the triangle inequality, x(0) is certainly the closest codeword

½½


to y (and therefore word MAP decoding is successful) if d(x(0) , y) < d(x(0) , x )/2. If p < δGV (R)/2, this happens with probability approaching one as N → ∞, and therefore the block error probability vanishes. However, the above argument overestimates the effect of noise. Although about N δGV (R)/2 incorrect bits may cause an unsuccessful decoding, they must occur in the appropriate positions for y to be closer to x than to x(0) . If they occur at uniformly random positions (as happens in a BSC), they will probably be harmless. The difference between the two situations is most significant in large-dimensional spaces, as shown by the analysis provided below. The distance between x(0) and y is a sum of N i.i.d. Bernoulli variables with parameter p (each bit is flipped with probability p). By the central limit theorem, N (p−ε) < d(x(0) , y) < N (p+ε) with probability approaching one in the N → ∞ limit, for any ε > 0. As for the remaining 2M −1 codewords, they are completely uncorrelated M with x(0) and, therefore, with y: {y, x(1) , · · · , x(2 −1) } are 2M i.i.d. random points drawn from the uniform distribution over {0, 1}N . The analysis of the previous section shows that, with probability approaching one as N → ∞, none of the codewords M {x(1) , · · · , x(2 −1) } lies within a ball of radius N δ centred on y when δ < δGV (R). In the opposite case, if δ > δGV (R), there is an exponential (in N ) number of these codewords within a ball of radius N δ. The performance of the RCE is easily deduced (see Fig. 6.4). If p < δGV (R), the transmitted codeword x(0) lies at a shorter distance than all the other ones from the received message y: decoding is successful. At a larger noise level, if p > δGV (R), there is an exponential number of codewords closer to y than the transmitted one is: decoding is unsuccessful. Note that the condition p < δGV (R) can be rewritten as R < CBSC (p), where CBSC (p) = 1−H2 (p) is the capacity of a BSC with flip probability p. 6.3.2

Symbol MAP decoding

In symbol MAP decoding, the i-th bit is decoded by first computing the marginal P (i) (xi |y) and then maximizing it with respect to xi . Using eqn (6.13), we get µ(i) y (xi ) =

µy (x) =

x\i

1 exp{−2B d(x, y)} , Z x

where we have introduced the parameter B≡ and the normalization constant Z≡

(6.14)

\i

1 log 2

1−p p

(6.15)

exp{−2B d(x, y)} .

(6.16)

x∈CN (i)

Equation (6.14) shows that the marginal distribution µy (xi ) sums contributions from all the codewords, not only the one closest to y. This makes the analysis of symbol MAP decoding slightly more involved than the word MAP decoding case.

Communicating over a binary symmetric channel 0.5

0.5

0.4

0.4

R − 1 + H(δ)

0.3

0.3

0.2

0.2

0.1

0.1

0 ♦ p -0.1

½½

0

δGV

-0.2 ? ? 0 0.2

R − 1 + H(δ)

δGV

♦p

-0.1 -0.2 0.4

0.6

0.8

1

δ

0

? 0.2

? 0.4

0.6

0.8

1

δ

by (d) (which counts the number of codeFig. 6.4 Logarithm of the distance enumerator N words at a distance d = N δ from the received message) divided by the block length N . Here, the rate is R = 1/2. We also show the distance of the transmitted codeword for two different noise levels: p = 0.03 < δGV (1/2) ≈ 0.110278644 (left) and p = 0.3 > δGV (R) (right). The tangent lines with slope 2B = log[(1 − p)/p] determine which codewords dominate the symbol MAP decoder.

Let us start by estimating the normalization constant Z. It is convenient to separate the contribution arising from the transmitted codeword x(0) from that arising from M the incorrect codewords x(1) , . . . , x(2 −1) :

Z = e−2Bd(x

(0)

,y)

+

N

y (d) e−2Bd ≡ Zcorr + Zerr , N

(6.17)

d=0

y (d) the number of incorrect codewords at a distance where we have denoted by N d from the vector y. The contribution from x(0) in the above expression is easily estimated. By the law of large numbers, d(x(0) , y) N p and therefore Zcorr is close to e−2N Bp with high probability. More precisely, for any ε > 0, e−N (2Bp+ε) ≤ Zcorr ≤ e−N (2Bp−ε) with probability approaching one in the N → ∞ limit.

y (d), For Zerr , we proceed in two steps: we first compute the distance enumerator N and then sum over d. The distance enumerator was computed in Section 6.2. As in the analysis of word MAP decoding, the fact that the distances are measured with respect to the channel output y and not with respect to a codeword does not change the result, M

y (d) because y is independent of the incorrect codewords x(1) , · · · , x(2 −1) . Therefore N is exponentially large in the interval δGV (R) < δ ≡ d/N < 1−δGV (R), while it vanishes with high probability outside the same interval. Moreover, if δGV (R) < δ < 1−δGV (R),

y (d) is tightly concentrated around its mean, given by eqn (6.11). The summation N over d in eqn (6.17) can then be evaluated by the saddle point method. This calculation is very similar to the estimation of the free energy of the random energy model (see Section 5.2). Roughly speaking, we have

½½


Zerr =

N

y (d) e−2Bd N N

1−δGV

. eN [(R−1) log 2+H(δ)2Bδ] dδ = eN φerr ,

(6.18)

δGV

d=0

where φerr ≡

max

[ (R − 1) log 2 + H(δ) − 2Bδ ] .

(6.19)

δ∈[δGV ,1−δGV ]

The reader can complete the mathematical details of the above derivation as outlined in Section 5.2. The bottom line is that Zerr is close to eN φerr with high probability as N → ∞. Let us examine the resulting expression given in eqn (6.19) (see Fig. 6.4). If the maximum is achieved in the interior of [δGV , 1 − δGV ], its location δ∗ is determined by the stationarity condition H (δ∗ ) = 2B, which implies δ∗ = p. In the opposite case, the maximum must be realized at δ∗ = δGV (remember that B > 0). Evaluating the right hand side of eqn (6.19) in these two cases, we get −δGV (R) log ((1 − p)/p) if p < δGV , (6.20) φerr = (R − 1) log 2 − log(1 − p) otherwise. We can now compare Zcorr and Zerr . At low noise levels (small p), the transmitted codeword x(0) is close enough to the received message y to dominate the sum in eqn (6.17). At higher noise levels, the exponentially more numerous incorrect codewords overcome the term due to x(0) . More precisely, with high probability we have Zcorr [1 + e−Θ(N ) ] if p < δGV , (6.21) Z= Zerr [1 + e−Θ(N ) ] otherwise, where the exponents Θ(N ) are understood to be positive. We now consider eqn (6.14), and once again separate out the contribution of the transmitted codeword, P (i) (xi |y) =

1 (0) [Zcorr I(xi = xi ) + Zerr,xi ] , Z

where we have introduced the quantity Zerr,xi = e−2Bd(z,y) I(zi = xi ) .

(6.22)

(6.23)

z∈CN \x(0)

Note that Zerr,xi ≤ Zerr . Together with eqn (6.21), this implies, if p < δGV (R), that (i) (0) (i) (0) µy (xi = xi ) = 1 − e−Θ(N ) and µy (xi = xi ) = e−Θ(N ) . Therefore, in this regime, (0) the symbol MAP decoder correctly outputs the transmitted bit xi . It is important to stress that this result holds with probability approaching one as N → ∞. Concretely, there exist bad choices of the code CN and particularly unfavourable channel realiza(i) (0) tions y such that µy (xi = xi ) < 1/2 and the decoder fails. However, the probability of such an event (i.e. the bit-error rate Pb ) vanishes in the large-block length limit N → ∞.


½½

What happens for p > δGV (R)? Arguing as for the normalization constant Z, it is easy to show that the contribution of incorrect codewords dominates the marginal distribution (6.22). Intuitively, this suggests that the decoder fails. A more detailed computation, sketched below, shows that the bit error rate in the N → ∞ limit is 0 if p < δGV (R), (6.24) Pb = p if δGV (R) < p < 1/2. Note that, above the threshold δGV (R), the bit error rate is the same as if the information message were transmitted without coding through the BSC: the code is useless. A complete calculation of the bit error rate Pb in the regime p > δGV (R) is rather lengthy. We shall provide here a heuristic, albeit essentially correct, justification, and leave a more detailed derivation to the exercise below. As already stressed, the contribution Zcorr of the transmitted codeword can be safely neglected in eqn (6.22). (0) Assume, without loss of generality, that xi = 0. The decoder will be successful if Zerr,0 > Zerr,1 and will fail in the opposite case. Two cases must be considered: either yi = 0 (this happens with probability 1 − p) or yi = 1 (probability p). In the first case, we have Zerr,0 = I(zi = 0) e−2Bdi (y,z) , z∈CN \x(0)

Zerr,1 = e−2B

I(zi = 1) e−2Bdi (y,z) ,

(6.25)

z∈CN \x(0)

where we have denoted by di (x, y) the number of of positions j, distinct from i, such that xj = yj . The sums in the above expressions are independent identically distributed random variables. Moreover, they are tightly concentrated around their mean. Since B > 0, this implies Zerr,0 > Zerr,1 with high probability. Therefore the decoder is successful in the case yi = 0. Analogously, the decoder fails with high probability if yi = 1, and hence the bit error rate converges to Pb = p for p > δGV (R).

Exercise 6.5 From a rigorous point of view, the weak point of the above argument is the lack of any estimate of the fluctuations of Zerr,0/1 . The reader may complete the derivation along the following lines: (a) Define X0 ≡ Zerr,0 and X1 ≡ e2B Zerr,1 . Prove that X0 and X1 are independent and identically distributed. (b) Define the P correct distance enumerators N0/1 (d) such that a representation of the form X0/1 = d N0/1 (d) exp(−2Bd) holds. (c) Show that a significant fluctuation of N0/1 (d) from its average is highly (more than exponentially) improbable (within an appropriate range of d). (d) Deduce that a significant fluctuation of X0/1 is highly improbable (the last two points can be treated along the lines already discussed for the random energy model in Chapter 5).

½½

6.3.3


Finite-temperature decoding (i)

The expression (6.14) for the marginal µy (xi ) is strongly reminiscent of a Boltzmann average. This analogy suggests a generalization which interpolates between the two ‘classical’ MAP decoding strategies discussed so far: finite-temperature decoding. We first define this new decoding strategy in the context of the BSC. Let β be a nonnegative number playing the role of an inverse temperature, and let y ∈ {0, 1}N be the channel output. We define the probability distribution µy,β (x) to be given by µy,β (x) =

1 e−2βBd(y,x) I(x ∈ CN ) , Z(β)

Z(β) ≡

e−2βBd(x,y) ,

(6.26)

x∈CN

where B is always related to the noise level p through eqn (6.15). This distribution depends upon the channel output y: for each received message y, the finite-temperature decoder constructs the appropriate distribution µy,β (x). For the sake of simplicity, we (i) shall not write this dependence explicitly. Let µy,β (xi ) be the marginal distribution of xi when x is distributed according to µy,β (x). The new decoder outputs (1) (N ) β x = arg max µy,β (x1 ) , . . . , arg max µy,β (xN ) . (6.27) x1

xN

As in the previous sections, readers are free to choose their favourite convention in the (i) (i) case of ties (i.e. for those i’s such that µy,β (0) = µy,β (1)). Two values of β are particularly interesting: β = 1 and β = ∞. If β = 1, the distribution µy,β (x) coincides with the distribution µy (x) of the channel input conditional on the output (see eqn (6.13)). Therefore, for any y, symbol MAP decoding coincides = xb . with finite-temperature decoding at β = 1: xβ=1 i If β = ∞, the distribution (6.26) is concentrated on those codewords which are the closest to y. In particular, if there is a unique closest codeword to y, finite-temperature decoding at β = ∞ coincides with word MAP decoding: xβ=∞ = xw .

Exercise 6.6 Using the approach developed in the previous section, analyse the performances of finite-temperature decoding for the RCE at any β.

The results of the above exercise are summarized in Fig. 6.5, which gives the finite-temperature decoding phase diagram. There exist three regimes, which are three distinct phases with very different behaviours. 1. A ‘completely ordered’ phase at low noise (p < δGV (R)) and low temperature (large enough β). In this regime, the decoder works: the probability distribution µy,β (x) is dominated by the transmitted codeword x(0) . More precisely, µy,β (x(0) ) = 1 − exp{−Θ(N )}. The bit and block error rates vanish as N → ∞. 2. A ‘glassy’ phase at higher noise (p > δGV (R)) and low temperature (large enough β). The transmitted codeword has a negligible weight µy,β (x(0) ) = exp{−Θ(N )}. The bit error rate is bounded away from 0, and the block error rate converges


½½

2.5 2 paramagnetic

1.5 1/β

1 0.5

ordered

glassy

0 0

0.1

0.2

0.3

0.4

0.5

p Fig. 6.5 Phase diagram for the rate-1/2 random code ensemble over a binary symmetric channel, using finite temperature decoding. Word MAP and bit MAP decoding correspond to 1/β = 0 and 1/β = 1, respectively. Note that the phase boundary of the error-free (ordered) phase is vertical in this interval of temperature.

to 1 as N → ∞. The measure µy,β (x) is dominated by the codewords closest to the received message y (which are distinct from the correct codeword, since p > δGV (R)). Its Shannon entropy H(µy,β ) is sublinear in N . This situation is closely related to the ‘measure condensation’ phenomenon that occurs in the low-temperature phase of the random energy model. 3. An ‘entropy-dominated’ (paramagnetic) phase at high temperature (small enough β). The bit and block error rates behave as in the glassy phase, and µy,β (x(0) ) = exp{−Θ(N )}. However, the measure µy,β (x) is now dominated by codewords whose distance d N δ∗ from the received message is larger than the minimal distance: δ∗ = pβ /[pβ + (1 − p)β ]. In particular, δ∗ = p if β = 1, and δ∗ = 1/2 if β = 0. In the first case we recover the result already obtained for symbol MAP decoding. In the second case, µy,β=0 (x) is the uniform distribution over the codewords, and the distance from the received message under this distribution is, with high probability, close to N/2. In this regime, the Shannon entropy H(µβ ) is linear in N . Finite-temperature decoding can be generalized to other channel models. Let µy (x) be the distribution of the transmitted message conditional on the channel output, given explicitly in eqn (6.5). For β > 0, we define the distribution1

1 The partition function Z(β) defined here differs by a multiplicative constant from the one defined in eqn (6.26) for a BSC.

½¾¼


µy,β (x) =

1 µy (x)β , Z(β)

Z(β) ≡

µy (x)β .

(6.28)

x

Once more, the decision of the decoder for the i-th bit is taken according to the rule (6.27). The distribution µy,β (x) is a ‘deformation’ of the conditional distribution µy (x). At large β, more weight is given to highly probable transmitted messages. At small β, the most numerous codewords dominate the sum. A little thought shows that, as for a BSC, the cases β = 1 and β = ∞ correspond, respectively, to symbol MAP and word MAP decoding. The qualitative features of the finite-temperature-decoding phase diagram are easily generalized to any memoryless channel. In particular, the three phases described above can be found in such a general context. Decoding is successful in the low-noise, large-β phase.

6.4

Error-free communication with random codes

As we have seen, the block error rate PB for communicating over a BSC with a random code and word MAP decoding vanishes in the large-block-length limit as long as R < CBSC (p), where CBSC (p) = 1 − H2 (p) is the channel capacity. This establishes the ‘direct’ part of Shannon’s channel coding theorem for the BSC case: error-free communication is possible at rates below the channel capacity. This result is in fact much more general. We describe here a proof for general memoryless channels, always based on random codes. For the sake of simplicity we shall restrict ourselves to memoryless channels with binary input and discrete output. Such are defined by a transition probability Q(y|x), where x ∈ {0, 1} and y ∈ Y, with Y being a finite alphabet. In order to handle this case, we must generalize the RCE: each codeword x(m) ∈ {0, 1}N , m = 0, . . . , 2M −1, is (m) (m) again constructed independently as a sequence of N i.i.d. bits x1 · · · xN . Unlike the (m) case of symmetric channels, xi is now drawn from an arbitrary distribution P (x), x ∈ {0, 1} instead of being uniformly distributed. It is important to distinguish P (x), which is an arbitrary single-bit distribution defining the code ensemble and will be chosen at our convenience for optimizing it, from the a priori source distribution µ0 (x) of eqn (6.5), which is a distribution over the codewords and models the behaviour of the information source. As in the previous sections, we shall assume the source distribution µ0 to be uniform over the codewords (see eqn (6.4)). On the other hand, the codewords themselves have been constructed using the single-bit distribution P (x). We shall first analyse the RCE for a generic distribution P (x), under word MAP decoding. The main result is the following. Theorem 6.1 Consider communication over a binary-input discrete memoryless channel with transition probability Q(y|x), using a code from the RCE with input bit distribution P (x) and word MAP decoding. If the code rate is smaller than the mutual information IX,Y between two random variables X, Y with joint distribution P (x)Q(y|x), then the block error rate vanishes in the large-block-length limit. Using this result, one can optimize the performance of the ensemble over the choice of the distribution P (·). More precisely, we maximixe the achievable rate for error-free communication, IX,Y . The corresponding optimal distribution P ∗ (·) depends upon

Error-free communication with random codes

½¾½

that channel: it is the best distribution adapted to the channel. Since the channel capacity is in fact defined as the maximum mutual information between the channel input and the channel output (see eqn (1.38)), the RCE with input bit distribution P ∗ (·) allows one to communicate without errors up to the channel capacity. The above theorem implies therefore the ‘direct part’ of Shannon’s theorem (Theorem 1.23). Proof Assume that the codeword x(0) is transmitted through the channel and the message y ∈ Y N is received. The decoder constructs the probability for x to be the channel input, conditional on the output y (see eqn (6.5)). Word MAP decoding consists in minimizing the cost function E(x) = −

N

log2 Q(yi |xi )

(6.29)

i=1

over the codewords x ∈ CN (note that we are using natural logarithms here). Decoding will be successful if and only if the minimum of E(x) is realized over the transmitted codeword x(0) . The problem therefore consists in understanding the behaviour of the M 2M random variables E(x(0) ), . . . , E(x(2 −1) ). Once more, it is necessary to single out E(x(0) ). This is the sum of N i.i.d. random (0) variables − log Q(yi |xi ), and it is therefore well approximated by its mean E E(x(0) ) = −N

P (x)Q(y|x) log2 Q(y|x) = N HY |X .

(6.30)

x,y

In particular (1 − δ)N HY |X < E(x(0) ) < (1 + δ)N HY |X with probability approaching one as N → ∞. As for the 2M − 1 incorrect codewords, the corresponding ‘log-likelihoods’ E(x(1) ), M . . . , E(x(2 −1) ) are i.i.d. random variables. We can therefore estimate the smallest of them by following the approach developed for the REM and already applied to the RCE for a BSC. In Section 6.8, we prove the following large-deviation result for the distribution of these variables. Lemma 6.2 Let εi = E(x(i) )/N . Then ε1 , . . . , ε2M −1 are i.i.d. random variables and . their distribution satisfies a large-deviation principle of the form P(ε) = 2−N ψ(ε) . The rate function is given by Q(y)D(py ||P ) , (6.31) ψ(ε) ≡ min {py (·)}∈Pε

y

where the minimum is taken over the set of probability distributions {py (·), y ∈ Y} in the subspace Pε defined by the constraint Q(y)py (x) log2 Q(y|x) , (6.32) ε=− xy

and where we have defined Q(y) ≡

x

Q(y|x)P (x).

½¾¾


The solution of the minimization problem formulated in this lemma is obtained through a standard Lagrange multiplier technique: py (x) =

1 P (x)Q(y|x)γ , z(y)

(6.33)

where the (ε-dependent) constants z(y) and γ are chosen in order to satisfy the nor malizations x py (x) = 1 for all y ∈ Y, and the constraint (6.32). The rate function ψ(ε) is convex, with a global minimum (corresponding to γ = 0) at ε∗ = − x,y P (x)Q(y) log2 Q(y|x), where its value is ψ(ε∗ ) = 0. This implies that, with high probability, all incorrect codewords will have costs E(x(i) ) = N ε in the range εmin − δ ≤ ε ≤ εmax + δ for all δ > 0, where εmin and εmax are the two solutions of ψ(ε) = R. Moreover, for any ε inside thar interval, the number of codewords with E(x(i) ) N ε is exponentially large, and close to 2N R−N ψ(ε) . So, with high probability, the incorrect codeword with minimum cost has a cost close to N εmin , while the correct codeword has a cost close to N HY |X . Therefore MAP decoding will find the correct codeword if and only if HY |X < εmin . Let us now show that the condition HY |X < εmin is in fact equivalent to R < IX,Y . It turns out that the value ε = HY |X is obtained using γ = 1 in eqn (6.33), and therefore py (x) = P (x)Q(y|x)/Q(y). The corresponding value of the rate function is ψ(ε = HY |X ) = HY − HY |X = IY |X . The condition for error-free communication, HY |X < εmin , can thus be rewritten as R < ψ(HY |X ), or R < IX,Y . Example 6.3 Reconsider a BSC with flip probability p. We have E(x) = −(N − d(x, y)) log(1 − p) − d(x, y) log p .

(6.34)

Up to a rescaling the cost coincides with the Hamming distance from the received message. If we take P (0) = P (1) = 1/2, the optimal types are p0 (1) = 1 − p0 (0) =

pγ (1 − p)γ + pγ

(6.35)

(see eqn (6.33)) and analogously for p1 (x). The corresponding cost is ε = −(1 − δ) log(1 − p) − δ log p ,

(6.36)

where have we defined δ = pγ /[(1 − p)γ + pγ ]. The large-deviation rate function is given, parametrically, by ψ(ε) = log 2 − H(δ). The reader will easily recognize the results already obtained in the previous section.

Exercise 6.7 Consider communication over a discrete memoryless channel with finite input and output alphabets X and Y, and transition probability Q(y|x), x ∈ X , y ∈ Y. Check that the above proof remains valid in this context.

Geometry again: Sphere packing

6.5

½¾¿


Coding has a lot to do with the optimal packing of spheres, which is a mathematical problem of considerable interest in various branches of science. Consider, for instance, communication over a BSC with flip probability p. A code of rate R and block length N NR consists of 2N R points {x(1) · · · x(2 ) } in the hypercube {0, 1}N . With each possible channel output y ∈ {0, 1}N , the decoder associates one of the codewords x(i) . Therefore we can think of the decoder as realizing a partition of the Hamming space into 2N R decision regions D(i) , i ∈ {1, . . . , 2N R }, each one associated with a distinct codeword. If we require each decision region {D(i) } to contain a sphere of radius ρ, the resulting code is guaranteed to correct any error pattern such that fewer than ρ bits are flipped. One often defines the minimum distance of a code as the smallest distance between any two codewords. If a code has a minimum distance d, the Hamming spheres of radius ρ = (d − 1)/2 do not overlap, and the code can correct ρ errors, whatever their positions. 6.5.1

The densest packing of Hamming spheres

We are thus led to consider the general problem of sphere packing on the hypercube {0, 1}N . A (Hamming) sphere of centre x(0) and radius r is defined as the set of points x ∈ {0, 1}N such that d(x, x(0) ) ≤ r. A packing of spheres of radius r and cardinality NS is specified by a set of centres x1 , . . . , xNS ∈ {0, 1}N such that the spheres of radius r centred on these points are disjoint. Let NNmax (δ) be the maximum cardinality of a packing of spheres of radius N δ in {0, 1}N . We define the corresponding rate as max (δ) ≡ N −1 log2 NNmax (δ) and would like to compute this quantity in the infiniteRN dimensional limit max Rmax (δ) ≡ lim sup RN (δ) . N →∞

(6.37)

The problem of determining the function Rmax (δ) is open: only upper and lower bounds are known. Here we shall derive the simplest of these bounds. Proposition 6.4 1 − H2 (2δ) ≤ Rmax (δ) ≤ 1 − H2 (δ)

(6.38)

The lower bound is often called the Gilbert–Varshamov bound, the upper bound is called the Hamming bound. Proof Lower bounds can be proved by analysing good packing strategies. A simple such strategy is to take the centres of the spheres as 2N R random points with uniform probability in the Hamming space. The minimum distance between any pair of points must be larger than 2N δ. This can be estimated by defining the distance enumerator M2 (d), which counts how many pairs of points have a distance d between them. It is straightforward to show that if d = 2N δ and δ is kept fixed as N → ∞, N R 2 . −N N = 2N [2R−1+H2 (2δ)] . 2 (6.39) E M2 (d) = d 2

½


As long as R < [1 − H2 (2δ)]/2, the exponent in the above expression is negative. Therefore, by the Markov inequality, the probability of having any pair of centres at a distance smaller than 2δ is exponentially small in the size. This implies that Rmax (δ) ≥

1 [1 − H2 (2δ)] . 2

(6.40)

A better lower bound can be obtained by a closer examination of the above (random) packing strategy. In Section 6.2, we derived the following result. If 2N R points are chosen from the uniform distribution in the Hamming space {0, 1}N , and one of them is considered, its closest neighbour is with high probability at a Hamming distance close to N δGV (R). In other words, if we draw around each point a sphere of radius δ, with δ < δGV (R)/2, and one of these spheres is selected randomly, with high probability it will not intersect any other sphere. This remark suggests the following trick (sometimes called expurgation in coding theory). Go through all of the spheres one by one and check if the chosen sphere intersects any other one. If the answer is positive, simply eliminate the sphere. This reduces the cardinality of the packing, but only by a fraction approaching 0 as N → ∞: the packing rate is thus unchanged. As δGV (R) is defined by R = 1 − H2 (δGV (R)), this proves the lower bound in eqn (6.38). The upper bound can be obtained from the fact that the total volume occupied by the spheres is not larger than the volume of the hypercube. If we denote by ΛN (δ) the volume of an N -dimensional Hamming sphere of radius N δ, we get NS ΛN (δ) ≤ 2N . . Since ΛN (δ) = 2N H2 (δ) , this implies the upper bound in eqn (6.38). Better upper bounds can be derived using more sophisticated mathematical tools. An important result of this type is the linear-programming bound, Rmax (δ) ≤ H2

1 − 2δ(1 − 2δ) 2

,

(6.41)

whose proof goes beyond our scope. On the other hand, no better lower bound than the Gilbert–Varshamov result is known. It is a widespread conjecture that this bound is indeed tight: in high dimensions, there is no better way to pack spheres than placing them randomly and expurgating the small fraction of them that are ‘squeezed’. The various bounds are shown in Fig. 6.6.

Exercise 6.8 Derive two simple alternative proofs of the Gilbert–Varshamov bound using the following hints: (a) Given a constant δ, we look at all of the ‘dangerous’ pairs of points whose distance is smaller than 2N δ. For each dangerous couple, we can expurgate one of its two points. The number of points expurgated is less than or equal to the number of dangerous pairs, which can be bounded using E M2 (d). What is the largest value of δ such that this expurgation procedure does not reduce the rate?


½

1 random packing expurgated packing Hamming bound linear programming bound

0.8

0.6 R(δ) 0.4

0.2

0 0

0.1

0.2

0.3

0.4

0.5

δ Fig. 6.6 Upper and lower bounds on the maximum packing rate Rmax (δ) of Hamming spheres of radius N δ. Random packing and expurgated random packing provide lower bounds. The Hamming and linear-programming bounds are upper bounds. (b) Construct a packing x1 . . . xN as follows. The first centre x1 can be placed anywhere in {0, 1}N . The second centre can be placed anywhere outside a sphere of radius 2N δ centred on x(0) . In general, the i-th centre xi can be at any point outside the spheres centred on x1 . . . xi−1 . This procedure stops when the spheres of radius 2N δ cover all the space {0, 1}N , giving a packing of cardinality N equal to the number of steps, and with a radius N δ.

6.5.2

Sphere packing and decoding over a BSC

Let us now see the consequences of Proposition 6.4 for coding over a BSC. If the transmitted codeword is x(i) , the channel output will be (with high probability) at a distance close to N p from x(i) . Clearly, R ≤ Rmax (p) is a necessary and sufficient condition for the existence of a code which corrects any error pattern such that fewer than N p bits are flipped. Note that this correction criterion is much stronger than requiring a vanishing (bit or block) error rate. The direct part of Shannon’s theorem shows the existence of codes with a vanishing (as N → ∞) block error probability for R < 1 − H2 (p) = CBSC (p). As shown by the linear-programming bound in Fig. 6.6, CBSC (p) lies above Rmax (p) for large enough p. Therefore, for such values of p, there is a non-vanishing interval of rates Rmax (p) < R < CBSC (p) such that one can correct N p errors with high probability but one cannot correct all error patterns involving that many bits. Let us show, for the BSC case, that the condition R < 1 − H2 (p) is actually a necessary one for achieving a vanishing block error probability (this is nothing but the

½


converse part of Shannon channel coding theorem, i.e Theorem 1.23). We define PB (k) to be the block error probability under the condition that k bits are flipped by the channel. If the codeword x(i) is transmitted, the channel output lies on the border of a Hamming sphere of radius k centred on x(i) : ∂Bi (k) ≡ {z : d(z, x(i) ) = k}. Therefore

PB (k) =

NR 2

1 2N R

≥ 1−

1−

i=1

1 2N R

NR 2

i=1

|∂Bi (k) ∩ D(i) | |∂Bi (k)|

|D(i) | , |∂Bi (k)|

(6.42)

(6.43)

where D(i) is the set of channel outputs that are decoded to x(i) (under word MAP decoding, these coincide with the points in {0, 1}N that are closer to x(i) than to any . other codeword). For a typical channel realization, k is close to N p, and |∂Bi (N p)| = 2N H2 (p) . Since {D(i) } is a partition of {0, 1}N , i |D(i) | = 2N . We deduce that for any ε > 0 and a large enough N , PB ≥ 1 − 2N (1−R−H2 (p)+ε) ,

(6.44)

and thus reliable communication is possible only if R ≤ 1 − H2 (p).

6.6

Other random codes

A major drawback of the random code ensemble is that specifying a particular code (an element of the ensemble) requires N 2N R bits. This information has to be stored somewhere when the code is used in practice, and the memory requirement goes soon beyond the capabilities of hardware. A much more compact specification is possible for the random linear code (RLC) ensemble. In this case the encoder is required to be a linear map, and all such maps are equiprobable. Concretely, the code is fully specified by an N × M binary matrix G = {Gij } (the generator matrix), and encoding is performed by left multiplication by G: x : {0, 1}M → {0, 1}N , z → G z ,

(6.45) (6.46)

where the multiplication has to be carried modulo 2. Endowing the set of linear codes with a uniform probability distribution is essentially equivalent to assuming the entries of G to be i.i.d. random variables, with Gij = 0 or 1 with probability 1/2. Note that only M N bits are required for specifying a code within this ensemble.

A remark on coding theory and disordered systems

½

Exercise 6.9 Consider a linear code with N = 4 and |C| = 8 defined by C = {(z1 ⊕ z2 , z2 ⊕ z3 , z1 ⊕ z3 , z1 ⊕ z2 ⊕ z3 ) | z1 , z2 , z3 ∈ {0, 1}} ,

(6.47)

where we have denoted by ⊕ the sum modulo 2. For instance, (0110) ∈ C because we can take z1 = 1, z2 = 1, and z3 = 0, but (0010) ∈ C. Compute the distance enumerator for x(0) = (0110).

It turns out that the RLC has extremely good performance. Like the original Shannon ensemble, it allows one to communicate without errors below capacity. Moreover, the rate at which the block error probability PB vanishes is faster for the RLC than for the RCE. This justifies the considerable effort devoted so far to the design and analysis of specific ensembles of linear codes satisfying additional computational requirements. We shall discuss some of the best such codes in the following chapters.

6.7

A remark on coding theory and disordered systems

We would like to stress here the fundamental similarity between the analysis of random code ensembles and the statistical physics of disordered systems. As should already be clear, there are several sources of randomness in coding theory: • First of all, the code used is chosen randomly from an ensemble. This was the original idea used by Shannon to prove the channel coding theorem. • The codeword to be transmitted is chosen with uniform probability from the code. This hypothesis is supported by the source–channel separation theorem. • The channel output is distributed, once the transmitted codeword is fixed, according to a probabilistic process which accounts for the channel noise. • Once all the above elements have been given, one is left with the decoding problem. As we have seen in Section 6.3.3, both classical MAP decoding strategies and finite-temperature decoding can be defined in a unified framework. The decoder constructs a probability distribution µy,β (x) over the possible channel inputs, and (i) estimates its single-bit marginals µy,β (xi ). The decision about the i-th bit depends (i)

upon the distribution µy,β (xi ). The analysis of a particular coding system can therefore be regarded as the analysis of the properties of the distribution µy,β (x) when the code, the transmitted codeword and the noise realization are distributed as described above. In other words, we are distinguishing between two levels of randomness: on the first level, we deal with the first three sources of randomness, and on the second level, we use the distribution µy,β (x). The deep analogy with the theory of disordered systems should be clear at this point. The code, channel input, and noise realization play the role of quenched disorder (the sample), while the distribution µy,β (x) is the analogue of the Boltzmann distribution. In both cases, the problem consists in studying the properties of a probability distribution which is itself a random object.

½

6.8


Appendix: Proof of Lemma 6.2

We estimate (to the leading exponential order in the large-N limit) the probability PN (ε) for one of the incorrect codewords, x, to have cost E(x) = N ε. The channel output y = (y1 · · · yN ) is a sequence of N i.i.d. symbols distributed according to Q(y) ≡ Q(y|x)P (x) , (6.48) x

and the cost can be rewritten as E(x) ≡ −

N

log Q(yi |xi )

i=1

= −N

1 I(xi = x, yi = y) . N Q(y) i=1 N

Q(y) log Q(y|x)

x,y

(6.49)

There are approximatively N Q(y) positions i such that yi = y, for y ∈ Y. We assume that there are exactly N Q(y) such positions, and that N Q(y) is an integer (of course, this hypothesis is false in general: it is, however, a routine exercise, left to the reader, to show that it can be avoided by a small technical detour). Furthermore, we introduce 1 I(xi = x, yi = y) . py (x) ≡ N Q(y) i=1 N

(6.50)

Under the above assumptions, the function py (x) is a probability distribution over x ∈ {0, 1} for each y ∈ Y. Looking at the subsequence of positions i such that yi = y, this function counts the fraction of the xi ’s such that xi = x. In other words py (·) is the type of the subsequence {xi |yi = y}. Because of eqn (6.49), the cost can be written in terms of these types as follows: Q(y)py (x) log Q(y|x) . (6.51) E(x) = −N xy

Therefore E(x) depends upon x uniquely through the types {py (·) : y ∈ Y}, and this dependence is linear in py (x). Moreover, according to our definition of the RCE, x1 , . . . , xN are i.i.d. random variables with distribution P (x). The probability PN (ε) that E(x)/N = ε can therefore be deduced from Corollary 4.5. To the leading exponential order, we get . (6.52) PN (ε) = exp{−N ψ(ε) log 2} , ψ(ε) ≡ min Q(y)D(py ||P ) such that ε = − Q(y)py (x) log2 Q(y|x) (6.53) . py (·)

y

xy

Notes The random code ensemble dates back to Shannon (1948), who used it (somewhat implicitly) in his proof of the channel coding thorem. A more explicit (and complete)

Notes

½

proof was provided by Gallager (1965). The reader can find alternative proofs in standard textbooks such as Cover and Thomas (1991), Csisz´ ar and K¨ orner (1981), and Gallager (1968). The distance enumerator is a code property that has been extensively investigated in coding theory. We refer for instance to Csisz´ ar and K¨ orner (1981), and Gallager (1968). A treatment of the random code ensemble by analogy with the random energy model was presented by Montanari (2001b). More detailed results in the same spirit can be found in Barg and Forney (2002) and Forney and Montanari (2001). The analogy between coding theory and the statistical physics of disordered systems was put forward by Sourlas (1989). Finite temperature decoding was introduced in by Rujan (1993). A key ingredient of our analysis was the assumption, already mentioned in Section 1.6.2, that any codeword is a priori equiprobable. The fundamental motivation for such an assumption is the source–channel separation theorem. In simple terms, one does not lose anything in constructing an encoding system in two blocks. First, an ideal source code compresses the data produced by the information source and outputs a sequence of i.i.d. unbiased bits. Then, a channel code adds redundancy to this sequence in order to counteract the noise on the channel. The theory of error-correcting codes focuses on the design and analysis of this second block, leaving the first one to source coding. The interested reader may find proofs of the separation theorem in Cover and Thomas (1991), Csiszár and K¨ orner (1981), and Gallager (1968). Sphere packing is a classical problem in mathematics, with applications in various branches of science. The book by Conway and Sloane (1998) provides both a very good introduction to this problem and some far-reaching results related to it and its connections to other fields, in particular to coding theory. Finding the densest packing of spheres in Rn is an open problem when n ≥ 4.


7 Number partitioning Number partitioning is one of the most basic optimization problems. It is very easy to state: ‘Given the values of N assets, is there a fair partition of them into two sets?’ Nevertheless, it is very difficult to solve: it is NP-complete, and the known heuristics are often not very good. It is also a problem with practical applications, for instance in multiprocessor scheduling. In this chapter, we shall pay special attention to the partitioning of a list of i.i.d. random numbers. It turns out that most heuristics perform poorly on this ensemble of instances. This motivates their use as a benchmark for new algorithms, as well as their analysis. On the other hand, it is relatively easy to characterize analytically the structure of random instances. The main result is that low-cost configurations (those with a small imbalance between the two sets) can be seen as independent energy levels. The model behaves very much like the random energy model of Chapter 5, although with a different energy distribution. The problem is defined in Section 7.1. Section 7.2 discusses algorithmic aspects: it introduces a complete algorithm and a smart heuristic. These are used in Section 7.3 in order to study numerically the partitioning of i.i.d. random numbers. Then we discuss, in Section 7.4, a simple model in the REM family, the random cost model, which can be analysed in detail by elementary methods. Section 7.5 provides a short description of rigorous results on the partitioning of i.i.d. random numbers: these results show that the random cost model provides a very good approximation to the original problem.

7.1

A fair distribution into two groups?

An instance of the number-partitioning problem is defined by a set of N positive integers S = {a1 , . . . , aN } indexed by i ∈ [N ] ≡ {1, . . . , N }. One would like to partition the integers into two subsets {ai : i ∈ A} and {ai : i ∈ B ≡ [N ] \ A} in such a way as to minimize the discrepancy between the sums of the elements in the two subsets. In other words, a configuration is given by A ⊆ [N ], and its cost is defined by ai − ai . (7.1) EA = i∈A

i∈B

A perfect partition is such that the total numbers in each subset equilibrate, which a is even, and E = 1 if means that EA ≤ 1 (actually, EA = 0 if A i i i ai is odd). As usual, one can define several versions of the problem, among which there are the following: (i) The decision problem: does there exist a perfect partition? (ii) The optimization problem: find a partition of lowest cost.

½¿¾

Number partitioning

There are several variants of the problem. So far, we have left the size of A free. This is called the unconstrained version. One can also study a constrained version where the cardinality difference |A| − |B| between the two subsets is fixed at some number D. If D = 0, the partition is said to be balanced. Here, for simplicity, we shall keep mainly to the unconstrained case.

Exercise 7.1 As a small warm-up, the reader can show that the following are true (maybe by writing a simple exhaustive search program): (a) (b) (c) (d)

The set S1 = {10, 13, 23, 6, 20} has a perfect partition. The set S2 = {6, 4, 9, 14, 12, 3, 15, 15} has a perfect balanced partition. In the set S3 = {93, 58, 141, 209, 179, 48, 225, 228}, the lowest possible cost is 5. In the set S4 = {2474, 1129, 1388, 3752, 821, 2082, 201, 739}, the lowest possible cost is 48.

7.2 7.2.1

Algorithmic issues An NP-complete problem

In order to understand the complexity of the problem, one must first measure its size. This is in turn given by the number of characters required for specifying a particular instance. In number partitioning, this depends crucially on how large the integers can be. Imagine that we restrict ourselves to the case ai ∈ {1, . . . , 2M }

∀ i ∈ {1, . . . , N } ,

(7.2)

so that each of the N integers can be encoded with M bits. The entire instance can then be encoded in N M bits. An exhaustive search obviously finds a solution in 2N operations for unbounded numbers (any M ). On the other hand, for bounded numbers (fixed M ) and N going to infinity, the algorithm defined in the exercise below finds a solution in a time of order N 2 2M . It turns out that no known algorithm solves the number-partitioning problem in a time bounded from above by a power of its size, N M . In fact, number partitioning is NP-complete and is considered to be among the fundamental problems in this class.

Exercise 7.2 Consider the following ‘transfer matrix’ (or ‘dynamic-programming’) approach to number partitioning with bounded numbers {ai }. For any k ∈ {1, . . . , N } and any q ∈ {1, . . . , N 2M }, define j Zk (q) =

1 0

if q can be written as otherwise.

Pk r=1

nr ar with nr ∈ {0, 1},

(7.3)

Write a recursion relation expressing Zk+1 in terms of Zk . Show that the set of numbers {ZN (q)}, q ∈ {1, . . . , N 2M } can be computed in a time of order N 2 2M . How could you then use ZN to solve the number-partitioning problem?

Partition of a random list: Experiments

7.2.2

½¿¿

A simple heuristic and a complete algorithm

There is no good algorithm for the number-partitioning problem. One of the best heuristics, due to Karmarkar and Karp (KK), uses the following idea. We start from a list a1 , . . . , aN which coincides with the original set of integers, and reduce it by erasing two elements ai and aj from the list and replacing them by the difference |ai − aj |, if this difference is non-zero. This substitution means that a decision has been made to place ai and aj in two different subsets (but without fixing which subsets they are in). One then iterates this procedure as long as the list contains two or more elements. If in the end one finds either an empty list or the list {1}, then there exists a perfect partitioning. In the opposite case, the remaining integer is the cost of one particular partitioning, but the problem could have better solutions. Of course, there is a lot of flexibility and ingenuity involved in the best choice of the elements ai and aj selected at each step. In the KK algorithm one chooses the two largest numbers. Example 7.1 Let us see how the KK algorithm works with the first list in Exercise 7.1: {10, 13, 23, 6, 20}. In the first iteration, we substitute 23 and 20 by 3, giving the list {10, 13, 6, 3}. The next step gives {3, 6, 3}, then {3, 3}, and then ∅, showing that there exists a perfect partition. Readers can find out for themselves how to systematically reconstruct the partition. A modification due to Korf transforms the KK heuristic into a complete algorithm, which will return the best partitioning (possibly in exponential time). Each time one eliminates two elements ai and aj , two new lists are built: a ‘left’ list, which contains |ai − aj | (this corresponds to placing ai and aj in different groups), and a ‘right’ list which contains ai + aj (it corresponds to placing ai and aj in the same group). By iterating in this way, one constructs a tree with 2N −1 terminal nodes, each one containing the cost of a valid partition. Vice versa, the cost of each possible partition is reported at one of the terminal nodes (note that each of the 2N possible partitions A is equivalent to its complement [N ] \ A). If one is interested only in the decision ‘is there a perfect partition?’, the tree can be pruned as follows. Each time one encounters a list whose largest element is larger than the sum of all other elements plus 1, this list cannot lead to a perfect partition. One can therefore avoid constructing the subtree whose root is such a list. Figure 7.1 shows a simple example of application of this algorithm.

7.3

Partition of a random list: Experiments

A natural way to generate random instances of the number-partitioning problem is to choose the N input numbers ai as i.i.d. random variables. Here we will be interested in the case where they are uniformly distributed in the set {1, . . . , 2M }. As we discussed in Chapter 3, one can use these random instances in order to test the typical performance of algorithms, but we will also be interested in natural probabilistic issues, such as the distribution of the optimal cost, in the limits where N and M go to ∞. It is useful to first get an intuitive feeling for the respective roles of N (the size of the set) and M (the number of digits of each ai in base 2). Consider the instances

½

Number partitioning 23

20

13

10

6

13 10 6 3

6 3 3

3 3

0

6

43 13 10 6

9 3

6

12

17 3

14

20

29 3

26

56 10 6

30 10 6

23 6 3

32

20 6

14

26

40 6

34

46

46 6

40

52

66 6

60

72

Fig. 7.1 A complete search algorithm. Starting from a list, one erases the two largest numbers ai and aj and generates two new lists: the left one contains |ai −aj |, and the right one contains ai + aj . At the bottom of the tree, every leaf contains the cost of a valid partition. In a search for a perfect partition, the tree can be pruned at the dashed leaves because the largest number is bigger than the sum of the others: the dash–dotted lists are not generated. The KK heuristic picks up only the left branch. In this example, it is successful and finds the unique perfect partition.

S2 , S3 , S4 of Exercise 7.1. Each of them contains N = 8 random numbers, but they are randomly generated with M = 4, M = 8, and M = 16, respectively. Clearly, the larger M is, the larger is the typical value of the ai , and the more difficult it is to distribute them fairly. Consider the costs of all possible partitions: it is reasonable to expect that in about half of the partitions, the most significant bit of the cost is 0. Among these, about one-half should have the second significant bit equal to 0. The number of partitions is 2N −1 , and this qualitative argument can thus be iterated roughly N times. This leads one to expect that, in a random instance with large N , there will be a significant chance of having a perfect partition if N > M . In contrast, for N < M , the typical cost of the best partition should behave like 2M −N . This intuitive reasoning turns out to be essentially correct, as far as the leading exponential behaviour in N and M is concerned. Here, we first provide some numerical evidence, obtained with the complete algorithm of Section 7.2.2 for relatively small systems. In the next section, we shall validate our conclusions by a sharper analytical argument. Figure 7.2 shows a numerical estimate of the probability pperf (N, M ) that a randomly generated instance has a perfect partition, plotted versus N . This was obtained by sampling nstat instances of the problem for each pair N , M considered (here nstat = 104 , 103 , and 102 when M = 8, 16, and 24, respectively), and solving each instance by complete enumeration. The probability pperf (N, M ) was estimated as the fraction of the sampled instances for which a perfect partitioning was found. The standard deviation of such an estimate is pperf (1 − pperf )/nstat . For a fixed value of M , pperf (N, M ) crosses over from a value close to 0 at small N to a value close to 1 at large N . The typical values of N where the crossover takes

½

Partition of a random list: Experiments 1

1

0.8

0.8

0.6

0.6

pperf

pperf 0.4

0.4

0.2

0.2

0

0 0

5

10

15

20

25

30

0.6

N

0.8

1

1.2

1.4

1.6

1.8

2

M/N

Fig. 7.2 A numerical study of randomly generated number-partitioning instances, where the P ai are uniformly distributed in {1, . . . 2M }, with i ai even. The fraction of samples with a perfect balanced partition is plotted versus N (left plot: from left to right, M = 8, 16, 24), and versus κ = M/N (right plot). In the limit N → ∞ at fixed κ, it turns out that the probability becomes a step function, equal to 1 for κ < 1, and to 0 for κ > 1 (see also Fig. 7.4).

place seem to grow proportionally to M . It is useful to look at the same data from a slightly different perspective by defining the ratio κ=

M , N

(7.4)

and considering pperf as a function of N and κ. The plot of pperf (κ, N ) versus κ at fixed N shows a very interesting behaviour; see Fig. 7.2, right frame. A careful analysis of the numerical data indicates that limN →∞ pperf (κ, N ) = 1 for κ < 1, and 0 for κ > 1. We stress that the limit N → ∞ is taken with κ kept fixed (and therefore letting M → ∞ in proportion to N ). As we shall see in the following, we face here a typical example of a phase transition, in the sense introduced in Chapter 2. The behaviour of a generic large instance changes completely when the control parameter κ crosses a critical value κc ≡ 1. For κ < 1, almost all instances of the problem have a perfect partition (in the large-N limit), for κ > 1 almost none of them can be partitioned perfectly. This phenomenon has important consequences for the computational difficulty of the problem. A good measure of the performance of Korf’s complete algorithm is the number R of lists generated in the tree before the optimal partition is found. In Fig. 7.3, we plot the quantity log2 R averaged over the same instances as those which we used for the estimation of pperf in Fig. 7.2. The size of the search tree first grows exponentially with N and then reaches a maximum around N ≈ M . We see a peak in log2 R plotted as a function of κ, somewhere around κ = κc = 1: problems close to the critical point are the hardest ones for the algorithm considered. A similar behaviour is found with other algorithms, and in fact we shall encounter it in many other decision problems, such as the satisfiability and colouring problems. When a class of random instances shows a phase transition as a function of one parameter, it is generally the case that the most difficult instances are found in the neighbourhood of the phase transition.

½

Number partitioning 1

20

0.8

15

0.6

log2 R 10

0.4

5

0.2

0

log2 R N −1

0 0

5

10

15

20

25

30

0.6

N

0.8

1

1.2

1.4

1.6

1.8

2

M/N

Fig. 7.3 Left: average of log2 R, where R is the size of the search tree for the KK algorithm. The three curves correspond to M = 8, 16, and 24 (from left to right). The size grows exponentially with N , and then reaches a maximum for N ≈ M . Right: the average of log2 R/(N − 1) plotted versus κ = M/N .

7.4 7.4.1

The random cost model Definition of the model

Consider, as before, the probability space of random instances constructed by taking the numbers aj as i.i.d., uniformly distributed in {1, . . . , 2M }. For a given partition A, the cost EA is a random variable with a probability distribution PA . Obviously, the costs of two partitions A and A are correlated random variables. The random cost approximation consists in neglecting these correlations. Such an approximation can be applied to any kind of problem, but it is not always a good one. Remarkably, as discovered by Mertens, the random cost approximation turns out to be ‘essentially exact’ for the partitioning of i.i.d. random numbers. In order to state precisely the above-mentioned approximation, one defines a random cost model (RCM), which is similar to the REM of Chapter 5. A sample is defined by the costs of all 2N −1 ‘partitions’ (here we identify the two complementary partitions A and [N ]\A). The costs are supposed to be i.i.d. random variables drawn from the probability distribution P. In order to mimic the random number-partitioning problem, P is taken to be the same as the distribution of the cost of a random partition A in the original problem: 1 PA . (7.5) P ≡ N −1 2 A

Here PA is the distribution of the cost of a partition A in the original numberpartitioning problem. Let us analyse the behaviour of P for large N . The cost of a randomly chosen partition in the original problem is given by | i σi ai |, where the σi are i.i.d.variables taking values ±1 with probability 1/2. For large N , the distribution of i σi ai is characterized by the central limit theorem, and P is obtained by restricting it to the positive domain. In particular, the cost of a partition will be, with high probability, of 2 , where order N αM

The random cost model 2 αM ≡ E a2 =

1 2M 1 M 1 2 + 2 + . 3 2 6

½

(7.6)

Moreover, for any 0 ≤ x1 < x2 , - x2 2 2 E P ∈ [x , x ] e−x /2 dx . 1 2 2 π N αM x1 Finally, the probability of a perfect partition P(E = 0) is just the probability of M M return to the origin for a random walk with steps σi ai ∈ {−2 , . . . , −1}∪{1, . . . , 2 }. Assuming for simplicity that i ai is even, we get 1 6 −M 2 P(0) 2 , (7.7) 2 πN 2πN αM 2 is the density of a normal random variable of mean 0 and variance where 1/ 2πN αM 2 N αM near the origin, and the extra factor of 2 comes from the fact that the random walk is on even integers only. As we shall show in the next sections, the RCM is a good approximation to the original number-partitioning problem. An intuitive explanation of this property can be found in the exercise below. Exercise 7.3 Consider two random, uniformly distributed, independent partitions A and A . Let P(E, E ) denote the joint probability of their energies when the numbers {ai } are i.i.d. and uniformly distributed over {1, . . . , 2M }. Show that P(E, E ) = P(E)P(E )[1+o(1)] in the large-N, M limit, if E, E < C 2M for some fixed C.

7.4.2

Phase transition

We can now proceed with the analysis of the RCM. We shall first determine the phase transition, then study the phase κ > 1, where, typically, no perfect partition can be found, and finally study the phase κ < 1, where an exponential number of perfect partitions exist. Consider a random instance of the RCM. The probability that no perfect partition exist is just the probability that each partition has a strictly positive cost. Since, within the RCM, the 2N −1 partitions have i.i.d. costs with distribution P, we have 2N −1

1 − pperf (κ, N ) = [1 − P(0)]

.

(7.8)

In the large-N limit with fixed κ, the zero-cost probability is given by eqn (7.7). In particular, P(0) 1. Therefore, 3 N −1 N (1−κ) 2 pperf (κ, N ) = 1 − exp[−2 + o(1) . (7.9) P(0)] + o(1) = 1 − exp − 2πN This expression predicts a phase transition for the RCM at κc = 1. Notice, in fact, that limN →∞ pperf (κ, N ) = 1 if κ < 1, and 0 if κ > 1. Moreover, eqn (7.9) describes

½

Number partitioning

1

0.8

0.6

pperf 0.4

0.2

0 -3

-2

-1

0

1

2

3

4

5

x Fig. 7.4 Finite-size scaling plot for the data of Fig. 7.2. The (estimated) probability of perfect partition pperf (N, M ) is plotted versus the rescaled variable x = N (κ − κc ) + (1/2) log2 N . The agreement with the theoretical prediction (7.10) is very good.

the precise behaviour of pperf (κ, N ) around the critical point κc for finite N . Let us define the variable x = N (κ − κc ) + (1/2) log2 N . In the limit N → ∞ and κ → κc at fixed x, one finds the following crossover behaviour: lim pperf (κ, N ) = 1 − exp −

N →∞

κ→κc

3 −x 2 . 2π

(7.10)

This is an example of finite-size scaling behaviour. In order to compare the above prediction with our numerical results for the original number-partitioning problem, we plot in Fig. 7.4 pperf (κ, N ) versus the scaling variable x. Here we have used the same data as that presented in Fig. 7.2, just changing the horizontal scale from N to x. The good collapse of the curves for various values of M provides evidence for the claim that the number-partitioning problem is indeed asymptotically equivalent to the RCM and undergoes a phase transition at κ = 1.

P Exercise 7.4 The argument above assumes that i ai is even. This is a condition which was imposed in the simulation whose results are P presented in Fig. 7.4. How should one modify the estimate of P(0) in eqn (7.7) when i ai is odd? Show that, in this case, if one keeps x = N (κ − κc ) + (1/2) log2 N , the scaling function becomes i h p the definition 1 − exp − 6/π 2−x . Run a simulation to check this prediction.

The random cost model

7.4.3

½

Study of the two phases

Let us now study the minimum cost in the phase κ > 1. The probability that all configurations have a cost larger than E is P(∀A : EA > E) =

1−

E

2N −1 P(E )

.

(7.11)

E =0

E This probability is non-trivial (i.e. it is bounded away from 0 and 1) if E =0 P(E ) = substituting P(E ) → P(0), which gives the O(2−N ). This sum can be estimated by √ N −1 M −N ) ∼ 2 N (note that, as this value of E is much condition E 1/(P(0)2 smaller than the scale over which P(E) varies significantly (see eqn (7.7)), the substitution of P(0) for P(E ) is indeed consistent). We therefore get, from eqn (7.11)

ε lim P ∀A : EA > N →∞ P(0)2N −1

= e−ε I(ε ≥ 0) .

(7.12)

In particular, the mean of the distribution on the right-hand side is equal to 1. This implies that the expectation of the lowest cost in the problem is E Egs = 2πN/3 2N (κ−1) . These predictions also fit the numerical results for number partitioning very well.

Exercise 7.5 Show that the probability density of the k-th lowest-cost configuration, in terms of the rescaled variable ε, is (εk−1 /(k − 1)!) exp(−ε) I(ε > 0). This is a typical case of extreme value statistics for bounded i.i.d. variables.

For the phase κ < 1, we already know that, for all but a vanishing fraction of samples, there exists at least one configuration with zero cost. It is instructive to count the number of zero-cost configurations. Since each configuration has zero cost independently with probability P(0), the number Z of zero-cost configurations is a binomial random variable with distribution N −1 2 2N −1 −Z P(0)Z [1 − P(0)] . (7.13) P (Z) = Z . In particular, for large N , Z concentrates around its average value Zav = 2N (1−κ) . One can define an entropy density of the ground state as sgs =

1 log2 Z . N

(7.14)

The RCM result in eqn (7.13) predicts that for κ < 1, the entropy density is close to (1 − κ) with high probability. Once again, numerical simulations of the original number-partitioning problem confirm this expectation.

½

Number partitioning

Exercise 7.6 Using the integral representation of the logarithm Z

∞

log2 x = 0

” 1 “ −t log 2 e − e−tx dt , t

(7.15)

compute E sgs directly. It will be useful to notice that the integral over t is dominated by very small values of t, of order 1/(2N −1 P(0)). One then easily finds E sgs (1/N ) log2 (2N −1 P(0)) 1 − κ.

7.5

Partition of a random list: Rigorous results

A detailed rigorous characterization of the phase diagram for the partitioning of random numbers confirms the predictions of the RCM. We shall first state some of the exact results known for the balanced partitioning of N numbers. For definiteness we keep as before to the case where the ai are i.i.d. and uniformly distributed in {1, . . . , 2M }, N and both N and i=1 ai are even. The following results hold in the ‘thermodynamic limit’ N, M → ∞ with fixed κ = M/N . Theorem 7.2 There is a phase transition at κ = 1. For κ < 1, with high probability, a randomly chosen instance has a perfect balanced partition. For κ > 1, with high probability, a randomly chosen instance does not have a perfect balanced partition. Theorem 7.3 In the phase κ < 1, the entropy density (7.14) of the number of perfect balanced partitions converges in probability to s = 1 − κ. Theorem 7.4 Define E = 2N (κ−1) 2πN/3, and let E1 ≤ · · · ≤ Ek be the k lowest costs, with k fixed. The k-tuple ε1 = E1 /E, . . . , εk = Ek /E then converges in distribution to (W1 , W1 + W2 , . . . , W1 + . . . Wk ), where the Wi are i.i.d. random variables with distribution P (Wi ) = e−Wi I(Wi ≥ 0). In particular, the (rescaled) optimal cost distribution converges to P (ε1 ) = e−ε1 I(ε1 ≥ 0). Note that these results all agree with the RCM. In particular, Theorem 7.4 states that, for fixed k and N → ∞, the lowest k costs are i.i.d. variables, as assumed in the RCM. This explains why the random cost approximation is so good. The proofs of these theorems (and of more detailed results concerning the scaling in the neighbourhood of the phase transition point κ = 1) are all based on the analysis of an integral representation for the number of partitions with a given cost, which we shall derive below. We shall then outline the general strategy by proving the existence of a phase transition, as in Theorem 7.2, and we refer the reader to the original literature for the other proofs. 7.5.1

Integral representation

For simplicity, we keep to the case where i ai is even: similar results can be obtained in the case of an odd sum (but the lowest cost is then equal to 1).

Partition of a random list: Rigorous results

Proposition 7.5 Given a set S = {a1 , . . . , aN } with partitions with cost E = 0 can be written as Z = 2N −1

N

π

−π j=1

cos(aj x)

½

i

ai even, the number Z of

dx . 2π

(7.16)

Proof We represent the partition A by writing %σi = 1 if i ∈ A, & and σi = −1 if N i ∈ B = [N ] \ A. We can write Z = 12 σ1 ,...,σN I σ a = 0 , where the factor j=1 j j

1/2 comes from the symmetry between A and B (the same partition is represented by the sequence σ1 , . . . , σN and by −σ1 , . . . , −σN ). We use the integral representation valid for any integer number a, π dx , (7.17) eixa I(a = 0) = 2π −π

which gives

1 Z= 2 σ ,...,σ 1

P

π

eix( N

−π

j

σj aj )

dx . 2π

(7.18)

The sum over the σi gives the integral representation (7.16)

Exercise 7.7 Show that a similar representation holds for the number of partitions with cost E ≥ 1, with an extra factor 2 cos(Ex) in the integrand. For the case of balanced partitions, find a similar representation with a two-dimensional integral.

The integrand of eqn (7.16) is typically exponential in N and oscillates wildly. It is thus tempting to compute the integral by the method of steepest descent. This strategy yields correct results for the phase κ ≤ 1, but it is not easy to control it rigorously. Hereafter, we shall use simple first- and second-moment estimates of the integral, which are powerful enough to derive the main features of the phase diagram. Finer control gives more accurate predictions which go beyond this presentation. 7.5.2

Moment estimates

We start by evaluating the first two moments of the number of perfect partitions Z. Proposition 7.6 In the thermodynamic limit, the first moment of Z behaves as 3 N (1−κ) (1 + Θ(1/N )) . (7.19) EZ = 2 2πN Proof The expectation value is taken over choices of ai where i ai is even. Let us use a modified expectation, denoted by Ei , over all choices of a1 , . . . , aN , without any parity constraint, so that the ai are i.i.d. Clearly, Ei Z = (1/2)E Z, because a perfect

½

Number partitioning

partition can be obtained only in the case where i ai is even, and this happens with probability 1/2. Because of the independence of the ai in the expectation Ei , one gets from eqn (7.16) π N dx . (7.20) [Ei cos(a1 x)] E Z = 2Ei Z = 2N 2π −π The expectation of the cosine is Ei cos(a1 x) = 2−M cos

%x

& sin(2M x/2) (2M + 1) ≡ g(x) . 2 sin(x/2)

(7.21)

A little thought shows that the integral in eqn (7.20) is dominated in the thermodynamic limit√by values of x very near to 0. More precisely, we can rescale the variable as x=x ˆ/(2M N ). We then have g(x) = 1 − x ˆ2 /(6N ) + Θ(1/N 2 ). The leading behaviour of the integral (7.20) at large N is thus given by 2 ∞ dˆ x x ˆ 3 N −M 1 N −M √ 2 , (7.22) EZ 2 exp − 6 2π 2πN N −∞ up to corrections of order 1/N .

Exercise 7.8 Show that, for E even, with E ≤ C2M for a fixed C, the number of partitions with cost E is also given by eqn (7.19) in the thermodynamic limit.

Proposition 7.7 When κ < 1, the second moment of Z behaves in the thermodynamic limit as 2 (7.23) E Z 2 = [E Z] (1 + Θ(1/N )) . Proof We again release the constraint of an even i ai , so that π dx1 π dx2 N [E cos(a1 x1 ) cos(a1 x2 )] E Z 2 = 22N −1 −π 2π −π 2π

(7.24)

The expectation of the product of the two cosines is E cos(a1 x1 ) cos(a1 x2 ) =

1 [g(x+ ) + g(x− )] , 2

(7.25)

where x± = x1 ± x2 . In order to find out which regions of the integration domain are important in the thermodynamic limit, one must be careful because the function g(x) is periodic with a period of 2π. The double integral is performed in the square [−π, +π]2 . The region √ of this square where g can be very close to 1 are the ‘centre’ where x1 , x2 = Θ(1/(2M N )), and the four corners, close to (±π, ±π), obtained from the centre by a shift of ±π in x+ or in x− . Because of the periodicity of g(x), the total contribution of

Notes

½

the four corners equals that of the centre. Therefore one can first compute the integral √ ˆ1(2) /(2M N ). The correct near the centre, using the change of variables x1(2) = x value of E Z 2 is equal to twice the result of this integral. The remaining part of the computation is straightforward, and indeed gives EZ 2 3 · 22N (1−κ) /(2πN ). In order for this argument to be correct, one must show that the contributions from outside the centre are negligible in the thermodynamic limit. The leading correction √ comes from regions where x+ = Θ(1/(2M N )), while x− is arbitrary. One can explicitly evaluate the integral in such a region by using the saddle point approximation. The result is of order Θ(2N (1−κ) /N ). Therefore, for κ < 1 the relative contributions from outside the centre (and the corners) are exponentially small in N . A careful analysis of the above two-dimensional integral can be found in the literature. Propositions 7.6 and 7.7 above have the following important implications. For κ > 1, E Z is exponentially small in N . Since Z is a non-negative integer, this implies (by the first-moment method) that, in most instances, Z is indeed 0. For κ < 1, E Z is exponentially large. Moreover, the normalized random variable Z/E Z has a small second moment, and therefore small fluctuations. An analysis similar to the one we did in Section 5.2.1 for the REM then shows that Z is positive with high probability. We have thus proved the existence of a phase transition at κc = 1, i.e. Theorem 7.2.

Exercise P −βEA7.9 Define as usual the partition function at inverse temperature β as Z(β) = . Using the integral representation Ae e−|U | =

Z

∞ −∞

dx 1 e−ixU π 1 + x2

(7.26)

P and the relation k∈Z 1/(1 + x2 k2 ) = π/(x tanh(π/x)), show that the ‘annealed average’ for i.i.d. numbers ai is r Ei (Z) = 2N (1−κ)

1 3 (1 + Θ(1/N )) . 2πN tanh(β/2)

(7.27)

Notes A nice elementary introduction to number partitioning is provided by Hayes (2002). The NP-complete nature of this problem is a classical result which can be found in textbooks such as Papadimitriou (1994) and Garey and Johnson (1979). The Karmarkar– Karp algorithm was introduced in a technical report (Karmarkar and Karp, 1982). Korf’s complete algorithm is given in Korf (1998). There has been a lot of work on the partitioning of random i.i.d. numbers. In particular, the case where ai is uniform in [0, 1] has been studied in detail. This can be regarded as equivalent to uniform costs in {1, . . . , 2M }, in the large-M limit. The scaling of the cost of the optimal solution in this case was studied as early as 1986 by Karmarkar et al. (1986). On the algorithmic side, this √ is a very challenging problem. As we have seen, the optimal partition has a cost O( N 2−N ); however, all known

½

Number partitioning

heuristics perform badly ( For instance, the KK heuristic finds a solu 'on this problem. tion with a cost Θ exp −0.72(log N )2 , which is very far from the optimal scaling (Yakir, 1996). The phase transition was identified numerically by Gent and Walsh (1998), and studied by statistical-physics methods by Ferreira and Fontanari (1998) and Mertens (1998). Mertens also introduced the random cost model (Mertens, 2000). His review paper (Mertens, 2001) provides a good summary of these publications, and will help to solve Exercises 7.3, 7.5, and 7.8. The parity questions discussed in Exercise 7.4 have been studied by Bauke (2002). Elaborating on these statistical-mechanics treatments, detailed rigorous results have been obtained for the unconstrained problem (Borgs et al., 2001) and more recently, for the constrained case (Borgs et al., 2003). These results go far beyond the theorems which we have stated here, and the interested reader is encouraged to study the original papers. Readers will also find there all the technical details needed to fully control the integral representation used in Section 7.5, and the solutions to Exercises 7.6 and 7.7.

8 Introduction to replica theory Over the past 30 years, the replica method has evolved into a rather sophisticated tool for attacking theoretical problems as diverse as spin glasses, protein folding, vortices in superconductors, and combinatorial optimization. In this book, we adopt a different (but equivalent and, in our view, more concrete) approach: the ‘cavity method’. In fact, readers can skip this chapter without great harm to their understanding of the rest of this book. It can be instructive, nevertheless, to have some knowledge of replicas: the replica method is an amazing construction which is incredibly powerful. It is not yet a rigorous method: it involves some formal manipulations, and a few prescriptions which may appear arbitrary. Nevertheless, these prescriptions are fully specified, and the method can be regarded as an ‘essentially automatic’ analytical tool. Moreover, several of its most important predictions have been confirmed rigorously through alternative approaches. Among its most interesting aspects is the role played by ‘overlaps’ among replicas. It turns out that the subtle probabilistic structure of the systems under study are often most easily phrased in terms of such variables. Here we shall take advantage of the simplicity of the random energy model defined in Chapter 5 to introduce replicas. This is the topic of Section 8.1. A more complicated spin model is introduced and discussed in Section 8.2. In Section 8.3, we study the relationship between the simplest replica-symmetry-breaking scheme and extreme value statistics. Finally, in Section 8.4, we briefly explain how to perform a local stability analysis in replica space. This is one of the most common consistency checks in the replica method.

8.1

Replica solution of the random energy model

As we saw in Section 5.1, a sample (or instance) of the REM is given by the values of 2N energy levels Ej , with j ∈ {1, . . . , 2N }. The energy levels are i.i.d. Gaussian random variables with mean 0 and variance N/2. A configuration of the REM is specified by the index j of one energy level. The partition function for a sample with energy levels {E1 . . . , E2N } is 2N exp (−βEj ) , (8.1) Z= j=1

and is itself a random variable (in physicists’ language, ‘Z fluctuates from sample to sample’). In Chapter 5 we showed that intensive thermodynamic potentials are selfaveraging, meaning that their distribution is sharply concentrated around the mean

½

Introduction to replica theory

value in the large-N limit. Among these quantities, a prominent role is played by the free-energy density f = −(βN )−1 log Z. Other potentials can in fact be computed from derivatives of the free energy. Unlike thermodynamic potentials, the partition function has a broad distribution even for large sizes. In particular, its average is dominated, in the low temperature phase, by extremely rare samples. In order to describe typical samples, one has to compute the average of the log-partition function, E log Z, i.e., up to a constant, the average free-energy density. It turns out that computing integer moments of the partition function E Z n , with n ∈ N, is much easier than computing the average log-partition function E log Z. This happens because Z is the sum of a large number of ‘simple’ terms. If, on the other hand, we were able to compute E Z n for any real n or, at least, for n small enough, the average log-partition function could be determined using, for instance, the relation 1 (8.2) E log Z = lim log(E Z n ) . n→0 n The idea is to carry out the calculation of E Z n ‘as if’ n were an integer. At a certain point, after obtaining a manageable enough expression, we ‘remember’ that n has in fact to be a real number and take this into account. As we shall see this whole line of approach has some of the flavour of an analytic continuation, but in fact it has quite a few extra grains of salt. . . The first step consists in noticing that Z n can be written as an n-fold sum N

2

n

Z =

exp (−βEi1 − · · · − βEin ) .

(8.3)

i1 ...in =1

This expression can be interpreted as the partition function of a new system. A configuration of this system is given by the n-tuple (i1 , . . . , in ), with ia ∈ {1, . . . , 2N }, and its energy is Ei1 ...in = Ei1 + · · · + Ein . In other words, the new system is formed from n statistically independent copies of the original one. We shall refer to such copies as replicas. In order to evaluate the average of eqn (8.3), it is useful to first rewrite it as N

n

Z =

2

N

2

i1 ...in =1 j=1

exp −βEj

n

I(ia = j)

.

(8.4)

a=1

By exploiting the linearity of the expectation, the independence of the Ej ’s, and the fact that they are Gaussian, one gets ⎛ ⎞ 2N n 2 N β exp ⎝ I(ia = ib )⎠ . (8.5) E Zn = 4 i ...i =1 1

n

a,b=1

E Z n can also be interpreted as the partition function of a new ‘replicated’ system. As before, a configuration is given by the n-tuple (i1 , . . . , in ), but now its energy is n Ei1 ...in = −N β/4 a,b=1 I(ia = ib ).

½


This replicated system has several interesting properties. First of all, it is no longer a disordered system: the energy is a deterministic function of the configuration. Second, replicas are no longer statistically independent: they do ‘interact’. This is due to the fact that the energy function cannot be written as a sum of single replica terms. The interaction amounts to an attraction between different replicas. In particular, the lowest-energy configurations are obtained by setting i1 = · · · = in . Their energy is Ei1 ...in = −N βn2 /4. Third, the energy itself depends upon the temperature, although in a very simple fashion. The effect of interaction is stronger at low temperature. The origin of the interaction among replicas is easily understood. For one given sample of the original problem, the Boltzmann distribution is concentrated at low temperature (β 1) on the lowest energy levels: all the replicas will tend to be in the same configuration with large probability. When averaging over sample realizations (i.e. over the energy levels E1 , . . . , E2N ), we do not see any longer which configuration i ∈ {1, . . . , 2N } has the lowest energy, but we still see that the replicas prefer to stay in the same state. There is no mystery in these remarks. The elements of the n-tuple (i1 . . . in ) are independent conditional on the sample, that is on the realization of the energy levels Ej , j ∈ {1, . . . , 2N }. If we do not condition on the realization, i1 , . . . , in become dependent. Given the configurations (i1 . . . in ) of the replicas, it is convenient to introduce an n × n matrix Qab = I(ia = ib ), with elements in {0, 1}. We shall refer to this matrix as the overlap matrix. The summand in eqn (8.5) depends upon the configuration (i1 . . . in ) only through the overlap matrix. We can therefore rewrite the sum over configurations as ⎞ ⎛ n 2 Nβ E Zn = (8.6) NN (Q) exp ⎝ Qab ⎠ . 4 Q

a,b=1

number of configurations (i1 . . . in ) whose overlap matrix is Here NN (Q) denotes the Q = {Qab }, and the sum Q runs over the symmetric {0, 1} matrices with ones on the diagonal. The number of such matrices is 2n(n−1)/2 , while the number of configurations of the replicated system is 2N n . It is therefore natural to guess that the number of configurations with a given overlap matrix satisfies a large-deviation principle of the . form NN (Q) = exp(N s(Q)).

Exercise 8.1 Show that the overlap matrix always has the following form: There exists a partition G1 , G2 , . . . , Gng of the n replicas (this means that G1 ∪ G2 ∪ · · · ∪ Gng = {1 . . . n} and Gi ∩ Gj = ∅) into ng groups such that Qab = 1 if a and b belong to the same group, and Qab = 0 otherwise. Prove that NN (Q) satisfies the large-deviation principle described above, with s(Q) = ng log 2.

Using this form of NN (Q), the replicated partition function can be written as . E Zn = exp (N g(Q)) , Q

g(Q) ≡

n β2 Qab + s(Q) . 4 a,b=1

(8.7)

½


The strategy of the replica method is to estimate the above sum using the saddle point method.1 The ‘extrapolation’ to non-integer values of n is discussed afterward. Note that this programme is completely analogous to the treatment of the Curie–Weiss model in Section 2.5.2 (see also Section 4.3 for related background), with the extra step of extrapolating to non-integer n. 8.1.1

The replica-symmetric saddle point

The function g(Q) is symmetric under permutation of replicas. Let π ∈ Sn be a permutation of n objects, and denote by Qπ the matrix with elements Qπab = Qπ(a)π(b) . Then g(Qπ ) = g(Q). This is a simple consequence of the fact that the n replicas were equivalent from the beginning. This symmetry is called replica symmetry, and is a completely generic feature of the replica method. When the dominant saddle point possesses this symmetry (i.e. when Qπ = Q for any permutation π), one says that the system is replica-symmetric (RS). In the opposite case, replica symmetry is spontaneously broken in the large-N limit. This is analogous to the spontaneous breaking of +/− symmetry in the Curie–Weiss model, which we discussed in Chapter 2 (see Section 2.5.2). In view of this permutation symmetry, the simplest idea is to seek a replicasymmetric saddle point. If Q is invariant under permutation, then, necessarily, Qaa = 1, and Qab = q0 for any pair a = b. We are left with two possibilities: • The matrix QRS,0 is defined by q0 = 0. In this case NN (QRS,0) = 2N (2N − 1) . . . (2N −n+1), which yields s(QRS,0 ) = n log 2 and g(QRS,0 ) = n β 2 /4 + log 2 . • The matrix QRS,1 is defined by q0 = 1. This means that i1 = · · · = in . There are, of course, NN (QRS,1 ) = 2N choices of the n-tuple (i1 . . . in ) compatible with this constraint, which yields s(QRS,1 ) = log 2 and g(QRS,1 ) = n2 β 2 /4 + log 2. Keeping for the moment to these RS saddle points, we need to find which one dominates the sum. In Fig. 8.1 we plot, for n = 3 and n = 0.5, g0 (n, β) ≡ g(QRS,0 ) and g1 (n, β) ≡ g(QRS,1 ) as functions of T = 1/β. Note that the expressions we have obtained for g0 (n, β) and g1 (n, β) are polynomials in n, which we can plot for non-integer values of n. When n > 1, the situation is always qualitatively the same as the one shown 4 log 2/n, we have g1 (β, n) > g0 (β, n) for the n = 3 case. If we let βc (n) = for β > βc (n), while g1 (β, n) < g0 (β, n) for β < βc (n). Assuming for the moment that the sum in eqn (8.7) is dominated by replica-symmetric terms, we have . E Z n = exp{N max[g0 (β, n), g1 (β, n)]}. The point βc (n) can therefore be interpreted as a phase transition in the system of n replicas. At high temperatures (β < βc (n)), the q0 = 0 saddle point dominates the sum: replicas are essentially independent. At low temperature, the partition function is dominated by q0 = 1: replicas are locked together. This fits nicely within our qualitative discussion of the replicated system in the previous section. The problems appear when we consider the n < 1 situation. In this case we still have a phase transition at βc (n) = 4 log 2/n, but the high- and low-temperature regimes 1 Speaking of ‘saddle points’ is a bit sloppy in this case, since we are dealing with a discrete sum. By this, we mean that we aim at estimating the sum in eqn (8.7) through a single ‘dominant’ term.


½

4 3.5 3 n=3 g(Q)

2.5 2 n = 0.5

1.5 1 0.5 0 0

0.5

1

1.5 T

2

2.5

3

Fig. 8.1 Rate function g(Q) for the REM (see eqn (8.7)) versus temperature; g(Q) is evaluated here at the two replica-symmetric saddle points QRS,0 (continuous curves) and QRS,1 (dashed curves), in the cases n = 3 and n = 0.5.

exchange their roles. At low temperature (β > βc (n)), we have g1 (β, n) < g0 (β, n), and at high temperature (β < βc (n)), we have g1 (β, n) > g0 (β, n). If we were to apply the usual prescription and pick out the saddle point which maximizes g(Q), we would obtain nonsense, physically (replicas become independent at low temperatures, and correlated at high temperatures, contrary to our general discussion) as well as mathematically (for n → 0, the function E Z n does not go to one, because g1 (β, n) is not linear in n at small n). As a matter of fact, the replica method prescribes that, in this regime n < 1, one must estimate the sum (8.7) using the minimum of g(Q)! There is no mathematical justification for this prescription in the present context. In the next example and the following chapters we shall outline some of the arguments employed by physicists in order to rationalize this choice.

Example 8.1 In order to get some understanding of this prescription, consider the following toy problem. We want to apply the replica recipe to the quantity Ztoy (n) = (2π/N )n(n−1)/4 (for a generic real n). For n integer, we have the following integral representation: P 2 dQab ≡ eN g(Q) dQab , (8.8) Ztoy (n) = e−(N/2) (ab) Qab (ab)

(ab)

where (ab) runs over all of the unordered pairs of indices a, b ∈ {1 . . . n} with a = b, and the integrals over Qab run over the real line. Now we try to evaluate the above integral by the saddle point method, and begin with the assumption that it is dominated by a replica-symmetric point Q∗ab = q0 for any a = b, yielding g(Q∗ ) = −n(n − 1)q02 /2. Next, we have to fix the value of q0 ∈ R. It is clear that the

½


. correct result is recovered by setting q0 = 0, which yields Ztoy (n) = 1. Moreover this ∗ is the unique choice such that g(Q ) is stationary. However, for n < 1, q0 = 0 corresponds to a minimum, rather than to a maximum of g(Q∗ ). A formal explanation of this odd behaviour is that the number of degrees of freedom, i.e. the number of matrix elements Qab with a = b, becomes negative for n < 1. This is one of the strangest aspects of the replica method, but it is unavoidable. Another puzzle which we shall discuss later, concerns the exchange of the order of the N → ∞ and n → 0 limits. Let us therefore select the saddle point q0 = 0, and use the trick (8.2) to evaluate the free-energy density. Assuming that the N → ∞ and n → 0 limits commute, we get the RS free energy 1 1 E log Z = lim lim log(E Z n ) N →∞ n→0 N n N β2 1 + log 2 . = lim g0 (n, β) = n→0 n 4

−βf ≡ lim

N →∞

(8.9)

Comparing this result with the exact free-energy density (eqn (5.15)),√we see that the RS result is correct only for the high-temperature phase β < βc = 2 log 2. It misses the phase transition. Within the RS framework, there is no way to get the correct solution for β > βc . 8.1.2

One-step replica symmetry breaking

For β > βc , the sum (8.7) is dominated by matrices Q which are not replica symmetric. The problem is to find these new saddle points, and they must make sense in the n → 0 limit. In order to improve on the RS result, one can the subspace of matrices to be optimized over (i.e. weaken the requirement of replica symmetry). The replica-symmetry-breaking (RSB) scheme, initially proposed by Parisi in the more complicated case of spin glass mean-field theory, prescribes a recursive procedure for defining larger and larger spaces of matrices Q where one searches for saddle points. The first step of this procedure is called one-step replica symmetry breaking (1RSB). In order to describe it, let us suppose that n is a multiple of x, divide the n replicas into n/x groups of x elements each, and set Qaa = 1 , Qab = q1 Qab = q0

if a and b are in the same group, if a and b are in different groups.

(8.10)

Since, in the case of the REM, the matrix elements are in {0, 1}, this Ansatz is distinct from the RS Ansatz only if q1 = 1 and q0 = 0. This corresponds, after a relabeling of the replica indices, to i1 = · · · = ix , ix+1 = · · · = i2x , etc. The number of choices of (i1 , . . . in ) which satisfy these constraints is NN (Q) = 2N (2N − 1) · · · (2N − n/x + 1), and therefore we get s(Q) = (n/x) log 2. The rate function in eqn (8.7) is given by g(QRSB ) = gRSB (β, n, x):


½

5

gRSB (β, n, x)/n

4 β=3

3 βc

2

β = 0.75

1 xs (3)

0 0

0.5

xs (βc )

1

xs (0.75)

1.5

2

2.5

3

x

Fig. 8.2 The rate function g(Q) (see eqn (8.7)), evaluated at the one-step replica-symmetry-breaking point, as a function of the replica-symmetry-breaking parameter x.

gRSB (β, n, x) =

n β2 nx + log 2 . 4 x

(8.11)

Following the discussion in Section 8.1.1, we should minimize gRSB (β, n, x) with respect to x, and then take the n → 0 limit. Notice that eqn (8.11) can be interpreted as an analytic function both of n and of x = 0. We shall therefore forget hereafter that n and x are integers with n a multiple of x. The derivative ∂gRSB (β, n, x)/∂x vanishes if x = xs (β), where √ βc 2 log 2 = . (8.12) xs (β) ≡ β β Substituting this into eqn (8.11), and assuming again that we can exchange the order of the limits n → 0 and N → ∞, we get 1 min gRSB (β, n, x) = β log 2 , (8.13) x n→0 n √ which is the correct result for β > βc : f = − log 2. In fact, we can recover the correct free energy of the REM over the whole temperature range if we accept that the inequality 1 ≤ x ≤ n, valid for n, x integers, becomes n = 0 ≤ x ≤ 1 in the limit n → 0 (we shall see later on some other arguments supporting this prescription). If the minimization is constrained to x ∈ [0, 1], we get a fully consistent answer: x = βc /β is the correct saddle point in the phase β > βc , while for β < βc the parameter x remains at the value x = 1. In Fig. 8.2, we sketch the function gRSB (β, n, x)/n for a few values of the temperature β. −βf = lim

8.1.3

Comments on the replica solution

One might think that the replica method is just a fancy way of reconstructing a probability distribution from its integer moments. We know how to compute the integer moments of the partition function E Z n , and we would like to infer the full distribution

½


of Z and, in particular, the value of E log Z. This is a standard topic in probability theory. It turns out that the probability distribution can be reconstructed if its integer moments do not grow too fast as n → ∞. A typical result is the following. Theorem 8.2. (Carleman) Let X be a real random variable with moments mn = E X n such that ∞

−1/2n

m2n

= ∞.

(8.14)

n=1

Then any variable with the same moments is distributed identically to X. For instance, if the moments do not grow faster than exponentially, i.e. E X n ∼ eαn , a knowledge of them completely determines the distribution of X. Let us try to apply the above result to the REM. The replica-symmetric calculation of Section 8.1.1 is easily turned into a lower bound: E Z n ≥ eng(QRS,0 ) ≥ eN β

2

n2 /4

.

(8.15)

Therefore the sum in eqn (8.14) converges, and the distribution of Z is not necessarily fixed by its integer moments.

Exercise 8.2 Assume that Z = e−F , where F is a Gaussian random variable, with probability density 2 1 p(F ) = √ e−F /2 . 2π

(8.16)

Compute the integer moments of Z. Do they satisfy the hypothesis of Carleman’s theorem? Show that the moments are unchanged if p(F ) is replaced by the density pa (F ) = p(F )[1 + a sin(2πF )], with |a| < 1 (from Feller (1968)).

In our replica approach, there exist several possible analytic continuations to noninteger n’s, and the whole issue is to find the correct one. Parisi’s Ansatz (and its generalization to higher-order RSB that we shall discuss below) gives a well-defined class of analytic continuations, which turns out to be the correct one in many different problems. The suspicious reader will notice that the moments of the REM partition function would not grow so rapidly if the energy levels had a distribution with bounded support. If for instance, we considered the Ei to be Gaussian random variables truncated to Ei ∈ [−Emax , Emax ], the partition function would be bounded from above by the n , and the whole constant Zmax = 2N eβEmax . Consequently, we would have E Z n ≤ Zmax distribution of Z could be recovered from its integer moments. In order to achieve such a goal, we would, however, need to know exactly all the moments 1 ≤ n < ∞ at fixed N (the system size). What we are instead able to compute, in general, is the large-N behaviour at any fixed n. In most cases, this information is insufficient to ensure a unique continuation to n → 0.


½

In fact, one can think of the replica method as a procedure for computing the quantity 1 log E Z n , N →∞ N

(8.17)

ψ(n) = lim

whenever the limit exists. In the frequent case where f = −(1/βN ) log Z satisfies a . large-deviation principle of the form PN (f ) = exp[−N I(f )], we have . . (8.18) E Z n = exp[−N I(f ) − N βnf ] df = exp{−N inf[I(f ) + βnf ]} . Therefore ψ(n) = − inf[I(f ) + βnf ]. In turn, the large-deviation properties of fN can be inferred from ψ(n) through the G¨ artner–Ellis theorem (Theorem 4.10). The typical value of the free-energy density is given by the location of the absolute minimum of I(f ). In order to compute it, one must in general use values of n which go to 0, and one cannot infer it from the integer values of n. 8.1.4

Condensation

As we discussed in Chapter 5, the appearance of a low-temperature ‘glass’ phase is associated with a condensation of the probability measure onto few configurations. We described this phenomenon quantitatively by the participation ratio Y . For the REM we obtained limN →∞ E Y = 1 − βc /β for any β > βc (see Proposition 5.3). Let us see how this result can be recovered in just a few lines from a replica computation. 2N The participation ratio is defined by Y = j=1 µ(j)2 , where µ(j) = e−βEj /Z is the Boltzmann probability of the j-th energy level. Therefore, ⎤ ⎡ 2N [definition of Y ] e−2βEi ⎦ E Y = lim E ⎣Z n−2 n→0

⎡ = lim E ⎣ n→0

= lim E n→0

i=1

e−β(Ei1 +···+Ein−2

i1 ...in−2

⎤

N

2 ) i=1

e−2βEi ⎦

[assume n ∈ N]

e−β(Ei1 +···+Ein ) I(in−1 = in )

i1 ...in

1 E = lim n→0 n(n − 1) a =b

1 n→0 n(n − 1)

= lim

a =b

E

a =b

e

I(ia = ib )

i1 ...in

E

[symmetrize]

e

i1 ...in

1 Qab n , n→0 n(n − 1)

= lim

−β(Ei1 +···+Ein )

−β(Ei1 +···+Ein )

e

I(ia = ib )

−β(Ei1 +···+Ein )

[denominator → 1]

i1 ...in

(8.19)

½


where the sums over the replica indices a, b run over a, b ∈ {1, . . . , n}, while the configuration indices ia are summed over {1, . . . , 2N }. In the last step, we have introduced the notation P (N β 2 /4) a,b Qab Q f (Q) NN (Q)e P , (8.20) f (Q)n ≡ (N β 2 /4) a,b Qab Q NN (Q)e and noticed that the sum over i1 , . . . , in can be split into a sum over the overlap matrices Q and a sum over the n-tuples i1 . . . in that have an overlap matrix Q. Note that · n can be interpreted as an expectation in the ‘replicated system’. . In the large-N limit, NN (Q) = eN s(Q) , and the expectation value in eqn (8.20) is given by a dominant2 (saddle point) term: f (Q)n f (Q∗ ). As argued in the previous sections, in the low-temperature phase β > βc , the saddle point matrix is given by the 1RSB expression (8.10): 1 Q1RSB ab n→0 n(n − 1)

E Y = lim

[saddle point]

a =b

1 n[(n − x)q0 + (x − 1)q1 ] n(n − 1) βc . = 1−x=1− β

= lim

n→0

[eqn (8.10)] [q0 = 0, q1 = 1]

(8.21)

This is exactly the result that we found in Proposition 5.3, using a direct combinatorial approach. It also confirms that the 1RSB Ansatz (8.10) makes sense only if 0 ≤ x ≤ 1 (the participation ratio Y is positive by definition). Compared with the computation in Section 5.3, the simplicity of the replica derivation is striking. At first sight, the manipulations in eqn (8.19) seem to require new assumptions with respect to the free-energy computation in the previous sections. Replicas are introduced in order to write the factor Z −2 in the participation ratio as the analytic continuation of a positive power Z n−2 . It turns out that this calculation is in fact equivalent to the one in eqn (8.2). This follows from the basic observation that expectation values can be obtained as derivatives of log Z with respect to some parameters.

Exercise 8.3 Using the replica method, show that, for T < Tc , 0 E@

1

N

2 X j=1

rA

µ(j)

=

Γ(r − x) (r − 1 − x)(r − 2 − x) . . . (1 − x) = , Γ(r)Γ(1 − x) (r − 1)(r − 2) . . . (1)

(8.22)

where Γ(x) denotes Euler’s gamma function.

2 If the dominant term corresponds to a non-replica-symmetric matrix Q∗ , all the terms obtained by permuting the replica indices contribute with an equal weight. Because of this fact, it is a good idea to compute averages of symmetric functions f (Q) = f (Qπ ). This is what we have done in eqn (8.19).

The fully connected p-spin glass model

½

Exercise 8.4 Using the replica method, show that, for T < Tc , ` ´ 3 − 5x + 2x2 E Y2 = . 3

8.2

(8.23)

The fully connected p-spin glass model

The replica method provides a compact and efficient way to compute –in a non-rigorous way– the free-energy density of the REM. The result proves to be exact, once replica symmetry breaking is used in the low-temperature phase. However, its power can be better appreciated on more complicated problems which cannot be solved by direct combinatorial approaches. In this section we shall apply the replica method to the ‘p-spin glass’ model. This model was invented in the course of the theoretical study of spin glasses. Its distinguishing feature is interactions which involve groups of p spins, with p ≥ 2. It generalizes ordinary spin glass models (see Section 2.6) in which interactions involve pairs of spins (i.e. p = 2). This provides an additional degree of freedom, the value of p, and different physical scenarios appear depending on whether p = 2 or p ≥ 3. Moreover, some pleasing simplifications show up for large p. In the p-spin model, one considers the space of 2N configurations of N Ising spins. The energy of a configuration σ = {σ1 , . . . , σN } is defined as Ji1 ...ip σi1 · · · σip , (8.24) E(σ) = − i1 αUB,1 (K), where

αUB,1 (K) ≡ −

log 2 , log(1 − 2−K )

(10.13)

E Z(F ) is exponentially small at large N . Equation (10.11) implies that the probability of a formula being SAT also vanishes at large N for such an α.

(N )

Theorem 10.5 If α > αUB,1 (K), then limN →∞ P{F is SAT} = 0, whence αs αUB,1 (K).

(K) ≤

One should not expect this bound to be tight. The reason is that, in the SAT phase, Z(F ) takes exponentially large values, and its fluctuations tend to be exponential in the number of variables.

Example 10.6 As a simple illustration, consider a toy example: the random 1-SAT ensemble SATN (1, α). A formula is generated by including each of the 2N literals as a clause independently with probability α/2 (we assume that α ≤ 2). In order for the formula to be SAT, for each of the N variables, at most one of the two corresponding literals must be included. We therefore have PN (K = 1, α) = (1 − α2 /4)N .

(10.14)

In other words, the probability for a random formula to be SAT goes exponentially fast to 0 for any α > 0: αs (K = 1) = 0. On the other hand, the upper bound deduced from EZ(F ) is αUB,1 (K) = 1. This is due to large fluctuations in the number of SAT assignments Z, as we shall see in the next exercise.

¾½¾

Satisfiability

Exercise 10.8 Consider the distribution of Z(F ) in the random 1-SAT ensemble. (a) Show that n

P {Z(F ) = 2 } =

N n

!

“

1−

α ”iN −n α ”2n h “ α 1− , 2 4

(10.15)

for any n ≥ 0. [Hint: If F is SAT, then Z(F ) = 2n , where n is the number of variables which do not appear in any clause.] (b) From this expression, deduce the large-deviation principle n o . P Z(F ) = 2N ν = exp{−N Iα (ν)} ,

(10.16)

where Iα (ν) ≡ −H(ν) − 2ν log(1 − α/2) − (1 − ν) log(α(1 − α/4)) .

(10.17)

What is the most probable value of ν? (c) Show that n o . E Z(F ) = exp N max[−Iα (ν) + ν log 2] . ν

(10.18)

What is the value of ν where the maximum is achieved, ν ∗ ? Show that Iα (ν ∗ ) > 0: ∗ . the probability of having Z(F ) = 2N ν is exponentially small, and therefore E Z(F ) is dominated by rare events.

Exercise 10.9 Repeat the derivation of Theorem 10.5 for the ensemble SATN (K, α) (i.e. compute E Z(F ) for this ensemble and find for what values of α this expectation is exponentially small). Show that the upper bound obtained in this case is α = 2K log 2. This is worse than the previous upper bound αUB,1 (K), although one would expect the threshold to be the same. Why? [Hint: The number of clauses M in a SATN (K, α) formula has a binomial distribution with parameters N and α. What values of M provide the dominant contribution to E Z(F )?]

In order to improve upon Theorem 10.5 using the first-moment method, one needs a better (but still simple) choice of the function U (F ). A possible strategy consists in defining some small subclass of ‘special’ SAT assignments, such that if a SAT assignment exists, then a special SAT assignment exists too. If the subclass is small enough, one can hope to reduce the fluctuations in U (F ) and sharpen the bound. One choice of such a subclass is ‘locally maximal’ SAT assignments. Given a formula F , an assignment x for this formula is said to be a locally maximal SAT assignment if

The phase transition in random

K (≥ 3)-SAT ¾½¿

and only if (1) it is a SAT assignment, and (2) for any i such that xi to 0, the assignment obtained by flipping it to xi = 1 is UNSAT. We define U (F ) as the number of locally maximal SAT assignments and apply the first-moment method to this function. This gives the following result. Theorem 10.7 For any K ≥ 2, let αUB,2 (K) be the unique positive solution of the equation Kα −K = 0. (10.19) α log(1 − 2 ) + log 2 − exp − K 2 −1 (N )

Then αs

(K) ≤ αUB,2 (K) for large enough N .

The proof is left as the following exercise.

Exercise 10.10 Consider an assignment x where exactly L variables are set to 0, the remaining N − L variables being set to 1. Without loss of generality, assume x1 , . . . , xL to be the variables set to zero. (a) Let p be the probability that a clause constrains the variable x1 , given that the clause is satisfied by the assignment x (By a clause that constrains x1 , we` mean the` clause ´ that ´ N −1 N −1 [(2K − 1) K ] . becomes unsatisfied if x1 is flipped from 0 to 1.) Show that p = K−1 (b) Show that the probability that the variable x1 is constrained by at least one of the M clauses, given that all these clauses are satisfied by the assignment x, is equal to q = 1 − (1 − p)M . (c) Let Ci be the event that xi is constrained by at least one of the M clauses. If C1 , . . . , CL were independent events, under the condition that x satisfies F , the probability that x1 , . . . xL are constrained would be equal to q L . Of course, C1 , . . . , CL are not independent. Find a heuristic argument to show that they are anti-correlated and that their joint probability is at most q L (consider, for instance the case L = 2). PN ` N ´ L (d) Assume the claim in (c) to be true. Show that E [U (F )] ≤ (1 − 2−K )M L=0 L q = (1 − 2−K )M [1 + q]N and finish the proof by working out the large-N asymptotics of this formula (with α = M/N fixed).

In Table 10.1, we report the numerical values of the upper bounds αUB,1 (K) and αUB,2 (K) for a few values of K. These results can be slightly improved upon by pursuing the same strategy further. For instance, one may strengthen the condition of maximality to flipping two or more variables. However, the quantitative improvement in the bound is rather small. 10.5.3

Lower bounds (N )

Two main strategies have been used to derive lower bounds on αc (K) in the largeN limit. In both cases, one takes advantage of Theorem 10.4: in order to show that (N ) αc (K) ≥ α∗ , it is sufficient to prove that a random SATN (K, M ) formula, with M = αN , is SAT with non-vanishing probability in the N → ∞ limit. The first approach consists in analysing explicit heuristic algorithms for finding SAT assignments. The idea is to prove that a particular algorithm finds a SAT assignment with positive probability as N → ∞ when α is smaller than some value.

Satisfiability

One of the simplest such bounds is obtained by considering unit clause propagation. Whenever there exists a unit clause, we assign the variables appearing in one such clause in such a way to satisfy it, and proceed recursively. Otherwise, we choose a variable uniformly at random from those which have not yet been fixed and assign it to 0 or 1 with probability 1/2. The algorithm halts if it finds a contradiction (i.e. a pair of opposite unit clauses) or if all the variables have been assigned. In the latter case, the assignment produced by the algorithm satisfies the formula. This algorithm is then applied to a random K-SAT formula with clause density α. It can be shown that a SAT assignment will be found with positive probability for α (N ) small enough: this gives the lower bound αc (K) ≥ ((K − 1)/(K − 2))K−2 2K−1 /K in the N → ∞ limit. In the exercise below, we give the main steps of the reasoning for the case K = 3; we refer to the literature for more detailed proofs.

Exercise 10.11 After T iterations, the formula will contain 3-clauses, as well as 2-clauses and 1-clauses. Denote by Cs (T ) the set of s-clauses, s = 1, 2, 3, and denote its size by Cs (T ) ≡ |Cs (T )|. Let V(T ) be the set of variables which have not yet been fixed, and let L(T ) be the set of literals on the variables of V(T ) (obviously, we have |L(T )| = 2|V(T )| = 2(N −T )). Finally, if a contradiction is encountered after Thalt steps, we adopt the convention that the formula remains unchanged for all T ∈ {Thalt , . . . , N }. (a) Show that, for any T ∈ {1, . . . , N }, each clause in Cs (T ) is uniformly distributed among the s-clauses over the literals in L(T ). (b) Show that the expected changes in the numbere of 3- and 2-clauses are given by E [C3`(T + 1) −Ć3 (T )] = −3C3 (T )/(N − T ) and E [C2 (T + 1) − C2 (T )] = 3C3 (T )/ 2(N − T ) − 2C2 (T )/(N − T ). (c) Show that, conditional on C1 (T ), C2 (T ), and C3 (T ), the change in the number of 1` d clauses is distributed as follows: C1 (T +1)−C1 (T ) = −I(C1 (T ) > 0)+B C2 (T ), 1/(N − ´ T ) . (We recall that B(n, p) denotes a binomial random variable with parameters n and p (see Appendix A)). (d) It can be shown that, as N → ∞ at fixed t = T /N , the variables Cs (T )/N for s ∈ {2, 3} concentrate around their expectation values, and these converge to smooth functions cs (t). Argue that these functions must solve`the ordinary differential equa´ tions (dc3 /dt) = −3c3 (t)/(1−t) and (dc2 /dt) = 3c3 (t)/ 2(1−t) −2c2 (t)/(1−t). Check that the solutions of these equations are c3 (t) = α(1 − t)3 and c2 (t) = (3α/2)t(1 − t)2 . (e) Show that the number of unit clauses follows a Markov process described by C1 (0) = 0, d

C1 (T +1)−C1 (T ) = −I(C1 (T ) > 0)+η(T ), where η(T ) is a Poisson-distributed random variable with mean c2 (t)/(1−t), and where t = T /N . Given C1 and a time T , show that the probability that there is no contradiction generated by the unit clause algorithm Q up to time T is Tτ=1 (1 − 1/(2(N − τ )))[C1 (τ )−1]I(C1 (τ )≥1) . (f ) Let ρ(T ) be the probability that there is no contradiction up to time T . Consider P (1−) T = N (1−); show that ρ(N (1−)) ≥ (1−1/(2N ))AN +B P( N C1 (τ ) ≤ AN +B). τ =1 Assume that α is such that, ∀t ∈ [0, 1 − ], c2 (t)/(1 − t) < 1. Show that there exist A, B P (1−) C1 (τ ) ≤ AN + B) is finite. Deduce that in the large-N such that limN →∞ P( N τ =1 limit, there is a finite probability that, at time N (1 − ), the unit clause algorithm has not produced any contradiction so far, and C1 (N (1 − )) = 0.

The phase transition in random

K (≥ 3)-SAT

(g) Conditionnal on the fact that the algorithm has not produced any contradiction and C1 (N (1 − )) = 0, consider the residual formula at time T = N (1 − ). Transform each 3-clause into a 2-clause by removing a uniformly random variable from it. Show that one obtains, for small enough, a random 2-SAT problem with a small clause density ≤ 32 /2, so that this is a satisfiable instance. (h) Deduce that, for α < 8/3, the unit clause propagation algorithm finds a solution with a finite probability

More refined heuristics have been analysed using this method and lead to better (N ) lower bounds on αc (K). We shall not elaborate on this approach here, but rather present a second strategy, based on a structural analysis of the problem. The idea is to use the second-moment method. More precisely, we consider a function U (F ) of the SAT formula F , such that U (F ) = 0 whenever F is UNSAT and U (F ) > 0 otherwise. We then make use of the following inequality: [E U (F )]2 . E[U (F )2 ]

P{F is SAT} = P{U (F ) > 0} ≥

(10.20)

The present strategy is more delicate to implement than the first-moment method, (N ) which we used in Section 10.5.2 to derive upper bounds on αc (K). For instance, the simple choice U (F ) = Z(F ) does not give any result: it turns out that the ratio [E Z(F )]2 /E[Z(F )2 ] is exponentially small in N for any non-vanishing value of α, so that the inequality (10.20) is useless. Again, one needs to find a function U (F ) whose fluctuations are smaller than those of the number Z(F ) of SAT assignments. More precisely, one needs the ratio [E U (F )]2 /E[U (F )2 ] to be non-vanishing in the N → ∞ limit. One successful idea uses a weighted sum of SAT assignments, U (F ) =

M

W (x, a) .

(10.21)

x a=1

Here the sum is over all the 2N assignments, and W (x, a) is a weight associated with clause a. This weight must be such that W (x, a) = 0 when the assignment x does not satisfy clause a, and W (x, a) > 0 otherwise. Let us choose a weight which depends on the number r(x, a) of variables which satisfy clause a in the assignment x: ϕ(r(x, a)) if r(x, a) ≥ 1, (10.22) W (x, a) = 0 otherwise. It is then easy to compute the first two moments of U (F ): E U (F ) = 2

N

−K

2

M

K K r=1

r

ϕ(r)

N ( ' N M [gϕ (N, L)] . E U (F )2 = 2N L L=0

,

(10.23)

(10.24)

Satisfiability

Here gϕ (N, L) is the expectation value of the product W (x, a)W (y, a) when a clause a is chosen uniformly at random, given that x and y are two assignments of N variables which agree on exactly L of them. In order to compute gϕ (N, L), it is convenient to introduce two binary vectors u, v ∈ {0, 1}K . They encode the following information. Consider a clause a, and fix us = 1 if, in the assignment x, the s-th variable of clause a satisfies the clause, and fix us = 0 otherwise. The components of v are defined similarly, but with the assignment y. Furthermore, we denote the Hamming distance between these vectors by d(u, v ), and their Hamming weights (number of non-zero components) by w(u), w(v ). Then gϕ (N, L) = 2−K

ϕ (w(u)) ϕ (w(v ))

u, v

L N

d(u,v) 1−

L N

K−d(u,v) .

(10.25)

runs over K-component vectors u, v with at least one non-zero Here the sum component. A particularly simple choice is ϕ(r) = λr . Denoting L/N by z, we find that %' & (K K (10.26) gw (N, L) = 2−K (λ2 + 1)z + 2λ(1 − z) − 2 [z + λ(1 − z)] + z k . The first two moments can be evaluated from Eqs. (10.23) and (10.24): . E U (F ) = exp{N h1 (λ, α)} ,

. E [U (F )2 ] = exp{N max h2 (λ, α, z)} , z

(10.27)

where the maximum is taken over z ∈ [0, 1], and ' ( h1 (λ, α) ≡ log 2 − αK log 2 + α log (1 + λ)K − 1 , (10.28) h2 (λ, α, z) ≡ log 2 − z log z − (1 − z) log(1 − z) − αK log 2 (10.29) & %' ( K K +α log (λ2 + 1)z + 2λ(1 − z) − 2 [z + λ(1 − z)] + z k . Evaluating the above expression for z = 1/2, one finds h2 (λ, α, 1/2) = 2h1 (λ, α). The interpretation is as follows. Setting z = 1/2 amounts to assuming that the second moment of U (F ) is dominated by completely uncorrelated assignments (two uniformly random assignments agree on about half of the variables). This results in the factorization of the expectation E [U (F )2 ] ≈ [E U (F )]2 . Two cases are possible: either the maximum of h2 (λ, α, z) over z ∈ [0, 1] is achieved only at z = 1/2, or it is not. (i) In the latter case, maxz h2 (λ, α, z) > 2h1 (λ, α) strictly, and therefore the ratio [E U (F )]2 /E[U (F )2 ] is exponentially small in N , and the second-moment inequality (10.20) is useless. (ii) If, on the other hand, the maximum of h2 (λ, α, z) is achieved only at z = 1/2, then the ratio [E U (F )]2 /E[U (F )2 ] is 1 to the leading exponential order. It is not difficult to work out the precise asymptotic behaviour (i.e. to compute the prefactor of the exponential). One finds that [E U (F )]2 /E[U (F )2 ] remains finite (N ) when N → ∞. As a consequence, α ≤ αc (K) for N large enough.

Notes

Table 10.1 Satisfiability thresholds for random K-SAT. We report the lower bound from Theorem 10.8 and the upper bounds obtained from Eqs. (10.13) and (10.19).

K αLB (K) αUB,1 (K) αUB,2 (K)

3 2.548 5.191 4.666

4 7.314 10.74 10.22

5 17.62 21.83 21.32

6 39.03 44.01 43.51

7 82.63 88.38 87.87

8 170.6 177.1 176.6

9 347.4 354.5 354.0

10 701.5 709.4 708.9

A necessary condition for the second case to occur is that z = 1/2 is a local maximum of h2 (λ, α, z). This implies that λ must be the (unique) strictly positive root of (1 + λ)K−1 =

1 . 1−λ

(10.30)

We have thus proved the following result. Theorem 10.8 Let λ be the positive root of Eq. (10.30), and let the function h2 ( · ) be defined as in Eq. (10.29). Assume that h2 (λ, α, z) achieves its maximum, as a function of z ∈ [0, 1] only at z = 1/2. Then a random SATN (K, α) is SAT with probability approaching one as N → ∞. Let αLB (K) be the largest value of α such that the hypotheses of this theorem are satisfied. The Theorem implies an explicit lower bound on the satisfiability threshold: (N ) αs (K) ≥ αLB (K) in the N → ∞ limit. Table 10.1 summarizes some of the values of the upper and lower bounds found in this section for a few values of K. In the large-K limit, the following asymptotic behaviours can be shown to hold: αLB (K) = 2K log 2 − 2(K + 1) log 2 − 1 + o(1) , 1 αUB,1 (K) = 2K log 2 − log 2 + o(1) . 2

(10.31) (10.32)

In other words, the simple methods set out in this chapter allow one to determine the satisfiability threshold with a relative error that behaves as 2−K in the large K-limit. More sophisticated tools, to be discussed in the following chapters, are necessary for obtaining sharp results at finite K.

Exercise 10.12 [Research problem] Show that the choice of weights ϕ(r) = λr is optimal: all other choices for ϕ(r) give a worse lower bound. What strategy could be followed to improve the bound further?

Notes The review paper by Gu et al. (1996) is a rather comprehensive source of information on the algorithmic aspects of satisfiability. The reader interested in applications will also find a detailed and referenced list of applications there.

Satisfiability

Davis and Putnam first studied an algorithm for satisfiability in Davis and Putnam (1960). This was based on a systematic application of the resolution rule. The backtracking algorithm discussed in this chapter was introduced in Davis et al. (1962). Other ensembles of random CNF formulae have been studied, but it turns out that it is not so easy to find hard formulae. For instance, take N variables, and generate M clauses independently according to the following rule. In a clause a, each of the variables appears as either xi or xi with the same probability p ≤ 1/2 in both cases, and does not appear with probability 1 − 2p. The reader is invited to study this ensemble; an introduction and a guide to the corresponding literature can be found in Franco (2000). Another useful ensemble is the ‘2 + p’ SAT problem, which interpolates between K = 2 and K = 3 by picking pM 3-clauses and (1 − p)M 2-clauses; see Monasson and Zecchina (1998), and Monasson et al. (1999). The polynomial nature of 2-SAT is discussed by Cook (1971). MAX-2SAT was shown to be NP-complete by Garey et al. (1976). Sch¨ oning’s algorithm was introduced in Sch¨ oning (1999) and discussed further in Sch¨ oning (2002). More general random-walk strategies for SAT are treated in Papadimitriou (1991), Selman and Kautz (1993), and Selman et al. (1994). The threshold αs = 1 for random 2-SAT was proved by Chvátal and Reed (1992), Goerdt (1996), and de la Vega (1992); see also de la Vega (2001). The scaling behaviour near to the threshold has been analysed using graph theoretical methods by Chayes et al. (2001). The numerical identification of the phase transition in random 3-SAT, together with the observation that difficult formulae are found near the phase transition, were done by Kirkpatrick and Selman (1994) and Selman and Kirkpatrick (1996). See also Selman et al. 1996). Friedgut’s theorem was proved by Friedgut (1999). Upper bounds on the threshold were discussed by Dubois and Boufkhad (1997) and Kirousis et al. (1998). Lower bounds for the threshold in random K-SAT, based on the analysis of search algorithms, were pioneered by Chao and Franco. The paper by Chao and Franco (1986) corresponds to Exercise 10.11, and a generalization can be found in Chao and Franco (1990). A review of this type of methods is provided by Achlioptas (2001). Backtracking algorithms were first analysed using an heuristic approach by Cocco and Monasson (2001a). Cocco et al. (2006) gives a survey of the analysis of algorithms based on statistical-physics methods. The idea of deriving a lower bound by the weighted second-moment method was introduced by Achlioptas and Moore (2007). The lower bound which we discuss here is derived by Achlioptas and Peres (2004); this paper also solves the first question of Exercise 10.12. A simple introduction to the use of the second moment method in various constraint satisfaction problems is provided by Achlioptas et al. (2005); see also Gomes and Selman (2005).

11 Low-density parity-check codes Low-density parity-check (LDPC) error-correcting codes were introduced in 1963 by Robert Gallager in his PhD thesis. The basic motivation came from the observation that random linear codes (see Section 6.6), had excellent theoretical performance (in terms of the number of channel errors they could correct) but were unpractical. In particular, no efficient algorithm was known for decoding. In retrospect, this is not surprising, since it was later shown that decoding for linear codes is an NP-hard problem. The idea was then to restrict the random linear code ensemble, introducing some structure that could be exploited for more efficient decoding. Of course, the risk is that such a restriction of the ensemble might spoil its performance. Gallager’s proposal was simple and successful (but ahead of its time): LDPC codes are among the most efficient codes around. In this chapter, we introduce one of the most important families of LDPC ensembles and derive some of their basic properties. As for any code, one can take two quite different points of view. The first is to study the performance of the code with respect to an appropriate metric, under optimal decoding, in which no constraint is imposed on the computational complexity of the decoding procedure. For instance, decoding by a scan of the whole, exponentially large codebook is allowed. The second approach consists in analysing the performance of the code under some specific, efficient decoding algorithm. Depending on the specific application, one may be interested in algorithms of polynomial complexity, or even require the complexity to be linear in the block length. Here we shall focus on performance under optimal decoding. We shall derive rigorous bounds, showing that appropriately chosen LDPC ensembles allow one to communicate reliably at rates close to Shannon’s capacity. However, the main interest of LDPC codes is that they can be decoded efficiently, and we shall discuss a simple example of a decoding algorithm with linear time complexity. A more extensive study of LDPC codes under practical decoding algorithms is deferred to Chapter 15. After defining LDPC codes and LDPC code ensembles in Section 11.1, we discuss some geometric properties of their codebooks in Section 11.2. In Section 11.3 we use these properties to derive a lower bound on the threshold for reliable communication. An upper bound follows from information-theoretic considerations. Section 11.4 discusses a simple decoding algorithm, which is shown to correct a finite fraction of errors.

¾¾¼

Low-density parity-check codes

11.1 11.1.1

Definitions Linear algebra with binary variables

Recall that a code is characterized by its codebook C, which is a subset of {0, 1}N . LDPC codes are linear codes, which means that the codebook is a linear subspace of {0, 1}N . In practice, such a subspace can be specified through an M × N matrix H, with binary entries Hij ∈ {0, 1}, where M < N . The codebook is defined as the kernel of H: C = { x ∈ {0, 1}N : Hx = 0 } .

(11.1)

Here and in all of this chapter, the multiplications and sums involved in Hx are understood as being computed modulo 2. The matrix H is called the parity check matrix of the code. The size of the codebook is |C| = 2N −rank(H) , where rank(H) denotes the rank of the matrix H (the number of linearly independent rows). As rank(H) ≤ M , we have |C| ≥ 2N −M . With a slight modification with respect to the notation of Chapter 1, we let L ≡ N − M . The rate R of the code therefore satisfies R ≥ L/N , equality being obtained when all of the rows of H are linearly independent. Given such a code, encoding can always be implemented as a linear operation. There exists an N × L binary matrix G, called the generator matrix, such that the codebook is the image of G: C = {x = Gz , where z ∈ {0, 1}L }. Encoding is therefore realized as the mapping z → x = Gz. (Note that the product H G is an M × L ‘null’ matrix with all entries equal to zero.) 11.1.2

Factor graph

In Section 9.1.2, we described the factor graph associated with one particular linear code (the Hamming code of eqn (9.8)). The recipe to build the factor graph, knowing H, is as follows. Let us denote by ia1 , . . . , iak(a) ∈ {1, . . . , N } the column indices such that H has a matrix element equal to 1 at row a and column iaj . The a-th coordinate of the vector Hx is then equal to xia1 ⊕ · · · ⊕ xiak(a) . Let µ0,H (x) be the uniform distribution over all codewords of the code H (hereafter, we shall often identify a code with its parity check matrix). It is given by µ0,H (x) =

M 1 I(xia1 ⊕ · · · ⊕ xiak = 0) . Z a=1

(11.2)

Therefore, the factor graph associated with µ0,H (x) (or with H) includes N variable nodes, one for each column of H, and M function nodes (also called, in this context, check nodes), one for each row. A factor node and a variable node are joined by an edge if the corresponding entry in H is non-vanishing. Clearly, this procedure can be inverted: with any factor graph with N variable nodes and M function nodes, we can associate an M × N binary matrix H, the adjacency matrix of the graph, whose non-zero entries correspond to the edges of the graph. 11.1.3

Ensembles with given degree profiles

In Chapter 9, we introduced the ensembles of factor ∞have N ∞graphs DN (Λ, P ). These variable nodes, and the two polynomials Λ(x) = n=0 Λn xn and P (x) = n=0 Pn xn

Definitions

¾¾½

PERMUTATION

Fig. 11.1 Construction of a graph/code with given degree profiles. Here, a graph with N = 7, M = 4, Λ(x) = 17 (2x + 2x2 + 3x3 ), and P (x) = 14 (3x3 + x6 ) is shown. The ‘sockets’ from the variable nodes and those from the check nodes are connected through a uniformly random permutation.

define the degree profiles: Λn is the probability that a randomly chosen variable node has degree n, and Pn is the probability that a randomly chosen function node has degree n. We define LDPCN (Λ, P ) to be the ensemble of linear codes whose parity check matrix is the adjacency matrix of a random graph from the ensemble DN (Λ, P ). We shall be interested in the limit N → ∞ while keeping the degree profiles fixed. Therefore each vertex has bounded degree, and hence the parity check matrix has a ‘low density.’ In order to eliminate trivial cases, we always assume that variable nodes have a degree at least 1, and function nodes at least 2. The numbers of parity check and variable nodes satisfy the relation M = N Λ (1)/P (1). The ratio L/N = (N −M )/N = 1−Λ (1)/P (1), which is a lower bound on the actual rate R, is called the design rate Rdes of the code (or of the ensemble). The actual rate of a code from the ensemble LDPCN (Λ, P ) is of course a random variable, but we shall see below that it is, in general, sharply concentrated ‘near’ Rdes . A special case which is often considered is that of ‘regular’ graphs: all variable nodes have degree l and all function nodes have degree k (i.e. P (x) = xk and Λ(x) = xl ). The corresponding code ensemble is usually denoted simply by LDPCN (l, k), or, more synthetically, (l, k). It has a design rate Rdes = 1 − (l/k). Generating a uniformly random graph from the ensemble DN (Λ, P ) is not a trivial task. The simplest way to bypass this problem is to substitute the uniformly random ensemble by a slightly different one which has a simple algorithmic description. One can, for instance, proceed as follows. First, separate the set of variable nodes uniformly at random into subsets of sizes N Λ0 , N Λ1 , . . . , N Λlmax , and attribute zero ‘sockets’ to the nodes in the first subset, one socket to each of the nodes in the second, and so on. Analogously, separate the set of check nodes into subsets of size M P0 , M P1 , . . . , M Pkmax and attribute 0, 1, . . . , kmax sockets to the nodes in each subset. At this point the variable nodes have N Λ (1) sockets, and so have the check nodes. Draw a uniformly random permutation from N Λ (1) objects and connect the sockets on the two sides accordingly (see Fig. 11.1).

¾¾¾


Exercise 11.1 In order to sample a graph as described above, one needs two routines. The first one separates a set of N objects uniformly into subsets of prescribed sizes. The second one samples a random permutation of N Λ (1) objects. Show that both of these tasks can be accomplished with O(N ) operations (having at your disposal a random number generator).

This procedure a flaw: it may generate multiple edges joining the same pair of nodes in the graph. In order to cure this problem, we shall agree that each time n edges join any two nodes, they must be erased if n is even, and they must be replaced by a single edge if n is odd. Of course, the resulting graph does not necessarily have the prescribed degree profile (Λ, P ), and even if we condition on this to be the case, its distribution is not uniform. We shall nevertheless insist on denoting the ensemble by LDPCN (Λ, P ). The intuition is that, for large N , the degree profile will be ‘close’ to the prescribed one and the distribution will be ‘uniform enough’ for our purposes. Moreover –and this is really important– this and similar graph generation techniques are used in practice.

Exercise 11.2 This exercise aims at proving that, for large N , the degree profile produced by the explicit construction described above is close to the prescribed degree profile. (a) Let m be the number of multiple edges appearing in the graph; compute its expectation. Show that E m = O(1) as N → ∞ with Λ and P fixed. (b) Let (Λ , P ) be the degree profile produced by the above procedure. Denote by d≡

X

|Λl − Λl | +

X

l

|Pk − Pk |

(11.3)

k

the ‘distance’ between the prescribed and the actual degree profiles. Derive an upper bound on d in terms of m and show that it implies E d = O(1/N ).

11.2

The geometry of the codebook

As we saw in Section 6.2, a classical approach to the analysis of error-correcting codes is to study the geometric properties of the corresponding codebooks. An important example of such properties is the distance enumerator Nx0 (d), giving the number of codewords at a Hamming distance d from x0 . In the case of linear codes, the distance enumerator does not depend upon the reference codeword x0 (the reader is invited to prove this statement). It is therefore customary to take the all-zeros codeword as the reference, and to use the term weight enumerator: N (w) = Nx0 (d = w) is the number of codewords having a weight (the number of ones in the codeword) equal to w. In this section, we want to estimate the expected weight enumerator N (w) ≡ E N (w) for a random code in the ensemble LDPCN (Λ, P ). In the case of the random code ensemble of Section 6.2, the corresponding N (w) grows exponentially in the block


¾¾¿

length N , and that most of the codewords have a weight w = N ω growing linearly with N . We shall in fact compute the exponential growth rate φ(ω), defined by . N (w = N ω) = eN φ(ω) . (11.4) In the jargon of statistical physics, N (w) is an ‘annealed average’ and hence it may be dominated by rare instances in the ensemble. On the other hand, one expects log N (w) to be tightly concentrated around its typical value N φq (ω). The typical exponent φq (ω) can be computed through a quenched calculation, for instance considering limN →∞ N −1 E log [1 + N (w)]. Of course, φq (ω) ≤ φ(ω) because of the concavity of the logarithm. In this chapter, we shall keep to the annealed calculation, which is much easier and gives an upper bound on the quenched result φq . Let x ∈ {0, 1}N be a binary word of length N and weight w. Note that Hx = 0 mod 2 if and only if the corresponding factor graph has the following property. Consider all the w variable nodes i such that xi = 1, and colour all edges incident on these nodes in red. Colour all of the other edges blue. All of the check nodes must then have an even number of incident red edges. A little thought shows that N (w) is the number of ‘coloured’ factor graphs that have this property for some set of w variable nodes, divided by the total number of factor graphs in the ensemble. We shall compute this number first for a graph with fixed degrees, i.e. for codes in the ensemble LDPCN (l, k), and then we shall generalize to arbitrary degree profiles. 11.2.1

Weight enumerator: Regular ensembles

In the fixed-degree case we have N variable nodes of degree l, and M function nodes of degree k. We denote by F = M k = N l the total number of edges. A valid coloured graph must have E = wl red edges. It can be constructed as follows. First, choose w variable nodes, which can be done in N w ways. Assign l red sockets to each node in this set, and l blue sockets to each node outside the set. Then, for each of the M function nodes, colour an even subset of its sockets red in such a way that the total number of red sockets is E = wl. Let mr be the number of function nodes with r red sockets. non-zero only when r is even, and they are constrained by The numbers mr can be k k m = M and r r=0 r=0 rmr = lw. The number of ways one can colour the sockets of the function nodes is thus mr (e) M k C(k, M, w) = m0 , . . . , mk r m0 ,...,mk r (11.5) k k % & % & ×I mr = M I rmr = lw , r=0

(e)

r=0

where the sum means that non-zero mr ’s appear only for r even. Finally, we join the variable-node and check node sockets in such a way that colours are matched. There are (lw)!(F − lw)! such matchings out of the total number F ! corresponding to different elements in the ensemble. Putting everything together, we get the final formula, (lw)!(F − lw)! N C(k, M, w) . (11.6) N (w) = w F!

¾¾


In order to compute the function φ(ω) in eqn (11.4), we need to work out the asymptotic behaviour of this formula when N → ∞ at fixed ω = w/N . Assuming that mr = xr M = xr N l/k, we can expand the multinomial factors using Stirling’s formula. This gives l k , (11.7) −xr log xr + xr log φ(ω) = max∗ (1 − l)H(ω) + r k r {xr } ∗ where the max all choices of x0 , x2 , x4 , . . . in [0, 1], subject to the two is taken over constraints r xr = 1 and r rxr = kω. The maximization can be done by imposing these constraints via two Lagrange multipliers. We obtain xr = Cz r kr I(r even), where C and z are two constants fixed by the equations

2 , (1 + + (1 − z)k (1 + z)k−1 − (1 − z)k−1 . ω=z (1 + z)k + (1 − z)k

C=

z)k

(11.8) (11.9)

Substituting the resulting xr back into the equation (11.7) for φ, this gives, finally, φ(ω) = (1 − l)H(ω) +

(1 + z)k + (1 − z)k l log − ωl log z , k 2

(11.10)

where z is the function of ω defined in eqn (11.9). We shall see in the next sections how to use this result, but let us first explain how it can be generalized. 11.2.2

Weight enumerator: General case

. We want to compute the leading exponential behaviour N (w) = exp[N φ(ω)] of the expected weight enumerator for a general LDPCN (Λ, P ) code. The idea of the approach is the same as the one we have just used for the case of regular ensembles, but the computation becomes heavier. It is therefore useful to adopt a more powerful formalism. Altogether, this subsection is more technical than the others: the reader who is not interested in the details can skip it and go to the results. We want to build a valid coloured graph; let us denote its number of red edges (which is no longer fixed by w) by E. There are coeff[ l (1 + xy l )N Λl , xw y E ] ways of choosing the w variable nodes in such a way that their degrees add up to E.1 As before, for each of the M function nodes, we colour an even subset of its sockets red insuch a way that the total number of red sockets is E. This can be done in coeff[ k qk (z)M Pk , z E ] ways, where qk (z) ≡ 12 (1+z)k + 12 (1−z)k . The numbers of ways one can match the red sockets in the variable and function nodes is still E!(F − E)!, where F = N Λ (1) = M P (1) is the total number of edges in the graph. This gives the result 1 We

denote the coefficient of xn in the formal power series f (x) by coeff[f (x), xn ].


¾¾

1 0.5 9

0 1

6

-0.5

0.5

3 0 -1

0 -0.5

-1

Im(z)

-1

-0.5 0 Re(z)

0.5

-0.5

-1

0 Re(z)

0.5

1

Fig. 11.2 Modulus of the function z −3ξ q4 (z)3/4 for ξ = 1/3.

N (w) =

F E!(F − E)! F! E=0 k l max max l N Λl w E M Pk E . (11.11) (1 + xy ) , x y coeff qk (z) ,z × coeff l=1

k=2

In order to estimate the leading exponential behaviour of N (w) at large N , when w = N ω, we set E = F ξ = N Λ (1)ξ. The asymptotic behaviour of the coeff[. . . , . . . ] terms can be estimated using the saddle point method. Here, we sketch the idea for the second of these terms. By the Cauchy theorem, 9 k 9 k max max f (z)N dz 1 M Pk E M Pk dz ≡ , = qk (z) ,z q (z) coeff k (1)ξ+1 N Λ 2πi z 2πi z k=2 k=2 (11.12) where the integral runs over any path encircling the origin of the complex z plane in the anticlockwise direction, and f (z) ≡

k max

1 z Λ (1)ξ

qk (z)Λ (1)Pk /P

(1)

.

(11.13)

k=2

In Fig. 11.2, we plot the modulus of the function f (z) for degree distributions Λ(x) = x3 , P (x) = x4 , and ξ = 1/3. The function has two saddle points of ±z∗ , where z∗ = z∗ (ξ) ∈ R+ solves the equation f (z∗ ) = 0, namely ξ=

k max k=2

ρk z∗

(1 + z∗ )k−1 − (1 − z∗ )k−1 . (1 + z∗ )k + (1 − z∗ )k

(11.14)

Im

¾¾


Here we have used the notation ρk ≡ kPk /P (1) introduced in Section 9.5 (analogously, we shall write λl ≡ lΛl /Λ (1)). This equation generalizes (11.9). If we take the integration contour in eqn (11.12) to be the circle of radius z∗ centred at z = 0, the integral is dominated by the saddle point at z∗ (together with the symmetric point −z∗ ). We therefore get coeff

k max

M Pk

qk (z)

,z

E

k=2

# $ kmax Λ (1) . = exp N −Λ (1)ξ log z∗ + Pk log qk (z∗ ) . P (1) k=2

Proceeding analogously with the second coeff[. . . , . . . ] term in eqn (11.11), we get . N (w = N ω) = exp{N φ(ω)}. The function φ is given by # φ(ω) = sup inf ξ

x,y,z

−Λ (1)H(ξ) − ω log x − Λ (1)ξ log(yz) +

l max l=2

$

kmax Λ (1) Λl log(1 + xy ) + Pk log qk (z) P (1) l

, (11.15)

k=2

where the minimization over x, y, z is understood to be taken over the positive real axis while ξ ∈ [0, 1]. The stationarity condition with respect to variations of z is given by eqn (11.14). Stationarity with respect to ξ, x, y yields ξ=

yz , 1 + yz

ω=

l max

Λl

l=1

xy l , 1 + xy l

ξ=

l max l=1

λl

xy l , 1 + xy l

(11.16)

respectively. If we use the first of these equations to eliminate ξ, we obtain the final parametric representation (in the parameter x ∈ [0, ∞[) of φ(ω) φ(ω) = −ω log x − Λ (1) log(1 + yz) +

l max

Λl log(1 + xy l )

(11.17)

l=1

+

kmax Λ (1) Pk log qk (z) , P (1) k=2

ω=

l max l=1

l

Λl

xy . 1 + xy l

(11.18)

The two functions y = y(x) and z = z(x) are solutions of the coupled equations kmax y=

k=2 kmax k=2

ρk p− k (z) ρk p+ k (z)

lmax ,

z=

l−1 /(1 + xy l ) l=1 λl xy lmax l l=1 λl /(1 + xy )

,

(11.19)

k−1 where we have defined p± ± (1 − z)k−1 ]/[(1 + z)k + (1 − z)k ]. k (z) ≡ [(1 + z)


¾¾

Exercise 11.3 The numerical solution of eqns (11.18) and (11.19) can be somewhat tricky. Here is a simple iterative procedure which usually works reasonably well. Readers are invited to try it with their favourite degree distributions Λ, P . First, solve eqn (11.18) for x at given y ∈ [0, ∞[ and ω ∈ [0, 1], using a bisection method. Next, substitute this value of x into eqn (11.19), and write the resulting equations as y = f (z) and z = g(y, ω). Define Fω (y) ≡ f (g(y, ω)). Solve the equation y = Fω (y) by iteration of the map yn+1 = Fω (yn ) Once the fixed point y∗ has been found, the other parameters are computed as follows z∗ = g(y∗ , ω), and x∗ is the solution of eqn (11.18) for y = y∗ . Finally, x∗ , y∗ , z∗ are substituted into eqn (11.17) to obtain φ(ω).

Examples of functions φ(ω) are shown in Figs 11.3–11.5. We shall now discuss these results, paying special attention to the region of small ω. 11.2.3

Short-distance properties

In the low-noise limit, the performance of a code depends a lot on the existence of codewords at a short distance from the transmitted one. For linear codes and symmetric communication channels, we can assume without loss of generality that the all-zeros codeword has been transmitted. Here, we shall work out the short-distance (i.e. for a mall weight ω) behaviour of φ(ω) for several LDPC ensembles. These properties will be used to characterize the performance of the code in Section 11.3. As ω → 0, solving eqns (11.18) and (11.19) yields y, z → 0. By Taylor expansion of these equations, we get y ρ (1)z ,

z λlmin xy lmin −1 ,

ω Λlmin xy lmin ,

(11.20)

where we have neglected higher-order terms in y, z. At this point we must distinguish whether lmin = 1, lmin = 2, or lmin ≥ 3. √ We start with the case lmin = 1. In this case x, y, z all scale like ω, and a short computation shows that ω 1 + O(ω) . (11.21) φ(ω) = − ω log 2 Λ21 In particular, φ(ω) is strictly positive for ω sufficiently small. The expected number of codewords within a small relative Hamming distance w = N ω from a given codeword is exponential in N . Furthermore, eqn (11.21) is reminiscent of the behaviour in the absence of any parity check, where one gets φ(ω) = H(ω) −ω log ω.

Exercise 11.4 In order to check eqn (11.21), compute the weight enumerator for the regular ensemble LDPCN (l = 1, k). Note that, in this case, the weight enumerator does not depend on the realization of the code, and admits the simple representation N (w) = coeff[qk (z)N/k , z w ].

An example of a weight enumerator for an irregular code with lmin = 1 is shown in Fig. 11.3. The behaviour (11.21) is quite bad for an error-correcting code. In order

¾¾


0.06 0.04 0.02 0.00 0.00

0.7 0.6

φ(ω)

0.5

0.02

0.04

0.4 0.3 0.2 0.1 0 0

0.2

0.4

0.6

0.8

1

ω

Fig. 11.3 Logarithm of the expected weight enumerator, φ(ω), plotted versus the reduced weight ω = w/N for the ensemble LDPCN ( 14 x + 14 x2 + 12 x3 , x6 ). Inset: small-weight region. φ(ω) is positive near to the origin, and in fact its derivative diverges as ω → 0: each codeword is surrounded by a large number of very close other codewords. This makes it a bad error correcting code.

to understand why, let us for a moment forget that this result was obtained by taking ω → 0 after N → ∞, and apply it in the regime N → ∞ at w = N ω fixed. We get w/2 N N (w) ∼ . (11.22) w It turns out that this result holds not only on average but for most codes in the ensemble. In other words, even at a Hamming distance of 2 from any given codeword, there are Θ(N ) other codewords. It is intuitively clear that discriminating between two codewords at Hamming distance Θ(1), given a noisy observation, is in most cases impossible. Because of these remarks, one usually discards lmin = 1 ensembles in error correction. Consider now the case lmin = 2. From eqn (11.20), we get P (1) 2Λ2 = log [λ (0)ρ (1)] . (11.23) φ(ω) A ω , A ≡ log P (1) Λ (1) As will be apparent in Chapter 15, the combination λ (0)ρ (1) has an important concrete interpretation. The code ensemble has significantly different properties depending on the sign of A. If A > 0, the expected number of codewords within a small (but of order Θ(N )) Hamming distance from any given codeword is exponential in the block length. The situation seems similar to the lmin = 1 case. Note, however, that φ(ω) goes much more quickly to 0 as ω → 0 in the present case. Assuming again that eqn (11.23) holds beyond the asymptotic regime in which it was derived, we get N (w) ∼ eAw .

(11.24)


¾¾

0.6 0.020 0.010 0.000

0.5 0.4

φ(ω)

0.00

0.01

0.02

0.3 0.2 0.1 0 0

0.2

0.4

0.6

0.8

1

ω

Fig. 11.4 Logarithm of the expected weight enumerator for the ensemble LDPCN (2, 4). The degree profiles are Λ(x) = x2 , meaning that all variable nodes have degree 2, and P (x) = x4 , meaning that all function nodes have degree 4. Inset: small-weight region. The constant A is positive, so there exist codewords at short distances.

In other words, the number of codewords around any particular codeword is o(N ) until we reach a Hamming distance d∗ log N/A. For many purposes d∗ plays the role of an ‘effective’ minimum distance. The example of the regular code LDPCN (2, 4), for which A = log 3, is shown in Fig. 11.4. If, on the other hand A < 0, then φ(ω) < 0 in some interval ω ∈]0, ω∗ [. The firstmoment method then shows that there are no codewords of weight ‘close to’ N ω for any ω in this range. A similar conclusion is reached if lmin ≥ 3, where one finds ω lmin − 2 ω log . (11.25) φ(ω) 2 Λlmin An example of a weight enumerator exponent for a code with good short-distance properties, the code LDPCN (3, 6), is given in Fig. 11.5. This discussion can be summarized as follows. Proposition 11.1 Consider a random linear code from the ensemble LDPCN (Λ, P ) with lmin ≥ 3. Let ω∗ ∈]0, 1/2[ be the first non-trivial zero of φ(ω), and consider any interval [ω1 , ω2 ] ⊂]0, ω∗ [. With high probability, there does not exist any pair of codewords with a distance belonging to this interval. The same result holds when lmin = 2 and λ (0)ρ (1) = (P (1)/P (1))(2Λ2 /Λ (1)) < 1. Note that our study deals only with weights w = ωN which grow linearly with N . The proposition excludes the existence of codewords of arbitrarily small ω, but it does not tell us anything about possible codewords of sublinear weight, i.e. w = o(N ) (for instance, with w finite as N → ∞). It turns out that if lmin ≥ 3, the code has with high probability no such codewords, and its minimum distance is at least N ω∗ . If, on the other hand, lmin = 2, the code typically has some codewords of finite weight

¾¿¼


0.6 0.5

φ(ω)

0.4

0.005 0.000 -0.005 -0.010 0.00

0.02

0.6

0.8

0.04

0.3 0.2 0.1 0 0

0.2

0.4

1

ω

Fig. 11.5 Logarithm of the expected weight enumerator for the ensemble LDPCN (3, 6). Inset: small-weight region; φ(ω) < 0 for ω < ω∗ ≈ 0.02. There are no codewords except for the ‘all-zeros’ one in the region ω < ω∗ .

w. However (if A < 0), they can be eliminated without changing the code rate by an ‘expurgation’ procedure similar to that described in Section 6.5.1. 11.2.4

Rate

The weight enumerator can also be used to obtain a precise characterization of the rate of an LDPCN (Λ, P ) code. For ω = 1/2, x = y = z = 1 satisfy eqns (11.18) and (11.19). This gives Λ (1) 1 = 1− log 2 = Rdes log 2 . (11.26) φ ω= 2 P (1) It turns out that, in most cases of practical interest, the curve φ(ω) has its maximum at ω = 1/2 (see for instance Figs 11.3–11.5). In such cases the result (11.26) shows that the rate equals the design rate. Proposition 11.2 Let R be the rate of a code from the ensemble LDPCN (Λ, P ), let Rdes = 1 − Λ (1)/P (1) be the associated design rate, and let φ(ω) be the function defined in eqns (11.17)–(11.19). Assume that φ(ω) achieves its absolute maximum over the interval [0, 1] at ω = 1/2. Then, for any δ > 0, there exists a positive N independent constant C1 (δ) such that P{|R − Rdes | ≥ δ} ≤ C1 (δ) 2−N δ/2 .

(11.27)

Proof Since we have already established that R ≥ Rdes , we only need to prove an upper bound on R. The rate is defined as R ≡ (log2 N )/N , where N is the total number of codewords. Markov’s inequality gives: P{R ≥ Rdes + δ} = P{N ≥ 2N (Rdes +δ) } ≤ 2−N (Rdes +δ) E N .

(11.28)

LDPC codes for the binary symmetric channel

¾¿½

. The expectation of the number of codewords is E N (w) = exp{N φ(w/N )}, and there are only N + 1 possible values of the weight w; therefore, . (11.29) E N = exp{N sup φ(ω)} . ω∈[0,1]

As sup φ(ω) = φ(1/2) = Rdes log 2 by hypothesis, there exists a constant C1 (δ) such that E N ≤ C1 (δ)2N (Rdes +δ/2) for any N . Plugging this into eqn (11.28), we get P{R ≥ Rdes + δ} ≤ C1 (δ) 2−N δ/2 .

(11.30)

11.3


Our study of the weight enumerator has shown that codes from the ensemble LDPCN (Λ, P ) with lmin ≥ 3 have a good short-distance behaviour. The absence of codewords within an extensive distance N ω∗ from the transmitted codeword guarantees that any error (even one introduced by an adversarial channel) that changes a fraction of the bits smaller than ω∗ /2 can be corrected. Here we want to study the performance of these codes in correcting typical errors introduced by a given (probabilistic) channel. We shall focus on the binary symmetric channel denoted by BSC(p), which flips each bit independently with probability p < 1/2. Supposing as usual that the all-zero codeword x(0) = 0 has been transmitted, let us denote the received message by y = (y1 . . . yN ). Its components are i.i.d. random variables taking the value 0 with probability 1 − p, and the value 1 with probability p. The decoding strategy which minimizes the block error rate is word MAP (or maximum-likelihood) decoding, which outputs the codeword closest to the channel output y. As already mentioned, we shall not bother about the practical implementation of this strategy and its computational complexity. The block error probability for a code C, denoted by PB (C), is the probability that there exists a ‘wrong’ codeword, distinct from 0, whose distance from y is smaller than d(0, y). Its expectation value over the code ensemble, PB = E PB (C), is an important indicator of the performance of the ensemble. We shall show that in the large-N limit, codes with lmin ≥ 3 undergo a phase transition, separating a low-noise phase, where p < pMAP , in which limN →∞ PB is zero, from a high-noise phase, where p > pMAP , in which the limit is not zero. Although the computation of pMAP is deferred to Chapter 15, we derive rigorous bounds here which imply that appropriate LDPC codes have very good performance, close to Shannon’s information-theoretic limit, under MAP decoding. 11.3.1

Lower bound on the threshold

We start by deriving a general bound on the block error probability PB (C) for BSC(p), valid for any linear code. Let N = 2N R be the size of the codebook C. By the union bound, 3 4 PB (C) = P ∃ α = 0 such that d(x(α) , y) ≤ d(0, y) ≤

N −1 α=1

3 4 P d(x(α) , y) ≤ d(0, y) .

(11.31)

¾¿¾


As the components of y are i.i.d. Bernoulli variables, the probability P{d(x(α) , y) ≤ d(0, y)} depends on the vector x(α) only through its weight w. Let x(w) be the vector formed by w ones followed by N −w zeros, and denote by N (w) the weight enumerator of the code C. Then PB (C) ≤

N

7 8 N (w) P d(x(w), y) ≤ d(0, y) .

(11.32)

w=1

8 7 u w−u I(u ≥ The probability P d(x(w), y) ≤ d(0, y) can be written as u w u p (1 − p) w/2), where u is the number of sites i ∈ {1, . . . , w} such that yi = 1. A good bound is provided by a standard method known as the Chernoff bound.

Exercise 11.5 Let X be a random variable. Show that, for any a and any λ > 0, ” “ P(X ≥ a) ≤ e−λa E eλX .

(11.33)

In our case this gives 8 7 P d(x(w), y) ≤ d(0, y) ≤ Eeλ[d(0,y)−d(x(w),y)] = [(1 − p) e−λ + p eλ ]w . The bound is optimized for λ =

1 2

log (1 − p)/p > 0, and gives

PB (C) ≤

N

N (w) e−γw ,

(11.34)

w=1

where γ ≡ − log 4p(1 − p) ≥ 0. The quantity 4p(1 − p) is sometimes referred to as the Bhattacharya parameter of the channel BSC(p).

Exercise 11.6 Consider the case of a general binary memoryless symmetric channel with a transition probability Q(y|x), x ∈ {0, 1}, y ∈ Y ⊆ R. First, show that eqn (11.31) remains valid if the Hamming distance d(x, y) is replaced by the log-likelihood dQ (x|y) = −

N X

log Q(yi |xi ) .

(11.35)

i=1

[Hint: Remember the general expressions (6.5) for the probability µy (x) = P(x|y) that the transmitted codeword was x, given that the received message is y.] Then repeat the derivation from eqn (11.31) to eqn (11.34). The final expression involves γ = − log BQ , P p where the Bhattacharya parameter is defined as BQ = y Q(y|1)Q(y|0).


¾¿¿

0.6

p = pLB

p < pLB

0.5

φ(ω)

0.4

p > pLB

0.3 0.2 0.1 0 0

0.2

0.4

0.6

0.8

1

ω

Fig. 11.6 Geometric construction yielding a lower bound on the threshold for reliable communication for the ensemble LDPCN (3, 6) used over the binary symmetric channel. In this case pLB ≈ 0.0438737, shown by the line labelled ‘p = pLB ’. The other two lines refer to p = 0.01 < pLB and p = 0.10 > pLB .

Equation (11.34) shows that the block error probability depends on two factors: the first is the weight enumerator, and the second, exp(−γw), is a channel-dependent term: as the weight of the codewords increases, their contribution is scaled down by an exponential factor because it is less likely that the received message y will be closer to a codeword of large weight than to the all-zero codeword. So far, the discussion has been valid for any given code. Let us now consider the average over LDPCN (Λ, P ) code ensembles. A direct averaging gives the bound PB ≡ EC PB (C) ≤

N w=1

N (w) e−γw

# $ . = exp N sup [φ(ω) − γω] .

(11.36)

ω∈]0,1]

As such, this expression is useless, because the supω [φ(ω) − γω], being greater than or equal to the value at ω = 0, is always positive. However, if we restrict to ensembles with lmin ≥ 3, we know that, with a probability going to one in the large-N limit, there exists no codeword in the ω interval ]0, ω∗ [. In such cases, the maximization over ω in eqn (11.36) can be performed in the interval [ω∗ , 1] instead of ]0, 1]. (By N ω −1 the Markov inequality, this is true whenever N w=1∗ N (w) → 0 as N → ∞.) The bound becomes useful whenever the supremum supω∈[ω∗ ,1] [φ(ω) − γω] is less than 0: then PB vanishes in the large-N limit. We have thus obtained the following result. Proposition 11.3 Consider the average block error rate PB for a random code in the ensemble LDPC N (Λ, P ), with lmin ≥ 3, used over a channel BSC(p), with p < 1/2. Let γ ≡ − log 4p(1 − p), and let φ(ω) be the the weight enumerator exponent defined in eqn (11.4) (φ(ω) can be computed using eqns (11.17)–(11.19)). If φ(ω) < γω for any ω ∈ (0, 1] such that φ(ω) ≥ 0, then PB → 0 in the large-block-length limit.

¾


This result has a pleasing geometric interpretation which is illustrated in Fig. 11.6. As p increases from 0 to 1/2, γ decreases from +∞ to 0. The condition φ(ω) < γω can be rephrased by saying that the weight enumerator exponent φ(ω) must lie below a straight line of slope γ through the origin. We denote by pLB the smallest value of p such that the line γω touches φ(ω). This geometric construction implies pLB > 0. Furthermore, for p large enough Shannon’s theorem implies that PB is bounded away from 0 for any non-vanishing rate R > 0. The MAP threshold pMAP for the ensemble LDPCN (Λ, P ) can be defined as the largest (or, more precisely, the supremum) value of p such that limN →∞ PB = 0. This definition has a very concrete practical meaning: for any p < pMAP one can communicate with an arbitrarily small error probability by using a code from the LDPCN (Λ, P ) ensemble, provided N is large enough. Proposition 11.3 then implies that pMAP ≥ pLB .

(11.37)

In general, one expects limN →∞ PB to exist (and to be strictly positive) for p > pMAP . However, there exists no proof of this statement. It is interesting to note that, at p = pLB , our upper bound on PB is dominated by codewords of weight w ≈ N ω ˜ , where ω ˜ > 0 is the value where φ(ω) − γω is maximum. This suggests that each time an error occurs, a finite fraction of the bits are decoded incorrectly and this fraction fluctuates little from transmission to transmission (or from code to code in the ensemble). The geometric construction also suggests the less obvious (but essentially correct) guess that this fraction jumps discontinuously from ˜ > ω∗ strictly: 0 to a finite value when p crosses the critical value pMAP . Finally, ω dominant error events are not triggered by the closest codewords!

Exercise 11.7 Let us study the case lmin = 2. Proposition 11.3 is no longer valid, but we can still apply eqn (11.36). (a) Consider the ensemble (2, 4) whose weight enumerator exponent is plotted in Fig. 11.4, the small-weight behaviour being given by eqn (11.24). At small enough p, it is reasonable to assume that the block error rate is dominated by small weight codewords. Estimate PB using eqn (11.36) under this assumption. (b) Show that p the above assumption breaks down for p ≥ ploc , where ploc ≤ 1/2 solves the equation 3 4p(1 − p) = 1. (c) Discuss the case of a general code ensemble with lmin = 2, and φ(ω) concave for ω ∈ [0, 1]. Draw a weight enumerator exponent φ(ω) such that the assumption of dominance by low-weight codewords breaks down before ploc . What do you expect for the average bit error rate Pb for p < ploc ? And for p > ploc ?

Exercise 11.8 Discuss the qualitative behaviour of the block error rate for cases where lmin = 1.


11.3.2

¾

Upper bound on the threshold

Let us consider as before communication over BSC(p), keeping for simplicity to regular codes LDPCN (l, k). Gallager has proved the following bound. Theorem 11.4 Let pMAP be the threshold for reliable communication over a binary symmetric channel using codes from the ensemble LDPCN (l, k), with design rate Rdes = 1 − k/l. Then pMAP ≤ pUB , where pUB ≤ 1/2 is the solution of 1 − (1 − 2p)k . (11.38) H(p) = (1 − Rdes )H 2 We shall not give a full proof of this result, but we shall show in this section a sequence of heuristic arguments which can be turned into a proof. The details can be found in the original literature. Assume that the all-zero codeword 0 has been transmitted and that a noisy vector y has been received. The receiver will look for a vector x at a Hamming distance of about N p from y, satisfying all the parity check equations. In other words, let us denote the syndrome by z = Hx, z ∈ {0, 1}M (here H is the parity check matrix and multiplication is performed modulo 2). This is a vector with M components. If x is a codeword, all parity checks are satisfied, and we have z = 0. There is at least one vector x fulfilling the conditions d(x, y) ≈ N p and z = 0: the transmitted codeword 0. Decoding is successful only if this is the unique such vector. The number of vectors x whose Hamming distance from y is close to N p is approximatively 2N H(p) . Let us now estimate the number of distinct syndromes z = Hx when x is on the sphere d(x, y) ≈ N p. Writing x = y ⊕ x , this is equivalent to counting the number of distinct vectors z = Hx when the weight of x is about N p. It is convenient to think of x as a vector of N i.i.d. Bernoulli variables of mean p: we are then interested in the number of distinct typical vectors z . Note that, since the code is regular, each entry zi is a Bernoulli variable of parameter k k n 1 − (1 − 2p)k p (1 − p)k−n = pk = . (11.39) n 2 n odd

If the bits of z were independent, the number of typical vectors z would be 2N (1−Rdes )H(pk ) (the dimension of z being M = N (1 − Rdes )). It turns out that correlations between the bits decrease this number, so we can use the i.i.d. estimate to get an upper bound. Let us now assume that for each z in this set, the number of reciprocal images (i.e. of vectors x such that z = Hx) is approximatively the same. If 2N H(p) 2N (1−Rdes )H(pk ) , for each z there is an exponential number of vectors x such that z = Hx. This will be true, in particular, for z = 0: the received message is therefore not uniquely decodable. In the alternative situation most of the vectors z correspond to (at most) a single x. This will be the case for z = 0: decoding can be successful. 11.3.3

Summary of the bounds

In Table 11.1 we consider a few regular LDPCN (Λ, P ) ensembles over the channel BSC(p). We show the window of possible values of the noise threshold pMAP , using

¾


Table 11.1 Bounds on the threshold for reliable communication over BSC(p) using LDPCN (l, k) ensembles with MAP decoding. The fourth and fifth columns list the lower bound (LB) of Proposition 11.3 and the upper bound (UB) of Theorem 11.4. The sixth column lists an improved lower bound obtained by Gallager.

l 3 3 3 4 5

k 4 5 6 6 10

Rdes 1/4 2/5 1/2 1/3 1/2

LB of Section 11.3.1 0.1333161 0.0704762 0.0438737 0.1642459 0.0448857

Gallager UB 0.2109164 0.1397479 0.1024544 0.1726268 0.1091612

Gallager LB 0.2050273 0.1298318 0.0914755 0.1709876 0.1081884

Shannon limit 0.2145018 0.1461024 0.1100279 0.1739524 0.1100279

the lower bound of Proposition 11.3 and the upper bound of Theorem 11.4. In most cases, the comparison is not satisfactory (the gap between the upper and lower bounds is close to a factor of 2). A much smaller uncertainty is achieved using an improved lower bound again derived by Gallager, based on a refinement of the arguments in the previous section. As we shall see in Chapter 15 by computing pMAP , neither of the bounds is tight. On the other hand, they are sufficiently good to show that, for large k, l the MAP threshold of these ensembles is close to Shannon, capacity (although bounded away from it). Indeed, by studying the asymptotic behaviour of these bounds, one can show that the MAP threshold of the (k, l) ensemble converges to pSh as k, l → ∞ with a fixed ratio l/k.

Exercise 11.9 Let pSh be the upper bound on pMAP provided by the Shannon channel coding theorem. Explicitly, pSh ≤ 1/2 is the solution of H(p) = 1−R. Prove that if R = Rdes (as is the case with high probability for LDPCN (l, k) ensembles), then pUB < pSh .

11.4

A simple decoder: Bit flipping

So far, we have analysed the behaviour of LDPC ensembles under the optimal (word MAP) decoding strategy. However, there is no known way of implementing this decoder with an efficient algorithm. The naive algorithm goes through each codeword x(α) , α = 0, . . . , 2N R − 1, and outputs the codeword of greatest likelihood Q(y|x(α) ). However, this approach takes a time which grows exponentially with the block length N . For large N (which is the regime where the error rate becomes close to optimal), this is unpractical. LDPC codes are interesting because there exist fast suboptimal decoding algorithms with performances close to the theoretical optimal performance, and therefore close to Shannon’s limit. Here we show one example of a very simple decoding method, called the bit flipping algorithm. After transmission through a BSC, we have received the message y, and we try to find the sent codeword x by use of the following algorithm.

A simple decoder: Bit flipping 0.05

0.8

0.04

0.6

0.03 E*

PB

1

¾

N=10000 N=20000 N=40000

0.4

0.02

0.2

N=10000 N=20000 N=40000

0.01

0 0.016

0.020

0.024 p

0.028

0.032

0 0.016

0.020

0.024

0.028

0.032

p

Fig. 11.7 Performance of the bit-flipping decoding algorithm on random codes from the (5, 10) regular LDPC ensemble, used over the channel BSC(p). Left: block error rate. Right: residual number of unsatisfied parity checks after the algorithm has halted. The statistical error bars are smaller than the symbols.

Bit-flipping decoder (received message y) 1: Set x(0) = y. 2: for t = 1, . . . , N : 3: find a bit belonging to more unsatisfied than satisfied parity checks; 4: if such a bit exists, flip it: xi (t + 1) = xi (t) ⊕ 1, and keep the other bits: xj (t + 1) = xj (t) for all j = i; 5: if there is no such bit, return x(t) and halt. The bit to be flipped is usually chosen uniformly at random from the bits satisfying the condition at step 3. However, this is irrelevant in the analysis below.

Exercise 11.10 Consider a code from the (l, k) regular LDPC ensemble (with l ≥ 3). Assume that the received message differs from the transmitted message in only one position. Show that the bit-flipping algorithm always corrects such an error.

Exercise 11.11 Assume now that the channel has introduced two errors. Draw the factor graph of a regular (l, k) code for which the bit-flipping algorithm is unable to recover from such an error event. What can you say of the probability of this type of graph in the ensemble?

In order to monitor the bit-flipping algorithm, it is useful to introduce the ‘energy.’ E(t) ≡ Number of parity check equations not satisfied by x(t) .

(11.40)

¾


This is a non-negative integer, and if E(t) = 0, the algorithm is halted and its output is x(t). Furthermore, E(t) cannot be larger than the number of parity checks M , and it decreases (by at least one) at each cycle. Therefore, the complexity of the algorithm is O(N ) (this is a commonly regarded as the ultimate goal for many communication problems). It remains to be seen if the output of the bit-flipping algorithm is related to the transmitted codeword. In Fig. 11.7, we present the results of a numerical experiment. We considered the (5, 10) regular ensemble and generated about 1000 random code and channel realizations for each value of the noise in a mesh. Then we applied the above algorithm and plotted the fraction of successfully decoded blocks, as well as the residual energy E∗ = E(t∗ ), where t∗ is the total number of iterations of the algorithm. The data suggests that bit-flipping is able to overcome a finite noise level: it recovers the original message with high probability when fewer than about 2.5% of the bits are corrupted by the channel. Furthermore, the curves for Pbf B under bit-flipping decoding become steeper and steeper as the size of the system is increased. It is natural to conjecture that asymptotically, a phase transition takes place at a well-defined noise bf level pbf : Pbf B → 0 for p < pbf and PB → 1 for p > pbf . Numerically, pbf = 0.025±0.005. This threshold can be compared with that for MAP decoding. The results in Table 11.1 imply 0.108188 ≤ pMAP ≤ 0.109161 for the (5, 10) ensemble. Bit-flipping is significantly suboptimal, but is still surprisingly good, given the extreme simplicity of the algorithm. Can we provide any guarantee on the performance of the bit-flipping decoder? One possible approach consists in using the expansion properties of the underlying factor graph. Consider a graph from the (l, k) ensemble. We say that it is an (ε, δ) expander if, for any set U of variable nodes such that |U | ≤ N ε, the set |D| of neighbouring check nodes has a size |D| ≥ δ|U |. Roughly speaking, if the factor graph is an expander with a large expansion constant δ, any small set of corrupted bits induces a large number of unsatisfied parity checks. The bit-flipping algorithm can exploit these checks to successfully correct the errors. It turns out that random graphs are very good expanders. This can be understood as follows. Consider a fixed subset U . As long as U is small, the subgraph induced by U and the neighbouring factor nodes D is a tree with high probability. If this is the case, elementary counting shows that |D| = (l − 1)|U | + 1. This would suggest that one can achieve an expansion factor of l − 1 (or close to it), for small enough ε. Of course, this argument has several flaws. First of all, the subgraph induced by U is a tree only if U has a sublinear size, but we are interested in all subsets U with |U | ≤ εN for some fixed N . Then, while most of the small subsets U are trees, we need to be sure that all subsets expand well. Nevertheless, one can prove that the heuristic expansion factor is essentially correct. Proposition 11.5 Consider a random factor graph F from the (l, k) ensemble. Then, for any δ < l − 1, there exists a constant ε = ε(δ; l, k) > 0 such that F is an (ε, δ) expander with probability approaching 1 as N → ∞. In particular, this implies that, for l ≥ 5, a random (l, k) regular factor graph is, with high probability an (ε, 43 l) expander. In fact, this is enough to ensure that the

Notes

¾

code will perform well at a low noise level. Theorem 11.6 Consider a regular (l, k) LDPC code C, and assume that the corresponding factor graph is an (ε, 34 l) expander. Then, the bit-flipping algorithm is able to correct any pattern of fewer than N ε/2 errors produced by a binary symmetric channel. In particular, PB (C) → 0 for communication over BSC(p) with p < ε/2. Proof As usual, we assume the channel input to be the all-zeros codeword 0. We denote by w = w(t) the weight of x(t) (the current configuration of the bit-flipping algorithm), and by E = E(t) the number of unsatisfied parity checks, as in eqn (11.40). Finally, we denote by F the number of satisfied parity checks among those which are neighbours of at least one corrupted bit in x(t) (a bit is ‘corrupted’ if it takes the value 1). Assume, first, that 0 < w(t) ≤ N ε at some time t. Because of the expansion property of the factor graph, we have E + F > 34 lw. On the other hand, every unsatisfied parity check is a neighbour of at least one corrupted bit, and every satisfied check which is a neighbour of a corrupted bit must involve at least two of them. Therefore E + 2F ≤ lw. Eliminating F from the above inequalities, we deduce that E(t) > 12 lw(t). Let Ei (t) be the number of unsatisfied checks involving the bit xi . Then, i:xi (t)=1

Ei (t) ≥ E(t) >

1 lw(t) . 2

(11.41)

Therefore, there must be at least one bit that has more unsatisfied than satisfied neighbours, and the algorithm does not halt. Let us now suppose that we start the algorithm with w(0) ≤ N ε/2. It must halt at some time t∗ , either with E(t∗ ) = w(t∗ ) = 0 (and therefore decoding is successful) or with w(t∗ ) ≥ N ε. In the second case, as the weight of x(t) changes by one at each step, we have w(t∗ ) = N ε. The above inequalities imply E(t∗ ) > N lε/2 and E(0) ≤ lw(0) ≤ N lε/2. This contradicts the fact that E(t) is a strictly decreasing function of t. Therefore the algorithm, when started with w(0) ≤ N ε/2, ends up in the state w = 0, E = 0. The approach based on expansion of the graph has the virtue of pointing out one important mechanism for the good performance of random LDPC codes, namely the local tree-like structure of the factor graph. It also provides explicit lower bounds on the critical noise level pbf for bit-flipping. However, these bounds turn out to be quite pessimistic. For instance, in the case of the (5, 10) ensemble, it has been proved that −12 . On the a typical factor graph is an (ε, 34 l) = (ε, 15 4 ) expander for ε < ε∗ ≈ 10 other hand, numerical simulations (see Fig. 11.7) show that the bit-flipping algorithm performs well up to noise levels much larger than ε∗ /2.

Notes Modern (post-Cook’s Theorem) complexity theory was first applied to coding by Berlekamp et al. (1978), who showed that maximum-likelihood decoding of linear codes is NP-hard.

¾


LDPC codes were first introduced by Gallager in his PhD thesis (Gallager, 1963; Gallager, 1962), which is in fact older than these complexity results. An excellent, detailed account of modern developments is provided by Richardson and Urbanke (2008). Gallager’s proposal did not receive enough consideration at the time. One possible explanation is the lack of computational power for simulating large codes in the 1960s. The rediscovery of LDPC codes in the 1990s (MacKay, 1999) was (at least in part) a consequence of the invention of turbo codes by Berrou and Glavieux (1996). Both of these classes of codes were soon recognized to be prototypes of a larger family: codes on sparse graphs. The major technical advance after this rediscovery was the introduction of irregular ensembles (Luby et al. 1997; Luby et al. 1998). There exists no formal proof of the ‘equivalence’ (whatever this means) of the various possible definitions of LDPC ensembles in the large-block-length limit. But, as we shall see in Chapter 15, the main property that enters in the analysis of LDPC ensembles is the local tree-like structure of the factor graph described in Section 9.5.1; and this property is rather robust with respect to a change of the ensemble. Gallager (1963) was the first to compute the expected weight enumerator for regular ensembles, and to use it in order to bound the threshold for reliable communication. General ensembles were considered by Litsyn and Shevelev (2003), Burshtein and Miller (2004), and Di et al. (2006). It turns out that the expected weight enumerator coincides with the typical (most likely) weight enumerator to leading exponential order for regular ensembles (in statistical-physics jargon, the annealed computation coincides with the quenched one). This is not the case for irregular ensembles, as pointed out by Di et al. (2004). Proposition 11.2 has, essentially, been known since Gallager (1963). The formulation quoted here is from Méasson et al. (2008). That paper contains some examples of ‘exotic’ LDPC ensembles such that the maximum of the expected weight enumerator is at a weight w = N ω∗ , with ω∗ = 1/2. A proof of the upper bound in Theorem 11.4 can be found in Gallager (1963). For some recent refinements, see Burshtein et al. (2002). Bit-flipping algorithms played an important role in the revival of LDPC codes, especially following the work of Sipser and Spielman (1996). These authors focused on explicit code construction based on expander graphs. They also provided bounds on the expansion of random LDPCN (l, k) codes. The lower bound on the expansion mentioned in Section 11.4 is taken from Richardson and Urbanke (2008).

12 Spin glasses We have already encountered several examples of spin glasses in Chapters 2 and 8. Like most problems in equilibrium statistical physics, they can be formulated in the general framework of factor graphs. Spin glasses are disordered systems, whose magnetic properties are dominated by randomly placed impurities. The theory aims at describing the behaviour of a typical sample of such materials. This motivates the definition and study of spin glass ensembles. In this chapter, we shall explore the glass phase of these models. It is useful to have a good understanding of glass phases, as we shall see them appearing in various problems from optimization and coding theory. In general, the occurrence of a glass phase is described physically in terms of a dramatic slowdown in a dynamical relaxation process. Here, we shall focus instead on purely static characterizations of glass phases, which can be applied to a broad class of problems. The focus of our presentation is on ‘mean-field models’, for at least two reasons: (i) a deep mathematical theory (still under developement) can provide a precise understanding of their behaviour; (ii) the ensembles of combinatorial optimization and coding problems to be considered in the following fall naturally into this class. We shall discuss the two types of spin glass transition that have been encountered in such models. In contrast to these ‘soluble’ cases, it must be stressed that very little is known (let alone proven) for realistic models of real spin glass materials. Even the existence of a spin glass phase has not been established rigorously in the latter case. We first discuss, in Section 12.1, how Ising models and their generalizations can be formulated in terms of factor graphs, and introduce several ensembles of these models. Frustration is a crucial feature of spin glasses; in Section 12.2, we discuss it in conjunction with gauge transformations. This section also explains how to derive some exact results solely with the use of gauge transformations. Section 12.3 describes the spin glass phase and the main approaches to its characterization. Finally, the phase diagram of a spin glass model with several glassy phases is traced in Section 12.4.

12.1 12.1.1

Spin glasses and factor graphs Generalized Ising models

Let us recall the main ingredients of magnetic systems with interacting Ising spins. The variables are N Ising spins σ = {σ1 , . . . , σN }, taking values in {+1, −1}. These are jointly distributed according to Boltzmann’s law for the energy function

¾

Spin glasses

Fig. 12.1 Factor graph representation of the SK model with N = 4 (left), and of the fully connected 3-spin model with N = 4 (right). The squares denote the interactions between the spins.

E(σ) = −

p max

Ji1 ...ip σi1 · · · σip .

(12.1)

p=1 i1 0 in Boolean form using xi and xj .]

Since 2-SAT is in P, and because of the equivalence demonstrated in the last exercise, one can check in polynomial time whether the energy function (12.3) is frustrated or not. This approach does not work when p ≥ 3, because K-SAT is NP-complete for K ≥ 3. However, as we shall see in Chapter 18, checking whether a spin glass energy function is frustrated remains a polynomial problem for any p. 12.2.1

Gauge transformation

When a spin glass sample has some negative couplings but is unfrustrated, one is in fact dealing with a ‘disguised ferromagnet’. By this we mean that, through a change of variables, the problem of computing the partition function for such a system can be reduced to the problem of computing the partition function of a ferromagnet. Indeed, by assumption, there exists a ground state spin configuration σi∗ ∈ {±1} such that, ∀(i, j) ∈ E, Jij σi∗ σj∗ > 0. Given a configuration σ, we define τi = σi σi∗ , and notice that τi ∈ {+1, −1}. Then the energy of the configuration is E(σ) = E∗ (τ ) ≡ − (i,j)∈E |Jij |τi τj . Obviously, the partition function for a system with the energy function E∗ ( · ) (which is a ferromagnet, since |Jij | > 0) is the same as for the original system. This change of variables is an example of a gauge transformation. In general, such a transformation amounts to changing all spins and, simultaneously, all couplings according to: (s) (s) Jij → Jij = Jij si sj , (12.4) σi → σi = σi si , where s = {s1 , . . . , sN } is an arbitrary configuration in {−1, 1}N . If we regard the partition function as a function of the coupling constants J = {Jij : (ij) ∈ E}, ⎞ ⎛ Z[J] = (12.5) exp ⎝β Jij σi σj ⎠ , {σi }

(ij)∈E

then we have Z[J] = Z[J (s) ] .

(12.6)

Spin glasses: Constraints and frustration

¾

The system with coupling constants J (s) is sometimes called the ‘gauge-transformed system’.

Exercise 12.6 Consider adding a uniform magnetic field (i.e. a linear term of the form P −B i σi ) to the energy function (12.3), and apply a generic gauge transformation to such a system. How must the uniform magnetic field be changed in order to keep the partition function unchanged? Is the new magnetic-field term still uniform?

Exercise 12.7 Generalize the above discussion of frustration and gauge transformations to the ±J 3-spin glass (i.e. a model of the type (12.1) involving only terms with p = 3).

12.2.2

The Nishimori temperature. . .

In many spin glass ensembles, there exists a special temperature (called the Nishimori temperature) at which some thermodynamic quantities, such as the internal energy, can be computed exactly. This nice property is particularly useful in the study of inference problems (a particular instance being symbol MAP decoding of errorcorrecting codes), since the Nishimori temperature arises naturally in these contexts. There are in fact two ways of deriving it: either as an application of gauge transformations (this is how it was discovered in physics), or by mapping the system onto an inference problem. Let us begin by taking the first point of view. Consider, for the sake of simplicity, the model (12.3). The underlying graph G = (V, E) can be arbitrary, but we assume that the couplings Jij on all the edges (ij) ∈ E are i.i.d. random variables taking values Jij = +1 with probability 1 − p and Jij = −1 with probability p. We denote by E the expectation with respect to this distribution. The Nishimori temperature for this system is given by TN = 1/βN , where βN = 1 log (1 − p)/p . It is chosen in such a way that the coupling-constant distribution 2 P(J) satisfies the condition P(J) = e−2βN J P(−J) .

(12.7)

An equivalent way of stating the same condition is to write P(J) =

eβN J Q(|J|) , 2 cosh(βN J)

(12.8)

where Q(|J|) denotes the distribution of the absolute values of the couplings (in the present example, this is a Dirac delta on |J| = 1). Let us now turn to the computation of the average, over the distribution of couplings, of the internal energy1 U ≡ EE(σ). More explicitly, 1 The same symbol U was used in Chapter 2 to denote the internal energy E(σ) (instead of its average). There should be no confusion with the present use.

¾

Spin glasses

⎧ ⎫ ⎨ 1 % ⎬ & P − U =E Jkl σk σl eβ (ij) Jij σi σj . ⎩ Z[J] ⎭ σ

(12.9)

(kl)

In general, it is very difficult to compute U . It turns out, however, that at the Nishimori temperature, the gauge invariance allows an easy computation. The average internal energy U can be expressed as U = E{ZU [J]/Z[J]}, where ZU [J] = − σ (kl) Jkl σk σl (ij) eβN Jij σi σj . Let s ∈ {−1, 1}N . By an obvious generalization of eqn (12.6), we have ZU [J (s) ] = ZU [J], and therefore U = 2−N E{ZU [J (s) ]/Z[J (s) ]} . (12.10) s

If the coupling constants Jij are i.i.d. with the distribution (12.8), then the gauge(s) = Jij are equally independent but with a distribution transformed constants Jij Ps (Jij ) =

eβN Jij si sj . 2 cosh βN

(12.11)

Equation (12.10) can therefore be written as U = 2−N s Es {ZU [J]/Z[J]}, where Es denotes the expectation with respect to the modified measure Ps (Jij ). Using eqn (12.11), and denoting by E0 the expectation with respect to the uniform measure over Jij ∈ {±1}, we get ⎧ ⎫ ⎨ eβN Jij si sj ZU [J] ⎬ U = 2−N E0 (12.12) ⎩ cosh βN Z[J] ⎭ s (ij) ⎧ ⎫ ⎨ ⎬ P Z [J] U eβN (ij) Jij si sj = 2−N (cosh βN )−|E| E0 (12.13) ⎩ s Z[J] ⎭ = 2−N (cosh βN )−|E| E0 {ZU [J]} .

(12.14)

It is easy to compute E0 ZU [J] = −2N (cosh βN )|E|−1 sinh βN . This implies our final result for the average energy at the Nishimori temperature: U = −|E| tanh(βN ) .

(12.15)

Note that this simple result holds for any choice of the underlying graph. Furthermore, it is easy to generalize it to other choices of the coupling distribution that satisfy eqn (12.8) and to models with multispin interactions of the form (12.1). An even wider generalization is treated below. 12.2.3

. . . and its relation to probability

The calculation of the internal energy in the previous subsection is straightforward but, in a sense, mysterious. It is hard to grasp what the fundamental reason is that makes things simpler at the Nishimori temperature. Here we discuss a more general

Spin glasses: Constraints and frustration

¾

derivation, in a slightly more abstract setting, which is related to the connection with inference problems mentioned above. Consider the following process. A configuration σ ∈ {±1} is chosen uniformly at random; we call the corresponding distribution P0 (σ). Next, a set of coupling constants J = {Ja } is chosen according to the conditional distribution P(J|σ) = e−βEJ (σ) Q0 (J) .

(12.16)

Here EJ (σ) is an energy function with coupling constants J, and Q0 (J) is some reference measure (which can be chosen in such a way that the resulting P(J|σ) is normalized). This can be interpreted as a communication process. The information source produces the message σ uniformly at random, and the receiver observes the couplings J. The joint distribution of J and σ is P(J, σ) = e−βEJ (σ) Q0 (J)P0 (σ). We shall denote the expectation with respect to this joint distribution by ‘Av’ in order to distinguish it from the thermal average (that over the Boltzmann measure, denoted by . ) and from the quenched average over the couplings, denoted by E. We assume that this process possesses a gauge symmetry: this assumption defines the Nishimori temperature. By this we mean that, given s ∈ {±1}N , there exists an invertible mapping J → J (s) such that Q0 (J (s) ) = Q0 (J) and EJ (s) (σ (s) ) = EJ (σ). It is then clear that the joint probability distribution of the coupling and the spins and the conditional distribution possess the same symmetry: P(σ (s) , J (s) ) = P(σ, J) ,

P(J (s) |σ (s) ) = P(J|σ) .

(12.17)

Let us introduce the quantity U(J) = Av(EJ (σ)|J) =

P(σ|J)EJ (σ) ,

(12.18)

σ

and denote J P(J|σ 0 ) U(J) by U (σ 0 ). This is nothing but the average internal energy for a disordered system with an energy function EJ (σ) and coupling distribution P(J|σ 0 ). For instance, if we take σ 0 as the ‘all-plus’ configuration, Q0 (J) to be proportional to the uniform measure over {±1}E , and EJ (σ) as given by eqn (12.3), then U (σ 0 ) is exactly the quantity U that we computed in the previous section. Gauge invariance implies that U(J) = U(J (s) ) for any s, and U (σ 0 ) does not depend upon σ 0 . We can therefore compute U = U (σ 0 ) by averaging over σ 0 . We obtain U = P0 (σ 0 ) P(J|σ 0 ) P(σ|J)EJ (σ) σ0

=

σ,J

J

P(σ, J)EJ (σ) =

σ

P(J|σ 0 )EJ (σ) ,

(12.19)

J

where we have used gauge invariance once more in the last step. The final expression is generally easy to evaluate, since the coublings Ja are generically independent under P(J|σ 0 ) In particular, it is straightforward to recover eqn (12.15) for the case treated in the last subsection.

¾

Spin glasses

Exercise 12.8 Consider a spin glass model on an arbitrary graph, with an energy given by eqn (12.3), and i.i.d. random couplings on the edges, drawn from the distribution P(J) = P0 (|J|)eaJ . Show that the Nishimori inverse P temperature is βN = a, and that the internal energy at this point is given by U = −|E| J P0 (|J|) J sinh(βN J). In the case where P is a Gaussian distribution of mean J0 , show that U = −|E|J0 .

12.3 12.3.1

What is a glass phase? Spontaneous local magnetizations

In physics, a ‘glass’ is defined through its dynamical properties. For classical spin models such as the ones we are considering here, one can define several types of physically meaningful dynamics. For definiteness, we shall use the single-spin-flip Glauber dynamics defined in Section 4.5. The main features of our discussion should remain unchanged as long as we keep to local dynamics (i.e. a bounded number of spins is flipped at each step), which obey detailed balance. Consider a system at equilibrium at time 0 (i.e. assume σ(0) to be distributed according to the Boltzmann distribution), and denote by · σ(0) the expectation with respect to Glauber dynamics conditional on the initial configuration. Within a ‘solid’2 phase, spins are correlated with their initial values on long time scales: lim lim σi (t)σ(0) ≡ mi,σ(0) = σi .

t→∞ N →∞

(12.20)

In other words, on arbitrary long but finite (in terms of the system size) time scales, the system converges to a ‘quasi-equilibrium’ state, which we shall call for brevity a ‘quasi-state’, with local magnetizations mi,σ(0) depending on the initial condition. The condition (12.20) is, for instance, satisfied by a d ≥ 2 Ising ferromagnet in zero external field, at temperatures below the ferromagnetic phase transition. In this case we have either mi,σ(0) = M (β) or mi,σ(0) = −M (β), depending on the initial condition, where M (β) is the spontaneous magnetization of the system. There are two quasistates, invariant under translation and related by a simple symmetry transformation. If the different quasi-states are neither periodic nor related by any symmetry, one may speak of a glass phase. It is also very important to characterize the glass phase at the level of equilibrium statistical mechanics, without introducing a specific dynamics. For the case of ferromagnets we have already seen the solution to this problem in Chapter 2. Let · B denote the expectation with respect to the Boltzmann measure for the energy function (12.1), after a uniform magnetic field has been added. We then define the two quasi-states by mi,± ≡ lim

lim σi B .

B→0± N →∞

(12.21)

A natural generalization to glasses consists in adding a small magnetic field which is not uniform. Let us add to the energy function (12.1) a term of the form − i si σi , 2 The name comes from the fact that in a solid, the preferred positions of the atoms are timeindependent; for instance, in a crystal, they are the vertices of a periodic lattice.

What is a glass phase?

¾

where s ∈ {±1}N is an arbitrary configuration. We denote by · ,s the expectation with respect to the corresponding Boltzmann distribution, and let mi,s ≡ lim

lim σi ,s .

→0± N →∞

(12.22)

The Edwards–Anderson order parameter, defined as 1 σi 2,s , →0± N →∞ N i

qEA ≡ lim

lim

(12.23)

where s is an equilibrium configuration sampled from the Boltzmann distribution, then signals the onset of the spin glass phase. The careful reader will notice that eqn (12.20) is not really completely defined. How should we take the N → ∞ limit? Do the limits exist, and how does the result depend on s? These are subtle questions. They underlie the problem of defining properly the pure states (extremal Gibbs states) in disordered systems. We shall come back to these issues in Chapter 22. An extremely fruitful idea is, instead, to study glassy phases by comparing several equilibrated (i.e. drawn from the Boltzmann distribution) configurations of the system: one can then use one configuration to define the direction of the polarizing field, as we just did for the Edwards–Anderson order parameter. Remarkably, this idea underlies the formal manipulations in the replica method. We shall explain below in greater detail two distinct criteria, based on this idea, which can be used to define a glass phase. Before this, let us discuss a criterion for stability of the high-temperature phase. 12.3.2

The spin glass susceptibility

We take a spin glass sample, with energy (12.1), and add to it a local magnetic field on site i, Bi . The magnetic susceptibility of spin j with respect to the field Bi is defined as the rate of change of mj = σj Bi with respect to Bi , dmj χji ≡ = β(σi σj − σi σj ) , (12.24) dBi Bi =0 where we have used the fluctuation–dissipation relation (2.44). The uniform (ferromagnetic) susceptibility defined in Section 2.5.1 gives the rate of change of the average magnetization with respect to an infinitesimal global uniform field, i.e. χ = (1/N ) i,j χji . Consider a ferromagnetic Ising model, as introduced in Section 2.5. Within the ferromagnetic phase (i.e. at zero external field and below the critical temperature), χ diverges with the system size N . One way to understand this divergence is the following. If we denote by m(B) the magnetization in infinite volume in a magnetic field B, and denote the spontaneous magnetization by M (β), we have χ = lim

B→0

1 M (β) [m(B) − m(−B)] = lim = ∞, B→0+ 2B B

within the ferromagnetic phase.

(12.25)

¾

Spin glasses

The above argument relates the divergence of the susceptibility to the existence of two distinct pure states of the system (‘plus’ and ‘minus’). What is the appropriate susceptibility for detecting a spin glass ordering? Following our previous discussion, we should consider the addition of a small non-uniform field Bi = si . The local magnetizations are given by χij sj + O(2 ) . (12.26) σi ,s = σi 0 + j

As suggested by eqn (12.25), we compare the local magnetizations obtained by perturbing the system in two different directions s and s : χij (sj − sj ) + O(2 ) . (12.27) σi ,s − σi ,s = j

How should we choose s and s ? A simple choice takes them as independent and uniformly random in {±1}N ; let us denote the expectation with respect to this distribution by Es . The above difference therefore becomes a random variable with zero mean. Its second moment allows us to define the spin glass susceptibility (sometimes called the non-linear susceptibility), 2 1 Es σi ,s − σi ,s . 2 →0 2N i

χSG ≡ lim

(12.28)

This is in a sense the equivalent of eqn (12.25) for the spin glass case. Using eqn (12.27), one gets the expression χSG = (1/N ) ij (χij )2 , which can also be written, using the fluctuation–dissipation relation, as χSG =

β2 2 [σi σj − σi σj ] . N i,j

(12.29)

Usually, a necessary condition for the system to be in a paramagnetic, non-solid phase is that χSG remains finite when N → ∞. We shall see below that this necessary condition of local stability is not always sufficient. Exercise 12.9 Another natural choice would consist in choosing s and s as independent configurations drawn from P the Boltzmann distribution. Show that with such a choice one would get χSG = (1/N ) i,j,k χij χjk χki . This susceptibility has not been studied in the literature, but it is reasonable to expect that it will lead generically to the same criterion of stability as the usual one given in eqn (12.29).

12.3.3

The overlap distribution function P (q)

One of the main indicators of a glass phase is the overlap distribution, which we defined in Section 8.2.2. Given a general magnetic model of the type (12.1), one generates two independent configurations σ and σ from the associated Boltzmann distribution and


¾

considers their overlap qσ,σ = N −1 i σi σi . The overlap distribution PN (q) is the distribution of qσ,σ when the couplings and the underlying factor graph are taken randomly from an ensemble. Its infinite-N limit is denoted by P (q). Its moments are given by 4 3 1 (12.30) σi1 . . . σir 2 . PN (q)q r dq = E r N i ,...,i .

1

r

In particular, the first moment PN (q) q dq = N −1 i m2i is the expected overlap, '. (2 . and the variance Var(q) ≡ PN (q) q 2 dq − PN (q) q dq is related to the spin glass susceptibility: 4 3 1 1 2 = χSG . [σi σj − σi σj ] Var(q) = E 2 N i,j N

(12.31)

How is a glass phase detected through the behaviour of the overlap distribution P (q)? We shall discuss here two scenarios that appear to be remarkably universal within mean-field models. In the next subsection, we shall see that the overlap distribution is in fact related to the idea, discussed in Section 12.3.1, of perturbing the system in order to explore its quasi-states. Generically, at small β, a system of the type (12.1) is found in a ‘paramagnetic’, or ‘liquid’, phase. In this regime, PN (q) becomes concentrated as N → ∞ on a single (deterministic) value q(β): with high probability, two independent configurations σ and σ have an overlap close to q(β). In fact, in such a phase, the spin glass susceptibility χSG is finite, and the variance of PN (q) therefore vanishes as 1/N . For β larger than a critical value βc , the distribution P (q) may acquire some structure, in the sense that several values of the overlap have non-zero probability in the N → ∞ limit. The temperature Tc = 1/βc is called the static (or equilibrium) glass transition temperature. For β > βc , the system is in an equilibrium glass phase. What does P (q) look like at β > βc ? Generically, the transition falls into one of the following two categories, the names of which come from the corresponding replicasymmetry-breaking patterns found in the replica approach: (i) Continuous (‘full replica-symmetry-breaking’, ‘FRSB’) glass transition. In Fig. 12.2, we sketch the behaviour of the thermodynamic limit of P (q) in this case. The delta function present at β < βc ‘broadens’ for β > βc , giving rise to a distribution with support in some interval [q0 (β), q1 (β)]. The width q1 (β) − q0 (β) vanishes continuously when β ↓ βc . Furthermore, the asymptotic distribution has a continuous density which is strictly positive in ]q0 (β), q1 (β)[, and two discrete (delta) contributions at q0 (β) and q1 (β). This type of transition has a ‘precursor’. If we consider the N → ∞ limit of the spin glass susceptibility, this diverges as β ↑ βc . This phenomenon is quite important for identifying the critical temperature experimentally, numerically, and analytically. (ii) Discontinuous (‘1RSB’) glass transition (see Fig. 12.3). Again, P (q) acquires a non-trivial structure in the glass phase, but the scenario is different. When β increases above βc , the δ-peak at q(β), which had unit mass at β ≤ βc , becomes a

¾

Spin glasses

P (q)

P (q)

q

q

P (q)

q Tc

0

T

Fig. 12.2 Typical behaviour of the order parameter P (q) (the asymptotic overlap distribution) at a continuous (FRSB) glass transition. Vertical arrows denote Dirac delta functions.

peak at q0 (β), with a mass 1 − x(β) < 1. Simultaneously, a second δ-peak appears at a value of the overlap q1 (β) > q0 (β) with mass x(β). As β ↓ βc , q0 (β) → q(βc ) and x(β) → 0. Unlike in a continuous transition, the width q1 (β) − q0 (β) does not vanish as β ↓ βc , and the open interval ]q0 (β), q1 (β)[ has vanishing probability in the N → ∞ limit. Furthermore, the thermodynamic limit of the spin glass susceptibility χSG has a finite limit as β ↑ βc . This type of transition has no ‘simple’ precursor (but we shall describe below a more subtle indicator). The two-peak structure of P (q) in a discontinuous transition has a particularly simple geometrical interpretation. When two configurations σ and σ are chosen independently with the Boltzmann measure, their overlap is (with high probability) approximately equal to either q0 or q1 . In other words, their Hamming distance is either N (1 − q1 )/2 or N (1 − q0 )/2. This means that the Boltzmann measure µ(σ) is concentrated in some regions of the Hamming space {−1, 1}N , called clusters. With high probability, two independent random configurations in the same cluster have a distance (close to) N (1 − q1 )/2, and two configurations in distinct clusters have a distance (close to) N (1 − q0 )/2. In other words, while the overlap does not concentrate in probability when σ and σ are drawn from the Boltzmann measure, it does when this measure is restricted to one cluster. In a more formal (but still imprecise) way, we might write µ(σ) ≈

wα µα (σ) ,

(12.32)

α

where the µα ( · ) are probability distributions concentrated on a single cluster, and the wα are the weights attributed by the Boltzmann distribution to each cluster.


¾

P (q)

P (q)

q

P (q)

q 0

q Tc

Td

T

Glass phase, trivial P (q). Fig. 12.3 Typical behaviour of the order parameter P (q) (the asymptotic overlap distribution) in a discontinuous (1RSB) glass transition. Vertical arrows denote Dirac delta functions.

7 2 8 According to this interpretation, x(β) = E α wα . Note that, since x(β) > 0 for β > βc , the weights are sizeable only for a finite number of clusters (if there were R clusters, all with the same weight wα = 1/R, one would have x(β) = 1/R). This is what we have found already in the REM, as well as in the replica solution of the completely connected p-spin model (see. Section 8.2). Generically, clusters already exist in some region of temperature above Tc , but the measure is not yet condensed on a finite number of them. The existence of clusters in this intermediate-temperature region can be detected instead using the tools described below. There is no clear criterion that allows one to distinguish a priori between systems that undergo one or the other type of transition. Experience gained with models solved via the replica and cavity methods indicates that a continuous transition typically occurs in standard spin glasses with p = 2-body interactions, but also, for instance, in the vertex-covering problem. A discontinuous transition is instead found in structural glasses, generalized spin glasses with p ≥ 3, random satisfiability, and coloring. To complicate things, both types of transition may occur in the same system at different temperatures (or when some other parameter is varied). This may lead to a rich phase diagram with several glass phases of different nature. It is natural to wonder whether gauge transformations might give some information about P (q). Unfortunately, it turns out that the Nishimori temperature never enters a spin glass phase: the overlap distribution at TN is concentrated on a single value, as suggested by the next exercise.

¾

Spin glasses

Exercise 12.10 Using the gauge transformation of Section 12.2.1, show that, at the Nishimori temperature, the overlap distribution PN (q) is equal to the distribution of the magP netization per spin m(σ) ≡ N −1 i σi . (In many spin glass models, one expects that this distribution of magnetization per spin will obey a large-deviation principle, and become concentrated on a single value as N → ∞.)

12.3.4

The -coupling method

The overlap distribution is in fact related to the idea of quasi-states introduced in Section 12.3.1. Let us again consider a perturbation of the Boltzmann distribution defined by adding to the energy a magnetic-field term − i si σi , where s = (s1 , . . . , sN ) is a generic configuration. We introduce the -perturbed energy of a configuration σ as E,s (σ) = E(σ) −

N

si σi .

(12.33)

i=1

It is important to realize that both the original energy E(σ) and the new term − i si σi are extensive, i.e. they grow in proportion to N as N → ∞. Therefore, in this limit the presence of the perturbation can be relevant. The -perturbed Boltzmann measure is 1 −βE,s (σ) e . (12.34) µ,s (σ) = Z,s In order to quantify the effect of the perturbation, let us measure the expected distance between σ and s, d(s, ) ≡

N 1 1 (1 − si σi s, ) N i=1 2

(12.35)

(note that i (1 − si σi )/2 is just the number of positions in which σ and s differ). For > 0, the coupling between σ and s is attractive; for < 0, it is repulsive. In fact, it is easy to show that d(s, ) is a decreasing function of . In the -coupling method, s is taken as a random variable, drawn from the (unperturbed) Boltzmann distribution. The rationale for this choice is that, in this way, s will point in the directions corresponding to quasi-states. The average distance induced by the -perturbation is then obtained, after averaging over s and over the choice of sample: 4 3 1 e−βE(s) d(s, ) . (12.36) d() ≡ E Z s There are two important differences between the -coupling method and the computation of the overlap distribution PN (q). (i) When PN (q) is computed, the two copies of the system are treated on an equal footing: they are independent and distributed according to Boltzmann law. In the -coupling method, one of the copies is distributed


¾

according to Boltzmann’s law, while the other follows a perturbed distribution depending on the first copy. (ii) In the -coupling method, the N → ∞ limit is taken at fixed . Therefore, the sum in eqn (12.36) can be dominated by values of the overlap q(s, σ) which would have been exponentially unlikely for the original (unperturbed) measure. When the N → ∞ limit P (q) is computed, such values of the overlap have a vanishing weight. The two approaches provide complementary information. Within a paramagnetic phase, d() remains a smooth function of in the neighbourhood of = 0, even after the N → ∞ limit has been taken: perturbing the system does not have any dramatic effect. But in a glass phase, d() becomes singular: it develops a discontinuity at = 0, which can be detected by defining ∆ = lim

lim d() − lim

→0+ N →∞

lim d() .

→0− N →∞

(12.37)

Notice that the limit N → ∞ is taken first: for finite N , there cannot be any discontinuity. One expects ∆ to be non-zero if and only if the system is in a ‘solid’ phase. In order to get an intuitive understanding of this, one can think of the process of adding a positive -coupling and then letting it tend to 0 as of a physical process. The system is first forced into an energetically favourable configuration (given by s). The forcing is then gradually removed and one checks whether any memory of the preparation is retained (∆ > 0) or whether the system ‘liquefies’ (∆ = 0). The advantage of the -coupling method with respect to the overlap distribution P (q) is twofold: • In some cases the dominant contribution to the Boltzmann measure comes from several distinct clusters, but a single cluster dominates over the others. More precisely, it may happen that the weights for subdominant clusters scale as wα = exp[−Θ(N θ )], with θ ∈]0, 1[. In this case, P (q) is a delta function and does not allow one to distinguish the state of the system from a purely paramagnetic phase. However, the -coupling method identifies the phase transition through a singularity in d() at = 0. • One can use the -coupling method to analyse a system undergoing a discontinuous transition when it is in a glass phase but in the T > Tc regime. In this case, the existence of clusters cannot be detected from P (q) because the Boltzmann measure is spread over an exponential number of them. This situation will be the subject of the next subsection. 12.3.5

1RSB clusters and the potential method

The 1RSB equilibrium glass phase corresponds to a condensation of the measure onto a small number of clusters of configurations. However, the most striking phenomenon is the appearance of clusters themselves. In the following chapters, we shall argue that this has important consequences for Monte Carlo dynamics, as well as for other algorithmic approaches to these systems. It turns out that the Boltzmann measure splits into clusters at a distinct temperature Td > Tc . In the region of temperatures [Tc , Td ], we shall say that the system is in a clustered phase, dynamical glass phase, or dynamical 1RSB phase. The phase transition at Td will be referred to

¾

Spin glasses

as the clustering or dynamical (glass) transition. In this regime, an exponential . number of clusters N = eN Σ carry a roughly equal weight. The rate of growth Σ is called the complexity3 or configurational entropy. The thermodynamic limit of the overlap distribution, P (q), does not show any signature of the clustered phase. In order to understand this point, it is useful to work through a toy example. Assume that the Boltzmann measure is entirely supported on exactly eN Σ sets of configurations in {±1}N (each set is a cluster), denoted by α = 1, . . . , eN Σ , and that the Boltzmann probability of each of these sets is w = e−N Σ . Assume furthermore that, for any two configurations belonging to the same cluster σ, σ ∈ α, their overlap is qσ,σ = q1 , whereas, if they belong to different clusters σ ∈ α, σ ∈ α , α = α , their overlap is qσ,σ = q0 < q1 . Although it might be difficult to actually construct such a measure, we shall neglect this problem for a moment, and compute the overlap distribution. The probability that two independent configurations fall into the same cluster is eN Σ w2 = e−N Σ . Therefore, we have PN (q) = (1 − e−N Σ ) δ(q − q0 ) + e−N Σ δ(q − q1 ) ,

(12.38)

which converges to δ(q − q0 ) as N → ∞: P (q) has a single delta function, as in the paramagnetic phase. A first signature of the clustered phase is provided by the -coupling method described in the previous subsection. The reason is very clear if we look at eqn (12.33): the -coupling ‘tilts’ the Boltzmann distribution in such a way that unlikely values of the overlap acquire a strictly positive probability. It is easy to compute the thermodynamic limit d∗ () ≡ limN →∞ d(). We get (1 − q0 )/2 for < c , (12.39) d∗ () = (1 − q1 )/2 for > c , where c = Σ/β(q1 − q0 ). As T ↓ Tc , clusters becomes less and less numerous and Σ → 0. Correspondingly, c ↓ 0 as the equilibrium glass transition is approached. The picture provided by this toy example is essentially correct, with the caveats that the properties of clusters will hold only within some accuracy and with high probability. Nevertheless, one expects d∗ () to have a discontinuity at some c > 0 for all temperatures in an interval ]Tc , Td ]. Furthermore, c ↓ 0 as T ↓ Tc . In general, the temperature Td computed through the -coupling method does not coincide with that of the clustering transition. The reason is easily understood. As illustrated by the above example, we are estimating the exponentially small probability P(q|s, J) that an equilibrated configuration σ has an overlap q with the reference configuration s, in a sample J. In order to do this, we compute the distance d() in a problem with a tilted measure. As we have seen already several times since Chapter 5, exponentially small (or large) quantities usually do not concentrate in probability, and d() may be dominated by exponentially rare samples. We have also learnt the cure for this problem: take logarithms! We therefore define4 the glass potential 3 This use of the term ‘complexity’ here, which is customary in statistical physics, should not be confused with its use in theoretical computer science. 4 One should introduce a resolution here, so that the overlap is actually constrained to be in some window around q. The width of this window can be let tend to 0 after N → ∞.


V (q) = − lim

N →∞

1 Es,J {log P(q|s, J)} . Nβ

¾

(12.40)

Here (as in the -coupling method), the reference configuration is drawn from the Boltzmann distribution. In other words, ⎫ ⎧ ⎬ ⎨ 1 e−βEJ (s) ( · · · ) . (12.41) Es,J ( · · · ) = EJ ⎭ ⎩ ZJ s . If, as expected, log P(q|s, J) concentrates in probability, one has P(q|s, J) = e−N V (q) with high probability. Exercise 12.11 Consider the following refined version of the toy model (12.38): P(q|s, J) = (1 − e−N Σ(s,J) )Gq0 (s,J);(b0 /N β) (q) + e−N Σ(s,J) ) Gq1 (s,J);(b1 /N β) (q), where Ga,b is a Gaussian distribution of mean a and variance b. We suppose that b0 , b1 are constants, but that Σ(s, J), q0 (s, J), q1 (s, J) fluctuate as follows: when J and s are distributed according to the correct joint distribution (12.41), then Σ(s, J), q0 (s, J), q1 (s, J) are independent Gaussian random variables of means Σ, q 0 , q 1 , respectively, and variances δΣ2 /N, δq02 /N, δq12 /N . Assuming for simplicity that δΣ2 < 2Σ, compute P (q) and d() for this model. Show that the glass potential V (q) is given by two arcs of parabolas as follows: j V (q) = min

ff

(q − q 0 )2 (q − q 1 )2 1 , + Σ 2b0 2b1 β

.

(12.42)

The glass potential V (q) has been computed using the replica method in only a small number of cases, mainly fully connected p-spin glasses. Here we shall just mention the qualitative behaviour that is expected on the basis of these computations. The result is summarized in Fig. 12.4. At small enough β, the glass potential is convex. When β is increased, one first encounters a value β∗ where V (q) ceases to be convex. When β > βd = 1/Td , V (q) develops a secondary minimum, at q = q1 (β) > q0 (β). This secondary minimum is in fact an indication of the existence of an exponential number of clusters, such that two configurations in the same cluster typically have an overlap q1 , while two configurations in distinct clusters have an overlap q0 . A little thought shows that the difference between the values of the glass potential at the two minima gives the complexity: V (q1 ) − V (q0 ) = T Σ. In models in which the glass potential has been computed exactly, the temperature Td computed in this way coincides with a dramatic slowing down of the relaxational dynamics. More precisely, a properly defined relaxation time for Glauber-type dynamics is finite for T > Td and diverges exponentially in the system size for T < Td . 12.3.6

Cloning and the complexity function

When the various clusters do not all have the same weight, the system is most appropriately described through a complexity function. Consider a cluster of configurations, called α. Its free energy Fα can be defined by restricting the partition function

¾

Spin glasses

0.7 0.6

V(q)

0.5 0.4 0.3 0.2 0.1 0 0

0.2

0.4

0.6 q

0.8

1

1.2

Fig. 12.4 Qualitative shapes of the glass potential V (q) at various temperatures. When the temperature is very high (not shown), V (q) is convex. Below T = Td , it develops a secondary minimum. The height difference between the two minima is V (q1 ) − V (q0 ) = T Σ. In the case shown here, q0 = 0 is independent of the temperature.

to configurations in the cluster α. One way of imposing this restriction is to choose a reference configuration σ 0 ∈ α, and restrict the Boltzmann sum to those configurations σ whose distance from σ 0 is smaller than N δ. In order to correctly identify clusters, one has to take (1 − q1 )/2 < δ < (1 − q∗ )/2, where q∗ > q1 is such that V (q∗ ) > V (q1 ). Let Nβ (f ) be the number of clusters such that Fα = N f (more precisely, this is an unnormalized measure attributing unit weight to the points Fα /N ). We expect it to satisfy a large-deviation principle of the form . Nβ (f ) = exp{N Σ(β, f )} .

(12.43)

The rate function Σ(β, f ) is the complexity function. If clusters are defined as above, with the cut-off δ in the appropriate interval, they are expected to be disjoint up to a subset of configurations of exponentially small Boltzmann weight. Therefore the total partition function is given by . −βFα . = eN [Σ(β,f )−βf ] df = eN [Σ(β,f∗ )−βf∗ ] , e (12.44) Z= α

where we have applied the saddle point method as in standard statistical-mechanics calculations (see Section 2.4). Here f∗ = f∗ (β) solves the saddle point equation ∂Σ/∂f = β. For several reasons, it is interesting to determine the full complexity function Σ(β, f ) as a function of f for a given inverse temperature β. The cloning method is a particularly efficient (although non-rigorous) way to do this computation. Here we sketch the basic idea: several applications will be discussed in the following chapters.


¾

One begins by introducing m identical ‘clones’ of the initial system. These are noninteracting except for the fact that they are constrained to be in the same cluster. In practice, one can constrain all their pairwise Hamming distances to be smaller than N δ, where (1 − q1 )/2 < δ < (1 − q∗ )/2. The partition function for the systems of m clones is therefore 3 4 exp − βE(σ (1) ) · · · − βE(σ (m) ) . (12.45) Zm = σ (1) ,...,σ (m)

where the prime reminds us that σ (1) , . . . σ (m) stay in the same cluster. By splitting the sum over the various clusters, we have Zm =

e−βE(σ

(1)

)···−βE(σ (m) )

α σ (1) ...σ (m) ∈α

=

% α

e−βE(σ)

&m .

(12.46)

σ∈α

At this point we can proceed as for the calculation of the usual partition function, and obtain ˆ ˆ . . e−βmFα = eN [Σ(β,f )−βmf ] df = eN [Σ(β,f )−βmf ] , (12.47) Zm = α

where f∗ = f∗ (β, m) solves the saddle point equation ∂Σ/∂f = βm. The free-energy density per clone of the cloned system is defined as Φ(β, m) = − lim

N →∞

1 log Zm . βmN

(12.48)

The saddle point estimate (12.47) implies that Φ(β, m) is related to Σ(β, f ) through Legendre transform: Φ(β, m) = f −

1 Σ(β, f ) , βm

∂Σ = βm . ∂f

(12.49)

If we forget that m is an integer, and admit that Φ(β, m) can be ‘continued’ to non-integer m, the complexity Σ(β, f ) can be computed from Φ(β, m) by inverting this Legendre transform. The similarity to the procedure used in the replica method is not fortuitous. Note, however, that replicas were introduced to deal with quenched disorder, while cloning is more general: it also applies to systems without disorder.

Exercise 12.12 In the REM, the natural definition of the overlap between two configurations i, j ∈ {1, . . . , 2N } is qi,j = I(i = j). Taking a configuration j0 as the reference, the -perturbed energy of a configuration j is E (, j) = Ej − N I(j = j0 ). (Note the extra

¾

Spin glasses

N multiplying , introduced in order to ensure that the new -coupling term is typically extensive.) √ (a) Consider the high-temperature phase where β < βc = 2 log 2. Show that the perturbed system has a phase transition at = (log 2)/β − β/4. (b) In the low temperature phase where β > βc , show that the phase transition takes place at = 0. Therefore, in the REM, clusters exist at any β, and every cluster is reduced to one single configuration: one must have Σ(β, f ) = log 2 − f 2 independently of β. Show that this is compatible with the cloning approach, through a computation of the potential Φ(β, m): ( Φ(β, m) =

12.4

2 − βm − log βm 4 √ − log 2

for m < for m >

βc , β βc . β

(12.50)

An example: The phase diagram of the SK model

Several mean-field models have been solved using the replica method. Sometimes a model may present two or more glass phases with different properties. Determining the phase diagram can be particularly challenging in those cases. A classical example is provided by the Sherrington–Kirkpatrick model with ferromagnetically biased couplings. As in the other examples in this chapter, this is a model for N Ising spins σ = (σ1 , . . . , σN ). The energy function is E(σ) = −

Jij σi σj ,

(12.51)

(i,j)

where (i, j) are unordered pairs, and the couplings Jij are i.i.d. Gaussian random variables with mean J0 /N and variance 1/N . In a sense, this model interpolates between the Curie–Weiss model treated in Section 2.5.2, corresponding to J0 → ∞, and the unbiased SK model considered in Chapter 8, corresponding to J0 = 0. The phase diagram can be plotted in terms of two parameters: the ferromagnetic bias J0 , and the temperature T . Depending on their values, the system can be found in one of four phases (see Fig. 12.5): the paramagnetic (P), ferromagnetic (F), symmetric spin glass (SG), or mixed ferromagnetic–spin glass (F-SG) phase. A simple characterization of these four phases can be obtained in terms of two quantities: the average magnetization and the overlap. In order to define these quantities, we must first observe that, since E(σ) = E(−σ), in the present model σi = 0 identically for all values of J0 , and T . In order to break this symmetry, we may add a magnetic-field term −B i σi and let B → 0 after taking the thermodynamic limit. We then define m = lim

lim Eσi B ,

B→0+ N →∞

q = lim

lim E(σi 2B ) ,

B→0+ N →∞

(12.52)

(which do not depend on i, because the coupling distribution is invariant under a permutation of the sites). In the P phase, one has m = 0, q = 0; in the SG phase, m = 0, q > 0, and in the F and F-SG phases, m > 0, q > 0.

An example: The phase diagram of the SK model

¾

2

T

1.5

P

1

F

0.5

SG F-SG

0 0

0.5

1 J0

1.5

2

Fig. 12.5 Phase diagram of the SK model in zero magnetic field. When the temperature T and the ferromagnetic bias J0 are varied, four distinct phases are encountered: paramagnetic (P), ferromagnetic (F), spin glass (SG) and mixed ferromagnetic–spin glass (F-SG). The full lines separate these various phases. The dashed line is the location of the Nishimori temperature.

A more complete description can be obtained in terms of the overlap distribution P (q). Because of the symmetry under spin inversion mentioned above, P (q) = P (−q) identically. The qualitative shape of P (q) in the thermodynamic limit is shown in Fig. 12.6. In the P phase, it consists of a single delta function with unit weight at q = 0: two independent configurations drawn from the Boltzmann distribution have, with high probability, an overlap close to 0. In the F phase, it is concentrated on two symmetric values q(J0 , T ) > 0 and −q(J0 , T ) < 0, each carrying a weight of one-half. We can summarize this behaviour by saying that a random configuration drawn from the Boltzmann distribution is found, with equal probability, in one of two different states. In the first one the local magnetizations are {mi }, and in the second one they are {−mi }. If one draws two independent configurations, they fall into the same state (corresponding to the overlap value q(J0 , T ) = N −1 i m2i ) or into opposite states (overlap −q(J0 , T )) with probability 1/2. In the SG phase, the support of P (q) is a symmetric interval [−qmax , qmax ], with qmax = qmax (J0 , T ). Finally, in the F-SG phase, the support is the union of two intervals [−qmax , −qmin ] and [qmin , qmax ]. In both the SG and the F-SG phase, the presence of a whole range of overlap values carrying non-vanishing probability, suggests the existence of a multitude of quasi-states (in the sense discussed in the previous section). In order to remove the degeneracy due to the symmetry under spin inversion, one sometimes defines an asymmetric overlap distribution by adding a magnetic-field term, and taking the thermodynamic limit as in eqn (12.52): P+ (q) = lim

lim PB (q) .

B→0+ N →∞

(12.53)

Somewhat surprisingly, it turns out that P+ (q) = 0 for q < 0, while P+ (q) = 2P (q) for q > 0. In other words, P+ (q) is equal to the distribution of the absolute value of the overlap.

¾

Spin glasses

P

P (q)

P (q)

F

q

q SG

P (q)

P (q)

q

M

q

Fig. 12.6 The typical shape of the distribution P (q) in each of the four phases of the SK model with ferromagnetically biased couplings.

Exercise 12.13 Consider the Curie–Weiss model in a magnetic field, (see Section 2.5.2). Draw the phase diagram and compute the asymptotic overlap distribution. Discuss its qualitative features for different values of the temperature and magnetic field.

We shall now add a few words for the reader interested in how one derives this diagram: Some of the phase boundaries have already been derived using the replica method in Exercise 8.12. The boundary P–F is obtained by solving the RS equation (8.68) for q, ω, x. The P–SG and F–M lines are obtained by the AT stability condition (8.69). Deriving the phase boundary between the SG and F–SG phases is much more challenging, because it separates glassy phases, and therefore it cannot be derived within the RS solution. It is known to be approximately vertical, but there is no simple expression for it. The Nishimori temperature is deduced from the condition (12.7), which gives TN = 1/J0 , and the line T = 1/J0 is usually called the ‘Nishimori line’. The internal energy per spin on this line is U/N = −J0 /2. Notice that the line does not enter any of the glass phases, as we know from general arguments. An important aspect of the SK model is that the appearance of the glass phase on the lines separating P from SG and on the line separating F from F–SG is a continuous transition. Therefore it is associated with a divergence of the non-linear susceptibility χSG . The following exercise, reserved for replica aficionados, sketches the main lines of the argument showing this.

Notes

¾

Exercise 12.14 P Let us see how to compute the non-linear susceptibility of the SK model, χSG = (β 2 /N ) i =j (σi σj − σi σj )2 , by the replica method. Show that

χSG

0 ! −1 X a b a b β2 X @ n = lim σi σi σj σj − n→0 N 2 i =j

(ab)

!−1

+ Z = N lim

n→0

n 4

X

!−1 X a b a c n σi σi σj σj 3 (abc) 1

σia σib σjc σjd A

(abcd)

e−N G(Q,λ) A(Q)

Y

(dQab dλab ) ,

(12.54)

(ab)

where we follow the notation of eqn (8.30), the sum over (a1 a2 . . . ak ) is understood to run over all the k-tuples of distinct replica indices, and A(Q) ≡

n 2

!−1

X (ab)

Q2ab −

!−1 X n Qab Qac + 3 (abc)

!−1

n 4

X

Qab Qcd .

(12.55)

(abcd)

Analyse the divergence of χSG along the following lines. The leading contribution to eqn (12.54) should come from the saddle point and be given, in the high-temperature phase, by A(Qab = q), where Qab = q is the RS saddle point. However, this contribution clearly vanishes when one takes the n → 0 limit. One must thus consider the fluctuations around the saddle point. Each of the terms of the type Qab Qcd in A(Q) gives a factor of 1/N times the appropriate matrix element of the inverse of the Hessian matrix. When this Hessian matrix is non-singular, these elements are all finite and one obtains a finite result in the N → ∞ limit. (The 1/N cancels the factor N in eqn (12.54).) When one reaches the AT instability line, the elements of the inverse of the Hessian matrix diverge, and therefore χSG also diverges.

Notes Lattice gas models of atomic systems, such as those discussed in the first two exercises, are discussed in statistical-physics textbooks; see, for instance, Ma (1985). The simple model of a glass in Exercise 12.3 was introduced and solved on sparse random graphs using the cavity method by Biroli and Mézard (2002). The order parameter for spin glasses defined by Edwards and Anderson (1975) is a dynamic order parameter which captures the long-time persistence of the spins. The static definition that we have introduced here should give the same result as the original dynamical definition (although of course we have no proof of this statement in general). A review of simulations of the Edwards–Anderson model can be found in Marinari et al. (1997). Mathematical results on mean-field spin glasses can be found in the book by Tanagrand (2003). A short recent survey is provided by Guerra (2005). Diluted spin glasses were introduced by Viana and Bray (1985).

¾

Spin glasses

The implications of the gauge transformation were derived by Hidetoshi Nishimori and his coworkers, and are explained in detail in his book (Nishimori, 2001). The notion of pure states in phase transitions, and the decomposition of Gibbs measures into pure states, is discussed in the book by Georgii (1988). We shall discuss this topic further in Chapter 22. The divergence of the spin glass susceptibility is especially relevant because this susceptibility can be measured in zero field. The experiments of Monod and Bouchiat (1982) present evidence of a divergence. This supports the existence of a spin glass transition in real (three-dimensional) spin glasses in zero magnetic field, at non-zero temperature. The existence of two transition temperatures Tc < Td was first discussed by Kirkpatrick and Wolynes (1987) and Kirkpatrick and Thirumalai (1987), who pointed out the relevance to the theory of structural glasses. In particular, Kirkpatrick and Thirumalai (1987) discussed the case of the p-spin glass. A review of this line of approach to structural glasses, and particularly its relevance to dynamical effects, is provided by Bouchaud et al. (1997). The -coupling method was introduced by Caracciolo et al. (1990). The idea of cloning in order to study the complexity function is due to Monasson (1995). Its application to studies of the glass transition without quenched disorder was developed by Mézard and Parisi (1999). The glass potential method was introduced by Franz and Parisi (1995).

13 Bridges: Inference and the Monte Carlo method We have seen in the last three chapters how problems with very different origins can be cast into the unifying framework of factor graph representations. The underlying mathematical structure, namely the locality of probabilistic dependencies between variables, is also present in many problems of probabilistic inference, which provides another unifying view of the field. We shall see through examples that several fundamental questions in physics, coding, and constraint satisfaction problems can be formulated as inference problems. On the other hand, locality is also an important ingredient that allows sampling from complex distributions using the Monte Carlo technique. It is thus very natural to apply this technique to our problems, and to study its limitations. We shall see here that the existence of large free-energy barriers, in particular in low-temperature regimes, considerably slows down the Monte Carlo sampling procedure. In Section 13.1, we present some basic terminology and simple examples of statistical inference problems. Statistical inference is an interesting field in itself, with many important applications (ranging from artificial intelligence to modelling and statistics). Here we emphasize the possibility of considering coding theory, statistical mechanics, and combinatorial optimization as inference problems. Section 13.2 develops a very general tool, the Markov chain Monte Carlo (MCMC) technique, already introduced in Section 4.5. This is often a very powerful approach. Furthermore, Monte Carlo sampling can be regarded as a statistical inference method, and Monte Carlo dynamics is a simple prototype of the local search strategies introduced in Sections 10.2.3 and 11.4. Many of the difficulties encountered in decoding, in constraint satisfaction problems, and in the study of glassy phases are connected to a dramatic slowing down of MCMC dynamics. We present the results of simple numerical experiments on some examples, and identify regions in the phase diagram where the MCMC slowdown implies poor performance as a sampling/inference algorithm. Finally, in Section 13.3, we explain a rather general argument to estimate the amount of time an MCMC algorithm as to be run in order to produce roughly independent samples with the desired distribution.

¾

Bridges: Inference and the Monte Carlo method p(r|c) \ c r=0 r=1

0 0.8 0.2

1 0.3 0.7

cloudy

0, 0 0.9 0.1

0, 1 0.2 0.8

1, 0 0.2 0.8

1, 1 0.1 0.9

0 0.1 0.9

1 0.7 0.3

sprinkler

rain p(w|r, s) \ r, s w=0 w=1

p(s|c) \ c s=0 s=1

wet

Fig. 13.1 The rain–sprinkler Bayesian network.

13.1 13.1.1

Statistical inference Bayesian networks

It is common practice in artificial intelligence and statistics, to formulate inference problems in terms of Bayesian networks. Although any such problem can also be represented in terms of a factor graph, it is worth to briefly introduce this alternative language. A famous toy example is the ‘rain–sprinkler’ network. Example 13.1 During a walk to the park, a statistician notices that the grass is wet. There are two possible reasons for this: either it rained during the night, or the sprinkler was activated in the morning to irrigate the lawn. Both events are, in turn, correlated with the weather condition in the last 24 hours. After a little thought, the statistician formalizes these considerations as the probabilistic model depicted in Fig. 13.1. The model includes four random variables: cloudy, rain, sprinkler, and wet, taking values in {0, 1} (which correspond, respectively, to false and true). The variables are organized as the vertices of a directed graph. A directed edge corresponds, intuitively, to a relation of causality. The joint probability distribution of the four variables is given in terms of conditional probabilities associated with the edges. Explicitly (variables are indicated by their initials), p(c, s, r, w) = p(c) p(s|c) p(r|c) p(w|s, r) .

(13.1)

The three conditional probabilities in this formula are given by the tables in Fig. 13.1. A ‘uniform prior’ is assumed on the event that the day was cloudy: p(c = 0) = p(c = 1) = 1/2. Given that wet grass was observed, we may want to know whether the most likely cause was the rain or the sprinkler. This amounts to computing the marginal probabilities c,r p(c, s, r, w = 1) , (13.2) p(s|w = 1) = c,r,s p(c, s , r, w = 1) c,s p(c, s, r, w = 1) . (13.3) p(r|w = 1) = c,r,s p(c, s , r, w = 1)

Statistical inference

cloudy

rain

¾

cloudy

sprinkler

rain

sprinkler

wet Fig. 13.2 Left: factor graph corresponding to the rain–sprinkler Bayesian network represented in Fig. 13.1. Right: factor graph for the same network under the observation of the variable w.

Using the numbers in Fig. 13.1, we get p(s = 1|w = 1) ≈ 0.63 and p(r = 1|w = 1) ≈ 0.48: the most likely cause of the wet grass is the sprinkler. In Fig. 13.2 we show the factor graph representation of eqn (13.1), and that corresponding to the conditional distribution p(c, s, r|w = 1). As is clear from the factor graph representation, the observation w = 1 induces a further dependency between the variables s and r, beyond that induced by their relation to c. The reader is invited to draw the factor graph associated with the marginal distribution p(c, s, r). In general, a Bayesian network describes the joint distributions of variables associated with the vertices of a directed acyclic graph G = (V, E). A directed graph is an ordinary graph with a direction (i.e. an ordering of the adjacent vertices) chosen on each of its edges, and no cycles. For such a graph, we say that a vertex u ∈ V is a parent of v, and write u ∈ π(v), if (u, v) is a (directed) edge of G. A random variable Xv is associated with each vertex v of the graph (for simplicity, we assume all the variables to take values in the same finite set X ). The joint distribution of {Xv , v ∈ V } is determined by the conditional probability distributions {p(xv |xπ(v) )}, where π(v) denotes the set of parents of the vertex v, and xπ(v) = {xu : u ∈ π(v)}: p(x) =

v∈π(G)

p(xv )

p(xv |xπ(v) ) ,

(13.4)

v∈G\π(G)

where π(G) denotes the set of vertices that have no parent in G. A general class of statistical inference problems can be formulated as follows. We are given a Bayesian network, i.e. a directed graph G plus the associated conditional probability distributions {p(xv |xπ(v) )}. A subset O ⊆ V of the variables is observed and takes values xO . The problem is to compute marginals of the conditional distribution p(xV \O |xO ). Given a Bayesian network G and a set of observed variable O, it is easy to obtain a factor graph representation of the conditional distribution p(xV \O |xO ), by a generalization of the procedure that we applied in Fig. 13.2. The general rule is as follows: (i) associate a variable node with each non-observed variable (i.e. each variable in xV \O ); (ii) for each variable in π(G)\O, add a degree-1 function node connected uniquely to

¾

Bridges: Inference and the Monte Carlo method

d1

f1

f2

f3

d2

f4

d3

d4

f5

d5

f6

f7

f8

Fig. 13.3 Left: toy version of the QMR-DT Bayesian network. Right: factor graph representation of the conditional distribution of the diseases d1 , . . . , d5 , given the findings f1 , . . . ,f8 .

that variable; (iii) for each non-observed vertex v which is not in π(G), add a function node and connect it to v and to all the parents of v; (iv) finally, for each observed variable u, add a function node and connect it to all the parents of u. Here is an example showing the practical utility of Bayesian networks. Example 13.2 The Quick Medical Reference, Decision Theoretic (QMR-DT) network is a two-level Bayesian network that was developed for automatic medical diagnosis. A schematic example is shown in Fig. 13.3. Variables in the top level, denoted by d1 , . . . , dN , are associated with diseases. Variables in the bottom level, denoted by f1 , . . . , fM , are associated with symptoms, or findings. Both diseases and findings are described by binary variables. An edge connects a disease di to a finding fa whenever that disease may be a cause for that finding. Such networks of implications are constructed on the basis of accumulated medical experience. The network is completed by two types of probability distribution. For each disease di , we are given an a priori occurrence probability p(di ). Furthermore, for each finding, we have a conditional probability distribution for that finding given a certain disease pattern. This usually takes the ‘noisy-OR’ form $ # N 1 (13.5) exp − θia di . p(fa = 0|d) = za i=1 This network is used for diagnostic purposes. The findings are set to values determined by observation of a patient. Given this pattern of symptoms, one aims to compute the marginal probability that any given disease is indeed present. 13.1.2

Inference in coding, physics, and optimization

Several of the problems encountered so far in this book can be recast in the language of statistical inference. Let us start with the decoding of error-correcting codes. As discussed in Chapter 6, in order to implement symbol MAP decoding, one has to compute the marginal distribution of input symbols, given the channel output. For instance, in the case of an LDPC code ensemble, dependencies between input symbols are induced by the parity check constraints. The joint probability distribution to be marginalized has

Statistical inference

¾

a natural graphical representation (although we have used factor graphs rather than Bayesian networks). The introduction of finite-temperature decoding allows us to view word MAP decoding as the zero-temperature limit case of a one-parameter family of inference problems. In statistical-mechanics models, one is mainly interested in the expectations and covariances of local observables with respect to the Boltzmann measure. For instance, the paramagnetic-to-ferromagnetic transition in an Ising ferromagnet, (see Section 2.5) can be located using the magnetization MN (β, B) = σi β,B . The computation of covariances, such as the correlation function Cij (β, B) = σi ; σj β,B , is a natural generalization of the simple inference problem discussed so far. Let us consider, finally, consider the case of combinatorial optimization. Assume, for the sake of definiteness, that a feasible solution is an assignment of the variables x = (x1 , x2 , . . . , xN ) ∈ X N and that its cost E(x) can be written as a sum of ‘local’ terms, Ea (xa ) . (13.6) E(x) = a

Here xa denotes a subset of the variables (x1 , x2 , . . . , xN ). Let µ∗ (x) denote the uniform distribution over the optimal solutions. The minimum energy can be computed as a sum of expectations with respect to this distribution: E∗ = a [ x µ∗ (x)Ea (xa )]. Of course, the distribution µ∗ (x) does not necessarily have a simple representation, and therefore the computation of E∗ can be significantly harder than simple inference.1 This problem can be overcome by ‘softening’ the distribution µ∗ (x). One possibility is to introduce a finite temperature and define µβ (x) = exp[−βE(x)]/Z, as already done in Section 4.6: if β is large enough, this distribution is concentrated on optimal solutions. At the same time, it has an explicit representation (apart from the value of the normalization constant Z) at any value of β. How large should β be in order to get a good estimate of E∗ ? The exercise below gives the answer under some rather general assumptions.

Exercise 13.1 Assume that the cost function E(x) takes integer values, and let U (β) = E(x)β . Owing to the form of eqn (13.6), the computation of U (β) is essentially equivalent to statistical inference. We assume, furthermore that ∆max = max[E(x) − E∗ ] is bounded by a polynomial in N . Show that 0≤

∂U 1 ≤ 2 ∆2max |X |N e−1/T , ∂T T

(13.7)

where T = 1/β. Deduce that by taking T = Θ(1/N ), one can obtain |U (β) − E∗ | ≤ ε for any fixed ε > 0.

1 Consider, for instance, the MAX-SAT problem, and let E(x) be the number of unsatisfied clauses under the assignment x. If the formula under study is satisfiable, then µ∗ (x) is proportional to the product of the characteristic functions associated with the clauses. In the opposite case, no explicit representation is known.

¾


In fact, a much larger temperature (smaller β) can be used in many important cases. We refer to Chapter 2 for examples in which U (β) = E∗ + E1 (N ) e−β + O(e−2β ), with E1 (N ) growing polynomially in N . In such cases one expects β = Θ(log N ) to be large enough.

13.2

The Monte Carlo method: Inference via sampling

Consider the statistical inference problem of computing the marginal probability µ(xi = x) from a joint distribution µ(x), x = (x1 , x2 , . . . , xN ) ∈ X N . Given L i.i.d. samples {x(1) , . . . , x(L) } drawn from the distribution µ(x), the desired marginal µ(xi = x) can be estimated as the the fraction of such samples for which xi = x. ‘Almost i.i.d.’ samples from p(x) can be produced, in principle, using the Markov chain Monte Carlo (MCMC) method of Section 4.5. Therefore the MCMC method can be viewed as a general-purpose inference strategy which can be applied in a variety of contexts. Notice that the locality of the interactions, expressed by the factor graph, is very useful since it allows one to generate ‘local’ changes in x (e.g. changing only one xi or a small number of them) easily. This will typically change the value of a few compatibility functions and hence produce only a small change in p(x) (i.e., in physical terms, in the energy of x). The possibility of generating, given x, a new configuration close in energy is important for the MCMC method to work. In fact, moves that increase the system energy by a large amount are typically rejected within MCMC rules. One should also be aware that sampling, for instance by the MCMC method, only allows one to estimate marginals or expectations which involve a small subset of variables. It would be very hard, for instance, to estimate the probability of a particular configuration x through the number L(x) of its occurrences in the samples. The reason is that at least 1/µ(x) samples would be required for us to obtain any accuracy, and this is typically a number exponentially large in N . 13.2.1

LDPC codes

Consider a code C from one of the LDPC ensembles introduced in Chapter 11, and assume that it has been used to communicate over a binary-input memoryless symmetric channel with transition probability Q(y|x). As shown in Chapter 6 (see eqn (6.5)), the conditional distribution of the channel input x, given the output y, reads µy (x) ≡ P(x|y) =

N 1 I(x ∈ C) Q(yi |xi ) . Z(y) i=1

(13.8)

We can use the explicit representation of the code membership function to write µy,β (x) =

M N 1 I(xia1 ⊕ · · · ⊕ xiak = 0) Q(yi |xi ) . Z(y) a=1 i=1

(13.9) (i)

in order to implement symbol MAP decoding, we must compute the marginals µy (xi ) of this distribution. Let us see how this can be done in an approximate way via MCMC sampling.


¾

Unfortunately, the simple MCMC algorithms introduced in Section 4.5 (single bit flip with acceptance test satisfying detailed balance) cannot be applied in the present case. In any reasonable LDPC code, each variable xi is involved in at least one parity check constraint. Suppose that we start the MCMC algorithm from a random configuration x distributed according to eqn (13.9). Since x has a non-vanishing probability, it satisfies all the parity check constraints. If we propose a new configuration where bit xi is flipped, this configuration will violate all the parity check constraints involving xi . As a consequence, such a move will be rejected by any rule satisfying detailed balance. The Markov chain is therefore reducible (each codeword forms a separate ergodic component), and useless for sampling purposes. In good codes, this problem is not easily cured by allowing for moves that flip more than a single bit. The reason is that a number of bits greater than or equal to the minimum distance must be flipped simultaneously. But as we saw in Section 11.2, if C is drawn from an LDPC ensemble with variable nodes of degree greater than or equal to 2, its minimum distance diverges with the block length (logarithmically if the minimum degree is 2, linearly if this is at least 3). Large moves of this type are likely to Nbe rejected, because they imply a large, uncontrolled variation in the likelihood i=1 Q(yi |xi ). A way out of this dilemma consists in ‘softening’ the parity check constraint by introducing a ‘parity check temperature’ β and the associated distribution µy,β (x) =

N M 1 −βEa (xia ...xia ) 1 k e Q(yi |xi ) . Z(y, β) a=1 i=1

(13.10)

Here the energy term Ea (xia1 . . . xiak ) takes values 0 if xia1 ⊕ · · · ⊕ xiak = 0, and 1 otherwise. In the limit β → ∞, the distribution (13.10) reduces to eqn (13.9). The (i) idea is now to estimate the marginals of eqn (13.10), µy,β (xi ), via MCMC sampling and then to use the decoding rule (β)

(i)

xi (y) ≡ arg max µy,β (xi ) . xi

(13.11)

For any finite β, this prescription is certainly suboptimal with respect to symbol MAP decoding. In particular, the distribution (13.10) gives non-zero weight to words x which do not belong to the codebook C. On the other hand, one may hope that for β large enough, the above prescription achieves a close-to-optimal bit error rate. We can simplify the above strategy further by giving up the objective of approx(i) imating the marginal µy,β (xi ) within any prescribed accuracy. Instead, we run the Glauber single-bit-flip MCMC algorithm for a fixed computer time and extract an (i) estimate of µy,β (xi ) from this run. Figure 13.4 shows the results of Glauber dynamics executed for 2LN steps starting from a uniformly random configuration. At each step, a bit i is chosen uniformly at random and flipped with a probability wi (x) =

µy,β (x(i) ) , µy,β (x(i) ) + µy,β (x)

(13.12)

where x is the current configuration and x(i) is the configuration obtained from x by flipping the i-th bit.

¾


0.35 0.30 0.25

0.35

β = 0.50 β = 0.75 β = 1.00 β = 1.50 β = 2.00

0.30 0.25

0.20

L = 50 L = 200 L = 800 L = 3200

Pb

Pb

0.20

0.15

0.15

0.10

0.10

0.05

0.05

0.00 0.00

0.05

0.10 p

0.15

0.00 0.00

0.05

0.10

0.15

p

Fig. 13.4 Decoding LDPC codes from the (3, 6) ensemble, used over a bynary symmetric channel with flip probability p, using MCMC sampling. The bit error rate is plotted versus p. The block length was fixed to N = 2000, and the number of sweeps was 2L. Left: for L = 100 and several values of the effective inverse temperature β. Right: improvement of the performance as the number of sweeps increases at fixed β = 1.5.

Exercise 13.2 Derive an explicit expression for wi (x), and show that this probability can be computed with a number of operations independent of the block length.

In this context, one often refers to a sequence of N successive updates as a sweep (on average, one flip is proposed for each bit in a sweep). The value of xi is recorded in each of the last L sweeps, and the decoder output is xi = 0 or xi = 1 depending on which value occurs more often in this record. The data in Fig. 13.4 refers to communication over a binary symmetric channel with flip probability p. In the left frame, we have fixed L = 100 and used several values of β. At small β, the resulting bit error rate is almost indistinguishable from that in the absence of coding, namely Pb = p. As β increases, parity checks are enforced more and more strictly and the error-correcting capabilities improve at low noise. The behaviour is qualitatively different for larger noise levels: for p 0.05, the bit error rate increases with β. The reason for this change is essentially dynamical. The Markov chain used for sampling from the distribution (13.10) decorrelates more and more slowly from its initial condition. Since the initial condition is uniformly random, thus yielding Pb = 1/2, the bit error rate obtained through our algorithm approaches 1/2 at large β (and above a certain threshold of p). This interpretation is confirmed by the data in the right frame of the figure. We shall see in Chapter 15 that in the large-block-length limit, the threshold for errorless bit MAP decoding in this case is predicted to be pc ≈ 0.101. Unfortunately, because of its slow dynamics, our MCMC decoder cannot be used in practice if the channel noise is close to this threshold. The sluggish dynamics of the single spin-flip MCMC for the distribution (13.10) is partially related to its reducibility for the model with hard constraints (13.9). An initial intuitive picture is as follows. At large β, codewords correspond to isolated


¾

‘lumps’ of probability with µy,β (x) = Θ(1), separated by improbable regions such that µy,β (x) = Θ(e−2β ) or smaller. In order to decorrelate, the Markov chain must spend a long time (at least of the order of the minimum distance of the code) in an improbable region, and this happens only very rarely. This rough explanation is neither complete nor entirely correct, but we shall refine it in Chapters 15 and 21. 13.2.2

Ising model

Some of the basic mechanisms responsible for the slowing down of Glauber dynamics can be understood with simple statistical-mechanics models. In this subsection, we consider a ferromagnetic Ising model with an energy function σi σj . (13.13) E(σ) = − (ij)∈G

Here G is an ordinary graph with N vertices, whose precise structure will depend on the particular example. The Monte Carlo method is applied to the problem of sampling from the Boltzmann distribution µβ (σ) at an inverse temperature β. As in the previous subsection, we focus on Glauber (or heat bath) dynamics, but rescale the time: in an infinitesimal interval dt, a flip is proposed with probability N dt at a uniformly random site i. The flip is accepted with the usual heat bath probability (here, σ is the current configuration and σ (i) is the configuration obtained by flipping the spin σi ), wi (σ) =

µβ (σ (i) ) . µβ (σ) + µβ (σ (i) )

(13.14)

Let us consider equilibrium dynamics first. We assume, therefore, that the initial configuration σ(0) is sampled from the equilibrium distribution µβ ( · ) and ask how many Monte Carlo steps must be performed in order to obtain an effectively independent random configuration. A convenient way of monitoring the equilibrium dynamics consists in computing the time correlation function CN (t) ≡

N 1 σi (0)σi (t) . N i=1

(13.15)

Here the average · is taken with respect to the realization of the Monte Carlo dynamics, as well as with respect to the initial state σ(0). Note that (1 − C(t))/2 is the average fraction of spins which differ in the configurations σ(0) and σ(t). One therefore expects C(t) to decrease with t, asymptotically reaching 0 when σ(0) and σ(t) are well decorrelated (in this model, each spin is equally likely to take a value +1 or −1). The reader may wonder how one can sample σ(0) from the equilibrium (Boltzmann) distribution. As already suggested in Section 4.5, within the Monte Carlo approach one can obtain an ‘almost’ equilibrium configuration by starting from an arbitrary configuration and running the Markov chain for sufficiently many steps. In practice, we initialize our chain from a uniformly random configuration (i.e. a β = 0, or infinitetemperature, equilibrium configuration) and run the dynamics for tw sweeps. We call

¾


1

1 L=3 L=5 L=7 L=9

0.6

L=3 L=4 L=7 L=9

0.8 C(t)

C(t)

0.8

0.6

0.4

0.4

0.2

0.2

0

0 1

2

4

8

16 t

32

64

1

10

100

1000

10000 100000

t

Fig. 13.5 Equilibrium correlation function for an Ising model on a two-dimensional grid of side L. Left: high temperature, T = 3. Right: low temperature, T = 2.

the configuration obtained after this process σ(0) and we then run for t more sweeps in order to measure C(t). The choice of tw is, of course, crucial: in general, the above procedure will produce a configuration σ(0) whose distribution is not the equilibrium one, and depends on tw . The measured correlation function will also depend on tw . Determining how large tw must be in order to obtain a good enough approximation to C(t) is a subject of intense theoretical work. A simple empirical rule is to measure C(t) for a given large tw , then double tw and check that nothing has changed.

Exercise 13.3 Using the instructions above, the reader is invited to write an MCMC code for the Ising model on a general graph and reproduce the data in the following examples.

Example 13.3 We begin by considering an Ising model on a two-dimensional grid of side L, with periodic boundary conditions. The vertex set is {(x1 , x2 ) : 1 ≤ xa ≤ L}. Edges join any two vertices at a (Euclidean) distance one, and also join the vertices (L, x2 ) to (1, x2 ) and (x1 , L) to (x1 , 1). We denote by CL (t) the correlation function for such a graph. In Chapter 2, we saw that this model √ undergoes a phase transition at a critical temperature 1/βc = Tc = 2/ log(1 + 2) ≈ 2.269185. The correlation functions plotted in Fig. 13.5 are representative of the qualitative behaviour in the hightemperature (left) and low-temperature (right) phases. At high temperature, CL (t) depends only mildly on the linear size L of the system. As L increases, the correlation function rapidly approaches a limit curve C(t) which decreases from 1 to 0 on a finite time scale (more precisely, for any δ > 0, CL (t) decreases below δ in a time which is bounded as L → ∞).


¾

1 N=10 N=20 N=30

N=10 N=20 N=30

0.8

0.2

C(t)

C(t)

0.3

0.6 0.4

0.1 0.2 0

0 1

2

4

8 t

16

32

1

10

100

1000

10000 100000

t

Fig. 13.6 Equilibrium correlation function for an Ising model on random graphs from the ensemble GN (2, M ), with M = 2N . Left: high temperature, T = 5. Right: low temperature, T = 2.

Fig. 13.7 A rooted ternary tree with n = 4 generations and N = 40 vertices.

At low temperature, there exists no limiting curve C(t) decreasing from 1 to 0 such that CL (t) → C(t) as L → ∞. The time required for the correlation function CL (t) to get close to 0 is much larger than in the high-temperature phase. More importantly, it depends strongly on the system size. This suggests that strong cooperative effects are responsible for the slowing down of the dynamics.

Example 13.4 We take G as a random graph from the ensemble GN (2, M ), with M = N α. As we shall see in Chapter 17, this model undergoes a phase transition when N → ∞ at a critical temperature βc , satisfying the equation 2α tanh β = 1. In Fig. 13.6, we present numerical data for a few values of N , and α = 2 (corresponding to a critical temperature Tc ≈ 3.915230). The curves presented here are representative of the high-temperature and lowtemperature phases. As in the previous example, the relaxation time scale depends strongly on the system size at low temperature.

¾


1

0.4

n=4 n=5 n=6 n=7

0.8

n=4 n=5 n=6 n=7

C(t)

C(t)

0.6

0.6 0.4

0.2

0.2

0

0 1

2

4

8

16

32

1

5

t

25

125

625

t

Fig. 13.8 Equilibrium correlation function for a ferromagnetic Ising model on a regular ternary tree. Left: high temperature, T = 2. Right: low temperature, T = 1.25.

Example 13.5 We take G as a rooted ternary tree with n generations (see Fig. 13.7). This tree contains N = (3n − 1)/(3 − 1) vertices and N − 1 edges. Using the methods of Chapter 17, one can show that this model undergoes a phase transition at a critical temperature βc , which satisfies the equation 3(tanh β)2 = 1. We therefore get Tc ≈ 1.528651. In this case the dynamics of a spin depends strongly upon its distance from the root. In particular, leaf spins are much less constrained than the others. In order to single out the ‘bulk’ behaviour, we modify the definition of the correlation function (13.15) by averaging only over the spins σi in the first n = 3 generations. We keep n fixed as n → ∞. As in the previous examples, CN (t) (see Fig. 13.8) has a well-defined N → ∞ limit in the high-temperature phase, and is strongly size-dependent at low temperature. We can summarize the last three examples by comparing the size dependence of the relaxation times scale in the respective low-temperature phases. A simple way to define such a time scale is to look for the smallest time such that C(t) decreases below some given threshold: τ (δ; N ) = min{ t > 0 such that CN (t) ≤ δ} .

(13.16)

In Fig. 13.9, we plot estimates obtained from the data presented in the previous examples, using δ = 0.2, and restricting ourselves to the data in the low-temperature (ferromagnetic) phase. The size dependence of τ (δ; N ) is very clear. However, it is much stronger for the random-graph and square-grid cases (and, in particular, in the former of these) than for the tree case. In fact, it can be shown that, in the ferromagnetic phase, ⎧ random graph, ⎨ exp{Θ(N √ )} (13.17) τ (δ; N ) = exp{Θ( N )} square lattice, ⎩ exp{Θ(log N )} tree. Section 13.3 will explain the origin of these different behaviours.


¾

random graph grid tree

10000

τ

1000 100 10 1 1

5

25

125 N

625

3125

Fig. 13.9 Size dependence of the relaxation time in ferromagnetic Ising models in the low-temperature phase. The various symbols refer to the various families of graphs considered in Examples 13.3–13.5. 0.20

0.012

0.16

0.008

0.12 E

0.004

0.000

0.08

0

1

2

0.04 0.00 0

1

2

3

4

5

6

α

Fig. 13.10 Minimization of the number of unsatisfied clauses in random 3-SAT formulae via Glauber dynamics. The number of variables N = 1000 was kept fixed. Here we take T = 0.25 and, from top to bottom L = 2.5 × 103 , 5 × 103 , 104 , 2 × 104 , 4 × 104 , and 8 × 104 iterations. The inset show the small-α regime in greater detail.

13.2.3

MAX-SAT

Given a satisfiability formula over N Boolean variables (x1 , . . . , xN ) = x, xi ∈ {0, 1}, the MAX-SAT optimization problem requires one to find a truth assignment which satisfies the largest number of clauses. We denote by xa the set of variables involved in the a-th clause, and by Ea (xa ) a function of the truth assignment that takes the

¾


0.20

0.012

0.16

0.008

0.12 E

0.004

0.000

0.08

0

1

2

0.04 0.00 0

1

2

3

4

5

6

α

Fig. 13.11 Same numerical simulation as in Fig. 13.10. Here we fix L = 4 · 104 and take, from top to bottom at large α, T = 0.15, 0.20, 0.25, 0.30, 0.35.

value 0 if the clause is satisfied and 1 otherwise. With these definitions, the MAX-SAT problem can be rephrased as the problem of minimizing an energy function of the form E(x) = a Ea (xa ), and we can therefore apply the general approach discussed after eqn (13.6). We thus consider the Boltzmann distribution µβ (x) = exp[−βE(x)]/Z and try to sample a configuration from µβ (x) at large enough β using the MCMC method. The assignment x ∈ {0, 1}N is initialized uniformly at random. At each time step, a variable index i is chosen uniformly at random and the corresponding variable is flipped according to the heat bath rule wi (x) =

µβ (x(i) ) . µβ (x) + µβ (x(i) )

(13.18)

As above, x(i) denotes the assignment obtained from x by flipping the i-th variable. The algorithm is stopped after LN steps (i.e. L sweeps), and the current assignment x∗ (and the corresponding cost E∗ = E(x∗ )) is stored in memory. In Fig. 13.10 and 13.11, we present the outcome of such an algorithm when applied to random 3-SAT formulae from the ensemble SATN (3, M ) with α = M/N . Here we focus on the mean cost E∗ of the returned assignment. One expects that, as N → ∞ with fixed L, the cost scales as E∗ = Θ(N ), and order-N fluctuations of E∗ away from the mean become exponentially unlikely. At low enough temperature, the behaviour depends dramatically on the value of α. For small α, E∗ /N is small and grows rather slowly with α. Furthermore, it seems to decrease to 0 as β increases. Our strategy is essentially successful and finds an (almost) satisfying assignment. Above α ≈ 2 to 3, E∗ /N starts to grow more rapidly with α, and does not seem to vanish as β → ∞. Even more striking is the behaviour as the number of sweeps L increases. In the small-α regime, E∗ /N decreases rapidly to some roughly L-independent saturation

Free-energy barriers

¾

E(x)

∆E Egs 0

x

Fig. 13.12 Random walk in a double-well energy landscape. After how many steps is the walker distributed (approximately) according to the equilibrium distribution?

value, already reached after about 103 sweeps. At large α there seems also to be an asymptotic value, but this is reached much more slowly, and even after 105 sweeps there is still space for improvement.

13.3


The examples above show that the time scale required for a Monte Carlo algorithm to produce (approximately) statistically independent configurations may vary wildly depending on the particular problem at hand. The same is true if we consider the time required in order to generate a configuration distributed approximately according to the equilibrium distribution, starting from an arbitrary initial condition. There exist various sophisticated techniques for estimating these time scales analytically, at least in the case of unfrustrated problems. In this section, we discuss a simple argument which is widely used in statistical physics as well as in probability theory, that of free-energy barriers. The basic intuition can be conveyed by simple examples. Example 13.6 Consider a particle moving on the line of integers, and denote its position by x ∈ Z. Each point x on the line has an energy E(x) ≥ Egs associated with it, as depicted in Fig. 13.12. At each time step, the particle attempts to move to one of the adjacent positions (either to the right or to the left) with probability 1/2. If we denote by x the position that the particle is trying to move to, the move is accepted according to the Metropolis rule 3 4 (13.19) w(x → x ) = min e−β[E(x )−E(x)] , 1 . The equilibrium distribution is of course given by Boltzmann’s law µβ (x) = exp[−βE(x)]/Z(β). Suppose we start with, say, x = 10. How many steps must we wait for in order that the random variable x will be distributed according to µβ (x)? It is intuitively

¾


2 k

3

1

2 1

n +1 Fig. 13.13 How much time does a random walk need to explore this graph?

clear that, in order to equilibrate, the particle must spend some amount of time both in the right and in the left well, and therefore it must visit the site x = 0. At equilibrium, this is visited on average a fraction µβ (0) of the time. This gives an estimate of the time needed to see the system change from one well to the other: τ≈

1 . µβ (0)

(13.20)

One is often interested in the low-temperature limit of τ . Assuming that E(x) diverges fast enough as |x| → ∞, the leading exponential behaviour of Z is . . Z(β) = e−βEgs , and therefore τ = exp{β∆E}, where ∆E = E(0) − Egs is the energy barrier to be crossed in order to pass from one well to the other. A low-temperature . asymptotics of the type τ = exp{β∆E} is referred to as an Arrhenius law.

Exercise 13.4 Consider a random walk on the graph in Fig. 13.13 (two cliques with n + 1 vertices, joined by a k-fold degenerate edge). At each time step, the walker chooses one of the adjacent edges uniformly at random and moves along it to the next node. What is the stationary distribution µeq (x), x ∈ {1, . . . , 2n}? Show that the probability to be at node 1 is 12 (k + n − 1)/(n2 + k − n). Suppose we start with a walker distributed according to µeq (x). Using an argument similar to that in the previous example, estimate the number of time steps τ that one must wait for in order to obtain an approximatively independent value of x. Show that τ 2n when n k, and interpret this result. In this case the k-fold degenerate edge joining the two cliques is called a bottleneck, and one speaks of an entropy barrier.

In order to obtain a precise mathematical formulation of the intuition gained in the preceding examples, we must define what we mean by ‘relaxation time’. We shall focus here on ergodic continuous-time Markov chains on a finite state space X . Such a chain is described by its transition rates w(x → y). If, at time t, the chain is in a state x(t) = x ∈ X , then, for any y = x, the probability that the chain is in a state y, ‘just after’ time t, is


P {x(t + dt) = y | x(t) = x} = w(x → y)dt .

¾

(13.21)

Such a chain is said to satisfy the detailed balance condition (or to be reversible) if, for any x = y, µ(x)w(x → y) = µ(y)w(y → x).

Exercise 13.5 Consider a discrete-time Markov chain and modify it as follows. Instead of waiting a unit time ∆t between successive steps, wait an exponentially distributed random time (i.e. ∆t is a random variable with pdf p(∆t) = exp(−∆t)). Show that the resulting process is a continuous-time Markov chain. What are the corresponding transition rates?

Let x → O(x) be an observable (a function of the state), define the shorthand O(t) = O(x(t)), and assume x(0) to be drawn from the stationary distribution. If the chain satisfies the detailed balance condition, one can show that the correlation function CO (t) = O(0)O(t) − O(0)O(t) is non-negative and monotonically decreasing, and that CO (t) → 0 as t → ∞. The exponential autocorrelation time for the observable O, denoted by τO,exp , is defined by 1 1 = − lim log CO (t) . t→∞ t τO,exp

(13.22)

The time τO,exp depends on the observable and tells us how fast its autocorrelation function decays to 0: CO (t) ∼ exp(−t/τO,exp ). It is meaningful to look for the ‘slowest’ observable and to define the exponential autocorrelation time (also called the inverse spectral gap, or, for brevity, the relaxation time) of the Markov chain as τexp = sup { τO,exp } . O

(13.23)

The idea of a bottleneck, and its relationship to the relaxation time, is clarified by the following theorem. Theorem 13.7 Consider an ergodic continuous-time Markov chain with state space X , and with transition rates {w(x → y)} satisfying detailed balance with respect to the stationary distribution µ(x). Given any two disjoint sets of states A, B ⊂ X , define the probability flux between them as W (A → B) = x∈A, y∈B µ(x) w(x → y). Then τexp ≥ 2

µ(x ∈ A) µ(x ∈ A) . W (A → X \A)

(13.24)

In other words, a lower bound on the correlation time can be constructed by looking for ‘bottlenecks’ in the Markov chain, i.e. partitions of the configuration space into two subsets. The lower bound will be good (and the Markov chain will be slow) if each of the subsets carries a reasonably large probability at equilibrium, but jumping from one to the other is unlikely.

¾


E(x)

∆E1 ∆ E2

Egs 0

x

Fig. 13.14 Random walk in an asymmetric double well.

Example 13.8 Consider a random walk in the double-well energy landscape of Fig. 13.12, where we confine the random walk to some large interval [−M : M ] in order to have a finite state space. Let us apply Theorem 13.7, by taking A = {x ≥ 0}. We have W (A → X \A) = µβ (0)/2 and, by symmetry, µβ (x ∈ A) = 12 (1 + µβ (0)). The inequality (13.24) yields τexp ≥

1 − µβ (0)2 . µβ (0)

(13.25)

Expanding the right-hand side in the low-temperature limit, we get τexp ≥ 2 eβ∆E (1 + Θ(e−cβ )).

Exercise 13.6 Apply Theorem 13.7 to a random walk in the asymmetric double-well energy landscape of Fig. 13.14. Does the Arrhenius law τexp ∼ exp(β∆E) apply to this case? What is the relevant energy barrier ∆E?

Exercise 13.7 Apply Theorem 13.7 to estimate the relaxation time of a random walk on the graph in Exercise 13.4.

Example 13.9 Consider Glauber dynamics for an Ising model on a two-dimensional L × L grid, with periodic boundary conditions, discussed earlier in Example 13.3. In the ferromagnetic phase, the distribution of the total magnetization M(σ) ≡ i σi , N = L2 , is concentrated around the values ±N M+ (β), where M+ (β) is the


¾

spontaneous magnetization. It is natural to expect that the bottleneck will correspond to the global magnetization changing sign. Assuming, for instance, that L is odd, let us define A = {σ : M(σ) ≥ 1} ,

A¯ = X \A = {σ : M(σ) ≤ −1} .

Using the symmetry under spin reversal, Theorem 13.7 yields ⎧ ⎫−1 ⎨ ⎬ τexp ≥ 2 µβ (σ) w(σ → σ (i) ) . ⎩ ⎭

(13.26)

(13.27)

σ :M(σ)=1i :σi =1

A good estimate of this sum can be obtained by noticing that, for any σ, w(σ → σ (i) ) ≥ w(β) ≡ 12 (1 − tanh 4β). Moreover, for any σ entering the sum, there 2 i such that σi = +1. We therefore get τexp ≥ are exactly (L + 1)/2 sites 2 (L w(β) σ :M(σ)=1 µβ (σ))−1 One suggestive way of writing this lower bound consists in defining a constrained free energy as follows: ⎧ ⎫ ⎨ ⎬ 1 exp[−βE(σ)] . (13.28) FL (m; β) ≡ − log ⎩ ⎭ β σ : M(σ)=m

If we denote the usual (unconstrained) free energy by FL (β), our lower bound can be written as τexp ≥

1 L2 w(β)

exp{β[FL (1; β) − FL (β)]} .

(13.29)

Apart from the pre-exponential factors, this expression has the same form as the Arrhenius law, the energy barrier ∆E being replaced by a ‘free-energy barrier’ ∆FL (β) ≡ FL (1; β) − FL (β). We are left with the task of estimating ∆FL (β). Let us start by considering the β → ∞ limit. In this regime, FL (β) is dominated by the all-plus and all-minus configurations, with energy Egs = −2L2 . Analogously, FL (1; β) is dominated by the lowest-energy configurations satisfying the constraint M(σ) = 1. An example of such a configuration is that shown in Fig. 13.15, whose energy is E(σ) = −2L2 + 2(2L + 2). Of course, all configurations obtained from that in Fig. 13.15, through a translation, rotation, or spin inversion have the same energy. We therefore find ∆FL (β) = 2(2L + 2) + Θ(1/β). It is reasonable to guess (and it can be proved rigorously) that the size dependence of ∆FL (β) remains unchanged through the whole low-temperature phase: ∆FL (β) 2γ(β)L ,

(13.30)

where the surface tension γ(β) is strictly positive at any β > βc , and vanishes as β ↓ βc . This, in turn, implies the following lower bound on the correlation time: τexp ≥ exp{2βγ(β)L + o(L)} .

(13.31)

This bound matches the numerical simulations in the previous section and can be proved to give the correct asymptotic size dependence.

¾


Fig. 13.15 Ferromagnetic Ising model on a 9 × 9 grid with periodic boundary conditions. Open circles correspond to σi = +1, and filled circles to σi = −1. The configuration shown here has an energy E(σ) = −122 and a magnetization M(σ) = +1.

Exercise 13.8 Consider the ferromagnetic Ising model on a random graph from GN (2, M ) that we studied in Example 13.4, and assume, for definiteness, N even. Arguing as above, show that τexp ≥ CN (β) exp{β[FN (0; β) − FN (β)]} .

(13.32)

Here FN (m; β) is the free energy of the model constrained to M(σ) = m, FN (β) is the unconstrained free energy, and CN (β) is a constant which grows, with high probability, slower than exponentially with N . For a graph G, let δ(G) be the minimum number of bicoloured edges if we colour half of the vertices red and half blue. Show that FN (0; β) − FN (β) = 2δ(GN ) + Θ(1/β) .

(13.33)

The problem of computing δ(G) for a given graph G is referred to as the balanced minimum cut (or graph-partitioning) problem, and is known to be NP-complete. For a random graph in GN (2, M ), it is known that δ(GN ) = Θ(N ) with high probability in the limit N → ∞, M → ∞, with α = M/N fixed and α > 1/2. (If α < 1/2, the graph does not contain a giant component and, obviously, δ(G) = o(N ).) This claim can be substantiated P through the following calculation. Given a spin configuration σ = (σ1 , . . . , σN ) with i σi = 0, let ∆G (σ) be the number of edges (i, j) in G such that σi = σj . Then P {δ(G) ≤ n} = P {∃ σ such that ∆G (σ) ≤ n} ≤

n X

E NG,m ,

(13.34)

m=0

where NG,m denotes the number of spin configurations with ∆G (σ) = m. Show that E NG,m =

! !−M N N N/2 2

M m

!„

N2 4

«m "

N 2

!

N2 − 4

#M −m .

(13.35)

Estimate this expression for large N and M with α = M/N fixed, and show that it implies δ(G) ≥ c(α)N with high probability, where c(α) > 0 for α > 1/2.

Notes

¾

Exercise 13.9 Repeat the same arguments as above for the case of a regular ternary tree described in Example 13.5, and derive a bound of the form (13.32). Show that, at low temperature, the Arrhenius law holds, i.e. τexp ≥ exp{β∆EN + o(β)}. How does ∆EN behave for large N ? [Hint: An upper bound can be obtained by constructing a sequence of configurations from the all-plus to the all-minus ground state such that any two consecutive configurations differ by a single spin flip.]

Notes For introductions to Bayesian networks, see Jordan (1998) and Jensen (1996). Bayesian inference was proved to be NP-hard by Cooper (1990). In Dagm and Luby (1993), it was shown that approximate Bayesian inference remains NP-hard. Decoding of LDPC codes via Glauber dynamics was considered by Franz et al. (2002). Satisfiability problems were considered by Svenson and Nordahl (1999). The Arrhenius law and the concept of an energy barrier (or ‘activation energy’) were discovered by the Swedish chemist Svante Arrhenius in 1889, in his study of chemical kinetics. An introduction to the analysis of Markov chain Monte Carlo methods (with special emphasis on enumeration problems) and their equilibration time can be found in Jerrum and Sinclair (1996) and Sinclair (1998). Geometric techniques for bounding the spectral gap are also discussed in Diaconis and Stroock (1991) and Diaconis and Saloff-Coste (1993). A book in preparation (Aldous and Fill, 2008) provides a complete exposition of the subject from a probabilistic point of view. For a mathematical-physics perspective, we refer to the lectures by Martinelli (1999). The rather basic Theorem 13.7 can be found in any of these references. For an early treatment of the Glauber dynamics of an Ising model on a tree, see Henley (1986). This paper contains a partial answer to Exercise 13.9. Rigorous estimates for this problem have been proved by Berger et al. (2005) and Martinelli et al. (2004).


Part IV Short-range correlations


14 Belief propagation Consider the ubiquitous problem of computing marginals of a graphical model with N variables x = (x1 , . . . , xN ) taking values in a finite alphabet X . The naive algorithm, which sums over all configurations, takes a time of order |X |N . The complexity can be reduced dramatically when the underlying factor graph has some special structure. One extreme case is that of tree factor graphs. On trees, marginals can be computed in a number of operations which grows linearly with N . This can be done through a ‘dynamic programming’ procedure that recursively sums over all variables, starting from the leaves and progressing towards the ‘centre’ of the tree. Remarkably, such a recursive procedure can be recast as a distributed ‘messagepassing’ algorithm. Message-passing algorithms operate on ‘messages’ associated with edges of the factor graph, and update them recursively through local computations done at the vertices of the graph. The update rules that yield exact marginals on trees have been discovered independently in several different contexts: statistical physics (under the name ‘Bethe–Peierls approximation’), coding theory (the ‘sum–product’ algorithm), and artificial intelligence (‘belief propagation’, BP). Here we shall adopt the artificial-intelligence terminology. This chapter gives a detailed presentation of BP and, more generally, messagepassing procedures, which provide one of the main building blocks that we shall use throughout the rest of the book. It is therefore important that the reader has a good understanding of BP. It is straightforward to prove that BP computes marginals exactly on tree factor graphs. However, it was found only recently that it can be extremely effective on loopy graphs as well. One of the basic intuitions behind this success is that BP, being a local algorithm, should be successful whenever the underlying graph is ‘locally’ a tree. Such factor graphs appear frequently, for instance in error-correcting codes, and BP turns out to be very powerful in this context. However, even in such cases, its application is limited to distributions such that far-apart variables become approximately uncorrelated. The onset of long-range correlations, typical of the occurrence of a phase transition, leads generically to poor performance of BP. We shall see several applications of this idea in the following chapters. We introduce the basic ideas in Section 14.1 by working out two simple examples. The general BP equations are stated in Section 14.2, which also shows how they provide exact results on tree factor graphs. Section 14.3 describes an alternative messagepassing procedure, the max-product (or, equivalently, min-sum) algorithm, which can be used in optimization problems. In Section 14.4, we discuss the use of BP in graphs with loops. In the study of random constraint satisfaction problems, BP messages

¾

Belief propagation

11 00 00 11 j

B ν →j

ν j←

11 00 00j 11

Fig. 14.1 Top: the factor graph of a one-dimensional Ising model in an external field. Bottom: the three messages arriving at site j describe the contributions to the probability distribution ν→j ), the right chain (b νj← ), and the external field B. of σj due to the left chain (b

become random variables. The study of their distribution provides a large amount of information about such instances and can be used to characterize the corresponding phase diagram. The time evolution of these distributions is known under the name of ‘density evolution’, and the fixed-point analysis of them is done by the replicasymmetric cavity method. Both are explained in Section 14.6.

14.1 14.1.1

Two examples Example 1: Ising chain

Consider the ferromagnetic Ising model on a line. The variables are Ising spins (σ1 , . . . , σN ) = σ, with σi ∈ {+1, −1}, and their joint distribution takes the Boltzmann form µβ (σ) =

1 −βE(σ) e , Z

E(σ) = −

N −1 i=1

σi σi+1 − B

N

σi .

(14.1)

i=1

The corresponding factor graph is shown in Figure 14.1. Let us now compute the marginal probability distribution µ(σj ) of spin σj . We shall introduce three ‘messages’ arriving at spin j, representing the contributions to µ(σj ) from each of the function nodes which are connected to i. More precisely, we define $ # j−1 j−1 1 exp β σi σi+1 + βB σi , ν →j (σj ) = Z→j σ ...σ i=1 i=1 1 j−1 ⎧ ⎫ −1 N ⎨ N ⎬ 1 ν j← (σj ) = exp β σi σi+1 + βB σi . (14.2) ⎩ ⎭ Zj← σ ...σ j+1

N

i=j

i=j+1

Messages are understood to be probability distributions and thus to be normalized. In the present case, the constants Z→j , Zj← are set by the conditions ν →j (+1) + ν →j (−1) = 1, and ν j← (+1) + ν j← (−1) = 1. In the following, when dealing with normalized distributions, we shall avoid writing the normalization constants explicitly

Two examples

¾

and instead use the symbol ∼ = to denote ‘equality up to a normalization’. With this notation, the first of the above equations can be rewritten as $ # j−1 j−1 ∼ (14.3) ν →j (σj ) = exp β σi σi+1 + βB σi . σ1 ...σj−1

i=1

i=1

By rearranging the summation over spins σi , i = j, the marginal µ(σj ) can be written as (14.4) µ(σj ) ∼ = ν →j (σj ) eβBσj ν j← (σj ) . In this expression, we can interpret each of the three factors as a ‘message’ sent to j from one of the three function nodes connected to the variable j. Each message coincides with the marginal distribution of σj in a modified graphical model. For instance, ν →j (σj ) is the distribution of σj in the graphical model obtained by removing all of the factor nodes adjacent to j except for the one on its left (see Fig. 14.1). This decomposition is interesting because the various messages can be computed iteratively. Consider, for instance, ν →i+1 . It is expressed in terms of ν →i as ν →i (σ ) eβσ σ+βBσ . (14.5) ν →i+1 (σ) ∼ = σ

Furthermore, ν →1 is the uniform distribution over {+1, −1}: ν →1 (σ) = 12 for σ = ±1. Equation (14.5) allows one to compute all of the messages ν →i , i ∈ {1, . . . , N }, in O(N ) operations. A similar procedure yields ν i← , by starting from the uniform distribution ν N ← and computing ν i−1← from ν i← recursively. Finally, eqn (14.4) can be used to compute all of the marginals µ(σj ) in linear time. All of the messages are distributions over binary variables and can thus be parameterized by a single real number. One popular choice for such a parameterization is to use the log-likelihood ratio1 u→i ≡

ν →i (+1) 1 log . 2β ν →i (−1)

(14.6)

In statistical-physics terms, u→i is an ‘effective (or local) magnetic field’: ν →i (σ) ∼ = eβu→i σ . Using this definition (and noticing that it implies ν →i (σ) = 21 (1+ σ tanh(βu→i ))), eqn (14.5) becomes u→i+1 = f (u→i + B) ,

(14.7)

where the function f (x) is defined as f (x) =

1 atanh [tanh(β) tanh(βx)] . β

(14.8)

The mapping u → f (u + B) is differentiable, with its derivative bounded by tanh β < 1. Therefore the fixed-point equation u = f (u + B) has a unique solution u∗ , and u→i goes to u∗ when i → ∞. Consider a very long chain, and a node 1 Note that our definition differs by a factor 1/2β from the standard definition of the log-likelihood in statistics. This factor is introduced to make contact with statistical-physics definitions.

¾

Belief propagation

1

2

2

1 a

a 0 3

b

0 c

6

3

b

6 c

4

5

4

5

Fig. 14.2 Left: a simple parity check code with seven variables and three checks. Right: the factor graph corresponding to the problem of finding the sent codeword, given a received message.

in the bulk j ∈ [εN, (1 − ε)N ]. Then, as N → ∞, both u→j and uj← converge to u∗ , so that σj → tanh[β(2u∗ + B)]. This is the bulk magnetization. If, on the other hand, we consider a spin on the boundary, we get a smaller magnetization σ1 = σN → tanh[β(u∗ + B)].

Exercise 14.1 Use the recursion (14.7) to show that, when N and j go to infinity, σj = M + O(λj , λN −j ), where M = tanh(2u∗ + B) and λ = f (u∗ + B). Compare this with the treatment of the one-dimensional Ising model in Section 2.5.

The above method can be generalized to the computation of joint distributions of two or more variables. Consider, for instance, the joint distribution µ(σj , σk ), for k > j. Since we already know how to compute the marginal µ(σj ), it is sufficient to consider the conditional distribution µ(σk |σj ). For each of the two values of σj , the conditional distribution of σj+1 , . . . , σN takes a form analogous to eqn (14.1) but with σj fixed. Therefore, the marginal µ(σk |σj ) can be computed through the same algorithm as before. The only difference is in the initial condition, which becomes ν →j (+1) = 1, ν →j (−1) = 0 (if we condition on σj = +1) and ν →j (+1) = 0, ν →j (−1) = 1 (if we condition on σj = −1).

Exercise 14.2 Compute the correlation function σj σk , when j, k ∈ [N ε, N (1 − ε)] and N → ∞. Check that when B = 0, σj σk = (tanh β)|j−k| . Find a simpler derivation of this last result.

Two examples

14.1.2

¾

Example 2: A tree-parity-check code

Our second example deals with a decoding problem. Consider the simple linear code whose factor graph is reproduced in the left frame of Fig. 14.2. It has a block length N = 7, and the codewords satisfy the three parity check equations x0 ⊕ x1 ⊕ x2 = 0 , x0 ⊕ x3 ⊕ x4 = 0 , x0 ⊕ x5 ⊕ x6 = 0 .

(14.9) (14.10) (14.11)

One of the codewords is sent through a channel of the type BSC(p), defined earlier. Assume that the received message is y = (1, 0, 0, 0, 0, 1, 0). The conditional distribution for x to be the transmitted codeword, given the received message y, takes the usual form µy (x) = P(x|y): µy (x) ∼ = I(x0 ⊕ x1 ⊕ x2 = 0)I(x0 ⊕ x3 ⊕ x4 = 0)I(x0 ⊕ x5 ⊕ x6 = 0)

6

Q(yi |xi ) ,

i=0

where Q(0|0) = Q(1|1) = 1 − p and Q(1|0) = Q(0|1) = p. The corresponding factor graph is drawn in the right frame of Fig. 14.2. In order to implement symbol MAP decoding, (see Chapter 6), we need to compute the marginal distribution of each bit. The computation is straightforward, but it is illuminating to recast it as a message-passing procedure similar to that in the Ising chain example. Consider, for instance, bit x0 . We start from the boundary. In the absence of the check a, the marginal of x1 would be ν1→a = (1 − p, p) (we use here the convention of writing distributions ν(x) over a binary variable as two-dimensional vectors (ν(0), ν(1))). This is interpreted as a message sent from variable 1 to check a. Variable 2 sends an analogous message ν2→a to a (in the present example, this happens to be equal to ν1→a ). Knowing these two messages, we can compute the contribution to the marginal probability distribution of variable x0 arising from the part of the factor graph containing the whole branch connected to x0 through the check a: I(x0 ⊕ x1 ⊕ x2 = 0) ν1→a (x1 )ν2→a (x2 ) . (14.12) ν a→0 (x0 ) ∼ = x1 ,x2

Clearly, ν a→0 (x0 ) is the marginal distribution of x0 in a modified factor graph that does not include either of the factor nodes b or c, and in which the received symbol y0 has been erased. This is analogous to the messages ν →j (σj ) used in the Ising chain example. The main difference is that the underlying factor graph is no longer a line, but a tree. As a consequence, the recursion (14.12) is no longer linear in the incoming messages. Using the rule (14.12), and analogous ones for ν b→0 (x0 ) and ν c→0 (x0 ), we obtain ν a→0 = (p2 + (1 − p)2 , 2p(1 − p)) , ν b→0 = (p2 + (1 − p)2 , 2p(1 − p)) , ν c→0 = (2p(1 − p), p2 + (1 − p)2 ) .

¾

Belief propagation

The marginal probability distribution of the variable x0 is finally obtained by taking into account the contributions of each subtree, together with the channel output for bit x0 : νb→0 (x0 )

νc→0 (x0 ) µ(x0 ) ∼ = Q(y0 |x0 ) ν a→0 (x0 )

2 2 2 2 ∼ = 2p (1 − p)[p + (1 − p) ] , 4p2 (1 − p)3 [p2 + (1 − p)2 ] . In particular, the MAP decoding of the symbol x0 is always x0 = 0 in this case, for any p < 1/2. An important fact emerges from this simple calculation. Instead of performing a summation over 27 = 128 configurations, we were able to compute the marginal at x0 by doing six summations (one for every factor node a, b, c and for every value of x0 ), each one over two summands (see eqn (14.12)). Such complexity reduction was achieved by merely rearranging the order of sums and multiplications in the computation of the marginal.

Exercise 14.3 Show that the message ν0→a (x0 ) is equal to (1/2, 1/2), and deduce that µ(x1 ) ∼ = ((1 − p), p).

14.2

Belief propagation on tree graphs

We shall now define belief propagation and analyse it in the simplest possible setting: tree-graphical models. In this case, it solves several computational problems in an efficient and distributed fashion. 14.2.1

Three problems

Let us consider a graphical model such that the associated factor graph is a tree (we call this model a tree-graphical model). We use the same notation as in Section 9.1.1. The model describes N random variables (x1 , . . . , xN ) ≡ x taking values in a finite alphabet X , whose joint probability distribution has the form µ(x) =

M 1 ψa (x∂a ) , Z a=1

(14.13)

where x∂a ≡ {xi | i ∈ ∂a}. The set ∂a ⊆ [N ], of size |∂a|, contains all variables involved in constraint a. We shall always use indices i, j, k, . . . for the variables and a, b, c, . . . for the function nodes. The set of indices ∂i involves all function nodes a connected to i. When the factor graph has no loops, the following are among the basic problems that can be solved efficiently with a message-passing procedure: 1. Compute the marginal distributions of one variable, µ(xi ), or the joint distribution of a small number of variables. 2. Sample from µ(x), i.e. draw independent random configurations x with a distribution µ(x).


¾

3. Compute the partition function Z or, equivalently, in statistical-physics language, the free entropy log Z. These three tasks can be accomplished using belief propagation, which is an obvious generalization of the procedure exemplified in the previous section. 14.2.2

The BP equations

Belief propagation is an iterative ‘message-passing’ algorithm. The basic variables on which it acts are messages associated with directed edges on the factor graph. For each edge (i, a) (where i is a variable node and a a function node) there exist, at the t-th (t) (t) iteration, two messages νi→a and ν a→i . Messages take values in the space of probability (t) (t) distributions over the single-variable space X . For instance, νi→a = {νi→a (xi ) : xi ∈ (t) (t) X }, with νi→a (xi ) ≥ 0 and xi νi→a (xi ) = 1. In tree-graphical models, the messages converge when t → ∞ to fixed-point values (see Theorem 14.1). These coincide with single-variable marginals in modified graphical (∞) models, as we saw in the two examples in the previous section. More precisely, νi→a (xi ) is the marginal distribution of variable xi in a modified graphical model which does not include the factor a (i.e. the product in eqn (14.13) does not include a). Analogously, (∞) ν a→i (xi ) is the distribution of xi in a graphical model where all factors in ∂i except a have been erased. Messages are updated through local computations at the nodes of the factor graph. By local we mean that a given node updates the outgoing messages on the basis of incoming ones at previous iterations. This is a characteristic feature of message-passing algorithms; the various algorithms in this family differ in the precise form of the update equations. The belief propagation (BP), or sum–product, update rules are (t) (t+1) ν b→j (xj ) , (14.14) νj→a (xj ) ∼ = b∈∂j\a (t) ν a→j (xj )

∼ =

ψa (x∂a )

x∂a\j

(t)

νk→a (xk ) .

(14.15)

k∈∂a\j

It is understood that, when ∂j \ a is an empty set, νj→a (xj ) is the uniform distribution. Similarly, if ∂a \ j is empty, then ν a→j (xj ) = ψa (xj ). A pictorial illustration of these rules is provided in Fig. 14.3. A BP fixed point is a set of t-independent mes(t) (t) sages νi→a = νi→a , ν a→i = ν a→i which satisfy eqns (14.14) and (14.15). From these, one obtains 2|E| equations (one equation for each directed edge of the factor graph) relating 2|E| messages. We shall often refer to these fixed-point conditions as the BP equations. After t iterations, one can estimate the marginal distribution µ(xi ) of variable i using the set of all incoming messages. The BP estimate is: (t−1) (t) ν a→i (xi ) . (14.16) νi (xi ) ∼ = a∈∂i

In writing the update rules, we have assumed that the update is done in parallel at all the variable nodes, then in parallel at all function nodes, and so on. Clearly, in this

¾

Belief propagation

b

j j

a k

a

(t+1)

Fig. 14.3 Left: the portion of the factor graph involved in the computation of νj→a (xj ). (t)

This message is a function of the ‘incoming messages’ νbb→j (xj ), with b = a. Right: the portion of the factor graph involved in the computation of the ‘incoming messages’

(t) νk→a (xk ),

(t) νba→j (xj ).

This message is a function of

with k = j.

case, the iteration number must be incremented either at variable nodes or at factor nodes, but not necessarily at both. This is what happens in eqns (14.14) and (14.15). Other update schedules are possible and sometimes useful. For the sake of simplicity, however, we shall stick to the parallel schedule. In order to fully define the algorithm, we need to specify an initial condition. It is a widespread practice to set initial messages to the uniform distribution over X (i.e. (0) νi→a (xi ) = 1/|X |). On the other hand, it can be useful to explore several distinct (random) initial conditions. This can be done by defining some probability measure P over the space M(X ) of distributions over X (i.e. the |X |-dimensional simplex) and (0) taking νi→a ( · ) as i.i.d. random variables with distribution P. The BP algorithm can be applied to any graphical model, irrespective of whether the factor graph is a tree or not. One possible version of the algorithm is as follows. BP 1: 2: 3: 4: 5: 6: 7: 8: 9: 10:

(graphical model (G, ψ), accuracy , iterations tmax ) Initialize BP messages as i.i.d. random variables with distribution P; For t ∈ {0, . . . , tmax } For each (j, a) ∈ E Compute the new value of ν a→j using eqn (14.15); For each (j, a) ∈ E Compute the new value of νj→a using eqn (14.14); Let ∆ be the maximum message change; If ∆ < return current messages; End-For; Return ‘Not Converged’;


¾

Among all message-passing algorithms, BP is uniquely characterized by the property of computing exact marginals on tree-graphical models. Theorem 14.1. (BP is exact on trees) Consider a tree-graphical model with diameter t∗ (which means that t∗ is the maximum distance between any two variable nodes). Then: 1. Irrespective of the initial condition, the BP update equations (14.14) and (14.15) converge after at most t∗ iterations. In other words, for any edge (ia), and any (t) (t) ∗ ∗ , ν a→i = ν a→i . t > t∗ , νi→a = νi→a 2. The fixed-point messages provide the exact marginals: for any variable node i, and (t) any t > t∗ , νi (xi ) = µ(xi ). Proof As exemplified in the previous section, on tree factor graphs BP is just a clever way to organize the sum over configurations to compute marginals. In this sense, the theorem is obvious. We shall sketch a formal proof here, leaving a few details to the reader. Given a directed edge i → a between a variable i and a factor node a, we define T(i → a) as the subtree rooted on this edge. This is the subtree containing all nodes w which can be connected to i by a non-reversing path2 which does not include the edge (i, a). Let t∗ (i → a) be the depth of T(i → a) (the maximal distance from a leaf to i). (t) We can show that, for any number of iterations t > t∗ (i → a), the message νi→a coincides with the marginal distribution of the root variable with respect to the graphical model T(i → a). In other words, for tree graphs, the interpretation of BP messages in terms of modified marginals is correct. This claim is proved by induction on the tree depth t∗ (i → a). The base step of the induction is trivial: T(i → a) is the graph formed by the unique node i. By definition, (t) for any t ≥ 1, νi→a (xi ) = 1/|X | is the uniform distribution, which coincides with the marginal of the trivial graphical model associated with T(i → a). The induction step is easy as well. Assuming the claim to be true for t∗ (i → a) ≤ τ , we have to show that it holds when t∗ (i → a) = τ + 1. To this end, take any t > τ + 1 (t+1) (t) and compute νi→a (xi ) using eqns (14.14) and (14.15) in terms of messages νj→b (xj ) in the subtrees for b ∈ ∂i \ a and j ∈ ∂b \ i. By the induction hypothesis, and since (t) the depth of the subtree T (j → b) is at most τ , νj→b (xj ) is the root marginal in such a subtree. It turns out that by combining the marginals at the roots of the subtrees T(j → b) using eqns (14.14) and (14.15), we can obtain the marginal at the root of T(i → a). This proves the claim. 14.2.3

Correlations and energy

The use of BP is not limited to computing one-variable marginals. Suppose we want to compute the joint probability distribution µ(xi , xj ) of two variables xi and xj . Since BP already enables to compute µ(xi ), this task is equivalent to computing the 2 A non-reversing path on a graph G is a sequence of vertices ω = (j , j , . . . , j ) such that n 0 1 (js , js+1 ) is an edge for any s ∈ {0, . . . , n − 1}, and js−1 = js+1 for s ∈ {1, . . . , n − 1}.

¿¼¼

Belief propagation

conditional distribution µ(xj | xi ). Given a model that factorizes as in eqn (14.13), the conditional distribution of x = (x1 , . . . , xN ) given xi = x takes the form µ(x|xi = x) ∼ =

M

ψa (x∂a ) I(xi = x) .

(14.17)

a=1

In other words, it is sufficient to add to the original graph a new function node of degree 1 connected to variable node i, which fixes xi = x. One can then run BP on (t) the modified factor graph and obtain estimates νj (xj |xi = x) for the conditional marginal of xj . This strategy is easily generalized to the joint distribution of any number m of variables. The complexity, however, grows exponentially in the number of variables involved, since we have to condition over |X |m−1 possible assignments. Happily, for tree-graphical models, the marginal distribution of any number of variables admits an explicit expression in terms of messages. Let FR be a subset of function nodes, let VR be the subset of variable nodes adjacent to FR , let R be the induced subgraph, and let xR be the corresponding variables. Without loss of generality, we shall assume R to be connected. Further, we denote by ∂R the subset of function nodes that are not in FR but are adjacent to a variable node in VR . Then, for a ∈ ∂R, there exists a unique i ∈ ∂a ∩ VR , which we denote by i(a). It then follows immediately from Theorem 14.1, and its characterization of messages, that the joint distribution of variables in R is µ(xR ) =

1 ∗ ψa (x∂a ) ν a→i(a) (xi(a) ) , ZR a∈FR

(14.18)

a∈∂R

∗ where ν a→i ( · ) are the fixed-point BP messages.

Exercise 14.4 Let us use the above result to write the joint distribution of the variables along a path in a tree factor graph. Consider two variable nodes i, j, and let R = (VR , FR , ER ) be the subgraph induced by the nodes along the path between i and j. For any function node a ∈ R, denote by i(a) and j(a) the variable nodes in R that are adjacent to a. Show that the joint distribution of the variables along this path, xR = {xl : l ∈ VR }, takes the form µ(xR ) =

Y 1 Y ˜ ψa (xi(a) , xj(a) ) ψ˜l (xl ) . ZR a∈F R

(14.19)

l∈VR

In other words, µ(xR ) factorizes according to the subgraph R. Write expressions for the compatibility functions ψã ( · , · ), ψ˜l ( · ) in terms of the original compatibility functions and the messages going from ∂R to VR .

A particularly useful case arises in the computation of the internal energy. In physics problems, the compatibility functions in eqn (14.13) take the form ψa (x∂a ) = e−βEa (x∂a ) , where β is the inverse temperature and Ea (x∂a ) is the energy function


¿¼½

characterizing constraint a. Of course, any graphical model can be written in this form (allowing for the possibility of Ea (x∂a ) = +∞ in the case of hard constraints), adopting for instance the convention β = 1, which we shall use from now on. The internal energy U is the expectation value of the total energy: U =−

x

µ(x)

M

log ψa (x∂a ) .

(14.20)

a=1

This can be computed in terms of BP messages using eqn (14.18) with FR = {a}. If, further, we use eqn (14.14) to express products of check-to-variable messages in terms of variable-to-check ones, we get M 1 ∗ ψa (x∂a ) log ψa (x∂a ) νi→a (xj ) , (14.21) U =− Z a=1 a x i∈∂a

∂a

∗ where Za ≡ x ψa (x∂a ) i∈∂a νi→a (xj ). Notice that in this expression the internal ∂a energy is a sum of ‘local’ terms, one for each compatibility function. On a loopy graph, eqns (14.18) and (14.21) are no longer valid, and, indeed, BP ∗ ∗ , ν a→i }. However, one can does not necessarily converge to fixed-point messages {νi→a replace fixed-point messages with BP messages after any number t of iterations and take these as definitions of the BP estimates of the corresponding quantities. From eqn (14.18), one obtains an estimate of the joint distribution of a subset of variables, which we shall call ν (t) (xR ), and from (14.21), an estimate of the internal energy.

14.2.4

Entropy

Remember that the entropy of a distribution µ over X V is defined as H[µ] = − x µ(x) log µ(x). In a tree-graphical model, the entropy, like the internal energy, has a simple expression in terms of local quantities. This follows from an important decomposition property. Let us denote by µa (x∂a ) the marginal probability distribution of all the variables involved in the compatibility function a, and by µi (xi ) the marginal probability distribution of variable xi . Theorem 14.2 In a tree-graphical model, the joint probability distribution µ(x) of all of the variables can be written in terms of the marginals µa (x∂a ) and µi (xi ) as µa (x∂a ) µi (xi )1−|∂i| . (14.22) µ(x) = a∈F

i∈V

Proof The proof is by induction on the number M of factors. Equation (14.22) holds for M = 1 (since the degrees |∂i| are all equal to 1). Assume that it is valid for any factor graph with up to M factors, and consider a specific factor graph G with M + 1 factors. Since G is a tree, it contains at least one factor node such that all its adjacent variable nodes have degree 1, except for at most one of them. Call such a factor node a, and let i be the only neighbour with degree larger than one (the case in which no such neighbour exists is treated analogously). Further, let x∼ be the vector of variables in

¿¼¾

Belief propagation

G that are not in ∂a \ i. Then (writing Pµ ( · ) for a probability under the distribution µ), the Markov property together with the Bayes rule yields Pµ (x) = Pµ (x∼ )Pµ (x|x∼ ) = Pµ (x∼ )Pµ (x∂a\i |xi ) = Pµ (x∼ )µa (x∂a )µi (xi )−1 . (14.23) The probability Pµ (x∼ ) can be written as P(x∼ ) ∼ = ψã (xi ) b∈F \a ψb (x∂b ), where ψa (x ). As the factor ψã has degree one, it can be erased and ψã (xi ) = ∂a

x∂a\i

incorporated into another factor as follows: take one of the other factors connected to i, c ∈ ∂i \ a, and change it to ψ˜c (x∂c ) = ψc (x∂c )ψã (xi ). In the reduced factor graph, the degree of i is smaller by one and the number of factors is M . Using the induction hypothesis, we get µb (x∂b ) µj (xj )1−|∂j| . (14.24) Pµ (x∼ ) = µi (xi )2−|∂i| b∈F \a

j∈V \i

The proof is completed by putting together eqns (14.23) and (14.24). As an immediate consequence of eqn (14.22), the entropy of a tree-graphical model can be expressed as sums of local terms: µa (x∂a ) log µa (x∂a ) − (1 − |∂i|) µi (xi ) log µi (xi ) . (14.25) H[µ] = − a∈F

i∈V

It is also easy to express the free entropy Φ = log Z in terms of local quantities. Recalling that Φ = H[µ]−U [µ] (where U [µ] is the internal energy given by eqn (14.21)), we get Φ = F[µ], where µa (x∂a ) − µa (x∂a ) log (1 − |∂i|)µi (xi ) log µi (xi ) . (14.26) F[µ] = − ψa (x∂a ) a∈F

i∈V

Expressing local marginals in terms of messages, via eqn (14.18), we can in turn write the free entropy as a function of the fixed-point messages. We introduce the function F∗ (ν), which yields the free entropy in terms of 2|E| messages ν = {νi→a ( · ), ν a→i ( · )}: F∗ (ν) = Fa (ν) + Fi (ν) − Fia (ν) , (14.27) a∈F

where

⎡

Fa (ν) = log ⎣

ψa (x∂a )

x∂a

Fai (ν) = log

i∈V

⎤ νi→a (xi )⎦ , Fi (ν) = log

i∈∂a

(ia)∈E

ν b→i (xi )

,

xi b∈∂i

νi→a (xi )

νa→i (xi )

.

(14.28)

xi

It is not hard to show that, by evaluating this functional at the BP fixed point ν ∗ , one gets F∗ (ν ∗ ) = F[µ] = Φ, thus recovering the correct free entropy. The function


4

a 1

4

a

d

0 2/3

3 c 2 b

d

1 2/3 0 1/3 1

e

b

5

1

¿¼¿

1/3

2

c

1 1/2

3

1 1/2

1/3 1 2/3 1

e

1/2 1/2

5

Fig. 14.4 Left: the factor graph of a small instance of the satisfiability problem with five variables and five clauses. A dashed line means that the variable appears negated in the adjacent clause. Right: the set of fixed-point BP messages for the uniform measure over solutions of this instance. All messages are normalized, and we show their weights for the value True. For any edge (a, i) (a being the clause and i the variable), the weight corresponding to the message νba→i is shown above the edge, and the weight corresponding to νi→a below the edge.

F∗ (ν) defined in eqn (14.27) is known as the Bethe free entropy (when multiplied by a factor −1/β, it is called the Bethe free energy). The above observations are important enough to be highlighted in a theorem. Theorem 14.3. (the Bethe free entropy is exact on trees) Consider a tree-graphical ∗ ∗ , ν a→i } be the fixedmodel. Let {µa , µi } denote its local marginals, and let ν ∗ = {νi→a ∗ point BP messages. Then Φ = log Z = F[µ] = F∗ (ν ). Notice that in the above statement, we have used the correct local marginals in F[ · ] and the fixed-point messages in F∗ ( · ). In Section 14.4 we shall reconsider the Bethe free entropy for more general graphical models, and regard it as a function over the space of all ‘possible’ marginals/messages.

Exercise 14.5 Consider the instance of the satisfiability problem shown in Fig. 14.4, left. Show by exhaustive enumeration that it has only two satisfying assignments, x = (0, 1, 1, 1, 0) and (0, 1, 1, 1, 1). Rederive this result using BP. Namely, compute the entropy of the uniform measure over satisfying assignments, and check that its value is indeed log 2. The BP fixed point is shown in Fig. 14.4, right.

Exercise 14.6 In many systems some of the function nodes have degree 1 and amount to a local redefinition of the reference Q measure over Q X . It is then convenient to single out these factors. Let us write µ(x) ∼ = a∈F ψa (x∂a ) i∈V ψi (xi ), where the second product runs over degree-1 function nodes (indexed by the adjacent variable node), and the factors ψa have degree at least 2. In the computation of F∗ , the introduction of ψi adds N extra factor nodes and subtracts N extra ‘edge’ terms corresponding to the edge between the variable node i and the function node corresponding to ψi .

¿

Belief propagation

Show that these two effects cancel, and that the net effect is to replace the variable-node contribution in eqn (14.27) with Fi (ν) = log

" X

ψi (xi )

xi

Y

# νba→i (xi ) .

(14.29)

a∈∂i

The problem of sampling from the distribution µ(x) over the large-dimensional space X N reduces to that of computing one-variable marginals of µ(x), conditional on a subset of the other variables. In other words, if we have a black box that computes µ(xi |xU ) for any subset U ⊆ V , it can be used to sample a random configuration x. The standard procedure for doing this is called sequential importance sampling. We can describe this procedureby the following algorithm in the case of tree-graphical models, using BP to implement such a ‘black box’. BP-Guided Sampling (fraphical model (G, ψ)) 1: initialize BP messages; 2: initialize U = ∅; 3: for t = 1, . . . , N : 4: run BP until convergence; 5: choose i ∈ V \ U ; 6: compute the BP marginal νi (xi ); 7: choose x∗i distributed according to νi ; 8: fix xi = x∗i and set U ← U ∪ {i}; 9: add a factor I(xi = x∗i ) to the graphical model; 10: end 11: return x∗ . 14.2.5

Pairwise models

Pairwise graphical models, i.e. graphical models such that all factor nodes have degree 2, form an important class. A pairwise model can be conveniently represented as an ordinary graph G = (V, E) over variable nodes. An edge joins two variables each time they are the arguments of the same compatibility function. The corresponding probability distribution reads µ(x) =

1 Z

ψij (xi , xj ) .

(14.30)

(ij)∈E

Function nodes can be identified with edges (ij) ∈ E. In this case belief propagation can be described as operating directly on G. Further, one of the two types of messages can be easily eliminated: here we shall work uniquely (t) with variable-to-function messages, which we will denote by νi→j (xi ), a shortcut for (t)

νi→(ij) (xi ). The BP updates then read

Optimization: Max-product and min-sum (t+1) νi→j (xi ) ∼ =

(t)

ψil (xi , xl ) νl→i (xl ) .

¿

(14.31)

l∈∂i\j xl

Simplified expressions can be derived in this case for the joint distribution of several variables (see eqn (14.18)), as well as for the free entropy.

Exercise 14.7 Show that, P for pairwise P models, the free entropy given in eqn (14.27) can be written as F∗ (ν) = i∈V Fi (ν) − (ij)∈E F(ij) (ν), where 2 Fi (ν) = log 4 2 F(ij) (ν) = log 4

XY xi j∈∂i

X

13

0

X

@

ψij (xi , xj )νj→i (xj )A5 ,

xj

3

νi→j (xi )ψij (xi , xj )νj→i (xj )5 .

(14.32)

xi ,xj

14.3

Optimization: Max-product and min-sum

Message-passing algorithms are not limited to computing marginals. Imagine that you are given a probability distribution µ( · ) as in eqn (14.13), and you are asked to find a configuration x which maximizes the probability µ(x). Such a configuration is called a mode of µ( · ). This task is important in many applications, ranging from MAP estimation (e.g. in image reconstruction) to word MAP decoding. It is not hard to devise a message-passing algorithm adapted to this task, which correctly solves the problem on trees. 14.3.1

Max-marginals

The role of marginal probabilities is played here by the max-marginals Mi (x∗i ) = max{µ(x) : xi = x∗i } . x

(14.33)

In the same way as the tasks of sampling and of computing partition functions can be reduced to computing marginals, optimization can be reduced to computing maxmarginals. In other words, given a black box that computes max-marginals, optimization can be performed efficiently. Consider first the simpler case in which the max-marginals are non-degenerate, i.e., for each i ∈ V , there exists an x∗i such that Mi (x∗i ) > Mi (xi ) (strictly) for any xi = x∗i . The unique maximizing configuration is then given by x∗ = (x∗1 , . . . , x∗N ). In the general case, the following ‘decimation’ procedure, which is closely related to the BP-guided sampling algorithm of Section 14.2.4, returns one of the maximizing configurations. Choose an ordering of the variables, say (1, . . . , N ). Compute M1 (x1 ), and let x∗1 be one of the values maximizing it: x∗ ∈ arg max M1 (x1 ). Fix x1 to take this

¿

Belief propagation

value, i.e. modify the graphical model by introducing the factor I(x1 = x∗1 ) (this corresponds to considering the conditional distribution µ(x|x1 = x∗1 )). Compute M2 (x2 ) for the new model, fix x2 to one value x∗2 ∈ arg max M2 (x2 ), and iterate this procedure, fixing all the xi ’s sequentially.

14.3.2

Message passing

It is clear from the above that max-marginals need only to be computed up to a multiplicative normalization. We shall therefore stick to our convention of denoting equality between max-marginals up to an overall normalization by ∼ =. Adapting the message-passing update rules to the computation of max-marginals is not hard: it is sufficient to replace sums with maximizations. This yields the following max-product update rules:

(t+1) νi→a (xi ) ∼ =

(t)

ν b→i (xi ) ,

(14.34)

b∈∂i\a (t) ν a→i (xi ) ∼ = max x∂a\i

⎧ ⎨ ψa (x∂a )

⎩

(t)

νj→a (xj )

⎫ ⎬

j∈∂a\i

⎭

.

(14.35)

The fixed-point conditions for this recursion are called the max-product equations. As in BP, it is understood that, when ∂j \ a is an empty set, νj→a (xj ) ∼ = 1 is the uniform distribution. Similarly, if ∂a \ j is empty, then ν a→j (xj ) ∼ = ψa (xj ). After any number of iterations, an estimate of the max-marginals is obtained as follows: νi (xi ) ∼ = (t)

(t−1)

ν a→i (xi ) .

(14.36)

a∈∂i

As in the case of BP, the main motivation for the above updates comes from the analysis of graphical models on trees. Theorem 14.4. (the max-product algorithm is exact on trees) Consider a treegraphical model with diameter t∗ . Then: 1. Irrespective of the initialization, the max-product updates (14.34) and (14.35) converge after at most t∗ iterations. In other words, for any edge (i, a) and any t > t∗ , (t) (t) ∗ ∗ and ν a→i = ν a→i . νi→a = νi→a 2. The max-marginals are estimated correctly, i.e., for any variable node i and any (t) t > t∗ , νi (xi ) = Mi (xi ).

The proof follows closely that of Theorem 14.1, and is left as an exercise for the reader.


¿

Exercise 14.8 The crucial property used in both Theorem 14.1 and Theorem 14.4 is the distributive property of the sum and the maximum with respect to the product. Consider, for instance, a function of the form f (x1 , x2 , x3 ) = ψ1 (x1 , x2 )ψ2 (x1 , x3 ). Then one can decompose the sum and maximum as follows: X

f (x1 , x2 , x3 ) =

x1 ,x2 ,x3

X

"

x1

max f (x1 , x2 , x3 ) = max

x1 ,x2 ,x3

x1

X

! ψ1 (x1 , x2 )

X

!# ψ2 (x1 , x3 )

,

(14.37)

x3

x2

»„

«„ «– max ψ2 (x1 , x3 ) .

max ψ1 (x1 , x2 ) x2

x3

(14.38)

Formulate a general ‘marginalization’ problem (with the ordinary sum and product substituted by general operations with a distributive property) and describe a message-passing algorithm that solves it on trees.

(t)

(t)

The max-product messages νi→a ( · ) and ν a→i ( · ) admit an interpretation which is (t) analogous to that of sum–product messages. For instance, νi→a ( · ) is an estimate of the max-marginal of variable xi with respect to the modified graphical model in which factor node a is removed from the graph. Along with the proof of Theorem 14.4, it is easy to show that, in a tree-graphical model, fixed-point messages do indeed coincide with the max-marginals of such modified graphical models. The problem of finding the mode of a distribution that factorizes as in eqn (14.13) has an alternative formulation, namely as minimizing a cost (energy) function that can be written as a sum of local terms: Ea (x∂a ) . (14.39) E(x) = a∈F

The problems are mapped onto each other by writing ψa (x∂a ) = e−βEa (x∂a ) (with β some positive constant). A set of message-passing rules that is better adapted to the latter formulation is obtained by taking the logarithm of eqns (14.34) and (14.35). This version of the algorithm is known as the min-sum algorithm: (t+1)

Ei→a (xi ) =

(t) (xi ) + C (t) , E i→a b→i

b∈∂i\a

⎡

(t) (xi ) E a→i

= min ⎣Ea (x∂a ) + x∂a\i

(14.40) ⎤

(t) Ej→a (xj )⎦

(t) . +C a→i

(14.41)

j∈∂a\i

The corresponding fixed-point equations are also known in statistical physics as the energetic cavity equations. Notice that, since the max-product marginals are relevant only up to a multiplicative constant, the min-sum messages are defined up to (t) an overall additive constant. In the following, we shall choose the constants Ci→a

(t) such that minx E (t+1) (xi ) = 0 and minx E

(t) (xi ) = 0, respectively. The and C i i a→i i→a a→i

¿

Belief propagation

analogue of the max-marginal estimate in eqn (14.36) is provided by the following log-max-marginal: (t−1) (t) (t)

(14.42) Ei (xi ) = E a→i (xi ) + Ci . a∈∂i

In the case of tree-graphical models, the minimum energy U∗ = minx E(x) can be ∗

∗ }. We obtain, ,E immediately written in terms of the fixed-point messages {Ei→a i→a in fact, Ea (x∗∂a ) , (14.43) U∗ = a

#

x∗∂a = arg min Ea (x∂a ) + x∂a

$ ∗

i→a (xi ) E

.

(14.44)

i∈∂a

In the case of non-tree graphs, this can be taken as a prescription to obtain a max(t) product estimate U∗ of the minimum energy. One just needs to replace the fixedpoint messages in eqn (14.44) with the messages obtained after t iterations. Finally, a minimizing configuration x∗ can be obtained through the decimation procedure described in the previous subsection.

Exercise 14.9 Show that U∗ is also given by U∗ = 2

X

a = min 4Ea (x∂a ) + x∂a

h ia = min xi

+

a∈F

a +

P

3 ∗ Ej→a (xj )5

j∈∂a ∗ Ei→a (xi )

P

∗ ba→i E (xi )

i .

i∈V

" ,

i = min xi

i −

X

P (ia)∈E

ia , where

# ∗ ba→i (xi ) , E

a∈∂i

(14.45)

h i ∗ ∗ ba→i (xi ) + Ei→a (xi ) , and show that the minima in [Hints: (i) Define x∗i (a) = arg min E eqn (14.45) are achieved at xi = x∗i (a) (for i and ai ) and at x∗∂a = {x∗i (a)}i∈∂a (for a ). P P ∗ ba→i (ii) Show that (ia) E (x∗i (a)) = i i .]

14.3.3

Warning propagation

A frequently encountered case is that of constraint satisfaction problems, where the energy function just counts the number of violated constraints: 0 if constraint a is satisfied, (14.46) Ea (x∂a ) = 1 otherwise. The structure of messages can be simplified considerably in this case. More precisely,

(0) ∈ {0, 1}, this condition is if the messages are initialized in such a way that E a→i preserved by the min-sum updates (14.40) and (14.41) at any subsequent time. Let us


¿

prove this statement by induction. Suppose it holds up to time t−1. From eqn (14.40), (t) it follows that Ei→a (xi ) is a non-negative integer. Now consider eqn (14.41). Since both (t)

(t) (xi ), the minimum of the rightEj→a (xj ) and Ea (x∂a ) are integers, it follows that E a→i hand side, is a non-negative integer as well. Further, since for each j ∈ ∂a \ i there (t) exists an x∗j such that Ej→a (x∗j ) = 0, the minimum in eqn (14.41) is at most 1, which proves our claim. This argument also shows that the outcome of the minimization in eqn (14.41) (t) depends only on which entries of the messages Ej→a ( · ) vanish. If there exists an assignment x∗j such that Ej→a (x∗j ) = 0 for each j ∈ ∂a \ i, and Ea (xi , x∗∂a\i ) = 0, then the value of the minimum is 0. Otherwise, it is 1. In other words, instead of keeping track of the messages Ei→a ( · ), one can use their ‘projections’ (14.47) Ei→a (xi ) = min {1, Ei→a (xi )} . (t)

Proposition 14.5 Consider an optimization problem with a cost function of the form (14.39) with Ea (x∂a ) ∈ {0, 1}, and assume the min-sum algorithm to be initialized

a→i (xi ) ∈ {0, 1} for all edges (i, a). Then, after any number of iterations, the with E function-node-to-variable-node messages coincide with those computed using the following update rules: ⎧ ⎫ ⎨ ⎬ (t+1)

(t) (xi ) + C (t) E , (14.48) Ei→a (xi ) = min 1, i→a b→i ⎩ ⎭ b∈∂i\a ⎫ ⎧ ⎬ ⎨ (t)

(t) ,

(t) (xi ) = min Ea (x ) + E (x ) +C (14.49) E ∂a a→i j→a j a→i x∂a\i ⎩ ⎭ j∈∂a\i

(t)

(t) are normalization constants determined by minx E

a→i (xi ) = 0 and where Ci→a , C i a→i minxi Ei→a (xi ) = 0. Finally, the ground state energy takes the same form as eqn. (14.45), with Ei→a ( · ) replacing Ei→a ( · ).

We call the simplified min-sum algorithm with the update equations (14.49) and (14.48) the warning propagation algorithm. The name is due to the fact that the messages Ei→a ( · ) can be interpreted as the following warnings: Ei→a (xi ) = 1 → ‘according to the set of constraints b ∈ ∂i \ a, the i-th variable should not take the value xi ’. Ei→a (xi ) = 0 → ‘according to the set of constraints b ∈ ∂i \ a, the i-th variable can take the value xi ’. Warning propagation provides a procedure for finding all direct implications of a partial assignment of the variables in a constraint satisfaction problem. For instance, in the case of the satisfiability problem, it finds all implications found by unit clause propagation (see Section 10.2).

¿½¼

14.4

Belief propagation

Loopy BP

We have seen how message-passing algorithms can be used efficiently in tree-graphical models. In particular, they allow one to exactly sample distributions that factorize according to tree factor graphs and to compute marginals, partition functions, and modes of such distributions. It would be very useful in a number of applications to be able to accomplish the same tasks when the underlying factor graph is no longer a tree. It is tempting to use the BP equations in this more general context, hoping to get approximate results for large graphical models. Often, we shall be dealing with problems that are NP-hard even to approximate, and it is difficult to provide general guarantees of performance. Indeed, an important unsolved challenge is to identify classes of graphical models where the following questions can be answered: ∗ ∗ , ν a→i } that reproduces the local marginals of 1. Is there any set of messages {νi→a µ( · ) by use of eqn (14.18), within some prescribed accuracy? 2. Do such messages correspond to an (approximate) fixed point of the BP update rules (14.14) and (14.15)? 3. Do the BP update rules have at least one (approximate) fixed point? Is it unique? 4. Does such a fixed point have a non-empty ‘basin of attraction’ with respect to eqns (14.14) and (14.15)? Does this basin of attraction include all possible (or all ‘reasonable’) initializations?

We shall not treat these questions in depth, as a general theory is lacking. We shall, rather, describe the sophisticated picture that has emerged, building on a mixture of physical intuition, physical methods, empirical observations, and rigorous proofs.

Exercise 14.10 Consider a ferromagnetic Ising model on a two-dimensional grid with periodic boundary conditions (i.e. ‘wrapped’ around a torus), as defined in Section 9.1.2 (see Fig. 9.7). Ising spins σi , i ∈ V , are associated with the vertices of the grid, and interact along the edges: µ(σ) =

1 β P(ij)∈E σi σj . e Z

(14.50)

(a) Describe the associated factor graph. (b) Write the BP equations. (c) Look for a solution that is invariant under translation, i.e. νi→a (σi ) = ν(σi ), νba→i (σi ) = νb(σi ): write down the equations satisfied by ν( · ), νb( · ). (d) Parameterize ν(σ) in terms of the log-likelihood h = (1/2β) log(ν(+1)/ν(−1)) and show that h satisfies the equation tanh(βh) = tanh(β) tanh(3βh). (e) Study this equation and show that, for 3 tanh β > 1, it has three distinct solutions corresponding to three BP fixed points. (f ) Consider the iteration of the BP updates starting from a translation-invariant initial condition. Does the iteration converge to a fixed point? Which one?

Loopy BP

¿½½

(g) Discuss the appearance of three BP fixed points in relation to the structure of the distribution µ(σ) and the paramagnetic–ferromagnetic transition. What is the approximate value of the critical temperature obtained from BP? Compare with the exact √ value βc = 12 log(1 + 2). (h) What results does one obtain for an Ising model on a d-dimensional (instead of twodimensional) grid?

14.4.1

The Bethe free entropy

As we saw in Section 14.2.4, the free entropy of a tree-graphical model has a simple expression in terms of local marginals (see eqn (14.26)). We can use it in graphs with loops with the hope that it provides a good estimate of the actual free entropy. In spirit, this approach is similar to the ‘mean-field’ free entropy introduced in Chapter 2, although it differs from it in several respects. In order to define precisely the Bethe free entropy, we must first describe a space of ‘possible’ local marginals. A minimalistic approach is to restrict ourselves to the ‘locally consistent marginals’. A set of locally consistent marginals is a collection of distributions bi ( · ) over X for each i ∈ V , and ba ( · ) over X |∂a| for each a ∈ F . Being distributions, they must be non-negative, i.e. bi (xi ) ≥ 0 and ba (x∂a ) ≥ 0, and they must satisfy the normalization conditions bi (xi ) = 1 ∀i ∈ V , ba (x∂a ) = 1 ∀a ∈ F . (14.51) xi

x∂a

To be ‘locally consistent’, they must satisfy the marginalization condition ba (x∂a ) = bi (xi ) ∀a ∈ F , ∀i ∈ ∂a .

(14.52)

x∂a\i

Given a factor graph G, we shall denote the set of locally consistent marginals by LOC(G), and the Bethe free entropy will be defined as a real-valued function on this space. It is important to stress that, although the marginals of any probability distribution µ(x) over x = (x1 , . . . , xN ) must be locally consistent, the converse is not true: one can find sets of locally consistent marginals that do not correspond to any distribution. In order to emphasize this point, locally consistent marginals are sometimes called ‘beliefs’.

Exercise 14.11 Consider the graphical model shown in Fig. 14.5, on binary variables (x1 , x2 , x3 ), xi ∈ {0, 1}. The figure also gives a set of beliefs in the vector/matrix form » bi =

bi (0) bi (1)

–

» , bij =

bij (00) bij (01) bij (10) bij (11)

– .

(14.53)

Check that this set of beliefs is locally consistent, but that they cannot be the marginals of any distribution µ(x1 , x2 , x3 ).

¿½¾

Belief propagation

b12

0.49 0.01 = 0.01 0.49

b2 =

0.5 0.5

0.5 b1 = 0.5

b31 =

0.01 0.49 0.49 0.01

b23 =

0.49 0.01 0.01 0.49

b3 =

0.5 0.5

Fig. 14.5 A set of locally consistent marginals, ‘beliefs’, that cannot arise as the marginals of any global distribution.

Given a set of locally consistent marginals b = {ba , bi }, we associate a Bethe free entropy with it exactly as in eqn (14.26): ba (x∂a ) − ba (x∂a ) log (1 − |∂i|) bi (xi ) log bi (xi ) . (14.54) F[b] = − ψa (x∂a ) a∈F

i∈V

The analogy with the naive mean-field approach suggests that stationary points (and, in particular, maxima) of the Bethe free entropy should play an important role. This is partially confirmed by the following result. Proposition 14.6 Assume ψa (x∂a ) > 0 for every a and x∂a . Then the stationary points of the Bethe free entropy F[b] are in one-to-one correspondence with the fixed points of the BP algorithm. As will become apparent from the proof, the correspondence between BP fixed points and stationary points of F[b] is completely explicit. Proof We want to check stationarity with respect to variations of b within the set LOC(G), which is defined by the constraints (14.51) and (14.52), as well as ba (x∂a ) ≥ 0, bi (xi ) ≥ 0. We thus introduce a set of Lagrange multipliers λ = {λi , i ∈ V ; λai (xi ), (a, i) ∈ E, xi ∈ X }, where λi corresponds to the normalization of bi ( · ) and λai (xi ) corresponds to the marginal of ba coinciding with bi . We then define the Lagrangian ⎡ ⎤ λi bi (xi ) − 1 − λai (xi ) ⎣ ba (x∂a ) − bi (xi )⎦ . L(b, λ) = F[b] − a∈F

xi

(ia),xi

x∂a\i

(14.55) Notice that we have not introduced a Lagrange multiplier for the normalization of ba (x∂a ), as this follows from the two constraints already enforced. The stationarity conditions with respect to bi and ba imply P λai (xi ) , ba (x∂a ) ∼ bi (xi ) ∼ = e−1/(|∂i|−1) = ψa (x∂a ) e− i∈∂a λai (xi ) . (14.56) a∈∂i

Loopy BP

¿½¿

The Lagrange multipliers must be chosen in such a way that eqn (14.52) is fulfilled. Any such set of Lagrange multipliers yields a stationary point of F[b]. Once the λai (xj ) have been found, the computation of the normalization constants in these expressions fixes λi . Conversely, any stationary point corresponds to a set of Lagrange multipliers satisfying the stated condition. ba (x∂a ) = It remains to show that sets of Lagrange multipliers such that x ∂a\i bi (xi ) are in one-to-one correspondence with BP fixed points. In order to see this, we define the messages P ψa (x∂a ) e− j∈∂a\i λaj (xj ) . (14.57) νi→a (xi ) ∼ = e−λai (xi ) , ν a→i (xi ) ∼ = x∂a\i

It is clear from the definition that such messages satisfy ψa (x∂a ) νi→a (xi ) . ν a→i (xi ) ∼ = x∂a\i

(14.58)

j∈∂a\i

Further, using the second equation of eqns (14.56) together with eqn. (14.57), we get ∼ νa→i (xi ). On the other hand, from the first of eqns (14.56) x∂a\i ba (x∂a ) = νi→a (xi )

together with eqn (14.57), we get bi (xi ) ∼ = b νi→b (xi )1/(|∂i|−1) . The marginalization condition thus implies νi→b (xi )1/(|∂i|−1) ∼ νa→i (xi ) . (14.59) = νi→a (xi )

b∈∂i

Taking the product of these equalities for a ∈ ∂i \ b, and eliminating a∈∂i\b νi→a (xi ) from the resulting equation (which is possible if ψa (x∂a ) > 0), we get ν a→i (xi ) . (14.60) νi→b (xi ) ∼ = a∈∂i\b

At this point we recognize in eqns (14.58) and (14.60) the fixed-point condition for BP (see eqns (14.14) and (14.15)). Conversely, given any solution of eqns (14.58) and (14.60), one can define a set of Lagrange multipliers using the first of eqns (14.57). It follows from the-fixed point condition that the second of eqns (14.57) is fulfilled as well, and that the marginalization condition holds. An important consequence of this proposition is the existence of BP fixed points. Corollary 14.7 Assume ψa (xa ) > 0 for every a and x∂a . The BP algorithm then has at least one fixed point. Proof Since F[b] is bounded and continuous in LOC(G) (which is closed), it takes its maximum at some point b∗ ∈ LOC(G). Using the condition ψa (xa ) > 0, it is easy to see that such a maximum is reached in the relative interior of LOC(G), i.e. that b∗a (x∂a ) > 0, b∗i (xi ) > 0 strictly. As a consequence, b∗ must be a stationary point and therefore, by Proposition 14.6, there is a BP fixed point associated with it. The ‘variational principle’ provided by Proposition 14.6 is particularly suggestive as it is analogous to naive mean-field bounds. For practical applications, it is sometimes

¿

Belief propagation

xl

xl

xi

xi

xj

xj

Fig. 14.6 Left: neighbourhood of a node i in a pairwise graphical model. Right: the modified graphical model used to define the message νi→j (xi ).

more convenient to use the free-entropy functional F∗ (ν) of eqn (14.27). This can be regarded as a function from the space of messages to reals F : M(X )|E| → R (remember is the set of directed edges in that M(X ) denotes the set of measures over X , and E the factor graph).3 It satisfies the following variational principle. Proposition 14.8 The stationary points of the Bethe free entropy F∗ (ν) are fixed points of belief propagation. Conversely, any fixed point ν of belief propagation such that F∗ (ν) is finite, is also a stationary point of F∗ (ν). The proof is simple calculus and is left to the reader. It turns out that for tree graphs and unicyclic graphs, F[b] is convex, and the above results then prove the existence and uniqueness of BP fixed points. But, for general graphs, F[b] is non-convex and may have multiple stationary points. 14.4.2

Correlations

What is the origin of the error made when BP is used in an arbitrary graph with loops, and under what conditions can it be small? In order to understand this point, let us consider for notational simplicity a pairwise graphical model (see eqn (14.2.5)). The generalization to other models is straightforward. Taking seriously the probabilistic interpretation of messages, we want to compute the marginal distribution νi→j (xi ) of xi in a modified graphical model that does not include the factor ψij (xi , xj ) (see Fig. 14.6). We denote by µ∂i\j (x∂i\j ) the joint distribution of all variables in ∂i \ j in the model where all the factors ψil (xi , xl ), l ∈ ∂i, have been removed. Then, νi→j (xi ) ∼ =

ψil (xi , xl )µ∂i\j (x∂i\j ) .

(14.61)

x∂i\j l∈∂i\j

Comparing this expression with the BP equations (see eqn (14.31)), we deduce that the messages {νi→j } solve these equations if 3 On

a tree, F∗ (ν) is (up to a change of variables) the Lagrangian dual of F(b).

Loopy BP

xl

¿

xl

xi

xi

xj

xj

Fig. 14.7 Left: modified graphical model used to define νl→i (xl ). Right: modified graphical model corresponding to the cavity distribution of the neighbours of i, µ∂i\j (x∂i\j ).

µ∂i\j (x∂i\j ) =

νl→i (xl ) .

(14.62)

l∈∂i\j

We can expect this to happen when two conditions are fulfilled: 1. Under µ∂i\j ( · ), the variables {xl : l ∈ ∂i \ j} are independent: µ∂i\j (x∂i\j ) = l∈∂i\j µ∂i\j (xl ). 2. The marginal of each of these variables under µ∂i\j ( · ) is equal to the corresponding message νl→i (xl ). In other words, the two graphical models obtained by removing all the compatibility functions that involve xi (namely, the model µ∂i\j ( · )) and by removing only ψil (xi , xl ) must have the same marginal for the variable xl ; see Fig. 14.7. These two conditions are obviously fulfilled for tree-graphical models. They are also approximately fulfilled if the correlations among the variables {xl : l ∈ ∂i} are ‘small’ under µ∂i\j ( · ). As we have seen, in many cases of practical interest (LDPC codes, random K-SAT, etc.) the factor graph is locally tree-like. In other words, when node i is removed, the variables {xl : l ∈ ∂i} are, with high probability, far apart from each other. This suggests that, in such models, the two conditions above may indeed hold in the large-size limit, provided far-apart variables are weakly correlated. A simple illustration of this phenomenon is provided in the exercises below. The following chapters will investigate this property further and discuss how to cope with cases in which it does not hold. Exercise 14.12 Consider an antiferromagnetic Ising model on a ring, with variables (σ1 , . . . , σN ) ≡ σ, σi ∈ {+1, −1} and distribution µ(σ) =

1 −β PN i=1 σi σi+1 , e Z

(14.63)

where σN +1 ≡ σ1 . This is a pairwise graphical model whose graph G is a ring over N vertices.

¿

Belief propagation

(a) Write the BP update rules for this model (see Section 14.2.5). (t)

(b) Express the update rules in terms of the log-likelihoods hi→ ≡ (t) (t) (t) (t) (t) 1 1 log((ν (+1))/(ν (−1))), and h ≡ log((ν (+1))/(ν (−1))). i→i+1 i→i+1 ←i i→i−1 i→i−1 2 2 (c) Show that, for any β ∈ [0, ∞), and any initialization, the BP updates converge to the unique fixed point h←i = hi→ = 0 for all i. (d) Assume that β = +∞ and N is even. Show that any set of log-likelihoods of the form hi→ = (−1)i a, h←i = (−1)i b, with a, b ∈ [−1, 1], is a fixed point. (e) Consider now the case where β = ∞ and N is odd, and show that the only fixed point is h←i = hi→ = 0. Find an initialization of the messages such that BP does not converge to this fixed point.

Exercise 14.13 Consider a ferromagnetic Ising model on a ring with a magnetic field. This is defined through the distribution µ(σ) =

PN 1 β PN e i=1 σi σi+1 +B i=1 σi , Z

(14.64)

where σN +1 ≡ σ1 . Notice that, with respect to the previous exercise, we have changed a sign in the exponent. (a, b) As in the previous exercise. (c) Show that, for any β ∈ [0, ∞), and any initialization, the BP updates converge to the unique fixed point h←i = hi→ = h∗ (β, B) for all i. (d) Let σi be the expectation of spin σi with respect to the measure µ( · ), and let σi BP be the corresponding BP estimate. Show that |σi − σi BP | = O(λN ) for some λ ∈ (0, 1).

14.5

General message-passing algorithms

Both the sum–product and the max-product (or min-sum) algorithm are instances of a more general class of message-passing algorithms. All of the algorithms in this family share some common features, which we now highlight. Given a factor graph, a message-passing algorithm is defined by the following ingredients: 1. An alphabet of messages M. This can be either continuous or discrete. The algo(t) (t) rithm operates on messages νi→a , ν a→i ∈ M associated with the directed edges in the factor graph. 2. Update functions Ψi→a : M|∂i\a| → M and Φa→i : M|∂a\i| → M that describe how to update messages. 3. An initialization, i.e. a mapping from the directed edges in the factor graph to (0) (0) M (this can be a random mapping). We shall denote by νi→a , ν a→i the image of such a mapping. 4. A decision rule, i.e. a local function from messages to a space of ‘decisions’ from which we are interested in making a choice. Since we shall be interested mostly

Probabilistic analysis

¿

in computing marginals (or max-marginals), we shall assume the decision rule to

i : M|∂i| → M(X ). be given by a family of functions Ψ Notice the characteristic feature of message-passing algorithms: messages going out from a node are functions of messages coming into the same node through the other edges. Given these ingredients, a message-passing algorithm with parallel updating may (0) (0) be defined as follows. Assign the values of initial messages νi→a , ν a→i according to an initialization rule. Then, for any t ≥ 0, update the messages through local operations at variable/check nodes as follows: (t+1)

(t)

νb→i : b ∈ ∂i \ a}) , νi→a = Ψi→a ({

(t) ν a→i

=

(t) Φa→i ({νj→a

: j ∈ ∂a \ i}) .

(14.65) (14.66)

Finally, after a pre-established number of iterations t, take the decision using the rules

i ; namely, return Ψ

i ({

νi (xi ) = Ψ νb→i : b ∈ ∂i})(xi ) . (t)

(t−1)

(14.67)

Many variants are possible concerning the update schedule. For instance, in the case of sequential updating one can pick out a directed edge uniformly at random and compute the corresponding message. Another possibility is to generate a random permutation of the edges and update the messages according to this permutation. We shall not discuss these ‘details’, but the reader should be aware that they can be important in practice: some update schemes may converge better than others.

Exercise 14.14 Recast the sum–product and min-sum algorithms in the general messagepassing framework. In particular, specify the alphabet of the messages, and the update and decision rules.

14.6


In the following chapters, we shall repeatedly be concerned with the analysis of message-passing algorithms on random graphical models. In this context, messages become random variables, and their distribution can be characterized in the largesystem limit, as we shall now see. 14.6.1

Assumptions

Before proceeding, it is necessary to formulate a few technical assumptions under which our approach works. The basic idea is that, in a ‘random graphical model’, distinct nodes should be essentially independent. Specifically, we shall consider below a setting which already includes many cases of interest; it is easy to extend our analysis to even more general situations.

¿

Belief propagation

A random graphical model is a (random) probability distribution on x = (x1 , . . . , xN ) of the form4 µ(x) ∼ =

a∈F

ψa (x∂a )

ψi (xi ) ,

(14.68)

i∈V

where the factor graph G = (V, F, E) (with variable nodes V , factor nodes F , and edges E) and the various factors ψa , ψi are independent random variables. More precisely, we assume that the factor graph is distributed according to one of the ensembles GN (K, α) or DN (Λ, P ) (see Chapter 9). The random factors are assumed to be distributed as follows. For any given degree

indexed by a ‘label’ J ∈ J, k, we are given a list of possible factors ψ (k) (x1 , . . . , xk ; J), (k) and a distribution PJb over the set of possible labels J. For each function node a ∈ F (k) of degree |∂a| = k, a label J a is drawn with distribution P , and the function ψa ( · ) Jb

is taken to be equal to ψ ( · ; J a ). Analogously, the factors ψi are drawn from a list of possible {ψ( · ; J)}, indexed by a label J which is drawn from a distribution PJ . The random graphical model is fully characterized by the graph ensemble, the set of (k)

{ψ( · ; J)}. distributions PJb , PJ , and the lists of factors {ψ (k) ( · ; J)}, We need to make some assumptions about the message update rules. Specifically, we assume that the variable-to-function-node update rules Ψi→a depend on i → a only through |∂i| and Ji , and the function-to-variable-node update rules Φa→i depend on a → i only through |∂a| and J a . With a slight misuse of notation, we shall denote the update functions by (k)

νb→i : b ∈ ∂i \ a}) = Ψl (

ν1 , . . . , ν l ; Ji ) , Ψi→a ({

Φa→i ({νj→a : j ∈ ∂a \ i}) = Φk (ν1 , . . . , νk ; J a ) ,

(14.69) (14.70)

νb→i : b ∈ ∂i \ a}, and {ν1 , . . . , νk } ≡ where l ≡ |∂i| − 1, k ≡ |∂a| − 1, {

ν1 , . . . , ν l } ≡ {

{νj→a : j ∈ ∂a \ i}. A similar notation will be used for the decision rule Ψ.

Exercise 14.15 Let G = (V, E) be a uniformly random graph with M = N α edges over ¯ Recall that an N vertices, and let λi , i ∈ V , be i.i.d. random variables uniform in [0, λ]. independent set for G is a subset of the vertices S ⊆ V such that if i, j ∈ S, then (ij) is not an edge. Consider the following weighted measure over independent sets: µ(S) =

Y 1 λi . I(S is an independent set) Z i∈S

(14.71)

4 Note that the factors ψ , i ∈ V , could have been included as degree-1 function nodes, as we i did in eqn (14.13); including them explicitly yields a description of density evolution which is more symmetric between variables and factors, and applies more directly to decoding.


¿

a

i

Fig. 14.8 A radius-2 directed neighbourhood Bi→a,2 (F ). (a) Write the distribution µ(S) as a graphical model with binary variables, and define the corresponding factor graph. (b) Describe the BP algorithm to compute its marginals. (c) Show that this model is a random graphical model in the sense defined above.

14.6.2

Density evolution equations

Consider a random graphical model, with factor graph G = (V, F, E), and let (i, a) (t) be a uniformly random edge in G. Let νi→a be the message sent by the BP algorithm (0) (0) in iteration t along edge (i, a). We assume that the initial messages νi→a , ν a→i are i.i.d. random variables, with distributions independent of N . A considerable amount (t) (t) of information is contained in the distributions of νi→a and ν a→i with respect to the realization of the model. We are interested in characterizing these distributions in the large-system limit N → ∞. Our analysis will assume that both the message alphabet M and the node label alphabet J are subsets of Rd for some fixed d, and that the update functions Ψi→a , Φa→i are continuous with respect to the usual topology of Rd . It is convenient to introduce the directed neighbourhood of radius t of a directed edge i → a, denoted by: Bi→a,t (G). This is defined as the subgraph of G that includes all of the variable nodes which can be reached from i by a non-reversing path of length at most t, whose first step is not the edge (i, a). It includes, as well, all of the function nodes connected only to those variable nodes; see Fig. 14.8. For illustrative reasons, we shall occasionally add a ‘root edge’, such as i → a in Fig. 14.8. Let us consider, to be definite, the case where G is a random factor graph from the ensemble DN (Λ, P ). In this case, Bi→a,t (F ) converges in distribution, when N → ∞, to the random tree ensemble Tt (Λ, P ) defined in Section 9.5.1.

Exercise 14.16 Consider a random graph from the regular ensemble DN (Λ, P )

¿¾¼

Belief propagation

a

a

a

i

i

i

(A)

(B)

(C)

Fig. 14.9 The three possible radius-1 directed neighbourhoods in a random factor graph from the regular graph ensemble DN (2, 3). with Λ2 = 1 and P3 = 1 (each variable node has degree 2 and each function node degree 3). The three possible radius-1 directed neighbourhoods appearing in such factor graphs are depicted in Fig. 14.9. (a) Show that the probability that a given edge (i, a) has neighbourhoods as in (B) or (C) in the figure is O(1/N ). d

(b) Deduce that Bi→a,1 (F ) → T1 , where T1 is distributed according to the tree model T1 (2, 3) (i.e. it is the tree in Fig. 14.9, labelled (A)). (c) Discuss the case of a radius-t neighbourhood.

For our purposes, it is necessary to include in the description of the neighbourhood Bi→a,t (F ) the value of the labels Ji , J b for function nodes b in this neighbourhood. It is understood that the tree model Tt (Λ, P ) includes labels as well: these have to be drawn as i.i.d. random variables independent of the tree and with the same distribution as in the original graphical model. (t) Now consider the message νi→a . This is a function of the factor graph G, of the (0) labels {Jj }, {J b }, and of the initial condition {νj→b }. However, a moment of thought shows that its dependence on G and on the labels occurs only through the radius-(t+1) directed neighbourhood Bi→a,t+1 (F ). Its dependence on the initial condition is only (0) through the messages νj→b for j, b ∈ Bi→a,t (F ). In view of the above discussion, let us pretend for a moment that the neighbourhood of (i, a) is a random tree Tt+1 with distribution Tt+1 (Λ, P ). We define ν (t) to be the message passed through the root edge of such a random neighbourhood after t messagepassing iterations. Since Bi→a,t+1 (F ) converges in distribution to the tree Tt+1 , we (t)

d

find that5 νi→a → ν (t) as N → ∞. (t) We have shown that, as N → ∞, the distribution of νi→a converges to that of a well-defined (N -independent) random variable ν (t) . The next step is to find a recursive characterization of ν (t) . Consider a random tree from the ensemble Tr (Λ, P ) and let 5 The

mathematically suspicious reader may wonder about the topology we are assuming for the message space. In fact, no assumption is necessary if the distribution of labels Ji , Jba is independent of N . If it is N -dependent but converges, then the topology must be such that the message updates are continuous with respect to it.


¿¾½

j → b be an edge directed towards the root, at a distance d from it. The directed subtree rooted at j → b is distributed according to Tr−d (Λ, P ). Therefore the message passed through it after r − d − 1 (or more) iterations is distributed as ν (r−d−1) . The degree of the root variable node i (including the root edge) has a distribution λl . Each check node connected to i has a number of other neighbours (distinct from i) which is a random variable distributed according to ρk . These facts imply the following distributional equations for ν (t) and ν (t) : d

(t)

(t)

ν1 , . . . , ν l ; J) , ν (t+1) = Ψl (

d (t) (t)

ν (t) = Φk (ν1 , . . . , νk ; J) .

(t)

(14.72)

(t)

Here ν b , b ∈ {1, . . . , l − 1}, are independent copies of ν (t) , and νj , j ∈ {1, . . . , k − 1}, are independent copies of ν (t) . As for l and k, these are independent random integers (k) distributed according to λl and ρk , respectively; J is distributed as PJb , and J is d

(0)

distributed as PJ . It is understood that the recursion is initiated with ν (0) = νi→a , d

(0)

ν (0) = ν a→i . In coding theory, the equations (14.72) are referred to as density evolution; sometimes, this term is also applied to the sequence of random variables {ν (t) , ν (t) }. In probabilistic combinatorics, they are also called recursive distributional equations. We have proved the following characterization of the distribution of messages. Proposition 14.9 Consider a random graphical model satisfying the assumptions in Section 14.6.1. Let t ≥ 0 and let (ia) be a uniformly random edge in the factor graph. (t) (t) Then, as N → ∞, the messages νi→a and ν i→a converge in distribution to the random (t) (t) and ν , respectively, defined through the density evolution equations variables ν (14.72). We shall discuss several applications of the idea of density evolution in the following chapters. Here we shall just mention that it allows one to compute the asymptotic distribution of message-passing decisions at a uniformly random site i. Recall that the general message-passing decision after t iterations is taken using the rule (14.67), with

l (

i ({

νb }) = Ψ ν1 , . . . , ν l ; Ji ) (where l ≡ |∂i|). Arguing as in the previous paragraphs, Ψ (t)

it is easy to show that in the large-N limit, νi ν (t) is distributed according to

l (

ν1 ν (t) = Ψ d

(t−1)

(t−1)

d

→ ν (t) , where the random variable

(t−1)

, . . . , ν l

; J) .

(14.73)

(t−1)

, . . . , ν l are i.i.d. copies of ν (t−1) , J is an independent copy of the As above, ν 1 variable-node label Ji , and l is a random integer distributed according to Λl . 14.6.3

The replica-symmetric cavity method

The replica-symmetric (RS) cavity method of statistical mechanics adopts a point of view which is very close to the previous one, but less algorithmic. Instead of considering the BP update rules as an iterative message-passing rule, it focuses on the fixed-point BP equations themselves.

¿¾¾

Belief propagation

The idea is to compute the partition function recursively, by adding one variable node at a time. Equivalently, one may think of taking one variable node out of the system and computing the change in the partition function. The name of the method comes exactly from this image: one digs a ‘cavity’ in the system. As an example, take the original factor graph, and delete the factor node a and all the edges incident on it. If the graph is a tree, this procedure separates it into |∂a| disconnected trees. Consider now the tree-graphical model described by the connected component containing the variable j ∈ ∂a. Denote the corresponding partition function, when the variable j is fixed to the value xj , by Zj→a (xj ). This partial partition function can be computed iteratively as ⎡ ⎤ ⎣ ψb (x∂b ) Zk→b (xk )⎦ . (14.74) Zj→a (xj ) = b∈∂j\a

x∂b\j

k∈∂b\j

The equations obtained by letting j → b be a generic directed edge in G are called the cavity equations, or Bethe equations. The cavity equations are mathematically identical to the BP equations, but with two important conceptual differences: (i) one is naturally led to think that the equations (14.74) must have a fixed point, and to give special importance to it; (ii) the partial partition functions are unnormalized messages, and, as we shall see in Chapter 19, their normalization provides useful information. The relation between BP messages and partial partition functions is Zj→a (xj ) . νj→a (xj ) = y Zj→a (y)

(14.75)

In the cavity approach, the replica symmetry assumption consists in pretending that, for random graphical models of the kind introduced above, and in the large-N limit, the following conditions apply: 1. There exists a solution (or quasi-solution6 ) to these equations. 2. This solution provides good approximations to the marginals of the graphical model. 3. The messages in this solution are distributed according to a density evolution fixed point. The last statement amounts to assuming that the normalized variable-to-factor messages νi→a (see eqn (14.75)), converge in distribution to a random variable ν that solves the following distributional equations: d

ν1 , . . . , ν k−1 ; J) , ν = Ψ(

. ν = Φ(ν1 , . . . , νl−1 ; J) d

(14.76)

Here we have used the same notation as in eqn (14.72): ν b , b ∈ {1, . . . , l − 1}, are (t) independent copies of ν (t) ; νj , j ∈ {1, . . . , k − 1}, are independent copies of ν (t) ; l 6 A quasi-solution is a set of messages ν j→a such that the average difference between the left- and right-hand sides of the BP equations goes to zero in the large-N limit.


¿¾¿

and k are independent random integers distributed according to λl and ρk respectively; and J and J are distributed as the variable and function node labels Ji and J a . Using the distributions of ν and ν , the expected Bethe free entropy per variable F/N can be computed by taking the expectation of eqn (14.27). The result is f RS = fvRS + nf ffRS − ne feRS ,

(14.77)

where nf is the average number of function nodes per variable, and ne is the average number of edges per variable. In the ensemble DN (Λ, P ) we have nf = Λ (1)/P (1) and ne = Λ (1); in the ensemble GN (K, α), nf = α and ne = Kα. The contributions of the variable nodes fvRS , function nodes ffRS , and edges feRS are RS ψ(x; J) ν 1 (x) · · · ν l (x) , fv = El,J,{bν } log ffRS = Ek,J,{ν} log b fe

= Eν,bν log

x1 ,...,xk

RS

x

ν1 (x1 ) · · · νk (xk ) ψ (k) (x1 , . . . , xk ; J)

,

ν(x)

ν (x)

.

(14.78)

x

In these expressions, E denotes the expectation with respect to the random variables given in subscript. For instance, if G is distributed according to the ensemble DN (Λ, P ), El,J,{bν } implies that l is drawn from the distribution Λ, J is drawn from PJ , and ν 1 , . . . , ν l are l independent copies of the random variable ν . Instead of estimating the partition function, the cavity method can be used to compute the ground state energy. One then uses min-sum-like messages instead of those in eqn (14.74). The method is then called the ‘energetic cavity method’; we leave to the reader the task of writing the corresponding average ground state energy per variable. 14.6.4

Numerical methods

Generically, the RS cavity equations (14.76), as well as the density evolution equations (14.72), cannot be solved in closed form, and one must use numerical methods to estimate the distribution of the random variables ν, ν . Here we limit ourselves to describing a stochastic approach that has the advantage of being extremely versatile and simple to implement. It has been used in coding theory under the name of ‘sampled density evolution’ or the ‘Monte Carlo method’, and is known in statistical physics as population dynamics, a name which we shall adopt in the following. The idea is to approximate the distribution of ν (or ν ) through a sample of (ideally) N i.i.d. copies of ν (or ν , respectively). As N becomes large, the empirical distribution of such a sample should converge to the actual distribution of ν (or ν ). We shall call νi } ≡ {

ν1 , . . . , ν N }) a population. the sample {νi } ≡ {ν1 , . . . , νN } (or {

The algorithm is described by the pseudocode below. As inputs, it requires the population size N , the maximum number of iterations T , and a specification of the ensemble of (random) graphical models. The latter is a description of the (edge-perspective)

¿

Belief propagation

degree distributions λ and ρ, the variable node labels PJ , and the factor node labels (k) PJb . Population dynamics (model ensemble, size N , iterations T ) (0) 1: Initialize {νi }; 2: for t = 1, . . . , T : 3: for i = 1, . . . , N : 4: Draw an integer k with distribution ρ; 5: Draw i(1), . . . , i(k − 1) uniformly in {1, . . . , N }; (k) 6: Draw J with distribution PJb ; (t) (t−1) (t−1)

,...,ν ; J); 7: Set ν = Φk (ν i

8: 9: 10: 11: 12: 13: 14: 15: 16:

i(1)

i(k−1)

end; for i = 1, . . . , N : Draw an integer l with distribution λ; Draw i(1), . . . , i(l − 1) uniformly in {1, . . . , N }; Draw J with distribution PJ ; (t) (t) (t) νi(1) , . . . , ν i(l−1) ; J); Set νi = Ψl (

end; end; (T ) (T ) νi }. return {νi } and {

(0)

(0)

In step 1, the initialization is done by drawing ν1 , . . . , νN independently with the same distribution P that was used for the initialization of the BP algorithm. (T ) It is not hard to show that, for any fixed T , the empirical distribution of {νi } (T ) (or {

νi }) converges, as N → ∞, to the distribution of the density evolution random variable ν (t) (or ν (t) ). The limit T → ∞ is trickier. Let us assume first that the density evolution has a unique fixed point, and ν (t) , ν (t) converge to this fixed point. We then (T ) expect the empirical distribution of {νi } also to converge to this fixed point if the N → ∞ limit is taken after T → ∞. When the density evolution has more than one fixed point, which is probably the most interesting case, the situation is more subtle. (T ) The population {νi } evolves according to a large but finite-dimensional Markov chain. Therefore (under some technical conditions) the distribution of the population is expected to converge to the unique fixed point of this Markov chain. This seems to imply that population dynamics cannot describe the multiple fixed points of density evolution. Luckily, the convergence of the population dynamics algorithm to its unique fixed point appears to happen on a time scale that increases very rapidly with N . For large N and on moderate time scales T , it converges instead to one of several ‘quasifixed points’ that correspond to the fixed points of the density evolution algorithm. In practice, one can monitor the effective convergence of the algorithm by computing, after any number of iterations t, averages of the form

Notes

ϕt ≡

N 1 (t) ϕ(νi ) , N i=1

¿

(14.79)

for a smooth function ϕ : √ M(X ) → R. If these averages are well settled (up to statistical fluctuations of order 1/ N ), this is interpreted as a signal that the iteration has converged to a ‘quasi-fixed point.’ The populations produced by the above algorithm can be used to to estimate expectations with respect to the density-evolution random variables ν, ν . For instance, the expression in eqn (14.79) is an estimate for E{ϕ(ν)}. When ϕ = ϕ(ν1 , . . . , νl ) is a function of l i.i.d. copies of ν, the above formula is modified to ϕt ≡

R 1 (t) (t) ϕ(νin (1) , . . . , νin (l) ) . R n=1

(14.80)

Here R is a large number (typically of the same order as N ), and in (1), . . . , in (l) are i.i.d. indices in {1, . . . , N }. Of course such estimates will be reasonable only if l N . A particularly important example is the computation of the free entropy (14.77). Each of the terms fvRS , ffRS and feRS can be estimated as in eqn (14.80). The precision of these estimates can be improved by repeating the computation for several iterations and averaging the result.

Notes The belief propagation equations have been rediscovered several times. They were developed by Pearl (1988) as an exact algorithm for probabilistic inference in acyclic Bayesian networks. In the early 1960s, Gallager had introduced them as an iterative procedure for decoding low-density-parity-check codes (Gallager, 1963). Gallager described several message-passing procedures, among them being the sum–product algorithm. In the field of coding theory, the basic idea of this algorithm was rediscovered in several works in the 1990s, in particular by Berrou and Glavieux (1996). In the physics context, the history is even longer. In 1935, Bethe used a free-energy functional written in terms of pseudo-marginals to approximate the partition function of the ferromagnetic Ising model (Bethe, 1935). Bethe’s equations were of the simple form discussed in Exercise 14.10, because of the homogeneity (translation invariance) of the underlying model. Their generalization to inhomogeneous systems, which has a natural algorithmic interpretation, waited until the application of Bethe’s method to spin glasses (Thouless et al., 1977; Klein et al., 1979; Katsura et al., 1979; Morita, 1979; Nakanishi, 1981). The review paper by Kschischang et al. (2001) gives a general overview of belief propagation in the framework of factor graphs. The role of the distributive property, mentioned in Exercise 14.8, was emphasized by Aji and McEliece (2000). On tree graphs, belief propagation can be regarded as an instance of the junction–tree algorithm (Lauritzen, 1996). This algorithm constructs a tree from the graphical model under study by grouping some of its variables. Belief propagation is then applied to this tree.

¿

Belief propagation

Although implicit in these earlier works, the equivalence between BP, the Bethe approximation, and the sum–product algorithm was only recognized in the 1990s. The turbodecoding and the sum–product algorithms were shown to be instances of BP by McEliece et al. (1998). A variational derivation of the turbo decoding algorithm was proposed by Montanari and Sourlas (2000). The equivalence between BP and the Bethe approximation was first put forward by Kabashima and Saad (1998) and, in a more general setting, by Yedidia et al. (2001) and Yedidia et al. (2005). The last of these papers proved, in particular, the variational formulation in Proposition 14.8. This suggests that one should look for fixed points of BP by seeking stationary points of the Bethe free entropy directly, without iterating the BP equations. An efficient such procedure, based on the observation that the Bethe free entropy can be written as a difference between a convex and a concave function, was proposed by Yuille (2002). An alternative approach consists in constructing convex surrogates of the Bethe free energy (Wainwright et al., 2005 a,b) which allow one to define provably convergent message-passing procedures. The Bethe approximation can also be regarded as the first step in a hierarchy of variational methods describing larger and larger clusters of variables exactly. This point of view was first developed by Kikuchi (1951), leading to the ‘cluster variational method’ in physics. The algorithmic version of this approach is referred to as ‘generalized BP’, and is described in detail by Yedidia et al. (2005). The analysis of iterative message-passing algorithms on random graphical models dates back to Gallager (1963). These ideas were developed into a systematic method, thanks also to efficient numerical techniques, by Richardson and Urbanke (2001 b), who coined the name ‘density evolution’. The point of view taken in this book, however, is closer to that of ‘local weak convergence’ (Aldous and Steele, 2003). In physics, the replica-symmetric cavity method for sparse random graphical models was first discussed by Mézard and Parisi (1987). The use of population dynamics first appeared in Abou-Chacra et al. (1973) and was developed further for spin glasses by Mézard and Parisi (2001), but that paper deals mainly with RSB effects, which will be the subject of Chapter 19.

15 Decoding with belief propagation As we have already seen, symbol MAP decoding of error-correcting codes can be regarded as a statistical inference problem. It is a very natural idea to accomplish this task using belief propagation. For properly constructed codes (in particular, LDPC ensembles), this approach has low complexity while achieving very good performance. However, it is clear that an error-correcting code cannot achieve good performance unless the associated factor graph has loops. As a consequence, belief propagation has to be regarded only as an approximate inference algorithm in this context. A major concern of the theory is to establish conditions for its optimality, and, more generally, the relation between message passing and optimal (exact symbol MAP) decoding. In this chapter, we discuss belief propagation decoding of the LDPC ensembles introduced in Chapter 11. The message-passing approach can be generalized to several other applications within information and communication theory: other code ensembles, source coding, channels with memory, etc. Here we shall keep to the ‘canonical’ example of channel coding, as most of the theory has been developed in this context. BP decoding is defined in Section 15.1. One of the main tools in the analysis is the ‘density evolution’ method, which we discuss in Section 15.2. This allows one to determine the threshold for reliable communication under BP decoding, and to optimize the code ensemble accordingly. The whole process is considerably simpler for an erasure channel, which is treated in Section 15.3. Finally, Section 15.4 explains the relation between optimal (MAP) decoding and BP decoding in the large-block-length limit: the two approaches can studied within a unified framework based on the Bethe free energy.

15.1

BP decoding: The algorithm

In this chapter, we shall consider communication over a binary-input, outputsymmetric, memoryless (BMS) channel. This is a channel in which the transmitted codeword is binary, x ∈ {0, 1}N , and the output y is a sequence of N letters yi from an alphabet Y ⊂ R. The probability of receiving letter y when bit x is sent, Q(y|x), possesses the symmetry property Q(y|0) = Q(−y|1). Let us suppose that an LDPC error-correcting code is used in this communication. The conditional probability for the channel input to be x ∈ {0, 1}N given the output y is P(x|y) = µy (x), where µy (x) =

N M 1 Q(yi |xi ) I(xia1 ⊕ · · · ⊕ xiak(a) = 0) . Z(y) i=1 a=1

(15.1)

¿

Decoding with belief propagation

The factor graph associated with this distribution is the usual one: an edge joins a variable node i to a check node a whenever the variable xi appears in the a-th parity check equation. Messages νi→a (xi ), ν a→i (xi ) are exchanged along the edges. We shall assume a parallel updating of BP messages, as introduced in Section 14.2: (t) (t+1) ν b→i (xi ) , (15.2) νi→a (xi ) ∼ = Q(yi |xi ) (t) ν a→i (xi )

∼ =

b∈∂i\a

I(xi ⊕ xj1 ⊕ · · · ⊕ xjk−1 = 0)

{xj }

(t)

νj→a (xj ) ,

(15.3)

j∈∂a\i

∼ denotes, where we have used the notation ∂a ≡ {i, j1 , . . . , jk−1 }, and the symbol = as before, ‘equality up to a normalization constant’. We expect that the asymptotic performance at large t and large N of such BP decoding, for instance its asymptotic bit error rate, should be insensitive to the precise update schedule. On the other hand, this schedule can have an important influence on the speed of convergence, and on performance at moderate N . Here we shall not address these issues. The BP estimate for the marginal distribution at node i and time t, also called the ‘belief’ or soft decision, is (t−1) (t) ν b→i (xi ) . (15.4) νi (xi ) ∼ = Q(yi |xi ) b∈∂i

Based on this estimate, the optimal BP decision for bit i at time t, sometimes called the hard decision, is (t)

(t)

x

i = arg max νi (xi ) . xi

(15.5)

In order to fully specify the algorithm, one must address two more issues: (1) How are the messages initialized? (2) After how many iterations t is the hard decision (15.5) taken? (0) (0) In practice, one usually initializes the messages to νi→a (0) = νi→a (1) = 1/2. One alternative choice, which is sometimes useful for theoretical reasons, is to take the (0) (0) messages νi→a ( · ) as independent random variables, for instance by choosing νi→a (0) uniformly on [0, 1]. In relation to the number of iterations, one would like to have a stopping criterion. In practice, a convenient criterion is to check whether x

(t) is a codeword, and to stop if this is the case. If this condition is not fulfilled, the algorithm is stopped after a fixed number of iterations tmax . On the other hand, for the purpose of performance analysis, we shall instead fix tmax and assume that the belief propagation algorithm is always run for tmax iterations, regardless of whether a valid codeword is reached at an earlier stage. Since the messages are distributions over binary-valued variables, we parameterize them by the log-likelihoods hi→a =

νi→a (0) 1 log , 2 νi→a (1)

ua→i =

ν a→i (0) 1 log . 2 ν a→i (1)

(15.6)

Analysis: Density evolution

a

¿

b

hi→a

ub→j

i

j

Fig. 15.1 Factor graph of a (2,3) regular LDPC code, and notation for the belief propagation messages.

Further, we introduce the a priori log-likelihood for bit i, given the received message yi : Bi =

Q(yi |0) 1 log . 2 Q(yi |1)

(15.7)

For instance, if communication takes place over a binary symmetric channel with flip probability p, one has Bi = 12 log((1 − p)/p) on variable nodes which have received yi = 0, and Bi = − 12 log((1 − p)/p) on those with yi = 1. The BP update equations (15.2) and (15.3) read in this notation (see Fig. 15.1) 4 3 (t) (t+1) (t) (t) (15.8) ub→i , ua→i = atanh tanh hj→a . hi→a = Bi + b∈∂i\a

j∈∂a\i

The hard-decision decoding rule depends on the overall BP log-likelihood (t) (t+1) hi = Bi + ub→i ,

(15.9)

b∈∂i

and is given by (using, for definiteness, a fair-coin outcome in the case of a tie) ⎧ (t) ⎪ if hi > 0, ⎨0 (t) (t) (15.10) x

i (y) = 1 if hi < 0, ⎪ ⎩ (t) 0 or 1 with probability 1/2 if hi = 0.

15.2


Let us study BP decoding of random codes from the ensemble LDPCN (Λ, P ) in the large-block-length limit. The code ensemble is specified by the degree distributions of the variable nodes Λ = {Λl } and of the check nodes P = {Pk }. We assume for (0) simplicity that messages are initialized to ua→i = 0. Because of the symmetry of the channel, under the above hypotheses, the bit (or block) error probability is independent of the transmitted codeword. An explicit derivation of this fact is outlined in Exercise 15.1 below. Thanks to this freedom, we can assume that the all-zero codeword has been transmitted. We shall first write the density evolution recursion as a special case of the recursion given in Section 14.6.2. It turns out that this recursion can be analysed in quite some detail, and, in particular,

¿¿¼


one can show that the decoding performance improves as t increases. The analysis hinges on two important properties of BP decoding and density evolution, related to the notions of ‘symmetry’ and ‘physical degradation’.

Exercise 15.1 The error probability is independent of the transmitted codeword. Assume (t) (t) that the codeword x has been transmitted, and let Bi (x), ua→i (x) and hi→a (x) be the corresponding channel log-likelihoods and messages. Because of the randomness in the channel realization, they are random variables. Furthermore, let σi = σi (x) = +1 if xi = 0, and = −1 otherwise. (a) Prove that the distribution of σi Bi is independent of x. (b) Use the equations (15.8) to prove, by induction over t, that the (joint) distribution of (t) (t) {σi hi→a , σi ua→i } is independent of x. (t)

(c) Use eqn (15.9) to show that the distribution of {σi hi } is independent of x for any b(t) (y) is t ≥ 0. Finally, prove that the distribution of the ‘error vector’ z (t) ≡ x ⊕ x independent of x as well. Write the bit and block error rates in terms of the distribution of z (t) .

15.2.1

Density evolution equations

Let us consider the distribution of messages after a fixed number t of iterations. As we saw in Section 14.6.2, in the large-N limit, the directed neighbourhood of any given edge is with high probability a tree, whose distribution converges to the model Tt (Λ, P ). This implies the following recursive distributional characterization for h(t) and u(t) : d

h(t+1) = B +

l−1

4 3 k−1 d (t) u(t) = atanh tanh hj .

(t)

ub ,

(15.11)

j=1

b=1 (t)

(t)

Here ub , b ∈ {1, . . . , l − 1}, are independent copies of u(t) ; hj , j ∈ {1, . . . , k − 1}, are independent copies of h(t) ; and l and k are independent random integers distributed according to λl and ρk , respectively. Finally, B = 12 log(Q(y|0)/Q(y|1)), where y is independently distributed according to Q(y|0). The recursion is initialized with u(0) = 0. Let us now consider the BP log-likelihood at site i. The same arguments as above (t) d (t) (t) imply hi → h∗ , where the distribution of h∗ is defined by (t+1) d

h∗

=B+

l

(t)

ub ,

(15.12)

b=1 (N,t)

where l is a random integer distributed according to Λl . In particular, if we let Pb be the expected (over an LDPCN (Λ, P ) ensemble) bit error rate for the decoding rule (15.10), then

Analysis: Density evolution (N,t)

lim Pb

N →∞

7 (t) 8 1 7 (t) 8 = P h∗ < 0 + P h∗ = 0 . 2

¿¿½

(15.13)

The suspicious reader will notice that this statement is non-trivial, because f (x) = I(x < 0) + 12 I(x = 0) is not a continuous function. We shall prove the statement below (t) using the symmetry property of the distribution of hi , which allows us to write the bit error rate as the expectation of a continuous function (see Exercise 15.2). 15.2.2

Basic properties: 1. Symmetry

A real random variable Z (or, equivalently, its distribution) is said to be symmetric if 8 7 (15.14) E {f (−Z)} = E e−2Z f (Z) for any function f such that one of the expectations exists. If Z has a density p(z), then the above condition is equivalent to p(−z) = e−2z p(z). Symmetric variables appear naturally in the description of BMS channels. Proposition 15.1 Consider a BMS channel with transition probability Q(y|x). Let Y be the channel output conditional on input 0 (this is a random variable with distribution Q(y|0)), and let B ≡ 21 log(Q(Y |0)/Q(Y |1)). Then B is a symmetric random variable. Conversely, if Z is a symmetric random variable, there exists a BMS channel whose log-likelihood ratio, conditional on the input being 0, is distributed as Z. Proof To avoid technicalities, we prove this claim when the output alphabet Y is a discrete subset of R. In this case, using channel symmetry in the form Q(y|0) = Q(−y|1), we get Q(y|1) Q(y|0) 1 1 log = log Q(y|0) f Q(y|1) f E {f (−B)} = 2 Q(y|0) 2 Q(y|1) y y 7 8 Q(y|0) Q(y|1) 1 f log = E e−2B f (B) . = Q(y|0) (15.15) Q(y|0) 2 Q(y|1) y We now prove the converse. Let Z be a symmetric random variable. We build a channel with output alphabet R as follows. Under input 0, the output is distributed as Z, and under input 1, it is distributed as −Z. In terms of densities, Q(z|0) = p(z) ,

Q(z|1) = p(−z) .

(15.16)

This is a BMS channel with the desired property. Of course, this construction is not unique. Example 15.2 Consider the binary erasure channel BEC(). If the channel input is 0, then Y can take two values, either 0 (with probability 1 − ) or ∗ (probability ). The distribution of B, PB = (1 − ) δ∞ + δ0 , is symmetric. In particular, this is true for the two extreme cases = 0 (a noiseless channel) and = 1 (a completely noisy channel, where PB = δ0 ).

¿¿¾


Example 15.3 Consider the binary symmetric channel BSC(p). The log-likelihood B can take two values, either b0 = 12 log((1 − p)/p) (input 0 and output 0) or −b0 (input 0 and output 1). Its distribution, PB = (1 − p) δb0 + p δ−b0 , is symmetric.

Example 15.4 Finally, consider the binary white-noise additive Gaussian channel BAWGN(σ 2 ). If the channel input is 0, the output Y has a probability density 1 (y − 1)2 q(y) = √ , (15.17) exp − 2σ 2 2πσ 2 i.e. it is a Gaussian of mean 1 and variance σ 2 . The output density for input 1 is determined by the symmetry of the channel; it is therefore a Gaussian of mean −1 and variance σ 2 . The log-likelihood for output y is easily checked to be b = y/σ 2 . Therefore B also has a symmetric Gaussian density, namely # 2 $ σ2 1 σ2 exp − b− 2 . (15.18) p(b) = 2π 2 σ

The variables appearing in the density evolution are symmetric as well. The argument is based on the symmetry of the channel log-likelihood, and the fact that symmetry is preserved by the operations in BP evolution: if Z1 and Z2 are two independent symmetric random variables (not necessarily identically distributed), it is straightforward to show that Z = Z1 + Z2 and Z = atanh[tanh Z1 tanh Z2 ] are both symmetric. Let us now consider the communication of the all-zero codeword over a BMS channel using an LDPC code, but let us first assume that the factor graph associated with the code is a tree. We apply BP decoding with a symmetric random initial condition (0) such as ua→i = 0. The messages passed during the decoding procedure can be regarded as random variables, because of the random received symbols yi (which yield random log-likelihoods Bi ). Furthermore, incoming messages at a given node are independent, since they are functions of Bi ’s (and of initial conditions) on disjoint subtrees. From the above remarks, and looking at the BP equations (15.8), it follows that the mes(t) (t) (t) sages ua→i and hi→a , as well as the overall log-likelihoods, hi are symmetric random variables at all t ≥ 0. Therefore we can state the following proposition. Proposition 15.5 Consider BP decoding of an LDPC code under the above assump(t) tions. If Bi→a,t+1 (F ) is a tree, then hi→a is a symmetric random variable. Analogously, (t) if Bi,t+1 (F ) is a tree, then Hi is a symmetric random variable. (t)

Proposition 15.6 The density-evolution random variables {h(t) , u(t) , H∗ } are symmetric.


¿¿¿

Exercise 15.2 Using Proposition 15.5 and the fact that, for any finite t, Bi→a,t+1 (F ) is a tree with high probability as N → ∞, show that ( lim

N →∞

(N,t) Pb

= lim E N →∞

)

N 1 X (t) f (hi ) N i=1

,

(15.19)

where f (x) = 1/2 for x ≤ 0 and f (x) = e−2x /2 otherwise.

Symmetry does not hold uniquely for the BP log-likelihood; it also holds for the actual (MAP) log-likelihood of a bit, as shown in the exercise below.

Exercise 15.3 Consider the actual (MAP) log-likelihood for bit i (as opposed to its BP approximation). This is defined as hi (y) =

P{xi = 0|y} 1 log . 2 P{xi = 1|y}

(15.20)

Q If we condition on the all-zero codeword being transmitted, so that P(y) = i Q(yi |0), then the random variable Hi = hi (y) is symmetric. This can be shown as follows. Q (a) Suppose that a codeword z = 0 has been transmitted, so that P(y) = i Q(yi |zi ), and define in this case a random variable associated with the log-likelihood of bit xi as (z) (z) d (z) d Hi = hi (y). Show that Hi = Hi if zi = 0, and Hi = −Hi if zi = 1. (b) Consider the following process. A bit zi is chosen uniformly at random. Then a codeword z is chosen uniformly at random conditional on the value of zi , and transmit(z) ted through a BMS channel, yielding an output y. Finally, the log-likelihood Hi is computed. If we hide the intermediate steps in a black box, this can be seen as a (z) communication channel zi → Hi . Show that this is a BMS channel. (c) Show that Hi is a symmetric random variable.

The symmetry property is a generalization of the Nishimori condition that we encountered for spin glasses. As can be recognized from eqn (12.7), the Nishimori condition is satisfied if and only if, for each coupling constant J, βJ is a symmetric random variable. Whereas symmetry occurs only at very special values of the temperature for spin glasses, it holds generically for decoding. The common mathematical origin of these properties can be traced back to the structure discussed in Section 12.2.3. 15.2.3

Basic properties: 2. Physical degradation

It turns out that, for large block lengths, BP decoding gets better as the number of iterations t increases (although it does not necessarily converge to the correct values). This is an extremely useful result, which does not hold when BP is applied to general inference problems. A precise formulation of this statement is provided by the notion

¿¿


x

x

y2

BMS(2)

y1

BMS(1)

C

y2

Fig. 15.2 The channel BMS(2) (top) is said to be physically degraded with respect to BMS(1) if it is equivalent to the concatenation of BMS(1) with a second channel C (bottom).

of physical degradation. We shall first define this notion in terms of BMS channels, and then extend it to all symmetric random variables. This allows us to apply it to the random variables encountered in BP decoding and density evolution. Let us start with the case of BMS channels. Consider two such channels, denoted as BMS(1) and BMS(2); we denote their transition matrices by {Q1 (y|x)}, {Q2 (y|x)} and the corresponding output alphabets by Y1 , Y2 . We say that BMS(2) is physically degraded with respect to BMS(1) if there exists a third channel C with input alphabet Y1 and output Y2 such that BMS(2) can be regarded as the concatenation of BMS(1) and C. By this we mean that passing a bit through BMS(1) and then feeding the output to C is statistically equivalent to passing the bit through BMS(2). If the transition matrix of C is {R(y2 |y1 )}, this can be written in formulae as

Q2 (y2 |x) =

R(y2 |y1 ) Q1 (y1 |x) ,

(15.21)

y1 ∈Y1

where, to simplify the notation, we have assumed Y1 to be discrete. A pictorial representation of this relationship is provided in Fig. 15.2. A formal way of expressing the same idea is that there exists a Markov chain X → Y1 → Y2 . Whenever BMS(2) is physically degraded with respect to BMS(1), we shall write BMS(1) & BMS(2) (which is read as: ‘BMS(1) is “less noisy than” BMS(2)’). Physical degradation is a partial ordering: if BMS(1) & BMS(2) and BMS(2) & BMS(3), then BMS(1) & BMS(3). Furthermore, if BMS(1) & BMS(2) and BMS(2) & BMS(1), then BMS(1) = BMS(2). However, given two binary memoryless symmetric channels, they are not necessarily ordered by physical degradation (i.e. it can happen that neither BMS(1) & BMS(2) nor BMS(2) & BMS(1)). Here are a few examples of channel pairs ordered by physical degradation.


¿¿

Example 15.7 Let 1 , 2 ∈ [0, 1] with 1 ≤ 2 . The corresponding erasure channels are then ordered by physical degradation: BEC(1 ) & BEC(2 ). Consider a channel C that has input and output alphabets Y = {0, 1, ∗} (the symbol ∗ representing an erasure). On the inputs 0, 1, it transmits the input unchanged with probability 1 − x and erases it with probability x. On the input ∗, it outputs an erasure. If we concatenate this channel with the output of BEC(1 ), we obtain a channel BEC(), with = 1 − (1 − x)(1 − ) (the probability that a bit is not erased is the product of the probability that it is not erased by each of the component channels). The claim is thus proved by taking x = (2 − 1 )/(1 − 1 ).

Exercise 15.4 If p1 , p2 ∈ [0, 1/2] with p1 ≤ p2 , then BSC(p1 ) BSC(p2 ). This can be proved by showing that BSC(p2 ) is equivalent to the concatenation of BSC(p1 ) with a second binary symmetric channel BSC(x). What value of the crossover probability x must one take?

Exercise 15.5 If σ12 , σ22 ∈ [0, ∞[ with σ12 ≤ σ22 , show that BAWGN(σ12 ) BAWGN(σ22 ).

If BMS(1) & BMS(2), most measures of the ‘reliability’ of the channel are ordered accordingly. Let us discuss two important such measures here: (1) the conditional entropy and (2) the bit error rate. (1) Let Y1 and Y2 be the outputs of passing a uniformly random bit through channels BMS(1) and BMS(2), respectively. Then H(X|Y1 ) ≤ H(X|Y2 ) (the uncertainty in the transmitted bit is larger for the ‘noisier’ channel). This follows immediately from the fact that X → Y1 → Y2 is a Markov chain by applying the data-processing inequality (see Section 1.4). (2) Assume that the outputs of channels BMS(1) and BMS(2) are y1 and y2 ,

a (ya ) = arg maxx P{X = respectively. The MAP decision rule for x, knowing ya , is x (a) xa (ya ) = x}. x|Ya = ya }, with a = 1, 2. The corresponding bit error rate is Pb = P{

(1) (2) Let us show that Pb ≤ Pb . As BMS(1) & BMS(2), there is a channel C such that (2) BMS(1) concatenated with C is equivalent to BMS(2). Then Pb can be regarded as the bit error rate for a non-MAP decision rule, given y1 . The rule is: transmit y1 through C, denote the output by y2 , and then compute x

2 (y2 ). This non-MAP decision rule cannot be better than the MAP rule applied directly to y1 . Since symmetric random variables can be associated with BMS channels (see Proposition 15.1), the notion of physical degradation of channels can be extended to symmetric random variables. Let Z1 , Z2 be two symmetric random variables and let BMS(1), BMS(2) be the associated BMS channels, constructed as in the proof of Proposition 15.1. We say that Z2 is physically degraded with respect to Z1 (and write Z1 & Z2 ) if BMS(2) is physically degraded with respect to BMS(1). It can be proved

¿¿


that this definition is in fact independent of the choice of BMS(1) and BMS(2) within the family of BMS channels associated with Z1 , Z2 . The interesting result is that BP decoding behaves in the intuitively most natural way with respect to physical degradation. As above, we now fix a particular LDPC code and look at BP messages as random variables due to the randomness in the received vector y. Proposition 15.8 Consider communication over a BMS channel using an LDPC code under the all-zero-codeword assumption, and BP decoding with the standard initial (0) (1) (t−1) (t) ! hi for any condition X = 0. If Bi,r (F ) is a tree, then hi ! hi ! · · · ! hi (0) (1) (t−1) (t) t ≤ r −1. Analogously, if Bi→a,r (F ) is a tree, then hi→a ! hi→a ! · · · ! hi→a ! hi→a for any t ≤ r − 1. We shall not prove this proposition in full generality here, but rather prove its most useful consequence for our purpose, namely the fact that the bit error rate decreases monotonically with t. (t) Proof Under the all-zero-codeword assumption, the bit error rate is P{

xi = (t) (t) 1} = P{hi < 0} (for the sake of simplicity, we neglect the case hi = 0 here). (t) Assume Bi,r (F ) to be a tree, and fix t ≤ r − 1. We want to show that P{hi < (t−1) < 0}. The BP log-likelihood after T iterations on the original graph, 0} ≤ P{hi (t) hi , is equal to the actual (MAP) log-likelihood for the reduced model defined on the tree Bi,t+1 (F ). More precisely, let us denote by Ci,t the LDPC code associated with the factor graph Bi,t+1 (F ), and imagine the following process. A uniformly random codeword in Ci,t is transmitted through the BMS channel, yielding an output y t . We define the log-likelihood ratio for bit xi as # $ P(xi = 0|y t ) 1 (t)

, (15.22) hi = log 2 P(xi = 1|y t ) (t)

i . Clearly, P{

xi = 1|xi = 0} = P{hi < 0}. and denote the MAP estimate for xi by x Instead of this MAP decoding, we can imagine that we scratch all the received symbols at a distance t from i, and then perform MAP decoding on the reduced information. We denote the resulting estimate by x

i . The vector of non-erased symbols is y t−1 . The corresponding log-likelihood is clearly the BP log-likelihood after t − 1 iterations. Therefore P{

xi = 1|xi = 0} = P{hi < 0}. By the optimality of the xi = xi }, which proves our claim. MAP decision rule, P{

xi = xi } ≤ P{

In the case of random LDPC codes, Bi,r (F ) is a tree with high probability for any fixed r, in the large-block-length limit. Therefore Proposition 15.8 has an immediate consequence in the asymptotic setting. (t−1)

Proposition 15.9 The density-evolution random variables are ordered by physical (0) degradation. Namely, h(0) ! h(1) ! · · · ! h(t−1) ! h(t) ! · · · . Analogously, h∗ ! (1) (t−1) (t) ! h∗ ! · · · . As a consequence, the asymptotic bit error rate after h∗ ! · · · ! h∗ (t) (N,t) a fixed number t of iterations, Pb ≡ limN →∞ Pb , decreases monotonically with t.

Analysis: Density evolution 0.20 0.6

0.6

0.15 0.3

0.15 0.3

0.10

0.0 0.0

0.1

0.2

Pb

Pb

0.20

¿¿

0.05

0.00 0.00

0.10

0.0 0.0

0.1

0.2

0.05

0.05

0.10 p

0.15

0.20

0.00 0.00

0.05

0.10

0.15

0.20

p

Fig. 15.3 Predicted performance of two LDPC ensembles with a binary symmetric channel. The curves were obtained through a numerical solution of density evolution, using a population dynamics algorithm with a population size of 5 × 105 . Left: the (3, 6) regular ensemble. Right: an optimized irregular ensemble with the same design rate Rdes = 1/2, and degree distribution pair Λ(x) = 0.4871 x2 +0.3128 x3 +0.0421 x4 +0.1580 x10 , P (x) = 0.6797 x7 + 0.3203 x8 . The dotted curves give the bit error rate obtained after t = 1, 2, 3, 6, 11, 21 and 51 iterations (from top to bottom), and the bold continuous lines refer to the limit t → ∞. In the insets we plot the expected conditional entropy EH(Xi |Y ).

(t)

Exercise 15.6 An alternative measure of the reliability of hi is provided by the conditional entropy. Assuming that a uniformly random codeword is transmitted, this is given (t) by Hi (t) = H(Xi |hi ). (a) Prove that if Bi,r (F ) is a tree, then Hi (t) decreases monotonically with t for t ≤ r − 1. (t)

(b) Assume that,Runder the all-zero-codeword assumption, hi that Hi (t) = log(1 + e−2z ) dpt (z) . [Hint: Remember that pt (.) is a symmetric distribution.]

15.2.4

has a density pt (.). Show

Numerical implementation, and threshold

Density evolution is a useful tool because it can be simulated efficiently. One can estimate numerically the distributions of the density evolution variables {h(t) , u(t) }, (t) and also {h∗ }. As we have seen, this gives access to the properties of BP decoding in (t) the large-block-length limit, such as the bit error rate Pb after t iterations. A possible approach1 consists in representing the distributions by samples of some fixed size S. This leads to the population dynamics algorithm discussed in Section 14.6.4. (t) (t) This algorithm generates, at each time t ∈ {0, . . . , T }, two populations {h1 , · · · , hNpop } 1 An alternative approach is as follows. Both of the maps (15.11) can be regarded as convolutions of probability densities for an appropriate choice of the message variables. For the first map, this is immediate in terms of log-likelihoods. For the second map, one can use variables r (t) = (sign h(t) , log | tanh h(t) |) and s(t) = (sign u(t) , log | tanh y (t) |)). By using a fast Fourier transform to implement convolutions, this can result in a significant speed-up of the calculation.

¿¿


Table 15.1 Belief propagation thresholds for a binary symmetric channel, for a few regular LDPC ensembles. The third column lists the design rate 1 − l/k.

l 3 3 3 4 (t)

k 4 5 6 6

Rdes 1/4 2/5 1/2 1/3

pd 0.1669(2) 0.1138(2) 0.0840(2) 0.1169(2)

Shannon limit 0.2145018 0.1461024 0.1100279 0.1739524

(t)

and {u1 , · · · , uNpop } which are approximately i.i.d. variables distributed as h(t) and u(t) , respectively. From these populations, one can estimate the bit error rate following eqn (15.13). More precisely, the population dynamics estimate is (t),pop dyn

Pb

=

l(n) R & 1 % (t−1) ϕ Bn + uin (j) , R n=1 j=1

(15.23)

where ϕ(x) ≡ 1 if x > 0, ϕ(0) = 1/2, and ϕ(x) = 0 otherwise. Here the Bn are distributed as 12 log(Q(y|0)/Q(y|1)), l(n) is distributed as Λl , and the indices in (1), . . . , in (l) are uniformly random in {1, . . . , Npop }. The parameter R is usually taken to be of the same order as the population size. In Fig. 15.3, we report the results of population dynamics computations for two different LDPC ensembles used with a binary symmetric channel with crossover prob(t) ability p. We consider two performance measures: the bit error rate Pb and the conditional entropy H (t) , which can also be easily estimated from the population. (t) As follows from Proposition 15.9, Pb and H (t) are monotonically decreasing functions of the number of iterations. One can also show that they are monotonically (t) increasing functions of p. Since Pb is non-negative and decreasing in t, it has a finite (t) BP limit PBP b ≡ limt→∞ Pb , which is itself non-decreasing in p (the limit curve Pb has (t) been estimated in Fig. 15.3 by choosing t large enough so that Pb is independent of t within the numerical accuracy). The BP threshold is defined as 7 8 . (15.24) pd ≡ sup p ∈ [0, 1/2] : PBP b (p) = 0 Here the subscript ‘d’ stands for ‘dynamical’: its intrinsic meaning and its relation to phase transitions in other combinatorial problems will be discussed in Chapter 21. Analogous definitions can be provided for other channel families such as the erasure channel BEC() and the Gaussian channel BAWGN(σ 2 ). In general, the definition (15.24) can be extended to any family of BMS channels BMS(p) indexed by a real parameter p which orders the channels with respect to physical degradation. Numerical simulation of the density evolution allows one to determine the BP threshold pd with good accuracy. We report the results of a few such simulations in Table 15.1. Let us stress that the threshold pd has an important practical meaning. For any p < pd , one can achieve an arbitrarily small bit error rate with high probability just by picking one random code from the ensemble LDPCN (Λ, P ) with large N and


¿¿

decoding it using BP with a large enough (but independent of N ) number of iterations. For p > pd , the bit error rate is asymptotically bounded from below by PBP b (p) > 0 for any fixed number of iterations (in practice, it turns out that doing more iterations, say N a , does not help). The value of pd is therefore a primary measure of the performance of a code. One possible approach to the design of good LDPC codes is to use random ensembles, and optimize the degree distribution. For instance, one can look for the degree distribution pair (Λ, P ) with the largest BP threshold pBP , given a certain design rate Rdes = 1 − P (1)/Λ (1). In the simple case of communication over a BEC, the optimization over the degree distributions can be carried out analytically, as we shall see in Section 15.3. For general BMS channels, it can be done numerically: one computes the threshold noise level for a given degree distribution pair using density evolution, and maximizes it by a local search procedure. Figure 15.3 shows an example of an optimized irregular ensemble with rate 1/2 for a BSC, including variable nodes of degrees 2, 3, 4 and 10 and check nodes of degrees 7 and 8. Its threshold is pd ≈ 0.097 (while the Shannon limit is 0.110). Note that this ensemble has a finite fraction of variable nodes of degree 2. We can use the analysis in Chapter 11 to compute its weight enumerator function. It turns out that the parameter A in eqn (11.23) is positive. This optimized ensemble has a large number of codewords with small weight. It is surprising, and not very intuitive, that a code such that there exist codewords at a sublinear distance from the transmitted codeword nevertheless has a large BP threshold pd . It turns out that this phenomenon is very general: code ensembles that approach the Shannon capacity turn out to have bad ‘short-distance properties’. In particular, the weight enumerator exponent, discussed in Section 11.2, is positive for all values of the normalized weight. Low-weight codewords do not spoil the performance in terms of pd . They are not harmless, though: they degrade the performance of the code at moderate block lengths N , below the threshold pd . Further, they prevent the block error probability from vanishing as N goes to infinity (in each codeword, a fraction 1/N of the bits is decoded incorrectly). This phenomenon is referred to as the error floor.

Exercise 15.7 While the BP threshold (15.24) was defined in terms of the bit error rate, any other ‘reasonable’ measure of the error in the decoding of a single bit would give the same result. This can be shown as follows. Let Z be a symmetric random variable and Pb ≡ P{Z < 0} + 12 P{Z = 0}. Show that, for any ∆ > 0, P{Z < ∆} ≤ (2 + e2∆ )Pb . Consider a sequence of symmetric random variables {Z (t) } such that the sequence of (t) Pb defined as before goes to 0. Show that the distribution of Z (t) becomes a Dirac delta at plus infinity as t → ∞.

15.2.5

Local stability

In addition to numerical computation, it is useful to derive simple analytical bounds on the BP threshold. A particularly interesting bound is provided by a local stability analysis. It applies to any BMS channel, and the result depends on the specific channel only through its Bhattacharya parameter B ≡ y Q(y|0)Q(y|1) ≤ 1. This

¿


parameter, which we have already encountered in Chapter 11, is a measure of the noise level of the channel. It preserves the ordering by physical degradation (i.e. the Bhattacharya parameters of two channels BMS(1) & BMS(2) satisfy B(1) ≤ B(2)), as can be checked by explicit computation. The local stability condition depends on the LDPC codethrough the fraction of vertices with degree 2, Λ2 = λ (0), and the value of ρ (1) = k Pk k(k − 1)/ k Pk k. It is expressed as follows. Theorem 15.10 Consider communication of the all-zero codeword over a binary memoryless symmetric channel with Bhattacharyia parameter B, using random elements from the ensemble LDPCN (Λ, P ) and belief propagation decoding in which the initial (0) (t,N ) be the bit messages ua→i are i.i.d. copies of a symmetric random variable. Let Pb (t) (t,N ) error rate after t iterations, and Pb = limN →∞ Pb . 1. If λ (0)ρ (1)B < 1, then there exists ξ > 0 such that, if Pb < ξ for some ξ, then (t) Pb → 0 as t → ∞. (t)

2. If λ (0)ρ (1)B > 1, then there exists ξ > 0 such that Pb > ξ for any t. (t)

Corollary 15.11 We define the local stability threshold ploc as ploc = inf

7

8 p | λ (0)ρ (1)B(p) > 1 .

(15.25)

The BP threshold pBP for decoding a communication over an ordered channel family BMS(p) using random codes from the ensemble LDPCN (Λ, P ) satisfies pd ≤ ploc .

We shall not give the full proof of this theorem, but will explain the stability argument that underlies it. If the minimum variable-node degree is 2 or larger, the d density evolution recursion (15.11) has as a fixed point h, u = Z∞ , where Z∞ is a random variable that takes the value +∞ with probability 1. The BP threshold pd is the largest value of the channel parameter such that {h(t) , u(t) } converge to this fixed point as t → ∞. It is then quite natural to ask what happens if the density evolution recursion is initiated with some random initial condition that is ‘close enough’ to Z∞ . To this end, we consider the initial condition 0 with probability , (15.26) X= +∞ with probability 1 − . This is nothing but the log-likelihood distribution for a bit transmitted through a binary erasure channel, with erasure probability . Let us now apply the density evolution recursion (15.11) with the initial condition l−1 d d u(0) = X. At the first step, we have h(1) = B + b=1 Xb , where the {Xb } are i.i.d.

¿

Analysis: Density evolution d

copies of X. Therefore h(1) = +∞ unless X1 = · · · = Xl−1 = 0, in which case h(1) = B. We therefore have B with prob. l−1 , (1) (15.27) With probability λl : h = +∞ with prob. 1 − l−1 , where B is distributed as the channel log-likelihood. Since we are interested in the behaviour ‘close’ to the fixed point Z∞ , we linearize with respect to , thus getting ⎧ with prob. λ2 + O(2 ), ⎨B (1) (15.28) h = +∞ with prob. 1 − λ2 + O(2 ), ⎩ ··· with prob. O(2 ). The last line has not been given in full here, but it will become necessary in subsequent iterations. It signals that h(1) could take some other value with a negligible probability. k−1 (1) Next, consider the first iteration at the check node side: u(1) = atanh{ j=1 tanh hj }. (1)

(1)

To first order in , we need to consider only two cases: Either h1 = · · · = hk−1 = +∞ (this happens with probability 1 − (k − 1)λ2 + O(2 )), or one of the log-likelihoods is distributed like B (with probability (k−1)λ2 +O(2 )). Averaging over the distribution of k, we get ⎧ with prob. λ2 ρ (1) + O(2 ), ⎨B (15.29) u(1) = +∞ with prob. 1 − λ2 ρ (1) + O(2 ), ⎩ ··· with prob. O(2 ). Repeating the argument t times (and recalling that λ2 = λ (0)), we get ⎧ with prob. (λ (0)ρ (1))t + O(2 ), ⎨ B1 + · · · + Bt (t) with prob. 1 − (λ (0)ρ (1))t + O(2 ), (15.30) h = +∞ ⎩ ··· with prob. O(2 ). 7 8 The bit error rate vanishes if and only P(t; ) = P h(t) ≤ 0 goes to 0 as t → ∞. The above calculation shows that 7 8 (15.31) P(t; ) = (λ (0)ρ (1))t P B1 + · · · + Bt ≤ 0 + O(2 ) . The probability of B1 +· · ·+Bt ≤ 0 can be computed, to leading exponential order, using the large-deviations estimates of Section 4.2. In particular, we saw in Exercise 4.2 that t 8 . 7 −zB ] . (15.32) P B1 + · · · + Bt ≤ 0 = inf E [e z≥0

We leave to the reader the exercise of showing that, since B is a symmetric random variable, E e−zB is minimized for z = 1, thus yielding 7 8 . P B1 + · · · + Bt ≤ 0 = Bt . (15.33) As a consequence, the order- coefficient in eqn (15.31) behaves, to leading exponential order, as (λ (0)ρ (1)B)t . Depending on whether λ (0)ρ (1)B < 1 or λ (0)ρ (1)B > 1, the density evolution may or may not converge to the error-free fixed point if initiated sufficiently close to it. The full proof relies on these ideas, but it requires one to control the terms of higher order in , and other initial conditions as well.

¿


15.3

BP decoding for an erasure channel

We now focus on an erasure channel BEC(). The analysis can be greatly simplified in this case: the BP decoding algorithm has a simple interpretation, and the density evolution equations can be studied analytically. This allows us to construct capacityachieving ensembles, i.e. codes which are, in the large-N limit, error-free up to a noise level given by Shannon’s threshold. 15.3.1

BP, the peeling algorithm, and stopping sets (0)

We consider BP decoding, with initial condition ua→i = 0. As can be seen from eqn (15.7), the channel log-likelihood Bi can take three values: +∞ (if a 0 has been received at position i), −∞ (if a 1 has been received at position i), or 0 (if an erasure occurred at position i). It follows from the update equations (15.8) that the messages exchanged at any subsequent time take values in {−∞, 0, +∞} as well. Consider first the equation for (t) (t) the check nodes. If one of the incoming messages hj→a is 0, then ua→i = 0 as well. (t)

(t)

If, on the other hand, hj→a = ±∞ for all incoming messages, then ua→i = ±∞ (the sign being the product of the incoming signs). Next, consider the update equation for (t) the variable nodes. If ub→i = 0 for all of the incoming messages, and Bi = 0 as well, (t+1) then of course hi→a = 0. If, on the other hand, some of the incoming messages or (t+1) the received value Bi takes the value ±∞, then hi→a takes the same value. Notice that there can never be contradictory incoming messages (i.e. both +∞ and −∞) at a variable node.

Exercise 15.8 Show that, if contradictory messages were sent to the same variable node, this would imply that the transmitted message was not a codeword.

The meaning of the three possible messages ±∞ and 0 and of the update equations (t) (t) is very clear in this case. Each time the message hi→a or ua→i is +∞ (or −∞), this means that the bit xi is 0 (or 1, respectively) in all codewords that coincide with the channel output at the non-erased positions: the value of xi is perfectly known. Vice (t) (t) versa, if hi→a = 0 (or ua→i = 0), the bit xi is currently considered equally likely to be either 0 or 1. The algorithm is very simple: each message changes value at most one time, either from 0 to +∞, or from 0 to −∞.

(t)

Exercise 15.9 To show this, consider the first time, t1 , at which a message hi→a changes from +∞ to 0. Find what has happened at time t1 − 1.

Therefore a fixed point is reached after a number of updates less than or equal to the number of edges N Λ (1). There is also a clear stopping criterion: if, in one update

BP decoding for an erasure channel (t)

¿

(t+1)

round, no progress is made (i.e. if hi→a = hi→a for all directed edges i → a), then no progress will be made in subsequent rounds. An alternative formulation of BP decoding is provided by the peeling algorithm. The idea is to view decoding as a linear-algebra problem. The code is defined through a linear system over Z2 , of the form Hx = 0. The output of an erasure channel fixes a fraction of the bits in the vector x (the non-erased bits). One is left with an inhomogeneous linear system L over the remaining erased bits. Decoding amounts to using this new linear system to determine the bits erased by the channel. If an equation in L contains a single variable xi with a non-vanishing coefficient, it can be used to determine xi , and replace it everywhere. One can then repeat this operation recursively until either all of the variables have been fixed (in which case the decoding is successful) or the residual linear system includes only equations over two or more variables (in which case the decoder gets stuck).

Exercise 15.10 An explicit characterization of the fixed points of the peeling algorithm can be given in terms of stopping sets (or 2-cores). A stopping set is a subset of variable nodes in the factor graph such that each check node has a number of neighbours in the subset which is either zero or at least 2. Let S be the subset of undetermined bits when the peeling algorithm stops. (a) Show that S is a stopping set. (b) Show that the union of two stopping sets is a stopping set. Deduce that, given a subset of variable nodes U , there exists a unique ‘largest’ stopping set contained in U that contains any other stopping set in U . (c) Let U be the set of erased bits. Show that S is the largest stopping set contained in U .

Exercise 15.11 We prove here that the peeling algorithm is indeed equivalent to BP decoding. As in the previous exercise, we denote by S the largest stopping set contained in the erased set U . (t)

(t)

(a) Prove that, for any edge (i, a) with i ∈ S, ua→i = ha→i = 0 at all times. (b) Vice versa, let S be the set of bits that are undetermined by BP after a fixed point is reached. Show that S is a stopping set. (c) Deduce that S = S (use the maximality property of S).

15.3.2

Density evolution

Let us study BP decoding of an LDPCN (Λ, P ) code after communication through a binary erasure channel. Under the assumption that the all-zero codeword has been transmitted, messages will take values in {0, +∞}, and their distribution can be parameterized by a single real number. We denote by zt the probability that h(t) = 0, and by zˆt the probability that u(t) = 0. The density evolution recursion (15.11) translates into the following recursion on {zt , zˆt }:

¿


1

1

0.8

0.8

= 0.4

= 0.45

0.6

0.6

F (z) 0.4

0.4

0.2

0.2

0

0 0

0.2

0.4

z

0.6

0.8

1

0

0.2

0.4

z

0.6

0.8

1

Fig. 15.4 Density evolution for the (3, 6) LDPC ensemble over the erasure channel BEC(), for two values of , below and above the BP threshold d = 0.4294.

zt+1 = λ(ˆ zt ) ,

zˆt = 1 − ρ(1 − zt ) .

(15.34)

We can eliminate zˆt from this recursion to get zt+1 = F (zt ), where we have defined F (z) ≡ λ(1 − ρ(1 − z)). The bit error rate after t iterations in the large-block-length (t) zt ). limit is Pb = Λ(ˆ In Fig. 15.4, we show as an illustration the recursion zt+1 = F (zt ) for the (3, 6) regular ensemble. The edge-perspective degree distributions are λ(z) = z 2 and ρ(z) = z 5 , so that F (z) = [1 − (1 − z)2 ]5 . Notice that F (z) is a monotonically increasing function with F (0) = 0 (if the minimum variable-node degree is at least 2) and F (1) = < 1. As a consequence, the sequence {zt } is decreasing and converges at large t to the largest fixed point of F . In particular, zt → 0 (and consequently = 0) if and only if F (z) < z for all z ∈ ]0, 1]. This yields the following explicit PBP b characterization of the BP threshold: d = inf

z : λ(1 − ρ(1 − z))

z ∈]0, 1] .

(15.35)

It is instructive to compare this characterization with the local stability threshold, which in this case reads loc = 1/λ (0)ρ (1). It is obvious that d ≤ loc , since loc = limz→0 z/λ(1 − ρ(1 − z)). Two cases are possible, as illustrated in Fig. 15.5: either d = loc or d < loc . Each case corresponds to a different behaviour of the bit error rate. If d = loc , then, BP 2 generically,2 PBP b () is a continuous function of at d , with Pb (d + δ) = Cδ + O(δ ) BP just above the threshold. If, on the other hand, d < loc , then Pb () is discontinuous BP,∗ at d , with PBP + Cδ 1/2 + O(δ) just above threshold. b (d + δ) = Pb 2 Other behaviours are possible, but they are not ‘robust’ with respect to a perturbation of the degree sequences.

BP decoding for an erasure channel

Pb

1

1

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0

¿

0 0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

Fig. 15.5 The bit error rate under BP decoding for the (3, 6) (left) and (2, 4) (right) ensembles. The prediction of density evolution (bold lines) is compared with numerical simulations (averaged over 10 code/channel realizations with block length N = 104 ). For the (3, 6) ensemble, where BP ≈ 0.4294 < loc = ∞, the transition is discontinuous. For the (2, 4) ensemble, where BP = loc = 1/4, the transition is continuous.

Exercise 15.12 Consider communication over a binary erasure channel using random elements from the regular (l, k) ensemble, in the limit k, l → ∞, with a fixed rate Rdes = 1−l/k. Prove that the BP threshold d tends to 0 in this limit.

15.3.3

Ensemble optimization

The explicit characterization (15.35) of the BP threshold for a binary erasure channel opens the way to the optimization of the code ensemble. One possible set-up is the following. We fix an erasure probability ∈ ]0, 1[: this is the estimated noise level in the channel that we are going to use. For a given degree sequence pair (λ, ρ),let d (λ, ρ) denote the corresponding BP threshold, and let R(λ, ρ) = 1−( k ρk /k)/( l λl /l) be the design rate. Our objective is to maximize the rate, while keeping d (λ, ρ) ≤ . Let us assume that the check node degree distribution ρ is given. Finding the optimal variable-node degree distribution can then be recast as an (infinite-dimensional) linear programming problem: ⎧ maximize ⎪ l λl /l , ⎪ ⎪ ⎨ subject to l λl = 1 ⎪ ≥ 0 ∀l, λ ⎪ l ⎪ ⎩ λ(1 − ρ(1 − z)) ≤ z ∀ z ∈ ]0, 1] .

(15.36)

Note thatthe constraint λ(1 − ρ(1 − z)) ≤ z conflicts with the requirement of maximizing l λl /l, since both are increasing functions in each of the variables λl . As is usual with linear programming, one can show that the objective function is maximized when the constraints are satisfied with equality, i.e. λ(1 − ρ(1 − z)) = z for all z ∈ ]0, 1]. This ‘matching condition’ allows one to determine λ for a given ρ.

¿


1 k=4 k=6 k=8 k = 10 k = 12

0.8

0.6

Pb 0.4

0.2

0 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Fig. 15.6 Belief propagation bit error rate for LDPCN (Λ, P ) ensembles from the capacity-achieving sequence (λ(k) , ρ(k) ) defined in the main text. The sequence is constructed in such a way as to achieve capacity at a noise level = 0.5 (the corresponding capacity is C() = 1 − = 0.5). The five ensembles considered here have design rates Rdes = 0.42253, 0.48097, 0.49594, 0.49894 and 0.49976 (for k = 4, 6, 8, 10 and 12, respectively).

We shall do this in the simple case where the check nodes have a uniform degree k, i.e. ρ(z) = z k−1 . The saturation condition implies λ(z) = (1/)[1 − (1 − z)1/(k−1) ]. By Taylor expanding this expression, we get, for l ≥ 2,

λl =

(−1)l Γ (1/(k − 1) + 1) . Γ(l) Γ (1/(k − 1) − l + 2)

(15.37)

In particular, λ2 = 1/((k − 1)), λ3 = (k − 2)/(2(k − 1)2 ), and λl λ∞ l−k/(k−1) as l → ∞. Unhappily, this degree sequence does not satisfy the normalization condition in (15.36). In fact, l λl = λ(1) = 1/. This problem can, however, be overcome by truncating the series and letting k → ∞, as shown in the exercise below. The final result is that a sequence of LDPC ensembles can be found that allows reliable communication under BP decoding, at a rate that asymptotically achieves the channel capacity C() = 1 − . This is stated more formally below. Theorem 15.12 Let ∈ (0, 1). Then there exists a sequence of degree distribution pairs (λ(k) , ρ(k) ), with ρ(k) (x) = xk−1 , such that d (λ(k) , ρ(k) ) > and R(λ(k) , ρ(k) ) → 1 − .

The precise construction of the sequence (λ(k) , ρ(k) ) is outlined in the next exercise. In Fig. 15.6, we show the BP error probability curves for this sequence of ensembles.

The Bethe free energy and MAP decoding

¿

ˆ (k) (z) = (1/)[1−(1−z)1/(k−1) ], and zL = PL λ ˆ (k) Exercise 15.13 Let ρ(k) (z) = z k−1 , λ l=2 l . (k) (k) ˆ /zL(k,) if Define L(k, ) as the smallest value of L such that zL ≥ 1. Finally, set λl = λ l (k) l ≤ L(k, ) and λl = 0 otherwise. (a) Show that λ(k) (1 − ρ(k) (1 − z)) < z for all z ∈ ]0, 1] and, as a consequence, d (λ(k) , ρ(k) ) > . [Hint: Use the fact that the coefficients λl in eqn (15.37) are non-negative and hence ˆ (k) (z)/zL(k,) .] λ(k) (x) ≤ λ ˆ (k) → 0 as k → ∞. Deduce that L(k, ) → ∞ and (b) Show that, for any sequence l(k), λ l(k) zL(k,) → 1 as k → ∞. (c) Prove that limk→∞ R(λ(k) , ρ(k) ) = limk→∞ 1 − zL(k,) = 1 − .

15.4


So far, we have studied the performance of LDPCN (Λ, P ) ensembles under BP messagepassing decoding, in the large-block-length limit. Remarkably, sharp asymptotic predictions can be obtained for optimal decoding as well, and they involve the same mathematical objects, namely distributions of messages. We shall focus here on symbol MAP decoding for a channel family {BMS(p)} ordered by physical degradation. As in Chapter 11, we can define a threshold pMAP depending on the LDPC ensemble, such that MAP decoding allows one to communicate reliably at all noise levels below pMAP . We shall compute pMAP using the Bethe free entropy. The free entropy of our decoding problem, averaged over the received signal, is defined as Ey log Z(y). Let us see how its value can be related to the properties of MAP decoding. A crucial step to understanding MAP decoding is estimating the typical number of inputs with non-negligible probability for a given channel output. We can quantify this precisely by introducing the ‘codeword entropy density’ hN = (1/N ) EHN (X|Y ), averaged over the code ensemble (throughout this section we shall use natural logarithms in the definition of the entropies, instead of logarithms to base 2). If hN is bounded away from 0 as N → ∞, the typical channel output is likely to correspond to an exponential number of inputs. If, on the other hand, hN → 0, the correct input has to be searched for among a subexponential number of candidates, and one may hope to be able to decode correctly. A precise relation to the error probability is provided by Fano’s inequality (1.28). Proposition 15.13 Denote by PN b the bit error probability for communication using a code of block length N . Then, H(PN b )≥

HN (X|Y ) . N

In particular, if the entropy density HN (X|Y )/N is bounded away from 0, so is PN b .

¿


Although this gives only a bound, it suggests that we can identify the MAP threshold as the largest noise level such that hN → 0 as N → ∞. In other words, we define 3 4 (15.38) pc ≡ sup p : lim hN = 0 , N →∞

and conjecture that, for LDPC ensembles, the bit error rate vanishes asymptotically if p < pc , thus implying pMAP = pc . Hereafter, we shall use pc (or c for a BEC) to denote the MAP threshold. The relation between this and similar phase transitions in other combinatorial problems will be discussed in Chapter 21. The conditional entropy HN (X|Y ) is directly related to the free entropy of the model defined in (15.1). More precisely, we have Q(y|0) log Q(y|0) , (15.39) HN (X|Y ) = Ey log Z(y) − N y

where Ey denotes the expectation with respect to the output vector y. In order to derive this expression, we first use the entropy chain rule to write (dropping the subscript N) H(X|Y ) = H(Y |X) + H(X) − H(Y ) .

(15.40)

Since the input message is uniform over the code, H(X) = N log |C|. Further, since |X) = the channel is memoryless and symmetric, H(Y i H(Yi |Xi ) = N H(Yi |Xi = 0) = −N y Q(y|0) log Q(y|0). Finally, rewriting the distribution (15.1) as p(x|y) =

|C| p(y, x) , Z(y)

(15.41)

we can identify (by Bayes’ theorem) Z(y) = |C| p(y). Equation (15.39) follows by putting together these contributions. The free entropy Ey log Z(y) is the non-trivial term in eqn (15.39). For LDPC codes, in the large-N limit, it is natural to compute it using the Bethe approximation of Section 14.2.4. Suppose u = {ua→i }, h = {hi→a } is a set of messages which solves the BP equations ⎧ ⎫ ⎨ ⎬ ub→i , ua→i = atanh tanh hj→a . (15.42) hi→a = Bi + ⎩ ⎭ b∈∂i\a

j∈∂a\i

Then the corresponding Bethe free entropy follows from eqn (14.28): log νua→i (xi )νhi→a (xi ) F(u, h) = − (ia)∈E

+

N i=1

log

xi

xi

Q(yi |xi )

a∈∂i

νua→i (xi ) +

M a=1

⎡

log ⎣

xa

Ia (x)

(15.43) ⎤ νhi→a (xi )⎦ .

i∈∂a

where we denote the distribution of a bit x whose log-likelihood ratio is u by νu (x), given by νu (0) = 1/(1 + e−2u ), νu (1) = e−2u /(1 + e−2u ).


¿

We are interested in the expectation of this quantity with respect to the code and channel realization, in the N → ∞ limit. As in Section 14.6.3, we assume that messages d d are asymptotically identically distributed, i.e. ua→i = u, hi→a = h, and that messages coming into the same node along distinct edges are asymptotically independent. Under these hypotheses, we get 1 RS Ey F(u, h) = fu,h + Q(y|0) log Q(y|0) , N →∞ N y lim

(15.44)

RS where the ‘shifted’ free-entropy density fu,h associated with the random variables u, h is defined by

RS

fu,h

l Q(y|x) = −Λ (1) Eu,h log νu (x)νh (x) + El,y,{ui } log νu (x) Q(y, 0) i=1 i x x k Λ (1) Ek E{hi } log I (x1 ⊕ · · · ⊕ xk = 0) νhi (xi ) . (15.45) + P (1) x ...x i=1

1

k

Here k and l are distributed according to Pk and Λl , respectively, and u1 , u2 , . . . and h1 , h2 , . . . are i.i.d. and distributed as u and h, respectively. RS is If the Bethe free entropy is correct, the shifted Bethe free-entropy density fu,h equal to the codeword entropy density hN . This reasonable assumption can be turned into a rigorous inequality. Theorem 15.14 If 3 u, h are symmetric random variables satisfying the distributional 4 k−1 l−1 d d and h = B + a=1 ua , then identities u = atanh i=1 tanh hi RS lim hN ≥ fu,h .

N →∞

(15.46)

It is natural to conjecture that the correct limit is obtained by optimizing the above lower bound, i.e. RS lim hN = sup fu,h ,

N →∞

(15.47)

u,h

where, once again, the sup is taken 4 over the pairs of symmetric random variables u, h 3 l−1 d d k−1 satisfying u = atanh i=1 tanh hi and h = B + a=1 ua . This conjecture has indeed been proved in the case of communication over a binary erasure channel for a large class of LDPC ensembles (including, for instance, regular ones). The above equation is interesting because it establishes a bridge between BP and MAP decoding. An example of application of this bridge is given in the next exercise.

¿


0.6

0.02 0.015

0.5

0.01 0.4

0.005

RS fu,h

p(h) 0.3

0

-0.005

0.2

-0.01 0.1

-0.015

0 -4

-3

-2

-1

0

1

2

3

4

h

5

-0.02 0.07 0.075 0.08 0.085 0.09 0.095

0.1

0.105 0.11

p

Fig. 15.7 Illustration of the RS cavity method applied to a (3, 6) regular code used over a binary symmetric channel. Left: a non-trivial distribution of the variables h found by population dynamics at a noise level p = 0.095. Right: shifted free entropy versus p, for the non-trivial solution (the normalization is such that the free entropy of the perfect-decoding phase u = h = ∞ is zero). When the noise level is increased, the non-trivial solution appears at pd , and its free entropy becomes positive at pc

Exercise 15.14 Proof that pd ≤ pc . (a) Recall that the pair of distributions “u, h = +∞ with probability one” constitute a RS = 0 at such a fixed density-evolution fixed point for any noise level. Show that fh,u point. (b) Use ordering by physical degradation to show that if any other fixed point exists, then the density evolution converges to it. (c) Deduce that pd ≤ pc .

Evaluating the expression in eqn (15.47) implies an a priori infinite-dimensional optimization problem. In practice, good approximations can be obtained through the following procedure: 1. Initialize h, u to a pair of symmetric random variables h(0) , u(0) . 2. Implement numerically the density evolution recursion (15.11) by population dynamics, and iterate it until an approximate fixed point is attained. d

RS at such a fixed point, after enforcing u = 3. Evaluate the functional fu,h 4 3 k−1 atanh i=1 tanh hi exactly.

The above procedure can be repeated for several different initializations u(0) , h(0) . The RS is then picked as an estimate for limN →∞ hN . largest of the corresponding values of fu,v While this procedure is not guaranteed to exhaust all of the possible densityevolution fixed points, it allows one to compute a sequence of lower bounds on the conditional entropy density. Further, by analogy with exactly solvable cases (such as that of a binary erasure channel), one expects a small, finite number of densityevolution fixed points. In particular, for regular ensembles and p > pd , a unique


¿

Table 15.2 MAP thresholds for a binary symmetric channel compared with the BP decoding thresholds, for a few regular LDPC ensembles.

l 3 3 3 4

k 4 5 6 6

Rdes 1/4 2/5 1/2 1/3

pd 0.1669(2) 0.1138(2) 0.0840(2) 0.1169(2)

pc 0.2101(1) 0.1384(1) 0.1010(2) 0.1726(1)

Shannon limit 0.2145018 0.1461024 0.1100279 0.1739524

Table 15.3 MAP thresholds for a binary erasure channel compared with the BP decoding thresholds, for a few regular LDPC ensembles.

l 3 3 3 4

k 4 5 6 6

Rdes 1/4 2/5 1/2 1/3

d 0.647426 0.517570 0.429440 0.506132

c 0.746010 0.590989 0.488151 0.665656

Shannon limit 0.750000 0.600000 0.500000 0.666667

(stable) fixed point is expected to exist in addition to the no-error one u, h = +∞. In Table 15.2 we present the corresponding MAP thresholds for a BSC and a few regular ensembles. The whole approach simplifies considerably in the case of communication over a binary erasure channel, as shown in the exercise below.

Exercise 15.15 Consider the erasure channel BEC(), and look for a fixed point of the density evolution equations (15.11) such that (i) h = 0 with probability z and h = ∞ with probability 1 − z, and (ii) u = 0 with probability zˆ and u = ∞ with probability 1 − zˆ. (a) Show that z and zˆ must satisfy the equations (15.34). (b) Show that the shifted free entropy (15.45) is equal to » – Λ (1) RS fu,h = Λ (1)z(1 − zˆ) + (P (1 − z) − 1) + Λ(ˆ z ) log 2 . P (1)

(15.48)

(c) Use this expression and the conjecture (15.47) to obtain the MAP thresholds for regular ensembles listed in Table 15.3.

The two problems of computing the BP and MAP thresholds are thus unified by the use of the RS cavity method. For any noise level p, there always exists a solution to the RS cavity equations in which the distribution of u is a point mass distribution at u = +∞ and the distribution of h is a point mass distribution at h = +∞. This RS = 0. solution corresponds to a perfect decoding; its shifted free entropy density is fu,h When p > pd , another solution to the RS cavity equations appears. Its shifted free entropy density f RS can be computed from eqn (15.44): it is initially negative and

¿


increases with p. The MAP threshold is the value p = pd above which f RS becomes positive. Figure 15.7 illustrates this behaviour. Still, this description leaves us with a puzzle. In the regime pd ≤ p < pc , the codeword entropy density associated with the solution h, u < ∞ is limN →∞ hN ≥ RS < 0. Analogously to what happens in the replica method (see Chapter 8), the fu,h solution should therefore be discarded as unphysical. It turns out that a consistent picture can be obtained only by including replica symmetry breaking, which will be the subject of Chapter 21.

Notes Belief propagation was first applied to the decoding problem by Robert Gallager in his PhD thesis (Gallager, 1963), and called the ‘sum–product’ algorithm there. Several low-complexity alternative message-passing approaches were introduced in the same work, along with the basic ideas of their analysis. The analysis of iterative decoding of irregular ensembles over an erasure channel was pioneered by Luby and co-workers (Luby et al., 1997, 1998, 2001a,b). These papers also presented the first examples of capacity-achieving sequences. Density evolution for general binary memoryless symmetric channels was introduced by Richardson and Urbanke (2001b). The whole subject is surveyed by Richardson and Urbanke (2001a, 2008). One important property that we have left out is ‘concentration’: the error probability under message-passing decoding is, for most of the codes, close to its ensemble average, which is predicted by density evolution. The design of capacity-approaching LDPC ensembles for general BMS channels was discussed by Chung et al. (2001) and Richardson et al. (2001). Since message passing allows efficient decoding, one may wonder whether the encoding (whose complexity is, a priori, O(N 2 )) might become the bottleneck. Luckily this is not the case: efficient encoding schemes were discussed by Richardson and Urbanke (2001c). The use of the RS replica method (equivalent to the cavity method) to characterize MAP decoding for sparse-graph codes was initiated by Kabashima and Saad (1999), who considered Sourlas’s LDGM codes. MN codes (a class of sparse-graph codes defined by MacKay and Neal (1996)) and turbo codes were studied shortly after, by Kabashima et al. (2000a,b) and by Montanari and Sourlas (2000) and Montanari (2000), respectively. Plain regular LDPC ensembles were considered first by Kabashima and Saad (2000), who considered the problem on a tree, and by Nakamura et al. (2001). The effect of replica symmetry breaking was first investigated by Montanari (2001b) and standard irregular ensembles were studied by Franz et al. (2002). The fact that the RS cavity method yields the exact value of the MAP threshold and that pMAP = pc has not yet been proven rigorously in a general setting. The first proof that it gives a rigorous bound was found by Montanari (2005) and subsequently generalized by Macris (2007). An alternative proof technique uses the ‘area theorem’ and the related ‘Maxwell construction’ (Méasson et al., 2005). Tightness of these bounds for a binary erasure channel was proved by Méasson et al. (2008). In this case the asymptotic codeword entropy density and the MAP threshold have been determined rigorously for a large family of ensembles.

Notes

¿

The analysis that we have described in this Chapter is valid in the large-blocklength limit N → ∞. In practical applications, a large block length implies some communication delay. This has motivated a number of studies aimed at estimating and optimizing LDPC codes at moderate block lengths. Some pointers to this large literature can be found in Di et al. (2002), Amraoui et al. (2004, 2007), Wang et al. (2006), K¨ otter and Vontobel (2003), and Stepanov et al. (2005).


16 The assignment problem Consider N ‘agents’ and N ‘jobs’, and suppose you are given an N × N matrix {Eij }, where Eij is the cost for having job j executed by agent i. Finding an assignment of agents to jobs that minimizes the cost is one of the most classical of combinatorial optimization problems. The minimum-cost (also referred to as ‘maximum-weight’) assignment problem is important both because of its many applications and because it can be solved in polynomial time. This has motivated a number of theoretical developments, from the algorithmic as well as the probabilistic viewpoint. Here we shall study the assignment problem as an application of message-passing techniques. It is, in fact, a success story of this approach. Given a generic instance of the assignment problem, the associated factor graph is not locally tree-like. Nevertheless, the min-sum algorithm can be proved to converge to an optimal solution in polynomial time. Belief propagation (the sum–product algorithm) can also be used for computing weighted sums over assignments, although much weaker guarantees exist in this case. A significant amount of work has been devoted to the study of random instances, mostly in the case where the costs Eij are i.i.d. random variables. Typical properties (such as the cost of the optimal assignment) can be computed heuristically within the replica-symmetric cavity method. It turns out that these calculations can also be made fully rigorous. In spite of the success of the replica-symmetric cavity method, one must be warned that apparently harmless modifications of the problem can spoil this success. One example is the generalization of minimal-cost assignment to multi-indices (say, matching agents with jobs and houses): such a ‘multi-assignment’ problem is not described by the replica-symmetric scenario. The more sophisticated ideas of replica symmetry breaking, described in Chapter 19 and after, are required. After defining the problem in Section 16.1, we compute in Section 16.2 the asymptotic optimal cost for random instances using the cavity method. This approach is based on a statistical analysis of the min-sum equations. In Section 16.3, we study the algorithmic aspects of the min-sum iteration and prove its convergence to the optimal assignment. Section 16.4 contains a combinatorial study of the optimal cost that confirms the cavity result and provides sharper estimates. In Section 16.5, we discuss a generalization of the assignment problem to a multi-assignment case.

¿

The assignment problem

i

j

11 00 00(i,j) 11

i

11 00 00 11

11 00 00 11 11 00 00 11 00 11

11 00 00 11 00 11

00 11 00 11 00 11 00 11

Agents

Jobs

j

11 00 00 11 11 00 00 11

Fig. 16.1 Left: graphical representation of a small assignment problem with three agents and three jobs. Each edge carries a cost (not shown); the problem is to find a perfect matching, i.e. a set of three edges which are vertex-disjoint, of minimal cost. Right: the factor graph corresponding to the representation (16.2) of this problem. The hatched squares indicate the function nodes associated with edge weights.

16.1

The assignment problem and random assignment ensembles

An instance of the assignment problem is determined by a cost matrix {Eij }, indexed by i ∈ A (the ‘agents’ set) and j ∈ B (the ‘jobs’ set), with |A| = |B| = N . We shall often identify A and B with the set {1, . . . , N } and use the terms ‘cost’ and ‘energy’ interchangeably. An assignment is a one-to-one mapping of agents to that is, a jobs; N permutation π of {1, . . . , N }. The cost of an assignment π is E(π) = i=1 Eiπ(i) . The optimization problem consists in finding a permutation that minimizes E(π). We shall often use a graphical description of the problem as a weighted complete bipartite graph over sets of vertices A and B. Each of the N 2 edges (i, j) carries a weight Eij . The problem is to find a perfect matching in this graph (a subset M of edges such that every vertex is adjacent to exactly one edge in M ), of minimal weight (see Fig. 16.1). In the following, we shall be interested in two types of questions. The first is to understand whether a minimum-cost assignment for a given instance can be found efficiently through a message-passing strategy. The second will be to analyse the typical properties of ensembles of random instances where the N 2 costs Eij are i.i.d. random variables drawn from a distribution with density ρ(E). One particularly convenient choice is that of exponentially distributed variables with a probability density function ρ(E) = e−E I(E ≥ 0). Although the cavity method allows more general distributions to be tackled, assuming exponential costs greatly simplifies rigorous combinatorial proofs.

Message passing and its probabilistic analysis

16.2 16.2.1

¿

Message passing and its probabilistic analysis Statistical-physics formulation and counting

Following the general statistical-physics approach, it is of interest to relax the optimization problem by introducing a finite inverse temperature β. The corresponding computational problem associates a weight with each possible matching, as follows. Consider the complete bipartite graph over the sets of vertices A (agents) and B (jobs). With each edge (i, j), i ∈ A, j ∈ B, we associate a variable (an ‘occupation number’) nij ∈ {0, 1}, encoding membership of edge (ij) in the matching: nij = 1 means that job j is done by agent i. We impose the condition that the subset of edges (i, j) with nij = 1 is a matching of the complete bipartite graph: nij ≤ 1 ∀i ∈ A , nij ≤ 1 ∀j ∈ B . (16.1) j∈B

i∈A

Let us denote by n = {nij : i ∈ A j ∈ B} the matrix of occupation numbers, and define the probability distribution ⎛ ⎞ 1 ⎝ ⎠ µ(n) = I nij ≤ 1 I nij ≤ 1 e−βnij (Eij −2γ) . (16.2) Z i∈A

j∈B

j∈B

i∈A

(ij)

The support of µ(n) corresponds to matchings, to the & ‘hard constraints’ that % thanks enforce the conditions (16.1). The factor exp 2βγ (ij) nij can be interpreted as a ‘soft constraint’: as γ → ∞, the distribution becomes concentrated on perfect matchings (the factor 2 is introduced here for future convenience). On the other hand, in the limit β → ∞, the distribution (16.2) becomes concentrated on the minimal-cost assignments. The optimization problem is thus recovered in the double limit of γ → ∞ followed by β → ∞. There is a large degree of arbitrariness in the choice of which constraint should be ‘softened’ and how. The present choice makes the whole problem as similar as possible to the general class of graphical models that we are studying in this book. The factor graph obtained from eqn (16.2) has the following structure (see Fig. 16.1). It contains N 2 variable nodes, each associated with an edge (i, j) in the complete bipartite graph over the sets of vertices A and B. It also includes N 2 function nodes of degree one, one for each variable node, and 2N function nodes of degree N , associated with the vertices in A and B. The variable node (i, j), i ∈ A, j ∈ B, is connected to the two function nodes corresponding to i and j, as well as to the function node corresponding to the edge (i, j). The first two function nodes enforce the hard constraints (16.1); the third one corresponds to the weight exp [−β(Eij − 2γ)nij ]. In the case of random instances, we shall be particularly interested in the thermodynamic limit N → ∞. In order for this limit to be non-trivial, the distribution (16.2) must be dominated neither by energy nor by entropy. Consider the case of i.i.d. costs Eij ≥ 0 with an exponential density ρ(E) = e−E . One can argue in this case that low-energy assignments have, with high probability, an energy of order O(1) as N → ∞. The ‘hand-waving’ reason is that for a given agent i ∈ A and any fixed k,

¿


the k lowest costs among those of the jobs that can be assigned to that agent (namely among {Eij : j ∈ B}) are of order O(1/N ). The exercise below sketches a more formal proof. Since the entropy1 is linear in N , we need to rescale the costs for the two contributions to be of the same order. To summarize, throughout our cavity analysis, we shall assume the edge cost to be drawn according to the ‘rescaled pdf’ ρˆ(E) = (1/N ) exp(−E/N ). This choice ensures that the occupied edges in the best assignment have a non-vanishing cost in the large-N limit.

Exercise 16.1 Assume the energies Eij to be i.i.d. exponential variables of mean η. Consider the ‘greedy mapping’ obtained by mapping each vertex iP ∈ A to that j = π1 (i) ∈ B which minimizes Eij , and call the corresponding energy E1 = i Ei,π1 (i) . (a) Show that E E1 = η. (b) Of course, π1 is not necessary injective, and is therefore not a valid matching. Let C be the number of collisions (i.e. the number of vertices j ∈ B such that there exist several i with π1 (i) = j). Show that E C = N (1 − 2/e) + O(1), and that C is tightly concentrated around its expectation. (c) Consider the following ‘fix’. Construct π1 in the greedy fashion described above, and let π2 (i) = π1 (i) whenever i is the unique vertex mapped to π1 (i). For each collision vertex j ∈ B, and each i ∈ A such that π1 (i) = j, let j be the vertex in B such that Eij takes the smallest value among P the vertices still unmatched. What is the expectation of the resulting energy E2 = i Ei,π2 (i) ? What is the number of residual collisions? (d) How can this construction be continued?

16.2.2

The belief propagation equations

The BP equations for this problem are a particular instantiation of the general ones given in eqns (14.14) and (14.15). We shall denote vertices in the sets A and B in the complete bipartite graph by i and j, respectively (see Fig. 16.1). To be definite, let us write explicitly the equations for updating messages flowing from right to left (i.e. from vertices j ∈ B to i ∈ A) in the graph in Fig. 16.1: νij→i (nij ) ∼ = ν j→ij (nij ) e−βnij (Eij −2γ) , 6 5 I nij + nkj ≤ 1 νkj→j (nkj ) . ν j→ij (nij ) ∼ = {nkj }

k∈A\i

(16.3) (16.4)

k∈A\i

The equations for messages moving from A to B, i.e. νij→j and ν i→ij , are obtained by inverting the roles of the two sets. Since the variables nij take values in {0, 1}, messages can be parameterized by a single real number, as usual. In the present case, it is convenient to introduce rescaled log-likelihood ratios as follows: 1 The total number of assignments is N !, which would imply an entropy of order N log N . However, if we limit the choices of π(i) to those j ∈ B such that the cost Eij is comparable to the lowest cost, the entropy becomes O(N ).

Message passing and its probabilistic analysis L

xj→i

1 ≡ γ + log β

ν j→ij (1) ν j→ij (0)

,

R

xi→j

1 ≡ γ + log β

ν i→ij (1) ν i→ij (0)

¿

.

(16.5)

Variable-to-function-node messages do not enter into this definition, but they are easily expressed in terms of the quantities xLi→j , xR i→j using eqn (16.3). The BP equations (16.3) and (16.4) can be written as 3 4 R 1 e−βEkj +βxk→j , xLj→i = − log e−βγ + β k∈A\i (16.6) 3 4 1 R −βγ −βEik +βxL k→i . + e xi→j = − log e β k∈B\j

The factor graph representation in Fig. 16.1, right frame, was necessary in order to write down the original BP equations. However, any reference to the factor graph disappears in the simplified form (16.6). This form can be regarded as a messagepassing procedure operating on the original complete bipartite graph (see Fig. 16.1, left frame).

Exercise 16.2 Marginals. Consider the expectation value of nij with respect to the meaL

R

sure (16.2). Show that its BP estimate is tij /(1 + tij ), where tij ≡ eβ(xj→i +xi→j −Eij ) .

The Bethe free entropy F(ν) can be computed using the general formulae (14.27) L and (14.28). Writing it in terms of the log-likelihood ratio messages {xR i→j , xj→i } is straightforward but tedious. The resulting BP estimate for the free entropy log Z is 6 5 R L log 1 + e−β(Eij −xi→j −xj→i ) F(x) = 2N βγ −

+

i∈A

i∈A,j∈B

⎡ log ⎣e

−βγ

+

j

e

⎤ −β(Eij −xL j→i )

⎦+

j∈B

log e

−βγ

+

e

−β(Eij −xR i→j )

(16.7) .

i

The exercise below provides a few guidelines for this computation.

Exercise 16.3 Consider the Bethe free entropy (14.27) for the model (16.2). (a) Show that it contains three types of function-node terms, one type of variable-node term, and three types of mixed (edge) terms. (b) Show that the function node term associated with the weight e−βnij (Eij −2γ) exactly cancels the mixed term involving this same factor node and the variable node (i, j). (c) Write explicitly each of the remaining terms, express them in terms of the messages L {xR i→j , xj→i }, and derive the result (16.7). [Hint: The calculation can be simplified by recalling that the expression (14.27) does not change value if each message is independently rescaled.]

¿


16.2.3

Zero temperature: The min-sum algorithm

The BP equations (16.6) simplify in the double limit of γ → ∞ followed by β → ∞ which is relevant to the minimum-cost assignment problem. Assuming that the L {xR i→j , xj→i } remain finite in this limit, we get xLj→i = min

k∈A\i

Ekj − xR k→j ,

xR i→j = min

k∈B\j

Eik − xLk→i .

(16.8)

Alternatively, the same equations can be obtained directly as min-sum update rules. This derivation is outlined in the exercise below.

Exercise 16.4 Consider the min-sum equations (14.41) and (14.40), applied to the graphical model (16.2). (a) Show that the message arriving at a variable node (ij) from the adjacent degree-1 b→ij (nij ) = nij (Eij − 2γ). factor node is equal to E (b) Write the update equations for the other messages, and eliminate the variable-tofunction-node messages Eij→i (nij ) and Eij→j (nij ) in favour of the function-to-variable ones. Show that the resulting equations for function-to-variable messages read as follows (see Fig. 16.1): bi→ij (1) = E

X k∈B\j

bi→ij (0) = E

X

bk→ik (0) , E ˆ ˜ bl→il (1) − E bl→il (0) + Eil − 2γ , bk→ik (0) + min E E − l∈B\j

k∈B\j

where we have adopted the notation [x]− = x if x < 0 and [x]− = 0 otherwise. L b b b b (c) Define xR i→j = Ei→ij (0)−Ei→ij (1)+γ and, analogously xi→j = Ej→ij (0)−Ej→ij (1)+γ. R L Write the above min-sum equations in terms of {xi→j , xj→i }. (d) Show that, in the large-γ limit, the update equations for these x-messages coincide with eqn (16.8).

The Bethe estimate for the ground state energy (the cost of the optimal assignment) can be obtained by taking the γ, β → ∞ limit of the free energy −F(x)/β, where F(x) is the Bethe approximation for the log-partition function log Z (see eqn (16.7)). Alternatively, we can use the fact that the min-sum equations estimate the maxmarginals of the graphical model (16.2). More precisely, for each pair (i, j), i ∈ A, j ∈ B, we define

i→ij (nij ) + E

j→ij (nij ) , Eij (nij ) ≡ nij (Eij − 2γ) + E ∗ nij ≡ arg min Eij (n) .

(16.9) (16.10)

n∈{0,1}

The interpretation of these quantities is that e−βEij (n) is the message-passing estimate for the max-marginal of nij with respect to the distribution (16.2). Let us neglect the case of a multiple optimal assignment (in particular, the probability of such an


¿

event vanishes for the random ensemblesthat we shall consider). Under the assumption that message-passing estimates are accurate, nij necessarily takes the value n∗ij in the optimal assignment; see Section 14.3. The resulting estimate of the ground state energy is Egs = ij n∗ij Eij . In the limit γ → ∞, eqn (16.10) reduces to a simple inclusion principle: an edge L ij is present in the optimal assignment (i.e. n∗ij = 1) if and only if Eij ≤ xR i→j + xj→i . We invite the reader to compare this result with that obtained in Exercise 16.2. 16.2.4

The distributional fixed point and ζ(2)

Let us now consider random instances of the assignment problem. For the sake of simplicity, we assume that the edge costs Eij are i.i.d. exponential random variables with mean N . We want to use the general density evolution technique of Section 14.6.2 to analyse the min-sum message-passing equations (16.8). The sceptical reader might notice that the assignment problem does not fit into the general framework for density evolution, since the associated graph (the complete bipartite graph) is not locally tree-like. The use of density evolution can nevertheless be justified, through the following limiting procedure. Given a threshold energy Emax , remove from the factor graph all of the variables (ij), i ∈ A, j ∈ B, such that Eij > Emax , and also remove the edges attached to them. Remembering that the typical edge costs are of order Θ(N ), it is easy to check that the resulting graph is a sparse factor graph and therefore density evolution applies. On the other hand, one can prove that the error made in introducing a finite cut-off Emax is bounded uniformly in N by a quantity that vanishes as Emax → ∞, which justifies the use of density evolution. In the following, we shall take the shortcut of writing density evolution equations for finite N without any cut-off and formally take the limit N → ∞ for them. Since the min-sum equations (16.8) involve minima, it is convenient to introduce (t) the distribution function AN,t (x) = P{xi→j ≥ x}, where t indicates the iteration number, and x(t) refers to right-moving messages (from A to B) when t is even and to left moving messages when t is odd. Then, the density evolution equations read AN,t+1 (x) = [1 − E AN,t (E − x)]N −1 , where E denotes the expectation with respect to E (that is, an exponential random variable of mean N ). In the cavity method, one seeks fixed points of this recursion. These are the distributions that solve AN (x) = [1 − E AN (E − x)]N −1 .

(16.11)

We now want to take the N → ∞ limit. Assuming the fixed point AN (x) has a (weak) limit A(x), we have E AN (E − x) =

1 N

∞

AN (y) e−(x+y)/N dy =

−x

1 N

∞

A(y) dy + o(1/N ) .

(16.12)

−x

It follows from eqn (16.11) that the limit message distribution must satisfy the equation ∞ A(x) = exp − A(y) dy . −x

(16.13)

¿


This equation has the unique solution A(x) = 1/(1 + ex ), corresponding to the density a(x) = A (x) = 1/[4 cosh2 (x)]. It can be shown that density evolution does indeed converge to this fixed point. Within the hypothesis of replica symmetry (seeSection 14.6.3), we can use the above fixed-point distribution to compute the asymptotic ground state energy (the minimum cost of the assignment). The most direct method is to use the inclusion principle: an L edge (ij) is present in the optimal assignment if and only if Eij ≤ xR i→j + xj→i . Therefore the conditional probability for (ij) to be in the optimal assignment, given its energy Eij = E, is given by 1 + (E − 1)eE q(E) = I(x1 + x2 ≥ E) a(x1 )a(x2 ) dx1 dx2 = (16.14) (eE − 1)2 The expected cost E∗ of the optimal assignment is equal to the number of edges, N 2 , times the expectation of the edge cost, times the probability that the edge is in the optimal assignment. Asymptotically we have E∗ = N 2 E{Eq(E)}: E∗ = N

2

∞

E e−E/N q(E) dE/N + o(N )

0 ∞

=N

E 0

1 + (E − 1)eE dE + o(N ) = N ζ(2) + o(N ) , (eE − 1)2

where ζ(2) ≡

∞ π2 1 ≈ 1.64493406684823 . = n2 6 n=1

(16.15)

Recall that this result holds when the edge weights are exponential random variables of mean N . If we reconsider the case of exponential random variables of mean 1, we get E∗ = ζ(2) + o(1). The reader can verify that the above derivation does not depend on the full distribution of the edge costs, but only on its behaviour near E = 0. More precisely, for any distribution of edge costs with a density ρ(E) such that ρ(0) = 1, the cost of the optimal assignment converges to ζ(2). Exercise 16.5 Suppose that the pdf of the costs ρ(E) has support R+ , and that ρ(E) E r , for some r > 0, when E ↓ 0. (a) Show that, in order to have an optimal weight of order N , the edge costs must be eij have density ρ (i.e. the typical eij , where the E rescaled by letting Eij = N r/r+1 E r/r+1 ). costs must be of order N (b) Show that, within the replica-symmetric cavity method, the asymptotic (N → ∞) message distribution satisfies the following distributional equation: j Z A(x) = exp −

∞

−x

ff (x + y)r A(y) dy

.

(16.16)


¿

(c) Assume that the solution A(x) to eqn (16.16) is unique and that replica symmetry holds. Show that the expected ground stateR energy (in the problem with rescaled edge costs) is E∗ = N r + o(N ), where r ≡ − A(x) log(A(x)) dx. As a consequence, the optimal cost in the initial problem is N r/(r+1) r (1 + o(1)). (d) Equation (16.16) can be solved numerically by use of the population dynamics algorithm of Section 14.6.3. Write the corresponding program and show that the costs of the optimal matching for r = 1, 2 are 1 ≈ 0.8086 and 2 ≈ 0.6382.

16.2.5

Non-zero temperature and stability analysis

The reader may wonder whether the heuristic discussion in the previous subsections can be justified. While a rigorous justification would lead us too far, we want to discuss, still at a heuristic level, the consistency of the approach. In particular, we want to argue that BP provides good approximations to the marginals of the distribution (16.2), and that density evolution can be used to analyse its behaviour sur random instances. Intuitively, two conditions should be satisfied for the approach to be valid: (i) the underlying factor graph should be locally tree-like; and (ii) the correlation between two variables nij , nkl should decay rapidly with the distance between edges (ij) and (kl) on such a graph. At first sight it appears that condition (i) is far from holding, since our factor graph is constructed from a complete bipartite graph. As mentioned in the previous subsection, the locally tree-like structure emerges if one notices that only edges with costs of order 1 are relevant (as above, we are assuming that the edge costs have been rescaled so that they are drawn with a probability density function ρ(E) ˆ = N −1 exp(−E/N )). In order to investigate this point further, we modify the model (16.2) by pruning from the original graph all edges with a cost larger than 2γ. In the large-β limit, this modification will become irrelevant since the Boltzmann weight (16.2) ensures that these ‘costly’ edges of the original problem are not occupied. In the modified problem, the degree of any vertex in the graph converges (as N → ∞) to a Poisson random variable with mean 2γ. The costs of ‘surviving’ edges converge to i.i.d. uniform random variables in the interval [0, 2γ]. For fixed β and γ, the asymptotic message distribution can be computed by the RS cavity method. The corresponding fixed-point equation reads k 1 d (16.17) e−β(Er −xr ) , x = − log e−βγ + β r=1 where k is a Poisson random variable with mean 2γ, the Er are i.i.d. and uniformly distributed on [0, 2γ], and the xr are i.i.d. with the same distribution as x. The fixedpoint distribution can be estimated easily using the population dynamics algorithm of Section 14.6.3. Some results are shown in Fig. 16.2. For large β, γ, the density estimated by this algorithm converges rapidly to the analytical result for β = γ = ∞, namely a(x) = 1/[4 cosh2 (x/2)]. The distribution of messages can be used to compute the expected Bethe free entropy. Assuming that the messages entering into eqn (16.7) are independent, we get E F(x) = N f RS (β, γ) + o(N ), where

¿


0.35

2

f RS (β)/β

0.30 0.25

a(x)

0.20 0.15

1 0 -1

0.10 -2

0.05 0.00 -10

-3

-5

0

5

10

0

1

2

x

3

4

5

T = 1/β

Fig. 16.2 Left: estimate of the probability distribution of the messages xi→j obtained by population dynamics. Here we consider the modified ensemble in which costly edges (with Eij > 2γ) have been removed. The three curves, from top to bottom, correspond to (β = 1, γ = 5), (β = 1, γ = 60), and (β = 10, γ = 60). The last curve is indistinguishable from the analytical result for (β = ∞, γ = ∞) namely a(x) = 1/[4 cosh2 (x/2)], also shown. The curves for larger γ are indistinguishable from the curve for γ = 60 on this scale. The algorithm used a population of size 105 , and the whole population wass updated 100 times. Right: free energy versus temperature T = 1/β, computed using eqn (16.18). The distribution of messages was obtained as above with γ = 40.

j1 k2 a1

x

j a2

1

2

x2

11 00 00 11

j

kr ar

11 00 00 11

r

xr

a r+1

11 00 00 11

Fig. 16.3 Part of the factor graph used to compute the correlation between xr and x1 .

⎡ f RS (β, γ) = 2βγ + 2E log ⎣e−βγ +

k

⎤

6 5 e−β(Ej −xj ) ⎦ − 2γE log 1 + e−β(E1 −x1 −x2 ) .

j=1

(16.18) Having a locally tree-like structure is only a necessary condition for BP to provide good approximations of the marginals. An additional condition is that the correlations of distinct variables nij , nkl decay rapidly enough with the distance between the nodes (ij), (kl) in the factor graph. Let us discuss here one particular measure of these correlations, namely the spin glass susceptibility defined in Section 12.3.2. In the present case it can be written as


χSG ≡

1 2 (ne nf − ne nf ) , N

¿

(16.19)

e,f

where the sum runs over all pairs of variable nodes e = (i, j) and f = (k, l) in the factor graph (or, equivalently, over all pairs of edges in the original bipartite graph with vertex sets A, B). If correlations decay fast enough for the system to be stable with respect to small perturbations, χSG should remain bounded as N → ∞. The intuitive explanation goes as follows. From the fluctuation–dissipation relation of Section 2.3, ne nf − ne nf is proportional to the change in nf when the cost of edge e is perturbed. The sign of such a change f , and therefore the resulting change in the will depend upon expected matching size f nf (namely f (ne nf −ne nf )) can be either positive or negative. Assuming that this sum obeys a central limit theorem, its typical size is given by the square root of f (ne nf − ne nf )2 . Averaging over the perturbed edge, we see that χSG measures the decay of correlations. We shall thus estimate χSG using the same RS cavity assumption that we used in our computation of the expectations ne . If the resulting χSG is infinite, the assumption will be falsified. In the opposite case, although nothing definite can be said, the assumption will be said to be ‘consistent’, and the RS solution will be said to be ‘locally stable’ (since it is stable to small perturbations). In order for the susceptibility to be finite, only pairs of variable nodes (e, f ) whose distance r in the factor graph is bounded should give a significant contribution to the susceptibility. We can then compute 1 (r) 2 (ne nf − ne nf ) (16.20) χSG ≡ N e,f : d(e,f )=r

for fixed r in the N → ∞ limit, and then sum the result over r. For any given r and large N , there is, with high probability, a unique path of length r joining e to f , all the others being of length Θ(log N ). On this path, we denote the variable nodes by (j1 , j2 , . . . , jr ) (with e = j1 , f = jr ), and denote the function nodes by (a2 , . . . , ar ) (see Fig. 16.3). We consider a fixed point of the BP algorithm and denote by xn the (log-likelihood) message passed from an to jn . The BP fixed-point equations (16.6) allow us to compute xr as a function of the message x1 arriving at j1 , and of all the messages coming in on the path {a2 , . . . , ar } from edges outside this path, which we denote by {yn,p }: k2 3 4 1 log e−βγ + e−β(E1 −x1 ) + e−β(E2,p −y2,p ) , β p=1 ... ... kr 3 4 1 xr = − log e−βγ + e−β(Er −xr ) + e−β(Er,p −yr,p ) . β p=1

x2 = −

(16.21)

In a random instance, the kn are i.i.d. Poisson random variables with mean 2γ, the variables En and En,p are i.i.d. random variables uniform on [0, 2γ], and the yn,p are

¿


i.i.d. random variables with the same distribution as the solution of eqn (16.17). We shall denote by Eout the expectation with respect to all of these variables outside the path. If we keep these variables fixed, a small change δx1 in the message x1 leads to a change δxr = (∂xr /∂x1 )δx1 = (∂x2 /∂x1 )(∂x3 /∂x2 ) . . . (∂xr /∂xr−1 )δx1 in xr . We leave it as an exercise to the reader to show that the correlation function is given by ne nf − ne nf = C

r ∂xr ∂xn =C , ∂x1 ∂xn−1 n=2

(16.22)

where the proportionality constant C is r-independent. Recalling that the expected number of variable nodes f such that d(e, f ) = r grows as (2γ)r , and using eqn (16.20), (r) we have E χSG = C eλr r , where # 2 $ r ∂xn 1 . λr (β, γ) = log(2γ) + log Eout r ∂xn−1 n=2

(16.23)

Therefore, a sufficient condition for the expectation of χSG to be finite is that λr (β, γ) (r) is negative and bounded away from 0 for large enough r (when this happens, E χSG decays exponentially with r). The exponent λr (β, γ) can be computed numerically through population dynamics: the population allows us to sample i.i.d. messages yn,p from the fixed-point message density, and the costs En and En,p are sampled uniformly in [0, 2γ]. The expectation (16.23) can be estimated through a numerical average over large enough populations. Note that the quantity that we are taking the expectation of depends exponentially on r. As a consequence, its expectation becomes more difficult to compute as r grows. In Fig. 16.4 we present some estimates of λr obtained by this approach. Since λr depends very weakly on r, we expect that λ∞ can be safely estimated from these data. The data are compatible with the following scenario: λ∞ (β, γ) is negative at all finite inverse temperatures β and vanishes as 1/β as β → ∞. This indicates that χSG is finite, so that the assumption of replica symmetry is consistent.

16.3

A polynomial message-passing algorithm

Remarkably, the min-sum message-passing algorithm introduced in Section 16.2.3 can be proved to return the minimum-cost assignment on any instance for which the minimum is unique. Let us state again the min-sum update equations of eqn (16.8), writing the iteration number explicitly: xLj→i (t + 1) = min Ekj − xR k→j (t) , k∈A\i

L xR i→j (t) = min Eik − xk→i (t) .

(16.24)

k∈B\j

Here, as before, A and B (with |A| = |B| = N ) are the two sets of vertices to be matched, and we continue to denote generic vertices in A and B by i and j, respectively. The algorithm runs as follows:


¿

0.0

λr

-1.0

-2.0

-3.0

-4.0 0

1

2

3

4

5

r

Fig. 16.4 The stability parameter λr defined in eqn (16.23), plotted versus r, for inverse temperatures β = 10, 5, 2, 1 (from top to bottom). The lines are guides to the eye. A negative asymptotic value of λr at large r shows that the spin glass susceptibility is finite. The data were obtained from a population dynamics simulation with a population of 106 , for γ = 20.

min-sum assignment (cost matrix E, iterations t∗ ) 1: Set xLj→i (0) = xR i→j (0) = 0 for any i ∈ A, j ∈ B 2: For all t ∈ {0, 1, . . . , t∗ }: 3: Compute the messages at time t + 1 using eqn (16.24) 4: Set π(i) = arg minj∈B Eij − xLj→i (t∗ ) for each i ∈ A; 5: Output the permutation π; This algorithm finds the optimal assignment if it is unique and if the number of iterations is large enough, as stated in the theorem below. Theorem 16.1 Let W ≡ maxij |Eij | and let be the gap between the cost E ∗ of the optimal assignment π ∗ and the next best cost: ≡ minπ( =π∗ ) (E(π) − E∗ ), where N E(π) ≡ i=1 Eiπ(i) . Then, for any t∗ ≥ 2N W/, the min-sum algorithm above returns the optimal assignment π ∗ . The proof is given in Section 16.3.2, and is based on the notion of a computation tree, explained in the present context in Section 16.3.1. For practical application of the algorithm to cases where one does not know the gap in advance, it is important to have a stopping criterion for the algorithm. This can be obtained by noticing that, after convergence, the messages become ‘periodicup-to-a-drift’ functions of t. More precisely, there exists a period τ and a drift C > 0 such that for any t > 2N W/, and any i ∈ A, xLj→i (t + τ ) = xLj→i (t) + C if j = arg mink∈B (Eik − xLk→i (t)), and xLj→i (t + τ ) = xLj→i (t) − C otherwise. If this happens, we write xL (t + τ ) = xL (t) + C. It turns out that (i) if for some time t0 , period τ , and constant C > 0, one has xL (t0 + τ ) = xL (t0 ) + C, then xL (t + τ ) = xL (t) + C for any t ≥ t0 ; and (ii) under the same conditions, the permutation returned by the min-sum algorithm is independent

¿


of t∗ for any t∗ ≥ t0 . We leave the proof of these statements as a (research-level) exercise for the reader. It can be seen immediately that they imply a clear stopping criterion: After any number of iterations t, check whether there exist t0 < t and C > 0, such that xL (t) = xL (t0 ) + C. If this is the case, halt the message-passing updates and return the resulting permutation as in step 4 of the above pseudocode. 16.3.1

The computation tree

As we saw in Fig. 16.1, an instance of the assignment problem corresponds to a weighted complete bipartite graph GN over sets of vertices A and B, with |A| = |B| = N . The analysis of the min-sum algorithm described above uses the notion of a computation tree in a crucial way. Given a vertex i0 ∈ A (the case i0 ∈ B is completely symmetric), the corresponding computation tree of depth t, denoted by Tti0 , is a weighted rooted tree of depth t and degree N , which is constructed recursively as follows. First, introduce the root î0 that is in correspondence with i0 ∈ A. For any j ∈ B, add a corresponding vertex ˆj to Tti0 and connect it to î0 . The weight of the resulting edge is taken to be Eî0 ,ˆj ≡ Ei0 ,j . At every subsequent generation, if î ∈ Tti0 corresponds to i ∈ A, and its direct ancestor is the ˆj that corresponds to j ∈ B, add N − 1 direct descendants of î to Tti0 . Each of these descendants kˆ corresponds to a distinct vertex k ∈ B \ j, and the associated weight is Ekˆˆj = Ekj . An alternative, more compact description of the computation tree Tti0 is to say that it is the tree of non-reversing paths on GN , rooted at i0 . Imagine iterating the min-sum equations (16.24) on the computation tree Tti0 (starting from the initial condition xî→ˆj (0) = 0). Since Tti0 has the same local structure as GN , for any s ≤ t the messages coming into the root î0 coincide with the messages along the corresponding edges in the original graph GN : xˆj→î0 (s) = xj→i0 (s). As the min-sum algorithm correctly finds the ground state on trees (see Theorem 14.4), the following property holds. ˆ Lemma 16.2 We define, for any i ∈ A, π t (i) = argminj∈B Ei,j − xL j→i (t) . Let i t ˆ ˆ denote the root of the computation tree Ti , and let j denote the direct descendant of i that corresponds to π t (i). We define an internal matching of a tree to be a subset of the edges such that each non-leaf vertex has one adjacent edge in the subset. Then the edge (î, ˆj) belongs to the internal matching with lowest cost in Tti (assuming that this is unique). Although it follows from general principles, it is instructive to rederive this result explicitly.

Exercise 16.6 Let r be an internal (non-leaf) vertex in the computation tree Tti , distinct from the root. Denote the set of its direct descendants by Sr (hence |Sr | = N − 1), and denote the tree induced by r and all its descendants by Tr . We define a ‘cavity internal matching’ in Tr as a subset of the edges of Tr such that each vertex in Tr distinct from r has degree 1. Denote the cost of the optimal cavity internal matching when vertex r is not matched by Ar , and denote its cost when vertex r is matched by Br .


2

Show that Ar =

X q∈Sr

Bq , Br = min 4Bq + Erq + q∈Sr

X

¿

3 Aq 5 .

(16.25)

q ∈Sr \{q}

Show that xr = Br − Ar satisfies the same equations as in eqn (16.24), and prove Lemma 16.2.

16.3.2

Proof of convergence of the min-sum algorithm

This subsection is a technical one, which gives a detailed proof of Theorem 16.1. It is convenient here to represent assignments as matchings, i.e. subsets of the edges such that each vertex is incident on exactly one edge in the subset. In particular, we denote the optimal matching on G by M∗ . If π ∗ is the optimal assignment, then M∗ ≡ {(i, π∗ (i)) : i ∈ A}. We denote by π the mapping returned by the min-sum algorithm. It is not necessarily injective, and therefore the subset of edges M = {(i, π(i)) : i ∈ A} is not necessarily a matching. The proof is by contradiction. Assume that π = π ∗ . Then there exists at least one vertex in A, which we denote by i0 , such that π(i0 ) = π ∗ (i0 ). Consider the depth-t

the optimal internal computation tree of i0 , Tti0 , denote its root by î0 , and denote by M

∗ the internal matching on Tt which is matching in this graph. Finally, denote by M i0 obtained by ‘lifting’ the optimal matching, M∗ . Let j = π(i0 ) ∈ B, and let ˆj ∈ Tti0 be

On the the neighbour of î0 whose projection on G is j. By Lemma 16.2, (î0 , ˆj) ∈ M. ∗

other hand, since π(i0 ) = π∗ (i0 ), (î0 , ˆj) ∈ M . The idea is to construct a new internal

on Tt , such that (i) (î0 , ˆj) ∈ M

, and (ii) the cost of M

is strictly smaller matching M i0

than the cost of M, thus leading to a contradiction.

in such a

is constructed by modifying M Intuitively, the improved matching M ∗

way as to ‘get closer’ to M . In order to formalize this idea, consider the symmetric

= M

'M

∗ , i.e. the set of edges which are either in M

or

and M

∗, P difference of M ∗

ˆ ˆ in M but not in both. The edge (i0 , j) belongs to P . We can therefore consider the

that contains (î0 , ˆj), which we shall call P.

A moment of connected component of P t

thought reveals that P is a path on Ti0 with end-points on its leaves (see Fig. 16.5).

and in M

∗ . We can then define Furthermore, its 2t edges alternate between edges in M

'P

(so that M

is obtained from M

by deleting the edges in P

∩M

and adding

=M M ∗

is smaller those in P ∩ M ). We shall now show that, if t is large enough, the cost of M

than the cost of M, in contradiction with the hypothesis.

onto the original complete bipartite graph G; call it Consider the projection of P

P ≡ ϕ(P) (see Fig. 16.5). This is a non-reversing path of length 2t on G. As such, it can be decomposed into m simple cycles2 {C1 , . . . , Cm } (possibly with repetitions) and at most one even-length path Q, whose lengths add up to 2N . Furthermore, the length of Q is at most 2N − 2, and the length of each of the cycles at most 2N . As a consequence, m > t/N . 2A

simple cycle is a cycle that does not visit the same vertex twice.

¿

l l


r

1

r

2

1

r2

r3

r2

l l l

r

3

r

2

r

3

r1

r3 r1 r2

r1

r2

l3

l2

l3

l2

l3

3

r

3

r3

2

l2

l

r3 r1

r1

r2

r3

1

P

2

3

l1

Fig. 16.5 Top left: an instance G of the assignment problem with 2N = 6 vertices (the costs are not shown). The optimal assignment π ∗ is composed of the thick edges. Right: the computation tree T2l1 . The matching π ∗ is ‘lifted’ to an internal matching in T2l1 composed of the thick edges. Notice that one edge in the original graph has many images in the computation tree. The dashed edges are those of the optimal internal matching in T2l1 , and the alternating path P is enclosed by the dotted line. Bottom left: the projection of P on the original graph; here, it consists of a single cycle.

Consider now a particular cycle, say Cs . Its edges alternate between edges belonging to the optimal matching M∗ and edges not belonging to it. As we have assumed that the second-best matching in G has a cost of at least above the best one, the total cost of the edges in Cs \ M∗ is at least above the total cost of the edges in Cs ∩ M∗ . As for the path Q, it again alternates between edges belonging to M∗ and edges outside of M∗ . We can order the edges in Q in such a way that the first edge is in M∗ and the last is not. By changing the last step, we can transform the path into an alternating cycle, to which the same analysis as above applies. This swapping changes the cost of edges not in Q by at most 2W . Therefore the cost of the edges in Q \ M∗ is at least the cost of the edges in Q ∩ M∗ plus − 2|W |.

denote the cost of the matching M

on Tt . By summing the cost difLet ET (M) i0

≥ ferences between the m cycles {C1 , . . . , Cm } and the path Q, we find that ET (M)

ET (M )+(m+1)−2W . Therefore, for t > 2N W/, ET (M) > ET (M ), in contradiction with our hypothesis. 16.3.3

A few remarks

The alert reader might be puzzled by the following observation. Consider a random instance of the assignment problem with i.i.d. edge weights, for example exponentially distributed. In Section 16.2.4, we analysed the min-sum algorithm using density evolution and showed that the only fixed point is given by the x-message density

Combinatorial results

¿

a(x) = 14 cosh2 (x/2). A little more work shows that, when initiated with x = 0 messages, the density evolution does indeed converge to such a fixed point. On the other hand, for such a random instance the maximum weight W and the gap between the two best assignments are almost surely finite and non-vanishing, and so the hypotheses of Theorem 16.1 apply. The proof in the previous subsection implies that the min-sum messages diverge: the messages xi→π∗ (i) diverge to +∞, while other ones diverge to −∞ (indeed, min-sum messages are just the difference between the cost of the optimal matching on the computation tree and the cost of the optimal matching that does not include the root). How can these two behaviours be compatible? The conundrum is that density evolution correctly predicts the distribution of messages as long as the number of iterations is kept bounded as N → ∞. On the other hand, the typical number of iterations required to see the divergence of messages discussed above is N W/. If the edge weights are exponentially distributed random variables of mean N , the typical gap is = Θ(1), while W = Θ(N log N ). Therefore the divergence sets in after t∗ = Θ(N 2 log N ) iterations. The two analyses therefore describe completely distinct regimes.

16.4


It turns out that a direct combinatorial analysis allows one to prove several nonasymptotic results for ensembles of random assignment problems. Although the techniques are quite specific, the final results are so elegant that they deserve to be presented. As a by-product, they also provide rigorous proofs of some of our previous results, such as the asymptotic optimal cost ζ(2) found in eqn (16.15). We shall consider here the case of edge weights given by i.i.d. exponential random variables with rate 1. Let us recall that an exponential random variable X with rate α has a density ρ(x) = α e−αx for x ≥ 0, and therefore its expectation is E[X] = 1/α. Equivalently, the distribution of X is given by P{X ≥ x} = e−αx for x ≥ 0. Exponential random variables have several special properties that make them particularly convenient in the present context. The most important one is that the minimum of two independent exponential random variables is again exponential. We shall use the following refined version of this statement. Lemma 16.3 Let X1 , . . . , Xn be n independent exponential random variables with respective rates α1 , . . . , αn . Then: n 1. The random variable X = min{X1 , . . . , Xn } is exponential with rate α ≡ i=1 αi . 2. The random variable I = arg mini Xi is independent of X, and has a distribution P{I = i} = αi /α. Proof First, note that the minimum of {X1 , . . . , Xn } is almost surely achieved by only one of the variables, and therefore the index I in item 2 of the lemma is well defined. An explicit computation yields, for any x ≥ 0 and i ∈ {1, . . . , n},

¿


P{I = i, X ≥ x} =

∞

αi e−αi z

x

P{Xj ≥ z} dz

j( =i)

∞

=

αi e−αz dz =

x

αi −αx e . α

(16.26)

By summing over i = 1, . . . , n, we get P{X ≥ x} = e−αx , which proves item 1. By taking x = 0 in the above expression, we get P{I = i} = αi /α. Using these two results, eqn (16.26) can be rewritten as P{I = i, X ≥ x} = P{I = i} P{X ≥ x}, which implies that X and I are independent. 16.4.1

The Coppersmith–Sorkin and Parisi formulae

The combinatorial approach is based on a recursion on the size of the problem. It is therefore natural to generalize the assignment problem by allowing for partial matching between two sets of unequal size as follows. Given a set of agents A and a set of jobs B (with |A| = M and |B| = N ), consider the complete bipartite graph G over the vertex sets A and B. A k-assignment between A and B is defined as a subset of k edges of G that has size k and is such that each vertex is adjacent to at most one edge. Given edge costs {Eij : i ∈ A, j ∈ B}, the optimal k-assignment is the assignment that minimizes the sum of the costs over the edges in the matching. The assignment problem considered so far is recovered by setting k = M = N . Below we shall assume, without loss of generality, that k ≤ M ≤ N . Theorem 16.4. (Coppersmith–Sorkin formula) Assume the edge costs {Eij : i ∈ A, j ∈ B} to be i.i.d. exponential random variables of rate 1, with |A| = M and |B| = N , and let Ck,M,N denote the expected cost of the optimal k-assignment. Then, Ck,M,N =

k−1

I(i + j < k)

i,j=0

1 . (M − i)(N − j)

(16.27)

This result, which we shall prove in the next two subsections, yields, as a special case, the expected cost CN of the complete matching over a bipartite graph with 2N vertices. Corollary 16.5. (Parisi formula) Let CN ≡ CN,N,N be the expected cost of the optimal complete matching between sets of vertices A, B with |A| = |B| = N , assuming that the edge weights are i.i.d. and exponentially distributed with a rate 1. Then

CN =

N 1 . 2 i i=1

(16.28)

In particular, the expected cost of the optimal assignment when N → ∞ is ζ(2).


¿

N −1 Proof By Theorem 16.4, we have CN = i,j=0 I(i + j < N ) (N − i)−1 (N − j)−1 . By simplifying equal terms, the difference CN +1 − CN can be written as N j=0

1 1 1 + − . (N + 1)(N + 1 − j) i=1 (N + 1 − i)(N + 1) r=1 (N + 1 − r)r N

N

(16.29)

By applying the identity 1/((N + 1 − r)r) = 1/((N + 1)r) + 1/((N + 1 − r)(N + 1)), this implies CN +1 − CN = 1/(N + 1)2 , which establishes Parisi’s formula. The rest of this section will describe the proof of Theorem 16.4. 16.4.2

From k-assignment to k + 1-assignment

The proof relies on two lemmas which relate the properties of the optimal k-assignment to those of the optimal (k + 1)-assignment. Let us denote the optimal k-assignment (viewed as a subset of the edges of the complete bipartite graph) by Mk . The first lemma applies to any realization of the edge costs, provided that no two subsets of the edges have equal cost (this happens with probability 1 within our random cost model). Lemma 16.6. (nesting lemma) Let k < M ≤ N , and assume that no linear combination of the edge costs {Eij : i ∈ A, j ∈ B} with coefficients in {+1, 0, −1} vanishes. Then every vertex that belongs to Mk also belongs to Mk+1 . The matching Mk consists of k edges which are incident on the vertices i1 , . . . , ik in set A and on j1 , . . . , jk in set B. We denote by E (k) the k × k matrix which is the restriction of E to the rows i1 , . . . , ik and the columns j1 , . . . , jk . The nesting lemma ensures that E (k+1) is obtained from E (k) by adding one row (ik+1 ) and one column (jk+1 ). Therefore we have a sequence of nested matrices E (1) ⊂ E (2) · · · ⊂ E (M ) = E containing the sequence of optimal assignments M1 , M2 , . . . , MM . Proof Colour all the edges in Mk red and all the edges in Mk+1 blue, and denote by Gk+ the bipartite graph induced by edges in Mk ∪ Mk+1 . Clearly, the maximum degree of Gk+ is at most 2, and therefore its connected components are either cycles or paths. We first show that no component of Gk+ can be a cycle. Assume, in contradiction, that the edges {u1 , v1 , u2 , v2 , . . . , up , vp } ⊆ Gk+ form such a cycle, with {u1 , . . . , up } ⊆ Mk and {v1 , . . . , vp } ⊆ Mk+1 . Since Mk is the optimal k-assignment, Eu1 + · · · + Eup ≤ Ev1 + · · · + Evp (in the opposite case, we could decrease its cost by replacing the edges {u1 , . . . , up } with {v1 , . . . , vp }, without changing its size). On the other hand, since Mk+1 is the optimal (k + 1)-assignment, the same argument implies Eu1 + · · · + Eup ≥ Ev1 + · · · + Evp . These two inequalities imply Eu1 + · · · + Eup = Ev1 + · · · + Evp , which is impossible by the non-degeneracy hypothesis. So far, we have proved that Gk+ consists of a collection of disjoint simple paths, formed by alternating blue and red edges. Along such paths, all vertices have degree 2 except for the two end-points which have degree 1. Since each path alternates between red and blue edges, the difference between the numbers of red and blue edges can be either 0 or ±1. We shall now show that there can exist only one such path, with one more blue edge than red edges.

¿


We first notice that Gk+ cannot contain even paths, with as many red as blue edges. This can be shown using the same argument that we used above in the case of cycles: either the cost of the blue edges along the path is lower than the cost of the red ones, which would imply that Mk is not optimal, or, vice versa, the cost of the red edges is lower, which would imply that Mk+1 is not optimal. We now exclude the existence of a path P of odd length with one more red edge than blue edges. Since the total number of blue edges is larger than the total number of red edges, there should exist at least one path P with odd length, with one more blue edge than red edges. We can then consider the double path P ∪ P , which contains as many red as blue edges, and apply to it the same argument as for cycles and even paths. We thus conclude that the symmetric difference of Mk and Mk+1 is a path of odd length, with one end-point i ∈ A and one end-point j ∈ B. These are the only vertices that are in Mk+1 but not in Mk . Conversely, there is no vertex that is in Mk but not in Mk+1 . Lemma 16.7 Let {ui : i ∈ A} and {vj : j ∈ B} be two collections of positive real numbers, and assume that the costs of the edges {Eij : i ∈ A, j ∈ B} are independent exponential random variables, the rate of Eij being ui vj . Denote by Ak = {i1 , . . . , ik } ⊆ A and Bk = {j1 , . . . , jk } ⊆ B the sets of vertices appearing in the optimal k-assignment Mk . Let Ik+1 = Ak+1 \ Ak and Jk+1 = Bk+1 \ Bk be the extra vertices which are added in Mk+1 . The conditional distribution of Ik+1 and Jk+1 is then P {Ik+1 = i, Jk+1 = j|Ak , Bk } = Qi,j , where Qij = % i ∈A\A

u i vj & %

ui k

&.

j ∈B\B

(16.30)

vj k

Proof Because of the nesting lemma, one of the following must be true: either the matching Mk+1 contains edges (Ik+1 , jb ) and (ia , Jk+1 ) for some ia ∈ Ak and jb ∈ Bk , or it contains the edge (Ik+1 , Jk+1 ). Let us fix ia and jb and condition on the first event E1 (ia , jb ) ≡ {Ak , Bk , (Ik+1 , jb ), (ia , Jk+1 ) ∈ Mk+1 } . Then, necessarily, EIk+1 ,jb = min{Eijb : i ∈ A \ Ak } (because, otherwise, we could decrease the cost of Mk+1 by making a different choice for Ik+1 ). Analogously, Eia ,Jk+1 = min{EiA j : j ∈ B \ Bk }. Since the two minima are taken over independent random variables, Ik+1 and Jk+1 are independent as well. Further, by Lemma 16.3, P {Ik+1 = i, Jk+1 = j | E1 (ia , jb )} =

ui vjb

i ∈A\Ak

ui vjb

uia vj = Qij . j ∈B\Bk uia vj

If we instead condition on the second event E2 ≡ {Ak , Bk , (Ik+1 , Jk+1 ) ∈ Mk+1 } ,


¿

then EIk+1 ,Jk+1 = min{Eij : i ∈ A \ Ak j ∈ B \ Bk } (because, otherwise, we could decrease the cost of Mk+1 ). By applying Lemma 16.3 again we get u i vj

P {Ik+1 = i, Jk+1 = j | E2 } =

i ∈A\Ak ,j ∈B\Bk

ui vj

= Qij .

Since the resulting probability is Qij irrespective of the conditioning, it remains the same when we condition on the union of the events {∪a,b E1 (ia , jb )} ∪ E2 = {Ak , Bk }. 16.4.3

Proof of Theorem 16.4

In order to prove the Coppersmith–Sorkin (C–S) formula (16.27), we shall consider the difference Dk,M,N ≡ Ck,M,N − Ck−1,M,N −1 , and establish in this subsection that 1 1 1 1 + + ··· + . (16.31) Dk,M,N = N M M −1 M −k+1 This immediately leads to the C–S formula, by recursion using as a base step the identity C1,M,N −k+1 = 1/(M (N − k + 1)) (which follows from the fact that this is the minimum of M (N − k + 1) i.i.d. exponential random variables with rate 1). Consider a random instance of the problem over vertex sets A and B with |A| = M and |B| = N , whose edge costs {Eij : i ∈ A, j ∈ B} are i.i.d. exponential random variables with rate 1. Let X be the cost of its optimal k-assignment. Let Y be the cost of the optimal (k − 1)-assignment for the new problem that is obtained by removing one fixed vertex, say the last one, from B. Our aim is to estimate the expectation value Dk,M,N = E(X − Y ), We shall use an intermediate problem with a cost matrix F of size (M + 1) × N , constructed as follows. The first M rows of F are identical to those of E. The elements of the matrix in its last line are N i.i.d. exponential random variables of rate λ, independent of E. We denote the cost of the edge (M + 1, N ) by W , and call the event ‘the optimal k-assignment in F uses the edge (M + 1, N )’ E. We claim that, as λ → 0, P(E) = λE[X − Y ] + O(λ2 ). Note first that, if E is true, then W + Y < X, and therefore ( ' P(E) ≤ P{W + Y < X} = E 1 − e−λ(X−Y ) = λ E[X − Y ] + O(λ2 ) . (16.32) Conversely, if W < X − Y , and all the edges from the vertex M + 1 in A to B \ {N } have a cost of at least X, then the optimal k-assignment in F uses the edge (M +1, N ). Therefore, using the independence of the edge costs, P(E) ≥ P{W < X − Y ; EM +1,j ≥ X for j ≤ N − 1} N −1 3 4 = E P{W < X − Y | X, Y } P{EM +1,j ≥ X| X} j=1

4 = E P{W < X − Y | X, Y } e−(N −1)λX 4 3 = E 1 − e−λ(X−Y ) e−(N −1)λX = λ E[X − Y ] + O(λ2 ) . 3

(16.33)

¿


We now turn to the evaluation of P(E), and show that P(E) =

k−1 M −r 6 15 1− . N M −r+λ r=0

(16.34)

Let us denote the M +1-th vertex in A by α. By Lemma 16.7, conditional on α ∈ / Mk−1 , the probability that α ∈ Mk is λ/(M − (k − 1) + λ). By recursion, this shows that the k−1 probability that α ∈ / Mk−1 is r=0 ((M − r)/(M − r + λ)). Since all of the N edges incident on α are statistically equivalent, we get eqn (16.34). k−1 Expanding eqn (16.34) as λ → 0, we get P(E) = (λ/N ) r=0 (1/(M − r)) + O(λ2 ). Since, as shown above, E[X − Y ] = limλ→0 P(E)/λ, this proves eqn (16.31), which establishes the C–S formula.

16.5

An exercise: Multi-index assignment

In Section 16.2.4, we computed the asymptotic minimum cost for random instances of the assignment problem using the cavity method under the replica-symmetric (RS) assumption. The result, namely that the cost converges to ζ(2) for exponential edge weights with mean 1, was confirmed by the combinatorial analysis of Section 16.4. This suggests that the RS assumption is correct for this ensemble, an intuition that is further confirmed by the fact that the min-sum algorithm finds the optimal assignment. Statistical physicists conjecture that there exists a broad class of random combinatorial problems which satisfy the RS assumption. On the other hand, many problems are thought not to satisfy it: the techniques developed for dealing with such problems will be presented in Chapter 19. It is important to have a feeling for the boundary separating RS from non-RS problems. This is a rather subtle point. Here we want to illustrate it by considering a generalization of random assignment: the multi-index random assignment (MIRA) problem. We shall consider studying the MIRA problem using the RS cavity method and detect the inconsistency of this approach. Since the present section is essentially an application of the methods developed above for the assignment problem, we shall skip all technical details. The reader may consider it as a long guided exercise. An instance of the multi-index assignment problem consists of d sets A1 , . . . , Ad , of N vertices, and a cost Ei for every d-tuple i = (i1 , . . . , id ) ∈ A1 × · · · × Ad . A = 0). A matching is a set of ‘hyperedge’ i can be occupied (ni = 1) or empty (ni hyperedges which are vertex disjoint; this means that i: a∈i ni ≤ 1 for each r and each a ∈ Ar . The cost of a matching is the sum of the costs of the hyperedges that it occupies. The problem is to find a perfect matching (i.e. a matching with N occupied hyperedges) with minimal total cost. In order to define a random ensemble of instances of the multi-index assignment problem, we proceed as for the assignment problem, and assume that the edge costs Ea are i.i.d. exponential random variables with mean N d−1 . Thus the costs have a density ρ(E) = N −d+1 e−E/N

d−1

I(E ≥ 0) .

(16.35)

An exercise: Multi-index assignment

¿

The reader is invited to check that under this scaling of the edge costs, the typical optimal cost is extensive, i.e. Θ(N ). The simple random assignment problem considered before corresponds to d = 2. We introduce a probability distribution on matchings that naturally generalizes eqn (16.2): & 1 −β Pi ni (Ei −2γ) % e I ni ≤ 1 , (16.36) µ(n) = Z i: a∈i a∈F

where F ≡ A1 ∪ · · · ∪ Ad . The associated factor graph has N d variable nodes, each of degree d, corresponding to the original hyperedges, and dN factor nodes, each of degree N , corresponding to the vertices in F . As usual i, j, · · · ∈ V denote the variable nodes in the factor graph and a, b, · · · ∈ F denote the function nodes enforcing the hard constraints. Using a parameterization analogous to that for the assignment problem, we find that the BP equations for this model take the form xb→i , hi→a = b∈∂i\a

xa→i = −

3 4 1 log e−βγ + e−β(Ej −hj→a ) . β

(16.37)

j∈∂a\i

In the large-β, γ limit they become xb→i , hi→a =

xa→i = min (Ej − hj→a ).

(16.38)

j∈∂a\i

b∈∂i\a

Finally, the Bethe free entropy can be written in terms of x-messages, yielding 4 3 −β(E −P i b∈∂i\a xb→i ) F[x] =N dβγ + log e−βγ + e a∈F

− (d − 1)

i∈∂a

4 3 P log 1 + e−β(Ei − a∈∂i xj→a) ) .

(16.39)

i∈V

Using the RS cavity method, we obtain the following equation for the distribution of x-messages in the N → ∞ limit: ⎧ ⎫ d−1 & % d−1 ⎨ % ⎬ & d−1 A(x) = exp − x+ tj I x + tj ≥ 0 dA(tj ) . (16.40) ⎩ ⎭ j=1

j=1

j=1

This reduces to eqn (16.13) in the case of simple assignment. Under the RS assumption, the cost of the optimal assignment is E∗ = N ∗ + o(N ), where d d & 1 % &2 % xj I xj > 0 dA(xj ) . (16.41) ∗ = 2 j=1 j j=1 These equations can be solved numerically to high precision and allow one to derive several consequences of the RS assumption. However, the resulting predictions

¿


(in particular, the cost of the optimal assignment) are wrong for d ≥ 3. There are two observations that show that the RS assumption is inconsistent: 1. Using the Bethe free-entropy expression (16.39), we can compute the asymptotic free-energy density as f RS (β) = limN →∞ F/N , for a finite β = 1/T . The resulting expression can be estimated numerically via population dynamics, for instance for d = 3. It turns out that the predicted entropy density s(T ) = f RS − β(df RS /dβ) becomes negative for T < Tcr ≈ 2.43. This implies that the prediction is incorrect: we are dealing with a statistical-mechanics model with a finite state space, and thus the entropy must be non-negative. 2. A local stability analysis can be performed analogously to what was done in Section 16.2.5. It turns out that, for d = 3, the stability coefficient λ∞ (see eqn (16.23)), becomes positive for T 1.6, indicating an instability of the putative RS solution to small perturbations. The same findings are generic for d ≥ 3. A more satisfactory set of predictions for such problems can be developed using the RSB cavity method, which will be treated in Chapter 19.

Notes Rigorous upper bounds on the cost of the optimal random assignment date back to Walkup (1979) and Karp (1987). The ζ(2) result for the cost was first obtained by Mézard and Parisi (1985) using the replica method. The cavity method solution was then found by Mézard and Parisi (1986) and Krauth and Mézard (1989), but the presentation in Section 16.2 is closer to that of Martin et al. (2005). The last of these papers deals with the multi-index assignment problem and contains answers to the exercise in Section 16.5, as well as a proper solution of this problem using the RSB cavity method. The first rigorous proof of the ζ(2) result was derived by Aldous (2001), using a method which can be regarded as a rigorous version of the cavity method. An essential step in developing this proof was the establishment of the existence of the limit, and its description as a minimum-cost matching on an infinite tree (Aldous, 1992). An extended review of the ‘objective method’ on which this convergence result is based can be found in Aldous and Steele (2003). A survey of the recursive distributional equations such as eqn (16.17) that occur in the replica-symmetric cavity method can be found in Aldous and Bandyopadhyay (2005). On the algorithmic side, the assignment problem has been a very well-studied problem for many years (Papadimitriou and Steiglitz, 1998), and there exist efficient algorithms based on ideas of network flows. The first BP algorithm was found by Bayati et al. (2005); it was then simplified by Bayati et al. (2006) into the O(N 3 ) algorithm presented in Section 16.3. This paper also shows that the BP algorithm is basically equivalent to Bertsekas’s auction algorithm (Bertsekas, 1988). The periodicup-to-a-shift stopping criterion is due to Sportiello (2004), and the explanation for the existence of diverging time scales for the onset of the drift can be found in Grosso (2004).

Notes

¿

Combinatorial studies of random assignments were initiated by Parisi’s conjecture (Parisi, 1998). This was generalized to the Coppersmith–Sorkin conjecture by Coppersmith and Sorkin (1999). The same paper also provides a nice survey of algorithmic bounds. In 2003, these conjectures were turned into theorems by two groups independently (Nair et al., 2003, 2006; Linusson and W¨ astlund, 2004). The simple derivation presented here is from Wästlund (2008).


17 Ising models on random graphs In this chapter, we shall consider two statistical-physics models for magnetic systems: the Ising ferromagnet and the Ising spin glass. While for physical applications one is mainly interested in three-dimensional Euclidean lattices, we shall study models on sparse random graphs. It turns out that this is a much simpler problem, and one can hope to obtain an exact solution in the thermodynamic limit. It is expected that, for a large family of ‘mean-field’ graphs, the qualitative behaviour of Ising models will be similar to that for sparse random graphs. For instance, ferromagnetic Ising models on any lattice of dimension d > 4 share many of the features of models on random graphs. A good understanding of which graphs are mean-field is, however, still lacking. Here we shall develop the replica-symmetric (RS) cavity approach, and study its stability. In the ferromagnetic case, this allows us to compute the asymptotic free energy per spin and the local magnetizations, at all temperatures. In the spin glass case, the ‘RS solution’ is correct only at high temperature, in the ‘paramagnetic’ phase. We shall see that the RS approach is inconsistent in the low-temperature ‘spin glass’ phase, and identify the critical transition temperature separating these two phases. Despite its inconsistency in the spin glass phase, the RS approach provides a very good approximation for many quantities of interest. Our study will be based mainly on non-rigorous physical methods. The results on ferromagnets and on the high-temperature phase of spin glasses, however, can be turned into rigorous statements, and we shall briefly outline the ideas involved in the rigorous approach. The basic notation and the BP (or cavity) formalism for Ising models are set in Section 17.1. Section 17.2 specializes to the case of random graph ensembles, and introduces the corresponding distributional equations. Finally, Sections 17.3 and 17.4 deal with the analysis of these equations and derive the phase diagram in the ferromagnetic and spin glass cases, respectively.

17.1

The BP equations for Ising spins

Given a graph G = (V, E), with |V | = N , we consider a model for N Ising spins σi ∈ {±1}, i ∈ V , with an energy function Jij σi σj . (17.1) E(σ) = − (i,j)∈E

The coupling constants Jij = Jji are associated with edges of G and indicate the strength of the interaction between spins connected by an edge. The graph G and the

¿

Ising models on random graphs

set of coupling constants J define a ‘sample’. Given a sample, the Boltzmann distribution is µG,J (σ) =

1 ZG,J

e−βE(σ) =

1 ZG,J

eβJij σi σj .

(17.2)

(i,j)∈E

As in the general model (14.13), this distribution factorizes according to a factor graph that has a variable node associated with each vertex i ∈ V , and a function node associated with each edge (i, j) ∈ E. It is straightforward to apply BP to such a model. Since the model (17.2) is pairwise (every function node has degree 2), the BP equations can be simplified following the strategy of Section 14.2.5. The message ν (ij)→j from function node (ij) to variable node j is related to the message νi→(ij) through (t) ν (ij)→j (σj ) ∼ =

(t)

eβJij σi σj νi→(ij) (σi ) .

(17.3)

σi

We can choose to work with only one type of message on each directed edge i → j of the original graph, say the variable-to-factor message. With a slight misuse of notation, we shall write νi→j (σi ) ≡ νi→(ij) (σi ). The BP update equations now read (see eqn(14.31)) (t+1) νi→j (σi ) ∼ =

(t)

eβJki σk σi νk→i (σk ) .

(17.4)

k∈∂i\j σk

Since spins are binary variables, one can parameterize messages by their log(t) likelihood ratio hi→j , defined through the relation 4 3 (t) (t) νi→j (σi ) ∼ = exp β hi→j σi .

(17.5)

We follow here the physics convention of rescaling the log-likelihood ratio by a factor 1/(2β). The origin of this convention lies in the idea of interpreting the distribution (17.5) as an ‘effective’ Boltzmann distribution for the spin σi , with energy function (t) (t) −hi→j σi . In statistical-physics jargon, hi→j is a local magnetic field. (t)

(t)

Two messages hi→j , hj→i ∈ R are exchanged along each edge (i, j). In this parameterization, the BP update equations (17.4) become (t+1) (t) f (Jki , hk→i ) , (17.6) hi→j = k∈∂i\j

where the function f has already been encountered in Section 14.1: f (J, h) =

1 atanh [tanh(βJ) tanh(βh)] . β

(17.7)

The local marginals and the free entropy can be estimated in terms of these messages. We shall assume here that h ≡ {hi→j } is a set of messages which solve the fixed-point equations. The marginal of spin σi is then estimated as

The BP equations for Ising spins

νi (σi ) ∼ = eβHi σi ,

Hi =

f (Jik , hk→i ) .

¿

(17.8)

k∈∂i

The free entropy log ZG,J is estimated using the general formulae (14.27) and (14.28). The result can be expressed in terms of the cavity fields {hi→j }. Using the shorthand θij ≡ tanh βJij , one gets

F[h] =

log cosh βJij −

(ij)∈E

+

i∈V

log

3

4 3 log 1 + θij tanh βhi→j tanh βhj→i

(ij)∈E

(1 + θij tanh βhj→i ) +

j∈∂i

4

(1 − θij tanh βhj→i )

.

(17.9)

j∈∂i

This expression is obtained by using the parameterization (17.5) in the general expressions (14.27) and (14.28), as outlined in the exercise below. As an alternative, one can use the expression (14.32) for pairwise models.

Exercise 17.1 In order to derive eqn (17.9), it is convenient to change the normalization of the compatibility functions by letting ψij (σi , σj ) = 1 + θij σi σj . This produces an overall change in the the energy that is taken into account by the first term in eqn (17.9). To get the other terms: (a) Show that νb(ij)→j (σj ) = 12 (1 + σj θij tanh βhi→j ). Note that F(ν), (see eqn (14.27)), is left unchanged by a multiplicative rescaling of the messages. As a consequence, we can use νb(ij)→j (σi ) = 1 + σj θij tanh βhi→j . (b) Show that, with this choice, the term Fi (ν) in eqn (14.27) is equal to the last term in eqn (17.9). P (c) Note that σi σi νi→(ij) (σi ) = tanh βhi→j . (d) Show that this implies (for a ≡ (ij)) Fa (ν) = Fai (ν) = Faj (ν) = log{1 + θij tanh βhi→j tanh βhj→i } .

(17.10)

Exercise 17.2 Show that the fixed points of the BP update equations (17.6) are stationary points of the free-energy functional (17.9). [Hint: Differentiate the right-hand side of eqn (17.9) with respect to tanh βhi→j .]

¿


Exercise 17.3 Show that the Bethe approximation to the internal energy, (see eqn (14.21)), is given by U =−

X

Jij

(ij)

θij + tanh βhi→j tanh βhj→i . 1 + θij tanh βhi→j tanh βhj→i

(17.11)

Check that U = −dF/dβ as it should, where F is given in eqn. (17.9). [Hint: Use the result of the previous exercise]

17.2

RS cavity analysis

We shall now specialize our analysis to the case of sparse random graphs. More precisely, we assume G to be a uniformly random graph with degree profile {Λk } (i.e., for each k ≥ 0, the number of vertices of degree k is N Λk ). The associated factor graph isa random factor graph from the ensemble DN (Λ, P ), where P (x) = x2 and Λ(x) = k≥0 Λk xk (see Section 9.2). As for the couplings Jij , we shall focus on two significant examples: (i) Ferromagnetic models, with Jij = +1 for any edge (i, j). (ii) Spin glass models. In this case the couplings Jij are i.i.d. random variables with Jij ∈ {+1, −1} uniformly at random. The general case of i.i.d. random couplings Jij can be treated within the same framework. Let us emphasize that the graph G and the couplings {Jij } are quenched random variables. We are interested in the properties of the measure (17.2) for a typical random realization of G, J. We shall pursue this goal by analysing distributions of BP messages (cavity fields). 17.2.1

Fixed-point equations

We choose an edge (i, j) uniformly at random in the graph G. The cavity field hi→j is a random variable (both because of the random choice of (i, j) and because of the randomness in the model). Within the assumptions of the RS cavity method, in the large-N limit, the distribution of h = hi→j satisfies the distributional equation d

h=

K−1

f (Ji , hi ) ,

(17.12)

i=1

where K is distributed according to the edge-perspective degree distribution: the prob∞ ability that K = k is λk = kΛk /( p=1 pΛp ). The fields h1 , . . . , hK are independent copies of the random variable h, and the couplings J1 , . . . JK are i.i.d. random variables distributed as the couplings in the model. In the physics literature, the distributional equation (17.12) is written formally in terms of the density a( · ) of the random variable h. Writing EJ for the expectation

RS cavity analysis

¿

over the i.i.d. coupling constants J1 , J2 , . . . , and enforcing the cavity equation through a Dirac delta, we have k ∞ k λk EJ δ h − f (Ji , hi ) a(hr ) dhr . (17.13) a(h) = r=1

i=1

k=1

Let us emphasize that, in writing such an equation, physicists do not assume that a genuine density a( · ) does exist. This and similar equations should be interpreted as a proxy for expressions such as eqn (17.12). Assuming that the distribution of h that solves eqn (17.12) has been found, the RS cavity method predicts the asymptotic free-entropy density f RS ≡ limN →∞ F/N as 4 3 1 1 (17.14) f RS = − Λ log cosh β + Λ EJ,h log 1 + tanh βJ tanh βh1 tanh βh2 2 2 k k 3 4 +Ek EJ,h log (1 + tanh βJi tanh βhi ) + (1 − tanh βJi tanh βhi ) . i=1

i=1

Here k is distributed according to {Λk }, the hi are i.i.d. random variables distributed as h, and the Ji are i.i.d. couplings (identically equal to +1 for ferromagnets, and uniform in {+1, −1} for spin glasses). Finally, Λ = k kΛk = Λ (1) denotes the average degree. 17.2.2

The paramagnetic solution

The RS distributional equation (17.12) always admits the solution ‘h = 0’, meaning that the random variable h is equal to 0 with probability 1. The corresponding distribution is a Dirac delta on h = 0: a = δ0 . This is usually referred to as the paramagnetic solution. Using eqn (17.14) we obtain the corresponding prediction for the free-entropy density 1 RS (β) = log 2 + Λ log cosh β . fpara 2

(17.15)

Exercise 17.4 In the context of this paramagnetic solution, derive expressions for the internal-energy density (−(Λ/2)EJ J tanh(βJ)) and the entropy density (log 2 + (Λ/2)EJ [log cosh(βJ) − βJ tanh(βJ)]).

In order to interpret this solution, recall that a( · ) is the asymptotic distribution of the cavity fields (BP messages) hi→j . The paramagnetic solution indicates that they vanish (apart, possibly, from a sublinear number of edges). Recalling the expression for local marginals in terms of cavity fields (see eqn (17.8)), this implies νi (σi = +1) = νi (σi = −1) = 1/2. One can similarly derive the joint distribution of two spins σi , σj connected by a single edge with a coupling Jij . A straightforward calculation yields νij (σi , σj ) =

7 8 1 exp βJij σi σj . 4 cosh β

(17.16)

¿


This is the same distribution as for two isolated spins interacting via the coupling Jij . In the paramagnetic solution, the effect of the rest of the graph on local marginals is negligible. The fact that h = 0 is a solution of the RS distributional equation (17.12) does not imply that the paramagnetic predictions are correct. We can distinguish three possibilities: 1. The paramagnetic predictions are correct. 2. The hypotheses of the RS cavity method do hold, but the distributional equation (17.12) admits more than one solution. In this case it is possible that behaviour of the model is correctly described by one of these solutions, although not the paramagnetic one. 3. The hypotheses of the RS method are incorrect. It turns out that, in the high-temperature phase, the paramagnetic solution is correct for both the ferromagnetic and the spin glass model. At low temperature (the critical temperature being different in the two cases), new solutions to the RS equations appear. Wheras for the ferromagnetic model correct asymptotic predictions are obtained by selecting the appropriate RS solution, this is not the case for spin glasses.

17.3

Ferromagnetic model

For the ferromagnetic Ising model (Jij = +1 identically), the paramagnetic solution correctly describes the asymptotic behaviour for β ≤ βc . The critical temperature depends on the graph ensemble only through the average degree from the edge perspective λ, and is the unique solution of the equation λ tanh βc = 1 .

(17.17)

The edge-perspective average degree λ is defined in terms of the degree distribution as Λk k(k − 1) , (17.18) λ= λk (k − 1) = k k Λk k k

or, more compactly, in terms of the degree generating function as λ = λ (1). In words, we choose an edge (i, j) of G uniformly at random, and select one of its end-points, say i, also at random. Then, λ is the expected number of edges incident on i distinct from (i, j). For β > βc , the RS distributional equation (17.12) admits more than one solution. While h = 0 is still a solution, the correct thermodynamic behaviour is obtained by selecting a different solution, the ‘ferromagnetic’ solution. 17.3.1

Local stability

When confronted with a distributional equation such as eqn (17.12), the simplest thing we can try to do is to take expectations. Recalling that k has a distribution λk and writing θ = tanh β, we obtain

Ferromagnetic model

E{h} = λ E{f (+1, h)} = λ E{θ h + O(h3 )} = λθ E{h} + E{O(h3 )} .

¿

(17.19)

If we neglect terms of order E{O(h3 )} (for instance, assuming that E{h3 } = O(E{h}3 )), this yields a linear iteration for the expectation of h. For λθ > 1, this iteration is unstable, indicating that h = 0 is an unstable fixed point and new fixed distributions appear. Indeed, a little more work shows that, for λθ < 1, h = 0 is the unique fixed point. It is enough to take the expectation of |h| and use the triangle inequality to obtain E|h| ≤ λ E|f (+1, h)| .

(17.20)

A little calculus shows that f (1, x) ≤ θx for x > 0, whence E|h| ≤ λθE|h| and therefore h = 0 identically. 17.3.2

Ferromagnetic susceptibility

There is a second, more physical, interpretation of the above calculation. Consider the ferromagnetic susceptibility, χ=

1 (σi σj − σi σj ) . N

(17.21)

i,j∈V

We expect this quantity to be bounded as N → ∞. This corresponds to the expectation that the total magnetization i σi behaves as in the central limit theorem. The idea is to assume that the paramagnetic solution is the correct one, and to check whether χ is indeed bounded. We have already seen that, in the paramagnetic phase, σi vanishes. Let us now consider two randomly chosen vertices i, j at a given distance r, and let us compute the expectation σi σj . With high probability, there is a single path with r edges joining i to j, and any finite neighbourhood of this path is a tree. Let us rename the two vertices 0 and r, and denote by σn , n ∈ {0, . . . , r}, the spins along the path joining them. Within the RS cavity approach, the marginal distribution of these spins is, (see eqn (14.18)): # µG (σ0 , . . . , σr ) ∼ = exp β

r−1

σp σp+1 + β

p=0

r

$ gp σp

.

(17.22)

p=0

The field gp is the effect of the rest of the graph on the distribution of the spin σp . Using eqn (14.18), one gets gp =

f (+1, hj→p ) ,

(17.23)

j∈∂p\path

where the sum extends over the set ∂p of neighbours of σp in the factor graph that are not on the path. Since, in the paramagnetic solution, the hj→p ’s vanish, all of the gp vanish as well. This implies σ0 σr = (tanh β)r .

¿


The susceptibility is then given by χ=

∞ 1 σi σj = N (r) (tanh β)r , N r=0

(17.24)

i,j∈V

where N (r) is the expected number of vertices j at distance r from a uniformly random . r vertex i. It is not hard to show that N (r) = λ for large r (the limit N → ∞ is taken before r → ∞). It follows that, for λ tanh β < 1, the susceptibility χ remains bounded. For λ tanh β > 1, the susceptibility is infinite and the paramagnetic solution cannot be correct. 17.3.3

The ferromagnetic phase

For β > βc (low temperature), one has to look for other solutions of the RS distributional equation (17.12). In general, this is done numerically, for instance using the population dynamics algorithm. In special cases, exact solutions can be found. As an example, assume that G is a random regular graph of degree k + 1. The corresponding factor graph is thus drawn from the ensemble DN (Λ, P ), with Λ(x) = xk+1 and P (x) = x2 . The local structure of the graph around a typical vertex is always the same: a regular tree of degree (k + 1). It is thus natural to seek a solution such that hi→j = h0 for each directed edge i → j. This corresponds to a solution of the distributional equation (17.12) where h = h0 identically (formally a = δh0 ). Equation (17.12) implies h0 = kf (+1, h0 ) .

(17.25)

This equation is easily solved. For β > βc , it has three solutions: the paramagnetic solution, and two opposite solutions ±h0 . In order to interpret these solutions, recall that eqn (17.8) implies σi = ±M , with M = tanh((k + 1)βf (+1, h0 )), where h0 is the positive solution of eqn (17.25). Figure 17.1 summarizes these results for the case k = 2. The reader might be puzzled by these results since, by symmetry, one necessarily has σi = 0. The correct interpretation of the ferromagnetic solution is similar to that of the Curie–Weiss model discussed in Section 2.5.2. In a typical configuration, the total magnetization i σi is, with high probability, close to +N M or −N M . Since each of these events occurs with probability 1/2, the average magnetization is 0. We can decompose the Boltzmann distribution as µ(σ) =

1 1 µ+ (σ) + µ− (σ) , 2 2

(17.26)

where µ+ ( · ) and µ− ( · ) are supported on configurations with i σi > 0 and i σi < 0, respectively, assuming N odd for simplicity. The expectation of a typical spin with respect to µ+ ( · ) or µ− ( · )) is then, asymptotically, σi + = +M or σi − = −M , respectively. The two components µ+ ( · ) and µ− ( · ) are often referred to as ‘pure states’, and are expected to have several remarkable properties. We will shall discuss decomposition into pure states further in the following chapters. It is interesting to notice that the

Ferromagnetic model

¿

2.5 2 h 1.5 1

M

0.5 0 0

0.5

1

1.5

2

2.5

T Fig. 17.1 Magnetization M and cavity field h0 in a ferromagnetic Ising model on a random regular graph of degree k + 1 = 3, as a function of the temperature. The critical temperature is Tc = 1/atanh(1/2) ≈ 1.82048.

cavity method does not reproduce the actual averages over the Boltzmann distribution, but rather the averages with respect to pure states. One can repeat the stability analysis for this new solution. The ferromagnetic susceptibility always involves the correlation between two spins in a one-dimensional problem, as in eqn (17.22). However, the fields gp are now non-zero. In particular, for p ∈ {1, . . . , r − 1}, we have gp = (k − 1)f (+1, h0 ). An explicit computation using the transfer matrix technique of Section 2.5.1 shows that the susceptibility of this ferromagnetic solution is finite in the whole low-temperature phase β > βc .

Exercise 17.5 Consider the case of a ferromagnetic Ising model on a random regular graph P in the presence of a positive external magnetic field. This is described by a term +B i σi in the exponent in eqn (17.2). Show that there is no phase transition: the cavity field h0 and the magnetization M are positive at all temperatures.

Exercise 17.6 Consider a ferromagnetic problem in which the Jij are i.i.d. random variables, but always positive (i.e. their distribution is supported on J > 0.) Write the RS distributional equation (17.12) in this case and perform a local stability analysis of the paramagnetic solution. Show that the critical temperature is determined by the equation λ EJ tanh(βJ) = 1. Compute the magnetization in the ferromagnetic phase (i.e. for β > βc ) using the population dynamics algorithm.

¿


17.3.4

Rigorous treatment

In the case of Ising ferromagnets, many predictions of the RS cavity method can be confirmed rigorously. While describing the proofs in detail would take us to far, we shall outline the main ideas here. To be concrete, we consider a model of the form (17.2), with Jij = +1, on a random graph with degree profile Λ such that λ is finite (i.e. Λ must have a second moment). Since we know that eqn (17.12) admits more than one solution at low temperature, we need to select a particular solution. This can be done by considering the density evolution iteration K−1 d (t) f (+1, hi ) (17.27) h(t+1) = i=1 (0)

with the initial condition h = +∞. In other words, we consider the sequence of distributions a( · ) that are obtained by iterating eqn (17.13) starting from δ+∞ . It can be shown that the sequence of random variables h(t) converges in distribution as t → ∞, and that the limit h(+) is a solution of eqn (17.12). It is important to stress that in general h(+) is a random variable with a highly non-trivial distribution. For λ tanh β < 1, the argument in Section 17.3.1 implies that h(+) = 0 with probability one. For λ tanh β > 1, h(+) is supported on non-negative values. Theorem 17.1 Let ZN (β) be the partition function of a ferromagnetic Ising model on a random graph with degree profile Λ. Under the hypotheses above, the RS cavity expression for the free entropy is exact. Precisely, 1 log ZN (β) = f RS (β) , N →∞ N lim

(17.28)

the limit holding almost surely for any finite β. Here f RS (β) is defined as in eqn (17.14), where the hi are i.i.d. copies of h(+) . Let us sketch the key ideas used in the proof, referring to the literature for a complete derivation. The first step consists in introducing a small positive magnetic field B that adds a term −B i σi to the energy (17.1). One then uses the facts that the free-entropy density is continuous in B (which also applies as N → ∞) and that log ZN (β, B) concentrates around E log ZN (β, B). This allows us to focus on E log ZN (β, B). It is easy to check that the cavity prediction is correct at β = 0 (since, in this limit, the spins become independent). The idea is then to compute the derivative of N −1 E log ZN (β, B) with respect to β and prove that this converges to the derivative of f RS (β). Some calculus shows that this is equivalent to proving that, for a uniformly random edge (i, j) in the graph, tanh β + tanh βh1 tanh βh2 lim Eσi σj = E , (17.29) N →∞ 1 + tanh β tanh βh1 tanh βh2 where h1 , h2 are i.i.d. copies of h(+) . The advantage of eqn (17.29) is that σi σj is a local quantity. One can then try to estimate it by looking at the local structure of the graph in a large but finite

Spin glass models

¿

neighbourhood around (i, j). The key problem is to prove that the rest of the graph ‘decouples’ from the expectation σi σj . One important ingredient in this proof is Griffiths’ inequality. For the reader’s reference, we recall it here for the case of a pairwise Ising model. Theorem 17.2 Consider a ferromagnetic Ising model, i.e. a Boltzmann distribution of the form ⎧ ⎫ ⎨ ⎬ 1 exp β Jij σi σj + Bi σi , (17.30) µG,J (σ) = ⎩ ⎭ ZG,J i

(i,j)∈G

with Jij , Bi ≥ 0. Then, for any U ⊆ V , in all of the Jij , Bi .

i∈U

σi is non-negative and non-decreasing

The strategy is then the following: 1. Given a certain neighbourhood S of the edge (i, j), one can consider two modified measures on the spins of S given by the Boltzmann measure where the spins outside of S have been fixed. The first measure (the ‘+ boundary condition’) has σi = 1 ∀i ∈ S. The second one (the ‘free boundary condition’) has σi = 0 ∀i ∈ S. The latter also amounts to considering the model on the subgraph induced by S. Griffiths’ inequality allows one to show that the true σi σj is bounded by the expectations obtained with the + and with free boundary conditions. 2. It can be shown that whenever the recursion (17.27) is initialized with h(0) supported on non-negative values, it converges to h(+) . 3. Finally one takes for S a ball of radius r centred on i. This is, with high probability, a tree. Using the result in item 2, one proves that the two expectations of σi σj with the + and free boundary conditions converge to the same value as r → ∞.

17.4

Spin glass models

We now turn to the study of the spin glass problem. Again, the paramagnetic solution turns out to describe correctly the system at high temperature, and the critical temperature depends on the ensemble only through the quantity λ. We shall see that the paramagnetic solution is unstable for β > βc , where λ (tanh βc )2 = 1 .

(17.31)

As in the previous section, we shall try to find another solution of the RS distributional equation (17.12) when β > βc . Such a solution exists and can be studied numerically; it yields predictions which are better than those of the paramagnetic solution. However, we shall argue that, even in the large-system limit, this solution does not correctly describe the model: the RS assumptions do not hold for β > βc , and replica symmetry breaking is needed. 17.4.1

Local stability

As with the ferromagnetic model, we begin with a local stability analysis of the paramagnetic solution, by taking moments of the RS distributional equation. Since f (J, h)

¿


is antisymmetric in J, any solution of eqn (17.12) has E{h} = 0 identically. The lowest non-trivial moment is therefore E{h2 }. Using the fact that E{f (Ji , hi )f (Jj , hj )} = 0 for i = j, and the Taylor expansion of f (J, h) around h = 0, we get E{h2 } = λ E{f (J, h)2 } = λ θ2 E{h2 } + E{O(h4 )} .

(17.32)

Assuming that E{O(h4 )} = O(E{h2 }2 ) (this step can be justified through a lengthier argument), we get a linear recursion for the second moment of h which is unstable for λθ2 > 1 (i.e. β > βc ). On the other hand, as in Section 17.3.1, one can use the d

inequality |f (J, h)| ≤ h(tanh β) to show that the paramagnetic solution h = δ0 is the only solution in the high-temperature phase λ θ2 < 1. 17.4.2

Spin glass susceptibility

An alternative argument for the appearance of the spin glass phase at β > βc is provided by computing the spin glass susceptibility. Recalling the discussion in Section 12.3.2, and using σi = 0 (as is the case in the paramagnetic case), we have χSG =

1 σi σj 2 . N

(17.33)

i,j∈V

Assuming the RS cavity approach to be correct, and using the paramagnetic solution, the computation of σi σj is analogous to the one we did in the ferromagnetic case. Denoting by ω the shortest path in G between i and j, the result is

σi σj =

tanh βJkl .

(17.34)

(kl)∈ω

Taking the square and splitting the sum according to the distance between i and j, we get χSG =

∞

N (r) (tanh β)2r ,

(17.35)

r=0

where N (r) is the average number of vertices at distance r from a random vertex i. We . r have N (r) = λ to the leading exponential order for large r. Therefore the above series is summable for λ θ2 < 1, yielding a bounded susceptibility. Conversely, if λ θ2 > 1 the series diverges, and the paramagnetic solution must be considered inconsistent. 17.4.3

Paramagnetic phase: Rigorous treatment

From a mathematical point of view, determining the free energy of the spin glass model on a sparse random graph is a challenging open problem. The only regime that is (relatively) well understood is the paramagnetic phase at zero external field.

Spin glass models

¿

Theorem 17.3 Let ZN (β) denote the partition function of the spin glass model with couplings ±1 on a random graph G, with a degree distribution Λ which has a finite second moment. If λ (tanh β)2 < 1, then 1 1 RS log ZN (β) = fpara (β) ≡ log 2 + Λ log cosh β , N 2 where the limit holds almost surely. lim

N →∞

(17.36)

Proof To keep things simple, we shall prove only a slightly weaker statement, namely the following. For any δ > 0, RS

RS

eN [fpara −δ] ≤ ZN ≤ eN [fpara +δ] ,

(17.37)

with high probability as N → ∞. In the proof, we shall denote by EJ the expectation with respect to the couplings Jij ∈ ±1, and write M = N Λ/2 for the number of edges RS

in G, so that eN fpara = 2N (cosh β)M . RS The probability of the event ZN ≥ eN [fpara +δ] is bounded from above using the Markov inequality. The annealed partition function is ⎧ ⎫ ⎨ ⎬ EJ eβJij σi σj = (cosh β)M = 2N (cosh β)M , (17.38) EJ {ZN } = ⎩ ⎭ σ σ (ij)∈E

whence ZN ≤ 2N (cosh β)M eN δ with high probability. The lower bound follows by applying the second-moment method to the random realization of the couplings Jij , given a typical random graph G: it is sufficient to show 2 } ≤ EJ {ZN }2 eN δ for any δ > 0. that EJ {ZN Surprisingly, the computation of the second moment reduces to computing the partition function of a ferromagnetic Ising model. The key identity is 1

1

2

2

EJ {eβJij (σi σj +σi σj ) } =

(cosh β)2 γτi τj e , cosh γ

(17.39)

where γ ≡ atanh((tanh β)2 ); σi1 , σi2 , σj1 , σj2 are Ising spins; and τi = σi1 σi2 andτj = σj1 σj2 . Using this identity, we find 3 4 1 1 2 2 2 (17.40) }= EJ eβJij (σi σj +σi σj ) EJ {ZN σ 1 ,σ 2

N

=2

(ij)∈E

(cosh β)2 cosh γ

M τ

e

γτi τj

N

=2

(ij)∈E

(cosh β)2 cosh γ

M f ZN (γ) ,

f ZN (γ)

where is the partition function of a ferromagnetic Ising model on the graph G at inverse temperature γ. This can be estimated through Theorem 17.1. As λ(tanh γ) = λ (tanh β)2 < 1, this ferromagnetic model is in its paramagnetic phase, and therefore f (γ) ≤ 2N (cosh γ)M eN δ with high probability. This in turn implies ZN 2 } ≤ 22N (cosh β)2M eN δ , EJ {ZN

which completes the proof.

(17.41)

¿


1.4

1.4

1.2

1.2

1.0

1.0

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0.0

0.0 -2

-1

0

1

2

-2

-1

h

0

1

2

h

Fig. 17.2 Distribution of cavity fields approximated by the population dynamics algorithm at T = 0.5 (left) and T = 0.1 (right), for a Jij = ±1 spin glass on a random regular graph with degree 3. Here, we plot a histogram with bin size ∆h = 0.01. A population of 104 fields {hi } was used in the algorithm, and the resulting histogram was averaged over 104 iterations.

17.4.4

An attempt to describe the spin glass phase

If β > βc , the RS distributional equation (17.12) admits a solution h that is not identically 0. The corresponding distribution a( · ) is symmetric under h → −h and can be approximated numerically using the population dynamics algorithm. This is usually referred to as ‘the RS spin glass solution’ (although it is far from obvious whether it is unique.) In Fig. 17.2 we plot the empirical distribution of cavity fields as obtained through the population dynamics algorithm, for a random regular graph of degree k+1 = 3. The corresponding √ critical temperature can be determined through eqn (17.31), yielding Tc = 1/ log( 2 + 1) ≈ 1.134592. Notice that the distribution a( · ) is extremely nontrivial, and indeed is likely to be highly singular. Once an approximation of the distribution of fields has been obtained, the free entropy can be estimated as f RS , given in eqn (17.14). Figure 17.3 shows the free energy F versus temperature. It is difficult to control the solutions of the RS distributional equation (17.12) analytically, with the exception of some limiting cases. One such case is the object of the next exercise.

Exercise 17.7 Consider the spin glass model on a random √ regular √ graph with degree k + 1, and assume that the couplings Jij take values {+1/ k, −1/ k} independently and uniformly at random. Argue that, as k → ∞, the following limiting behaviour holds. (a) The solution a( · ) of the RS equation (17.12) converges to a Gaussian with mean 0 and variance q, where q solves the equation Z q=

2

(tanh βh)2 e−h

/(2q)

dh √ = Eh tanh2 (βh) . 2πq

(17.42)

Spin glass models

¿

-1.2

-1.25

−f RS /β

-1.3

-1.35

-1.4

-1.45

-1.5 0.2

0.4

0.6

0.8

1

T

1.2

1.4

1.6

1.8

2

Fig. 17.3 The free-energy density −f RS /β of a spin glass model with Jij = ±1 on a random regular graph with degree 3. Here, we plot results obtained for the paramagnetic solution (dashed line) and the RS spin glass solution (full black line). The critical temperature Tc ≈ 1.134592, (see eqn (17.31)), is indicated by the vertical line.

(b) Show that the RS prediction for the free entropy per spin converges to the following value (here the limit k → ∞ is taken after N → ∞): RS fSK (β) = Eh log(2 cosh(βh)) +

β2 (1 − q)2 . 4

(17.43)

Notice that these results are identical to those obtained by the replica-symmetric analysis of the SK model (see eqn (8.68)).

The second limit in which a solution can be found analytically is that of zero temperature, T → 0 (or, equivalently, β → ∞). Assume that h has a finite, nonvanishing limit as β → ∞. It is easy to show that the limiting distribution must satisfy a distributional equation that is formally identical to (17.12), but with the function f ( · ) replaced by its zero-temperature limit f (J, h) = sign(J h) min[|J|, |h|] .

(17.44)

Since J ∈ {+1, −1}, if h takes values on the integers, then f (J, h) ∈ {+1, 0, −1}. As a consequence, eqn (17.12) admits a solution with support on the integers. For the sake of simplicity, we shall restrict ourselves to the case of a regular graph with degree k + 1. The distribution of h can be formally written as a(h) =

k r=−k

pr δr (h) .

(17.45)

¿


∞ Let us denote by p+ = r=1 pr the probability that the field h is positive, and by p− the probability that it is negative. Notice that the distribution of f (J, h) depends only on p+ , p0 and p− and not on the individual weights pr . By the symmetry of a( · ), pr = p−r , and therefore p+ = (1 − p0 )/2. As a consequence, the RS cavity equation (17.13) implies the following equation for p0 : k/2

p0 =

q=0

k 2q

2q 2q k−2q 1 − p0 p0 . q 2

(17.46)

Exercise 17.8 (a) Show that the probability that h = r (and by symmetry, the probability that h = −r) is given by: (k−r)/2

pr = p−r =

X q=0

! «2q+r „ k 1 − p0 pk−2q−r 2 2q + r 0

2q + r q

! .

(17.47)

d

(b) Consider the distribution a∗ ( · ) of the local field H acting on a spin, defined by H = Pk+1 Pk+1 r=1 f (Jr , hr ). Show that it takes the form a(h) = r=−k−1 sr δr (h), where sr is given by the same formula (17.47) as for pr above, with k replaced by k + 1. (c) Show that the ground state energy per spin is

E0 = k

k+1 X r=−k−1

sr |r| − (k + 1)

k X r=−k

pr |r| −

k+1 p0 (2 − p0 ) . 2

(17.48)

As an example, consider the case of a random regular graph with degree k + 1 = 3. In this case, solving eqn (17.46) yields p0 = 1/3, p1 = 2/9, and p2 = 1/9. The resulting ground state energy per spin is E0 = −23/18. 17.4.5

Instability of the RS spin glass solution

It turns out that the solution of the RS distributional equation discussed in the previous subsection does not correctly describe the model in the thermodynamic limit. In particular, the RS spin glass solution predicts an unacceptable negative entropy at low temperatures, and its prediction for the ground state energy per spin, E0 = −23/18 = −1.27777 . . . , differs from the best numerical estimate −1.2716 ± 0.0001. Therefore the assumptions of the RS cavity method cannot be correct for the low-temperature phase λ(tanh β)2 > 1. On the other hand, physicists think that the approach can be rescued: the asymptotic free-energy density can be computed by introducing replica symmetry breaking. This will be the subject of Chapters 19 to 22. Here we want to show how the inconsistency of this ‘RS solution’ can be detected, by the computation of the spin glass susceptibility. The computation is similar to that for the paramagnetic solution, but one has to deal with the presence of non-zero values

Spin glass models

¿

1.6

1.4

1.2

k ρ2

1

0.8

0.6

0.4

0.2

0 0.6

0.8

1

1.2

T

1.4

1.6

1.8

2

Fig. 17.4 Instability of the ‘RS solution’ for the Ising spin glass on a random regular graph with degree k + 1 = 3. The plot shows kρ2 versus the temperature, where ρ2 is the rate of exponential decay of the spin glass correlation function E(σ0 σr − σ0 σr )2 at large r. A value of kρ2 < 1 corresponds to a bounded spin glass susceptibility. This is the case for the paramagnetic solution at T > Tc . For T < Tc ≈ 1.134592, the bottom curve is the result for the RS cavity solution (with a field distribution a(h) estimated from population dynamics), and the top curve is the result for the paramagnetic solution. Both ‘solutions’ yield kρ2 > 1, and are thus unstable.

in the support of the cavity field distribution a( · ). As in Section 17.4.2, the first step is to compute the correlation between two spins σ0 and σr at a distance r in G. This is found by considering the joint distribution of the spins along the shortest path between 0 and r. In the cavity method, this has the form $ # r−1 r (17.49) Jp σp σp+1 + β gp σp . µG,J (σ0 , . . . , σr ) ∼ = exp β p=0

p=0

Here the couplings Jp are i.i.d. and uniformly random in {+1, −1}. The fields gp are i.i.d. variables whose distribution is determined by d

g=

k−1

f (Jq , hq ) ,

(17.50)

q=1

where the hq are k − 1 independent copies of the random variable h that solves eqn (17.12), and the Jr are again independent and uniformly random in {+1, −1}. (To be precise, g0 and gr have a different distribution from the others, as they are sums of k i.i.d. terms instead of k − 1. However, this difference is irrelevant for our computation of the leading exponential order of the correlation at large r.) The solution of the one-dimensional problem (17.22) can be obtained by the transfer matrix method discussed in Section 2.5.1 (and, in the BP context, in Section 14.1).

¿


We denote by zp+ (σ) the partition function of the partial chain including the spins σ0 , σ1 , . . . , σp , where we have fixed σ0 = +1, and σp = σ. We can then immediately derive the recursion + zp+1 (σ) = Tp (σ, σ )zp+ (σ ) , (17.51) σ ∈{+1,−1}

where the p-th transfer matrix takes the form βJ +βg p+1 e p e−βJp +βgp+1 . Tp = e−βJp −βgp+1 eβJp −βgp+1

(17.52)

Similarly, we denote by zp− (σ) the partition function conditional on σ0 = −1. This satisfies the same recursion relation as zp+ , but with a different initial condition. We have, in fact, z0+ (1) = 1, z0+ (−1) = 0, and z0− (1) = 0, z0− (−1) = 1. The joint probability distribution of σ0 and σr is given by µG,J (σ0 , σr ) ∼ = eβg0 I(σ0 = +1) zr+ (σr ) + e−βg0 I(σ0 = −1) zr− (σr ) .

(17.53)

From this expression, one finds, after a few lines of computation, 4[zr+ (1)zr− (−1) − zr+ (−1)zr− (1)] σ0 σr − σ0 σr = ' (2 . eβg0 (zr+ (1) + zr+ (−1)) + e−βg0 (zr− (1) + zr+ (−1))

(17.54)

Approximate samples of the random variables g1 , g2 , . . . gr can be obtained through the population dynamics algorithm. This allows one to evaluate the correlation function (17.54) and estimate its moments. In order to compute the spin glass susceptibility (see Section 17.4.2), one needs to estimate the growth rate of the second moment at large r. We define ρ through the leading exponential behaviour at large r, 2 . E (σ0 σr − σ0 σr ) = ρ2r ,

(17.55)

where E refers here to the expectation with respect to the fields g1 , . . . , gr (i.e. to the graph and the couplings outside the path between 0 and r). In a regular graph with degree (k + 1), the number of neighbours at a distance r from a random vertex i grows like k r . As a consequence, the series for the spin glass susceptibility is summable (thus suggesting that the susceptibility is bounded) if kρ2 < 1. For the paramagnetic solution, ρ = tanh β, and this condition reduces to β < βc (see eqn (17.31)). In Figure 17.4 we plot kρ2 versus the temperature, as computed numerically for the RS spin glass field distribution a( · ). In the whole phase T < Tc , this is larger than one, showing that the RS ‘solution’ is in fact unstable. This analysis is easily generalized to irregular graph ensembles, by replacing k with the average (edge-perspective) degree λ. Physicists point out that the RS spin glass solution, although wrong, looks ‘less wrong’ than the paramagnetic solution. Indeed, kρ2 is smaller in the RS spin glass solution. Also, if one computes the entropy density −dF/dT at zero temperature, one finds that it is negative in both solutions, but it is larger for the RS spin glass solution.

Notes

¿

Table 17.1 Estimates of the ground state energy density of the Ising spin glass on a random regular graph of degree k + 1.

k=2 k=4

Paramagnetic −3/2 = −1.5 −5/2 = −2.5

RS −23/18 ≈ −1.2778 −1.69133

1RSB −1.2723 −1.6752

2RSB −1.67316(4)

Numerics −1.2716(1) −1.673(1)

In the next few chapters we shall discuss a one-step replica-symmetry-breaking formalism that goes further in the same direction. The instability parameter kρ2 becomes smaller than in the RS case, but it is still larger than one. In this respect, the Ising spin glass is a particularly complicated model. It is expected that the free-entropy density and similar asymptotic quantities wil be predicted correctly only in the limit of full replica symmetry breaking. We refer to Chapter 22 for a further discussion of this point. Table 17.1 gives the one- and two-step RSB estimates of the ground state energy in the cases k = 2 and 4, and compares them with the best available numerical results.

Exercise 17.9 Consider the large-degree limit as in Exercise 17.7. Show that the stability condition kρ2 < 1 becomes, in this limit, β 2 Eh (1 − tanh2 (βh))2 < 1, where h is a Gaussian random variable with zero mean and a variance q satisfying eqn (17.42). This is nothing but the de Almeida–Thouless condition discussed in Chapter 8.

Let us conclude by warning the reader on one point. Although local stability is a necessary consistency check for the RS solution, it is by no means sufficient. Indeed, models with p-spin interactions and p ≥ 3 (such as the XORSAT model treated in the next chapter) often admit a locally stable ‘RS solution’ that is nevertheless incorrect.

Notes Ising ferromagnets on random graphs have appeared in several papers (e.g. Johnston and Plech´ ac, 1998; Dorogotsev et al., 2002; Leone et al., 2004). The application of belief propagation to this model was considered by Looij and Kappen (2005). The rigorous cavity analysis of the Ising ferromagnet presented in this chapter can be found in Dembo and Montanari (2008c), which also proves the exponential convergence of BP. The problem of spin glasses on random graphs was first studied using the replica method by Viana and Bray (1985), who worked out the paramagnetic solution and located the transition. The RS cavity solution that we have described here was first discussed for Erd¨ os–Renyi graphs by Mézard and Parisi (1987); see also Kanter and Sompolinsky (1987). Expansions around the critical point were developed by Goldschmidt and De Dominicis (1990). The related (but different) problem of a spin glass on a regular tree was introduced by Thouless (1986) and further studied by Chayes et al. (1986) and Carlson et al. (1990). One-step replica symmetry breaking for spin glasses on sparse random graphs was studied by Wong and Sherrington (1988), Goldschmidt and Lai (1990) and Monasson

¼¼


(1998) using replicas, and by Goldschmidt (1991) and Mézard and Parisi (2001, 2003) using the cavity method. The 2RSB ground state energy given in Table 17.1 was obtained by Montanari (2003), and the numerical values are from Boettcher (2003).

Part V Long-range correlations


18 Linear equations with Boolean variables Solving a system of linear equations over a finite field F is arguably one of the most fundamental operations in mathematics. Several algorithms have been devised to accomplish such a task in polynomial time. The best known is Gaussian elimination, which has O(N 3 ) complexity (here N is number of variables in the linear system, and we assume the number of equations to be M = Θ(N )). As a matter of fact, one can improve over Gaussian elimination, and the best existing algorithm for general systems has complexity O(N 2.376... ). Faster methods also exist for special classes of instances. The set of solutions of a linear system is an affine subspace of FN . Despite this apparent simplicity, the geometry of affine or linear subspaces of FN can be surprisingly rich. This observation is systematically exploited in coding theory. Linear codes are just linear spaces over finite fields. Nevertheless, they are known to achieve the Shannon capacity on memoryless symmetric channels, and their structure is far from trivial, as we have already seen in Chapter 11. From a different point of view, linear systems are a particular example of constraint satisfaction problems. We can associate with a linear system a decision problem (establishing whether it has a solution), a counting problem (counting the number of solutions), and an optimization problem (minimizing the number of violated equations). While the first two are polynomial, the latter is known to be NP-hard. In this chapter, we consider a specific ensemble of random linear systems over Z2 (the field of integers modulo 2), and discuss the structure of its set of solutions. The definition of this ensemble is motivated mainly by its analogy with other random constraint satisfaction problems, which also explains the name ‘XOR-satisfiability’ (or ‘XORSAT’). In the next section, we provide the precise definition of the XORSAT ensemble and recall a few elementary properties of linear algebra. We also introduce one of the main objects of study of this chapter: the SAT–UNSAT threshold. Section 18.2 takes a detour into the properties of belief propagation for XORSAT. These are shown to be related to the correlation structure of the uniform measure over solutions and, in Section 18.3, to the appearance of a 2-core in the associated factor graph. Sections 18.4 and 18.5 build on these results to compute the SAT–UNSAT threshold and characterize the structure of the solution space. While many results can be derived rigorously, XORSAT offers an ideal playground for understanding the non-rigorous cavity method, which will be developed further in the following chapters. This is the subject of Section 18.6.

Linear equations with Boolean variables

18.1 18.1.1

Definitions and general remarks Linear systems

Let H be an M × N matrix with entries Hai ∈ {0, 1}, a ∈ {1, . . . , M }, i ∈ {1, . . . , N }, and let b be an M -component vector with binary entries ba ∈ {0, 1}. An instance of the XORSAT problem is given by a pair (H, b). The decision problem requires us to find an N -component vector x with binary entries xi ∈ {0, 1} which solves the linear system Hx = b mod 2, or to show that the system has no solution. The name ‘XORSAT’ comes from the fact that the sum modulo 2 is equivalent to the ‘exclusive OR’ operation: the problem is whether there exists an assignment of the variables x which satisfies a set of XOR clauses. We shall thus say that an instance is SAT or UNSAT whenever the linear system has or does not have, respectively, a solution. We shall, furthermore, be interested in the set of solutions, denoted by S; in its size Z = |S|; and in the properties of the uniform measure over S. This is defined by µ(x) =

1 I( Hx = b Z

mod 2 ) =

M 1 ψa (x∂a ) , Z a=1

(18.1)

where ∂a = (ia (1), . . . , ia (K)) is the set of non-vanishing entries in the a-th row of H, and ψa (x∂a ) is the characteristic function for the a-th equation in the linear system (explicitly, ψa (x∂a ) = I(xi1 (a) ⊕ · · · ⊕ xiK (a) = ba ), where we denote the sum modulo 2 by ⊕ as usual). In the following, we shall omit the specification that operations are carried mod 2 when this is clear from the context. When H has a row weight p (i.e. each row has p non-vanishing entries), the problem is related to a p-spin glass model. Writing σi = 1 − 2xi and Ja = 1 − 2ba , we can associate with an XORSAT instance the energy function E(σ) =

M %

1 − Ja

a=1

& σj ,

(18.2)

j∈∂a

which counts (twice) the number of violated equations. This can be regarded as a pspin glass energy function with binary couplings. The XORSAT decision problem asks whether there exists a spin configuration σ with zero energy or, in physical jargon, whether the above energy function is ‘unfrustrated’. If there exists such a configuration, log Z is the ground state entropy of the model. A natural generalization is the MAX-XORSAT problem. This requires us to find a configuration which maximizes the number of satisfied equations, i.e. minimizes E(σ). In the following, we shall use the language of XORSAT but, of course, all statements have their direct counterpart in the context of p-spin glasses. Let us recall a few well-known facts of linear algebra that will be useful in the following: (i) The image of H is a vector space of dimension rank(H) (rank(H) is the number of independent rows in H); the kernel of H (the set S0 of x which solves the homogeneous system Hx = 0) is a vector space of dimension N − rank(H). (ii) As a consequence, if M ≤ N and H has rank M (all of its rows are independent), then the linear system Hx = b has a solution for any choice of b.

Definitions and general remarks

(iii) Conversely, if rank(H) < M , then the linear system has a solution if and only if b is in the image of H. If the linear system has at least one solution x∗ , then the set of solutions S is an affine space of dimension N − rank(H): we have S = x∗ + S0 and Z = 2N −rank(H) . We shall denote by µ0 ( · ) the uniform measure over the set S0 of solutions of the homogeneous linear system: µ0 (x) =

1 I( Hx = 0 Z0

mod 2 ) =

M 1 0 ψ (x ) , Z0 a=1 a ∂a

(18.3)

where ψa0 is given by the same expression as for ψa but with ba = 0. Note that µ0 is always well defined as a probability distribution, because the homogeneous system has at least the solution x = 0, whereas µ is well defined only for SAT instances. The linear structure has several important consequences: • If y is a solution of the inhomogeneous system, and if x is a uniformly random solution of the homogeneous linear system (with distribution µ0 ), then x = x ⊕ y is a uniformly random solution of the inhomogeneous system (its probability distribution is µ). • Under the measure µ0 , there exist only two sorts of variables xi , those which are ‘frozen to 0’, (i.e. take the value 0 in all of the solutions) and those which are ‘free’ (they take the value 0 in one half of the solutions and 1 in the other half). Under the measure µ (when it exists), a bit can be frozen to 0, frozen to 1, or free. These facts are proved in the next exercise.

Exercise 18.1 Let f : {0, 1}N → {0, 1} be a linear function (explicitly, f (x) is the sum of a subset xi(1) , . . . , xi(n) of the bits, mod 2). (a) If x is drawn from the distribution µ0 , f (x) becomes a random variable taking values in {0, 1}. Show that if there exists a configuration y with µ0 (y) > 0 and f (y) = 1, then P{f (x) = 0} = P{f (x) = 1} = 1/2. In the opposite case, P{f (x) = 0} = 1. (b) Suppose that there exists at least one solution to the system Hx = b, so that µ exists. Consider the random variable f (x) obtained by drawing x from the distribution µ. Show that one of the following three cases occurs: P{f (x) = 0} = 1, P{f (x) = 0} = 1/2, or P{f (x) = 0} = 0. These results apply in particular to the marginal of bit i, using f (x) = xi .

Exercise 18.2 Show that: (a) If the number of solutions of the homogeneous system is Z0 = 2N −M , then the inhomogeneous system is satisfiable (SAT), and has 2N −M solutions, for any b. (b) Conversely, if the number of solutions of the homogeneous system is Z0 > 2N −M , then the inhomogeneous system is SAT only for a fraction 2N −M /Z0 of the b’s.


x3 x2

x6

x5

x1 x4 Fig. 18.1 Factor graph for a 3-XORSAT instance with N = 6, M = 6 (see Exercise 18.3).

The distribution µ admits a natural factor graph representation: variable nodes are associated with variables and factor nodes with linear equations; see Fig. 18.1. Given an XORSAT formula F (i.e. a pair H, b), we denote the associated factor graph by G(F ). It is remarkable that one can identify subgraphs of G(F ) that serve as witnesses of the satisfiability or unsatisfiability of F . By this we mean that the existence of such subgraphs implies the satisfiability or unsatisfiability of F . The existence of a simple witness for unsatisfiability is intimately related to the polynomial nature of XORSAT. Such a witness is obtained as follows. Given a subset L of the clauses, draw the factor graph including all of the clauses in L, all of the adjacent variable nodes, and the edges between them. If this subgraph has even degree at each of the variable nodes, and if ⊕a∈L ba = 1, then L is a witness for unsatisfiability. Such a subgraph is sometimes called a frustrated hyperloop (in analogy with the frustrated loops that appear in spin glasses, where function nodes have degree 2).

Exercise 18.3 Consider a 3-XORSAT instance defined through the 6 × 6 matrix 2 6 6 6 H=6 6 4

0 1 0 1 0 1

1 0 1 0 1 0

0 0 0 1 0 1

1 1 0 0 1 0

1 0 1 0 0 1

0 1 1 1 1 0

3 7 7 7 7 . 7 5

(18.4)

(a) Compute rank(H) and find the two solutions of the homogeneous linear system. (b) Show that the linear system Hx = b has a solution if and only if b1 ⊕ b4 ⊕ b5 ⊕ b6 = 0. How many solutions does it have in this case?

Definitions and general remarks

(c) Consider the factor graph associated with this linear system, (see Fig. 18.1). Show that each solution of the homogeneous system must correspond to a subset U of variable nodes with the following property: the subgraph induced by U and including all of the adjacent function nodes has even degree at the function nodes. Find one subgraph with this property.

18.1.2

Random XORSAT

The random K-XORSAT ensemble is defined by taking b uniformly at random in {0, 1}M , and H uniformly at random from the N × M matrices with entries in {0, 1} which have exactly K non-vanishing elements per row. Each equation thus involves N K-tuples, and the resulting factor K distinct variables chosen uniformly from the K graph is distributed according to the ensemble GN (K, M ). N possible rows A slightly different ensemble is defined by including each of the K N with K non-zero entries independently with probability p = N α/ K . The corresponding factor graph is then distributed according to the ensemble GN (K, α). Given the relation between homogeneous and inhomogeneous systems described above, it is quite natural to introduce an ensemble of homogeneous linear systems. This is defined by taking H to be distributed as above, but with b = 0. Since a homogeneous linear system always has at least one solution, this ensemble is sometimes referred to as SAT K-XORSAT or, in its spin interpretation, as the ferromagnetic K-spin model. Given a K-XORSAT formula F , we shall denote by F0 the formula corresponding to the homogeneous system. We are interested in the limit of large systems N , M → ∞ with α = M/N fixed. By applying Friedgut’s theorem (see Section 10.5), it is possible to show that, for K ≥ 3, the probability for a random formula F to be SAT has a sharp threshold. More precisely, (N ) (N ) (N ) there exists an αs (K) such that for α > (1 + δ)αs (K) or α < (1 − δ)αs (K), P{F is SAT} → 0 or P{F is SAT} → 1, respectively, as N → ∞. (N ) A moment of thought reveals that αs (K) = Θ(1). Let us give two simple bounds to convince the reader of this statement. Upper bound. The relation between the homogeneous system and the original linear system derived in Exercise 18.2 implies that P{F is SAT} = 2N −M E{1/Z0 }. As Z0 ≥ (N ) 1, we get P{F is SAT} ≤ 2−N (α−1) and therefore αs (K) ≤ 1. Lower bound. For α < 1/K(K − 1), the factor graph associated with F is formed, with high probability, by finite trees and unicyclic components. This corresponds to the matrix H being decomposable into blocks, each one corresponding to a connected component. The reader can show that, for K ≥ 3, both a tree formula and a unicyclic component correspond to a linear system of full rank. Since each block has full rank, (N ) H has full rank as well. Therefore αs (K) ≥ 1/K(K − 1).


Exercise 18.4 There is no sharp threshold for K = 2. (a) Let c(G) be the cyclic number of the factor graph G (the number of edges, minus the number of vertices, plus the number of connected components) of a random 2-XORSAT formula. Show that P{F is SAT} = E 2−c(G) . (b) Argue that this implies that P{F is SAT} is bounded away from 1 for any α > 0. (c) Show that P{F is SAT} is bounded away from 0 for any α < 1/2. [Hint: Remember the geometrical properties of G discussed in Sections 9.3.2 and 9.4.]

(N )

In the next sections we shall show that αs (K) has a limit αc (K) and compute it explicitly. Before delving into this, it is instructive to derive two improved bounds.

(N )

Exercise 18.5 In order to obtain a better upper bound on αs

(K), proceed as follows:

(a) Assume that, for any α, Z0 ≥ 2 with probability larger than some ε > 0 at large (N ) N . Show that αs (K) ≤ α∗ (K), where α∗ (K) is the smallest value of α such that 1 − α − fK (α) ≤ 0. N fK (α)

(b) Show that the above assumption holds with fK (α) = e−Kα , and that this yields α∗ (3) ≈ 0.941. What is the asymptotic behaviour of α∗ (K) for large K? How can you improve the exponent fK (α)?

(N )

Exercise 18.6 A better lower bound on αs (K) can be obtained through a first-moment calculation. In order to simplify the calculations, we consider here a modified ensemble in which the K variables entering into each equation are chosen independently and uniformly at random (they do not need to be distinct). The scrupulous reader can check at the end that returning to the original ensemble needs only small changes. (a) Show that for a positive random variable Z, (EZ)(E[1/Z]) ≥ 1. Deduce that P{F is SAT} ≥ 2N −M /E ZF0 . (b) Prove that E ZF 0 =

N X w=0

N w

!"

1 2

„

2w 1+ 1− N

«K !#M .

(18.5)

´˜ ˆ ` (c) Let gK (x) = H(x)+α log 12 1 + (1 − 2x)K , and define α∗ (K) to be the largest value (N ) of α such that the maximum of gK (x) is achieved at x = 1/2. Show that αs (K) ≥ α∗ (K). For K = 3, we get α∗ (3) ≈ 0.889.

Belief propagation

18.2 18.2.1

Belief propagation BP messages and density evolution

Equation (18.1) provides a representation of the uniform measure over solutions of an XORSAT instance as a graphical model. This suggests that we should apply messagepassing techniques. We shall describe belief propagation here and analyse its behaviour. (N ) While this may seem at first sight a detour from the objective of computing αs (K), it will instead provide some important insight. Let us assume that the linear system Hx = b admits at least one solution, so that the model (18.1) is well defined. We shall first study the homogeneous version Hx = 0, i.e. the measure µ0 , and then pass to µ. Applying the general definitions of Chapter 14, the BP update equations (14.14) and (14.15) for the homogeneous problem read (t) (t) (t+1) (t) ν b→i (xi ) , ν a→i (xi ) ∼ ψa0 (x∂a ) νj→a (xj ) . νi→a (xi ) ∼ = = b∈∂i\a

x∂a\i

j∈∂a\i

(18.6) These equations can be considerably simplified using the linear structure. We have seen that under µ0 , there are two types of variables, those ‘frozen to 0’ (i.e. equal to 0 in all solutions) and those which are ‘free’ (equally likely to be 0 or 1). BP aims at determining whether any single bit belongs to one class or the other. Now consider BP messages, which are also distributions over {0, 1}. Suppose that at time t = 0 they also take one of two possible values that we denote by ∗ (corresponding to the uniform distribution) and 0 (distribution entirely supported on 0). It is not hard to show that this remains true at all subsequent times. The BP update equations (18.6) simplify under this initialization (they reduce to the erasure decoder of Sect. 15.3): • At a variable node, the outgoing message is 0 unless all the incoming messages are ∗. • At a function node, the outgoing message is ∗ unless all the incoming messages are 0. (The message coming out of a degree-1 variable node is always ∗.) These rules preserve a natural partial ordering. Given two sets of messages ν = νi→a }, let us say that ν (t) ! ν(t) if, for each directed edge i → a where {νi→a }, ν = { (t) (t) the message νi→a = 0, then νi→a = 0 as well. It follows immediately from the update rules that if at some time t the messages are ordered as ν (t) ! ν(t) , then this order is preserved at all later times: ν (s) ! ν(s) for all s > t. This partial ordering suggests that we should pay special attention to the two (0) (0) ‘extremal’ initial conditions, namely νi→a = ∗ for all directed edges i → a or νi→a = 0 for all i → a. The fraction of edges Qt that carry a message 0 at time t is a deterministic quantity in the N → ∞ limit. It satisfies the recursion }, Qt+1 = 1 − exp{−KαQK−1 t

(18.7)

with Q0 = 1 for the ‘0’ initial condition and Q0 = 0 for the ‘∗’ initial condition. The density evolution recursion (18.7) is represented pictorially in Fig. 18.2.

½¼

Linear equations with Boolean variables 1

1 α>αd(k)

0.8

0.8 α=αd(k)

0.6 Qt

F(Q)

0.6 α 0 for α > αd (K). Here Q > 0 is the largest positive solution of Q = 1−exp{−Kα QK−1 }. The critical value αd (K) of the density of equations α = M/N separating these two regimes is given by 7 K−1 8 . (18.8) αd (K) = sup α such that ∀x ∈]0, 1] : x < 1 − e−Kα x We get, for instance, αd (K) ≈ 0.818469, 0.772280, and 0.701780 for K = 3, 4, and 5, respectively, and αd (K) = log K/K[1 + o(1)] as K → ∞. We have therefore found two regimes for the homogeneous random XORSAT problem in the large-N limit. For α < αd (K), there is a unique BP fixed point with all messages1 equal to ∗. The BP prediction for single-bit marginals that corresponds to this fixed point is νi (xi = 0) = νi (xi = 1) = 1/2. For α > αd (K), there exists more than one BP fixed point. We have found two of them: the all-∗ one, and one with a density of ∗’s equal to Q. Other fixed points of the inhomogeneous problem can be constructed as follows for α ∈ ]αd (K), αs (K)[. Let x(∗) be a solution of the inhomogeneous problem, and let ν, ν be a BP fixed point in the homogeneous case. Then the messages ν (∗) , ν (∗) , defined by (∗)

(∗)

νj→a (xj = 0) = νj→a (xj = 1) = 1/2

if νj→a = ∗,

(∗) νj→a (xj )

if νj→a = 0,

= I(xj =

(∗) xj )

(18.9)

(and similarly for ν (∗) ) are a BP fixed point for the inhomogeneous problem. For α < αd (K), the inhomogeneous problem admits, with high probability, a unique BP fixed point. This is a consequence of the result obtained in the following exercise: 1 While a vanishing fraction of messages ν i→a = 0 is not excluded by our argument, it can be ruled out by a slightly lengthier calculation.

Belief propagation

0

½½

0 0

xi 0

0

t 0 0

0

Fig. 18.3 Factor graph for a 3-XORSAT instance with the depth t = 1 neighbourhood of vertex i, Bi,t (G), indicated. Fixing all of the variables outside Bi,t (G) to 0 does not imply that xi must be 0 in order to satisfy the homogeneous linear system.

Exercise 18.7 Consider a BP fixed point ν (∗) , νb(∗) for the inhomogeneous problem, and (∗) (∗) assume all the messages to be of one of three types: νj→a (xj = 0) = 1, νj→a (xj = 0) = 1/2, (∗)

and νj→a (xj = 0) = 0. Assume furthermore that the messages are not ‘contradictory,’ i.e. (∗)

(∗)

that there exists no variable node i such that νba→i (xi = 0) = 1 and νbb→i (xi = 0) = 0. Construct a non-trivial BP fixed point for the homogeneous problem.

18.2.2

Correlation decay

The BP prediction is that for α < αd (K), the marginal distribution of any bit xi is uniform under either of the measures µ0 or µ. The fact that the BP estimates do not depend on the initialization is an indication that the prediction is correct. Let us prove that this is indeed the case. To be definite, we consider the homogeneous problem (i.e. µ0 ). The inhomogeneous case follows, using the general remarks in Section 18.1.1. We start from an alternative interpretation of Qt . Let i ∈ {1, . . . , N } be a uniformly random variable index, and consider the ball of radius t around i in the factor graph G, Bi,t (G) (see Fig. 18.3). Set all of the variables xj outside this ball to xj = 0, and (N ) let Qt be the probability that, under this condition, all of the solutions of the linear system Hx = 0 have xi = 0. The convergence of Bi,t (G) to the tree model T(K, α) (N ) discussed in Section 9.5 then implies that, for any given t, limN →∞ Qt = Qt . It also determines the initial condition to be Q0 = 1. Consider now the marginal distribution µ0 (xi ). If xi = 0 in all of the solutions of Hx = 0, then, a fortiori, xi = 0 in all of the solutions that fulfil the additional (N ) condition xj = 0 for j ∈ Bi,t (G). Therefore we have P {µ0 (xi = 0) = 1} ≤ Qt . By taking the N → ∞ limit, we get

½¾

Linear equations with Boolean variables 0000000000000000000 1111111111111111111 1111111111111111111 0000000000000000000 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 i 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111

x

xi

t

t

x(2)

x(1)

Fig. 18.4 A thought experiment: fix variables ‘far’ from i to two different assignments and check their influence on xi . For α < αd , there is no influence. (N )

lim P {µ0 (xi = 0) = 1} ≤ lim Qt

N →∞

N →∞

= Qt .

(18.10)

Letting t → ∞ and noticing that the left-hand side does not depend on t, we get P {µ0 (xi = 0) = 1} → 0 as N → ∞. In other words, all but a vanishing fraction of the bits are free for α < αd (K). The number Qt also has another interpretation, which generalizes to the inhomogeneous problem. We choose a solution x(∗) of the homogeneous linear system and, instead of fixing the variables outside the ball of radius t to 0, we fix them to (∗) (N ) (∗) is the probability that xi = xi , under this xj = xj , j ∈ Bi,t (G). Then Qt condition. The same argument holds in the inhomogeneous problem, with the measure µ: if x(∗) is a solution of Hx = b and we fix the variables outside Bi,t (G) to (∗) (∗) (N ) xj = xj , the probability that xi = xi under this condition is again Qt . The fact that limt→∞ Qt = 0 when α < αd (K) thus means that a spin decorrelates from the whole set of variables at distance larger than t, when t is large. This formulation of the correlation decay is rather specific to XORSAT, because it relies on the dichotomous nature of this problem: either the ‘far away’ variables completely determine xi , or they have no influence on it and it is uniformly random. A more generic formulation of the correlation decay, which generalizes to other problems which do not have this dichotomy property, consists in comparing two different choices x(1) , x(2) of the reference solution (see Fig. 18.4). For α < αd (K), the correlations decay even in the worst case: # $ lim E

N →∞

(1)

(2)

sup |µ(xi |x∼i,t ) − µ(xi |x∼i,t )|

= Qt → 0

(18.11)

x(1) ,x(2)

as t → ∞. In Chapter 22, we shall discuss weaker (non-worst-case) definitions of correlation decay, and their relation to phase transitions.

18.3 18.3.1

Core percolation and BP The 2-core and peeling

What happens for α > αd (K)? A first hint is provided by the instance in Fig. 18.1. In (t) this case, the configuration of messages νi→a = 0 on all directed edges i → a is a fixed

Core percolation and BP

½¿

point of the BP update for the homogeneous system. A moment of thought shows that this happens because G has the property that each variable node has degree at least 2. We shall now see that, for α > αd (K), G has with high probability a subgraph (called the 2-core) with the same property. We have already encountered similar structures in Section 15.3, where we identified them as being responsible for errors in the iterative decoding of LDPC codes over an erasure channel. Let us recall the relevant points2 from that discussion. Given a factor graph G, a stopping set is a subset of the function nodes such that all of the variable nodes have a degree greater than or equal to 2 in the induced subgraph. The 2-core is the largest stopping set. It is unique and can be found by the peeling algorithm, which amounts to iterating the following procedure: find a variable node of degree 0 or 1 (a ‘leaf’), and erase it together with the factor node adjacent to it, if there is one. The resulting subgraph, the 2-core, will be denoted as K2 (G). The peeling algorithm is of direct use for solving the linear system: if a variable has degree 1, the unique equation where it appears allows one to express it in terms of other variables. It can thus be eliminated from the problem. The 2-core of G is the factor graph associated with the linear system obtained by iterating this procedure, which we shall refer to as the ‘core system’. The original system has a solution if and only if the core does. We shall refer to solutions of the core system as core solutions. 18.3.2

Clusters

Core solutions play an important role, as the set of solutions can be partitioned according to their core values. Given an assignment x, we denote by π∗ (x) its projection onto the core, i.e. the vector of those entries in x that corresponds to vertices in the core. Suppose that the factor graph has a non-trivial 2-core, and let x(∗) be a core solution. We define the cluster associated with x(∗) as the set of solutions to the linear system such that π∗ (x) = x(∗) (the reason for the name ‘cluster’ will become clear in Section 18.5). If the core of G is empty, we shall adopt the convention that the entire set of solutions forms a unique cluster. Given a solution x(∗) of the core linear system, we shall denote the corresponding cluster by S(x(∗) ). One can obtain the solutions in S(x(∗) ) by running the peeling algorithm in the reverse direction, starting from x(∗) . In this process, one finds variables which are uniquely determined by x(∗) ; they form what is called the ‘backbone’ of the graph. More precisely, we define the backbone B(G) as the subgraph of G that is obtained by augmenting K2 (G) as follows. Set B0 (G) = K2 (G). For any t ≥ 0, pick a function node a which is not in Bt (G) and which has at least K − 1 of its neighbouring variable nodes in Bt (G), and build Bt+1 (G) by adding a (and its neighbouring variables) to Bt (G). If no such function node exists, set B(G) = Bt (G) and halt the procedure. This definition of B(G) does not depend on the order in which function nodes are added. The backbone contains the 2-core, and is such that any two solutions of the linear system which belong to the same cluster coincide on the backbone. We have thus found that the variables in a linear system naturally divide into three possible types: the variables in the 2-core K2 (G), those in B(G) \ K2 (G) which are not 2 Note that the structure causing decoding errors was the 2-core of the dual factor graph that is obtained by exchanging variable and function nodes.


b

4

3

c

2 a

1

Fig. 18.5 The factor graph of an XORSAT problem, its core (enclosed by the dash–dotted line), and its backbone (enclosed by the dashed line).

in the core but are fixed by the core solution, and the variables which are not uniquely determined by x(∗) . This distinction is based on the geometry of the factor graph, i.e. it depends only the matrix H, and not on the value of the right-hand side b in the linear system. We shall now see how BP finds these structures. 18.3.3

Core, backbone, and belief propagation

Consider the homogeneous linear system Hx = 0, and suppose that the BP algorithm (0) is run with the initial condition νi→a = 0. Denote by νi→a , ν a→i the fixed point reached by BP (with the measure µ0 ) under this initialization (the reader is invited to show that such a fixed point is indeed reached after a number of iterations at most equal to the number of messages). The fixed-point messages νi→a , ν a→i can be exploited to find the 2-core K2 (G), using the following properties (which can be proved by induction over t): (i) νi→a = ν a→i = 0 for each edge (i, a) in K2 (G); (ii) a variable i belongs to the core K2 (G) if and only if it receives messages ν a→i = 0 from at least two of the neighbouring function nodes a ∈ ∂i; and (iii) if a function node a ∈ {1, . . . , M } has νi→a = 0 for all of the neighbouring variable nodes i ∈ ∂a, then a ∈ K2 (G). The fixed-point BP messages also contain information about the backbone: a variable i belongs to the backbone B(G) if and only if it receives at least one message ν a→i = 0 from its neighbouring function nodes a ∈ ∂i.

Exercise 18.8 Consider an XORSAT problem described by the factor graph in Fig. 18.5. (a) Using the peeling and backbone construction algorithms, check that the core and backbone are those described in the caption. (b) Compute the BP messages found for the homogeneous problem as a fixed point of the BP iteration starting from the all-0 configuration. Check the core and backbone that you obtain from these messages.

The SAT–UNSAT threshold in random XORSAT

(c) Consider the general inhomogeneous linear system with the same factor graph. Show that there exist two solutions to the core system: x1 = 0, x2 = bb ⊕ bc , x3 = ba ⊕ bb ⊕ bc , x4 = ba ⊕ bb and x1 = 0, x2 = bb ⊕ bc ⊕ 1, x3 = ba ⊕ bb ⊕ bc , x4 = ba ⊕ bb ⊕ 1. Identify the two clusters of solutions.

18.4


We shall now see how a sharp characterization of the core size in random linear systems provides the clue to the determination of the satisfiability threshold. Remarkably, this characterization can again be achieved through an analysis of BP. 18.4.1

The size of the core

Consider an homogeneous linear system over N variables drawn from the random (t) K-XORSAT ensemble, and let {νi→a } denote the BP messages obtained from the (0) initialization νi→a = 0. The density evolution analysis of Section 18.2.1 implies that the fraction of edges carrying a message 0 at time t (which we called Qt ) satisfies the recursion equation (18.7). This recursion holds for any given t asymptotically as N → ∞. (t) It follows from the same analysis that, in the large-N limit, the messages ν a→i (t)

t ≡ QK−1 entering a variable node i are i.i.d. with P{

νa→i = 0} = Q . Let us assume t for a moment that the limits t → ∞ and N → ∞ can be exchanged without much harm. This means that the fixed-point messages ν a→i entering a variable node i are

≡ QK−1 , where Q is the largest solution asymptotically i.i.d. with P{

νa→i = 0} = Q of the fixed-point equation b

= QK−1 . Q

Q = 1 − e−KαQ ,

(18.12)

The number of incoming messages with ν a→i = 0 therefore converges to a Poisson

The expected number of variable nodes in the core random variable with mean KαQ. will be E|K2 (G)| = N V (α, K) + o(N ), where V (α, K) is the probability that such a Poisson random variable is greater than or equal to 2; that is, b

b

e−KαQ . V (α, K) = 1 − e−KαQ − KαQ

(18.13)

In Fig. 18.6, we plot V (α) as a function of α. For α < αd (K), the peeling algorithm erases the whole graph, and there is no core. The size of the core jumps to some finite value at αd (K), and when α → ∞ the core is the full graph. Is K2 (G) a random factor graph or does it have any particular structure? By construction, it cannot contain variable nodes of degree zero or one. Its expected degree profile (the expected fraction of nodes of any given degree) is asymptotically

l is the probability that a Poisson random variable of parameter

≡ {Λ

l }, where Λ Λ

conditioned to be at least 2, is equal to l. Explicitly, Λ

0 = Λ

1 = 0, and KαQ,

l = Λ

1

eKαQb

1

l (KαQ)

− 1 − KαQ l!

for l ≥ 2.

(18.14)

Somewhat surprisingly, K2 (G) does not have any more structure than that determined by its degree profile. This fact is stated more formally in the following theorem.


0.8 0.7 0.6

V (α)

0.5 0.4

C(α)

0.3 0.2

Σ(α)

0.1 0.0 -0.1 0.70

αs

αd 0.75

0.80

0.85

0.90

0.95

1.00

α Fig. 18.6 The core of the random 3-XORSAT formulae contains N V (α) variables and N C(α) equations. These numbers are plotted versus the number of equations per variable of the original formula, α. The number of solutions to the XORSAT linear system is Σ(α) = V (α) − C(α). The core appears for α ≥ αd , and the system becomes UNSAT for α > αs , where αs is determined by Σ(αs ) = 0.

Theorem 18.1 Consider a factor graph G from the ensemble GN (K, N α) with K ≥ 3. Then (i) K2 (G) = ∅ with high probability for α < αd (K). (ii) For α > αd (K), |K2 (G)| = N V (α, K) + o(N ) with high probability.

l − ε and Λ

l + ε with (iii) The fraction of vertices of degree l in K2 (G) is between Λ −Θ(N ) . probability greater than 1 − e (iv) Conditional on the number of variable nodes n = |K2 (G)|, the degree profile being

K2 (G) is distributed according to the ensemble Dn (Λ,

xK ). Λ,

We shall not provide a proof of this theorem. The main ideas have already been presented in the previous pages, except for one important mathematical point: how to exchange the limits N → ∞ and t → ∞. The basic idea is to run BP for a large but fixed number of steps t. At this point the resulting graph is ‘almost’ a 2-core, and one can show that a sequential peeling procedure stops in fewer than N ε steps. In Fig. 18.7, we compare the statement in this theorem with numerical simulations. The probability that G contains a 2-core Pcore (α) increases from 0 to 1 as α ranges from 0 to ∞, with a threshold that becomes sharper and sharper as the size N increases. The threshold behaviour can be accurately described using finite-size scaling. Setting α = αd (K) + β(K) z N −1/2 + δ(K) N −2/3 (with properly chosen β(K) and δ(K)), one can show that Pcore (α) approaches a K-independent non-trivial limit that depends smoothly on z.


1

1

0.8

0.8 100

100

0.6 Pcore

200 300

0.6 Pcore

200 300

0.4

400

0.4

400

600

600

0.2

0.2

0

0 0.6

0.7

0.8

α

0.9

1

-3

-2

-1

0

z

1

2

3

Fig. 18.7 The probability that a random graph from the ensemble GN (K, α) with K = 3 (or equivalently, the factor graph of a random 3-XORSAT formula) contains a 2 core. Left: the outcome of numerical simulations is compared with the asymptotic threshold αd (K). Right: scaling plot (see text).

18.4.2

The threshold

l , we can compute Knowing that the core is a random graph with degree distribution Λ the expected number of equations in the core. This is given by the number of vertices times their average degree, divided by K, which yields N C(α, K) + o(N ), where

− e−KαQb ) . C(α, K) = αQ(1

(18.15)

In Fig. 18.6, we plot C(α, K) versus α. If α < αd (K), there is no core. For α ∈ ]αd , αs [, the number of equations in the core is smaller than the number of variables V (α, K). Above αc , there are more equations than variables. A linear system has a solution if and only if the associated core problem has a solution. In a large random XORSAT instance, the core system involves approximately N C(α, K) equations between N V (α, K) variables. We shall show that these equations are, with high probability, linearly independent as long as C(α, K) < V (α, K), which implies the following result. Theorem 18.2. (XORSAT satisfiability threshold) For K ≥ 3, let

+ (K − 1)(1 − Q)) , Σ(K, α) = V (K, α) − C(K, α) = Q − αQ(1

(18.16)

are the largest solution of eqn (18.12). Let αs (K) = inf{α : Σ(K, α) < 0}. where Q, Q Consider a random K-XORSAT linear system with N variables and N α equations. The following results hold with a probability going to 1 in the large-N limit: (i) The system has a solution when α < αs (K). (ii) The system has no solution when α > αs (K). (iii) For α < αs (K), the number of solutions is 2N (1−α)+o(N ) , and the number of clusters is 2N Σ(K,α)+o(N ) . Note that the the last expression in eqn (18.16) can be obtained from eqns (18.13) and (18.15) using the fixed-point condition (18.12).


1 50 100 200

0.8

400

0.6

Psat 0.4

0.2

αs (3) 0 0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

α Fig. 18.8 The probability that a random 3-XORSAT formula with N variables and N α equations is SAT, estimated numerically by generating 103 –104 random instances.

K αd αs 0

αd

3 0.81847 0.91794

4 0.77228 0.97677

5 0.70178 0.99244

αs

Fig. 18.9 Left: a pictorial view of the phase transitions in random XORSAT systems. The satisfiability threshold is αs . In the ‘Easy-SAT’ phase, for α < αd , there is a single cluster of solutions. In the ‘Hard-SAT’ phase, for αd < α < αs , the solutions of the linear system are grouped into well-separated clusters. Right: the thresholds αd , αs for various values of K. At large K, one has αd (K) log K/K and αs (K) = 1 − e−K + O(e−2K ).

The predictions of this theorem are compared with numerical simulations in Fig. 18.8, and Fig. 18.9 summarizes the results for the thresholds for XORSAT. Proof We shall convey only the basic ideas of the proof here and refer to the literature for technical details. Let us start by proving (ii), namely that for α > αs (K), random XORSAT instances are with high probability UNSAT. This follows from a linear-algebra argument. Let H∗ denote the 0–1 matrix associated with the core, i.e. a matrix including those rows/columns such that the associated function/variable nodes belong to K2 (G). Notice that if a given row is included in H∗ then all the columns corresponding to nonzero entries of that row are also in H∗ . As a consequence, a necessary condition for the rows of H to be independent is that the rows of H∗ are independent. This is, in turn, impossible if the number of columns in H∗ is smaller than its number of rows. Quantitatively, one can show that M − rank(H) ≥ rows(H∗ ) − cols(H∗ ) (with the obvious meanings of rows( · ) and cols( · )). For large random XORSAT systems, Theo-


11 00 11 00

Fig. 18.10 Adding a function node involving a variable node of degree one. The corresponding linear equation is independent of the other equations.

rem 18.1 implies that rows(H∗ )−cols(H∗ ) = −N Σ(K, α)+o(N ) with high probability. According to our discussion in Section 18.1.1, among the 2M possible choices of the right-hand side vector b, only 2rank(H) are in the image of H and thus lead to a solvable system. In other words, conditional on H, the probability that random XORSAT is solvable is 2rank(H)−M . By the above argument this is, with high probability, smaller than 2N Σ(K,α)+o(N ) . Since Σ(K, α) < 0 for α > αs (K), it follows that the system is UNSAT with high probability. In order to show that a random system is satisfiable with high probability when α < αs (K), one has to prove the following facts: (i) if the core matrix H∗ has maximum rank, then H has maximum rank as well; (ii) if α < αs (K), then H∗ has maximum rank with high probability. As a by-product, the number of solutions is 2N −rank(H) = 2N −M . (i) The first step follows from the observation that G can be constructed from K2 (G) through an inverse peeling procedure. At each step, one adds a function node which involves at least a degree-one variable (see Fig. 18.10). Obviously this newly added equation is linearly independent of the previous ones, and therefore rank(H) = rank(H∗ ) + M − rows(H∗ ). (ii) Let n = cols(H∗ ) be the number of variable nodes and m = rows(H∗ ) the number of function nodes in the core K2 (G). Let us consider the homogeneous system on the core, H∗ x = 0, and denote by Z∗ the number of solutions to this system. We shall show that, with high probability, this number is equal to 2n−m . This means that the dimension of the kernel of H∗ is n − m and therefore H∗ has full rank. We know from linear algebra that Z∗ ≥ 2n−m . To prove the reverse inequality, we use a first-moment method. According to Theorem 18.1, the core is a uniformly random

+ o(1). factor graph with n = N V (K, α) + o(N ) variables and degree profile Λ = Λ We denote the expectation value with respect to this ensemble by E. We shall use a first-moment analysis below to show that, when α < αc (K), E {Z∗ } = 2n−m [1 + oN (1)] . −n+m

(18.17)

The Markov inequality P{Z∗ > 2 }≤2 E{Z∗ } then implies the bound. The surprise is that eqn (18.17) holds, and thus a simple first-moment estimate allows us to establish that H∗ has full rank. We saw in Exercise 18.6 that the same n−m

¾¼

Linear equations with Boolean variables 0.06

0.04

0.02

φ(ω) 0.00

-0.02

-0.04 0.0

0.1

0.2

0.3

0.4

ω Fig. 18.11 The exponential rate φ(ω) of the weight enumerator of the core of a random 3-XORSAT formula. From top to bottom, α = αd (3) ≈ 0.818469, 0.85, 0.88, 0.91, and 0.94 (recall that αs (3) ≈ 0.917935). Inset: blow-up of the small-ω region.

approach, when applied directly to the original linear system, fails above some α∗ (K) which is strictly smaller than αs (K). Reducing the original graph to its 2-core has drastically reduced the fluctuations of the number of solutions, thus allowing a successful application of the first-moment method. We now turn to the proof of eqn (18.17), and we shall limit ourselves to the computation of E{Z∗ } to the leading exponential order, when the core size and degree

and P (x) = xK . This probprofiles take their typical values n = N V (K, α), Λ = Λ, lem is equivalent to computing the expected number of codewords in the LDPC code defined by the core system, which we have already done in Section 11.2. The result takes the typical form 3 4 . sup φ(ω) . (18.18) E{Z∗ } = exp N ω∈[0,V (K,α)]

Here, φ(ω) is the exponential rate for the number of solutions with weight N ω. Adapting eqn (11.18) to the present case, we obtain the following parametric expression: φ(ω) = −ω log x − η(1 − e−η ) log(1 + yz) η ηl + e−η log(1 + xy l ) + (1 − e−η ) log qK (z) , l! K

(18.19)

l≥2

ω=

l≥2

e−η

η l xy l , l! 1 + xy l

(18.20)

∗ and qK (z) = [(1 + z)K + (1 − z)K ]/2, and y = y(x), z = z(x) are where η = KαQ the solution of l l−1 /(1 + xy l )] (1 + z)K−1 − (1 − z)K−1 l≥1 [η /l!] [xy , y= z= . (18.21) l l (1 + z)K−1 + (1 − z)K−1 l≥1 [η /l!] [1/(1 + xy )]

The Hard-SAT phase: Clusters of solutions

¾½

With a little work one can see that ω∗ = V (K, α)/2 is a local maximum of φ(ω), . with φ(ω∗ ) = Σ(K, α) log 2. As long as ω∗ is a global maximum, E{Z∗ |n, Λ} = . n−m . It turns out (seeFig. 18.11) that the only other local maxiexp{N φ(ω∗ )} = 2 . mum is at ω = 0, corresponding to φ(0) = 0. Therefore E{Z∗ |n, Λ} = 2n−m as long as φ(ω∗ ) = Σ(K, α) > 0, i.e. for any α < αs (K) Note that the actual proof of eqn (18.17) is more complicated, because it requires estimating the subexponential factors. Nevertheless, it can be carried out successfully.

18.5

The Hard-SAT phase: Clusters of solutions

In random XORSAT, the whole regime α < αs (K) is SAT. This means that, with high probability, there exist solutions to the random linear system, and the number of . solutions is in fact Z = eN (1−α) . Note that the number of solutions does not show any precursor of the SAT–UNSAT transition at αs (K) (recall that αs (K) < 1), nor does it carry any trace of the sudden appearence of a non-empty 2-core at αd (K). On the other hand, the threshold αd (K) separates two phases, which we shall call the Easy-SAT phase (for α < αd (K)) and the Hard-SAT phase (for α ∈ ]αd (K), αs (K)[). These two phases differ in the structure of the solution space, as well as in the behaviour of some simple algorithms. In the Easy-SAT phase there is no core, solutions can be found in (expected) linear time using the peeling algorithm, and they form a unique cluster. In the Hard-SAT phase, the factor graph has a large 2-core, and no algorithm is known that finds a solution in linear time. The solutions are partitioned into 2N Σ(K,α)+o(N ) clusters. Until now, we have used the name ‘cluster’ fairly arbitrarily, and it has only denoted a subset of solutions that coincide in the core. The next result shows that distinct clusters are ‘far apart’ in Hamming space. Proposition 18.3 In the Hard-SAT phase, there exists a δ(K, α) > 0 such that, with high probability, any two solutions in distinct clusters have a Hamming distance larger than N δ(K, α). Proof The proof follows from the computation of the weight enumerator exponent φ(ω) (see eqn (18.20) and Fig. 18.11). One can see that for any α > αd (K), φ (0) < 0, and, as a consequence, there exists a δ(K, α) > 0 such that φ(ω) < 0 for 0 < ω < δ(K, α). This implies that if x∗ , x∗ are two distinct solutions of the core linear system, then either d(x∗ , x∗ ) = o(N ) or d(x, x ) > N δ(K, α). It turns out that the first case can be excluded along the lines of the minimal-distance calculation of Section 11.2. Therefore, if x, x are two solutions belonging to distinct clusters, then d(x, x ) ≥ d(π∗ (x), π∗ (x )) ≥ N δ(K, α). This result suggests that we can regard clusters as ‘lumps’ of solutions well separated from each other. One aspect which has been conjectured, but not proved, concerns the assertion that clusters form ‘well-connected components’. By this we mean that any two solutions in a cluster can be joined by a sequence of other solutions, whereby two successive solutions in the sequence differ in at most sN variables, with sN = o(N ) (a reasonable expectation is sN = Θ(log N )).

¾¾

18.6


An alternative approach: The cavity method

The analysis of random XORSAT in the previous sections relied heavily on the linear structure of the problem, as well as on the very simple instance distribution. This section describes an alternative approach that is potentially generalizable to more complex situations. The price to be paid is that this second derivation relies on some assumptions about the structure of the solution space. The observation that our final results coincide with those obtained in the previous section gives some credibility to these assumptions. The starting point is the observation that BP correctly computes the marginals of µ( · ) (the uniform measure over the solution space) for α < αd (K), i.e. as long as the set of solutions forms a single cluster. We want to extend its domain of validity to α > αd (K). If we index the clusters by n ∈ {1, . . . , N }, the uniform measure µ( · ) can be decomposed into a convex combination of uniform measures over each single cluster: µ( · ) =

N

wn µn ( · ) .

(18.22)

n=1

Note that in the present case, wn = 1/N is independent of n and the measures µn ( · ) are obtained from each other via a translation, but this will not be true in more general situations. Consider an inhomogeneous XORSAT linear system, and denote by x(∗) one of its solutions in cluster n. The distribution µn has single-variable marginals µn (xi ) = (∗) I(xi = xi ) if node i belongs to the backbone, and µn (xi = 0) = µn (xi = 1) = 1/2 on the other nodes. In fact, we can associate with each solution x(∗) a fixed point of the BP equations. We have already described this in Section 18.2.1 (see eqn (18.9)). At this fixed point, (∗) messages take one of the following three values: νi→a (xi ) = I(xi = 0) (which we shall (∗) (∗) (∗) (∗) denote by νi→a = 0), νi→a (xi ) = I(xi = 1) (denoted by νi→a = 1), and νi→a (xi = 0) = (∗) (∗) νi→a (xi = 1) = 1/2 (denoted by νi→a = ∗). An analogous notation will be used for function-to-variable-node messages. The solution can be written most easily in terms of the latter, ⎧ (∗) ⎪ ⎨ 1 if xi = 1 and i, a ∈ B(G), (∗) (18.23) ν a→i = 0 if x(∗) = 0 and i, a ∈ B(G), i ⎪ ⎩ ∗ otherwise. (∗)

Note that these messages depend only on the value of xi on the backbone of G, and hence they depend on x(∗) only through the cluster that it belongs to. Conversely, for any two distinct clusters, the above definition gives two distinct fixed points. Because (n) (n) of this observation, we shall denote these fixed points by {νi→a , ν a→i }, where n is a cluster index.


¾¿

11 00 00 11 00 11 11 00 00 11

core 11frozen11 00 00 11 00 00 11 00 1111111 11 00 11 0000000 00 11 0000000 1111111 00 11 1111111 0000000 0000000 1111111 0000000 1111111 00 11 0000000 001111111 11 0000000 001111111 11 0000000 1111111 0000000 1111111 0000000 1111111 0000000 1111111

1

a

b

2

Fig. 18.12 Left: a set of BP messages associated with one cluster (cluster number n) of (n) solutions. An arrow along an edge means that the corresponding message (either νi→a or (n) νba→i ) takes a value in {0, 1}. The other messages are equal to ∗. Right: a small XORSAT instance. The core is the whole graph. In the homogeneous problem, there are two solutions, which form two clusters: x1 = x2 = 0 and x1 = x2 = 1. Besides the two corresponding BP fixed points described in Proposition 18.4, and the all-∗ fixed point, there exist other fixed points such as νba→1 = ν1→b = νbb→2 = ν2→a = 0, νba→2 = ν2→b = νbb→1 = ν1→a = ∗.

Let us recall the BP fixed-point condition: ∗ if ν b→i = ∗ for all b ∈ ∂i\a, νi→a = (18.24) otherwise, any ‘non-∗’ ν b→i ∗ if ∃j ∈ ∂a\i such that ν j→a = ∗, (18.25) ν a→i = otherwise. ba ⊕ νj1 →a ⊕ · · · ⊕ νjl →a Below, we shall denote these equations symbolically as νb→i } , νi→a = f{

ν a→i = ˆf{νj→a } .

(18.26)

Let us summarize our findings. Proposition 18.4 With each cluster n, we can associate a distinct fixed point of the (n) (n) (n) BP equations (18.25) {νi→a , ν a→i }, such that ν a→i ∈ {0, 1} if i, a are in the backbone (n) and ν a→i = ∗ otherwise. Note that the converse of this proposition is false: there may exist solutions to the BP equations which are not of the above type. One of them is the all-∗ solution. Non-trivial solutions exist as well, as shown in Fig. 18.12.


An introduction to the 1RSB cavity method in the general case will be presented in Chapter 19. Here we shall give a short informal preview in the special case of XORSAT: the reader will find a more formal presentation in the next chapter. The first two assumptions of the 1RSB cavity method can be summarized as follows (all statements are understood to hold with high probability). Assumption 1 In the case of a large random XORSAT instance, for each cluster ‘n’ of solutions, the BP solution ν (n) , ν (n) provides an accurate ‘local’ description of the measure µn ( · ). This means that, for instance, the one-point marginals are given by µn (xj ) ∼ = (n)

a→j (xj ) + o(1), but also that local marginals inside any finite cavity are well a∈∂j ν approximated by eqn (14.18). Assumption 2 For a large random XORSAT instance in the Hard-SAT phase, the number of clusters eN Σ is exponential in the number of variables. Further, the number of solutions of the BP equations (18.25) is, to the leading exponential order, the same as the number of clusters. In particular, it is the same as the number of solutions constructed in Proposition 18.4. (n)

A priori, one might have hoped to identify the set of messages {νi→a } for each cluster. The cavity method gives up this ambitious objective and aims instead to (n) compute the distribution of νi→a for any fixed edge i → a, when n is a cluster index drawn with distribution {wn }. We thus want to compute the quantities 3 3 4 4 (n) (n)

a→i (

Qi→a (ν) = P νi→a = ν , ν ) = P ν a→i = ν , Q (18.27) for ν, ν ∈ {0, 1, ∗}. Computing these probabilities rigorously is still a challenging task. In order to proceed, we make some assumptions about the joint distribution of the (n) messages νi→a when n is a random cluster index (chosen with probability wn ). The simplest idea would be to assume that messages on ‘distant’ edges are independent. For instance, let us consider the set of messages entering a given variable node i. Their only correlations are induced through BP equations along the loops to which i belongs. Since in random K-XORSAT formulae such loops have, with high probability, a length of order log N , one might think that messages coming into a given node are asymptotically independent. Unfortunately, this assumption is false.

a→i (1) > 0 for at least

a→i (0), Q The reason is easily understood if we assume that Q two of the function nodes a adjacent to a given variable node i. This would imply that, (n) (n) with positive probability, a randomly sampled cluster has νa→i = 0 and νb→i = 1. But there does not exist any such cluster, because in such a situation there is no consistent prescription for the marginal distribution of xi under µn ( · ). Our assumption will be that the next simplest thing happens: messages are independent conditional on the fact that they do not contradict each other. Assumption 3 Consider the Hard-SAT phase of a random XORSAT problem. Denote by i ∈ G a uniformly random node, denote by n a random cluster index with (n) distribution {wn }, and let be an integer ≥ 1. Then the messages {νj→b }, where


(j, b) are all of the edges at distance from i directed towards i, are asymptotically independent under the condition of being compatible.

Here ‘compatible’ means the following. Consider the linear system Hi, xi, = 0 for the neighbourhood of radius around node i. If this admits a solution under the boundary condition xj = νj→b for all of the boundary edges (j, b) on which {νj→b } ∈ {0, 1}, then the messages {νj→b } are said to be compatible. Given the messages νj→b at the boundary of a radius- neighbourhood, the BP equations (18.24) and (18.25) allow one to determine the messages inside this neighbourhood. Consider, in particular, two nested neighbourhoods at distances and + 1 from i. The inward messages on the boundary of the larger neighbourhood completely determines those on the boundary of the smaller neighbourhood. A little thought shows that if the messages on the outer boundary are distributed according to Assumption 3, then the distribution of the resulting messages on the inner boundary also satisfies the same assumption. Further, the distributions are consistent if and only if the following ‘survey propagation’ equations are satisfied by the one-message marginals: Qi→a (ν) ∼ =

b→i (

νb ) I(ν = f{

νb }) I({

νb }b∈∂i\a ∈ COMP) , Q

(18.28)

{b νb } b∈∂i\a

a→i (

ν) = Q

Qj→a (νj ) I(

ν = ˆf{νj }) .

(18.29)

{νj } j∈∂a\i

Here, {

νb } ∈ COMP if and only if the messages are compatible (i.e. they do not contain both a 0 and a 1). Since Assumptions 1, 2, and 3 above hold only with high probability, and asymptotically in the system size, the equalities in eqns (18.28) and (18.29) must also be interpreted as approximate. The equations should be satisfied within any given accuracy ε, with high probability as N → ∞.

Exercise 18.9 Show that eqns (18.28) and (18.29) can be written explicitly as Qi→a (0) ∼ =

Y

b b→i (0) + Q b b→i (∗)) − (Q

b∈∂i\a

Qi→a (1) ∼ =

Y

b b→i (∗) , Q

(18.30)

b b→i (∗) , Q

(18.31)

b∈∂i\a

b b→i (1) + Q b b→i (∗)) − (Q

b∈∂i\a

Qi→a (∗) ∼ =

Y

Y

Y

b∈∂i\a

b b→i (∗) , Q

b∈∂i\a

where the symbol ∼ = hides a global normalization constant, and

(18.32)


9 8 = < Y Y 1 b a→i (0) = (Qj→a (0) + Qj→a (1)) + (Qj→a (0) − Qj→a (1)) , Q ; 2 : j∈∂a\i

j∈∂a\i

9 8 = < Y Y 1 b a→i (1) = Q (Qj→a (0) + Qj→a (1)) − (Qj→a (0) − Qj→a (1)) , ; 2 : j∈∂a\i

b a→i (∗) = 1 − Q

Y

(18.33)

j∈∂a\i

(18.34) (Qj→a (0) + Qj→a (1)) .

(18.35)

j∈∂a\i

The final step of the 1RSB cavity method consists in looking for a solution of eqns (18.28) and (18.29). There are no rigorous results on the existence or number of such solutions. Further, since these equations are only approximate, approximate solutions should be considered as well. In the present case, a very simple (and somewhat degenerate) solution can be found that yields correct predictions for all of the quantities of interest. In this solution, the message distributions take one of two possible forms: on some edges, one has Qi→a (0) = Qi→a (1) = 1/2 (with some misuse of notation, we shall write Qi→a = 0 in this case), and on some other edges, Qi→a (∗) = 1 (we shall

a→i . A little algebra shows that then write Qi→a = ∗). Analogous forms hold for Q this is a solution if and only if the η’s satisfy

b→i = ∗ for all b ∈ ∂i\a, ∗ if Q (18.36) Qi→a = 0 otherwise,

j→a = ∗, if ∃j ∈ ∂a\i such that Q

a→i = ∗ (18.37) Q 0 otherwise. These equations are identical to the original BP equations for the homogeneous problem (this feature is very specific to XORSAT and will not generalize to more advanced applications of the method). However, the interpretation is now completely different. (n) On the edges where Qi→a = 0, the corresponding message νi→a depends on the cluster (n) (n) n; νi→a = 0 in half of the clusters, and νi→a == 1 in the other half. These edges are those inside the core, or in the backbone but directed ‘outward’ with respect to the core, as shown in Fig. 18.12. On the other edges, the message does not depend upon (n) the cluster, and νi→a = ∗ for all n’s. A concrete interpretation of these results is obtained if we consider the one-bit marginals µn (xi ) under the single-cluster measure. According to Assumption 1 above, (n) we have µn (xi = 0) = µn (xi = 1) = 1/2 if ν a→i = ∗ for all a ∈ ∂i. If, on the other (n) hand, ν a→i = 0 or 1 for at least one a ∈ ∂i, then µn (xi = 0) = 1 or µn (xi = 0) = 0, respectively. We thus recover the full solution discussed in the previous sections: inside a given cluster n, the variables in the backbone are completely frozen, either to 0 or to 1. The other variables have equal probability to be 0 or 1 under the measure µn .

Notes

The cavity approach allows one to compute the complexity Σ(K, α), as well as many other properties of the measure µ( · ). We shall see this in the next chapter.

Notes Random XORSAT formulae were first studied as a simple example of random satisfiability by Creignou and Daudé (1999). This work considered the case of ‘dense formulae’ where each clause includes O(N ) variables. In this case the SAT–UNSAT threshold is at α = 1. In the field of coding theory, this model had been characterized since the work by Elias (1955) (see Chapter 6). The case of sparse formulae was addressed using moment bounds by Creignou et al. (2003). The replica method was used by Ricci-Tersenghi et al. (2001), Franz et al. (2001a), and Franz et al. (2001b) to derive the clustering picture, determine the SAT–UNSAT threshold, and study the glassy properties of the clustered phase. The fact that, after reducing the linear system to its core, the first-moment method provides a sharp characterization of the SAT–UNSAT threshold was discovered independently by two groups: Cocco et al. (2003) and Mézard et al. (2003). The latter also discussed the application of the cavity method to the problem. The full second-moment calculation that completes the proof can be found for the case K = 3 in Dubois and Mandler (2002). Papers by Montanari and Semerjian (2005), Montanari and Semerjian (2006a), and Mora and Mézard (2006) were devoted to finer geometrical properties of the set of solutions of random K-XORSAT formulae. Despite these efforts, it remains to be proved that clusters of solutions are indeed ‘well connected’. Since the locations of various transitions are known rigorously, a natural question is to study the critical window. Finite-size scaling of the SAT–UNSAT transition was investigated numerically by Leone et al. (2001). A sharp characterization of finite-size scaling for the appearence of a 2-core, corresponding to the clustering transition, was achieved by Dembo and Montanari (2008a).


19 The 1RSB cavity method The effectiveness of belief propagation depends on one basic assumption: when a function node is pruned from the factor graph, the adjacent variables become weakly correlated with respect to the resulting distribution. This hypothesis may break down either because of the existence of small loops in the factor graph or because variables are correlated at large distances. In factor graphs with a locally tree-like structure, the second scenario is responsible for the failure of BP. The emergence of such long-range correlations is a signature of a phase transition separating a ‘weakly correlated’ and a ‘highly correlated’ phase. The latter is often characterized by the decomposition of the (Boltzmann) probability distribution into well-separated ‘lumps’ (pure Gibbs states). We considered a simple example of this phenomenon in our study of random XORSAT. A similar scenario holds in a variety of problems, from random graph colouring to random satisfiability and spin glasses. The reader should be warned that the structure and organization of the pure states in such systems is far from being fully understood. Furthermore, the connection between long-range correlations and pure-state decomposition is more subtle than is suggested by the above remarks. Despite these complications, physicists have developed a non-rigorous approach to deal with this phenomenon: the ‘one-step replica symmetry breaking’ (1RSB) cavity method. This method postulates a few properties of the pure-state decomposition, and, on this basis, allows one to derive a number of quantitative predictions (‘conjectures’ from a mathematical point of view). Examples include the satisfiability threshold for random K-SAT and for other random constraint satisfaction problems. The method is rich enough to allow some self-consistency checks of such assumptions. In several cases in which the 1RSB cavity method has passed this test, its predictions have been confirmed by rigorous arguments (and there is no case in which they have been falsified so far). These successes encourage the quest for a mathematical theory of Gibbs states on sparse random graphs. This chapter explains the 1RSB cavity method. It alternates between a general presentation and a concrete illustration of the XORSAT problem. We strongly encourage readers to read the previous chapter on XORSAT before the present one. This should help them to gain some intuition about the whole scenario. We start with a general description of the 1RSB glass phase, and the decomposition into pure states, in Section 19.1. Section 19.2 introduces an auxiliary constraint satisfaction problem in order to count the number of solutions of BP equations. The 1RSB analysis amounts to applying belief propagation to this auxiliary problem. One can then apply the methods of Chapter 14 (for instance, density evolution) to the auxiliary problem. Section 19.3 illustrates this approach with the XORSAT problem

¿¼

The 1RSB cavity method

and shows how the 1RSB cavity method recovers the rigorous results of the previous chapter. In Section 19.4, we show how the 1RSB formalism, which in general is rather complicated, simplifies considerably when the temperature of the auxiliary constraint satisfaction problem takes the value x = 1. Section 19.5 explains how to apply this formalism to optimization problems (leveraging the min-sum algorithm), leading to the survey propagation algorithm. Section 19.6 describes the physical intuition which underlies the whole method. The appendix in Section 19.7 contains some technical aspects of the survey propagation equations applied to XORSAT, and their statistical analysis.

19.1 19.1.1

Beyond BP: Many states Bethe measures

The main lesson of the previous chapters is that in many cases, the probability distribution specified by graphical models with a locally tree-like structure takes a relatively simple form, which we shall call a Bethe measure (or Bethe state). Let us first define precisely what we mean by this, before we proceed to discuss what kinds of other scenarios can be encountered. As in Chapter 14, we consider a factor graph G = (V, F, E), with variable nodes V = {1, . . . , N }, factor nodes F = {1, . . . , M }, and edges E. The joint probability distribution over the variables x = (x1 , . . . , xN ) ∈ X N takes the form µ(x) =

M 1 ψa (x∂a ) . Z a=1

(19.1)

Given a subset of variable nodes U ⊆ V (which we shall call a ‘cavity’), the induced subgraph GU = (U, FU , EU ) is defined as the factor graph that includes all of the factor nodes a such that ∂a ⊆ U , and the adjacent edges. We also write (i, a) ∈ ∂U νa→i } is a set of probability if i ∈ U and a ∈ F \ FU . Finally, a set of messages {

distributions over X , indexed by directed edges a → i in E with a ∈ F , i ∈ V . Definition 19.1. (Informal) The probability distribution µ is a Bethe measure (or Bethe state) if there exists a set of messages {

νa→i } such that, for ‘almost all’ of the ‘finite-size’ cavities U , the distribution µU ( · ) of the variables in U can be approximated as ψa (x∂a ) ν a→i (xi ) + err(xU ) , (19.2) µU (xU ) ∼ = a∈FU

(ia)∈∂U

where err(xU ) is a ‘small’ error term, and ∼ = denotes, as usual, equality up to a normalization. A formal definition should specify what is meant by ‘almost all’, ‘finite-size’ and ‘small’. This can be done by introducing a tolerance N (with N ↓ 0 as N → ∞) and a size LN (where LN is bounded as N → ∞). One then requires that some norm of err( · ) (e.g. an Lp norm) is smaller than N for a fraction larger than 1 − N of all possible

Beyond BP: Many states

b

b

j

a

¿½

a

i

j

i

Fig. 19.1 Two examples of cavities. The right-hand one is obtained by adding the extra function node a. The consistency of the Bethe measure in these two cavities implies the BP equation for νba→i ; see Exercise 19.1.

cavities U of size |U | < LN . The underlying intuition is that the measure µ( · ) is well approximated locally by the given set of messages. In the following, we shall follow the physicists’ habit of leaving implicit the various approximation errors. Notice that the above definition does not make use of the fact that µ factorizes as in eqn (19.1). It thus applies to any distribution over x = {xi : i ∈ V }. If µ( · ) is a Bethe measure with respect to the message set {

νa→i }, then the consistency of eqn (19.2) for different choices of U implies some non-trivial constraints on the messages. In particular, if the loops in the factor graph G are not too small (and under some technical conditions on the functions ψa ( · )), then the messages must be close to satisfying the BP equations. More precisely, we define a quasi-solution of the BP equations as a set of messages which satisfy almost all of the equations within some accuracy. The reader is invited to prove this statement in the exercise below.

Exercise 19.1 Assume that G = (V, F, E) has girth larger than 2, and that µ( · ) is a Bethe measure with respect to the message set {b νa→i }, where νba→i (xi ) > 0 for any (i, a) ∈ E, and ψa (x∂a ) > 0 for any a ∈ F . For U ⊆ V and (i, a) ∈ ∂U , define a new subset of variable nodes as W = U ∪ ∂a (see Fig. 19.1). Applying eqn (19.2) to the subsets of variables U and W , show that a message must satisfy (up to an error term of the same order as err( · )) νba→i (xi ) ∼ =

X x∂a\i

ψa (x∂a )

Y n Y j∈∂a\i

o νbb→j (xj ) .

(19.3)

b∈∂j\a

Show that these equations are equivalent to the BP equations Q (14.14) and (14.15). [Hint: Define, for k ∈ V , c ∈ F and (k, c) ∈ E, νk→c (xk ) ∼ = d∈∂k\c νbd→k (xk ).]

It would be pleasant if the converse was true, i.e. if each quasi-solution of the BP equations corresponded to a distinct Bethe measure. In fact, such a relation will be at the heart of the assumptions of the 1RSB method. However, one should keep in mind that this is not always true, as the following example shows.

¿¾


Example 19.2 Let G be a factor graph with the same degree K ≥ 3 at both factor and variable nodes. Consider binary variables (X = {0, 1}) and, for each a ∈ F , let ψa (xi1 (a) , . . . , xiK (a) ) = I(xi1 (a) ⊕ · · · ⊕ xiK (a) = 0) .

(19.4)

Given a perfect matching M ⊆ E, a solution of the BP equations can be constructed as follows. If (i, a) ∈ M, then let ν a→i (xi ) = I(xi = 0) and νi→a (0) = νi→a (1) = 1/2. If, on the other hand, (i, a) ∈ M, then let ν a→i (0) = ν a→i (1) = 1/2 and νi→a (0) = I(xi = 0). Check that this is a solution of the BP equations and that all the resulting local marginals coincide with the ones of the measure µ(x) ∼ = I(x = 0), independently of M. If we take, for instance, G to be a random regular graph with degree K ≥ 3, both at factor nodes and at variable nodes, then the number of perfect matchings of G is, with high probability, exponential in the number of nodes. Therefore we have constructed an exponential number of solutions of the BP equations that describe the same Bethe measure. 19.1.2

A few generic scenarios

Bethe measures are a conceptual tool for describing distributions of the form (19.1). Inspired by the study of glassy phases (see Section 12.3), statistical mechanics studies have singled out a few generic scenarios in this respect, that we informally describe below. RS (replica-symmetric). This is the simplest possible scenario: the distribution µ( · ) is a Bethe measure. A slightly more complicated situation (which we still place in the ‘replica-symmetric’ family) arises when µ( · ) decomposes into a finite set of Bethe measures related by ‘global symmetries’, as in the Ising ferromagnet discussed in Section 17.3. d1RSB (dynamic one-step replica symmetry breaking). There exist an exponentially large (in the system size N ) number of Bethe measures. The measure µ decomposes into a convex combination of these Bethe measures, i.e. wn µn (x) , (19.5) µ(x) = n

with weights wn exponentially small in N . Furthermore µ( · ) is itself a Bethe measure. s1RSB (static one-step replica symmetry breaking). As in the d1RSB case, there exist an exponential number of Bethe measures, and µ decomposes into a convex combination of such states. However, a finite number of the weights wn are of order 1 as N → ∞, and (unlike in the previous case) µ is not itself a Bethe measure. In the following, we shall focus on the d1RSB and s1RSB scenarios, which are particularly interesting and can be treated in a unified framework (we shall sometimes refer to both of them as ‘1RSB’). More complicated scenarios, such as ‘full RSB’,

Beyond BP: Many states

¿¿

are also possible. We shall not discuss such scenarios here because, so far, we have a relatively poor control of them in sparse graphical models. In order to proceed further, we shall make a series of assumptions about the structure of Bethe states in the 1RSB case. While further research work is required to formalize these assumptions completely, they are precise enough for deriving several interesting quantitative predictions. To avoid technical complications, we assume that the compatibility functions ψa ( · ) are strictly positive. (The cases with ψa ( · ) = 0 should be treated as limiting cases of n n , ν a→i } of the BP equations such models.) Let us index the various quasi-solutions {νi→a by n. With each of them we can associate a Bethe measure, and we can compute the corresponding Bethe free entropy Fn = F(ν n ). The three postulates of the 1RSB scenario are listed below. Assumption 1 There exist exponentially many quasi-solutions of the BP equations. The number of such solutions with free entropy F(ν n ) ≈ N φ is (to leading exponential order) exp{N Σ(φ)}, where Σ( · ) is the complexity function.1 This can be expressed more formally as follows. There exists a function Σ : R → R+ (the complexity) such that, for any interval [φ1 , φ2 ], the number of quasi-solutions of the BP equations with F(ν n ) ∈ [N φ1 , N φ2 ] is exp{N Σ∗ + o(N )}, where Σ∗ = sup{Σ(φ) : φ1 ≤ φ ≤ φ2 }. We shall also assume in the following that Σ(φ) is ‘regular enough’, without going into the details. Among Bethe measures, a special role is played by those which have short range correlations (are extremal ). We have already mentioned this point in Chapter 12, and shall discuss the relevant notion of correlation decay in Chapter 22. We denote the set of extremal measures by E. Assumption 2 The ‘canonical’ measure µ, defined as in eqn (19.1), can be written as a convex combination of extremal Bethe measures µ(x) =

wn µn (x) ,

(19.6)

n∈E

with weights related to the Bethe free entropies wn = eFn /Ξ, Ξ ≡

n∈E

eFn .

Note that Assumption 1 characterizes the number of (approximate) BP fixed points, and Assumption 2 expresses the measure µ( · ) in terms of extremal Bethe measures. While each such measure gives rise to a BP fixed point by the arguments in the previous subsection, it is not clear that the converse holds. The next assumption implies that this is the case, to the leading exponential order. Assumption 3 To leading exponential order, the number of extremal Bethe measures equals the number of quasi-solutions of the BP equations: the number of extremal Bethe measures with free entropy ≈ N φ is also given by exp{N Σ(φ)}. 1 As we are interested only in the leading exponential behaviour, the details of the definitions of quasi-solutions become irrelevant, as long as (for instance) the fraction of violated BP equations vanishes in the large-N limit.


19.2

The 1RSB cavity equations

Within the three assumptions described above, the complexity function Σ(φ) provides basic information on how the measure µ decomposes into Bethe measures. Since the number of extremal Bethe measures with a given free-entropy density is exponential in the system size, it is natural to treat them within a statistical-physics formalism. The BP messages of the original problem will be the new variables, and Bethe measures will be the new configurations. This is what 1RSB is about. We introduce the auxiliary statistical-physics problem through the definition of a canonical distribution over extremal Bethe measures: we assign to a measure n ∈ E the probability wn (x) = exFn /Ξ(x). Here x plays the role of an inverse temperature (and is often called the Parisi 1RSB parameter). 2 The partition function of this generalized problem is . exFn = eN [xφ+Σ(φ)] dφ . (19.7) Ξ(x) = n∈E

According to Assumption 2 above, extremal Bethe measures contribute to µ with a weight wn = eFn /Ξ. Therefore the original problem is described by the choice x = 1. But varying x will allow us to recover the full complexity function Σ(φ). . If Ξ(x) = eN F(x) , a saddle point evaluation of the integral in eqn (19.7) gives Σ as the Legendre transform of F: F(x) = xφ + Σ(φ) , 19.2.1

∂Σ = −x . ∂φ

(19.8)

Counting BP fixed points

In order to actually estimate Ξ(x), we need to consider the distribution induced by wn (x) in the messages ν = {νi→a , ν a→i }, which we shall denote by Px (ν). The fundamental observation is that this distribution can be written as a graphical model, whose variables are BP messages. A first family of function nodes enforces the BP equations, and a second one implements the weight exF(ν) . Furthermore, it turns out that the topology of the factor graph in this auxiliary graphical model is very close to that of the original factor graph. This suggests that we should use the BP approximation in this auxiliary model in order to estimate Σ(φ). The 1RSB approach can be therefore summarized in one sentence: Introduce a Boltzmann distribution over Bethe measures, write it in the form of a graphical model, and use BP to study this model. This programme is straightforward, but one must be careful not to confuse the two models (the original one and the auxiliary one), and their messages. Let us first simplify the notations for the original messages. The two types of messages entering the mai and νi→a = mia ; BP equations of the original problem will be denoted by ν a→i =

mai by

m. Each we shall denote the set of all of the mia by m and the set of all of the

2 It turns out that the present approach is equivalent to the cloning method discussed in Chapter 12, where x is the number of clones.


11 00 11 00

11 00 00 11 00 11

a a

11 00 00 11 i

i ai

11 00 00 11

11 00 00 11

Fig. 19.2 A part of the original factor graph (left), and the corresponding auxiliary factor graph (right).

of these 2|E| messages is a normalized probability distribution over the alphabet X . With this notation, the original BP equations read

mbi (xi ) ,

mai (xi ) ∼ ψa (x∂a ) mja (xj ) . (19.9) mia (xi ) ∼ = = {xj }j∈∂a\i

b∈∂i\a

j∈∂a\i

Hereafter, we shall write them in the compact form mia = fi {

mbi }b∈∂i\a ,

mai = ˆfa {mja }j∈∂a\i .

(19.10)

Each message set (m,

m) is given a weight proportional to exF(m,bm) , where the free entropy m) is written in terms of BP messages as F(m,

m) = Fa ({mja }j∈∂a ) + Fi ({

mbi }b∈∂i ) − Fia (mia ,

mai ) . (19.11) F(m,

a∈F

i∈V

(ia)∈E

The functions Fa , Fi , Fia were obtained in (14.28). Let us copy them here for convenience: ⎡ ⎤ ψa (x∂a ) mja (xj )⎦ , Fa ({mja }j∈∂a ) = log ⎣ Fi ({

mbi }b∈∂i ) = log Fia (mia ,

mai ) = log

x∂a

j∈∂a

mbi (xi )

,

xi b∈∂i

(19.12)

mia (xi )

mai (xi )

.

(19.13)

xi

We now consider the 2|E| messages m and

m as variables in our auxiliary graphical model. The distribution induced by wn (x) in such messages takes the form m) = Px (m,

1 Ψa ({mja ,

mja }j∈∂a ) Ψi ({mib ,

mib }b∈∂i ) Ψia (mia ,

mia ) , Ξ(x) a∈F

i∈V

(ia)∈E

(19.14)


where we have introduced the compatibility functions % & xFa ({mja } j∈∂a ) Ψa = e I

mai = ˆfa {mja }j∈∂a\i ,

(19.15)

i∈∂a

Ψi =

xFi ({bm } bi b∈∂i ) mbi }b∈∂i\a e I mia = fi {

,

(19.16)

a∈∂i

Ψia = e−xFia (mia ,bmai ) .

(19.17)

The corresponding factor graph is depicted in Fig. 19.2 and can be described as follows: • For each edge (i, a) of the original factor graph, we introduce a variable node in the mai ). Furthermore, auxiliary factor graph. The associated variable is the pair (mia ,

we introduce a function node connected to this variable, contributing to the weight through a factor Ψia = e−xFai . • For each function node a of the original graph, we introduce a function node in the auxiliary graph and connect it to all of the variable nodes corresponding to edges (i, a), i ∈ ∂a. The compatibility function Ψa associated with this function node has two roles: (i) it enforces the |∂a| BP equations expressing the variables {

mai }i∈∂a in terms of the {mia }i∈∂a (see eqn (19.9)); (ii) it contributes to the weight through a factor exFa . • For each variable node i of the original graph, we introduce a function node in the auxiliary graph, and connect it to all of the variable nodes corresponding to edges (i, a), a ∈ ∂i. The compatibility function Ψi has two roles: (i) it enforces mbi }b∈∂i (see the |∂i| BP equations expressing the variables {mib }b∈∂i in terms of {

eqn (19.9)); (ii) it contributes to the weight through a factor exFi . Note that we were a bit sloppy in eqns (19.15)–(19.17). The messages mia ,

mai are in general continuous, and indicator functions should therefore be replaced by delta functions. This might, in turn, pose some definition problems (what is the reference measure on the messages? can we hope for exact solutions of BP equations?). One should consider the above as a shorthand for the following procedure. First, we discretize the messages (and BP equations) in such a way that they can take a finite number q of values. We compute the complexity by letting N → ∞ at fixed q, and take the limit q → ∞ at the end. It is easy to define several alternative, and equally reasonable, limiting procedures. We expect all of them to yield the same result. In practice, the ambiguities in eqns (19.15)–(19.17) are resolved on a case-by-case basis. 19.2.2

Message passing in the auxiliary model

The problem of counting the number of Bethe measures (more precisely, computing the complexity function Σ(φ)) has been reduced to one of estimating the partition function Ξ(x) of the auxiliary graphical model (19.14). Since we are interested in the case of locally tree-like factor graphs G, the auxiliary factor graph is locally treelike as well. We can therefore apply BP to estimate its free-entropy density F(x) = limN N −1 log Ξ(x). This will give us the complexity through the Legendre transform of eqn (19.8).


i ν(ia)→i e−xFai (mia ,bmai ) νi→(ia) (ia)

νa→(ia)

ν(ia)→a

a

Fig. 19.3 Messages in the auxiliary graphical model.

In the following, we denote by i ∈ V and a ∈ F a generic variable node and function node in the graph G, and by (ia) ∈ E an edge in G. By extension, we denote the corresponding nodes in the auxiliary graph in the same way. The messages appearing in the BP analysis of the auxiliary model can be classified as follows (see Fig. 19.3): mai ) and • From the variable node (ia), two messages are issued: ν(ia)→a (mia ,

mai ). ν(ia)→i (mia ,

mai ). • From the function node a, |∂a| messages to nodes i ∈ ∂a are issued: ν a→(ai) (mia ,

mai ). • From the function node i, |∂i| messages to nodes a ∈ ∂i are issued: ν i→(ai) (mia ,

• From the degree-one function node connected to the variable node (ia), a message is issued towards this variable node. This message is simply e−xFia (mia ,bmai ) . The BP equations for the variable node (ia) take a simple form: ν(ia)→a (mia ,

mai ) ∼ mai ) e−xFia (mia ,bmai ) , = ν i→(ia) (mia ,

mai ) ∼ mai ) e−xFia (mia ,bmai ) . ν(ia)→i (mia ,

= ν a→(ia) (mia ,

(19.18)

We can use these equations to eliminate the messages ν i→(ia) , ν a→(ia) in favour of ν(ia)→a , ν(ia)→i . In order to emphasize this choice (and to simplify the notation) we define mai ) ≡ ν(ia)→a (mia ,

mai ) , Qia (mia ,

ai (mia ,

Q mai ) ≡ ν(ia)→i (mia ,

mai ) .

(19.19)

We can now write the remaining BP equations of the auxiliary graphical model in

ai ( · , · ). The BP equation associated with the function node terms of Qia ( · , · ) and Q corresponding to i ∈ V reads ∼ Qia (mia ,

mai ) = I mic = fi ({

mdi }d∈∂i\c ) 7

{mib ,b mbi }b∈∂i\a

c∈∂i

mbi }b∈∂i ) − Fai (mia ,

mai )] × exp x [Fi ({

8

bi (mib ,

Q mbi ) ,

b∈∂i\a

and the equation associated with the function node corresponding to a ∈ F is

(19.20)


ai (mia ,

mai ) ∼ Q =

⎡

⎤ & % ⎣ I

maj = ˆfa ({mka }k∈∂a\j ) ⎦

{mja ,b maj }j∈∂a\i

j∈∂a

8 ( 7 ' mai ) Qja (mja ,

maj ) . × exp x Fa {mja }j∈∂a − Fai (mia ,

(19.21)

j∈∂a\i

Equations (19.20) and (19.21) can be simplified further, using the following lemma. mai (xi ) > 0. Under the condition mia = Lemma 19.3 Assume that xi mia (xi )

mdi }d∈∂i\a ) (which holds when the indicator functions in eqn (19.20) evaluate to fi ({

mbi }b∈∂i )−Fai (mia ,

mai ) can be expressed in terms of {

mbi }b∈∂i\a . 1), the difference Fi ({

Explicitly, we have mbi }b∈∂i\a ) ≡

mbi (xi ) . (19.22) eFi −Fia = zia ({

xi b∈∂i\a

Analogously, under the condition

mai = ˆfa ({mka }k∈∂a\i ) (which holds when the indicator functions in eqn (19.21) evaluate to 1), the difference Fa ({mja }j∈∂a ) − mai ) depends only on {mja }j∈∂a\i . Explicitly, Fai (mia ,

eFa −Fia = zâi ({mja }j∈∂a\i ) ≡

x∂a

ψa (x∂a )

mja (xj ) .

(19.23)

j∈∂a\i

Proof Let us consider eqn (19.22) first. From the definition (14.28), it follows that

mbi (xi ) Fi −Fia . (19.24) = xi b∈∂i e mai (xi ) xi mia (xi )

mci }c∈∂i\a ) in the denominator, and keeping track of the norSubstituting mia = fi ({

malization constant, we get

mbi (xi ) , (19.25) mia (xi )

mai (xi ) = xi b∈∂i mai (xi ) xi b∈∂i\a

x i

which implies eqn (19.22). The derivation of eqn (19.23) is completely analogous and is left as an exercise for the reader. Note that the functions zia ( · ) and zâi ( · ) appearing in eqns (19.22) and (19.23) are in fact the normalization constants hidden by the ‘∼ =’ notation in eqns (19.9). Because of this lemma, we can seek a solution of eqns (19.20) and (19.21) where

ai depends only on

mai . Hereafter, we shall focus on Qia depends only on mia , and Q this case, and, with some misuse of notation, we shall write mai ) = Qia (mia ) , Qia (mia ,

ia (mia ,

ai (

Q mai ) = Q mai ) .

(19.26)


The BP equations for the auxiliary graphical model (19.20), (19.21) then become x

bi (

Qia (mia ) ∼ I (mia = gi ({

mbi })) [zia ({

mbi })] mbi ) , (19.27) Q = {b mbi }

ai (

mai ) ∼ Q =

b∈∂i\a

I (

mai = fa ({mja })) [ˆ zai ({mja })]

{mja }

x

Qja (mja ) ,

(19.28)

j∈∂a\i

where {

mbi } is a shorthand for {

mbi }b∈∂i\a and {mja } is a shorthand for {mja }j∈∂a\i . mbi }) and zâi ({mja }) are given in eqns (19.22) and (19.23). The expressions for zia ({

Equations (19.27) and (19.28) are the 1RSB cavity equations. As we did in the case of the ordinary BP equations, we can consider them as an update rule for a message-passing algorithm. This will be discussed further in the following sections.

a→i ( · ) is used, to emphasize the fact that 1RSB Sometimes the notation Qi→a ( · ), Q messages are associated with directed edges. mai (xi ) > Notice that our derivation was based on the assumption that xi mia (xi )

0. This condition holds if, for instance, the compatibility functions of the original model are bounded away from 0. Under this condition, we have shown the following. Proposition 19.4 If the 1RSB cavity equations (19.27) and (19.28) have a solution

Q, this corresponds to a solution to the BP equations of the auxiliary graphical Q, model. Conversely, if the BP equations of the auxiliary graphical model admit a solution satisfying the condition (19.26), then the resulting messages must be a solution of the 1RSB cavity equations. The assumption (19.26)—which is suggestive of a form of ‘causality’— cannot be further justified within the present approach, but alternative derivations of the 1RSB equations confirm its validity. 19.2.3

Computing the complexity

We now compute the free entropy of the auxiliary graphical model within the BP approximation. We expect the result of this procedure to be asymptotically exact for a wide class of locally tree-like graphs, thus yielding the correct free-entropy density F(x) = limN N −1 log Ξ(x).

ai } to be a solution (or a quasi-solution) of the 1RSB cavity equaAssume {Qia , Q tions (19.27) and (19.28). We use the general form (14.27) of the Bethe free entropy, but take into account the degree-one factor nodes using the simplified expression derived in Exercise 14.6. The various contributions to the free entropy are: • The contribution from the function node a (here {mia } is a shorthand for {mia }i∈∂a ): ⎧ ⎫ ⎨ ⎬ FRSB = log exFa ({mia }) Qia (mia ) . (19.29) a ⎩ ⎭ {mia }

i∈∂a

• The contribution from the function node i (here {

mai } is a shorthand for {

mai }a∈∂i ): ⎧ ⎫ ⎨ ⎬

ai (

FRSB = log exFi ({bmai }) mai ) . Q (19.30) i ⎩ ⎭ {b mai }

a∈∂i

¼


• The contribution from the variable node (ia): ⎧ ⎫ ⎨ ⎬

ai (

= log exFia (mia ,bmai ) Qia (mia )Q mai ) . FRSB ia ⎩ ⎭

(19.31)

mia ,b mai

• The contributions from the two edges a–(ai) and i–(ai) are both equal to −FRSB ia . The Bethe free entropy of the auxiliary graphical model is equal to

= FRSB ({Q, Q}) FRSB + FRSB − FRSB (19.32) a i ia . a∈F

19.2.4

i∈V

(ia)∈E

Summary

The 1RSB cavity equations (19.27) and (19.28) are the BP equations for the auxiliary

ai (

graphical model defined in eqn (19.14). They relate 2|E| messages {Qia (mia ), Q mai )}. Each such message is a probability distribution of ordinary BP messages mia (xi ) and

mai (xi ), respectively. These elementary messages are, in turn, probability distributions on variables xi ∈ X .

ai }, one can estimate the Given a solution (or an approximate solution) {Qia , Q free-entropy density of the auxiliary model as

+ errN , log Ξ(x) = FRSB ({Q, Q})

(19.33)

is given by eqn (19.32). For a large class of locally tree-like models, where FRSB ({Q, Q}) we expect the BP approximation to be asymptotically exact for the auxiliary model. This means that the error term errN is o(N ). For such models, the free-entropy density is given by its 1RSB cavity expression

. The complexity Σ(φ) is then computed F(x) = f RSB (x) ≡ limN →∞ FRSB ({Q, Q})/N through the Legendre transform (19.8). 19.2.5

Random graphical models and density evolution

Let us consider the case where G is a random graphical model as defined in Section 14.6.1. The factor graph is distributed according to one of the ensembles GN (K, α)

inor DN (Λ, P ). The function nodes are taken from a finite list {ψ (k) (x1 , . . . , xk ; J)} (k) dexed by a label J with distribution PJb . Each factor ψa ( · ) is taken to be equal to ψ (k) ( · · · ; J a ) independently with the same distribution. We also introduce explicitly a degree-one factor ψi (xi ) connected to each variable node i ∈ V . This is also drawn independently from a list of possible factors {ψ(x; J)}, indexed by a label J with distribution PJ . For a random graphical model, the measure µ( · ) becomes random, and so does its decomposition into extremal Bethe states in particular the probabilities {wn } and n n

ai } become the message sets {νi→a , ν a→i }. In particular, the 1RSB messages {Qia , Q random. It is important to keep in mind the ‘two levels’ of randomness. Given an edge n is random if the Bethe state n is drawn from the distribution (ia), the message νi→a wn . The resulting distribution Qia (m) becomes a random variable when the graphical model is itself random.


½

ai (

The distributions of Qia (m) and Q m) can then be studied through the density evo(0)

d

lution method of Section 14.6.2. Let us assume an i.i.d. initialization Qia ( · ) = Q(0) ( · ) d (0) (t)

(0) ( · ) =

(t) ( · ) the 1RSB messages along (and Q Q ( · )), and denote by Qia ( · ) and Q ai ai edge (ia) after t parallel updates using the 1RSB equations (19.27) and (19.28). If (ia) (t) is a uniformly random edge then, as N → ∞, Qia ( · ) converges in distribution3 to

(t) ( · ) converges in distribution to Q

(t) ( · )). The distributions Q(t) ( · ) Q(t) ( · ) (and Q ia (t)

( · ) are themselves random variables that satisfy the following equations: and Q (t+1)

Q

d

(m) ∼ =

x

I (m = f({

mb }; J)) z({

mb }; J)

{b mb } d

(t) (

m) ∼ Q =

l−1

(t) (

Q b mb ) ,

(19.34)

b=1

k−1 & % (t)

zˆ({mj }; J)

x I

m = ˆf({mj }; J) Qj (mj ) , {mj }

(19.35)

j=1

where k 3and l4are distributed according to the edge-perspective degree ρk and 4 3 profiles (t) (t) (t)

λl , the Qb are k − 1 independent copies of Q ( · ), and the Qj are l − 1 independent copies of Q(t) ( · ). The functions z and zˆ are given by z({

mb }; J) =

ψ(x, J)

x

= zˆ({mj }; J)

x1 ,...,xk

l−1

mb (x) ,

b=1

ψ (k) (x1 , . . . , xk ; J)

k−1

mj (xj ) .

(19.36)

j=1

In the 1RSB cavity method, the actual distribution of Qi→a is assumed to coincide with one of the fixed points of the above density evolution equations. As in the RS case, one hopes that, on large enough instances, the message-passing algorithm will converge to messages distributed according to this fixed-point equation (meaning that there is no problem in exchanging the limits t → ∞ and N → ∞). This can be checked numerically. For random graphical models, the 1RSB free-entropy density converges to a finite

by limit f RSB (x). This can be expressed in terms of the distributions of Q and Q. taking the expectation of eqns (19.29)–(19.31) and assuming that the 1RSB messages coming into the same node are i.i.d. As in eqn (14.77), the result takes the form f RSB = fvRSB + nf ffRSB − ne feRSB .

(19.37)

Here nf is the average number of function nodes per variable (equal to Λ (1)/P (1) for a graphical model in the ensemble DN (Λ, P ), and to α for a graphical model in the ensemble GN (K, α)), and ne is the number of edges per variable (equal to Λ (1) and 3 We shall not discuss the measure-theoretic subtleties related to this statement. Let us just mention that weak topology is understood on the space of messages Q(t) .

¾


to Kα for these two ensembles). The contributions from variable nodes fvRSB , function nodes ffRSB , and edges feRSB are

feRSB

b m,m

19.2.6

x ⎫ ⎬

l (

1 (

m1 ) . . . Q ml )

m1 (x) . . .

ml (x)ψ(x; J) Q , ⎩ ⎭ x∈X {b m1 ,...,b ml } ⎧ ⎨ = Ek,J,{Q} log Q1 (m1 ) . . . Qk (mk ) b ⎩ {m1 ,...,mk } ⎡ ⎤x ⎫ ⎬

⎦ ×⎣ m1 (x1 ) . . . mk (xk )ψ (k) (x1 , . . . , xk ; J) , ⎭ x1 ,...,xk ∈X ⎧ x ⎫ ⎨ ⎬

m)Q(m) = EQ,Q log

m(x)m(x) Q(

. (19.38) b ⎩ ⎭

fvRSB = El,J,{Q} b log

ffRSB

⎧ ⎨

x∈X

Numerical implementation

Needless to say, it is extremely challenging to find a fixed point of the density evolution

A equations (19.34) and (19.35), and thus determine the distributions of Q and Q. simple numerical approach consists in generalizing the population dynamics algorithm described in the context of the RS cavity method (see Section 14.6.3). There are two important issues related to such a generalization:

· )), which is itself a distribution of mes(i) We seek the distribution of Q( · ) (and Q( sages. If we approximate Q( · ) by a sample (a ‘population’), we thus need two levels of populations. In other words, we seek a population {msr } with N M items. For each r ∈ {1, . . . , N }, the set of messages {msr }, s ∈ {1, . . . , M }, represents a distribution Qr ( · ) (ideally, it would be an i.i.d. sample from this distribution). At the next level, the population {Qr ( · )} , r ∈ {1, · · · , N }, represents the distribution of Q( · ) (ideally, an i.i.d. sample). Analogously, for function-to-variable messages, we use a population {

msr }, with r ∈ {1, . . . , N } and s ∈ {1, . . . , M }.

x appearing in eqns (19.34) and (ii) The reweighting factors z({

mb }; J)x and zˆ({mj }; J) (19.35) do not have any analogue in the RS context. How can one take such factors

· ) are represented as populations? One possibility into account when Q( · ) and Q( is to generate an intermediate weighted population, and then sample from it with a probability proportional to the weight. This procedure is summarized in the following pseudocode.


¿

1RSB Population dynamics (model ensemble, sizes N, M , iterations T ) 1: Initialize {msr }; 2: for t = 1, . . . , T : 3: for r = 1, . . . , N : 4: Draw an integer k with distribution ρ; 5: Draw i(1), . . . , i(k − 1) uniformly in {1, . . . , N }; (k) 6: Draw J with distribution PJb ; 7: for s = 1, . . . , M : 8: Draw s(1), . . . , s(k − 1) uniformly in {1, . . . , M }; s(1) s(k−1)

; J) 9: Compute

ms = ˆf(m , . . . , m temp

10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25:

i(1) s(1)

i(k−1) s(k−1)

x Compute W s = zˆ(mi(1) , . . . , mi(k−1) ; J) end; Generate the new population mstemp , W s }s∈[M ] ) {

msr }s∈[M ] = Reweight({

end; for r = 1, . . . , N : Draw an integer l with distribution λ; Draw i(1), . . . , i(l − 1) uniformly in {1, . . . , N }; Draw J with distribution P ; for s = 1, . . . , M : Draw s(1), . . . , s(l − 1) uniformly in {1, . . . , M }; s(1) s(k−1) mi(1) , . . . ,

mi(l−1) ; J) Compute mstemp = f(

Compute W s = z(

mi(1) , . . . ,

mi(l−1) ; J)x end; Generate the new population {msr }s∈[M ] = Reweight({mstemp , W s }s∈[M ] ) s(1)

s(l−1)

end; return {

msr } and {msr }.

The reweighting procedure is given by the following pseudocode: Reweight (population of messages and weights {(mstemp , W s )}s∈[M ] ) 1: for s = 1, . . . , M , set ps ≡ W s / s W s ; 2: for s = 1, . . . , M : 3: Draw i ∈ {1, . . . , M } with distribution ps ; 4: Set msnew = mitemp ; 5: end; 6: return {msnew }s∈[M ] . In the large-N, M limit, the populations generated by this algorithm should con (T ) ( · ) (see eqns (19.34) and verge to i.i.d. samples distributed as Q(T ) ( · ) and Q (19.35)). If we let T grow, they should represent accurately the fixed points of the density evolution, although the caveats expressed in the RS case should be repeated


here. Among the other quantities, the populations generated by this algorithm allow one to estimate the 1RSB free-entropy density (19.37). Suppose we have generated a population of messages {

msr ( · )}, where each message is a probability distribution on X . The corresponding estimate of fvRSB is ⎧ x ⎫ N M ⎨ 1 ⎬ s(1) 1 s(l)

f RSB = El,J log

m (x) · · ·

m (x) ψ(x; J) . v r(1) r(l) ⎩Ml ⎭ Nl s(1),...,s(l)=1 x∈X

r(1)...r(l)=1

Similar expressions are easily written for ffRSB and feRSB . Their (approximate) evaluation can be accelerated considerably by summing over a random subset of the l-tuples r(1), . . . , r(l) and s(1), . . . , s(l). Further, as in the RS case, it is beneficial to average over iterations (or, equivalently, over T ) in order to reduce statistical errors at small computational cost.

19.3

A first application: XORSAT

Let us apply the 1RSB cavity method to XORSAT. This approach was introduced in Section 18.6, but we now want to show how it follows as a special case of the formalism developed in the previous sections of this chapter. Our objective is to exemplify the general ideas with a well understood problem, and to build some basic intuition that will be useful in more complicated applications. As in Chapter 18, we consider the distribution over x = (x1 , . . . , xN ) ∈ {0, 1}N specified by µ(x) =

M 1 I xi1 (a) ⊕ · · · ⊕ xik (a) = ba . Z a=1

(19.39)

As usual, ⊕ denotes the sum modulo 2 and, for each a ∈ {1, . . . , M }, ∂a = {i1 (a), . . . , iK (a)} is a subset of {1, ·, N }, and ba ∈ {0, 1}. Random K-XORSAT formulae are generated by choosing both the index set {i1 (a), . . . , iK (a)} and the right-hand side ba uniformly at random. 19.3.1

BP equations

The BP equations read mia (xi ) =

1

mbi (xi ) , zia

(19.40)

b∈∂i\a

mai (xi ) =

1 I xi1 (a) ⊕ · · · ⊕ xiK (a) = ba mja (xj ) . zâi x ∂a\i

(19.41)

j∈∂a\i

As in Section 18.6, we shall assume that messages can take only three values, which we denote by the shorthands mia = 0 if (mia (0) = 1, mia (1) = 0), mia = 1 if (mia (0) = 0, mia (1) = 1), and mia = ∗ if (mia (0) = mia (1) = 1/2).


Consider the first BP equation (19.40), and denote by n0 , n1 , and n∗ the numbers of messages of type 0, 1, and ∗, respectively, in the set of incoming messages {

mbi }, b ∈ ∂i\a. Equation (19.40) can then be rewritten as ⎧ ⎧ −n 0 if n0 > 0, n1 = 0, 2 ∗ if n0 > 0, n1 = 0, ⎪ ⎪ ⎪ ⎪ ⎨ ⎨ −n∗ if n0 = 0, n1 > 0, 1 if n0 = 0, n1 > 0, 2 zia = (19.42) mia = ⎪ ∗ if n0 = 0, n1 = 0, ⎪ 21−n∗ if n0 = 0, n1 = 0, ⎪ ⎪ ⎩ ⎩ ? if n0 > 0, n1 > 0, 0 if n0 > 0, n1 > 0. The computation of the normalization constant zia will be useful in the 1RSB analysis. Notice that if n0 > 0 and n1 > 0, a contradiction arises at node i and therefore mia is not defined. However, we shall see that, because zia = 0 in this case, this situation does not create any problem within 1RSB.

1 , and n

∗ the numbers of In the second BP equation (19.41), we denote by n

0 , n messages of type 0, 1, and ∗, respectively, among {mja }, j ∈ ∂a\i. We then obtain ⎧ ⎨ 0 if n∗ = 0 and n1 has the same parity as ba ,

mai = 1 if n∗ = 0 and n1 does not have the same parity as ba , (19.43) ⎩ ∗ if n∗ > 0. In all three cases, zâi = 1. In Section 18.6 we studied the equations (19.40) and (19.41) above and deduced that, for typical random instances with α = M/N < αd (K), they have a unique mai = ∗ on each edge. solution, with mia =

Exercise 19.2 Evaluate the Bethe free entropy for this solution, and show that it yields the free-entropy density f RS = (1 − α) log 2.

19.3.2


We now assume that the BP equations (19.42) and (19.43) have many solutions, and apply the 1RSB cavity method to study their statistics.

ai are distributions over {0, 1, ∗}. A little effort The 1RSB messages Qia and Q shows that eqn (19.27) yields ⎧ ⎫ ⎨ ⎬ % % & & 1

bi (0) + 2−x Q

bi (∗) −

bi (∗) Q 2−x Q Qia (0) = , (19.44) ⎭ Zia ⎩ b∈∂i\a b∈∂i\a ⎧ ⎫ ⎨ ⎬ % % & & 1

bi (1) + 2−x Q

bi (∗) −

bi (∗) Q 2−x Q Qia (1) = , (19.45) ⎭ Zia ⎩ b∈∂i\a

Qia (∗) =

1 x 2 Zia

b∈∂i\a

bi (∗) . 2−x Q

(19.46)

b∈∂i\a

For instance, eqn (19.44) follows from the first line of eqn (19.42): mia = 0 if all of the incoming messages are

mbi ∈ {∗, 0} (first term), unless they are all equal to ∗

The 1RSB cavity method x (subtracted term). The reweighting zia = 2−xn∗ decomposes into factors associated with the incoming ∗ messages. The second group of 1RSB equations, eqn (19.28), takes the form ⎫ ⎧ ⎬ ⎨

ai (0) = 1 (Qja (0) + Qja (1)) + s(ba ) (Qja (0) − Qja (1)) , Q ⎭ 2 ⎩ j∈∂a\i

j∈∂a\i

⎫ ⎧ ⎬ ⎨ 1

ai (1) = (Qja (0) + Qja (1)) − s(ba ) (Qja (0) − Qja (1)) , Q ⎭ 2 ⎩ j∈∂a\i

ai (∗) = 1 − Q

(19.47)

j∈∂a\i

(19.48) (Qja (0) + Qja (1)) ,

(19.49)

j∈∂a\i

where s(ba ) = +1 if ba = 0, and s(ba ) = −1 otherwise. Notice that, if one takes x = 0, the two sets of equations coincide with those obtained in Section 18.6 (see eqn (18.35)) (the homogeneous linear system, ba = 0, was considered there). As in that section, we look for solutions such that the messages

ai ( · )) take two possible values: either Qia (0) = Qia (1) = 1/2, Qia ( · ) (and similarly Q or Qia (∗) = 1. This assumption is consistent with the 1RSB cavity equations (19.44) and (19.49). Under this assumption, the x dependency drops from these equations and we recover the analysis in Section 18.6. In particular, we can repeat the density evolution analysis discussed there. If we denote by Q∗ the probability that a randomly chosen edge carries the 1RSB message Qia (0) = Qia (1) = 1/2, then the fixed-point equation of density evolution reads }. Q∗ = 1 − exp{−kαQk−1 ∗

(19.50)

For α < αd (K), this equation admits only the solution Q∗ = 0, implying Qia (∗) = 1 with high probability. This indicates (once more) that the only solution of the BP equations in this regime is mia = ∗ for all (i, a) ∈ E. For α > αd , a pair of non-trivial solutions (with Q∗ > 0) appear, indicating the existence of a large number of BP fixed points (and hence Bethe measures). Considerations of stability under density evolution suggest that we should select the largest one. It will also be useful in the following to introduce the probability

∗ = Qk−1 Q ∗

(19.51)

ai (0) = Q

ai (1) = 1/2. that a uniformly random edge carries a message Q 19.3.3

Complexity

We can now compute the Bethe free entropy (19.32) of the auxiliary graphical model. The complexity will be computed through the Legendre transform of the 1RSB free entropy (see eqn (19.8)).


Let us start by computing the contribution FRSB defined in eqn (19.29). Consider a the weight eFa ({mia }) =

I(xi1 (a) ⊕ · · · ⊕ xiK (a) = ba )

x∂a

mia (xi ) .

(19.52)

i∈∂a

1 , and n

∗ denote the number of variable nodes i ∈ ∂a such that mia = 0, 1, Let n

0 , n and ∗, respectively, for i ∈ ∂a. We then obtain ⎧

∗ > 0, ⎨ 1/2 if n

1 has the same parity as ba , if n

∗ = 0 and n (19.53) eFa ({mia }) = 1 ⎩

1 does not have the same parity as ba . 0 if n

∗ = 0 and n Taking the expectation of exFa ({mia }) with respect to {mia }, distributed independently according to Qia ( · ), and assuming that Qia (0) = Qia (1) (which is the case in our solution), we get # Fa

RSB

= log

$ 1 1 (1 − Qia (∗)) + x 1 − (1 − Qia (∗)) . 2 2 i∈∂a

(19.54)

i∈∂a

The first term corresponds to the case n

∗ = 0 (the factor 1/2 being the probability

∗ > 0. Within our solution, either that the parity of n

1 is ba ), and the second to n Qia (∗) = 0 or Qia (∗) = 1. Therefore only one of the above terms survives: either = − log 2, or Qia (∗) = 1 for some i ∈ ∂a, Qia (∗) = 0 for all i ∈ ∂a, yielding FRSB a = −x log 2. implying FRSB a Until now, we have considered a generic K-XORSAT instance. For random instances, we can take the expectation with respect to Qia (∗), independently distributed as in the density-evolution fixed point. The first case, namely Qia (∗) = 0 for all i ∈ ∂a = − log 2), occurs with probability Qk∗ . The second, i.e. Qia (∗) = 1 for (and thus FRSB a = −x log 2), occurs with probability 1 − Qk∗ . Altogether, we some i ∈ ∂a (and FRSB a obtain ' ( E{FRSB } = − Qk∗ + x(1 − Qk∗ ) log 2 + oN (1) . (19.55) a will concenAssuming the messages Qia ( · ) to be short-range correlated, a∈F FRSB a trate around its expectation. We then have, with high probability, ' ( 1 RSB Fa = −α Qk∗ + x(1 − Qk∗ ) log 2 + oN (1) . N

(19.56)

a∈F

The contributions from variable-node and edge terms can be computed along similar lines. We shall just sketch these computations, and invite the reader to work out the details. , i ∈ V , defined in eqn (19.30). Assume that Consider the contribution FRSB i

ai (∗) = 1 and Q

ai (0) = Q

ai (1) = 1/2 for n∗ and n0 , respectively, of the neighQ = −(n∗ x + n0 − 1) log 2 if n0 ≥ 1, and bouring function nodes a ∈ ∂i. Then FRSB i


FRSB = −(n∗ − 1)x log 2 otherwise. Averaging these expressions over n0 (a Poissoni

∗ ) and n∗ (Poisson with mean kα(1− Q

∗ )), distributed random variable with mean kαQ we obtain 6 5 6 4 35 1 RSB

∗ ) − e−kαQb∗ x log 2 + oN (1) .

∗ − 1 + e−kαQb∗ + kα(1 − Q Fi = − kαQ N i∈V

(19.57) defined in eqn (19.31). If Let us consider, finally, the edge contribution FRSB (ia) F F

ai (0) = Q

ai (1) = 1/2, then either e ai = 1 or e ia = 0, Qia (0) = Qia (1) = 1/2 and Q each with probability 1/2. As a consequence, FRSB (ia) = − log 2. If either Qia (∗) = 1 or RSB F

ai (∗) = 1 (or both), e ia = 1/2 with probability 1, and therefore FRSB = −x log 2. Q (ia) Altogether, we obtain, with high probability, 3 4 1 RSB

∗ + (1 − Q∗ Q

∗ )x log 2 + oN (1). F(ia) = −kα Q∗ Q (19.58) N (ia)∈E

The free entropy (19.32) of the auxiliary graphical model is obtained by collecting the various terms. We obtain FRSB (x) = N f RSB (x) + o(N ), where f RSB (x) = [Σtot + x φtyp ] log 2 and b

∗ − αQk∗ + 1 − e−kαQ∗ ,

∗ − kαQ Σtot = kαQ∗ Q

∗ + αQk∗ − α + e−kαQb∗ .

∗ + kαQ φtyp = −kαQ∗ Q

(19.59) (19.60)

∗ = Qk−1 , a condition that has a Here Q∗ is the largest solution of eqn (19.50), and Q ∗ pleasing interpretation, as shown in the exercise below. b ∗ ) defined in eqn (19.59). Show that the Exercise 19.3 Consider the function Σtot (Q∗ , Q b∗ = stationary points of this function coincide with the solutions of eqn (19.50) and that Q . Qk−1 ∗

Because of the linear dependence on x, the Legendre transform (19.8) is straightforward: Σtot if φ = φtyp , (19.61) Σ(φ) = −∞ otherwise. This means that there are 2N Σtot Bethe measures, and these all have entropy N φtyp log 2. Furthermore, Σtot + φtyp = 1 − α, confirming that the total entropy is (1 − α) log 2. This identity can be also written in the form 1 1 1 = N Σtot × N φtyp , 2 2 2N (1−α)

(19.62)

which is nothing but the decomposition (19.6) into extremal Bethe measures. Indeed, if x is a solution of the linear system, then µ(x) = 1/2N (1−α) , wn ≈ 1/2N Σtot , and

The special value x = 1

(assuming the µn to have disjoint supports) µn (x) ≈ 1/2N φtyp for the state n which contains x. Note that the value of Σ that we find here coincides with the result that we obtained in Section 18.5 for the logarithm of the number of clusters in random XORSAT formulae. This provides an independent check of our assumptions and, in particular, it shows that the number of clusters is, to leading order, the same as the number of Bethe measures. In particular, the SAT–UNSAT transition occurs at the value of α where the complexity Σtot vanishes. At this value, each cluster still contains a large number, 2N (1−αs ) , of configurations.

Exercise 19.4 Repeat this 1RSB cavity analysis for a linear Boolean system described by a factor graph from the ensemble DN (Λ, P ). (This means a random system of linear equations, where the fraction of equations involving k variables is Pk , and the fraction of variables which appear in exactly equations is Λ .) b ∗ satisfy (a) Show that Q∗ and Q b∗ ) , b ∗ = ρ(Q∗ ) , Q∗ = 1 − λ(1 − Q Q

(19.63)

where λ and ρ are the edge-perspective degree profiles. (b) Show that the complexity is given by Σtot = 1 −

Λ (1) b ∗ ) − Λ (1)(1 − Q∗ )Q b∗ , P (Q∗ ) − Λ(1 − Q P (1)

(19.64)

and the internal entropy of the clusters is φtyp = 1 − Λ (1)/P (1) − Σtot . (c) In the case where all variables have a degree strictly larger than 1 (so that λ(0) = 0), b ∗ = 1, Σtot = 1 − Λ (1)/P (1), φtyp = 0. argue that the relevant solution is Q∗ = Q What is the interpretation of this result in terms of the core structure discussed in Section 18.3?

19.4


Let us now return to the general formalism. The case x = 1 plays a special role, in that the weights {wn (x)} of various Bethe measures in the auxiliary model coincide with those appearing in the decomposition (19.6). This fact manifests itself in some remarkable properties of the 1RSB formalism.

19.4.1

Back to BP

Consider the general 1RSB cavity equations (19.27) and (19.28). Using the explicit form of the reweighting factors eFi −Fia and eFa −Fia provided in eqns (19.22) and (19.23), they can be written, for x = 1, as


Qia (mia ) ∼ =

I (mia = gi ({

mbi }))

xi {b mbi }

ai (

mai ) ∼ Q =

ψa (x∂a )

x∂a

bi (

mbi )

mbi (xi ) , Q

b∈∂i\a

I (

mai = fa ({mja }))

{mja }

Qja (mja ) mja (xj ) .

(19.65) (19.66)

j∈∂a\i

We introduce the messages obtained by taking the averages of the 1RSB messages

ai }: {Qia , Q av (xi ) ≡ νi→a

Qia (mia ) mia (xi ) ,

av ν a→i (xi ) ≡

mia

ai (

mai )

mai (xi ) . Q

b mai

The interpretation of these quantities is straightforward. Given an extremal Bethe n n ( · ) (or ν a→i ( · )) be the measure sampled according to the distribution wn , let νi→a corresponding message along the directed edge i → a (or a → i, respectively). Its av av ( · ) (or ν a→i ( · ), expectation, with respect to the random choice of the measure, is νi→a respectively). Using the eqns (19.9), one finds that eqns (19.65) and (19.66) imply

av (xi ) ∼ νi→a =

av ν b→i (xi ) ,

b∈∂i\a

av (xi ) ∼ ν a→i =

{xj }j∈∂a\i

ψa (x∂a )

(19.67)

av νj→a (xj ) ,

(19.68)

j∈∂a\i

which are nothing but the ordinary BP equations. This suggests that even if µ( · ) decomposes into an exponential number of extremal Bethe measures µn ( · ), it is itself a (non-extremal) Bethe measure. In particular, there exists a quasi-solution of the BP equations associated with it that allows one to compute its marginals. The reader might be disappointed by these remarks. Why should we insist on the 1RSB cavity approach if, when the ‘correct’ weights are used, one recovers the much simpler BP equations? There are at least two answers: 1. The 1RSB approach provides a much more refined picture: decomposition into extremal Bethe states, long-range correlations, and complexity. This is useful and interesting per se. 2. In the case of a static (s1RSB) phase, it turns out that the region x = 1 corresponds to an ‘unphysical’ solution of the 1RSB cavity equations, and that (asymptotically) correct marginals are instead obtained by letting x = x∗ , for some x∗ ∈ [0, 1). In such cases it is essential to resort to the full 1RSB formalism (see Section 19.6 below). 19.4.2

A simpler recursion

As we stressed above, controlling (either numerically or analytically) the 1RSB distributional recursions (19.34) and (19.35) is a difficult task. In the case x = 1, they simplify considerably and lend themselves to a much more accurate numerical study. This observation can be very useful in practice.


As in Section 19.2.5, we consider a random graphical model. We shall also assume a ‘local uniformity condition’. More precisely, the original model µ( · ) is a Bethe measure av av (xi ) = 1/q and ν a→i (xi ) = 1/q, where q = |X | is the size of for the message set νi→a the alphabet. While such a local uniformity condition is not necessary, it considerably simplifies the derivation below. The reader can find a more general treatment in the literature. Consider eqns (19.34) and (19.35) at x = 1. The normalization constants can be easily computed using the uniformity condition. We can then average over the structure of the graph and the function node distribution: let us denote the averaged

av . They satisfy the following equations: distributions by Qav and Q ⎧ ⎫ l−1 ⎨ ⎬ l−2 (t)

(m) = E q I (m = f({

m }; J)) z({

m }) (

m ) , (19.69) Q Q(t+1) b b b av av ⎩ ⎭ b=1 {b mb } ⎧ ⎫ k−1 ⎨ q k−2 % ⎬ & (t) ˆf({mj }; J)

(t) (

zˆ({mj }; J)

Q m ) = E I

m = Q (m ) , (19.70) j av av ⎩ ψk ⎭ {mj }

j=1

where the expectations are taken over l, k, J, J, distributed according to the random graphical model. Here, ψ k = x1 ,...,xk−1 ψ(x1 , . . . , xk−1 , x; J) can be shown to be independent of x (this is necessary for the uniformity condition to hold). Equations (19.69) and (19.70) are considerably simpler than the original distribu (t) tional equations (19.34) and (19.35) in that Q(t) av ( · ) and Qav ( · ) are non-random. On the other hand, they still involve a reweighting factor that is difficult to handle. It turns out that this reweighting can be eliminated by introducing a new pair of distributions for each x ∈ X :

(t) (m) ,

x(t) (m) ≡ q m(x) Q R av

Rx(t) (m) = q m(x) Q(t) av (m) .

(19.71) (t)

One can show that eqns (19.69) and (19.70) imply the following recursions for Rx

x(t) : and R ⎧ ⎫ l−1 ⎨ ⎬

(t) (

Rx(t+1) (m) = E I (m = g({

mb }; J)) , (19.72) R x mb ) ⎩ ⎭ b=1 {b mb } ⎧ ⎫ ⎨ ⎬ & k−1 %

(t) (

R π({xj }|x; J) I

m = f ({mj }; J) Rx(t)j (mj ) . (19.73) x m) = E ⎩ ⎭ {xj }

{mj }

j=1

k, J and, for any x, J,

the distriHere E denotes the expectation with respect to l, J,

bution π({xj }|x; J) is defined by

= π(x1 , . . . , xk−1 |x; J)

ψ(x1 , . . . , xk−1 , x; J)

y1 ,...,yk−1

ψ(y1 , . . . , yk−1 , x; J)

.

(19.74)


Exercise 19.5 Prove eqns (19.72) and (19.73). It might be useful to recall the following explicit expressions for the reweighting factors z and zˆ: z({b mb }) m(x) =

l−1 Y

b mb (x) ,

(19.75)

b=1

b b zˆ({mj }; J) m(x) =

X

b ψ(x1 , . . . , xk−1 , x; J)

{xi },x

k−1 Y

mj (xj ) .

(19.76)

j=1

Equations (19.72) and (19.73) have a simple operational description. Let J and k be drawn according to their distribution, and, given x, let us generate x1 , . . . , xk−1 accord Then we draw independent messages m1 , . . . , mk−1 ing to the kernel π(x1 , . . . , xk |x; J). (t) (t)

with distributions Rx1 , . . . , Rxk−1 , respectively. According to eqn (19.73),

m = f ({mj }; J) (t)

then has the distribution Rx . For eqn (19.72), we draw J and l according to their

x(t) . ml−1 with distribution R distribution. Given x, we draw l − 1 i.i.d. messages

m1 , . . . ,

(t+1)

. Then m = g({

mb }; J) has the distribution Rx We shall see in Chapter 22 that this procedure does indeed coincide with that for computing ‘point-to-set correlations’ with respect to the measure µ( · ). To summarize, for x = 1 we have succeeded in simplifying the 1RSB density evolution equations in two directions: (i) the resulting equations do not involve ‘distributions of distributions’; and (ii) we have got rid of the reweighting factor. A third crucial simplification is the following. Theorem 19.5 The 1RSB equations have a non-trivial solution (meaning a solution different from the RS one) if and only if eqns (19.72) and (19.73), when initialized (0) so that Rx is a singleton distribution on m(y) = I(y = x), converge as t → ∞ to a non-trivial distribution. This theorem resolves (in the case x = 1) the ambiguity in the initial condition of the 1RSB iteration. In other words, if the 1RSB equations admit a non-trivial solution, this solution can be reached if we iterate the equations starting from the initial condition mentioned in the theorem. We refer the reader to the literature for a proof.

Exercise 19.6 Show that the free entropy of the auxiliary model FRSB (x), evaluated at x = 1, coincides with the RS Bethe free entropy. Further, its derivative with respect to x at x = 1 can be expressed in terms of the fixed(∞) bx(∞) . In particular, the complexity and internal free entropy point distributions Rx and R can be computed from the fixed points of the simplified equations (19.72) and (19.73).

Survey propagation

The conclusion of this section is that 1RSB calculations at x = 1 are not technically harder than RS ones. In view of the special role played by the value x = 1, this observation can be exploited in a number of contexts.

19.5

Survey propagation

The 1RSB cavity method can be applied to other message-passing algorithms whenever these have many fixed points. A particularly important case is the min-sum algorithm of Section 14.3. This approach (in both its RS and its 1RSB version) is sometimes referred to as the energetic cavity method because, in physics terms, the min-sum algorithm aims at computing the ground state configuration and its energy. We shall call the corresponding 1RSB message-passing algorithm SP(y) (‘survey propagation at finite y’). 19.5.1

The SP(y) equations

The formalism follows closely that used for counting the solutions of BP equations. To emphasize the similarities, let us adopt the same notation for the min-sum messages as for the BP ones. We define mja (xj ) ≡ Ei→a (xi ) ,

a→i (xi ) ,

mai (xi ) ≡ E

(19.77)

and write the min-sum equations (14.41) and (14.40) as mbi }b∈∂i\a ,

mai = ˆfae {mja }j∈∂a\i . mia = fie {

(19.78)

The functions fie and ˆfae are defined by eqns (14.41) and (14.40), which we reproduce here: mia (xi ) =

mbi (xi ) − uia , (19.79) b∈∂i\a

⎡

mai (xi ) = min ⎣Ea (x∂a ) + x∂a\i

⎤ mja (xj )⎦ − u âi ,

(19.80)

j∈∂a\i

where uia , u âi are normalization constants (independent of xi ) which ensure that mai (xi ) = 0 and minxi mia (xi ) = 0. minxi

mai }, we associate the Bethe energy With any set of messages {mia ,

m) = Fea ({mia }i∈∂a ) + Fei ({

mai }a∈∂i ) − Feia (mia ,

mai ) , (19.81) Fe (m,

a∈F

i∈V

where the various terms are (see eqn (14.45)) 5 6 mia (xi ) , Fea = min Ea (x∂a ) + x∂a

Feia

j∈∂a

' ( = min mia (xi ) +

mai (xi ) . xi

(ia)∈E

Fei = min xi

5

6

mai (xi ) ,

a∈∂i

(19.82)

Having set up the message-passing algorithm and the associated energy functional, we can repeat the programme developed in the previous sections. In particular, in analogy with Assumption 1, we have the following.


Assumption 4 There exist exponentially many quasi-solutions {mn } of the min-sum equations. The number of such solutions with Bethe energy Fe (mn ) ≈ N is (to leading exponential order) exp{N Σe ()}, where Σe () is the energetic complexity function. In order to estimate Σe (), we introduce an auxiliary graphical model, whose varimai }. These are forced to satisfy (within some ables are the min-sum messages {mia ,

accuracy) the min-sum equations (19.79) and (19.80). Each solution is given a weight e e−yF (m,bm) , with y ∈ R. The corresponding distribution is Py (m,

m) =

1 Ψa ({mja ,

mja }j∈∂a ) Ψi ({mib ,

mib }b∈∂i ) Ψia (mia ,

mia ) , Ξ(y) a∈F

i∈V

(ia)∈E

(19.83) where Ψa =

% & −yFe ({mja } j∈∂a ) e a I

mai = ˆfae {mja }j∈∂a\i ,

(19.84)

i∈∂a

Ψi =

−yFe ({bm } mbi }b∈∂i\a e i bi b∈∂i ) , I mia = fie {

(19.85)

a∈∂i

Ψia = eyFia (mia ,bmai ) . e

(19.86)

Since the auxiliary graphical model is again locally tree-like, we can hope to derive asymptotically exact results through belief propagation. The messages of the auxiliary

ai ( · ), are distributions over the minproblem, which will be denoted as Qia ( · ) and Q sum messages. The SP(y) equations are obtained by making, further, the independence assumption (19.26). The reader will certainly have noticed that the whole procedure is extremely close to our study in Section 19.2.2. We can apply our previous analysis verbatim to derive the SP(y) update equations. The only step that requires some care is the formulation of the proper analogue of Lemma 19.3. This becomes the following. mai (xi ) < ∞ for at least one value of xi ∈ X . Lemma 19.6 Assume that mia (xi ) +

mbi }b∈∂i\a ), then If mia = fie ({

Fei − Feia = uia ({

mbi }b∈∂i\a ) ≡ min xi

3

4

mbi (xi ) .

(19.87)

b∈∂i\a

Analogously, if

mai = fae ({mja }j∈∂a\i ), then 3 4 Fea − Feia = u âi ({mja }j∈∂a\i ) ≡ min Ea (x∂a ) + mka (xk ) . x∂a

(19.88)

k∈∂a\i

Using this lemma, the same derivation as in Section 19.2.2 leads to the following result.

Survey propagation

Proposition 19.7 The SP(y) equations are (with the shorthands {

mbi } for {

mbi }b∈∂i\a and {mja } for {mja }j∈∂a\i ) Qia (mia ) ∼ =

{b mbi }

ai (

mai ) ∼ Q =

I (mia = gie ({

mbi })) e−yuia ({bmbi })

bi (

mbi ) , Q

(19.89)

b∈∂i\a

I (

mai = fae ({mja })) e−yûai ({mja })

{mja }

Qja (mja ) .

(19.90)

j∈∂a\i

In the following we shall reserve the name survey propagation (SP) for the y = ∞ case of these equations. 19.5.2

Energetic complexity

The Bethe free entropy for the auxiliary graphical model is given by

= FRSB,e ({Q, Q})

a∈F

FRSB,e + a

i∈V

FRSB,e − i

FRSB,e , ia

(19.91)

(ia)∈E

and allows us to count the number of min-sum fixed points. The various terms are formally identical to those in eqns (19.29), (19.30), and (19.31), provided F· ( · ) is replaced everywhere by −Fe· ( · ), and x by y. We reproduce these equations here for convenience: ⎧ ⎫ ⎨ ⎬ e = log e−yFa ({mia }) Qia (mia ) , (19.92) FRSB,e a ⎩ ⎭ i∈∂a {mia } ⎧ ⎫ ⎨ ⎬ e

ai (

FRSB,e = log e−yFi ({bmai }) mai ) , Q (19.93) i ⎩ ⎭ a∈∂i {b mai } ⎧ ⎫ ⎨ ⎬ e

ai (

= log e−yFia (mia ,bmai ) Qia (mia )Q mai ) . FRSB,e (19.94) ia ⎩ ⎭ mia ,b mai

Assuming that the Bethe free entropy gives the correct free entropy of the auxiliary model, the energetic complexity function Σe () can be computed from FRSB,e (y)

through a Legendre transform: in the large-N limit, we expect FRSB,e ({Q, Q}) = e N F (y) + o(N ), where

= Σe () − y , Fe ({Q, Q})

∂Σe = y. ∂

(19.95)

Finally, the 1RSB population dynamics algorithm can be used to sample (approximately) the SP(y) messages in random graphical models.


19.5.3

Constraint satisfaction and binary variables

In Section 14.3.3, we noticed that the min-sum messages simplify significantly when we are dealing with constraint satisfaction problems. In such problems, the energy function takes the form E(x) = a Ea (x∂a ), where Ea (x∂a ) = 0 if constraint a is satisfied by the assignment x, and Ea (x∂a ) = 1 otherwise. As discussed in Section 14.3.3, the min-sum equations then admit solutions with

mai (xi ) ∈ {0, 1}. Furthermore, we do not need to keep track of the variable-to-function-node messages mia (xi ), but only of their ‘projection’ on {0, 1}. In other words, in constraint satisfaction problems the min-sum messages take 2|X | − 1 possible values (the all-1 message cannot appear). As a consequence, the

ai ( · ) and Qia ( · ) simplify considerably: they are points in a (2|X | −1)SP(y) messages Q dimensional simplex. If the min-sum messages are interpreted in terms of warnings, as we did in Section 14.3.3, then SP(y) messages keep track of the warnings’ statistics (over pure states). One can use this interpretation to derive the SP(y) update equations directly without going through the whole 1RSB formalism. Let us illustrate this approach with the important case of binary variables |X | = 2. The min-sum messages

m and m (once projected) can take three values: (

m(0),

m(1)) ∈ {(0, 1), (1, 0), (0, 0)}. We shall denote them by 0 (interpreted as a warning ‘take value 0’), 1 (interpreted as a warning ‘take value 1’) and ∗ (interpreted as a warning ‘you can take any value’), respectively. Warning propagation (WP) can be described in words as follows. Consider the message from variable node i to function node a. This depends on

1 , and all of the messages to i from function nodes b ∈ ∂i \ a. Suppose that n

0 , n

0 > n

1 , i n

∗ of these messages are of type 0, 1, and ∗, respectively, for i ∈ ∂a. If n

0 , it sends a 1 message to a. If n

1 = n

0 , it send a ∗ sends a 0 message to a. If n

1 > n message to a. The ‘number of contradictions’ among the messages that it receives is n1 , n

0 ). Fei − Feia = uia = min(

Now consider the message from function node a to variable node i. This depends on the messages coming from the neighbouring variables j ∈ ∂a \ i. We partition the neighbours into subsets P∗ , P0 , P1 , where Pm is the set of indices j such that mja = m. For each value of xi ∈ {0, 1}, the algorithm computes the minimal value of Ea (x∂a ) such that the variables in P0 and P1 are fixed to 0 and to 1, respectively. More explicitly, let us define a function ∆P (xi ) as follows: ∆P (xi ) =

min

{xj }j∈P∗

Ea (xi , {xj }j∈P∗ , {xk = 0}k∈P0 , {xl = 1}l∈P1 ) .

(19.96)

The following table then gives the outgoing message

mai and the number of contradicâi , as a function of the values of ∆P (0) and ∆P (1): tions at a, Fea − Feai = u ∆P (0) 0 0 1 1

∆P (1) 0 1 0 1

mai ∗ 0 1 ∗

u âi 0 0 0 1

Survey propagation

Having established the WP update rules, we can immediately write the SP(y) equations. Consider a node, and one of its neighbours to which it sends messages. For each possible configuration of incoming warnings at this node, denoted by input, we have found the rules to compute the outgoing warning output = O; UT(input) and the number of contradictions δFe (input). SP(y) messages are distributions over (0, 1, ∗), namely

ai (0), Q

ai (1), Q

ai (∗)). Notice that these messages are (Qia (0), Qia (1), Qia (∗)) and (Q only marginally more complicated than ordinary BP messages. Let P(input) denote the probability of a given input assuming independent warnings with distribution

ai ( · ). The probability of an outgoing message output ∈ {0, 1, ∗} is then Qia ( · ) or Q P(output) ∼ =

P(input)I(O; UT(input) = output)e−yδF

e

(input)

.

(19.97)

input

Depending on whether the node we are considering is a variable or function node, this

ai ( · ). probability distribution corresponds to the outgoing message Qia ( · ) or Q It can be shown that the Bethe energy (19.82) associated with a given fixed point of the WP equations coincides with the total number of contradictions. This can be expressed as the number of contradictions at function nodes, plus those at variable nodes, minus the number of edges (i, a) such that the warning in direction a → i contradicts the warning in direction i → a (the last term avoids double counting). It follows that the Bethe free entropy of the auxiliary graphical model FRSB,e (y) weights each WP fixed point depending on its number of contradictions, as it should. 19.5.4

XORSAT again

Let us now apply the SP(y) formalism to random K-XORSAT instances. We let the energy function E(x) count the number of unsatisfied linear equations: Ea (x∂a ) =

0 if xi1 (a) ⊕ · · · ⊕ xiK (a) = ba , 1 otherwise.

(19.98)

The simplifications discussed in the previous subsection apply to this case. The 1RSB population dynamics algorithm can be used to compute the free-entropy density Fe (y). Here we limit ourselves to describing the results of this calculation for the case K = 3. Let us stress that the problem we are considering here is different from the one investigated in Section 19.3. Whereas there we were interested in the uniform measure over solutions (thus focusing on the satisfiable regime α < αs (K)), here we are estimating the minimum number of unsatisfied constraints (which is most interesting in the unsatisfiable regime α > αs (K)). It is easy to show that the SP(y) equations always admit a solution in which Qia (∗) = 1 for all (i, a), indicating that the min-sum equations have a unique solution. This corresponds to a density-evolution fixed point where Q(∗) = 1 with probability 1, yielding an Fe (y) independent of y. For y smaller than an α-dependent threshold y∗ (α), this is the only solution that we find. For larger values of y, the SP(y) equations have a non-trivial solution. Figure 19.4 shows the result for the free-entropy density Fe (y) for three values of α.

Fe (y)

The 1RSB cavity method 0.10

0.07

0.08

0.06

0.06

0.05

0.04

Σe ()

0.04

0.02

0.03

0.00

0.02

-0.02

0.01

-0.04 0.0

0.5

1.0

1.5

2.0

y

2.5

3.0

3.5

4.0

0.00 0.00

0.02

0.04

0.06

0.08

0.10

Fig. 19.4 Random 3-XORSAT at α = 0.87, 0.97 and 1.07. Recall that, for K = 3, αd (K) ≈ 0.818 and αs (K) ≈ 0.918. Left: free-entropy density Fe (y) as a function of y, obtained using the population dynamics algorithm, with N = 2 × 104 and t = 5 × 103 (α increases from bottom to top). Right: complexity Σe () as a function of energy density (equal to the number of violated constraints per variable); α increases from left to right. Only the concave part of Σe () is physically meaningful.

Above this threshold, the density evolution converges to a ‘non-trivial’ 1RSB fixed point. The complexity functions Σe () can be deduced by taking the Legendre transform (see eqn (19.95)-, which requires differentiating Fe (y) and plotting (, Σe ) in parametric form. The derivative can be computed numerically in a number of ways: 1. Compute analytically the derivative of FRSB,e (y) with respect to y. This turns out

and can therefore to be a functional of the fixed-point distributions of Q and Q, be easily evaluated. 2. Fit the numerical results for the function Fe (y) and differentiate the fitting function 3. Approximate the derivative as a difference at nearby values of y. In the present case, we have followed the second approach using the parametric form Ffit (y) = a + b e−y + c e−2y + d e−3y . As shown in Fig. 19.4, the resulting parametric curve (, Σe ) is multiple-valued (this is a consequence of the fact that Fe (y) is not concave). Only the concave part of Σe () is retained as physically meaningful. Indeed, the convex branch is ‘unstable’ (in the sense that further RSB would be needed), and it is not yet understood whether it has any meaning. For α ∈ [αd (K), αs (K)[, Σe () remains positive down to = 0. The intercept Σe ( = 0) coincides with the complexity of clusters of SAT configurations, as computed in Chapter 18 (see Theorem 18.2). For α > αs (K) (in the UNSAT phase), Σe () vanishes at gs (K, α) > 0. The energy density gs (K, α) is the minimal fraction of violated equations in a random XORSAT linear system. Note that Σe () is not defined above a second energy density d (K, α). This indicates that we should take Σe () = −∞ there: above d (K, α), one recovers a simple problem with a unique Bethe measure. Figure 19.5 shows the values of gs (K, α) and d (K, α) as functions of α for K = 3 (random 3-XORSAT).

The nature of 1RSB phases

0.25 0.20 0.15

d (K, α)

0.10

gs (K, α) 0.05 0.00 0.6

0.8

1

α

1.2

1.4

Fig. 19.5 Asymptotic ground state energy (= minimal number of violated constraints) per variable, gs (K, α), for random K = 3-XORSAT formulae; gs (K, α) vanishes for α < αs (K). The dashed line shows d (K, α), which is the highest energy density e such that configurations with E(x) < N e are clustered. It vanishes for α < αd (K).

19.6


In the previous sections, we discussed how to compute the complexity function Σ(φ) (or its ‘zero-temperature’ version, the energetic complexity Σe ()). Here we want to come back to the problem of determining some qualitative properties of the measure µ( · ) for random graphical models, on the basis of its decomposition into extremal Bethe measures, wn µn (x) . (19.99) µ(x) = n∈E

Assumptions 2 and 3 imply that, in this decomposition, we introduce a negligible error if we drop all of the states n except the ones with free entropy φn ≈ φ∗ , where φ∗ = argmax {φ + Σ(φ) : Σ(φ) ≥ 0} .

(19.100)

In general, Σ(φ) is strictly positive and continuous in an interval [φmin , φmax ] with Σ(φmax ) = 0, and Σ(φ) = x∗ (φmax − φ) + O((φmax − φ)2 )

(19.101)

for φ close to φmax . It turns out that the decomposition (19.99) has different properties depending on the result of the optimization (19.100). One can distinguish two phases (see Fig. 19.6): d1RSB (dynamic one-step replica symmetry breaking), when the maximum is achieved in the interior of [φmin , φmax ] and, as a consequence Σ(φ∗ ) > 0; and s1RSB (static onestep replica symmetry breaking), when the maximum is achieved at φ∗ = φmax and therefore Σ(φ∗ ) = 0 (this case occurs iff x∗ ≤ 1).

The 1RSB cavity method Σ

T3

T1

T2 φ3

φ2

φ

1

Fig. 19.6 A sketch of the complexity Σ versus the free-entropy density φ in a finite-temperature problem with a 1RSB phase transition, at three temperatures T1 < T2 < T3 . A random configuration x with distribution µ(x) is found with high probability in a cluster with free-entropy density φ1 , φ2 , or phi3 , respectively. T2 and T3 are above the condensation transition: φ2 and φ3 are the points where ∂Σ/∂φ = −1. T1 is below the condensation transition: φ1 is the largest value of φ where Σ is positive.

19.6.1

Dynamical 1RSB

We assume here that Σ∗ = Σ(φ∗ ) > 0. We can then restrict the sum (19.99) to those states n such that φn ∈ [φ∗ − ε, φ∗ + ε], if we allow an exponentially small error. To the leading exponential order, there are eN Σ∗ such states whose weights are wn ∈ [e−N (Σ∗ +ε ) , e−N (Σ∗ −ε ) ]. Different states are expected to have essentially disjoint support. By this we mean that there exist subsets {Ωn }n∈E of the configuration space X N such that, for any m ∈ E, µm (Ωm ) ≈ 1 ,

wn µn (Ωm ) ≈ 0 .

(19.102)

n∈E\m

Further, different states are separated by ‘large free-energy barriers’. This means that one can choose the above partition in such a way that only an exponentially small (in N ) fraction of the probability measure is on its boundaries. This structure has two important consequences: Glassy dynamics. Let us consider a local Markov chain dynamics that satisfies detailed balance with respect to the measure µ( · ). As an example, we can consider the Glauber dynamics introduced in Chapter 4 (in order to avoid trivial reducibility effects, we can assume in this discussion that the compatibility functions ψa (x∂a ) are bounded away from 0). Imagine initiating the dynamics at time 0 with an equilibrated configuration x(0) distributed according to µ( · ). This is essentially equivalent to picking a state n uniformly at random from among the typical states, and then sampling x(0) from µn ( · ). Because of the exponentially large barriers, the dynamics will stay confined in Ωn for an exponentially large time, and equilibrate among states only on larger time scales. This can be formalized as follows. We denote by D(x, x ) the Hamming distance in N X . Take two i.i.d. configurations with distribution µ, and let N d0 be the expectation


value of their Hamming distance. Analogously, take two i.i.d. configurations with distribution µn , and let N d1 be the expectation value of their Hamming distance. When the state n is chosen randomly with distribution wn , we expect d1 not to depend on the state n, asymptotically, for large sizes. Furthermore, we expect d1 < d0 . We can then consider the (normalized) expected Hamming distance between configurations at time t in Glauber dynamics, d(t) = D(x(0), x(t))/N . For any ε < d0 − d1 , the correlation time τ (ε) ≡ inf{t : d(t) ≥ d0 − ε} is expected to be exponentially large in N. Short-range correlations in finite-dimensional projections. We described the motivation for the 1RSB cavity method by considering the emergence of long-range correlations due to decomposition of µ( · ) into many extremal Bethe measures. Surprisingly, such correlations cannot be detected by probing a bounded (when N → ∞) number of variables. More precisely, if i(1), . . . , i(k) ∈ {1, · · · , N } are uniformly random variable indices, then, in the d1RSB phase, N →∞

E|f1 (xi(1) )f2 (xi(2) ) · · · fk (xi(k) ) − f1 (xi(1) )f2 (xi(2) ) · · · fk (xi(k) )| → 0 . (Here · denotes the expectation with respect to the measure µ, and E denotes the expectation with respect to the graphical model for a random ensemble.) This finding can be understood intuitively as follows. If there are long-range correlations among subsets of k variables, then it must be true that conditioning on the values of k − 1 of those variables changes the marginal distribution of the k-th one. On the other hand, we think that long-range correlations arise because far-apart variables ‘know’ that the whole system is in the same state n. But conditioning on a bounded number (k − 1) of variables cannot select in any significant way from among the eN Σ∗ relevant states, and thus cannot change the marginal of the k-th variable. An alternative argument makes use of the observation that if x(1) and x(2) are two i.i.d. configurations with distribution µ( · ), then their distance D(x(1) , x(2) ) concentrates in probability. This is due to the fact that the two configurations will be, with high probability, in different states n1 = n2 (the probability of n1 = n2 being e−N Σ∗ ), whose distance depends weakly on the pair of states. Let us note, finally, that the absence of long range correlations among bounded subsets of variables is related to the observation that µ( · ) is itself a Bethe measure (although a non-extremal one) in a d1RSB phase (see Section 19.4.1). Indeed, each BP equation involves a bounded subset of the variables and can be violated only because of correlations among them. As we shall discuss in Section 22.1.2, long-range correlations in a d1RSB phase can be probed through more sophisticated ‘point-to-set’ correlation functions. 19.6.2

Static 1RSB

In this case the decomposition (19.99) is dominated by a few states of near-to-maximal free entropy φn ≈ φmax . If we ‘zoom in’ near the edge by letting φn = φmax + sn /N , then the ‘free-entropy shifts’ sn form a point process with density exp(−x∗ s). The situation is analogous to the one we found in the random energy model for T < Tc . Indeed, it is expected that the weights {wn } will converge to the same universal


Poisson-Dirichlet process found there, and to depend on the details of the model only through the parameter x∗ (we have already discussed this universality using replicas in Chapter 8). In particular, if x(1) and x(2) are two i.i.d. replicas with distribution µ, and n1 and n2 are the states that they belong to, then the probability for them to belong to the same state is $ # 2 wn = 1 − x∗ . (19.103) E {Pµ (n1 = n2 )} = E n∈E

Here E denotes the expectation with respect to the graphical-model distribution. As a consequence, the distance D(x(1) , x(2) ) between two i.i.d. replicas does not concentrate (the overlap distribution is non-trivial). This, in turn, can only be true if the two-point correlation function does not vanish at large distances. Long-range correlations of this type make BP break down. The original graphical model µ( · ) is no longer a Bethe measure: its local marginals cannot be described in terms of a set of messages. The 1RSB description, according to which µ( · ) is a convex combination of Bethe measures, is unavoidable. At this point, we are left with a puzzle. How do we circumvent the argument given in Section 19.4.1 that, if the ‘correct’ weight x = 1 is used, then the marginals as computed within 1RSB still satisfy the BP equations? The conundrum is that, within an s1RSB phase, the parameter x = 1 is not the correct one to use in the 1RSB cavity equations (although it is the correct one for weighting states). In order to explain this, let us first note that, if the complexity is convex and behaves as in eqn (19.101) near its edge, with a slope −x∗ > −1, then the optimization problem (19.100) has the same result as φ∗ = argmax {xφ + Σ(φ) : Σ(φ) ≥ 0} ,

(19.104)

for any x ≥ x∗ . Therefore, in the 1RSB cavity equations, we could in principle use any value of x greater than or equal to x∗ (this would select the same states). However, the constraint Σ(φ) ≥ 0 cannot be enforced locally, and does not show up in the cavity equations. If one performs the computation of Σ within the cavity method using a value x > x∗ , then one finds a negative value of Σ, which must be rejected (it is believed to be related to the contribution of some exponentially rare instances). Therefore, in order to ensure that one studies the interval of φ such that Σ(φ) ≥ 0, one must impose x ≤ x∗ in the cavity method. In order to select the states with free-entropy density φmax , we must thus choose the Parisi parameter that corresponds to φmax , namely x = x∗ . 19.6.3

When does 1RSB fail?

The 1RSB cavity method is a powerful tool, but does not always provide correct answers, even for locally tree-like models, in the large-system limit. The main assumption of the 1RSB approach is that once we pass to the auxiliary graphical model (which ‘enumerates’ BP fixed points), a simple BP procedure is asymptotically exact. In other words, the auxiliary problem has a simple ‘replica-symmetric’ structure and no glassy

Appendix: The SP(y) equations for XORSAT

phase. This is correct in some cases, such as random XORSAT and SAT close to their SAT–UNSAT threshold, but it may fail in others. One mechanism leading to a failure of the 1RSB approach is that the auxiliary graphical model is incorrectly described by BP. This may happen because the measure of the auxiliary model decomposes into many Bethe states. In such a case, one must introduce a second auxiliary model, dealing with the multiplicity of BP fixed points in the first auxiliary model. This is usually referred to as ‘two-step replica symmetry breaking’ (2RSB). Obviously one can find situations in which it is necessary to iterate this construction, leading to an R-th level auxiliary graphical model (R-RSB). Continuous (or full) RSB corresponds to the large-R limit. While such developments are conceptually clear (at least from a heuristic point of view), they are technically challenging. So far, only limited results have been obtained beyond 1RSB. A brief survey is given in Chapter 22.

19.7

Appendix: The SP(y) equations for XORSAT

This appendix provides some technical details of the 1RSB treatment of random KXORSAT within the ‘energetic’ formalism. The results of this approach were discussed in Section 19.5.4. In particular, we shall derive the behaviour of the auxiliary free entropy Fe (y) at large y, and deduce the behaviour of the complexity Σe () at small . This section can be regarded as an exercise in applying the SP(y) formalism. We shall skip many details and just give the main intermediate results of the computation. XORSAT is a constraint satisfaction problem with binary variables. We can thus apply the simplified method of Section 19.5.3. The projected min-sum messages can take three values: 0, 1, ∗. By exploiting the symmetry of XORSAT between 0 and 1, SP(y) messages can be parameterized by a single number, for instance the sum of their weights on 0 and 1. We shall therefore write Qia (0) = Qia (1) = ζia /2 (thus

ai (0) = Q

ai (1) = ηai /2 (whence Q

ai (∗) = 1 − ηai ). implying Qia (∗) = 1 − ζia ) and Q In terms of these variables, the SP(y) equation at function node a reads

ηai =

ζja .

(19.105)

j∈∂a\i

The SP(y) equation at variable node i is a little more complicated. Let us consider

bi , b ∈ ∂i \ a. Each of them is parameterized by all of the |∂i| − 1 incoming messages Q a number ηbi . We let η = {ηbi , b ∈ ∂i \ a} and define the function Bq (η) as follows: Bq (η) =

I(|S| = q)

S⊂{∂i\a}

(1 − ηbi )

ηcj .

(19.106)

c∈S

b∈∂i\{S∪{a}}

q+r

Let Aq,r (η) = Bq+r (η) update equation:

ζia = (|∂i|−1)/2 q=0

2−(q+r) . After some thought, one can obtain the following

q

2

|∂i|−2 |∂i|−1 q=0

Aq,r (η)e−yq |∂i|−2 |∂i|−1

r=q+1

Aq,q (η)e−yq + 2

q=0

r=q+1

Aq,r (η)e−yq

.

(19.107)


The auxiliary free entropy FRSB,e (y) has the general form (19.91), with the various contributions expressed as follows in terms of the parameters {ζia , ηai }: RSB,e

eFa

1 = 1 − (1 − e−y ) ζia , 2

RSB,e

eFai

i∈∂a

RSB,e

eF i

=

di d i −q

1 = 1 − ηai ζia (1 − e−y ) , 2

Aq,r ({ηai }a∈∂i ) e−y min(q,r) .

(19.108)

q=0 r=0

Let us consider random K-XORSAT instances with a constraint density α. Equations (19.105) and (19.107) are promoted to distributional relations that determine the asymptotic distribution of η and ζ on a randomly chosen edge (i, a). The 1RSB population dynamics algorithm can be used to approximate these distributions. We encourage the reader to implement it, and obtain a numerical estimate of the auxiliary free-entropy density Fe (y). It turns out that, at large y, one can control the distributions of η and ζ analytically, provided their qualitative behaviour satisfies the following assumptions (which can be checked numerically): • With probability t one has η = 0, and with probability 1−t, η = 1−e−y ηˆ, where t has a limit in ]0, 1[, and ηˆ converges to a random variable with support on [0, ∞[, as y → ∞. ˆ • With probability s, one has ζ = 0, and with probability 1 − s, ζ = 1 − e−y ζ, ˆ where s has a limit in ]0, 1[, and ζ converges to a random variable with support on [0, ∞[, as y → ∞. Under these assumptions, we shall expand the distributional version of eqns (19.105) and (19.107) keeping terms up to first order in e−y . We shall use t, s, ηˆ, ζˆ to denote the limit quantities mentioned above. It is easy to see that t and s must satisfy the equations (1 − t) = (1 − s)k−1 and

∗ and s = e−Kα(1−t) . These are identical to eqns (19.50) and (19.51), whence t = 1 − Q s = 1 − Q∗ . Equation (19.105) leads to the distributional equation d

ηˆ = ζˆ1 + · · · + ζˆK−1 ,

(19.109)

ˆ where ζˆ1 , . . . , ζˆK−1 are K − 1 i.i.d. copies of the random variable ζ. The update equation (19.107) is more complicated. There are, in general, l inputs to a variable node, where l is Poisson-distributed with mean Kα. Let us denote by m the number of incoming messages with η = 0. The case m = 0 yields ζ = 0 and is taken care of in the relation between t and s. If we condition on m ≥ 1, the distribution of m is 1 λm −λ e I(m ≥ 1) , (19.110) P(m) = m! 1 − e−λ where λ = Kα(1 − t). Conditional on m, eqn (19.107) simplifies as follows: d • If m = 1, ζˆ = ηˆ.

Notes

• If m = 2, ζˆ = 1 identically. • If m ≥ 3, ζˆ = 0 identically. The various contributions to the free entropy (19.37) are given by 5 6 ˆ + o(e−y ) , ffRSB,e = (1 − s)k − log 2 + e−y (1 + Kζ) ( λ2 −λ ' e − log 2 + e−y (1 + 2ˆ η ) 2 ∞ ( λm −λ ' e (1 − m) log 2 + e−y m(1 + ˆ η ) + o(e−y ) , + m! m=3 5 6 ˆ + o(e−y ) , = (1 − t)(1 − s) − log 2 + e−y (1 + ˆ η + ζ)

(19.111)

fvRSB,e =

feRSB,e

(19.112) (19.113)

ˆ are the expectation values of ηˆ and ζ. ˆ This gives the free-entropy where ˆ η and ζ RSB,e e RSB,e RSB,e −y + αfv − Kα fe = Σ0 + e 0 + o(e−y ), where density F (y) = ff k−1 λ λ log 2 , (19.114) Σ0 = 1 − − e−λ 1 + k k k λ 1 − e−λ 1 + λ . (19.115) 0 = k 2 Taking the Legendre transform (see eqn (19.95)), we obtain the following behaviour of the energetic complexity as → 0: Σe () = Σ0 + log

0 e + o() ,

(19.116)

This shows, in particular, that the ground state energy density is proportional to (α − αs )/| log(α − αs )| close to the SAT–UNSAT transition (when 0 < α − αs 1).

Exercise 19.7 In the other extreme case, show that at large α one gets gs (K, α) = α/2 + √ √ 2α∗ (K) + o( α), where the positive constant ∗ (K) is the absolute value of the ground state energy of the fully connected K-spin model studied in Section 8.2. This indicates that there is no interesting intermediate asymptotic regime between M = Θ(N ) (discussed in the present chapter) and M = Θ(N K−1 ) (discussed with the use of the replica method in Chapter 8).

Notes The cavity method originated as an alternative to the replica approach in the study of the Sherrington–Kirkpatrick model (Mézard et al., 1985b). The 1RSB cavity method for locally tree-like factor graphs was developed in the context of spin glasses by Mézard and Parisi (2001). Its application to zero-temperature problems (counting solutions of the min-sum equations) was also first described in the context of spin glasses, by Mézard and Parisi (2003). The presentation in this chapter differs in its scope from


those studies, which were more focused on computing averages over random instances. For a rigorous treatment of the notion of a Bethe measure, we refer to Dembo and Montanari (2008b). The idea that the 1RSB cavity method is in fact equivalent to applying BP on an auxiliary model appeared in several papers treating the cases of colouring and satisfiability with y = 0 (Parisi, 2002; Braunstein and Zecchina, 2004; Maneva et al., 2005). The treatment presented here generalizes these studies, with the important difference that the variables of our auxiliary model are messages on the edges rather than quantities defined on vertices. The analysis of the x = 1 case is strictly related to the problem of reconstruction on a tree. This has been studied by Mézard and Montanari (2006), where the reader will find the proof of Theorem 19.5 and the expression for the free entropy obtained in Exercise 19.6. The SP(y) equations for one single instance were written down first in the context of the K-satisfiability problem by Mézard and Zecchina (2002); see also Mézard et al. (2003a). The direct derivation of the SP(y) equations for binary-variable problems described in Section 19.5.3 was done originally for the satisfiability problem by Braunstein et al. (2005); see also Braunstein and Zecchina (2004) and Maneva et al. (2005). The application of the 1RSB cavity method to the random XORSAT problem, and its comparison with the exact results, was done by Mézard et al. (2003b). An alternative to the cavity approach followed throughout this book is provided by the replica method of Chapter 8. As we saw, the replica method was first invented in order to treat fully connected models (i.e. models on complete graphs) (see Mézard et al., 1987), and was subsequently developed in the context of sparse random graphs (Mézard and Parisi, 1985; De Dominicis and Mottishaw, 1987; Mottishaw and De Dominicis, 1987; Wong and Sherrington, 1988; Goldschmidt and Lai, 1990). The technique was further improved in Monasson (1998), which offers a very lucid presentation of the method.

20 Random K-satisfiability This chapter applies the cavity method to the random K-satisfiability problem. We shall study both the phase diagram (in particular, we shall determine the SAT–UNSAT threshold αs (K)) and the algorithmic applications of message passing. The whole chapter is based on heuristic derivations: the rigorization of the whole approach is still in its infancy. Neither the conjectured phase diagram nor the efficiency of messagepassing algorithms has yet been confirmed rigorously. But the computed value of αs (K) is conjectured to be exact, and the low-complexity message-passing algorithms that we shall describe turn out to be particularly efficient in finding solutions. We start in Section 20.1 by writing the BP equations, following the approach set out in Chapter 14. The statistical analysis of such equations provides a first (replicasymmetric) estimate of αs (K). This, however, turns out to be incorrect. The reason for this failure is traced back to the incorrectness of the replica-symmetric assumption close to the SAT–UNSAT transition. The system undergoes a ‘structural’ phase transition at a clause density smaller than αs (K). Nevertheless, BP empirically converges for a wide range of clause densities, and it can be used to find SAT assignments for large instances provided the clause density α is not too close to αs (K). The key idea is to use BP as a heuristic guide in a sequential decimation procedure. In Section 20.2, we apply the 1RSB cavity method developed in Chapter 19. A statistical analysis of the 1RSB equations gives the values for αs (K) summarized in Table 20.1. From the algorithmic point of view, one can use SP instead of BP as a guide in the decimation procedure. We shall explain and study numerically the corresponding ‘survey-guided decimation’ algorithm, which is presently the most efficient algorithm for finding SAT assignments in large random satisfiable instances with a clause density close to the threshold αs (K). This chapter focuses on K-SAT with K ≥ 3. The K = 2 problem is quite different: satisfiability can be proved in polynomial time, the SAT–UNSAT phase transition is driven by a very different mechanism, and the threshold is known to be αs (2) = 1. It turns out that a (more subtle) qualitative difference also distinguishes K = 3 from K ≥ 4. In order to illustrate this point, we shall use both 3-SAT and 4-SAT as running examples. The problem of colouring random graphs turns out to be very similar to that of random K-satisfiability. Section 20.4 presents a few highlights in the study of random graph colourings. In particular, we emphasize how the techniques used for Ksatisfiability are successful in this case as well.

Random K-satisfiability

20.1

Belief propagation and the replica-symmetric analysis

We have already studied some aspects of random K-SAT in Chapter 10, where we derived, in particular, some rigorous bounds on the SAT–UNSAT threshold αs (K). Here we shall study the problem using message-passing approaches. Let us start by summarizing our notation. An instance of the K-satisfiability problem is defined by M clauses (indexed by a, b, . . . ∈ {1, . . . , M }) over N Boolean variables x1 , . . . , xN taking values in {0, 1}. We denote by ∂a the set of variables in clause a, and by ∂i the set of clauses in which variable xi appears. Further, for each i ∈ ∂a, we introduce a number Jai , which takes the value 1 if xi occurs negated in clause a, and takes the value 0 if the variable occurs unnegated. It will be convenient to distinguish elements of ∂a according to the value of Jai . We let ∂0 a ≡ {i ∈ ∂a such that Jai = 0} and ∂1 a = {i ∈ ∂a such that Jai = 1}. Similarly we denote by ∂0 i and ∂1 i the neighbourhoods of i: ∂0 i = {a ∈ ∂i such that Jai = 0} and ∂1 i = {a ∈ ∂i such that Jai = 1}. As usual, the indicator function over clause a being satisfied is denoted by ψa ( · ): ψa (x∂a ) = 1 if clause a is satisfied by the assignment x, and ψa (x∂a ) = 0 if it is not. Given a SAT instance, we begin by studying the uniform measure over SAT assignments, µ(x) =

M 1 ψa (x∂a ) . Z a=1

(20.1)

We shall represent this distribution with a factor graph, as in Fig. 10.1, and in this graph we draw dashed edges when Jai = 1 and full edges when Jai = 0. 20.1.1

The BP equations

The BP equations for a general model of the form (20.1) have already been written down in Chapter 14. Here we want to rewrite them in a more compact form that is convenient for both analysis and implementation. They are best expressed using the following notation. Consider a variable node i connected to a factor node a and partition its neighbourhood as ∂i = {a} ∪ Sia ∪ Uia , where (see Fig. 20.1) if Jai = 0 then if Jai = 1 then

Sia = ∂0 i \ {a}, Uia = ∂1 i , Sia = ∂1 i \ {a}, Uai = ∂0 i .

(20.2)

Since the variables xi are binary, the BP messages νi→a ( · ) and ν a→i ( · ) at any time can be parameterized by a single real number. We fix the parameterization by letting ζia ≡ νi→a (xi = Jai ) (which obviously implies νi→a (xi = 1 − Jai ) = 1 − ζia ) and ζâi ≡ ν a→i (xi = Jai ) (yielding ν a→i (xi = 1 − Jai ) = 1 − ζâi ). A straightforward calculation allows us to express the BP equations (here in fixedpoint form) in terms of these variables:


Uja

Uia

Sia

Sja

i j

a Fig. 20.1 The set Sia contains all checks b in ∂i \ a such that Jbi = Jai ; the set Uia contains all checks b in ∂i \ a such that Jbi = 1 − Jai .

5

6 5

6 − ζˆbi ) 6 5 6 5 6 5 6, ζia = 5 ˆ ˆ ˆ ˆ b∈Sia ζbi b∈Uia (1 − ζbi ) + b∈Uia ζbi b∈Sia (1 − ζbi ) 1 − j∈∂a\i ζja , (20.3) ζâi = 2 − j∈∂a\i ζja b∈Sia

ζˆbi

b∈Uia (1

with the convention that a product over zero terms is equal to 1. Note that evaluating the right-hand sides takes O(|∂i|) and O(|∂a|) operations, respectively. This should be contrasted with the general implementation of the BP equations (see Chapter 14), |∂a| ) at function which requires O(|∂i|) operations at variable nodes but nodes. O(2 The Bethe free entropy takes the usual form, F = a∈F Fa + i∈V Fi − (ia)∈E Fia (see eqn (14.27)). The various contributions can be expressed in terms of the parameters ζia , ζâi as follows: Fa = log 1 − ζia , Fi = log (1 − ζˆbi ) + (1 − ζâi ) ζâi ζˆbi , Fai

5

i∈∂a

a∈∂0 i

= log (1 − ζia )(1 − ζâi ) + ζia ζâi

6

b∈∂1 i

a∈∂0 i

.

Given the messages, the BP estimate for the marginal on site i is νi (xi ) ∼ ν a→i (xi ) . =

b∈∂1 i

(20.4)

(20.5)

a∈∂i

20.1.2

Statistical analysis

Let us now consider a random K-SAT formula, i.e. a uniformly random formula with N variables and M = N α clauses. The resulting factor graph will be distributed


according to the ensemble GN (K, M ). Given a variable index i, the numbers |∂0 i| and |∂1 i| of variables in which xi occurs unnegated or negated converge to independent Poisson random variables of mean Kα/2. If (i, a) is a uniformly random edge in the graph, the corresponding fixed-point messages ζia and ζâi are random variables (we assume here that an ‘approximate’ fixed point exists). Within the RS assumption, they converge in distribution, as N → ∞, to random variables ζ and ζˆ whose distributions satisfy the RS distributional equations d 1 − ζ1 . . . ζK−1 , ζˆ = 2 − ζ1 . . . ζK−1 d

ζ=

(20.6)

ζˆ1 . . . ζˆp (1 − ζˆp+1 ) . . . (1 − ζˆp+q ) . ζˆ1 . . . ζˆp (1 − ζˆp+1 ) . . . (1 − ζˆp+q ) + (1 − ζˆ1 ) . . . (1 − ζˆp )ζˆp+1 . . . ζˆp+q

(20.7)

Here p and q are two i.i.d. Poisson random variables with mean Kα/2 (corresponding to the sizes of S and U), ζ1 , . . . , ζK−1 are i.i.d. copies of ζ, and ζˆ1 , . . . , ζˆp+q are i.i.d. ˆ copies of ζ. The distributions of ζ and ζˆ can be approximated using the population dynamics algorithm. The resulting samples can then be used to estimate the free-entropy density, as outlined in the exercise below.

Exercise 20.1 Argue that, within the RS assumptions, the large-N limit of the Bethe free-entropy density is given by limN →∞ F/N = f RS = fvRS + αfcRS − KαfeRS , where " fvRS = E log

p Y

a=1

fcRS feRS

Y

p+q

ζâ

a=p+1

(1 − ζâ ) +

p Y a=1

= E log [1 − ζ1 · · · ζK−1 ] , i h = E log (1 − ζ1 )(1 − ζˆ1 ) + ζ1 ζˆ1 .

Y

p+q

(1 − ζâ )

# ζâ ,

a=p+1

(20.8)

Here E denotes the expectation with respect to ζ1 , . . . , ζK , which are i.i.d. copies of ζ; ˆ and p and q, which are i.i.d. Poisson random ζˆ1 , . . . , ζˆp+q , which are i.i.d. copies of ζ; variables with mean Kα/2.

Figure 20.2 shows an example of the entropy density found within this approach for 3-SAT. For each value of α in a mesh, we used a population of size 104 , and ran the algorithm for 3 × 103 iterations. Messages were initialized uniformly in ]0, 1[, and the first 103 iterations were not used for computing the free entropy. The predicted entropy density is strictly positive and decreasing in α for α ≤ α∗ (K), with α∗ (3) ≈ 4.6773. Above α∗ (K), the RS distributional equations do not seem to admit any solution with ζ, ζˆ ∈ [0, 1]. This is revealed numerically by the fact that the denominator of eqn (20.7) vanishes during the population updates. Since we find an RS entropy density which is positive for all α < α∗ (K), the value α∗ (K) is the RS prediction for the SAT–UNSAT threshold. It turns out that α∗ (K) can be computed without population dynamics, as outlined by the exercise below.


0.7 0.6 0.5 0.4

f RS (α) 0.3 0.2 0.1 0 -0.1 0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

α Fig. 20.2 RS prediction for the asymptotic entropy density of random 3-SAT formulae, plotted versus the clause density α for 3-SAT. The result is expected to be correct for α ≤ αc (3) = αd (3) ≈ 3.86.

Exercise 20.2 How to compute α∗ (K)? The idea is that, above this value of the clause density, any solution of the RS distributional equations has ζˆ = 0 with positive probability. In this case the denominator of eqn (20.7) vanishes with positive probability, leading to a contradiction. We start by regularizing eqn (20.7) with a small parameter . Each ζî is replaced by max(ζî , ). Let us denote by x the probability that ζˆ is of order , and by y the probability that ζ is of order 1 − . Consider the limit → 0. (a) Show that x = y K−1 (b) Show that 1 − 2y = e−Kαx I0 (Kαx), where I0 (z) is the Bessel function with Taylor P 2p 2 expansion I0 (t) = ∞ p=0 (1/p! ) (t/2) . [Hint: Suppose that there are p variables among ζˆ1 . . . ζˆp and q among ζˆp+1 . . . ζˆp+q that are of order . Show that this update equation gives ζ = O() if p > q , ζ = 1−O() if p < q , and ζ = O(1) if p = q . ] (c) Let α∗ (K) be the largest clause density such that the two equations derived in (a) and (b) admit the unique solution x = y = 0. Show that, for α ≥ α∗ (K), a new solution appears with x, y > 0. (d) By solving the above equations numerically, show that α∗ (3) ≈ 4.6673 and α∗ (4) ≈ 11.83.

Unhappily, this RS computation is incorrect when α is large enough, and, as a consequence, the prediction for the SAT–UNSAT phase transition is wrong as well. In particular, it contradicts the upper bound αUB,2 (K) found in Chapter 10 (for instance, in the two cases K = 3 and 4, one has αUB,2 (3) ≈ 4.66603 < α∗ (3) and αUB,2 (4) ≈ 10.2246 < α∗ (4)). The largest α such that the RS entropy density is correct is nothing but the value for the condensation transition αc (K). We shall discuss


1.0

1.0

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0.0

0.0 3.6

3.8

4.0 α

4.2

4.4

8.5

9

9.5

10

10.5

11

α

Fig. 20.3 Empirical probability that BP converges to a fixed point, plotted versus the clause density α, for 3-SAT (left plot) and 4-SAT (right plot). The statistics are over 100 instances, with N = 5 × 103 variables (dashed curve) and N = 104 variables (full curve). There is an indication of a phase transition occurring for αBP ≈ 3.85 (K = 3) and αBP ≈ 10.3 (K = 4). The data points show the empirical probability that BP-guided decimation finds a SAT assignment, computed over 100 instances with N = 5 × 103 . The vertical lines correspond to the SAT–UNSAT threshold.

this phase transition further below and in Chapter 22. There is another way to realize that something is wrong with the RS assumption close to the SAT–UNSAT phase transition. The idea is to look at the BP iteration. 20.1.3

BP-guided decimation

The simplest experiment consists in iterating the BP equations (20.3) on a randomly generated K-SAT instance. We start from uniformly random messages, and choose the following convergence criterion, defined in terms of a small number δ: the iteration is halted at the first time t∗ (δ) such that no message has changed by more than δ over the last iteration. If a large time tmax is fixed, one can estimate the probability of convergence within tmax iterations by repeating the same experiment many times. Figure 20.3 shows this probability for δ = 10−2 and tmax = 103 , plotted versus α. The probability curves show a sharp decrease around a critical value of α, αBP , which is robust to variations of tmax and δ. This numerical result is indicative of a threshold behaviour: the typical convergence time t∗ (δ) stays finite (or grows moderately) with N when α < αBP . Above αBP , BP fails to converge in a time tmax on a typical random instance. When it converges, BP can be used to find a SAT assignment, using it as a heuristic guide for a sequential decimation procedure. Each time the value of a new variable has to be fixed, BP is iterated until the convergence criterion, with parameter δ, is met (alternatively, one may be more bold and use the BP messages after a time tmax even when they have not converged). Then one uses the BP messages in order to decide (i) which variable to fix, and (ii) what value the variable should take. In the present implementation, these decisions were taken on the basis of a simple statistic: the bias of the variables. Given the BP estimate νi ( · ) of the marginal of xi , we define the bias as πi ≡ νi (0) − νi (1). The algorithm is given below.


BP-Guided Decimation (SAT formula F, accuracy , iterations tmax ) 1: For all n ∈ {1, . . . , N }: 2: Call BP(F, , tmax ); 3: If BP does not converge, return ‘NOT found’ and exit; 4: For each variable node j, compute the bias πj ; 5: Find a variable i(n) with the largest absolute bias |πi(n) |; 6: If πi(n) ≥ 0, fix xi(n) to x∗i(n) = 0; 7: Otherwise, fix xi(n) to x∗i(n) = 1; 8: Replace F by the formula obtained after this reduction 9: End-For; 10: Return the assignment x∗ A pseudocode for BP was given in Section 14.2. Let us emphasize that the same decimation procedure could be used not only with BP but also with other types of guidance, if we had some way to estimate the marginals. The empirical success probability of the BP-guided decimation algorithm on random formulae is shown in Fig. 20.3 (estimated from 100 instances of size N = 5 × 104 ) for several values of α. The qualitative difference between 3-SAT and 4-SAT emerges clearly from this data. For 3-SAT, the decimation procedure returns a SAT assignment almost every time it converges, i.e. with probability close to one for α 3.85. For 4-SAT, BP converges most of the time if α 10.3. This value is larger than the conjectured SAT–UNSAT threshold αs (4) ≈ 9.931 (and also larger than the best rigorous upper bound αUB,2 (4) ≈ 10.2246.) On the other hand, the BP-guided decimation finds SAT assignments only when α 9.25. It is believed that the cases K ≥ 5 behave like K = 4. 20.1.4

On the validity of the RS analysis

These experiments suggest that something is not correct in the RS assumptions when α is large enough. The precise mechanism by which they are incorrect depends, however, on the value of K. For K = 3, the BP fixed point become unstable, and this leads to errors in decimations. In fact, the local stability of the BP fixed point can be computed along the lines of Section 17.4.2. The result is that it become unstable at αst (3) ≈ 3.86. In contrast, for K ≥ 4 the fixed point remains stable but does not correspond to the correct marginals. Local stability is not a good enough test in this case. Correspondingly, one can define two types of threshold: (i) A stability threshold αst (K), beyond which BP does not have a locally stable fixed point. (ii) A 1RSB condensation threshold αc (K), beyond which there is no BP fixed point that gives a correct estimate of the local marginals and free entropy. Clearly, one should have αc (K) ≤ αst (K). Our study suggests that αc (3) = αst (3) 3.86, whereas, for K ≥ 4, there is a strict inequality αc (K) < αst (K). The reason for the failure of BP is the decomposition of the measure (20.1) into many pure states. This happens at a third critical value αd (K) ≤ αc (K), referred to as the dynamical transition, in accordance with our discussion of spin glasses in


Section 12.3: αd (K) is the critical clause density above which Glauber dynamics will become inefficient. If αd (K) < α < αc (K), one expects, as we discussed in Section 19.4.1, that there will exist many pure states, and many quasi-solutions to the BP equations, among which one will give the correct marginals. At this point, the reader might well be discouraged. This is understandable: we started by seeking one threshold (the SAT–UNSAT transition αs (K)) and rapidly ended up defining a number of other thresholds, αd (K) ≤ αc (K) ≤ αst (K) ≤ αs (K), to describe a zoology of exotic phenomena. It turns out that, while an understanding of the proliferation of pure states is necessary to get the correct value of αs (K), one does not need a detailed description of the clusters, which is a challenging task. Luckily, there exists a shortcut, through the use of the energetic cavity method. It turns out that the sketchy description of clusters that we get from this method, as if looking at them from far away, is enough to determine αs . Even more than that, the sketch is a very useful and interesting one. In Section 20.3, we shall discuss a more detailed picture obtained through the fully fledged 1RSB cavity method applied to the model (20.1).

20.2

Survey propagation and the 1RSB phase

The use of the energetic 1RSB cavity method can be justified in two ways. From the first point of view, we are changing the problem. Instead of computing marginals of the distribution (20.1), we consider the problem of minimizing the energy function

E(x) =

M

Ea (x∂a ) .

(20.9)

a=1

Here Ea (x∂a ) = 0 if clause a is satisfied by the assignment x, and Ea (x∂a ) = 1 otherwise. The SAT–UNSAT threshold αs (K) is thus identified as the critical value above which the ground state energy min E(x) vanishes. With the cavity method, we shall estimate the ground state energy density, and find that it vanishes below some threshold. This is then identified as αs (K). This identification amounts to assuming that, for generic large random K-SAT problems, there is no interval of α where the ground state energy is positive but sublinear in N . This assumption is reasonable, but of course it does not hold in more general situations. If, for instance, we added to a random K-SAT formula a small unsatisfiable subformula (including o(N ) variables), our approach would not detect the change, whereas the formula would always be unsatisfiable. For α < αs (K), the cavity method provides a rough picture of zero-energy pure states. This brings us to the second way of justifying this ‘sketch’. We saw that describing a pure (Bethe) state in a locally tree-like graph amounts to assigning a set of cavity messages, i.e. of marginal distributions for the variables. The simplified description of the energetic 1RSB method only distinguishes between marginals that are concentrated on a single value and marginals that are not. The concentrated marginals are described exactly, while the other ones are just summarized by a single statement, ‘not concentrated’.


20.2.1

The SP(y) equations

The satisfiability problem involves only hard constraints and binary variables. We can thus use the simplified SP(y) equations of Section 19.5.3. The messages are triples:

ai (0), Q

ai (1), Q

ai (∗)) (Qia (0), Qia (1), Qia (∗)) for variable-to-function messages, and (Q for function-to-variable messages. In the case of K-satisfiability, these can be further simplified. The basic observation

ai (1) = 0, and if Jai = 1, then Q

ai (0) = 0. This can be shown is that if Jai = 0, then Q either by starting from the general formalism in Section 19.5.3 or by reconsidering the interpretation of warning propagation messages. Recall that a ‘0’ message means that the constraint a ‘forces’ variable xi to take the value 0 in order to minimize the system’s energy. In K-SAT this can happen only if Jai = 0, because xi = 0 is then the value that satisfies the clause a. With this remark in mind, the function-to-variablenode message can be parameterized by a single real number. We shall choose this

ai (1) if Jai = 1, and denote it by Q

ai . This

ai (0) if Jai = 0 and Q number to be Q

number Qai is the probability that there is a warning sent from a to i which forces the value of variable xi . Analogously, it is convenient to adopt a parameterization of the variable-to-function message Qia (m) which takes into account the value of Jai . More precisely, recall that Qia is supported on three types of messages: m(0) = 0, m(1) > 0; m(0) = m(1) = 0; or m(0) > 0, m(1) = 0. Let us denote the corresponding weights by Qia (0), Qia (∗) and Qia (1). If Jai = 0, we then define QSia ≡ Qia (0), Q∗ia ≡ Qia (∗), and QU ia ≡ Qia (1). Conversely, if Jai = 1, we let QSia ≡ Qia (1), Q∗ia ≡ Qia (∗), and QU ia ≡ Qia (0). We summarize this notation below with the corresponding interpretations. We emphasize that ‘probability’ refers here to the random choice of a pure state (see Section 19.1). • • • •

QSia : QU ia : Q∗ia :

ai : Q

probability probability probability probability

that that that that

xi xi xi xi

is is is is

forced by the clauses b ∈ ∂i \ a to satisfy a; forced by the clauses b ∈ ∂i \ a to violate a; not forced by the clauses b ∈ ∂i \ a; forced by clause a to satisfy it.

The 1RSB cavity equations have been written in Section 19.5.3.

b Exercise 20.3 Write the 1RSB equations explicitly in terms of the messages QS , QU , Q∗ , Q by applying the procedure of Section 19.5.3.

Alternatively, the 1RSB cavity equations can be guessed, bearing the above interpretation in mind. Clause a forces variable xi to satisfy it if and only if all of the other variables entering into clause a are forced (by some other clause) not to satisfy a. This means that

ai = QU (20.10) Q ja . j∈∂a\i


Consider, on the other hand, variable node i, and assume for definiteness that Jia = 0 (the opposite case gives rise to identical equations). Remember that, in this case, Sia denotes the subset of clauses b = a in which Jib = 0, and Uia the subset in which Jib = 1. Assume that the clauses in ΩS ⊆ Sia and ΩU ⊆ Uia force xi to satisfy them. Then xi is forced to either satisfy or violate a, depending whether |ΩS | > |ΩU | or |ΩS | < |ΩU |. Finally, xi is not forced if |ΩS | = |ΩU |. The energy shift is equal to the number of ‘forcing’ clauses in ∂i \ a that are violated when xi is chosen to satisfy the largest number of them, namely min(|ΩU |, |ΩS |). We thus get the equations S ∼

bi ) ,

bi QU e−y|Ω | (1 − Q (20.11) Q ia = |ΩU |>|ΩS |

QSia ∼ =

b∈ΩU ∪ΩS

e−y|Ω

U

|

|ΩS |>|ΩU |

Q∗ia ∼ =

bi Q

b∈ΩU ∪ΩS

e−y|Ω

|ΩU |=|ΩS |

U

|

bi Q

b∈ΩU ∪ΩS

b ∈ΩU ∪ΩS

bi ) , (1 − Q

(20.12)

b ∈ΩU ∪ΩS

bi ) . (1 − Q

(20.13)

b ∈ΩU ∪ΩS

∗ S The overall normalization is fixed by the condition QU ia + Qia + Qia = 1. As usual, eqns (20.10–(20.13) can be understood either as defining a mapping from

ai , Qia } onto itself or as a set of fixed-point conditions. In the space of messages {Q both cases, they are referred to as the SP(y) equations for the satisfiability problem. From the computational point of view, these equations involve a sum over 2|∂i|−1 terms. This is often too much if we want to iterate the SP(y) equations on large KSAT formulae: the average degree of a variable node in a random K-SAT formula with clause density α is Kα. Further, in the most interesting regime—close to the SAT– K UNSAT threshold—α = Θ(2K ), and the sum is over 2Θ(K2 ) terms, which rapidly becomes impractical. It is thus important to notice that the sums can be computed efficiently by interpreting them as convolutions.

Exercise 20.4 Consider a sequence of independent Bernoulli random variables X1 , . . . , Xn , . .P . , with means η1 , . . . , ηn , . . . , respectively. Let Wn (m) be the probability that the sum n b=1 Xb is equal to m. (a) Show that these probabilities satisfy the recursion Wn (m) = ηn Wn−1 (m − 1) + (1 − ηn )Wn−1 (m) , for m ∈ {0, . . . , n}. Argue that these identities can be used, together with the initial condition W0 (m) = I(m = 0), to compute Wn (m) in O(n2 ) operations. (b) How can one compute the right-hand sides of eqns (20.11)–(20.13) in O(|∂i|2 ) operations?

20.2.2

The free entropy FRSB,e

provides Within the 1RSB energetic cavity method, the free entropy FRSB,e ({Q, Q}) detailed information on the minimal energy of (Bethe) pure states. These pure states


are nothing but metastable minima of the energy function (i.e. minima whose energy cannot be decreased with a bounded number of spin flips).

ai } that The 1RSB free entropy is expressed in terms of a set of messages {Qia , Q provide a (quasi-)solution of the SP(y) equations (20.10)–(20.13). Following the general theory in Section 19.5.2, it can be written in the form

= FRSB,e ({Q, Q})

FRSB,e + a

a∈C

FRSB,e − i

i∈V

FRSB,e . ia

(20.14)

(i,a)∈E

Equation (19.94) yields

ai QU . = 1 − (1 − e−y )Q ia

RSB,e

eFia

(20.15)

defined in eqn (19.92) can be computed as follows. The The contribution FRSB,e a reweighting Fea ({mia }) is always equal to 0, except in the case where all of the variables in clause a receive a warning requesting that they point in the ‘wrong direction’, namely the direction which does not satisfy the clause. Therefore, RSB,e

e Fa

= 1 − (1 − e−y )

QU ia .

i∈∂a

defined in eqn (19.93) depends on the messages sent Finally, the contribution FRSB,e i from the check nodes b ∈ ∂i. Let us denote by ΩS ⊆ ∂0 i the subset of check nodes b ∈ ∂0 i such that clause b forces xi to satisfy it. Similarly, we denote by ΩU ⊆ ∂1 i the subset of ∂1 i such that clause b forces xi to satisfy it. We then have RSB,e

e Fi

=

ΩU ,ΩS

e−y min(Ω

S

U

,Ω )

⎡

bi ⎣ Q

b∈ΩU ∪ΩS

⎤

bi )⎦ . (1 − Q

(20.16)

b ∈ΩU ∪ΩS

Exercise 20.5 Show that, for any i ∈ ∂a, FRSB,e = FRSB,e . a ia

20.2.3

The large-y limit: The SP equations

Consider now the case of satisfiable instances. A crucial problem is to characterize satisfying assignments and to find them efficiently. This amounts to focusing on zeroenergy assignments, which are selected by taking the y → ∞ limit within the energetic cavity method. We can take the limit y → ∞ in the SP(y) equations (20.11)–(20.13). This yields


ai = Q

QU ja ,

(20.17)

j∈∂a\i

∼ QU ja =

⎡

bj ) ⎣1 − (1 − Q

b∈Sja

QSja ∼ =

⎡

bj ) ⎣1 − (1 − Q

b∈Uja

Q∗ja ∼ =

⎤

bj )⎦ , (1 − Q

b∈Uja

(20.18)

⎤

bj )⎦ , (1 − Q

(20.19)

b∈Sja

bj ) , (1 − Q

(20.20)

b∈∂j\a S ∗ where the normalization is always fixed by the condition QU ja + Qja + Qja = 1. The y = ∞ equations have a simple interpretation. Consider a variable xj appearing in clause a, and assume that it receives a warning from clause b = a independently

bj . Then with probability Q b∈Sja (1 − Qbj ) is the probability that variable j receives no warning forcing it in the direction which satisfies clause a. The product

bj ) is the probability that variable j receives no warning forcing it in (1 − Q b∈Uja the direction which violates clause a. Therefore QU ja is the probability that variable j receives at least one warning forcing it in the direction which violates clause a, conditional on the fact that there are no contradictions in the warnings received by j from

ai is the clauses b = a. Analogous interpretations hold for QSja and Q∗ja . Finally, Q probability that all variables in ∂a \ i are forced in the direction which violates clause a, under the same condition of no contradiction. Note that the y = ∞ equations are a relatively simple modification of the BP equations in eqn (20.3). However, the interpretation of the messages is very different in the two cases. Finally, the free entropy in the y = ∞ limit is obtained as RSB,e RSB,e FRSB,e + Fi − Fia , (20.21) FRSB,e = a a∈C

where

i∈V

(i,a)∈E

4 3 U

, = log 1 − Q Q FRSB,e ai ia ia # $ RSB,e

= log (1 − Qbi ) + (1 − Qbi ) − (1 − Qbi ) , Fi b∈∂0 i

FRSB,e a

⎧ ⎫ ⎨ ⎬ = log 1 − QU . ja ⎩ ⎭

b∈∂1 i

(20.22) (20.23)

b∈∂i

(20.24)

j∈∂a

Exercise 20.6 Show that, if the SP messages satisfy the fixed-point equations (20.17)– P P + a (1 − |∂a|)FRSB,e . (20.20), the free entropy can be rewritten as FRSB,e = i FRSB,e a i


20.2.4

The SAT–UNSAT threshold

The SP(y) equations (20.10)–(20.13) always admit a ‘no warning’ fixed point corre ai = 0, QS = QU = 0, and Q∗ = 1 for each (i, a) ∈ E. Other fixed sponding to Q ia ia ia points can be explored numerically by iterating the equations on large random formulae. Within the cavity approach, the distribution of the message associated with a uniformly random edge (i, a) satisfies a distributional equation. As explained in Sec ai and (QU , QS , Q∗ ) tion 19.2.5, this distributional equation is obtained by promoting Q ia ia ia to random variables and reading eqns (20.10)–(20.13) as equalities in distribution. The distribution can then be studied by the population dynamics of Section 19.2.6. It ob = 0 and viously admits a no-warning (or ‘replica-symmetric’) fixed point, with Q (QU , QS , Q∗ ) = (0, 0, 1) identically, but (as we shall see) in some cases one also finds a different, ‘non-trivial’ fixed-point distribution. Given a fixed point, the 1RSB free-entropy density Fe (y) can be estimated by taking the expectation of eqn (20.14) (with respect to both degrees and fields) and dividing by N . When evaluated at the no-warning fixed point, the free-entropy density Fe (y) vanishes. This means that the number of clusters of SAT assignments is subexponential, so that the corresponding complexity density vanishes. To a first approximation, this solution corresponds to low-energy assignments forming a single cluster. Note that the energetic cavity method counts the number of clusters of SAT assignments, and not the number of SAT assignments itself (which is actually exponentially large). Figure 20.4 shows the outcome of a population dynamics computation. We plot the free-entropy density Fe (y) as a function of y for random 3-SAT, for a few values of the clause density α. These plots were obtained by initializing the population dynamics

i } uniformly random in [0, 1]. For α < αd,SP 3.93, recursion with i.i.d. messages {Q

are the iteration converges to the ‘no-warning’ fixed point where all of the messages Q equal to 0. For α > αd,SP , and when y is larger than a critical value yd (α), the iteration converges to a non-trivial fixed point. This second solution has a non-vanishing value of the free-entropy density Fe (y). The energetic complexity Σe () is obtained from Fe (y) via the Legendre transform (19.95). In practice, the Legendre transform is computed by fitting the population dynamics data, and then transforming the fitting curve. Good results were obtained with a fit of r∗ ψr e−ry with r∗ between 2 and 4. The resulting curves Σe () the form Fefit (y) = r=0 (or, more precisely, their concave branches) are shown in Fig. 20.5.

Exercise 20.7 Show that Σe ( = 0) = limy→∞ Fe (y)

The energetic complexity Σe () is the exponential growth rate of the number of (quasi-) solutions of the min-sum equations with energy density u. As can be seen in Fig. 20.5, for α = 4.1 or 4.2 (and in general, in an interval above αd (3)), one finds Σe ( = 0) > 0. The interpretation is that there exist exponentially many solutions of the min-sum equations with zero energy density.


0.02 0.015

Fe (y)

0.01 0.005 0 -0.005 0

1

2

3

4

5

y Fig. 20.4 1RSB free-entropy density for 3-SAT, computed from a population dynamics analysis of the SP equation, at α = 4.1, 4.2, and 4.3 (from top to bottom). For each α, y, a population of size 12 000 was iterated 12 × 106 times. The resulting Fe was computed by averaging over the last 8 × 106 iterations.

0.02

Σe () 0.01

0.001

0.002

0.003

0.004

0.005

Fig. 20.5 Energetic complexity density Σe plotted versus energy density , for the 3-SAT problem at α = 4.1, 4.2, and 4.3 (from top to bottom). These curves were obtained from Fig. 20.4, using a Legendre transform of free-entropy fits. .

In contrast, when α = 4.3 the curve starts at a positive or, equivalently, the 1RSB complexity curve has Σe ( = 0) < 0. Of course, the typical number of min-sum solutions cannot decrease exponentially. The result Σe ( = 0) < 0 is interpreted as a consequence of the fact that a typical random formula does not admit any (approximate) solutions of the min-sum equations with energy density = 0. Given the correspondence between min-sum fixed points and clusters of low-energy assignments, this in turn implies that a typical random formula does not have any SAT assignment.


Table 20.1 Predictions of the 1RSB cavity method for the SAT–UNSAT threshold of random K-satisfiability.

K αs (K)

3 4.2667

4 9.931

5 21.117

6 43.37

7 87.79

8 176.5

9 354.0

10 708.9

From Fig. 20.5, one would expect that the SAT–UNSAT transition would lie between α = 4.2 and α = 4.3. A more precise estimate can be obtained by plotting Fe (y → ∞) versus α, and locating the value of α where it vanishes. For 3-SAT, we obtain the estimate for the SAT–UNSAT threshold αs (3) = 4.26675 ± 0.00015. The predictions of this method for αs (K) are shown in Table 20.1. In practice, reliable estimates can be obtained with population dynamics only for K ≤ 7. The reason is that αs (K) increases exponentially with K, and the size of the population needed in order to achieve a given precision will increase accordingly (the average number of independent messages entering into the distributional equations is Kα). For large K, one can formally expand the distributional equations, which yields a series for αs (K) in powers of 2−K . The first two terms (seven terms have been computed) of this expansion are 1 αs (K) = 2K log 2 − (1 + log 2) + O(2−K K 2 ) . 2

20.2.5

(20.25)

SP-guided decimation

The analysis in the last few pages provides a refined description of the set of solutions of random formulae. This knowledge can be exploited to efficiently find some solutions, much in the same way as we used belief propagation in Section 20.1.3. The basic strategy is again to use the information provided by the SP messages as a clever heuristic in a decimation procedure. The first step consists in finding an approximate solution of the SP(y) equations (20.10)–(20.13), or of their simplified y = ∞ version (eqns (20.17)–(20.20)), for a given instance of the problem. To be definite, we shall focus on the latter case, since y = ∞ selects zero-energy states. We can seek solutions of the SP equations by iteration, exactly as we would do with BP. We initialize the SP messages, generally as i.i.d. random variables with some common distribution, and then update them according to eqns (20.17)–(20.20). Updates can, for instance, be implemented in parallel until a convergence criterion has been met. Figure 20.6 shows the empirical probability that the iteration converges before tmax = 1000 iterations on random formulae as a function of the clause density α. As a convergence criterion, we required that the maximal difference between any two subsequent values of a message was smaller than δ = 10−2 . Messages were initialized

ai ∈ [0, 1] independently and uniformly at random. It is by drawing, for each edge, Q clear that SP has better convergence properties than BP for K = 3 and, indeed, it converges even for α larger than the SAT–UNSAT threshold.


1

1

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0

0 3.6

3.8

4 α

4.2

4.4

9

9.5

10

10.5

11

11.5

α

Fig. 20.6 Empirical convergence probability of SP (initialized with uniformly random messages) plotted versus the clause density α for 3-SAT (left), and 4-SAT (right). The average is over 100 instances, with N = 5 × 103 (solid line) and N = 104 (dashed line) variables. The data points show the empirical probability that SP-guided decimation finds a SAT assignment, computed over 100 instances with N = 5 × 103 . The vertical lines are the predicted SAT–UNSAT thresholds.

The numerics suggest the existence of two thresholds αd,SP (K) and αu,SP (K) that characterize the convergence behaviour as follows (all of the statements below should be interpreted as holding with high probability in the large-N limit):

ai = 0 • For α < αd,SP , the iteration converges to the trivial fixed point defined by Q for all edges (i, a) ∈ G. • For αd,SP < α < αu,SP , the iteration converges to a ‘non-trivial’ fixed point. • For αu,SP < α, the iteration does not converge. In the interval αd,SP (K) < α < αU,SP (K), it is expected that an exponential number of fixed points will exist, but most of them will be degenerate and correspond

ai = 0 or 1 for all of the edges (i, a). to ‘disguised’ WP fixed points. In particular, Q On the other hand, the fixed point actually reached by iteration is stable with respect to changes in the initialization. This suggests the existence of a unique non-degenerate fixed point. The threshold αd,SP (K) is conjectured to be the same as that defined for the distributional equation in the previous section; this is why we have used a similar symbol. In particular, αd,SP (K = 3) ≈ 3.93 and αd,SP (K = 4) ≈ 8.30. We have also obtained αu,SP (K = 3) ≈ 4.36 and αu,SP (K = 4) ≈ 9.7. SP can be used in a decimation procedure. After iterating the SP equations until convergence, one computes the SP marginal for each variable i ∈ {1, . . . , N }: ∼

(1 − Qai ) 1 − (1 − Qai ) , wi (1) = a∈∂0 i

wi (0) ∼ =

ai ) (1 − Q

a∈∂1 i

wi (∗) ∼ =

ai ) , (1 − Q

a∈∂i

a∈∂1 i

1−

(1 − Qai ) ,

a∈∂0 i

(20.26)


with the normalization condition wi (1) + wi (0) + wi (∗) = 1. The interpretation of these SP marginals is the following: wi (1) and wi (0) are the probabilities that the variable i receives a warning forcing it to take the value xi = 1 or xi = 0, respectively, conditional on the fact that it does not receive contradictory warnings. The bias of the variable is then defined as πi ≡ wi (0) − wi (1). The variable with the largest absolute bias is selected, and fixed according to the sign of the bias. This procedure is then iterated as with BP-guided decimation. It typically happens that, after some fraction of the variables have been fixed with this method, the SP iteration on the reduced instance converges to the trivial fixed

ai = 0. According to our interpretation, this means that the resulting problem point Q is described by a unique Bethe measure, and the SAT assignments are no longer clustered. In fact, in agreement with this interpretation, it is found that, typically, simple algorithms are able to solve the reduced problem. A possible approach is to run BP-guided decimation. An even simpler alternative is to apply a simple local search algorithm, such as Walksat or simulated annealing. The pseudocode for this algorithm is as follows. SP-Guided Decimation (formula F, SP parameter , tmax , WalkSAT parameters f , p) 1: Set U = ∅; 2: Repeat until FAIL or U = V : 3: Call SP(F, , tmax ). If it does not converge, FAIL; 4: For each i ∈ V \ U , compute the bias πi ; 5: Let j ∈ V \ U have the largest value of |πi |; 6: If |πj | ≤ 2K call WalkSAT(F, f, p); 7: Else fix xj according to the sign of πj , and define F as the new formula obtained after fixing xj ; 8: End-Repeat; 9: Return the current assignment; SP (formula F, accuracy , iterations tmax ) 1: Initialize SP messages to i.i.d. random variables; 2: For t ∈ {0, . . . , tmax } 3: For each (i, a) ∈ E

ai using eqn (20.10) 4: Compute the new value of Q 5: For each (i, a) ∈ E 6: Compute the new value of Qai using eqns (20.11)–(20.13) 7: Let ∆ be the maximum difference from the previous iteration; 8: If ∆ < return current messages; 9: End-For; 10: Return ‘Not Converged’; The pseudocode of WalkSAT was given in Section 10.2.3. In Fig. 20.6, we plot the empirical success probability of the SP-guided-decimation


1

1

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0

0 3.6

3.7

3.8

3.9

4 α

4.1

4.2

4.3

4.4

9

9.2

9.4

9.6

9.8

10

10.2

10.4

α

Fig. 20.7 Performance of BP-guided decimation and SP-guided decimation on 3-SAT (left plot) and 4-SAT (right plot) problems. The plots show the probability of finding a SAT assignment versus the clause density, averaged over 100 instances with N = 5 × 103 variables. The SP-based algorithm (dotted line) performs better than the BP-based one (full line). The vertical lines are the SAT–UNSAT thresholds.

algorithm for random 3-SAT and 4-SAT formulae as a function of the clause density α. A careful study suggests that the algorithm finds a satisfying assignment with high probability when α 4.252 (for K = 3) and α 9.6 (for K = 4). These values are slightly smaller than the conjectured locations of the SAT–UNSAT threshold αs (3) ≈ 4.2667 and αs (4) ≈ 9.931. Apart from the SP routine (which builds upon the statistical-mechanics insight), the above algorithm is quite naive and could be improved in a number of directions. One possibility is to allow the algorithm to backtrack, i.e. to release some variables that were fixed at a previous stage of the decimation. Further, we have not used, at any step, the information provided by the free entropy Fe (y = ∞), which can be computed at little extra cost. Since this gives an estimate of the logarithm of the number of clusters of solutions, it can also be reasonable to make choices that maximize the value of Fe in the resulting formula. As can be deduced from Fig. 20.7, SP-guided decimation outperforms BP-guided Decimation. Empirically, this algorithm, or small variations of it, provides the most efficient procedure for solving large random K-SAT formulae close to the SAT–UNSAT threshold. Furthermore, it has extremely low complexity. Each SP iteration requires O(N ) operations, which yields O(N tmax ) operations per SP call. In the implementation outlined above, this implies an O(N 2 tmax ) complexity. This can, however, be reduced to O(N tmax ) by noticing that fixing a single variable does not affect the SP messages significantly. As a consequence, SP can be called every N δ decimation steps for some small δ. Finally, the number of iterations required for convergence seems to grow very slowly with N , if it does at all. One should probably think of tmax as a big constant, or tmax = O(log N ) In order to get a better understanding of how SP-guided decimation works, it is useful to monitor the evolution of the energetic-complexity curve Σe () while decimating. When SP iteration has converged for a given instance, one can use eqn (20.21) to compute the free entropy, and then compute the curve Σe () by means of a Legendre

Some ideas about the full phase diagram

0.02

0.015

Σe ()

0.025

0.01

0.02 0.015 0.01

0.005 0.005 0 0

0.02 0.04 0.06 0.08 0.1 0.12 0.14

0 0

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

0.004

Fig. 20.8 Decimation process: the complexity versus energy density (1/N times the number of violated clauses) measured on a single instance of random 3-SAT with N = 10 000 and α = 4.2 (top curve), and on the decimated instances obtained after fixing 1000, 2000, and 3000 variables with the SP-guided decimation procedure (from top to bottom). For comparison, the inset shows the same complexity versus the total energy after fixing 1000, 2000, and 3000 randomly chosen variables to arbitrary values

transform. In Fig. 20.8, we consider a run of the SP-guided decimation algorithm on one random 3-SAT formula with N = 104 at α = 4.2. Here, the complexity curve of the residual formula (N Σe () versus the number of violated clauses N ) is plotted every 1000 decimation steps. One can notice two main effects: (i) the zero-energy complexity N Σe (0) decreases, showing that some clusters of solutions are lost during the decimation; and (ii) the number of violated clauses in the most numerous metastable clusters, the ‘threshold energy’, decreases as well,1 implying that the problem becomes simpler: the true solutions are less and less hidden among metastable minima. The important point is that the effect (ii) is much more pronounced than (i). After about half of the variables have been fixed, the threshold energy vanishes. SP converges to the trivial fixed point, the resulting instance becomes ‘simple’ and is solved easily by Walksat.

20.3 20.3.1

Some ideas about the full phase diagram Entropy of clusters

The energetic 1RSB cavity method has given two important results: on the one hand, a method to locate the SAT–UNSAT transition threshold αs , which is conjectured to be exact, and on the other, a powerful message-passing algorithm: SP. These results 1 Because of the instability of the 1RSB solution at large energies (see Chapter 22), the threshold energies obtained within the 1RSB approach are not exact. However, one expects the actual behaviour to be quantitatively close to the 1RSB description.


0.04

0.08

0.03

Σ(φ)0.02

f RS

Σ(x = 0)

0.06

0.04

F(x = 0)

F(x = 1)

F(x∗ )

0.01 0.02

αs

αd

0 0 -0.01

αc

Σ(x = 1)

-0.02 -0.02

8 0

0.01

0.02

0.03

0.04

0.05

φ

0.06

0.07

0.08

0.09

8.5

9

9.5

10

10.5

α

Fig. 20.9 1RSB analysis of random 4-SAT. Left: complexity versus internal entropy density of clusters, for α = 9.3, 9.45, 9.6, 9.7, 9.8, and 9.9 (from top to bottom). When SAT configurations are sampled uniformly, one finds either configurations in an exponentially large number of clusters (the dot on the curve for α = 9.45, which is the point where dΣ/dφ = −1), or a condensed phase where the measure is dominated by a few clusters (the squares on the curves for α ≥ 9.6). Right: complexity Σ(x) and free-entropy density F(x) at a few key values of x: x = 0 corresponds to the maximum of Σ(φ), x = 1 to the point where dΣ/dφ = −1, and x = x∗ to Σ(φ) = 0. The dynamical transition is at αd ≈ 9.38, the condensation transition at αc ≈ 9.547, and the SAT–UNSAT transition at αs ≈ 9.931.

were obtained at a cost: we completely forgot about the size of the clusters of SAT assignments, their ‘internal entropy’. In order to get a finer understanding of the geometry of the set of solutions in the SAT phase, we need to get back to the uniform measure over SAT assignments of eqn (20.1), and use the 1RSB method of Section 19.2. Our task is, in principle, straightforward: we need to estimate the 1RSB free entropy F(x), and evaluate the Legendre transform (19.8) in order to get the complexity function Σ(φ). Recall that Σ(φ) is the exponential growth rate of the number of clusters with free entropy N φ (in the present case, since we are restricting ourselves to SAT configurations, the free entropy of a cluster is equal to its entropy). This is a rather demanding task from the numerical point of view. Let us understand why: each BP message is parameterized by one real number in [0, 1], as we saw in eqn (20.3). A 1RSB message characterizes the distribution of this number, so it is a pdf on [0, 1]. One such distribution is associated with each directed edge of the factor graph. For a study of the phase diagram, one needs to perform a statistical analysis of the 1RSB messages. Within the population dynamics approach, this means that we must use a (large) population of distribution functions. For each value of x, the algorithm must be run for a large enough number of iterations to estimate F(x). This is at the limit of what can be done numerically. Fortunately, it can be complemented by two simpler computations: the SP approach, which gives the results corresponding to x = 0, and a study of the x = 1 case using the simplification described in Section 19.4.

Some ideas about the full phase diagram

20.3.2

The condensation transition for K ≥ 4

We shall not provide any technical details of these computations, but focus instead on the main results using K = 4-SAT as a running example. As shown by Fig. 20.9, this system displays the full scenario of phase transitions described in Section 19.6. Upon increasing the clause density α, one finds first an RS phase for α < αd , then a d1RSB phase with exponentially many relevant states for αd < α < αc , and then an s1RSB phase with condensation of the measure on a few states for αc < α < αs . The system becomes UNSAT for α > αs . Figure 20.9 shows the evolution of the complexity versus the internal entropy density of the clusters as α increases (note that increasing α plays the same role as decreasing the temperature in the general scenario sketched in Fig. 19.6). For a given α, almost all clusters have an internal entropy density φ0 corresponding to the maximum of Σ(φ). The complexity at the maximum, Σ(φ0 ) = F(x = 0), is equal to the complexity at zero energy density that we found by the energetic 1RSB cavity method. When SAT configurations are sampled uniformly, almost all of them are found in clusters of internal entropy density φ1 such that Σ(φ) + φ is maximum, conditional on the fact that Σ(φ) ≥ 0. In the d1RSB phase, one has Σ(φ1 ) > 0, and in the s1RSB phase one has Σ(φ1 ) = 0. The condensation point αc can therefore be found through a direct (and more precise) study at x = 1. Indeed it is identified as the value of the clause density such that the two equations Σ(φ) = 0 and dΣ/dφ = −1 admit a solution.

Exercise 20.8 Using the Legendre transform in eqn (19.8), show that this condensation point αc is the point where the 1RSB free-entropy function F(x) satisfies F(1) − F (1) = 0 (where the prime means a derivative with respect to x). As we saw in Section 19.4, the value of F(1) is equal to the RS free entropy. The value of the internal entropy F (1) can be obtained explicitly from the x = 1 formalism. Writing down the full x = 1 formalism for random satisfiability, including this computation of F (1), is an interesting (non-trivial) exercise.

The dynamical transition point αd is defined as the smallest value of α such that there exists a non-trivial solution to the 1RSB equations at x = 1 (in practice, it is best studied using the point-to-set correlation, which will be described in Chapter 22). Notice from Fig. 20.9 that there can exist clusters of SAT assignments even at α < αd : for α = 4.3, there exists a branch of Σ(φ) around the point φ0 where it is maximum, but this branch disappears, if one increases φ, before one can find a point where dΣ/dφ = −1. The interpretation of this regime is that an exponentially small fraction of the solutions are grouped into well-separated clusters. The vast majority of the solutions belong instead to a single, well-connected ‘replica-symmetric’ cluster. As we saw in the case of the energetic cavity method, the first occurrence of the clusters around φ0 occurs at the value αd,SP , which is around 8.3 for 4-SAT. The same scenario has been found in studies of random K-SAT with K = 5 and 6, and it is expected to hold for all K ≥ 4. The situation is somewhat different for K = 3, as the condensation point αc coincides with αd : the 1RSB phase is always condensed. Table 20.2 summarizes the values of the thresholds.

Random K-satisfiability Table 20.2 Predictions of the 1RSB cavity method for the non-trivial SP, dynamical, condensation, and SAT–UNSAT thresholds of random K-satisfiability.

K 3 4 5 6

20.4

αd 3.86 9.38 19.16 36.53

αc 3.86 9.547 20.80 43.08

αs 4.2667 9.931 21.117 43.37

An exercise: Colouring random graphs

Recall that a proper q-colouring of a graph G = (V, E) is an assignment of colours {1, . . . , q} to the vertices of q in such a way that no edge has its two adjacent vertices of the same colour. Hereafter, we shall refer to a proper q-colouring as simply a ‘colouring’ of G. Colourings of a random graph can be studied following the approach just described for satisfiability, and reveal a strikingly similar behaviour. Here we shall just present some key steps of this analysis: this section can be seen as a long exercise in applying the cavity method. We shall focus on the case of random regular graphs, which is technically simpler. In particular, many results can be derived without resorting to a numerical solution of the cavity equations. The reader is encouraged to work out the many details which have been left out. We shall adopt the following description of the problem: with each vertex i ∈ V of a graph G = (V, E), we associate a variable xi ∈ {1, . . . , q}. The energy of a colour assignment x = {x1 , . . . , xN } is given by the number of edges whose vertices have the same colour: I(xi = xj ) . (20.27) E(x) = (ij)∈E

If the graph is colourable, we are also interested in the uniform measure over proper colourings: 1 1 I(E(x) = 0) = I(xi = xj ) , (20.28) µ(x) = Z Z (ij)∈E

where Z is the number of proper colourings of G. The factor graph associated with µ( · ) is easily constructed. We associate one variable node with each vertex of i ∈ G and one function node with each edge (ij) ∈ C, and connect this function node to the two variable nodes corresponding to i and j. The probability distribution µ(x) is therefore a pairwise graphical model. We shall assume that G is a random regular graph of degree c. Equivalently, the corresponding factor graph is distributed according to the ensemble DN (Λ, P ), with Λ(x) = xc and P (x) = x2 . The important technical simplification is that, for any fixed r, the radius-r neighbourhood around a random vertex i is, with high probability, a tree of degree c, i.e. it is non-random. In other words, the neighbourhood of most of the nodes is the same. Let us start with the RS analysis of the graphical model (20.28). As we saw in Section 14.2.5, we can get rid of function-to-variable-node messages, and work with variable-to-function-node messages νi→j (xi ). The BP equations read

An exercise: Colouring random graphs

νi→j (x) ∼ =

(1 − νk→i (x)) .

(20.29)

k∈∂i\j

Because of the regularity of the graph, there exist solutions of these equations such that messages take the same value on all edges. In particular, eqn (20.29) admits the solution νi→j ( · ) = νunif ( · ), where νunif ( · ) is the uniform message: νunif (x) = 1/q for x ∈ {1, . . . , q}. The corresponding free-entropy density (equal here to the entropy density) is 1 c RS . (20.30) f = log q + log 1 − 2 q It can be shown that this coincides with the ‘annealed’ estimate N −1 log EZ. It decreases with the degree c of the graph and becomes negative for c larger than cUB (q) ≡ 2 log q/ log(q/(q − 1)), similarly to what we saw in Fig. 20.2. The Markov inequality implies that, with high probability, a random c-regular graph does not admit a proper q-colouring for c > cUB (q). Further, the RS solution is certainly incorrect for c > cUB (q). A stability analysis of this solution shows that the spin glass susceptibility diverges as c ↑ cst (q), with cst (q) = q 2 − 2q + 2. For q ≥ 4, cst (q) > cUB (q). In order to correct the above inconsistencies, we have to resort to the energetic 1RSB approach. Let us focus on the y → ∞ limit (or, equivalently, the zero-energy limit). In this limit, we obtain the SP equations. These can be written in terms of messages Qi→j ( · ) that have the following interpretation: • Qi→j (x) = probability that, in the absence of (i, j), xi is forced to the value x, • Qi→j (∗) = probability that, in the absence of (i, j), xi is not forced. Recall that ‘probability’ is interpreted here with respect to a random Bethe state. An SP equation expresses the message Qi→j ( · ) in terms of the c − 1 incoming messages Qk→i ( · ), with k ∈ ∂i \ j. To keep the notation simple, we fix an edge i → j and denote it by 0, and use 1, . . . , c − 1 to label the edges k → i with k ∈ ∂i \ j. Then, for any x in {1, . . . , q},we have (x ...x )∈N (x) Q1 (r1 )Q2 (x2 ) · · · Qc−1 (xc−1 ) , (20.31) Q0 (x) = 1 c−1 (x1 ...xc−1 )∈D Q1 (r1 )Q2 (x2 ) · · · Qc−1 (xc−1 ) where: • D is the set of tuples (x1 , . . . , xc−1 ) ∈ {∗, 1, . . . , q}n such that there exists z ∈ {1, . . . , q} with z = x1 , . . . , xc−1 . According to the interpretation above, this means that there is no contradiction among the warnings to i. • N (x) is the set of tuples (x1 , . . . , xc−1 ) ∈ D such that, for any z = x, there exists k ∈ {1, . . . , c − 1} such that xk = z. In other words, x is the only colour for vertex i that is compatible with the warnings. Q0 (∗) is determined by the normalization condition Q0 (∗) + x Q0 (x) = 1. On a random regular graph of degree c, these equations admit a solution with Qi→j ( · ) = Q( · ) independent of the edge (i, j). Furthermore, if we assume this solution


c = 17

0.15

q 3 4 5 6 10

cd,SP 5 9 13 17 36

cd 5 9 14 18 39

cc 6 10 14 19 43

cs 6 10 15 20 44

c = 18

0.1

0.05

c = 19

Σ(φ) 0

c = 20

-0.05 0

0.02

0.04

0.06

φ

0.08

0.1

0.12

0.14

Fig. 20.10 Results of a 1RSB analysis of proper q-colourings of random regular graphs. The table gives the thresholds: the threshold for the appearance of non-trivial SP solutions cd,SP , the dynamical threshold cd , the condensation threshold cc , and the threshold for the colourable/uncolourable transition cs . The figure on the right shows the complexity of the clusters as a function of their internal entropy density. Here, q = 6 and the degrees of the graphs are c = 17 (RS), c = 18 (d1RSB), c = 19 (s1RSB) and c = 20 (uncolourable). The circles denote the points of slope −1 on the complexity curves.

to be symmetric under permutation of colours, the corresponding message can be parameterized by a single number a ∈ [0, 1/q]: Q(x) = a for x ∈ {1, . . . , q} , Q(∗) = 1 − qa . Plugging this Ansatz into eqn (20.31), we get q−1 r q−1 (1 − (r + 1)a)c−1 r=0 (−1) . a = q−1 rq r c−1 r=0 (−1) r+1 (1 − (r + 1)a)

(20.32)

(20.33)

The complexity Σe ( = 0), yielding the exponential growth rate of the number of clusters of proper colourings, is given by Σe (e = 0) = limy→∞ Fe (y). We find that e

Σ ( = 0; c, q) = log

q−1 r=0

q c c (1 − (r + 1)a) − log(1 − qa2 ) . r+1 2

r

(−1)

(20.34) Given the number of colours q, one can study what happens when the degree c grows (which amounts to increasing the density of constraints). The situation is very similar to that found for the satisfiability problem. For c ≥ cd,SP (q), there exists a pair of nontrivial solution to eqn (20.33) with a > 0. The complexity Σe (e = 0) can be computed from eqn (20.34) (evaluated for the largest solution a of eqn (20.33)), and is decreasing

Notes

in c. It becomes negative for c ≥ cs (q). The degree cs (q) is thus the 1RSB prediction for the SAT–UNSAT threshold. When c < cs (q), the uniform measure over valid colourings can be studied, and, in particular, one can characterize the distribution of the entropy of the clusters. Figure 20.10 shows the complexity as a function of the internal entropy density of the clusters. The similarity to Fig. 20.9 is obvious. One can define two particularly relevant thresholds: cd is the smallest degree such that the 1RSB equations at x = 1 have a non-trivial solution, and cc is the smallest degree such that the uniform measure over proper colourings is ‘condensed’. The table in Fig. 20.10 gives some examples of these thresholds. An asymptotic analysis for large q shows that cd,SP = q(log q + log log q + 1 − log 2 + o(1)) , cd = q(log q + log log q + O(1)) , cc = 2q log q − log q − 2 log 2 + o(1) , cs = 2q log q − log q − 1 + o(1) .

(20.35) (20.36) (20.37) (20.38)

These predictions can be rephrased into a statement on the chromatic number, i.e. the minimal number of colours needed to colour a graph. Because of the heuristic nature of the approach, we formulate it as a conjecture. Conjecture 20.1 With high probability, the chromatic number of a random regular graph with N vertices and degree c ≥ 4 is equal to χchrom (c), where χchrom (c) = max{q : Σe ( = 0; c, q) > 0} .

(20.39)

Here Σe ( = 0; c, q) is given by eqn (20.34), where a is the largest solution of eqn (20.33) in the interval [0, 1/q]. Using the numbers in Fig. 20.10, this conjecture predicts, for instance, that χchrom (c) = 3 for c = 4, 5, χchrom (c) = 4 for c = 6, 7, 8, 9, and χchrom (c) = 5 for 10 ≤ c ≤ 14. On the side of rigorous results, a clever use of the first- and second- moment methods allows one to prove the following result. Theorem 20.2 With high probability, the chromatic number of a random regular graph with N vertices and degree c is either k, k + 1 or k + 2, where k is the smallest integer such that c < 2k log k. Furthermore, if c > (2k − 1) log k, then with high probability the chromatic number is either k or k + 1. One can check explicitly that the results of the 1RSB cavity conjecture agree with this theorem, which proves the correct leading behaviour at large c. Although this presentation was focused on random regular graphs, a large class of random graph ensembles can be analysed along the same lines.

Notes Random K-satisfiability was first analysed using the replica-symmetric cavity method by Monasson and Zecchina (1996, 1997). The resulting equations are equivalent to a density evolution analysis of belief propagation. BP was used as an algorithm for


finding SAT assignments by Pumphrey (2001). That study concluded that BP was ineffective in solving satisfiability problems, mainly because it assigned variables in a one-shot fashion, unlike the case when it is used in decimation. The 1RSB cavity method was applied to random satisfiability in Mézard et al. (2002) and Mézard and Zecchina (2002), where the value of αc was computed for 3-SAT. This approach was applied to larger K by Mertens et al. (2006), who also derived the largeK asymptotics. The SP(y) and SP equations for satisfiability were first written down in Mézard and Zecchina (2002), where SP-guided decimation was introduced. A more algorithmic presentation of SP was then developed by Braunstein et al. (2005), together with an optimized source code for SP and decimation (Braunstein et al., 2004). The idea of backtracking was suggested by Parisi (2003), but its performance has not been systematically studied yet. The condensation phenomenon was discussed by Krzakala et al. (2007), in relation to studies of the entropic complexity in the colouring problem (Mézard et al., 2005b; Krzakala and Zdeborov´ a, 2007) and in the satisfiability problem (Montanari et al., 2008). The analysis in this chapter was heuristic, and is waiting for a rigorous proof. Let us point out that one important aspect of the whole scenario has been established rigorously for K ≥ 8: it has been shown that in some range of clause density below αs (K), the SAT assignments are grouped into exponentially many clusters, well separated from each other (Mézard et al., 2005a; Achlioptas and Ricci-Tersenghi, 2006; Daudé et al., 2008). This result can be obtained by a study of the ‘x-satisfiability’ problem, which requires one to determine whether a formula has two SAT assignments differing in xN variables. Bounds on the x-satisfiability threshold can be obtained through the first- and second- moment methods. The colouring problem was first studied with the energetic 1RSB cavity method by Mulet et al. (2002) and Braunstein et al. (2003): these papers contain the derivation of the SAT/UNSAT threshold and the SP equations. A detailed study of the entropy of clusters, and the computation of the other thresholds, was carried out by Krzakala and Zdeborov´ a (2007). These papers also study the case of Erd¨ os–Rényi graphs. Theorem 20.2 was proven by Achlioptas and Moore (2004), and its analogue for Erd¨ os–Rényi graphs by Achlioptas and Naor (2005). The validity of the RS solution for colouring graphs of low enough degree was proven by Bandyopadhyay and Gamarnik (2006). Linear relaxations of satisfiability have been studied by Gamarnik (2004).

21 Glassy states in coding theory In Chapter 15, we studied the problem of decoding random LDPC codes, and found two phase transitions that characterize the performance of the code in the large-blocklength limit. Consider, for instance, communication over a binary symmetric channel with crossover probability p. Under belief propagation decoding, the bit error rate vanishes in the large-block-length limit below a first threshold pd and remains strictly positive for p > pd . On the other hand, the minimal bit error rate achievable with the same ensemble (i.e. the bit error rate under symbol MAP decoding) vanishes up to a larger noise level pc and is bounded away from 0 for p > pc . In principle, one should expect every decoding algorithm to have a different threshold. This suggests that we should not attach too much importance to the BP threshold pd . On the contrary, we shall see in this chapter that pd is, in some sense, a ‘universal’ characteristic of the code ensemble: above pd , the decoding problem is plagued by an exponential number of metastable states (Bethe measures). In other words, the phase transition which takes place at pd is not only algorithmic, it is a structural phase transition. This transition turns out to be a dynamical 1RSB glass transition, and this suggests that pd is the largest possible threshold for a large class of local decoding algorithms. We have already seen, in the last section of Chapter 15, that the two thresholds pd and pc are closely related and can both be computed formally within the RS cavity method, i.e. in terms of the density-evolution fixed point. The analysis below will provide a detailed explanation of this connection in terms of the glass transition studied in Chapter 19. In Section 21.1, we start by considering a numerical investigation of the role of metastable states in decoding. Section 21.2 considers the particularly instructive case of a binary erasure channel, where the glassy states can be analysed relatively easily using the energetic 1RSB cavity method. The analysis of general memoryless channels is described in Section 21.3. Finally, Section 21.4 describes the connection between metastable states, which are one of the main subjects of study in this chapter, and trapping sets (subgraphs of the original factor graph that are often regarded as responsible for coding failures).

21.1

Local search algorithms and metastable states

The codewords of an LDPC code are solutions of a constraint satisfaction problem. The variables are the bits of a word x = (x1 , x2 , . . . , xN ), with xi ∈ {0, 1}, and the constraints are the parity check equations, i.e. a set of linear equations mod 2. This is

Glassy states in coding theory

analogous to the XORSAT problem considered in Chapter 18, although the ensembles of linear systems used in coding are different. An important difference from XORSAT is that we are looking for a specific solution of the linear system, namely the transmitted codeword. The received message y gives us a hint of where to look for this solution. For notational simplicity, we shall assume that the output alphabet Y is discrete, and the channel is a binary-input, memoryless, output-symmetric (BMS) channel (see Chapter 15)) with transition probability1 Q(y|x). The probability that x is the transmitted codeword, given the received message y, is given by the usual formula (15.1), P(x|y) = µy (x), where µy (x) ∼ =

N

Q(yi |xi )

i=1

M

I(xia1 ⊕ · · · ⊕ xiak(a) = 0) .

(21.1)

a=1

It is natural to associate an optimization problem with the code. We define the energy E(x) of a word x (also called a ‘configuration’) as twice the number of parity check equations violated by x (the factor 2 is introduced for future simplification). Codewords coincide with the global minima of this energy function, with zero energy. We already know that decoding consists in computing marginals of the distribution µy (x) (symbol MAP decoding) or finding its argmax (word MAP decoding). In the following, we shall discuss two closely related problems: (i) optimizing the energy function E(x) within a subset of the configuration space defined by the received word and the channel properties; and (ii) sampling from a ‘tilted’ Boltzmann distribution associated with E(x). 21.1.1

Decoding by constrained optimization

Let us start by considering the word MAP decoding problem. We shall exploit our knowledge of BMS channels. Conditional on the received word y = (y1 , y2 , . . . , yN ), the log-likelihood for x to be the channel input is Ly (x) =

N

log Q(yi |xi ) .

(21.2)

i=1

We shall later use the knowledge that the input word was a codeword, but Ly (x) is well defined for any x ∈ {0, 1}N , regardless of whether it is a codeword or not, so let us first characterize its properties. Assume, without loss of generality, that the codeword 0 was transmitted. By the law of large numbers, for large N the log-likelihood of this codeword is close to −N h, where h is the channel entropy: h = − y Q(y|0) log Q(y|0). The probability of an order-N deviation away from this value is exponentially small in N . This suggests that we should look for the transmitted codeword among those x such that Ly (x) is close to h. The corresponding typical-pairs decoding strategy goes as follows. Given the channel output y, look for a codeword x ∈ C such that Ly (x) ≥ −N (h + δ). We shall 1 Throughout this chapter, we adopt a different notation for the channel transition probability from that in the rest of the book, in order to avoid confusion with 1RSB messages.


refer to this condition as the ‘distance constraint’. For instance, in the case of a binary symmetric channel, it amounts to constraining the Hamming distance between the codeword x and the received codeword y to be small enough. If exactly one codeword satisfies the distance constraint, return it. If there is no such codeword, or if there are several of them, declare an error. Here δ > 0 is a parameter of the algorithm, which should be thought of as going to 0 after N → ∞.

Exercise 21.1 Show that the block error probability of typical-pairs decoding is independent of the transmitted codeword. [Hint: Use the linear structure of LDPC codes, and the symmetry property of a BMS channel.]

Exercise 21.2 This exercise is aimed at convincing the reader that typical-pairs decoding is ‘essentially’ equivalent to maximum-likelihood (or MAP) decoding. (a) Show that the probability that no codeword exists with Ly (x) ∈ [−N (h+δ), −N (h−δ)] is exponentially small in N . [Hint: Apply Sanov’s Theorem, (see Section 4.2), to the type of the received codeword.] (b) Obtain an upper bound for the probability that maximum-likelihood decoding succeeds and typical-pairs decoding fails in terms of the probability that there exists an incorrect codeword x with Ly (x) ≥ −N (h + δ), but no incorrect codeword Ly (x) ≥ −N (h − δ). (c) Estimate the last probability for Shannon’s random code ensemble. Show, in particular, that it is exponentially small for all noise levels strictly smaller than the MAP threshold and δ small enough.

Since codewords are global minima of the energy function E(x), we can rephrase typical-pairs decoding as an optimization problem: Minimize E(x)

subject to Ly (x) ≥ −N (h + δ) .

(21.3)

Neglecting exponentially rare events, we know that there always exists at least one solution with cost E(x) = 0, corresponding to the transmitted codeword. Therefore, typical-pairs decoding is successful if and only if the minimum is non-degenerate. This happens with high probability for p < pc . In contrast, for p > pc , the optimization admits other minima with zero cost (incorrect codewords). We have already explored this phenomenon in Chapters 11 and 15, and we shall discuss it further below. For p > pc , there exists an exponential number of codewords whose likelihood is greater than or equal to the likelihood of the transmitted one. Similarly to what we have seen in other optimization problems (such as MAXXORSAT and MAX-SAT), there exists generically an intermediate regime pd < p < pc , which is characterized by an exponentially large number of metastable states. For these values of p, the global minimum of E(x) is still the transmitted codeword, but is ‘hidden’ by the proliferation of deep local minima. Remarkably, the threshold for the appearance of an exponential number of metastable states coincides with the BP


threshold pd . Thus, for p ∈ ]pd , pc [, MAP decoding would be successful, but messagepassing decoding fails. In fact, no practical algorithm which succeeds in this regime is known. A ‘cartoon’ of this geometrical picture is presented in Fig. 21.1. At this point, the reader might be puzzled by the observation that finding configurations with E(x) = 0 is, per se, a polynomial task. Indeed, it amounts to solving a linear system modulo 2, and can be done by Gaussian elimination. However, the problem (21.3) involves the condition Ly (x) ≥ −N (h + δ), which is not a linear constraint modulo 2. If one resorts to local-search-based decoding algorithms, the proliferation of metastable states for p > pd can block the algorithm. We shall discuss this phenomenon for two local search strategies: ∆-local search and simulated annealing. 21.1.2

∆-local-search decoding

A simple local search algorithm consists in starting from a word x(0) such that Ly (x(0)) ≥ −N (h + δ) and then recursively constructing x(t + 1) by optimizing the energy function within a neighbourhood of radius ∆ around x(t), as in the following pseudocode. ∆ local search (channel output y, search size ∆, likelihood resolution δ) 1: Find x(0) such that Ly (x(0)) ≥ −N (h + δ); 2: for t = 0, . . . , tmax − 1: 3: Choose a uniformly random connected set U ⊂ {1, . . . , N } of variable nodes in the factor graph with |U | = ∆; 4: Find the configuration x that minimizes the energy subject to xj = xj for all j ∈ U ; 5: If Ly (x ) ≥ −N (h + δ), set x(t + 1) = x ; otherwise, set x(t + 1) = x(t); 6: end; 7: return x(tmax ). (Recall that a set of variable nodes U is ‘connected’ if, for any i, j ∈ U , there exists a path in the factor graph connecting i to j such that all variable nodes along the path are in U as well.)

Exercise 21.3 A possible implementation of step 1 consists in setting xi (0) arg maxx Q(yi |x). Show that this choice meets the likelihood constraint.

=

If the factor graph has bounded degree (which is the case with LDPC ensembles), and ∆ is bounded as well, each execution of the cycle above implies a bounded number of operations. As a consequence, if we let tmax = O(N ), the algorithm has linear complexity. A computationally heavier variant consists in choosing U at step 3 greedily. This means going over all such subsets and then taking the one that maximizes the decrease in energy |E(x(t + 1)) − E(x(t))|.


Fig. 21.1 Three possible ‘cartoon’ landscapes for the energy function E(x) (the number of violated checks), plotted in the space of all configurations x with Ly (x) ≥ N (h − δ). Left: the energy has a unique global minimum with E(x) = 0 (the transmitted codeword) and no (deep) local minima. Centre: many deep local minima appear, although the global minimum remains non-degenerate. Right: more than one codeword is compatible with the likelihood constraint, and the global minimum E(x) = 0 becomes degenerate.

Obviously, the energy E(x(t)) of the configuration produced after t iterations is a non-increasing function of t. If it vanishes at some time t ≤ tmax , then the algorithm implements a typical-pairs decoder. Ideally, one would like a characterization of the noise levels and code ensembles such that E(x(tmax )) = 0 with high probability. The case ∆ = 1 was analysed in Chapter 11, under the name of the ‘bit-flipping’ algorithm, for communicating over the channel BSC(p). We saw that there exists a threshold noise level p1 such that if p < p1 , the algorithm returns the transmitted codeword with high probability. It is reasonable to expect that the algorithm will be unsuccessful with high probability for p > p1 . Analogously, one can define thresholds p∆ for each value of ∆. Determining these thresholds analytically is an extremely challenging problem. One line of approach could consist in first studying ∆-stable configurations. We say that a configuration x is ∆-stable if, for any configuration x such that Ly (x ) ≥ −N (h + δ) and d(x, x ) ≤ ∆, E(x ) ≥ E(x).

Exercise 21.4 Show that, if no ∆-stable configuration exists, then the greedy version of the algorithm above will find a codeword after at most M steps (M being the number orf parity checks).

While this exercise hints at a connection between the energy landscape and the difficulty of decoding, one should be aware that the problem of determining p∆ cannot be reduced to determining whether ∆-stable states exist or to estimating their number. The algorithm in fact fails if, after a number t of iterations, the distribution of x(t) is (mostly) supported in the basin of attraction of ∆-stable states. The key difficulty is, of course, to characterize the distribution of x(t). 21.1.3

Decoding by simulated annealing

A more detailed understanding of the role of metastable configurations in the decoding problem can be obtained through an analysis of the MCMC decoding procedure that


we discussed in Section 13.2.1. We thus soften the parity check constraints through the introduction of an inverse temperature β = 1/T (this should not be confused with the temperature introduced in Chapter 6, which instead multiplied the log-likelihood of the codewords). Given the received word y, we define the following distribution over the transmitted message x (see eqn (13.10)):

µy,β (x) ≡

N 1 exp{−βE(x)} Q(yi |xi ) . Z(β) i=1

(21.4)

This is the ‘tilted Boltzmann form’ that we alluded to before. In the low-temperature limit, it reduces to the familiar a posteriori distribution which we would like to sample: µy,β=∞ (x) is supported on the codewords, and gives each of them a weight proportional to its likelihood. At infinite temperature, β = 0, the distribution factorizes over the bits xi . More precisely, under µy,β=0 (x), the bits xi are independent random variables with marginals Q(yi |xi )/(Q(yi |0) + Q(yi |1)). Sampling from this measure is very easy. For β ∈ ]0, ∞[, µy,β ( · ) can be regarded as a distribution of possible channel inputs for a code with ‘soft’ parity check constraints. Note that, unlike the β = ∞ case, this distribution depends in general on the actual parity check matrix and not just on the codebook C. This is actually a good feature of the tilted measure: the performance of practical algorithms does indeed depend upon the representation of the parity check matrix of C. It is therefore necessary to take it into account. We sample from µy,β ( · ) using Glauber dynamics, (see Section 13.2.1). We have already seen in that section that decoding through sampling at a fixed β fails above a certain noise level. Let us now try to improve on that method using a simulatedannealing procedure in which β is increased gradually according to an annealing schedule β(t), with β(0) = 0. The following decoder uses as input the received word y, the annealing schedule, and some maximal numbers of iterations tmax , n. Simulated Annealing Decoder (y, {β(t)}, tmax , n) 1: Generate x∗ (0) from µy,0 ( · ); 2: for t = 0, . . . tmax − 1: 3: Set x(0; t) = x∗ (t − 1); 4: Let x(j; t), j ∈ {1, . . . , n} be the configurations produced by n successive Glauber updates at β = β(t); 5: Set x∗ (t) = x(n; t); 6: end 7: return x(tmax ). The algorithmic complexity of the decoder is proportional to the total number of Glauber updates ntmax . If we want the algorithm to be efficient, this should grow linearly or slightly superlinearly with N . The intuition is that the first (small-β) steps allow the Markov chain to equilibrate across the configuration space, whereas, as β gets larger, the sample becomes concentrated onto (or near to) codewords. Hopefully, at each stage x∗ (t) will be approximately distributed according to µy,β(t) ( · ).

Local search algorithms and metastable states 0.30 0.25 0.20

0.30 τ =1 τ =10 2 τ =10

0.25 0.20

eann 0.15

eann 0.15

0.10

0.10

0.05

0.05

0.00 0.0

0.2

0.4

0.6

T

0.8

1.0

1.2

0.00 0.0

τ =10 2 τ =103 τ =10

0.2

0.4

0.6

0.8

1.0

1.2

T

Fig. 21.2 Decoding random codes from the (5, 6) LDPC ensemble using simulated annealing. Here we consider a block length N = 12 000 and transmission over BSC(p) with p = 0.12 (left) and 0.25 (right). The system was annealed through tmax = 1200 temperature values equally spaced between T = 1.2 and T = 0. At each temperature, n = N τ updates were executed. The statistical errors are comparable to the size of the jumps along the curves.

Figure 21.2 shows the result obtained by use of this simulated-annealing decoder, using random LDPC codes from the (5, 6) regular ensemble, over a binary symmetric channel with crossover probabilities p = 0.12 and 0.25 (for this ensemble, pd ≈ 0.139 and pc ≈ 0.264). The annealing schedule was linear in the temperature, namely β(t) = 1/T (t), where 8 7 t , (21.5) T (t) = T (0) − T (0) − T (tmax ) tmax with T (0) = 1.2 and T (tmax ) = 0. The performance of the decoding can be evaluated through the number of violated checks in the final configuration, which is half of E(x(tmax )). The figure shows the energy density averaged over 10 repetitions of the decoding experiment (each time with a new code randomly chosen from the ensemble), e(t) = (1/N )E(x(t)), versus the temperature T (t). As the number of updates performed at each temperature increases, the number of violated checks per variable seems to converge to a well-defined limiting value that depends on t only through the corresponding temperature: 1 E(x(t)) → eann (β(t)) . (21.6) N Further, E(x(t))/N seems to concentrate around its mean as N → ∞. At small p, eann (β) quickly converges to 0 as β → ∞: a codeword (the transmitted one) is found efficiently. In fact, even at β = 1, the numerical result for eann (β) is indistinguishable from 0. We expect that eann (β) will coincide within the numerical accuracy with the theoretical prediction for the equilibrium average 1 lim E(x)β . (21.7) eeq (β) ≡ N N →∞ This agrees with the above observations, since eeq (β) = O(e−10β ) (the lowest excitation over the ground state amounts to flipping a single bit; its energy is equal to 10). The

¼¼


numerics thus suggest that x(tmax ) is indeed approximately distributed according to µy,β(t) ( · ). At large p, eann (β) has instead a non-vanishing β → ∞ limit: the annealing algorithm does not find any codeword. The returned word x∗ (tmax ) typically violates Θ(N ) parity checks. On the other hand, in the equilibrated system at β = ∞, the energy vanishes by construction (we know that the transmitted codeword satisfies all checks). Therefore the simulation has fallen out of equilibrium at some finite β, thus yielding a distribution of x(tmax ) which is very different from µy,β=∞ ( · ). The data in Fig. 21.2 show that the energy varies very slowly at low temperatures, which confirms the fact that the system is out of equilibrium. We shall argue below that this slowing down is in fact due to a dynamical glassphase transition occuring at a well-defined temperature Td = 1/βd . Below this temperature, x(tmax ) gets trapped with high probability into a pure state corresponding to a deep local minimum of E(x) with positive energy, and never reaches a global minimum of the energy (i.e. a codeword). This is related to the ‘energy landscape’ picture discussed in the previous subsection. Indeed, the success of the simulated-annealing decoder for p ≤ pd can be understood as follows. At small noise, the ‘tilting’ factor i Q(yi |xi ) effectively selects a portion of the configuration space around the transmitted codeword (more or less like the likelihood constraint above), and this portion is small enough that there is no metastable state inside it. An interesting aspect of simulated-annealing decoding is that it can be analysed on the basis of a purely static calculation. Indeed, for any β ≤ βd , the system is still in equilibrium and its distribution is simply given by eqn (21.4). Its study, and the determination of βd , will be the subject of the next sections. Before moving to this analysis, let us make a last remark about simulated annealing: for any finite β, the MCMC algorithm is able to equilibrate if it is iterated a large number of times (a direct consequence of the fact that Glauber dynamics is irreducible and aperiodic). This raises a paradox, as it seems to imply that the annealing energy always coincides with the equilibrium energy, and the system never falls out of equilibrium during the annealing process. The solution to the conundrum is that in the previous discussion, we tacitly assumed that the number of Monte Carlo steps cannot grow exponentially with the system size. To be more precise, one can, for instance, define the annealing energy as eann (β) ≡

lim

lim

tmax →∞ N →∞

1 EN (x(tβ = (1 − β(0)/β)tmax )) , N

(21.8)

where we have assumed β(tmax ) = ∞ The important point is that the limit N → ∞ is taken before tmax → ∞: in such a case, simulated annealing can be trapped in metastable states.

21.2

The binary erasure channel

If communication takes place over the binary erasure channel BEC(), the analysis of metastable states can be carried out in detail by adopting the point of view of constrained optimization introduced in Section 21.1.1.


¼½

Suppose that the all-zero codeword x∗ = (0, . . . , 0) has been sent, and let y ∈ {0, ∗}N be the channel output. We shall denote by U = U (y) the set of erased bits. The log-likelihood for the word x to be the input can take two possible values: Ly (x) = |U | log if xi = 0 for all i ∈ U , and Ly (x) = −∞ otherwise. Of course, the input codeword belongs to the first set: Ly (x∗ ) = |U | log . The strategy of Section 21.1.1 reduces therefore to minimizing E(x) (i.e. minimizing the number of violated parity checks) among all configurations x such that xi = 0 on all of the non-erased positions. When the noise is smaller than the MAP threshold, there is a unique minimum with energy 0, namely the transmitted codeword x∗ . Our aim is to study the possible existence of metastable states, using the energetic cavity method of Section 19.5. This problem is closely related to XORSAT, whose analysis was presented in Chapters 18 and Chapter 19. Once all the non-erased bits have been fixed to xi = 0, decoding amounts to solving a homogeneous system of linear equations for the remaining bits. If one uses a code from the ensemble LDPCN (Λ, P ), the degree profiles of the remaining nodes are Λ(x) and R(x), where the probability of a check node to have degree k, Rk , is given in terms of the original Pk by Rk =

k max

k k (1 − )k −k , P k k

k =k

(21.9)

and the corresponding edge-perspective degree profile is given as usual by rk = kRk / p pRp .

Exercise 21.5 Show that r(u) =

P k

rk uk−1 = ρ(1 − (1 − u)).

Assuming as usual that the number of metastable states—solutions of the min-sum equations—of energy N e grows like exp(N Σe (e)), we shall use the 1RSB energetic cavity method to compute the energetic complexity Σe (e). This can be done using the SP(y) equations on the original factor graph. As our problem involves only hard constraints and binary variables, we can use the simplified formalism of Section 19.5.3. Each min-sum message can take three possible values, 0 (the meaning of which is ‘take value 0’), 1 (‘take value 1’) and ∗ (‘you can take any value’). The SP(y) messages are distributions on these three values or, equivalently, normalized triplets. 21.2.1

The energetic 1RSB equations

Let us now turn to the statistical analysis of these messages. We denote by Q =

the messages from (Q0 , Q1 , Q∗ ) the messages from variable to check nodes, and by Q check to variable nodes. We first note that, if a bit is not erased, then it sends a sure 0 message Q = (1, 0, 0) to all its neighbouring checks. This means that the distribution of Q has a mass of at least 1 − on sure 0 messages. We can write (1, 0, 0) with probability (1 − ) , Q= (21.10) Q with probability .

¼¾


and Q

can then be obtained exactly as in SecThe distributional equations for Q tions 19.5 and 19.7.

e and Q b satisfy the equations Exercise 21.6 Show that the distributions of Q d eσ = b1 , · · · , Q b l−1 ) , Q Fl,σ (Q 1 1 0 1 Qk−1 i Qk−1 e i 1 e ei ei b0 Q i=1 (Q0 + Q1 ) + 2 Qi=1 (Q0 − Q1 ) 2 Q d @ 1 k−1 (Q @Q e i0 + Q e i1 ) − 1 k−1 (Q e i0 − Q e i1 ) A , b1 A = i=1 i=1 2 2 Qk−1 b e Q∗ 1 − i=1 (1 − Q∗,i )

0

(21.11) (21.12)

where we have defined, for σ ∈ {0, 1, ∗}, b1 , . . . , Q b l−1 ) ≡ Fl,σ (Q b a }) ≡ Zl,σ ({Q

b a }) Zl,σ ({Q , b a }) + Zl,1 ({Q b a }) + Zl,∗ ({Q b a }) Zl,0 ({Q (σ) X

e−y min(|Ω0 |,|Ω1 |)

Ω0 ,Ω1 ,Ω∗

Y

a∈Ω0

b a0 Q

Y a∈Ω1

b a1 Q

Y

(21.13) b a∗ . Q

(21.14)

a∈Ω∗

P(σ) Here we have denoted by Ω0 ,Ω1 ,Ω∗ the sum over partitions of {1, . . . , l − 1} = Ω0 ∪ Ω1 ∪ Ω∗ such that |Ω0 | > |Ω1 | (for the case σ = 0), |Ω0 | = |Ω1 | (for σ = ∗), or |Ω0 | < |Ω1 | (for σ = 1). Furthermore, k and l are random integers, with distributions rk and λl respectively, the e and the {Q b a } are k − 1 i.i.d. copies of Q. b e i } are l − 1 i.i.d. copies of Q, {Q

Given a solution of the 1RSB equations, we can compute the Bethe free-entropy

of the auxiliary problem. Within the 1RSB cavity method, we can density FRSB,e (Q, Q) estimate the free-entropy density of the auxiliary model using the Bethe approximation

This gives access to the energetic complexity function as Fe (y) = (1/N )FRSB,e (Q, Q). e Σ (e) through the Legendre transform Fe (y) = Σe (e) − y e. Within the 1RSB cavity method, we can estimate the latter using the Bethe approximation: Fe (y) = f RSB,e (y).

Exercise 21.7 Computation of the free entropy. Using eqn (19.91), show that the Bethe free entropy of the auxiliary graphical model is N f RSB,e + o(N ), where e Q) b + E log zv ({Q b a }; l) f RSB,e = −Λ (1) E log ze (Q, +

Λ (1) e i }; k) . E log zf ({Q P (1)

(21.15)

Here the expectations are taken over l (with distribution Λl ), k (with distribution Rk , b a . The contributions of edges (ze ), e Q, b and their i.i.d. copies Q ei , Q defined in eqn (21.9)), Q, variable nodes (zv ) and function nodes (zf ) take the form


” “ e Q) b = 1 + (e−y − 1) Q e0 Q e1 Q b1 + Q b0 , ze (Q, X Y b Y b Y b −y min(|Ω |,|Ω |) 0 1 b i }; l) = b0 b1 b∗ e zv ({Q , Q Q Q Ω0 ,Ω1 ,Ω∗ b∈Ω0

e i }; k) = 1 + 1 (e−y − 1) zf ({Q 2

b∈Ω1

(

k Y

i=1

b∈Ω∗

e i0 + Q e i1 ) − (Q

k Y

¼¿

(21.16) (21.17)

) e i0 − Q e i1 ) (Q

,

(21.18)

i=1

where the sum in the second equation runs over the partitions Ω0 ∪ Ω1 ∪ Ω∗ = [l].

21.2.2

The BP threshold and the onset of metastability

A complete study of the distributional equations (21.11) and (21.12) is a rather challenging task. On the other hand, they can be solved approximately through population dynamics. It turns out that the distribution obtained numerically shows different sym (or Q)

to metry properties depending on the value of . We define a distribution Q 1 , and ‘positive’ if Q 0 > Q 1 . We know from the analysis of 0 = Q be ‘symmetric’ if Q BP decoding that directed edges in the graph can be divided into two classes: those that eventually carry a message 0 under BP decoding, and those that instead carry a message ∗ even after a BP fixed point has been reached. It is natural to think that (i.e., even among edges of the first class correspond to a positive 1RSB message Q metastable states, the corresponding bits are biased towards 0), while edges of the second class correspond instead to a symmetric message Q. and Q.

This suggests the following hypothesis concerning the distributions of Q ˆ ˆ We assume that there exist weights ξ, ξ ∈ [0, 1] and random distributions b, b, c, ˆc, ˆ are symmetric, c, ˆc are positive, and such that b, b b with probability ξ, d (21.19) Q= c with probability 1 − ξ, ˆ with probability ξ, ˆ b d

= Q (21.20) ˆ ˆc with probability 1 − ξ.

respectively, are symIn other words ξ and ξˆ denote the probabilities that Q and Q, metric. to be symmetric, all of the inputs Q

i Equation (21.11) shows that, in order for Q

must be symmetric. On the other hand, eqn (21.12) implies that Q is symmetric if at a is symmetric. Using the result of Exercise 21.5, we thus find least one of the inputs Q that our Ansatz is consistent only if the weights ξ, ξˆ satisfy the equations ˆ , ξ = λ(ξ)

ξˆ = 1 − ρ(1 − ξ) .

(21.21)

ˆ these coincide with the density-evolution fixed-point If we define z ≡ ξ and zˆ ≡ ξ, conditions for BP (see eqns (15.34)). This is not surprising, in view of the physical discussion which lead us to introduce the Ansatz in eqns (21.19) and (21.20): ξ corresponds to the fraction of edges that remain erased at the BP fixed point. On the


0.06

0.03

= 0.45

0.03 0

0.02

0.06

= 0.50

Σ(e)0.03

0.01

ed

0

ec

0.06

0

0.03

d

= 0.55

0 0

0.005

0.01

0.015

0.02

0

0.2

e

c

0.4

0.6

0.8

Fig. 21.3 Metastable states for random elements of the (3, 6) regular ensemble used over BEC() (for this ensemble, d ≈ 0.4294 and c ≈ 0.4882). Left: complexity as a function of the energy density for three values of the channel parameter above d . Right: the maximum and minimum energy densities ed and ec of the metastable states as a function of the erasure probability.

other hand, we shall see that this observation implies that BP ceases to converge to the correct fixed point at the same threshold noise d where metastable states start to appear. For ≤ d , eqns (21.21) admit the unique solution ξ = ξˆ = 0, corresponding to the fact that BP decoding recovers the full transmitted message. As a consequence, we d d

·) = ˆc( · ) to have an almost surely positive mean. In fact, it can take Q( · ) = c( · ), Q( is not hard to check that a consistent solution of eqns (21.11) and (21.12) is obtained by taking

=Q = (1, 0, 0) Q

almost surely.

(21.22)

Since the cavity fields do not fluctuate from state to state (their distribution is almost surely a point mass), the structure of this solution indicates that no metastable state is present for ≤ d . This is confirmed by the fact that the free-entropy density of this solution, Fe (y), vanishes for all y. Above a certain noise threshold, i.e. for > d , eqns (21.21) still possess the solution ξ = ξˆ = 0, but a new solution with ξ, ξˆ > 0 appears as well. We have discussed this new solution in the density evolution analysis of BP decoding: it is associated with the fact that the BP iterations have a fixed point at which a finite fraction of the bits remain undetermined. Numerical calculations show that that, for > d , the iteration and of eqns (21.11) and (21.12) converges to a non-trivial distribution. In particular, Q

are found to be symmetric with probability ξ > 0 and ξˆ > 0, respectively, where Q the values of ξ, ξˆ are the non-trivial solution of eqns (21.21). The free entropy of the auxiliary model, Fe (y), can be computed using eqn (21.15). Its Legendre transform is the energetic-complexity curve Σe (e).

1

The binary erasure channel 0.12

0.12

0.10

0.10 τ = 10 2 τ = 10 3 τ = 10

0.08 0.06

0.08 τ = 10 2 τ = 10 3 τ = 10

0.06

eann

eann

0.04

0.04

0.02

0.02

0

0 0

0.2

0.4

0.6

T

0.8

1.0

0

0.2

0.4

0.6

0.8

T

Fig. 21.4 Decoding random codes from the (3, 6) regular ensemble used over BEC(). In both cases, N = 104 , and the annealing schedule consisted of tmax = 103 equidistant temperatures in T = 1/β ∈ [0, 1]. At each value of the temperature, n = N τ Monte Carlo updates were performed. Left, = 0.4 < d . Right, = 0.6 > d ; the horizontal line corresponds to the energy density of the highest metastable states ed ( = 0.6).

Figure 21.3 shows a typical outcome of such a calculation for LDPC ensembles when > d . In this whole regime, there exists a zero-energy word, the transmitted (all-0) codeword. This is described by the solution ξ = ξˆ = 0. On top of this, the non-trivial solution gives a complexity curve Σe (e) which is positive in an interval of energy densities (ec , ed ). A positive complexity means that an exponential number of metastable states is present. But, when their energy density e is strictly positive, these metastable states violate a finite fraction of the parity checks. As increases, both ed and ec decrease. At c , ec vanishes continuously, and ec = 0 and ed > 0 for all ≥ c . In other words, at noise levels larger than c , there appears an exponential number of zero-energy states. These are codewords that are separated by energy barriers with a height Θ(N ). Consistently with this interpretation, Σ(e = 0) = RS RS , where fh,u is the RS free-entropy density (15.48) estimated for the non-trivial fh,u fixed point of density evolution. The notion of metastable states thus allows us to compute the BP and MAP thresholds within a unified framework. The BP threshold is the noise level where an exponential number of metastable states appears. This shows that this threshold not only is associated with a specific decoding algorithm, but also has a structural, geometric meaning. On the other hand, the MAP threshold coincides with the noise level where the energy of the lowest-lying metastable states vanishes. Figure 21.4 shows the results of some numerical experiments with the simulatedannealing algorithm of Section 21.1.3. Below the BP threshold, and for a slow enough annealing schedule, the algorithm succeeds in finding a codeword (a zero-energy state) in linear time. Above the threshold, even at the slowest annealing rate, we could not find a codeword. Furthermore, the residual energy density at zero temperature is close to ed , suggesting that the optimization procedure is indeed trapped among the highest metastable states. This suggestion is confirmed by Fig. 21.5, which compares the dependence of ed with the residual energy under simulated annealing. Once again,

1


0.04 τ = 10 τ = 102 τ = 103 ed

0.03

eann (T = 0)

0.02

0.01

0 0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Fig. 21.5 Decoding random codes from the (3, 6) regular ensemble used over BEC(). Here, we plot the minimum energy density achieved through simulated annealing versus the channel parameter. The continuous line shows the energy of the highest-lying metastable states. The size and annealing schedule were as in Fig. 21.4.

there is rough agreement between the two (let us stress that one should not expect perfect agreement between the residual energy in Fig. 21.5 and ed : the former does indeed depend on the whole dynamical annealing process).

21.3

General binary memoryless symmetric channels

One would like to generalize to other channel models the above analysis of metastable states in the constrained-optimization formulation of decoding. In general, the computation is technically more intricate than for a BEC. The reason is that, for general channels, the distance condition Ly (x) ≥ −N (h + δ) cannot be written in terms of ‘local’ binary constraints. As a consequence, one cannot use the simplified approach of Section 19.5.3, and the general 1RSB formalism is required. We shall follow this line of approach, but rather than push it to the point of determining the full complexity function, we shall only determine whether the model (21.4) undergoes a dynamical phase transition as β increases from 0 to ∞, and locate the critical point βd (p) (here p denotes the channel parameter). This is, indeed, the most important piece of information for our purposes. If a dynamical phase transition occurs at some βd < ∞, then for β > βd the measure (21.4) decomposes into an exponential number of metastable pure states. As β crosses βd , the system is trapped in one of these and falls out of equilibrium. Upon further cooling (increase of β), the energy density of the annealed system remains higher than the equilibrium energy density and does not vanish as β → ∞. This analysis allows us to determine the noise threshold of the simulated-annealing decoder as the largest noise level p such that there is no finite βd . In the following, we first write the general 1RSB equations at finite β, and present some results obtained by solving them numerically. Finally, we give a heuristic argument showing that βd (p) goes to infinity for exactly p ↓ pd .


21.3.1

The 1RSB cavity approach

We shall apply the 1RSB cavity approach of Chapter 19 to the decoding problem. Given a code and a received message y, we want to study the probability distribution µy,β (x) defined in eqn (21.4), and understand whether it decomposes into exponentially many extremal Bethe measures. The BP equations are simple generalizations of those written in Chapter 15 for the case β = ∞. In terms of the log-likelihoods νi→a (0) ν a→i (0) 1 1 log , ua→i = log , 2 νi→a (1) 2 ν a→i (1) Q(yi |0) 1 ≡ B(yi ) , Bi = log 2 Q(yi |1)

hi→a =

(21.23)

they read hi→a = Bi +

ub→i ≡ fi ({ub→i }) ,

(21.24)

b∈∂i\a

4 3 ua→i = atanh tanh β tanh hj→a ≡ ˆfa ({hj→a }) .

(21.25)

j∈∂a\i

The corresponding Bethe free entropy is given by the following equation (unlike in Chapter 15, we use natural logarithms here):

F(u, h) = −

log

(ia)∈E

+

M a=1

⎡

log ⎣

ν ua→i (xi )νhi→a (xi ) +

xi

exp(−βEa (x∂a ))

x∂a

⎤

N i=1

log

xi

Q(yi |xi )

ν ua→i (xi )

a∈∂i

νhi→a (xi )⎦ .

(21.26)

i∈∂a

As in eqn (15.44), we introduce a ‘shifted’ free-entropy density φ, defined as φ=

1 F(u, h) − Q(y|0) log Q(y|0) . N y

(21.27)

Recall that the 1RSB cavity approach assumes that, to leading exponential order, the number N (φ) of Bethe measures with a shifted free-entropy density equal to φ is equal to the number of quasi-solutions of eqns (21.24) and (21.25). We shall write, as . usual, N (φ) = exp(N Σ(φ)), and our aim is to compute the complexity Σ(φ), using as in Chapter 19 an auxiliary graphical model which counts the number of solutions of the BP equations, weighted by a factor exp(N xφ). If the free entropy of the auxiliary model is F(x) = limN →∞ FRSB (x)/N , then Σ(φ) is given by the Legendre transform F(x) = xφ + Σ(φ), ∂Σ/∂φ = −x. For a given code and received y, the basic objects involved in the 1RSB approach

bj . are the distributions of the fields hi→a and ub→j , denoted respectively by Qia and Q They satisfy the following 1RSB equations:


Qia (hi→a ) ∼ =

ai (ua→i ) ∼ Q =

δ (hi→a = fi ({ub→i })) (zia )x

bi (ub→i ) , dQ

(21.28)

b∈∂i\a

% & δ ua→i = ˆfa ({hj→a }) (ˆ zai )x dQja (hj→a ) .

(21.29)

j∈∂a\i

Exercise 21.8 Show that the factors zia and zâi in these equations, defined in eqns (19.22) and (19.23), are given by zia ({ub→i }, Bi ) =

P 2 cosh(Bi + b∈∂i\a ub→i ) Q , b∈∂i\a (2 cosh(ub→i ))

zâi ({hj→a }) = 1 + e−2β .

(21.30) (21.31)

Although in this case zâi is a constant and can be absorbed in the normalization, we shall keep it explicitly in the following. We now turn to the statistical analysis of the above equations. If we pick out a uniformly random edge in the factor graph of a code from the ensemble LDPCN (Λ, P ), the

and Q themselves become random objects, which satisfy the distributional densities Q equations % & l−1 d 1

a (ua ) , z({ua }; B(y))x δ h − fl−1 ({ua }; B(y)) Q(h) = dQ (21.32) Z a=1 % & k−1 d 1

zˆ({hi })x δ u − ˆfk−1 ({hi }) Q(u) = dQi (hi ) (21.33) Z

i=1

and the {Qi }

a } are l − 1 i.i.d. copies of Q, where k, l, y are random variables, the {Q are k − 1 i.i.d. copies of Q. Further, l is drawn from the edge-perspective variabledegree profile λ, k is drawn from the edge-perspective check-degree profile ρ, and y is drawn from Q( · |0), the distribution of the channel output upon input 0. The k−1 l−1 functions ˆfk−1 ({hi }) = atanh(tanh β i=1 tanh(hi )) and fl−1 ({ua }; B) = B − a=1 ua are defined analogously to eqns (21.24) and (21.25). The functions z( · ) and zˆ( · ) are given similarly by the expressions in eqns (21.30) and (21.31). The 1RSB free-entropy density (i.e. the entropy density of the auxiliary model) is

where f RSB (Q, Q)

is the expected free-entropy density, estimated as F(x) = f RSB (Q, Q),

and Q and Q are distributed according to the ‘correct’ solution of the distributional equations (21.32) and (21.33):

= −Λ (1) E log ze (Q, Q)

+ E log zv ({Q

a }; l, y) + f RSB (Q, Q)

Λ (1) E log zf ({Qi }; k) . P (1)

and l i.i.d. copies of Here the expectation is taken with respect to k i.i.d. copies of Q d d d Q, and with respect to k = P· , l = Λ· , and y = Q( · |0). Finally, ze , zv , zf read


= ze (Q, Q)

a }; l, y) = zv ({Q

dQ(h) dQ(u)

l

a (ua ) dQ

a=1

zf ({Qi }; k) =

l

⎡ dQi (hi ) ⎣

× I

νh (x)νu (x)

,

x=0 1 l Q(y|x)

Q(y|0)

k

xi = even

−2β

+e

x νua (x)

,

(21.35)

νhi (xi )

i

(21.34)

a=1

{x1 ,··· ,xk } i=1

i=1

x

1

x=0

I

x xi = odd

. (21.36)

i

A considerable amount of information is contained in the 1RSB free-energy density F(x). For instance, one could deduce the energetic complexity from it by taking the appropriate β → ∞ limit. Here, we shall not attempt to develop a full solution of the 1RSB distributional equations, but shall use them to detect the occurrence of a dynamical phase transition.

21.3.2

The dynamical phase transition

The location of the dynamical phase transition βd (p) is determined by the smallest value of β such that the distributional equations (21.32) and (21.33) have a nontrivial solution at x = 1. For β > βd (p), the distribution (21.4) decomposes into an exponential number of pure states. As a consequence, we expect simulated annealing to fall out of equilibrium when βd (p) is crossed. In the top part of Fig. 21.6, we show the result of applying such a technique to the (5, 6) regular ensemble used for communication over BSC(p). At small p, no dynamic phase transition is revealed through this procedure at any positive temperature. Above a critical value of the noise level p, the behaviour changes dramatically, and a phase transition is encountered at a critical point βd (p) that decreases monotonically for larger p. By changing both β and p, one can identify a phase transition line that separates the ergodic and non-ergodic phases. Remarkably, the noise level at which a finite βd appears is numerically indistinguishable from pd ≈ 0.145. Does the occurrence of a dynamical phase transition for p pd indeed influence the behaviour of the simulated-annealing decoder? Some numerical confirmation was presented earlier in Fig. 21.2. Further support in favour of this thesis is provided by the bottom part of Fig. 21.6, which plots the residual energy density of the configuration produced by the decoder as β → ∞. Above pd , this becomes strictly positive and only slowly dependent on the cooling rate. It is compared with the equilibrium value of the internal energy at βd (p). This would be the correct prediction if the energy of the system did not decrease any more after the system fell out of equilibrium at βd (p). Although we do not expect this to be strictly true, the resulting curve provides a good first estimate.

½¼


e−2β 0.10 0.08 0.06 0.04 0.02

pd p

0.00 0

eann

0.1

0.2

0.3

0.4

0.5

0.18 τ = 10 τ = 102 τ = 103 ε d ( Td )

0.16 BSC 0.14 0.12 0.1 0.08 0.06 0.04 0.02

pd

0 0

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5

p

Fig. 21.6 Top: location of the dynamic phase transition for random codes from the (5, 6) ensemble used over BSC(p) (the circles were obtained through sampled density evolution; the dashed line is a guide for the eye). Bottom: residual energy density after simulated annealing, as measured in numerical simulations. The dashed line gives the equilibrium energy at the dynamical transition temperature Td .

21.3.3

Metastable states and the BP threshold

One crucial element of this picture can be confirmed analytically, for a generic BMS channel family ordered by physical degradation with respect to p: at zero temperature, the dynamical transition, which signals the proliferation of metastable Bethe states, occurs exactly at the decoding threshold pd . More precisely, the argument below proves that at β = ∞ there cannot exist any non-trivial x = 1 solution of eqns (21.32) and (21.33) for p < pd , while there exists one for p > pd . We expect that, for most channel families, the same situation should hold for β large enough (dependent on p), but this has not been proven yet. Let us consider the 1RSB equations (21.32) and (21.33) in the case β = ∞. Assuming that the degree profiles are such that l ≥ 2 and k ≥ 2 (a reasonable requirement


½½

for useful code ensembles), it is clear that they have a special ‘no-error’ solution as

sociated with the sent codeword in which Q(h) = δ∞ (h) and Q(u) = δ∞ (h) almost surely. It is a simple exercise to check that the (shifted) free-entropy density of this solution is equal to 0. The important question is whether there exist other solutions beyond the ‘noerror’ one. We can make use of the simplification occuring at x= 1. As we saw in av (xi ) ≡ νia Qia (νia )νia (xi ) Section 19.4.1, the expectation values of the messages, νi→a av

and ν a→i (xi ) ≡ bmai Qai (

νai )

νai (xi ), satisfy the BP equations. Let us first study the case p < pd . We have seen in Chapter 15 that there is a unique solution of the BP equations: the no-error solution. This shows that in this low-noise regime, there cannot exist any non-trivial 1RSB solution. We conclude that there is no glass phase in the regime p < pd We now turn to the case p > pd (always with β = ∞), and use the analysis of BP presented in Chapter 15. That analysis revealed that, when p > pd , the density evolution of the BP messages admits at least one ‘replica-symmetric’ fixed point distinct from the no-error one. We shall now use this replica-symmetric fixed point in order to construct a nontrivial 1RSB solution. The basic intuition behind this construction is that each Bethe measure consists of a single configuration, well separated from other ones. Indeed, we expect that each Bethe measure can be identified with a zero-energy configuration, i.e. with a codeword. If this is true, then, with respect to each of these Bethe measures, the local distribution of a variable is deterministic, either a unit mass on 0 or a unit mass

is supported on on 1. Therefore we seek a solution where the distribution of Q and Q functions of the form 1 ˜ δ+∞ (h) + (1 + tanh h) 2 1

˜) δ+∞ (u) + Q(u) = (1 + tanh u 2

Q(h) =

1 ˜ δ−∞ (h) , (1 − tanh h) 2 1 (1 + tanh u ˜) δ−∞ (u) , 2

(21.37) (21.38)

˜ and u where h ˜ are random variables. Exercise 21.9 Show that this Ansatz solves eqns (21.32) and (21.33) at β = ∞ if and ˜ and u only if the distributions of h ˜ satisfy d ˜= B(y) + h

l−1 X a=1

u ˜,

d

u ˜ = atanh

h k−1 Y

i ˜i . tanh h

(21.39)

i=1

˜ and u It is easy to check that the random variables h ˜ satisfy the same equations as the fixed point of the density evolution for BP (see eqn (15.11)). We conclude that, for p > pd and x = 1, a solution to the 1RSB equations is given by the Ansatz in ˜ and u eqns (21.37) and (21.38) if h ˜ are drawn from the fixed-point distributions of eqn (15.11).

½¾

Glassy states in coding theory 0.15

0.2

0.1

0.15

F(x)

Σ(φ)

0.05

0.1

0

0.05

-0.05 0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

0 -0.2

-0.15

-0.1

-0.05

0

φ

x

Fig. 21.7 Left: the free entropy of the auxiliary model, F(x), as a function of the weight parameter x, for a (3, 6) code on a BSC (recall that pd ≈ 0.084 and pc ≈ 0.101 in this case). From bottom to top, p = 0.090, 0.095, 0.100, 0.105, 0.110. Right: the complexity Σ(φ) plotted versus the shifted free-entropy density φ. From left to right, p = 0.090, 0.095, 0.100, 0.105, 0.110.

It turns out that a similar solution is easily found for any value of x > 0, provided β = ∞. The only place where x plays a role is in the reweighting factor of eqn (21.35): when x = 1, the only modification in the distributional equations (21.39) is that B(y) must be multiplied by x. Therefore one can obtain the 1RSB solution for any x > 0 if one knows the solution to the RS cavity equations (i.e. the fixed point of the density evolution for BP) for a slightly modified problem in which B(y) is changed to xB(y). Technically, this is equivalent to studying the modified measure µy (x) ∼ =

M

I(xia1 ⊕ · · · ⊕ xiak(a) = 0)

a=1

N

Q(yi |xi )x ,

(21.40)

i=1

within the RS approach of Chapter 15 (such a modified measure has already been introduced in Chapter 6). Let us assume that we have found a non-trivial fixed point for this auxiliary prob(x) (x) lem, characterized by the distributions aRS (h) and âRS (u), and denote by f RS (x) the corresponding value of the free-entropy density defined in eqn (15.45). The 1RSB equations with reweighting parameter x have a solution of the type (21.37) and (21.38), (x) ˜ is distributed according to a(x) ( · ), and u ˜ is distributed according to âRS ( · ). provided h RS RSB The 1RSB free-entropy density F(x) = E F (x)/N is given simply by F(x) = f RS (x) .

(21.41)

Therefore the problem of computing F(x), and its Legendre transform, the complexity Σ(φ), reduce to a replica-symmetric computation. This is a simple generalization of the problem considered in Chapter 15, where the decoding measure is modified by raising it to the power x, as in eqn (21.40). Notice, however, that the interpretation is now different. In particular, x has to be properly chosen in order to focus on dominant pure states. The problem can easily be studied numerically using the population dynamics algorithm. Figure 21.7 shows an example of the complexity Σ(φ) for a binary symmetric

0.05

Metastable states and near-codewords

½¿

100 10−1

Pb

10−2 10−3 10−4 10−5 0.0

0.2

0.4

0.6

0.8

1.0

Fig. 21.8 Bit error probability for 40 random elements of the (3, 6) regular ensemble with N = 2500 used over the BEC(). The continuous curve corresponds to the average error probability.

channel. The regime pd < p < pc is characterized by the existence of a band of metastable states with a negative shifted free entropy φ ≤ φ0 < 0. They are, in principle, irrelevant when compared with the ‘no-error’ solution, which has φ = 0, confirming that MAP decoding will return the transmitted codeword. In fact, they are even unphysical: φ is nothing but the conditional entropy density of the transmitted codeword given the received message. As a consequence, it must be non-negative. However, the solution extends to β < ∞, where it makes perfect sense (it describes non-codeword metastable configurations), thus solving the puzzle. The appearance of metastable states coincides with the noise threshold above which BP decoding fails. When p > pc , the top end of the band φ0 becomes positive: the ‘glassy’ states dominate the measure and MAP decoding fails.

21.4

Metastable states and near-codewords

In a nutshell, the failure of BP decoding for p > pd can be traced back to configurations (words) x that (i) are deep local minima of the energy function E(x) (which counts the number of violated parity checks), and (ii) have a significant weight under the measure i Q(y| xi ). Typically, such configurations are not codewords, although they can be very close to codewords from the energy point of view. An interesting qualitative analogy can be drawn between this analysis, and various notions that have been introduced to characterize the error floor. Let us start by describing the error floor problem. We saw that, for p < pd , the bit error rate under BP decoding vanishes when the block length N → ∞. Unhappily, the block length cannot be taken arbitrarily large, because of two types of practical


considerations. First, coding a block of N bits simultaneously implies a communication delay proportional to N . Second, any hardware implementation of BP decoding becomes increasingly difficult as N gets larger. Depending on the application, one can be forced to consider a maximum block length between 103 and 105 . This brings up the problem of characterizing the bit error rate at moderate block length. Figure 21.8 shows the outcomes of numerical simulations for random elements of the (3, 6) ensemble used over an erasure channel. One can clearly distinguish two regimes: a rapid decrease of the error probability in the ‘waterfall region’ d ≈ 0.429 (in physics terms, the ‘critical regime’); and a flattening at lower noise values, in the ‘error floor’. It is interesting to note that the error floor level is small but highly dependent (in relative terms) on the graph realization. We know that the error floor should vanish when codes with larger and larger block length are taken, but we would like a prediction of its value given the graph G. With the notable exception of an erasure channel, this problem is largely open. However, several heuristics have been developed. The basic intuition is that the error floor is due to small subgraphs of the factor graph that are prone to error. If U is the set of variable nodes in such a subgraph, we can associate with it a configuration x that takes the value 1 on U and 0 otherwise (throughout our analysis, we are assuming that the codeword 0 has been transmitted). This x need not be a codeword but should in some sense be ‘close’ to one. Once a class F of such subgraphs has been identified, the error probability is estimated by assuming that any type of error is unlikely, and errors in different subsets are roughly independent: P {BP decoder fails on U } . (21.42) PB (G) ≈ U ∈F

If the subsets U are small, each of the terms on the right-hand side can be evaluated efficiently via importance sampling. It is interesting to take a look at some definitions of the class of subgraphs F that have been introduced in the literature. In each case, the subgraph is characterized by two integers (w, e) that describe how dangerous/close to codewords it is (small w or e corresponding to dangerous subgraphs). In practice, the sum in eqn (21.42) is restricted to small w, e. • Trapping sets (or near-codewords). A trapping set is a subgraph including the variable nodes in U , all of the adjacent check nodes, and the edges that connect them. It is a (w, e) near-codeword if the number of variable nodes is |U | = w and the number of check nodes of odd degree is e. In our framework, a trapping set is simply a configuration x with a weight (number of non-zero entries) equal to w and an energy E(x) = 2e. Notice that hardly any restriction is imposed on trapping sets. Special constraints are sometimes added, depending on the channel model and on the decoding algorithm (if not BP). • Absorbing sets. A (w, e) absorbing set is a (w, e) trapping set that satisfies two further requirements: (i) each variable node is adjacent to more check nodes of even degree (with respect to the subgraph) than of odd degree, and (ii) it does not contain

Notes

a (w , e) absorbing set with w < w. The first condition implies that the corresponding configuration x is a local minimum of E(x), stable with respect to one flip. The connection between small weak subgraphs and the error probability is still somewhat vague. The ‘energy landscape’ E(x) might provide some hints towards bridging this gap.

Notes This chapter is largely based on the analysis of metastable states given by Montanari (2001a, b) and Franz et al. (2002). One-step replica symmetry breaking was also investigated by Migliorini and Saad (2006). The approach was extended to asymmetric channels by Neri et al. (2008). The typical-pairs decoding presented here is slightly different from the original procedure of Aji et al. (2001). Stopping sets were introduced by Di et al. (2002), and inspired much of the subsequent research on error floors. The idea that small subgraphs of the factor graph are responsible for error floors was first convincingly demonstrated for general channel models by MacKay and Postol (2003) and Richardson (2003). Absorbing sets were defined by Dolecek et al. (2007). After its invention, simulated annealing was the object of a significant amount of work in the fields of operations research and probability. A review can be found in Aarts et al. (2003). A detailed comparison between 1RSB analysis and simulatedannealing experiments for models on sparse graphs was presented by Montanari and Ricci-Tersenghi (2004).


22 An ongoing story This book describes a unified approach to a number of important problems in information theory, physics, and computer science. We have presented a consistent set of methods to address these problems, but the field is far from being fully understood, and there remain many open challenges. This chapter provides a concise description of some of these challenges, as well as a survey of recent progress. Our ambition is to set an agenda for the newly developed field that we have been describing. We shall distinguish roughly three types of directions. The first one, to be discussed in Section 22.1, is the main challenge. It aims at a better qualitative understanding of models on sparse random graphs. At the core of the cavity method lies the postulate that such systems can have only a limited number of ‘behaviours’ (phases). Each phase corresponds to a different pattern of replica symmetry breaking (replica-symmetric (RS), one-step replica symmetry breaking (1RSB), etc.). In turn, these phases also have a description in terms of pure states decomposition, as well as in terms of long-range correlations. Understanding the fundamental reasons and conditions for the universality of these phases, as well as the equivalence between their characterizations, would be extremely important. The second direction, described in Section 22.2, concerns the development of the cavity formalism itself. We have focused mainly on systems in which either the RS or the 1RSB cavity method is expected to be asymptotically exact in the large-size limit. This expectation is, in part, based on some internal consistency checks of the 1RSB approach. An important such check is to verify that the 1RSB ‘solution’ is stable with respect to small perturbations. Whenever this test is passed, physicists feel confident enough that the cavity method provides exact conjectures (thresholds, minimum cost per variable, etc.). If the test is not passed, higher-order RSB is thought to be needed. The situation is much less satisfactory in this case, and the cavity method poses some technical problems even at the heuristic level. Section 22.3 lists a number of fascinating questions that arise in the connection between the existence of glassy-phase transitions and algorithmic slowdown. These are particularly important in view of applications in computer science and information theory: sparse graphical models can be useful for a number of practically relevant tasks, as the example of the use of LDPC codes in channel coding has shown. There is some empirical evidence that phase transitions have an impact on the behaviour and efficiency of algorithms. Physicists hope that this impact can be understood (to some extent) in a unified way, and that it is ultimately related to the geometric structure of the set of solutions and to correlation properties of the measure. While some general arguments in favour of this statement have been put forward, the actual understanding

An ongoing story

is still very poor.

22.1

Gibbs measures and long-range correlations

At an abstract level, the cavity method explored in the last few chapters relies on a (yet unproven) structural theorem. Consider a generic graphical model, for a probability distribution on N variables, x, taking values in a discrete space X N : µ(x) =

1 ψa (x∂a ) . Z

(22.1)

a∈F

The cavity method postulates that, for large classes of models taken from appropriate ensembles, the model is qualitatively described in the large-N limit by one out of a small number of generic scenarios, or phases. The postulated qualitative features of such phases are then cleverly used to derive quantitative predictions (e.g. the locations of phase transitions). Needless to say, we are not able to state precisely, let alone prove, such a structural theorem in this generality. The complete set of necessary hypotheses is unknown. However, we have discussed several examples, from XORSAT to diluted spin glasses and error-correcting codes. In principle, it is not necessary that the factor graph be locally tree-like, but, in practice, locally tree-like models are the ones that we can control most effectively. Such a structure implies that when one digs a cavity in the graph, the variables on the boundary of the cavity are far apart. This leads to a simple structure of their correlation in the large-system limit, and hence to the possibility of writing asymptotically exact recursion equations. Here, we do not want to discuss the necessary hypotheses in any more detail. It would certainly be a significant achievement to prove such a structural theorem even in a restricted setting (say, for the uniform measure over solutions of random K-SAT formulae). We want, instead, to convey some important features of the phases postulated within the cavity approach. In particular, there is a key aspect that we want to stress. Each of the various phases mentioned can be characterized from two complementary points of view: 1. In terms of decomposition of the distribution µ( · ) into ‘lumps’ or ‘clusters’. Below, we shall propose a precise definition of these lumps, and they will be called pure states. 2. In terms of correlations among far-apart variables on the factor graph. We shall introduce two notions of correlation decay that differ in a rather subtle way but correspond to different phases. These two characterizations are, in turn, related to the various aspects of the cavity method. 22.1.1

On the definition of pure states

The notion of a pure state is a crucial one in rigorous statistical mechanics. Unfortunately, standard definitions are tailored to translation-invariant models on infinite graphs. The graphical models that we have in mind are sparse random graphs (in this


class we include labelled random graphs, where the labels specify the nature of function nodes), and standard approaches do not apply to them. In particular, we need a concrete definition that is meaningful for finite graphs. Consider a sequence of finite graphical models {µN ( · )}, indexed by the number of variable nodes N . A pure-state decomposition is defined by assigning, for each N , a partition of the configuration space X N into NN subsets Ω1,N , . . . , ΩNN ,N : X N = Ω1,N ∪ · · · ∪ ΩNN ,N .

(22.2)

The pure-state decomposition must meet the following conditions: 1. The measure of each subset in the partition is bounded away from 1: max{µN (Ω1,N ), . . . , µN (ΩN ,N )} ≤ 1 − δ .

(22.3)

2. The subsets are separated by ‘bottlenecks.’ More precisely, for Ω ⊆ X N , we define its -boundary as ∂ Ω ≡ {x ∈ X N : 1 ≤ d(x, Ω) ≤ N } .

(22.4)

where d(x, Ω) is the minimum Hamming distance between x and any configuration x ∈ Ω. Then we require lim max

N →∞

r

µN (∂ Ωr,N ) = 0, µN (Ωr,N )

(22.5)

for some > 0. Note that the measure of ∂ Ωr,N can be small for two reasons, either because Ωr,N is small itself (and therefore has a small boundary) or because the boundary of Ωr,N is much smaller than its interior. Only the last situation corresponds to a true bottleneck, as is enforced by the denominator µN (Ωr,N ) in eqn (22.5). 3. The conditional measure on the subset Ωr,N , defined by µrN (x) ≡

1 µN (x)I(x ∈ Ωr,N ) , µN (Ωr,N )

(22.6)

cannot be decomposed further according to the two conditions above. Given such a partition, the distribution µN ( · ) can be written as a convex combination of distributions with disjoint support µN ( · ) =

NN

wr µrN ( · ) ,

wr ≡ µN (Ωr,N ) .

(22.7)

r=1

Note that this decomposition is not necessarily unique, as shown by the example below. The non-uniqueness is due to the fact that sets of configurations of X N with negligeable weight can be attributed to one state or another. On the other hand, the conditional measures µrN ( · ) should depend only weakly on the precise choice of decomposition.

¾¼

An ongoing story

Example 22.1 Consider a ferromagnetic Ising model on a random regular graph of degree (k + 1). The Boltzmann distribution reads ⎧ ⎫ ⎨ ⎬ 1 exp β µN (x) = xi xj , (22.8) ⎩ ⎭ ZN (β) (i,j)∈E

with xi ∈ X = {+1, −1}. To avoid irrelevant complications, let us assume that N is odd. Following the discussion in Section 17.3, we expect this distribution to admit a non-trivial pure-state decomposition for k tanh β > 1, with a partition Ω+ ∪ Ω− = X N . Here Ω+ and Ω− ) are the sets of configurations for which i xi is positive or negative, respectively. With respect to this decomposition, w+ = w− = 1/2. Of course, an (asymptotically) equivalent decomposition is obtained by letting Ω+ be the set of configurations with i xi ≥ C for some fixed C. It is useful to recall that the condition (22.5) implies that any ‘local’ Markov dynamics that satisfies detailed balance with respect to µN ( · ) is slow. More precisely, assume that µN (∂ Ωr,N ) ≤ exp{−∆(N )} . µN (Ωr,N )

(22.9)

Then any Markov dynamics that satisfies detailed balance with respect to µN and flips at most N variables at each step has a relaxation time larger than C exp{∆(N )} (where C is an N -independent constant that depends on the details of the model). Moreover, if the dynamics is initialized in x ∈ Ωr,N , it will take a time of order C exp{∆(N )} to get to a distance N from Ωr,N . In many cases based on random factor graph ensembles, we expect eqn (22.9) to hold with a ∆(N ) which is linear in N . In fact, in the definition of pure-state decomposition, we might require a bound of the form (22.9) to hold, for some function ∆(N ) (e.g. ∆(N ) = N ψ , with some appropriately chosen ψ). This implies that pure states are stable on time scales shorter than exp{∆(N )}. 22.1.2

Notions of correlation decay

The above discussion of relaxation times brings up a second key concept: correlation decay. According to an important piece of wisdom in statistical mechanics, physical systems that have only short-range correlations should relax rapidly to their equilibrium distribution. The hand-waving reason is that, if different degrees of freedom (particles, spins, etc.) are independent, then the system relaxes on microscopic time scales (namely the relaxation time of a single particle, spin, etc.). If the degrees of freedom are not independent, but their correlations are short-ranged, they can be coarse-grained in such a way that they become nearly independent. Roughly speaking, this means that one can construct ‘collective’ variables from blocks of original variables. Such conditional variables take |X |B values, where B is the block size, and are nearly independent under the original (Boltzmann) distribution.


¾½

As we are interested in models on non-Euclidean graphs, the definition of correlation decay must be specified. We shall introduce two distinct types of criteria. Although they may look similar at first sight, it turns out that they are not, and each of them will characterize a distinct generic phase. The simplest approach, widely used in physics, consists in considering two-point correlation functions. Averaging them over the two positions defines a susceptibility. For instance, in the case of Ising spins xi ∈ X = {1, −1}, we have already discussed the spin glass susceptibility χSG =

1 (xi xj − xi xj )2 , N

(22.10)

i,j∈V

where · denotes the expectation value with respect to µ. When χSG is bounded as N → ∞, this is an indication of short-range correlations. Through the fluctuation– dissipation theorem (see Section 2.3), this is equivalent to stability with respect to local perturbations. Let us recall the mechanism of this equivalence. Imagine a perturbation of the model (22.16) that acts on a single variable xi . Stability requires that the effect of such a perturbation on the expectation of a global observable j f (xj ) should be bounded. The change in the marginal at node j due to a perturbation at i is proportional to the covariance xi xj − xi xj . As in Section 12.3.2, the average effect of the perturbation xj , j = i, often vanishes (more pre at i on the variables cisely, limN →∞ (1/N ) j∈V xi xj − xi xj = 0) because terms related to different vertices j cancel. The typical effect of the perturbation is captured by the spin glass susceptibility. Generalizing this definition to arbitrary alphabets is easy. We need to use a measure of how much the joint distribution µij ( · , · ) of xi and xj is different from the product of the marginals µi ( · ) times µj ( · ). One such measure is provided by the variation distance ||µij ( · , · ) − µi ( · )µj ( · )|| ≡

1 |µij (xi , xj ) − µi (xi )µj (xj )| . 2 x ,x i

(22.11)

j

We then define the two-point correlation by averaging this distance over the vertices i, j: χ(2) ≡

1 ||µij ( · , · ) − µi ( · )µj ( · )|| . N

(22.12)

i,j∈V

Exercise 22.1 Consider again the case of Ising variables, where X = {+1, −1}. Show that χSG = o(N ) if and only if χ(2) = o(N ). [Hint: Let Cij ≡ xi xj − xi xj . Show that Cij = 2||µij ( · , · ) − µi ( · )µj ( · )||. Then use 2 } and χ(2) = N E{|Cij |}/2, the expectation E being over uniformly random χSG = N E{Cij i, j ∈ V .]

¾¾

An ongoing story

Of course, one can define l-point correlations in an analogous manner: χ(l) ≡

1 N l−1

||µi(1)...i(l) ( · · · ) − µi(1) ( · ) · · · µi(l) ( · )|| .

(22.13)

i(1),...,i(l)∈V

The l-point correlation χ(l) has a useful interpretation in terms of a thought experiment. Suppose you are given an N -dimensional distribution µ(x) and have access to the marginal µi(1) ( · ) at a uniformly random variable node i(1). You want to test how stable this marginal is with respect to small perturbations. Perturbations affect l − 1 randomly chosen variable nodes i(2), . . . , i(l), changing µ(x) into µ (x) ∼ = µ(x)(1 + δ2 (xi(2) )) · · · (1 + δl (xi(l) )). The effect of the resulting perturbation on µi(1) , to first order in the product δ2 · · · δl , is bounded in terms of the expectation by χ(l) (this is again a version of the fluctuation–dissipation theorem). Definition 22.2. (First type of correlation decay) The graphical model given by µ( · ) is said to be stable to small perturbations if, for all finite l, χ(l) /N → 0 as N → ∞. In practice, in sufficiently homogeneous (mean-field) models, this type of stability is equivalent to that found using only l = 2. Let us now introduce another type of criterion for correlation decay. Again we look at a variable node i, but now we want to check how strongly xi is correlated with all of the ‘far apart’ variables. Of course, we must define what ‘far apart’ means. We fix an integer , define B(i, ) as the ball of radius centred on i, and define B(i, ) as its complement, i.e. the subset of variable nodes j such that d(i, j) ≥ . We then want to estimate the correlation between xi and xB(i,) = {xj : j ∈ B(i, )}. This amounts to measuring the distance between the joint distribution µi,B(,) ( · , · ) and the product of the marginals µi ( · )µB(,) ( · ). If we use the total variation distance defined in eqn (22.11), we obtain the following point-to-set correlation function: Gi () ≡ ||µi,B(i,) ( · , · ) − µi ( · )µB(i,) ( · )|| .

(22.14)

The function Gi () can be interpreted according to two distinct but equally suggestive thought experiments. The first one comes from the theory of structural glasses (it is meant to elucidate the kind of long-range correlations arising in a fragile glass). Imagine that we draw a reference configuration x∗ from the distribution µ( · ). Now, we generate a second configuration x as follows. Variables outside the ball, with i ∈ B(i, ), are forced to the reference configuration, xi = x∗i . Variables at a distance smaller than (denoted by xB(i,) ) are instead drawn from the conditional distribution µ(xB(i,) |x∗B(i,) ). If the model µ( · ) has some form of rigidity (long-range correlations), then xi should be close to x∗i . The correlation Gi () measures how much the distributions of xi and x∗i differ. The second experiment is closely related to the first one, but has the flavour of a statistics (or computer science) question. Someone draws the configuration x∗ as above from the distribution µ( · ). They then reveal to you the values of far-apart variables in the reference configuration, i.e. the values of x∗j for all j ∈ B(i, ). They


¾¿

ask you to reconstruct the value of x∗i , or to guess it as well as you can. The correlation function Gi () measures how likely you are to guess correctly (assuming unbounded computational power), compared with the case in which no variable has been revealed to you. This discussion suggests the following definition. Definition 22.3. (Second type of correlation decay) The graphical model µ( · ) is said to satisfy the non-reconstructibility (or extremality) condition if, for all i’s, Gi () → 0 as → ∞. (More precisely, we require that there exists a function δ(), with lim→∞ δ() = 0, such that Gi () ≤ δ() for all i and N .) In the opposite case, i.e. if Gi () remains bounded away from zero at large distance, the model is said to be reconstructible. 22.1.3

Generic scenarios

We shall now describe the correlation decay properties and the pure-state decomposition for the three main phases that we have encountered in the previous chapters: the RS, dynamical 1RSB, and static 1RSB phases. When we are dealing with models on locally tree-like random graphs, each of these phases can also be studied using the appropriate cavity approach, as we shall recall. Here we focus on phases that appear ‘generically’. This means that we exclude (i) critical points, which are obtained by fine-tuning some parameters of the model; and (ii) multiplicities due to global symmetries, for instance in the zero-field ferromagnetic Ising model. Of course, there also exist other types of generic phases, such as the higherorder RSB phases that will be discussed in the next section, and maybe more that have not been explored yet. • Replica-symmetric. In this phase, there exists no non-trivial decomposition into pure states of the form (22.7). In other words, NN = 1 with high probability. Correlations decay according to both criteria: the model is stable to small perturbations and it satisfies the non-reconstructibility condition. Therefore it is short-range correlated in the strongest sense. Finally, the replica-symmetric cavity method of Chapter 14 yields asymptotically exact predictions. • Dynamical 1RSB. In this phase, the measure µ( · ) admits a non-trivial decomposition of the form (22.7) into an exponential number of pure states: NN = eN Σ+o(N ) with high probability for some Σ > 0. Furthermore, most of the measure is carried by states of equal size. More precisely, for any δ > 0, all but an exponentially small fraction of the measure is contained in states Ωr,N such that −Σ − δ ≤

1 log µ(Ωr,N ) ≤ −Σ + δ . N

(22.15)

From the correlation point of view, this phase is stable to small perturbations, but it is reconstructible. In other words, a finite number of probes would fail to reveal long-range correlations. But long-range correlations of the point-to-set type are in fact present, and they are revealed, for instance, by a slowdown of reversible Markov dynamics. The glass order parameter, namely the overlap distribution P (q), is trivial in this phase (as implied by eqn (12.31)), but its glassy nature

An ongoing story

UNSAT

αd (K)

αc (K)

αs (K)

α

Fig. 22.1 A pictorial view of the different phases in K-SAT with K ≥ 4, depending on the number of clauses per variable α. From left to right: replica-symmetric, dynamical 1RSB, static 1RSB, and UNSAT.

can be found through the -coupling method of Section 12.3.4. The model can be solved exactly (in the sense of determining its asymptotic free-energy density) within the 1RSB cavity method. The thermodynamically dominant states, i.e. those satisfying eqn (22.15), correspond to the 1RSB parameter x = 1. • Static 1RSB. This is the ‘genuine’ 1RSB phase analogous to the low-temperature phase of the random energy model. The model admits a non-trivial pure-state decomposition with wildly varying weights. For any δ > 1, a fraction 1 − δ of the measure is contained in the k(N, δ) pure states with the largest weights. The number k(N, δ) converges, when N → ∞, to a finite random variable (taking integer values). If we order the weights according to their magnitudes w(1) ≥ w(2) ≥ w(3) ≥ · · · , they converge to a Poisson–Dirichlet process (see Chapter 8). This phase is not stable to small perturbations, and it is reconstructible: it has long-range correlations according to both criteria. The asymptotic overlap distribution function P (q) has two delta-function peaks, as in Fig. 12.3. Again, it can be solved exactly within the 1RSB cavity method. These three phases are present in a variety of models, and are often separated by phase transitions. The ‘clustering’ or ‘dynamical’ phase transition separates the RS and dynamical 1RSB phases, and a condensation phase transition separates the dynamical 1RSB from the static 1RSB phase. Figure 22.1 describes the organization of the various phases in random K-SAT with K ≥ 4, as discussed in Section 20.3. For α < αd (K), the model is RS; for αd (K) < α < αc (K), it is dynamically 1RSB; for αc (K) < α < αs (K), it is statically 1RSB; and for αs (K) < α, it is UNSAT. Figure 22.2 shows the point-toset correlation function in random 4-SAT. It clearly develops long-range correlations at α ≥ αd ≈ 9.38. Notice the peculiar development of correlations through a plateau whose width increases with α, and diverges at αd . This is typical of the dynamical 1RSB transition.

22.2

Higher levels of replica symmetry breaking

For some of the models studied in this book, the RS or the 1RSB cavity method is thought to yield asymptotically exact predictions. However, in general, higher orders of RSB are necessary. We shall sketch how to construct these higher-order solutions


0.6

Gi () 0.4

0.2

0 0

30

60

90

120

150

Fig. 22.2 The point-to-set correlation function defined in eqn (22.14), plotted versus distance for random 4-satisfiability, at clause densities α = 9.30, 9.33, 9.35, and 9.40 (from bottom to top).

hierarchically in locally tree-like graphical models. In particular, understanding the structure of the two-step replica-symmetry-breaking (2RSB) solution allows us to derive a ‘stability criterion’ for the 1RSB approach. It is on the basis of this criterion that, for instance, our derivation of the SAT–UNSAT threshold in Chapter 20 is conjectured to give an exact result. 22.2.1

The high-level picture

Let us first briefly summarize the RS/1RSB approach. Consider an ensemble of graphical models defined through the distribution (22.1) with a locally tree-like factor graph structure. Within the RS cavity method, the local marginals of µ( · ) are accurately νa→i }. Given a small (tree-like) described in terms of the message sets {νi→a } and {

subgraph induced by the vertex set U ⊂ V , the effect of the rest of the graph G \ GU on U is described by a factorized measure on the boundary of U . One-step replica symmetry breaking relaxes this assumption, by allowing longrange correlations, with a peculiar structure. Namely, the probability distribution µ( · ) is assumed to decompose into a convex combination of Bethe measures µr ( · ). Within each ‘state’ r, the local marginals of the measure restricted to this state are well r } (by ‘well described’, we mean that the described in terms of a set of messages {νi→a description becomes asymptotically exact at large N ). Sampling a state r at random defines a probability distribution P({ν}, {

ν }) over messages. This distribution is then found to be described by an ‘auxiliary’ graphical model, which is easily deduced from the original one. In particular, the auxiliary factor graph inherits the structure of the original one, and therefore it is again locally tree-like. 1RSB amounts to using the RS cavity method to study this auxiliary graphical model over messages. In some cases, 1RSB is expected to be asymptotically exact in the thermodynamic limit. However, this is not always the case: it may fail because the measure P({ν}, {

ν }) decomposes into multiple pure states. Higher-order RSB is used to study this type of situation by iterating the above construction.

An ongoing story

d2

d1 d0

Fig. 22.3 Cartoon of the distribution µ(x) for a model described by two-step replica symmetry breaking. The probability mass is concentrated in the grey ‘lumps’ of radius d2 , which are organized into ‘clouds’ of radius d1 > d2 . The dashed circle corresponds to the typical distance d0 between clouds.

More precisely, the 2RSB method starts from the ‘auxiliary’ distribution P({ν}, {

ν }). Instead of studying this with the RS method as we havedone so far, we use instead the 1RSB method to study P({ν}, {

ν }) (therefore introducing an auxiliary auxiliary model, which is studied by the RS method). The 2RSB Ansatz admits a hand-waving interpretation in terms of the qualitative features of the original model µ( · ). Consider 1RSB again. The interpretation was that µ( · ) is a convex combination of ‘pure states’ µr ( · ), each forming a well-separated lump in configuration space. Within 2RSB, the lumps have a hierarchical organization, i.e. they are grouped into ‘clouds’. Each lump is addressed by giving a ‘cloud index’ r1 , and, within the cloud, a ‘lump index’ r2 . The measure thus decomposes as µ(x) =

wr1 ,r2 µr1 ,r2 (x) .

(22.16)

r1 ∈S1 , r2 ∈S2 (r1 )

Here S2 (r1 ) is the set of indices of the lumps inside cloud r1 . A pictorial sketch of this interpretation is shown in Fig. 22.3. Even the most forgiving reader should be puzzled by all this. For instance, what is the difference between N1 clouds, each involving N1 lumps, and just N1 N2 lumps? In order to distinguish between these two cases, one can look at a properly defined distance, say the Hamming distance divided by N , between two i.i.d. configurations drawn with distribution µ( · ) (in physics jargon, two replicas). If the two configurations are conditioned to belong to the same lump, to different lumps within the same cloud, or to different clouds, the normalized distances concentrate around three values d2 , d1 , d0 , respectively, with d2 < d1 < d0 . As in the case of 1RSB, one could in principle distinguish dynamic and static 2RSB phases depending on the numbers of relevant clouds and lumps within clouds. For instance, in the most studied case of static 2RSB, these numbers are subexponential. As a consequence, the asymptotic distribution of the distance between two replicas has non-zero weight on each of the three values d0 ,


r1 r2 r3 wr Fig. 22.4 Hierarchical structure of the distribution µ(x) within k-step replica symmetry breaking. Here k = 3.

d1 , d2 (in other words, the overlap distribution P (q) is a combination of three delta functions). Of course, this whole construction can be bootstrapped further, by having clouds grouped into larger structures, etc. Within k-RSB, the probability distribution µ( · ) is a convex combination of ‘states’ µr ( · ), where r = (r1 , r2 , . . . , rk ) indexes the leaves of a k-generation tree. The indices r1 , r2 , . . . , rk correspond to the nodes encountered along the path between the root and the leaf. This translates into a hierarchy of auxiliary graphical models. If k is allowed to be arbitrarily large, this hierarchy is expected to determine the asymptotic properties of a large class of models. In particular, one can use it to compute the free entropy per variable φ ≡ limN →∞ N −1 log ZN . The resulting description of µ(x) has a natural ultrametric structure, as discussed in Chapter 8 and recalled in Fig. 22.4. This structure is captured by the generalized random energy model (GREM), a simple model that generalizes the REM discussed in Chapter 5. While presenting the solution of the GREM would take us too far, it is instructive to give its definition. Example 22.4 The GREM is a simple model for the probability distribution µ( · ) within k-step RSB. Its definition involves one parameter N ∈ N that corresponds to the system size, and several others (denoted by {a0 , a1 , . . . , ak−1 }, {d0 , d2 , . . . , dk−1 }, and {Σ0 , Σ1 , . . . , Σk−1 }) that are thought of as being fixed as N → ∞. States are associated with the leaves of a k-generation tree. Each leaf is indexed by the path r = (r0 , . . . , rk−1 ) that connects it to the root (see Fig. 22.4). The GREM does not describe the structure of each state µr ( · ) (which can be thought of as supported on a single configuration). It describes only the distribution of distances between the states, and the distribution of the weights wr appearing in the decomposition (22.16). A node at level i has exp{N Σi } offspring. The total number of states is therefore exp{N (Σ0 + · · · + Σk−1 )}. Two random configurations drawn from states r and s have a distance di(r,s) , where i(r, s) is the largest integer i such that ri = si . Finally, the weight of state r has the form

An ongoing story

wr =

1 exp{−β(Er(0) + · · · + Er(k−1) )} , 0 k−1 Z

(22.17)

(i)

where the Er are independent normal random variables with mean 0 and variance N ai . The interested reader is invited to derive the thermodynamic properties of the GREM, for instance the free energy as a function of the temperature. 22.2.2

What does 2RSB look like?

Higher-order RSB has been studied in some detail in many ‘fully connected’ models such as the p-spin Ising model considered in Chapter 8. In contrast, if one considers models on sparse graphs as we do here, any cavity calculation beyond 1RSB is technically very challenging. In order to understand why, it is interesting to have a superficial look at how a 2RSB cavity calculation would be formally set up, without any attempt at justifying it. For the sake of simplicity, we shall consider a model of the form (22.1) with pairwise interactions. Therefore all of the factor nodes have degree 2, and BP algorithms can be simplified by using only one type of message passed along the edges of an ordinary graph (see Section 14.2.5). Consider a variable node 0 ∈ V of degree (l + 1), and denote l of its neighbours by {1, . . . , l}. We let ν1 , . . . , νl be the messages from 1, . . . , l, respectively, and let ν0 be the message from 0 to its (l + 1)-th neighbour. As we saw in Section 14.2.5, the RS cavity equation (i.e. the BP fixed-point equation) at node 0 reads ν0 (x0 ) =

k 1 ψ0i (x0 , xi )νi (xi ) , z{νi } i=1 x

(22.18)

i

where z{νi } is determined by the normalization condition of ν0 ( · ). In order to lighten the notation, it is convenient to introduce a function f0 that, when evaluated on l messages ν1 , . . . , νl , returns the message ν0 as above. We shall therefore write eqn (22.18) in shorthand form as ν0 = f0 {νi }. Each νi is a point in a (|X |−1)-dimensional simplex. The 1RSB cavity equations are obtained from eqn (22.18) by promoting the messages νi to random variables with distribution Qi ( · ) (see Chapter 19). The equations depend on the 1RSB parameter (a real number), which we denote here by x1 . Adopting a continuous notation for the distributions of messages, we get Q0 (ν0 ) =

1 Z{Qi }

z{νi }x1 δ(ν0 − f0 {νi })

l

dQi (νi ) ,

(22.19)

i=1

Analogously to the replica-symmetric case (eqn (22.18)), we shall write Q0 = F0 {Qi } as a shorthand for this equation. The function F0 takes as its argument l distributions Q1 , . . . , Ql and evaluates a new distribution Q0 (each of the Qi ’s is a distribution over a (|X | − 1)-dimensional simplex). At this point, the formal similarity of eqns (22.18) and (22.19) should be clear. The 2RSB cavity equations are obtained by promoting the distributions Qi to random


variables (taking values in the set of distributions over a |X |-dimensional simplex).1 Their probability distributions are denoted by Qi , and the resulting equations depend on one further real parameter x2 . Formally, the 2RSB equation can be written as Q0 (Q0 ) =

1 Z{Qi }

Z{Qi }x2 /x1 δ(Q0 − F0 {Qi })

l

dQi (Qi ) .

(22.20)

i=1

This equation might look scary, as the Qi ( · ) are distributions over distributions over a compact subset of the reals. It is useful to rewrite it in a mathematically more correct form. This is done by requiring, for any measurable set of distributions A (see the footnote), the following equality to hold: Q0 (A) =

1 Z{Qi }

Z{Qi }x2 /x1 I(F0 {Qi } ∈ A)

l

dQi (Qi ) .

(22.21)

i=1

The interpretation of the 2RSB messages Qi is obtained by analogy with the 1RSB 1 case. Let α1 be the index of a particular cloud of states, and let Qα i ( · ) be the distribution of the message νi over the lumps in cloud α1 . Then Qi is the distribution of 1 Qα i when one picks out a cloud index α1 randomly (each cloud being sampled with a weight that depends on x1 .) In principle, eqn (22.20) can be studied numerically by generalizing the population dynamics approach of Chapter 19. In the present case, one can think of two implementations: for one given instance, one can generalize the SP algorithm, but this generalization involves, on each directed edge of the factor graph, a population of populations. If, instead, one wants to perform a statistical analysis of these messages, seeking a fixed point of the corresponding density evolution, one must use a population of populations of populations! This is obviously challenging from the point of view of computer resources (both memory and time). To the best of our knowledge, it has been tried only once, in order to compute the ground state energy of the spin glass on random 5-regular graphs. Because the graph is regular, it looks identical at any finite distance from any given point. One can therefore seek a solution such that the Qi on all edges are the same, and one is back to the study of populations of populations. The results have been summarized in Table 17.1: if one looks at the ground state energy, the 2RSB method provides a small correction of order 10−4 to the 1RSB value, and this correction seems to be in agreement with the numerical estimates of the ground state. 22.2.3

Local stability of the 1RSB phase

The above discussion of 2RSB will help us to check the stability of the 1RSB phase. The starting point consists in understanding the various ways in which the 2RSB formalism can reduce to the 1RSB one. 1 The mathematically inclined reader might be curious about the precise definition of a probability distribution over the space of distributions. It turns out that, given a measure space Ω (in our case a (|X | − 1)-dimensional simplex), the set of distributions over Ω can be given a measurable structure that makes 2RSB equations well defined. This is done by using the smallest σ-field under which the mapping Q → Q(A) is measurable for any A ⊆ Ω that is measurable.

¿¼

An ongoing story

The first obvious reduction consists in taking the 2RSB distribution Qi to be a Dirac delta at Q∗i . In other words, for any continuous functional F on the space of distributions, (22.22) F(Qi ) dQi (Qi ) = F(Q∗i ) . It is not hard to check that, if {Q∗i } solves the 1RSB equation (22.19), this choice of {Qi } solves eqn (22.20) independently of x2 . There exists, however, a second reduction, which corresponds to taking Qi ( · ) to be a non-trivial distribution, but supported on Dirac deltas. Let us denote by δν ∗ a 1RSB distribution which is a Dirac delta on the message ν = ν ∗ . Given a set of messages {Q∗i } that solves the 1RSB equation (22.19), we construct Qi ( · ) as a superposition of Dirac deltas over all values of ν ∗ , each one appearing with a weight Q∗i (ν ∗ ). Again, this distribution is defined more precisely by its action on a continuous functional F(Q): F(Qi ) dQi (Qi ) = F(δν ∗ ) dQ∗i (ν ∗ ) . (22.23)

Exercise 22.2 Suppose that {Q∗i } solves the analogue of the 1RSB equation (22.19) in which the parameter x1 has been changed to x2 . Show that the Qi defined by eqn (22.23) solves eqn (22.20) independently of x1 . [Hint: Show that, when evaluated on Dirac deltas, the normalization Z appearing in eqn (22.19) is related to the normalization z in eqn (22.18) by Z{δνi } = (z{νi })x1 .]

In view of the interpretation of the 2RSB messages Qi outlined in the previous subsection, and illustrated schematically in Fig. 22.3, these two reductions correspond to qualitatively different limiting situations. In the first case, described by eqn (22.22), the distribution over clouds becomes degenerate: there is essentially one cloud (by this we mean that the number of clouds is not exponentially large in N : the corresponding complexity vanishes). In the second case, described by eqn (22.23), it is the distribution within each cloud that trivializes: there is only one cluster (in the same sense as above) in each cloud. What are the implications of these remarks? Within the 1RSB approach, one needs to solve eqn (22.19) in the space of distributions over BP messages: let us call this the ‘1RSB space’. When passing to 2RSB, one seeks a solution of eqn (22.20) within a larger ‘2RSB space’, namely the space of distributions over distributions over BP messages. Equations (22.22) and (22.23) provide two ways of embedding the 1RSB space in the 2RSB space. When one finds a 1RSB solution, one should naturally ask whether there exists a proper 2RSB solution as well (i.e. a solution outside the 1RSB subspace). If this is not the case, physicists usually conjecture that the 1RSB solution is asymptotically correct (for instance, it yields the correct free energy per spin). This check has been carried out for models on complete graphs (e.g. the fully connected p-spin glasses). So far, the difficulties of studying the 2RSB equations have prevented its implementation for sparse factor graphs.


¿½

Fig. 22.5 Cartoon of the two types of local instabilities from a 1RSB solution towards 2RSB.

Luckily there is a convenient (albeit less ambitious) alternative: check the local stability of 1RSB solutions with respect to higher-order RSB. Given a 1RSB solution, we look at it as a point in the 2RSB space according to the two possible embeddings, and we study the effect of a small perturbation. More precisely, consider the iteration of the 2RSB equations (22.20):

(t+1) Qi→j (Q0 )

1 = Z{Ql→i }

Z{Ql→i }r δ(Qi→j − Fi {Ql→i })

(t)

dQl→i (Ql→i ) .

l∈∂i\j

Given the factor graph G, we initiate this iteration from a point close to the 1RSB solution described by either of the embeddings (22.22) or (22.23) and see if the iteration converges back to the 1RSB fixed point. This is studied by linearizing the iteration in terms of an appropriate ‘perturbation’ parameter. If the iteration does not converge to the 1RSB fixed point, the 1RSB solution is said to be unstable. The instability is said to be of ‘type I’ if it occurs when the embedding (22.22) is used, and of ‘type II’ if it occurs for the embedding (22.23). An alternative approach to checking the local stability of a 1RSB solution consists in computing the spin glass susceptibility, which describes the reaction of the model (22.16) to a perturbation that acts on a single variable xi . As we discussed above, the effect of this perturbation (studied to linear order) remains finite when the spin glass susceptibility χ(2) is finite. One should therefore compute χ(2) assuming that the 1RSB solution is correct, and check that it is finite. However, the 1RSB picture implies a second condition: each single lump r should also be stable to small perturbations. More precisely, we define χSG,r as the spin glass susceptibility with respect to the measure µr ( · ) restricted to the state r. If we denote by · r the expectation value with respect to µr , the ‘intra-state’ susceptibility χSG,intra is a weighted average of χSG,r over the states:

¿¾

An ongoing story

χSG,intra =

wr χSG,r ,

(22.24)

r

χSG,r =

2 1 xi xj r − xi r xj r . N i,j

(22.25)

Within the susceptibility approach, the second condition consists in computing χSG,intra with the 1RSB approach and requiring that it stays finite as N → ∞. It is generally believed that these two approaches to the local stability of the 1RSB phase coincide. Type I stability should be equivalent to χ(2) being finite; it means that the system is stable with respect to the grouping of states into clusters. Type II stability should be equivalent to χSG,intra being finite; it means that the system is stable towards a splitting of the states into substates. A pictorial representation of the nature of the two instabilities in the spirit of Fig. 22.3 is shown in Fig. 22.5. The two approaches to stability computations have been developed in several special cases, and are conjectured to coincide in general. Remarkably, 1RSB is unstable in several interesting cases, and higher-order RSB would be needed to obtain exact predictions. Stability computations are somewhat involved, and a detailed description is beyond our scope. Nevertheless, we would like to give an example of the results that can be obtained through a local stability analysis. Consider random K-SAT formulae, with N variables and M = N α clauses. Let es (α) denote the minimum number of unsatisfied clauses per variable, in the large-system limit. The limit es (α) can be computed along the lines of Chapter 20 using the 1RSB cavity method: for a given α, one computes the energetic complexity density Σe (e) versus the density of violated clauses e. Then es (α) is found as the minimal value of u such that Σe (e) > 0. It vanishes for α < αs (K) (the SAT–UNSAT threshold) and, for α > αs (K), departs continuously from 0, increasing monotonically. The stability computation shows that, for a given α, there is in general an instability of type II which appears above some value e = eG (α): only the part of Σe (e) with e ≤ eG (α) is in a locally stable 1RSB phase. When α < αm (K), eG (α) = 0 and the whole 1RSB computation is unstable. For α > αG (K), eG (α) < es (α) (the ground state energy density), and again 1RSB is unstable (this implies that the 1RSB prediction for es (α) is not correct). The conclusion is that the 1RSB calculation is stable only in an interval ]αm (K), αG (K)[. Figure 22.6 summarizes this discussion for 3-SAT. For all values of K, the stable interval ]αm (K), αG (K)[ contains the SAT–UNSAT threshold αs (K). The stability check leads to the conjecture that the 1RSB prediction for αs (K) is exact. Let us stress, however, that stability has been checked only with respect to small perturbations. A much stronger argument would be obtained if one could do the 2RSB computation and show that it has no solution apart from the two ‘embedded 1RSB solutions’ that we discussed above. 22.2.4

Open problems within the cavity method

The main open problem is, of course, to prove that the 1RSB cavity approach yields correct predictions for some models. This has been achieved so far only for a class of


¿¿

Σe 0.02

0.01

0.002

0.004

e

0.014 0.012 0.010

e d (α) e G (α) e s (α)

energy

0.008

1RSB stable region

0.006 0.004 0.002 0

αd −0.002

3.9

αm 4

4.1

4.2

αs

α

4.3

αG 4.4

4.5

4.6

Fig. 22.6 Top: the energetic complexity Σe in a random 3-SAT problem, computed within the 1RSB cavity method, plotted versus the density e of violated clauses, for α = 4.1, 4.2, and 4.3 (from top to bottom). The curves reproduce Fig. 20.5, but now the stable and unstable regions are shown. The thick lines, below eG (α), give the part of the complexity curve for which the 1RSB computation is locally stable (absent for α = 4.1 < αm (3), where the whole curve is unstable). This is the only part that is computed reliably by 1RSB; the parts shown by dashed lines are unstable. Bottom: for the same random 3-SAT problem, plotted versus the clause density α, the continuous line gives the minimum density of unsatisfied clauses as predicted within 1RSB (this is the value of e where Σe (e) starts to become positive). The dotted line gives the threshold energy density as predicted within 1RSB (the maximal value of e where Σe (e) exists). The grey area indicates the region of local stability of the 1RSB solution. The ground state energy density predicted by 1RSB is wrong for α > αG (although probably very close to the actual value), because in this region there is an instability towards higher-order RSB. It is conjectured that the stable region, αm < α < αs , is in a 1RSB phase: if this conjecture holds, the 1RSB prediction of αs for the SAT–UNSAT threshold is correct. For K = 3, one has αm (3) = 4.153(1), αs (3) = 4.2667(1), and αG (3) = 4.390(5).

An ongoing story

models on the complete graph. Here we want to point out a number of open questions that wait for an answer, even at a heuristic level, within the 1RSB cavity method itself. Distributional equations. Cavity predictions are expressed in terms of fixed points of equations of the form (22.19). When we are considering models on ensembles of random graphs, this can be read as an equation for the probability distribution of Q0 ( · ) (which is taken to be identical to those of Q1 ( · ), . . . , Qk ( · ).) Currently such equations are mostly studied using the population dynamics method of Section 14.6.4. The main alternative explored so far has been to formally expand the equations for large degrees. Population dynamics is powerful and versatile. However, in many cases, this approach is too coarse, particularly as soon as one wants to study k-RSB with k ≥ 2. It is intrinsically hampered by statistical errors that are of the order of the inverse square root of population size. In some models (for instance, for graph ensembles with large but bounded average degree), statistical fluctuations are too large for the population sizes that can be implemented on ordinary PCs (typically 107 –108 elements). This limits the possibility of distinguishing, for instance, 2RSB from 1RSB effects, because high precision is generally required to see the difference. Furthermore, metastability is the crux (and the limit) of the whole population dynamics approach. Therefore it would be interesting to make progress in two directions: • Analytical tools and generic results for the cavity equations; this could provide important guiding principles for any numerical study. • New efficient and stable numerical methods. A step forward has been made with the reconstruction algorithm discussed in Theorem 19.5, but unfortunately it is limited to one value of the rescaling parameter, x = 1. Local stability. Local stability criteria provide important guidance in heuristic studies. It is important to put these results on firmer foundations. Two specific tasks could be, for instance: • Prove that if all 1RSB solutions of the cavity equations are locally unstable, then there must exist a 2RSB solution outside the 1RSB subspace. • Prove that if a solution of the cavity equations is locally unstable, it does not describe the model correctly. Occurrence of k-RSB. A number of random graphical models have been studied within the cavity (or replica) method. In most cases, it has been found that the system is either RS, 1RSB, or FRSB. The cases in which a 2RSB phase is found are rare, and they always involve some kind of special construction of the compatibility function (for instance, a fully connected model which is a superposition of two p-spin glass interactions, with p1 = 3 and p2 = 16, displays 2RSB). Therefore one should: • Find a ‘natural’ model for which 2RSB is asymptotically exact, or understand why this is impossible.

Phase structure and the behaviour of algorithms

Full replica symmetry breaking. We saw that k-RSB provides, as k increases, a sequence of ‘nested’ schemes aimed at computing various quantities such as local marginals and the free-entropy density in the large-system limit. A k-th order scheme includes all of the lower l-RSB schemes with l < k as nested subspaces of the set of feasible solutions to the cavity equations. On the other hand, as the number of steps increases, the description of the set of feasible solutions becomes more and more complicated (distributions of distributions of . . . ). Surprisingly, in the case of fully connected models, there exists a compact description of the space of feasible solutions in the FRSB limit k → ∞. An outstanding problem is to find an analogous description in the case of models on sparse graphs. This would allow us to look for the best solution in the k-RSB space for all k. Therefore we need to: • Find a description of the space of full replica-symmetry-breaking messages for models on sparse graphs. Variational aspects. It is widely believed that if one finds a consistent solution of the cavity k-RSB equations, the free-energy density computed with this solution is always a lower bound on the correct free-energy density of the model (in particular, the k-RSB ground state energy density prediction is a lower bound on the true value). This should hold for a large class of models.While this has been proven in some specific cases, one would like to: • Find a general proof that the free energy computed with the cavity method is a lower bound on the correct free energy of the model.

22.3


A good part of this book has been devoted to the connection between the various phases in random graphical models and the behaviour of algorithms. There exists by now substantial evidence (empirical, heuristic, and in some cases rigorous) that such a connection exists. For instance, we have seen with the example of codes in Chapter 21 how the appearance of a 1RSB phase, and the corresponding proliferation of metastable states, determines the noise threshold where BP decoding fails. Developing a broader understanding of this connection, and determining the class of algorithms to which it applies, is a very important problem. We propose here a list of broad research problems, whose advancement will probably help to clarify this issue. We always have in mind a graphical model of the form (22.1), with a locally tree-like factor graph. Impact of the dynamical transition on Monte Carlo dynamics. Consider the problem of sampling from the distribution (22.1) using a Monte Carlo Markov chain (MCMC) algorithm. The Markov chain is assumed to flip a sublinear (o(N )) number of variables at each step, and to satisfy detailed balance with respect to the probability distribution µ( · ). One expects that, if the system is in a 1RSB phase, the relaxation time of this algorithm will increase rapidly (probably exponentially) with the system size. Intuitive

An ongoing story

arguments in favour of this statement can be obtained from each of the two characterizations of the 1RSB phases introduced in Section 22.1. The argument is different depending on whether we start from the pure-state decomposition or from the characterization in terms of correlations. In the first case, the relaxation time is estimated through the time to cross a bottleneck (see also Chapter 13). In the second case, one can define a correlation length ∗i through the point-to-set correlation function Gi () (see eqn (22.14)). In order for the system to relax, information has to travel a distance ∗i . But if ∗i diverges with the size, so must the relaxation time. This picture is intuitively satisfying, but it is far from being proved, and needs to be formulated more precisely. For instance, it often happens that in RS phases there exist small isolated metastable states that make the relaxation time (the inverse spectral gap of the MCMC) formally large. But even in such cases, numerical simulations indicate that Glauber dynamics equilibrates rapidly within the RS phase. This observation is probably related to the fact that the initial condition is chosen uniformly at random, and that equilibration is only checked for on local observables. A number of questions arise: • Why is metastability irrelevant ‘in practice’ in an RS phase? Is it because of local measurements? Or because of the uniform initial condition? If the latter is true, what is so special about the uniform initial condition? • Within an RS phase, can one approximate partition functions efficiently? Message passing and the estimation of marginals. For a number of models on sparse random graphs within the RS and (sometimes) dynamical 1RSB phases, messagepassing methods such as belief propagation and survey propagation show good performance empirically. More precisely, they return good approximations of local expectation values if initialized from uniform messages. Current rigorous techniques for analysing BP often aim at proving that it is accurate regardless of the initialization. As a consequence, results are dominated by the behaviour under worst-case initializations that are not used in practice. As an illustration, consider applying BP to the uniform measure over solutions of a random K-SAT formula. An analysis under worst-case initialization allows one to prove that BP is accurate only for α ≤ (2 log K)/K[1 + o(1)]. This threshold is embarrassingly small when compared with the dynamical transition point that terminates the RS phase at αd (K) = 2K log K/K[1 + o(1)]. In general, we have no good mathematical control of when BP or SP converges and/or gives good approximations of marginals. Empirically, it seems that SP is able to converge in some regions of 1RSB phases where BP does not. We have no real understanding of this fact beyond the hand-waving argument that 1RSB correctly captures the structure of correlations in these phases. Here are a number of open questions about these issues: • Why is the performance of BP and SP on random instances, with uniformly random initialization, much better than in the worst case? What is special about the uniform initialization? What are the features of random instances that make them easier? Can these features be characterized and checked efficiently?


• Under what conditions do BP (and SP) algorithms converge and give good approximations to local marginals? When their naive iteration does not converge, can one systematically either force convergence or use time averages of the messages? • It seems that, on sparse random graphical models, BP and SP outperform local MCMC algorithms. In particular, these message-passing algorithms can have (at least in principle), good performance within the dynamical 1RSB phase. Can one demonstrate this possibility convincingly for some model? Message-passing algorithms and optimization. If one is seeking a solution to a random constraint satisfaction problem using message passing, the main approach so far has been the use of decimation: one first computes all local marginals, then decides, based on this knowledge, how to fix a variable, and then iterates the procedure. In general, this procedure converges when the number of constraints per variable is not too large, but it fails above a critical value of this number, which is strictly smaller than the SAT–UNSAT threshold. No one knows how to determine this threshold analytically. An alternative to decimation is the reinforcement method: instead of fixing a variable based on a knowledge of local marginals, it modifies some local factors applying to each of the individual variables, based on this same information. So far, optimizing this modification is an art, and its critical threshold cannot be estimated either. Some of the questions that arise are the following: • How can we predict the performance of BP+decimation or SP+decimation? For instance, empirically these methods find solutions to random K-SAT formulae with high probability for α < αBP (K) (or α < αSP (K)), but we have no prediction for these algorithmic thresholds. In what class of problems is SP better than BP? • Similar questions for BP+reinforcement and SP+reinforcement. • We need to find new ways to use the information about local marginals found by message passing in order to exhibit solutions. • In an UNSAT phase, the message-passing procedure is able to give an estimate of the minimal number of violated constraints. Is it possible to use this information, and that contained in the messages, in order to prove unsatisfiability for one given instance? The above questions focus on sparse random instances. Message-passing techniques have been (partially) understood and sharpened for this type of instance. They naturally arise in a large class of applications where the graphical model is random or pseudorandom by design. The theory of sparse-graph codes is a clear example in this direction. In the limit of large block lengths, random constructions have proved to be generally superior to deterministic ones. More recently, sparse-graph constructions have been proposed for data compression (both lossless and lossy), online network measurements, multiterminal communications, distributed storage, group testing, etc. On the other hand, being able to deal with structured graphs would open up an even broader class of applications. When applied to structured problems, message-passing algorithms often fail to converge. This is typically the reason why the decimation method may fail, even when the marginals of the original problem are well estimated by message passing: the instance found after fixing many variables is no longer random.

An ongoing story

Finding appropriate modifications of message passing for structured graphs would therefore be very interesting. The question that arises, therefore, is: • How can we use message passing in order to improve the solution of some general classes of (non-random) constraint satisfaction problems? Can it be coupled efficiently to other general methods (such as MCMC)?

Notes The present chapter has inevitably been elliptic. We shall provide a few pointers to recent research here, without any ambition to be comprehensive. The connection between correlation lengths and phase transitions is a classical topic in statistical mechanics which has recently been revived by the interest in the glass transition. A good starting point for learning about this subject in the context of glasses is the paper by Bouchaud and Biroli (2004), which describes the point-toset correlation function using a thought-experiment where one freezes the variables outside a ball (see Section 22.1.2). The description of point-to-set correlations in terms of ‘reconstruction’ problems is taken from Evans et al. (2000). This paper studied the reconstruction phase transition for Ising models on trees. Results for a wide class of models on trees were surveyed by Mossel and Peres (2003) and Mossel (2004). We also refer to Gerschenfeld and Montanari (2007) for the generalization to non-tree graphs. The connection between ‘reconstruction’ and the ‘dynamical’ 1RSB phase transition was first pointed out by Mézard and Montanari (2006). The implications of this phase transition for dynamics were explored by Berger et al. (2005), Martinelli et al. (2004), and Montanari and Semerjian (2006b). The definition of pure states presented in this chapter and the location of the dynamical and condensation phase transitions for random K-SAT and the colouring of random graphs are from Krzakala et al. (2007). The GREM was introduced by Derrida (1985) and studied in detail by Derrida and Gardner (1986). A 2RSB phase in fully connected models has been found by Crisanti and Leuzzi (2007). There are very few results about higher-order RSB in models on sparse random graphs. For spin glasses, one can use perturbative expansions close to the critical point (Viana and Bray, 1985) or for large degrees (Goldschmidt and De Dominicis, 1990). The 2RSB computation of the ground-state energy for spin glasses mentioned in Section 22.2 is from Montanari (2003). The method for verifying the local stability of the 1RSB solution for sparse systems was first devised by Montanari and Ricci-Tersenghi (2003), and applied to random satisfiability problems by Montanari et al. (2004). A complete list of stability thresholds, including their asymptotic behaviour, for random K-SAT can be found in Mertens et al. (2006). The interpretation of 1RSB instability in terms of susceptibilities was discussed by Rivoire et al. (2003). The fact that the free energy computed with the cavity (or replica) method is a lower bound on the true value can be proven in some fully connected models using the inequalities of Guerra (2003). The same strategy also yields rigorous bounds in some diluted systems (Franz and Leone, 2003; Franz et al., 2003; Panchenko and Talagrand, 2004) but it still relies on some details of the structure of the models, and a general proof applicable to all cases is lacking.

Notes

The reinforcement algorithm was introduced and discussed for SAT by Chavas et al. (2005). There exist only scarce results on the algorithmic consequences of the structure of the solution space. Some recent analyses can be found in Altarelli et al. (2007), Montanari et al. (2007), Ardelius and Aurell (2006) and Alava et al. (2007). The convergence and correctness of BP for random K-satisfiability at small enough α was proven by Montanari and Shah (2007). This book has covered only a small subset of the problems that lie at the intersection between information theory, computer science and statistical physics. It would be difficult to provide an exhaustive list of references on the topics that we did not touch: we shall instead limit ourselves to a few ‘access points’. The vertex-covering problem has been studied in detail by the replica and the cavity method (Weigt and Hartmann, 2000; Zhou, 2003; Hartmann and Weigt, 2005). As we mentioned, channel coding is only one of the fundamental problems addressed by information theory. Data compression, in particular in its ‘lossy’ version, is a key component in many modern technologies, and presents a number of open problems (Ciliberti et al., 2005; Wainwright and Maneva, 2005). Some other statistical problems such as group testing are similar in spirit to data compression (Mézard et al., 2007). Modern wireless and wireline communication systems are intrisically multi-user systems. Finding optimal coding schemes in a multi-user context is a very open subject of great practical interest. Even the information-theoretic capacity of such systems is unknown. Two fields that have benefited from tools or from analogies with statistical mechanics are multi-user detection (Tanaka, 2002; Guo and Verd´ u, 2002) and networking (Kelly, 1991). Also within a communications context, a large effort has been devoted to characterizing large communication networks such as the Internet. A useful review was provided by Kleinberg et al. (1999). Statistical mechanics concepts have been applied to the analysis of fluctuations in financial markets (Bouchaud and Potters, 2003) and for modelling interactions among economic agents (Challet et al., 2005). Finally, biology presents a number of problems in which randomness, interaction between different components, and robustness play important roles. Stochastic models on networks, and inference algorithms have been studied in a number of contexts, from neural networks (Baldassi et al., 2007; Coolen et al., 2000) to phylogeny (Mossel, 2003) and gene expression (Friedman et al., 2000). A few of these topics, and others, are reviewed in recent summer school proceedings (Bouchaud et al., 2007).


Appendix A Symbols and notation In this appendix, we summarize the conventions that we have adopted throughout the book for symbols and notation. Secs. A.1 and A.2 deal with equivalence relations and orders of growth. Sec. A.3 presents the notation used in combinatorics and probability. The table in Section A.4 gives the main mathematical notation, and that in Section A.5 gives the notation for information theory. The table in Section A.6 summarizes the notation used for factor graphs and graph ensembles. The table in Section A.7 focuses on the notation used in message passing, belief propagation and survey propagation, and the cavity method.

A.1

Equivalence relations

As usual, the symbol ‘=’ denotes equality. We also use ‘≡’ for definitions and ‘≈’ for ‘numerically close to’. For instance, we may say that the Euler–Mascheroni constant is given by n 1 − log n ≈ 0.5772156649 . (A.1) γE ≡ lim n→∞ k k=1

d

When dealing with two random variables X and Y , we write X = Y if X and Y have the same distribution. For instance, given n + 1 i.i.d. Gaussian variables X0 , . . . , Xn , with zero mean and unit variance, we can write d 1 X0 = √ (X1 + · · · + Xn ) . n

(A.2)

We have adopted several equivalence symbols to denote the asymptotic behavior of functions as their argument tends to some limit. For the sake of simplicity, we assume here that the argument is an integer n → ∞. The limit to be considered in each . particular case should be clear from the context. We write f (n) = g(n) if f and g are equal ‘to the leading exponential order’ as n → ∞, i.e. if lim

n→∞

f (n) 1 log = 0. n g(n)

(A.3)

For instance, we may write

n n/2

. = 2n .

(A.4)

Symbols and notation

We write instead f (n) ∼ g(n) if f and g are asymptotically equal ‘up to a constant’, i.e. if lim

n→∞

f (n) =C, g(n)

for some constant C = 0. For instance, we have n 1 ∼ n−1/2 . 2n n/2

(A.5)

(A.6)

Finally, the symbol ‘’ is reserved for asymptoric equality, i.e. if lim

n→∞

f (n) = 1. g(n)

(A.7)

For instance, we have n 1 2 . 2n n/2 πn

(A.8)

The symbol ‘∼ =’ denotes equality up to a constant. If p( · ) and q( · ) are two measures on the same finite space X (not necessarily normalized), we write p(x) ∼ = q(x) if there exists C > 0 such that p(x) = C q(x) ,

(A.9)

for any x ∈ X . The definition generalizes straightforwardly to infinite sets X : the Radon–Nikodyn derivative between p and q is a positive constant.

A.2

Orders of growth

We have used a few symbols to denote the order of growth of functions when their arguments tend to some definite limit. For the sake of definiteness, we refer here to functions of an integer n → ∞. As above, the adaptation to any particular context should be straightforward. We write f (n) = Θ(g(n)), and say that f (n) is of order g(n), if there exist two positive constants C1 and C2 such that C1 g(n) ≤ |f (n)| ≤ C2 g(n) ,

(A.10)

for any n large enough. For instance, we have n

k = Θ(n2 ) .

(A.11)

f (n) = 0. g(n)

(A.12)

k=1

We write instead f (n) = o(g(n)) if lim

n→∞

Combinatorics and probability

For instance, n

k −

k=1

1 2 n = o(n2 ) . 2

(A.13)

Finally, f (n) = O(g(n)) if there exist a constant C such that |f (n)| ≤ C g(n)

(A.14)

for any n large enough. For instance, n3 sin(n/10) = O(n3 ) .

(A.15)

Note that both f (n) = Θ(g(n)) and f (n) = o(g(n)) imply f (n) = O(g(n)). As the last example shows, the converse is not necessarily true.

A.3

Combinatorics and probability

The standard notation is used for multinomial coefficients. For any n ≥ 0, l ≥ 2, and n1 , . . . , nl ≥ 0 such that n1 + · · · + nl = n, we have n! n . (A.16) ≡ n1 , n2 , . . . , nl n1 !n2 ! . . . nl ! For binomial coefficients (i.e. for l = 2), the usual shorthand is n! n n = ≡ . k, l − k k k!(n − k)!

(A.17)

In combinatorics, certain quantities are most easily described in terms of their generating functions. Given a formal power series f (x), coeff{f (x), xn } denotes the coefficient of the monomial xn in the series. More formally, f (x) = fn xn ⇒ fn = coeff{f (x), xn } . (A.18) n

For instance, coeff{(1 + x)m , xn } =

m . n

(A.19)

Some standard random variables are as follows: • A Bernoulli p variable is a random variable X taking values in {0, 1} such that P(X = 1) = p. • B(n, p) denotes a binomial random variable with parameters n and p. This is defined as a random variable taking values in {0, . . . , n} and having a probability distribution n k p (1 − p)n−k . (A.20) P{B(n, p) = k} = k


• A Poisson random variable X with parameter λ takes integer values and has the following probability distribution:

P{X = k} =

λk −λ e . k!

(A.21)

The parameter λ is the mean of X. Finally, we have used the symbol δa for the Dirac ‘delta function’. This is in fact a measure that attributes unit mass to the point a. In formulae, for any set A,

δa (A) = I(a ∈ A) .

A.4

Summary of mathematical notation

= ≡ ≈ d

= . = ∼ ∼ = Θ(f ) o(f ) argmaxf (x) · · N R β ↓ βc β ↑ βc ]a, b[ ]a, b] Z2 a⊕b I(·) A!0

(A.22)

Equal. Defined as. Numerically close to. Equal in distribution. Equal to the leading exponential order. Asymptotically equal up to a constant. Equal up to a normalization constant (for probabilities; see Eq. (14.3)). Of the same order as f (see Sec. A.2). Grows more slowly than f (see Sec. A.2). The set of values of x where the real-valued function f reaches its maximum. Integer part. x is the largest integer n such that n ≤ x. x is the smallest integer n such that n ≥ x. The set of integer numbers. The set of real numbers. β goes to βc through values > βc . β goes to βc through values < βc . Open interval of real numbers x such that a < x < b. Interval of real numbers x such that a < x ≤ b. The field of integers modulo 2. The sum modulo 2 of the two integers a and b. Indicator function: I(A) = 1 if the logical statement A is true, and I(A) = 0 if the statement A is false. The matrix A is positive semidefinite.

Information theory

A.5

Information theory

HX IXY H(p) M(X ) C & B

A.6

Factor graphs

GN (k, M ) GN (k, α) DN (Λ, P ) Tr (Λ, P ) Tr (k, α) Λ(x) P (x) λ(x) ρ(x) Bi,r (F ) Bi→a,t (F )

A.7

Entropy of the random variable X (See Eq. (1.7)). Mutual information of the random variables X and Y (see Eq. (1.25)). Entropy of a Bernoulli variable with parameter p. Space of probability distributions over a finite set X . Codebook. BMS(1) & BMS(2): channel BMS(2) is physically degraded with respect to BMS(1). Bhattacharya parameter of a channel.

Random k-factor graph with M function nodes and N variable nodes. Random k-factor graph with N variable nodes. Each function node is present independently with probability N α/ N k . Degre- constrained random factor graph ensemble. Degree-constrained random tree factor graph ensemble. Shorthand for the random tree factor graph Tr (Λ(x) = ekα(x−1) , P (x) = xk ). Degree profile of variable nodes. Degree profile of function nodes. Edge-perspective degree profile of variable nodes. Edge-perspective degree profile of function nodes. Neighbourhood of radius r of variable node i. Directed neigbourhood of an edge.

Cavity and message-passing methods

νi→a (xi ) ν a→i (xi ) Φ F(ν) Fe (ν) f RS Qi→a (ν)

a→i (

Q ν) x F(x)

BP messages (variable to function node). BP messages (function to variable node). Free entropy. Bethe free entropy (as a function of messages). Bethe energy (as a function of min-sum messages). Bethe (RS) free-entropy density. 1RSB cavity message/SP message (variable to function node). 1RSB cavity message/SP message (function to variable node). Parisi 1RSB parameter. Free-entropy density of the auxiliary model counting BP fixed points.


Σ(φ) FRSB (Q) f RSB y Fe (y) Σe (e) FRSB,e (Q) f RSB,e

Complexity. 1RSB cavity free entropy (Bethe free entropy of the auxiliary model, a function of the messages). 1RSB cavity free-entropy density. Zero-temperature Parisi 1RSB parameter (y = limβ→∞ βx). Free-entropy density of the auxiliary model counting min-sum fixed points. Energetic complexity. Energetic 1RSB cavity free entropy (Bethe free entropy of the auxiliary model, a function of the messages). Energetic 1RSB cavity free-entropy density.

References Aarts, E. H. L., Korst, J. H. M., and van Laarhoven, P. J. M. (2003). Simulated annealing. In Local Search in Combinatorial Optimization, pp. 91–120. Princeton University Press. Abou-Chacra, R., Anderson, P. W., and Thouless, D. J. (1973). A self-consistent theory of localization. J. Phys. C , 6, 1734–1752. Achlioptas, D. (2001). Lower bounds for random 3-SAT via differential equations. Theor. Comput. Sci., 265, 159–185. Achlioptas, D. (2007). Personal communication. Achlioptas, D. and Moore, C. (2004). The chromatic number of random regular graphs. In Proc. RANDOM’04, Lecture Notes in Computer Science, Vol. 3122, pp. 219–228. Springer, Berlin. Achlioptas, D. and Moore, C. (2007). Random k-SAT: Two moments suffice to cross a sharp threshold. SIAM J. Comput., 36, 740–762. Achlioptas, D. and Naor, A. (2005). The two possible values of the chromatic number of a random graph. Ann. Math., 162, 1333–1349. Achlioptas, D. and Peres, Y. (2004). The threshold for random k-SAT is 2k log 2 − O(k). J. Am. Math. Soc., 17, 947–973. Achlioptas, D. and Ricci-Tersenghi, F. (2006). On the solution-space geometry of random constraint satisfaction problems. In Proc. 38th ACM Symposium on Theory of Computing, STOC, Seattle, WA. Achlioptas, D., Naor, A., and Peres, Y. (2005). Rigorous location of phase transitions in hard optimization problems. Nature, 435, 759–764. Aizenman, M., Sims, R., and Starr, S. L. (2005). Mean-field spin glass models from the Cavity-ROSt perspective. In Mathematical Physics of Spin Glasses, Cortona. Eprint arXiv:math-ph/0607060. Aji, S.M. and McEliece, R.J. (2000). The generalized distributive law. IEEE Trans. Inf. Theory, 46, 325–343. Aji, A., Jin, H., Khandekar, A., MacKay, D., and McEliece, R. (2001). BSC thresholds for code ensembles based on ‘Typical Pairs’ decoding. In Codes, Systems, and Graphical Models, IMA Volumes in Mathematics and Its Applications, Vol.123, pp. 1–37. Springer, Berlin. Alava, M., Ardelius, J., Aurell, E., Kaski, P., Krishnamurthy, S., Orponen, P., and Seitz, S. (2007). Circumspect descent prevails in solving random constraint satisfaction problems. Eprint arXiv:0711.4902. Aldous, D. (1992). Asymptotics in the random assignment problem. Probab. Theory Relat. Fields, 93, 507–534. Aldous, D. (2001). The ζ(2) limit in the random assignment problem. Random Struct. Algorithms, 18, 381–418.

References

Aldous, D. and Bandyopadhyay, A. (2005). A survey of max-type recursive distributional equations. Ann. Appl. Probab., 15, 1047–1110. Aldous, D. and Fill, J. (2008). Reversible Markov chains and random walks on graphs. Book in preparation. Available online at http://www.stat.berkeley.edu/users/aldous/RWG/book.html. Aldous, D. and Steele, J. M. (2003). The objective method: Probabilistic combinatorial optimization and local weak convergence. In Probability on Discrete Structures (ed. H. Kesten), pp. 1–72. Springer, Berlin. Altarelli, F., Monasson, R., and Zamponi, F. (2007). Relationship between clustering and algorithmic phase transitions in the random k-XORSAT model and its NPcomplete extensions. J. Phys. A, 40, 867–886. Proc. International Workshop on Statistical-Mechanical Informatics, Kyoto. Amraoui, A., Montanari, A., Richardson, T. J., and Urbanke, R. (2004). Finite-length scaling for iteratively decoded LDPC ensembles. IEEE Trans. Inf. Theory, in press. Eprint http://arxiv.org/abs/cs/0406050. Amraoui, A., Montanari, A., and Urbanke, R. (2007). How to find good finite-length codes: From art towards science. Eur. Trans. Telecommun., 18, 491–508. Applegate, D., Bixby, R., Chv´ atal, V., and Cook, W. The traveling salesman problem. Available online at http://www.tsp.gatech.edu/. Ardelius, J. and Aurell, E. (2006). Behavior of heuristics on large and hard satisfiability problems. Phys. Rev. E , 74, 037702. Baldassi, C., Braunstein, A., Brunel, N., and Zecchina, R. (2007). Efficient supervised learning in networks with binary synapses. Proc. Natl. Acad. Sci., 1049, 11079– 11084. Balian, R. (1992). From Microphysics to Macrophysics: Methods and Applications of Statistical Physics. Springer, New York. Bandyopadhyay, A. and Gamarnik, D. (2006). Counting without sampling: New algorithms for enumeration problems using statistical physics. In Proc. 17th annual ACM-SIAM Symposium on Discrete Algorithms. ACM, New York. Barg, A. (1998). Complexity issues in coding theory. In Handbook of Coding Theory (ed. V. S. Pless and W. C. Huffman), Chapter 7. Elsevier Science, Amsterdam. Barg, A. and Forney, G. D. (2002). Random codes: minimum distances and error exponents. IEEE Trans. Inf. Theory, 48, 2568–2573. Bauke, H. (2002). Statistische Mechanik des Zahlenaufteilungsproblems. Available at: http://tina.nat.uni-magdeburg.de/heiko/Publications/index.php. Baxter, R. J. (1982). Exactly Solved Models in Statistical Mechanics. Academic Press, London. Bayati, M., Shah, D., and Sharma, M. (2005). Maximum weight matching via maxproduct belief propagation. In Proc. IEEE International Symposium on Information Theory, Adelaide, pp. 1763–1767. Bayati, M., Shah, D., and Sharma, M. (2006). A simpler max-product maximum weight matching algorithm and the auction algorithm. In Proc. IEEE International Symposium on Information Theory, Seattle, pp. 557–561. Bender, E. A. and Canfield, E. R. (1978). The asymptotic number of labeled graphs with given degree sequence. J. Comb. Theory (A), 24, 296–307.

References

Berger, N., Kenyon, C., Mossel, E., and Peres, Y. (2005). Glauber dynamics on trees and hyperbolic graphs. Probab. Theory Relat. Fields, 131, 311–340. Berlekamp, E., McEliecee, R. J., and van Tilborg, H. C. A. (1978). On the inherent intractability of certain coding problems. IEEE Trans. Inf. Theory, 29, 384–386. Berrou, C. and Glavieux, A. (1996). Near optimum error correcting coding and decoding: Turbo codes. IEEE Trans. Commun., 44, 1261–1271. Bertsekas, D. P. (1988). The auction algorithm: A distributed relaxation method for the assignment problem. Ann. Oper. Res., 14, 105–123. Bethe, H. A. (1935). Statistical theory of superlattices. Proc. R. Soc. London A, 150, 552–558. Binder, K. and Young, A. P. (1986). Spin glasses: Experimental facts, theoretical concepts, and open questions. Rev. Mod. Phys., 58, 801–976. Biroli, G. and Mézard, M. (2002). Lattice glass models. Phys. Rev. Lett., 88, 025501. Boettcher, S. (2003). Numerical results for ground states of mean-field spin glasses at low connectivities. Phys. Rev. B , 67, 060403. Bollob´ as, B. (1980). A probabilistic proof of an asymptotic formula for the number of labelled regular graphs. Eur. J. Combinatorics, 1, 296–307. Bollob´ as, B. (2001). Random Graphs. Cambridge University Press, Cambridge. Bollobas, B., Borgs, C., Chayes, J. T., Kim, J. H., and Wilson, D. B. (2001). The scaling window of the 2-SAT transition. Random Struct. Algorithms, 18, 201–256. Borgs, C., Chayes, J., and Pittel, B. (2001). Phase transition and finite-size scaling for the integer partitioning problem. Random Struct. Algorithms, 19, 247. Borgs, C., Chayes, J., Mertens, S., and Pittel, B. (2003). Phase diagram for the constrained integer partitioning problem. Random Struct. Algorithms, 24, 315–380. Bouchaud, J.-P. and Biroli, G. (2004). On the Adam–Gibbs–Kirkpatrick–Thirumalai– Wolynes scenario for the viscosity increase in glasses. J. Chem. Phys, 121, 7347– 7354. Bouchaud, J.-P. and Mézard, M. (1997). Universality classes for extreme value statistics. J. Phys. A: Math. Gen., 30, 7997–8015. Bouchaud, J.-P. and Potters, M. (2003). Theory of Financial Risk and Derivative Pricing. Cambridge University Press, Cambridge. Bouchaud, J.-P., Cugliandolo, L., Kurchan, J., and Mézard, M. (1997). Out of equilibrium dynamics in spin glasses and other glassy systems. In Spin Glasses and Random Fields (ed. A.P. Young), pp.161–224. World Scientific, Singapore. Bouchaud, J.-P., Mézard, M., and Dalibard, J. (eds.) (2007). Complex Systems: Lecture Notes of the Les Houches Summer School 2006. Elsevier, Amsterdam. Boyd, S. P. and Vandenberghe, L. (2004). Convex Optimization. Cambridge University Press, Cambridge. Braunstein, A. and Zecchina, R. (2004). Survey propagation as local equilibrium equations. J. Stat. Mech., 06007. Braunstein, A., Mulet, R., Pagnani, A., Weigt, M., and Zecchina, R. (2003). Polynomial iterative algorithms for coloring and analyzing random graphs. Phys. Rev. E , 68, 036702. Braunstein, A., Mézard, M., and Zecchina, R. (2004). C code for the SP algorithm. Available online at http://www.ictp.trieste.it/∼zecchina/SP/.

¼

References

Braunstein, A., Mézard, M., and Zecchina, R. (2005). Survey propagation: an algorithm for satisfiability. Random Struct. Algorithms, 27, 201–226. Burshtein, D. and Miller, G. (2004). Asymptotic enumeration methods for analyzing LDPC codes. IEEE Trans. Inf. Theory, 50, 1115–1131. Burshtein, D., Krivelevich, M., Litsyn, S., and Miller, G. (2002). Upper bounds on the rate of LDPC codes. IEEE Trans. Inf. Theory, 48, 2437–2449. Caracciolo, S., Parisi, G., Patarnello, S., and Sourlas, N. (1990). 3d Ising spin glass in a magnetic field and mean-field theory. Europhys. Lett., 11, 783. Carlson, J. M., Chayes, J. T., Chayes, L., Sethna, J. P., and Thouless, D. J. (1990). Bethe lattice spin glass: The effects of a ferromagnetic bias and external fields. I. Bifurcation. J. Stat. Phys., 61, 987–1067. Challet, D., Marsili, M., and Zhang, Y.-C. (2005). Minority Games. Oxford University Press, Oxford. Chao, M.-T. and Franco, J. (1986). Probabilistic analysis of two heuristics for the 3-satisfiability problem. SIAM J. Comput, 15, 1106–1118. Chao, M. T. and Franco, J. (1990). Probabilistic analysis of a generalization of the unit-clause literal selection heuristics for the k-satisfiability problem. Inf. Sci , 51, 289–314. Chavas, J., Furtlehner, C., Mézard, M., and Zecchina, R. (2005). Survey-propagation decimation through distributed local computations. J. Stat. Mech., 11016. Chayes, J. T., Chayes, L., Sethna, J. P., and Thouless, D. J. (1986). A mean field spin glass with short-range interactions. Commun. Math. Phys., 106, 41–89. Chung, S.-Y., Forney, G. D., Richardson, T. J., and Urbanke, R. (2001). On the design of low-density parity-check codes within 0.0045 dB of the Shannon limit. IEEE Commun. Lett., 5, 58–60. Chv´ atal, V. and Reed, B. (1992). Mick gets some (and the odds are on his side). In Proc. 33rd IEEE Symposium on Foundations of Computer Science, FOCS, Pittsburgh, pp. 620–627. Ciliberti, S., Mézard, M., and Zecchina, R. (2005). Lossy data compression with random gates. Phys. Rev. Lett., 95, 038701. Clifford, P. (1990). Markov random fields in statistics. In Disorder in Physical Systems: A Volume in Honour of John M. Hammersley (eds. G. Grimmett and D. Welsh), pp. 19–32. Oxford University Press, Oxford. Cocco, S. and Monasson, R. (2001a). Statistical physics analysis of the computational complexity of solving random satisfiability problems using backtrack algorithms. Eur. Phys. J. B , 22, 505–531. Cocco, S. and Monasson, R. (2001b). Trajectories in phase diagrams, growth processes, and computational complexity: How search algorithms solve the 3satisfiability problem. Phys. Rev. Lett., 86, 1654–1657. Cocco, S., Dubois, O., Mandler, J., and Monasson, R. (2003). Rigorous decimationbased construction of ground pure states for spin-glass models on random lattices. Phys. Rev. Lett., 90, 047205. Cocco, S., Monasson, R., Montanari, A., and Semerjian, G. (2006). Approximate analysis of search algorithms with physical methods. In Computational Complexity and Statistical Physics (eds. A. Percus, G. Istrate, and C. Moore), Santa Fe Studies

References

½

in the Science of Complexity, pp. 1–37. Oxford University Press, Oxford. Conway, J. H. and Sloane, N. J. A. (1998). Sphere Packings, Lattices and Groups. Springer, New York. Cook, S. A. (1971). The complexity of theorem-proving procedures. In Proc. 3rd ACM Symposium on the Theory of Computing, STOC, Shaker Heights, OH, pp. 151–158. Coolen, A.C.C., Kuehn, R., and Sollich, P. (2005). Theory of Neural Information Processing Systems. Oxford University Press, Oxford. Cooper, G. F. (1990). The computational complexity of probabilistic inference using Bayesian belief networks. Artif. Intell./, 42, 393–405. Coppersmith, D. and Sorkin, G. B. (1999). Constructive bounds and exact expectations for the random assignment problem. Random Struct. Algorithms, 15, 133–144. Cover, T. M. and Thomas, J. A. (1991). Elements of Information Theory. Wiley, New York. Creignou, N. and Daudé, H. (1999). Satisfiability threshold for random XOR-CNF formulas. Discrete Appl. Math., 96–97, 41–53. Creignou, N., Daudé, H., and Dubois, O. (2003). Approximating the satisfiability threshold for random k-XOR-formulas. Combinatorics, Probab. Comput., 12, 113– 126. Crisanti, A. and Leuzzi, L. (2007). Amorphous–amorphous transition and the twostep replica symmetry breaking phase. Phys. Rev. B , 76, 184417. Csiszár, I. and K¨ orner, J. (1981). Information Theory: Coding Theorems for Discrete Memoryless Systems. Academic Press, New York. Dagum, P. and Luby, M. (1993). Approximating probabilistic inference in Bayesian belief networks is NP-hard. Artif. Intell., 60, 141–153. Darling, R. W. R. and Norris, J. R. (2005). Structure of large random hypergraphs. Ann. Appl. Probab., 15, 125–152. Daudé, H., Mézard, M., Mora, T., and Zecchina, R. (2008). Pairs of SAT assignments in random boolean formulae. Theor. Comput. Sci., 393, 260–279. Davis, M. and Putnam, H. (1960). A computing procedure for quantification theory. J. Assoc. Comput. Mach., 7, 201–215. Davis, M., Logemann, G., and Loveland, D. (1962). A machine program for theoremproving. Commun. ACM , 5, 394–397. de Almeida, J. R. L. and Thouless, D. (1978). Stability of the Sherrington–Kirkpatrick solution of a spin glass model. J. Phys. A, 11, 983–990. De Dominicis, C. and Mottishaw, P. (1987). Replica symmetry breaking in weak connectivity systems. J. Phys. A, 20, L1267–L1273. de la Vega, W. F. (1992). On random 2-SAT. Unpublished manuscript. de la Vega, W. F. (2001). Random 2-SAT: Results and problems. Theor. Comput. Sci., 265, 131–146. Dembo, A. and Montanari, A. (2008a). Finite size scaling for the core of large random hypergraphs. Ann. Appl. Probab.. Dembo, A. and Montanari, A. (2008b). Graphical models, Bethe states and all that. In preparation. Dembo, A. and Montanari, A. (2008c). Ising models on locally tree-like graphs.

¾

References

Eprint arxiv:0804.4726. Dembo, A. and Zeitouni, O. (1998). Large Deviations Techniques and Applications. Springer, New York. Derrida, B. (1980). Random-energy model: Limit of a family of disordered models. Phys. Rev. Lett., 45, 79. Derrida, B. (1981). Random-energy model: An exactly solvable model of disordered systems. Phys. Rev. B , 24, 2613–2626. Derrida, B. (1985). A generalization of the random energy model which includes correlations between energies. J. Physique Lett., 46, L401–L407. Derrida, B. and Gardner, E. (1986). Solution of the generalised random energy model. J. Phys. C , 19, 2253–2274. Derrida, B. and Toulouse, G. (1985). Sample to sample fluctuations in the random energy model. J. Physique Lett., 46, L223–L228. Di, C., Proietti, D., Richardson, T. J., Telatar, E., and Urbanke, R. (2002). Finite length analysis of low-density parity-check codes on the binary erasure channel. IEEE Trans. Inf. Theory, 48, 1570–1579. Di, C., Montanari, A., and Urbanke, R. (2004). Weight distribution of LDPC code ensembles: Combinatorics meets statistical physics. In Proc. IEEE International Symposium on Information Theory, Chicago, p. 102. Di, C., Richardson, T. J., and Urbanke, R. (2006). Weight distribution of low-density parity-check codes. IEEE Trans. Inf. Theory, 52, 4839–4855. Diaconis, P. and Saloff-Coste, L. (1993). Comparison theorems for reversible Markov chains. Ann. Appl. Probab., 3, 696–730. Diaconis, P. and Stroock, D. (1991). Geometric bounds for eigenvalues of Markov chains. Ann. Appl. Probab., 1, 36–61. Dolecek, L., Zhang, Z., Anantharam, V., and Nikolić, B. (2007). Analysis of absorbing sets for array-based LDPC codes. In Proc. IEEE International Conference on Communications, ICC, Glasgow. Dorogotsev, S. N., Goltsev, A. V., and Mendes, J. F. F. (2002). Ising models on networks wth arbitrary distribution of connections. Phys. Rev. E , 66, 016104. Dubois, O. and Boufkhad, Y. (1997). A general upper bound for the satisfiability threshold of random r-sat formulae. J. Algorithms, 24, 395–420. Dubois, O. and Mandler, J. (2002). The 3-XORSAT threshold. In Proc. 43rd IEEE Symposium on Foundations of Computer Science, FOCS, pp. 769–778. Duchet, P. (1995). Hypergraphs. In Handbook of Combinatorics (eds. R. Graham, M. Grotschel, and L. Lovasz), pp. 381–432. MIT Press, Cambridge, MA. Durrett, R. (1995). Probability: Theory and Examples. Duxbury Press, New York. Edwards, S. F. and Anderson, P. W. (1975). Theory of spin glasses. J. Phys. F , 5, 965–974. Elias, P. (1955). Coding for two noisy channels. In Third London Symposium on Information Theory, pp. 61–76. Ellis, R. S. (1985). Entropy, Large Deviations and Statistical Mechanics. Springer, New York. Erd¨ os, P. and Rényi, A. (1960). On the evolution of random graphs. Publ. Math. Sci. Hung. Acad. Sci, 5, 17–61.

References

¿

Euler, L. (1736). Solutio problematis ad geometriam situs pertinentis. Comment. Acad. Sci. U. Petrop., 8, 128–140. Reprinted in Opera Omnia Ser. I-7, pp. 1–10, 1766. Evans, W., Kenyon, C., Peres, Y., and Schulman, L. J. (2000). Broadcasting on trees and the Ising model. Ann. Appl. Probab., 10, 410–433. Feller, W. (1968). An Introduction to Probability Theory and Its Applications. Wiley, New York. Ferreira, F. F. and Fontanari, J. F. (1998). Probabilistic analysis of the number partitioning problem. J. Phys. A, 31, 3417–3428. Fischer, K. H. and Hetz, J. A. (1993). Spin Glasses. Cambridge University Press, Cambridge. Flajolet, P. and Sedgewick, R. (2008). Analytic Combinatorics. Cambridge University Press, Cambridge. Forney, G. D. (2001). Codes on graphs: Normal realizations. IEEE Trans. Inf. Theory, 47, 520–548. Forney, G. D. and Montanari, A. (2001). On exponential error bounds for random codes on the DMC. Available online at http://www.stanford.edu /∼montanar/PAPERS/. Franco, J. (2000). Some interesting research directions in satisfiability. Ann. Math. Artif. Intell., 28, 7–15. Franz, S. and Leone, M. (2003). Replica bounds for optimization problems and diluted spin systems. J. Stat. Phys, 111, 535–564. Franz, S. and Parisi, G. (1995). Recipes for metastable states in spin glasses. J. Physique I , 5, 1401. Franz, S., Leone, M., Ricci-Tersenghi, F., and Zecchina, R. (2001a). Exact solutions for diluted spin glasses and optimization problems. Phys. Rev. Lett., 87, 127209. Franz, S., Mézard, M., Ricci-Tersenghi, F., Weigt, M., and Zecchina, R. (2001b). A ferromagnet with a glass transition. Europhys. Lett., 55, 465. Franz, S., Leone, M., Montanari, A., and Ricci-Tersenghi, F. (2002). Dynamic phase transition for decoding algorithms. Phys. Rev. E , 22, 046120. Franz, S., Leone, M., and Toninelli, F. (2003). Replica bounds for diluted nonPoissonian spin systems. J. Phys. A, 36, 10967–10985. Friedgut, E. (1999). Sharp thresholds of graph proprties, and the k-sat problem. J. Am. Math. Soc., 12, 1017–1054. Friedman, N., Linial, M., Nachman, I., and Peér, D. (2000). Using bayesian networks to analyze expression data. J. Comput. Biol., 7, 601–620. Galavotti, G. (1999). Statistical Mechanics: A Short Treatise. Springer, New York. Gallager, R. G. (1962). Low-density parity-check codes. IEEE Trans. Inf. Theory, 8, 21–28. Gallager, R. G. (1963). Low-Density Parity-Check Codes. MIT Press, Cambridge, MA. Available online at http://web./gallager/www/pages/ldpc.pdf. Gallager, R. G. (1965). A simple derivation of the coding theorem and some applications. IEEE Trans. Inf. Theory, IT-11, 3–18. Gallager, R. G. (1968). Information Theory and Reliable Communication. Wiley, New York.

References

Gamarnik, D. (2004). Linear phase transition in random linear constraint satisfaction problems. Probab. Theory Relat. Fields, 129, 410–440. Gardner, E. (1985). Spin glasses with p-spin interactions. Nucl. Phys. B , 257, 747– 765. Garey, M. R. and Johnson, D. S. (1979). Computers and Intractability: A Guide to the Theory of NP-Completeness. W. H. Freeman, New York. Garey, M. R., Johnson, D. S., and Stockmeyer, L. (1976). Some simplified NPcomplete graph problems. Theor. Comput. Sci., 1, 237–267. Gent, I. P. and Walsh, T. (1998). Analysis of heuristics for number partitioning. Comput. Intell., 14(3), 430–451. Georgii, H.-O. (1988). Gibbs Measures and Phase Transitions. Walter de Gruyter, Berlin. Gerschenfeld, A. and Montanari, A. (2007). Reconstruction for models on random graph. In Proc. 48th IEEE Symposium on Foundations of Computer Science, FOCS, Providence, RI, pp. 194–205. Goerdt, A. (1996). A threshold for unsatisfiability. J. Comput. Syst. Sci., 53, 469– 486. Goldschmidt, Y. Y. (1991). Spin glass on the finite-connectivity lattice: The replica solution without replicas. Phys. Rev. B , 43, 8148–8152. Goldschmidt, Y. Y. and De Dominicis, C. (1990). Replica symmetry breaking in the spin-glass model on lattices with finite connectivity: Application to graph partitioning. Phys. Rev. B , 41, 2184–2197. Goldschmidt, Y. Y. and Lai, P. Y. (1990). The finite connectivity spin glass: Investigation of replica symmetry breaking of the ground state. J. Phys. A, 23, L775–L782. Gomes, C. P. and Selman, B. (2005). Can get satisfaction. Nature, 435, 751–752. Gross, D. J. and Mézard, M. (1984). The simplest spin glass. Nucl. Phys., B240 [FS12], 431–452. Grosso, C. (2004). Cavity method analysis for random assignment problems. Tesi di Laurea, Universit` a degli Studi di Milano. Gu, J., Purdom, P. W., Franco, J., and Wah, B. W. (1996). Algorithms for the satisfiability (SAT) problem: A survey. In Satisfiability Problem: Theory and Applications (eds. D. Du, J. Gu, and P. M. Pardalos), pp. 19–151. American Mathematical Society, Providence, RI. Guerra, F. (2003). Broken replica symmetry bounds in the mean field spin glass model. Commun. Math. Phys., 233, 1–12. Guerra, F. (2005). Spin glasses. Eprint arXiv:cond-mat/0507581. Guo, D. and Verd´ u, S. (2002). Multiuser detection and statistical mechanics. In Communications Information and Network Security (eds. V. Bhargava, H. Poor, V. Tarokh, and S. Yoon), Chapter 13, pp. 229–277. Kluwer Academic, Dordrecht. Hartmann, A. K. and Rieger, H. (eds.) (2002). Optimization Algorithms in Physics. Wiley-VCH, Berlin. Hartmann, A. K. and Rieger, H. (2004). New Optimization Algorithms in Physics. Wiley-VCH, Berlin. Hartmann, A. K. and Weigt, M. (2005). Phase Transitions in Combinatorial Optimization Problems. Wiley-VCH, Weinheim.

References

Hayes, B. (2002). The easiest hard problem. Am. Sci., 90(2), 113–117. Henley, C. L. (1986). Ising domain growth barriers on a Cayley tree at percolation. Phys. Rev. B , 33, 7675–7682. Huang, K. (1987). Statistical Mechanics. Wiley, New York. Janson, S., Luczak, T., and Ruci´ nski, A. (2000). Random Graphs. Wiley, New York. Jaynes, E. T. (1957). Information theory and statistical mechanics. Phys. Rev., 106, 620–630. Jensen, F. V. (ed.) (1996). An Introduction to Bayesian Networks. UCL Press, London. Jerrum, M. and Sinclair, A. (1996). The Markov chain Monte Carlo method: An approach to approximate counting and integration. In Approximation Algorithms for NP-Hard Problems, pp. 482–520. PWS Publishing, Boston, MA. Johnston, D. A. and Plech´ ac, P. (1998). Equivalence of ferromagnetic spin models on trees and random graphs. J. Phys. A, 31, 475–482. Jordan, M. (ed.) (1998). Learning in Graphical Models. MIT Press, Boston, MA. Kabashima, Y. and Saad, D. (1998). Belief propagation vs. TAP for decoding corrupted messages. Europhys. Lett., 44, 668–674. Kabashima, Y. and Saad, D. (1999). Statistical mechanics of error correcting codes. Europhys. Lett., 45, 97–103. Kabashima, Y. and Saad, D. (2000). Error-correcting code on a cactus: A solvable model. Europhys. Lett., 51, 698–704. Kabashima, Y., Murayama, T., and Saad, D. (2000a). Typical performance of Gallager-type error-correcting codes. Phys. Rev. Lett., 84, 1355–1358. Kabashima, Y., Murayama, T., Saad, D., and Vicente, R. (2000b). Regular and irregular Gallager-type error correcting codes. In Advances in Neural Information Processing Systems 12 (eds. S. A. Solla, T. K. Leen, and K.R. M¨ uller). MIT Press, Cambridge, MA. Kanter, I. and Sompolinsky, H. (1987). Mean field theory of spin-glasses with finite coordination number. Phys. Rev. Lett., 58, 164–167. Karmarkar, N. and Karp, R. M. (1982). The differencing method of set partitioning. Technical Report CSD 82/113, University of California, Berkeley. Karmarkar, N., Karp, R. M., Lueker, G. S., and Odlyzko, A. M. (1986). Probabilistic analysis of optimum partitioning. J. Appl. Probab., 23, 626–645. Karo´ nski, M. and Luczak, T. (2002). The phase transition in a random hypergraph. J. Comput. Appl. Math., 142, 125–135. Karp, R. M. (1987). An upper bound on the expected cost of an optimal assignment. In Proc. Japan–U.S. Joint Seminar, New York (eds D. S. Johnson, T. Nishizeki, A. Nozaki, and H. S. Wilf), Perspectives in Computing, Vol. 15, pp 1–4, Academic Press, Orlando. Katsura, S., Inawashiro, S., and Fujiki, S. (1979). Spin glasses for the infinitely long ranged bond Ising model without the use of the replica method. Physica, 99A, 193–216. Kelly, F. P. (1991). Network routing. Phil. Trans. R. Soc. London A, 337, 343–367. Kikuchi, R. (1951). A theory of cooperative phenomena. Phys. Rev., 81, 988–1003. Kirkpatrick, S. and Selman, B. (1994). Critical behavior in the satisfiability of random

References

boolean expressions. Science, 264, 1297–1301. Kirkpatrick, S. and Sherrington, D. (1978). Infinite ranged models of spin glasses. Phys. Rev. B , 17, 4384–4403. Kirkpatrick, T. R. and Thirumalai, D. (1987). p-spin interaction spin glass models: Connections with the structural glass problem. Phys. Rev. B , 36, 5388–5397. Kirkpatrick, T. R. and Wolynes, P. G. (1987). Connections between some kinetic and equilibrium theories of the glass transition. Phys. Rev. A, 35, 3072–3080. Kirkpatrick, S., Gelatt, C. D., and Vecchi, M. P. (1983). Optimization by simulated annealing. Science, 220, 671–680. Kirousis, L. M., Kranakis, E., Krizanc, D., and Stamatiou, Y. (1998). Approximating the unsatisfiability threshold of random formulas. Random Struct. Algorithms, 12, 253–269. Klein, M. W., Schowalter, L. J., and Shukla, P. (1979). Spin glasses in the Bethe– Peierls–Weiss and other mean field approximations. Phys. Rev. B , 19, 1492–1502. Kleinberg, J. M., Kumar, R., Raghavan, P., Rajagopalan, S., and Tomkins, A. S. (1999). The Web as a graph: Measurements, models, and methods. In 5th Conference on Computing and Combinatorics, Tokyo, pp. 1–17. Korf, R. E. (1998). A complete anytime algorithm for number partitioning. Artif. Intell., 106, 181:203. K¨ otter, R. and Vontobel, P. O. (2003). Graph covers and iterative decoding of finitelength codes. In Proc. 3rd Int. Conf. on Turbo Codes and Related Topics, Brest, France, pp. 75–82. Krauth, W. (2006). Statistical Mechanics: Algorithms and Computations. Oxford University Press, Oxford. Krauth, W. and Mézard, M. (1989). The cavity method and the Traveling Salesman Problem. Europhys. Lett., 8, 213–218. Krzakala, F. and Zdeborova, L. (2007). Phase transitions in the coloring of random graphs. Phys. Rev. E , 76, 031131. Krzakala, F., Montanari, A., Ricci-Tersenghi, F., Semerjian, G., and Zdeborova, L. (2007). Gibbs states and the set of solutions of random constraint satisfaction problems. Proc. Natl. Acad. Sci., 104, 10318–10323. Kschischang, F. R., Frey, B. J., and Loeliger, H.-A. (2001). Factor graphs and the sum–product algorithm. IEEE Trans. Inf. Theory, 47, 498–519. Lauritzen, S. L. (1996). Graphical Models. Oxford University Press, Oxford. Leone, M., Ricci-Tersenghi, F., and Zecchina, R. (2001). Phase coexistence and finitesize scaling in random combinatorial problems. J. Phys. A, 34, 4615–4626. Leone, M., Vázquez, A., Vespignani, A., and Zecchina, R. (2004). Ferromagnetic ordering in graphs with arbitrary degree distribution. Eur. Phys. J. B , 28, 191– 197. Linusson, S. and W¨ astlund, J. (2004). A proof of Parisi’s conjecture on the random assignment problem. Probab. Theory Relat. Fields, 128, 419–440. Litsyn, S. and Shevelev, V. (2003). Distance distributions in ensembles of irregular low-density parity-check codes. IEEE Trans. Inf. Theory, 49, 3140–3159. Luby, M., Mitzenmacher, M., Shokrollahi, A., Spielman, D. A., and Stemann, V. (1997). Practical loss-resilient codes. In Proc. 29th ACM Symposium on Theory of

References

Computing, STOC, pp. 150–159. Luby, M., Mitzenmacher, M., Shokrollahi, A., and Spielman, D. A. (1998). Analysis of low density codes and improved designs using irregular graphs. In Proc. 30th ACM Symposium on Theory of Computing, STOC, pp. 249–258. Luby, M., Mitzenmacher, M., Shokrollahi, A., and Spielman, D. A. (2001a). Efficient erasure correcting codes. IEEE Trans. Inf. Theory, 47(2), 569–584. Luby, M., Mitzenmacher, M., Shokrollahi, A., and Spielman, D. A. (2001b). Improved low-density parity-check codes using irregular graphs. IEEE Trans. Inf. Theory, 47, 585–598. Ma, S.-K. (1985). Statistical Mechanics. World Scientific, Singapore. MacKay, D. J. C. (1999). Good error correcting codes based on very sparse matrices. IEEE Trans. Inf. Theory, 45, 399–431. MacKay, D. J. C. (2002). Information Theory, Inference & Learning Algorithms. Cambridge University Press, Cambridge. MacKay, D. J. C. and Neal, R. M. (1996). Near Shannon limit performance of low density parity check codes. Electron. Lett., 32, 1645–1646. MacKay, D. J. C. and Postol, M. S. (2003). Weaknesses of Margulis and Ramanujan– Margulis low-density parity check codes. Elect. Notes in Theor. Computer Sci., 74, 97–104. Macris, N. (2007). Sharp bounds on generalised EXIT function. IEEE Trans. Inf. Theory, 53, 2365–2375. Maneva, E., Mossel, E., and Wainwright, M. J. (2005). A new look at survey propagation and its generalizations. In Proc. 16th ACM–SIAM Symposium on Discrete Algorithms, SODA, Vancouver, pp. 1089–1098. Marinari, E., Parisi, G., and Ruiz-Lorenzo, J. (1997). Numerical simulations of spin glass systems. In Spin Glasses and Random Fields (ed. A. Young). World Scientific, Singapore. Martin, O. C., Mézard, M., and Rivoire, O. (2005). Random multi-index matching problems. J. Stat. Mech., 09006. Martinelli, F. (1999). Lectures on Glauber dynamics for discrete spin models. In Lectures on Probability Theory and Statistics, Saint-Flour 1997, Lecture Notes in Mathematics, pp. 93–191. Springer, Berlin. Martinelli, F., Sinclair, A., and Weitz, D. (2004). Glauber dynamics on trees: Boundary conditions and mixing time. Commun. Math. Phys, 250, 301–334. McEliece, R. J., MacKay, D. J. C., and Cheng, J.-F. (1998). Turbo decoding as an instance of Pearl’s ‘belief propagation’ algorithm. IEEE J. Sel. Areas Commun., 16, 140–152. Méasson, C., Montanari, A., and Urbanke, R. (2008). Maxwell construction: The hidden bridge between iterative and maximum a posteriori decoding. IEEE Trans. Inf. Theory,, 54, 5277–5307. Méasson, C., Montanari, A., Richardson, T., and Urbanke, R. (2005). The generalized area theorem and some of its consequences. Submitted. Eprint http://arxiv.org/abs/cs/0511039. Mertens, S. (1998). Phase transition in the number partitioning problem. Phys. Rev. Lett., 81(20), 4281–4284.

References

Mertens, S. (2000). Random costs in combinatorial optimization. Phys. Rev. Lett., 84(7), 1347–1350. Mertens, S. (2001). A physicist’s approach to number partitioning. Theor. Comput. Sci., 265, 79–108. Mertens, S., Mézard, M., and Zecchina, R. (2006). Threshold values of random KSAT from the cavity method. Random Struct. Algorithms, 28, 340–373. Mézard, M. and Montanari, A. (2006). Reconstruction on trees and spin glass transition. J. Stat. Phys., 124, 1317–1350. Mézard, M. and Parisi, G. (1985). Replicas and optimization. J. Physique Lett., 46, L771–L778. Mézard, M. and Parisi, G. (1986). Mean-field equations for the matching and the traveling salesman problems. Europhys. Lett., 2, 913–918. Mézard, M. and Parisi, G. (1987). Mean-field theory of randomly frustrated systems with finite connectivity. Europhys. Lett., 3, 1067–1074. Mézard, M. and Parisi, G. (1999). Thermodynamics of glasses: A first principles computation. Phys. Rev. Lett., 82, 747–751. Mézard, M. and Parisi, G. (2001). The Bethe lattice spin glass revisited. Eur. Phys. J. B , 20, 217–233. Mézard, M. and Parisi, G. (2003). The cavity method at zero temperature. J. Stat. Phys., 111, 1–34. Mézard, M. and Zecchina, R. (2002). The random K-satisfiability problem: From an analytic solution to an efficient algorithm. Phys. Rev. E , 66, 056126. Mézard, M., Parisi, G., and Virasoro, M. A. (1985a). Random free energies in spin glasses. J. Physique Lett., 46, L217–L222. Mézard, M., Parisi, G., and Virasoro, M. A. (1985b). SK model: The replica solution without replicas. Europhys. Lett., 1, 77–82. Mézard, M., Parisi, G., Sourlas, N., Toulouse, G., and Virasoro, M. A. (1985c). Replica symmetry breaking and the nature of the spin glass phase. J. Physique, 45, 843–854. Mézard, M., Parisi, G., and Virasoro, M. A. (1987). Spin Glass Theory and Beyond. World Scientific, Singapore. Mézard, M., Parisi, G., and Zecchina, R. (2003a). Analytic and algorithmic solution of random satisfiability problems. Science, 297, 812–815. Mézard, M., Ricci-Tersenghi, F., and Zecchina, R. (2003b). Two solutions to diluted p-spin models and XORSAT problems. J. Stat. Phys., 111, 505–533. Mézard, M., Mora, T., and Zecchina, R. (2005a). Clustering of solutions in the random satisfiability problem. Phys. Rev. Lett., 94, 197205. Mézard, M., Palassini, M., and Rivoire, O. (2005b). Landscape of solutions in constraint satisfaction problems. Phys. Rev. Lett., 95, 200202. Mézard, M., Tarzia, M., and Toninelli, C. (2007). Statistical physics of group testing. J. Phys. A, 40, 12019–12029. Proc. International Workshop on StatisticalMechanical Informatics, Kyoto. Migliorini, G. and Saad, D. (2006). Finite-connectivity spin-glass phase diagrams and low-density parity check codes. Phys. Rev. E , 73, 026122. Molloy, M. and Reed, B. (1995). A critical point for random graphs with a given

References

degree sequence. Random Struct. Algorithms, 6, 161–180. Monasson, R. (1995). Structural glass transition and the entropy of metastable states. Phys. Rev. Lett., 75, 2847–2850. Monasson, R. (1998). Optimization problems and replica symmetry breaking in finite connectivity spin glasses. J. Phys. A, 31, 513–529. Monasson, R. and Zecchina, R. (1996). Entropy of the K-satisfiability problem. Phys. Rev. Lett., 76, 3881–3885. Monasson, R. and Zecchina, R. (1997). Statistical mechanics of the random Ksatisfiability problem. Phys. Rev. E , 56, 1357–1370. Monasson, R. and Zecchina, R. (1998). Tricritical points in random combinatorics: The (2 + p)-sat case. J. Phys. A, 31, 9209–9217. Monasson, R., Zecchina, R., Kirkpatrick, S., Selman, B., and Troyansky, L. (1999). Determining computational complexity from characteristic phase transitions. Nature, 400, 133–137. Monod, P. and Bouchiat, H. (1982). Equilibrium magnetization of a spin glass: is mean-field theory valid? J. Physique Lett., 43, 45–54. Montanari, A. (2000). Turbo codes: The phase transition. Eur. Phys. J. B, 18, 121–136. Montanari, A. (2001a). Finite size scaling and metastable states of good codes. In Proc. 39th Allerton Conference on Communications, Control and Computing, Monticello, IL. Montanari, A. (2001b). The glassy phase of Gallager codes. Eur. Phys. J. B , 23, 121–136. Montanari, A. (2003). 2RSB population dynamics for spin glasses. Unpublished. Montanari, A. (2005). Tight bounds for LDPC and LDGM codes under MAP decoding. IEEE Trans. Inf. Theory, 51, 3221–3246. Montanari, A. and Ricci-Tersenghi, F. (2003). On the nature of the low-temperature phase in discontinuous mean-field spin glasses. Eur. Phys. J. B , 33, 339–346. Montanari, A. and Ricci-Tersenghi, F. (2004). Cooling-schedule dependence of the dynamics of mean-field glasses. Phys. Rev. B , 70, 134406. Montanari, A. and Semerjian, G. (2005). From large scale rearrangements to mode coupling phenomenology in model glasses. Phys. Rev. Lett., 94, 247201. Montanari, A. and Semerjian, G. (2006a). On the dynamics of the glass transition on Bethe lattices. J. Stat. Phys., 124, 103–189. Montanari, A. and Semerjian, G. (2006b). Rigorous inequalities between length and time scales in glassy systems. J. Stat. Phys., 125, 23–54. Montanari, A. and Shah, D. (2007). Counting good truth assignments for random satisfiability formulae. In Proc. 18th Symposium on Discrete Algorithms, SODA, New Orleans, pp. 1255–1265. Montanari, A. and Sourlas, N. (2000). The statistical mechanics of turbo codes. Eur. Phys. J. B , 18, 107–119. Montanari, A., Parisi, G., and Ricci-Tersenghi, F. (2004). Instability of one-step replica-symmetry breaking in satisfiability problems. J. Phys. A, 37, 2073–2091. Montanari, A., Ricci-Tersenghi, F., and Semerjian, G. (2007). Solving constraint satisfaction problems through belief-propagation-guided decimation. In Proc. 45th

References

Allerton Conference on Communications, Control and Computing, Monticello, IL. Montanari, A., Ricci-Tersenghi, F., and Semerjian, G. (2008). Cluster of solutions and replica symmetry breaking in random k-satisfiability. J. Stat. Mech., 04004. Mooij, J. M. and Kappen, H. K. (2005). On the properties of Bethe aproximation and loopy belief propagation on binary networks. J. Stat. Mech., 11012. Mora, T. and Mézard, M. (2006). Geometrical organization of solutions to random linear Boolean equations. J. Stat. Mech., 10007. Mora, T. and Zdeborov´ a, L. (2007). Random subcubes as a toy model for constraint satisfaction problems. Eprint arXiv:0710.3804. Morita, T. (1979). Variational principle for the distribution function of the effective field for the random Ising model in the Bethe approximation. Physica, 98A, 566– 572. Mossel, E. (2003). Phase transitions in phylogeny. Trans. Am. Math. Soc., 356, 2379–2404. Mossel, E. (2004). Survey: Information flow on trees. In Graphs, Morphisms, and Statistical Physics, pp. 155–170. American Mathematical Society, Providence, RI. Mossel, E. and Peres, Y. (2003). Information flow on trees. Ann. Appl. Probab., 13, 817–844. Mottishaw, P. and De Dominicis, C. (1987). On the stability of randomly frustrated systems with finite connectivity. J. Phys. A, 20, L375–L379. Mulet, R., Pagnani, A., Weigt, M., and Zecchina, R. (2002). Coloring random graphs. Phys. Rev. Lett., 89, 268701. Nair, C., Prabhakar, B., and Sharma, M. (2003). Proofs of the Parisi and Coppersmith–Sorkin conjectures for the finite random assignment problem. In Proc. 44th IEEE Symposium on Foundations of Computer Science, FOCS, pp. 168–178. Nair, C., Prabhakar, B., and Sharma, M. (2006). Proofs of the Parisi and Coppersmith-Sorkin random assignment conjectures. Random Struct. Algorithms, 27, 413–444. Nakamura, K., Kabashima, Y., and Saad, D. (2001). Statistical mechanics of lowdensity parity check error-correcting codes over Galois fields. Europhys. Lett., 56, 610–616. Nakanishi, K. (1981). Two- and three-spin cluster theory of spin glasses. Phys. Rev. B , 23, 3514–3522. Neri, I., Skantzos, N. S., and Bollé, D. (2008). Gallager error correcting codes for binary asymmetric channels. Eprint arXiv:0803.2580. Nishimori, H. (2001). Statistical Physics of Spin Glasses and Information Processing. Oxford University Press, Oxford. Norris, J. R. (1997). Markov Chains. Cambridge University Press, Cambridge. Panchenko, D. and Talagrand, M. (2004). Bounds for diluted mean-field spin glass models. Probab. Theor. Relat. Fields, 130, 319–336. Papadimitriou, C. H. (1991). On selecting a satisfying truth assignment. In Proc. 32nd IEEE Symposium on Foundations of Computer Science, FOCS, pp. 163–169. Papadimitriou, C. H. (1994). Computational Complexity. Addison-Wesley, Reading, MA. Papadimitriou, C. H. and Steiglitz, K. (1998). Combinatorial Optimization. Dover

References

Publications, Mineola, NY. Parisi, G. (1979). Toward a mean field theory for spin glasses. Phys. Lett., 73A, 203–205. Parisi, G. (1980a). A sequence of approximated solutions to the SK model for spin glasses. J. Phys. A, 13, L115–L121. Parisi, G. (1980b). The order parameter for spin glasses: A function on the interval [0, 1]. J. Phys. A, 13, 1101–1112. Parisi, G. (1983). Order parameter for spin glasses. Phys. Rev. Lett., 50, 1946–1948. Parisi, G. (1988). Statistical Field Theory. Addison-Wesley, Reading, MA. Parisi, G. (1998). A conjecture on random bipartite matching. Eprint arXiv:cond-mat/9801176. Parisi, G. (2002). On local equilibrium equations for clustering states. Eprint arXiv:cs/0212047v1. Parisi, G. (2003). A backtracking survey propagation algorithm for K-satisfiability. Eprint arxiv:cond-mat/0308510. Pearl, J. (1988). Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, San Francisco. Pitman, J. and Yor, M. (1997). The two-parameter Poisson–Dirichlet distribution derived from a stable subordinator. Ann. Probab., 25, 855–900. Prim, R. C. (1957). Shortest connection networks and some generalizations. Bell Syst. Tech. J., 36, 1389–1401. Pumphrey, S. J. (2001). Solving the satisfiability problem using message passing techniques. Part III Physics Project Report, Department of Physics, University of Cambridge. Reif, F. (1965). Fundamentals of Statistical and Thermal Physics. McGraw-Hill, New York. Ricci-Tersenghi, F., Weigt, M., and Zecchina, R. (2001). Exact solutions for diluted spin glasses and optimization problems. Phys. Rev. E , 63, 026702. Richardson, T. J. (2003). Error floors of LDPC codes. In Proc. 41st Allerton Conference on Communications, Control and Computing, Monticello, IL. Richardson, T. and Urbanke, R. (2001a). An introduction to the analysis of iterative coding systems. In Codes, Systems, and Graphical Models, IMA Volumes in Mathematics and Its Applications, Vol. 123, pp. 1–37. Springer, Berlin. Richardson, T. J. and Urbanke, R. (2001b). The capacity of low-density parity check codes under message-passing decoding. IEEE Trans. Inf. Theory, 47, 599–618. Richardson, T. J. and Urbanke, R. (2001c). Efficient encoding of low-density paritycheck codes. IEEE Trans. Inf. Theory, 47, 638–656. Richardson, T. J. and Urbanke, R. (2008). Modern Coding Theory. Cambridge University Press, Cambridge. Available online at http://lthcwww.epfl .ch/mct/index.php. Richardson, T. J., Shokrollahi, A., and Urbanke, R. (2001). Design of capacityapproaching irregular low-density parity-check codes. IEEE Trans. Inf. Theory, 47, 610–637. Rivoire, O., Biroli, G., Martin, O. C., and Mézard, M. (2003). Glass models on Bethe lattices. Eur. Phys. J. B , 37, 55–78.

References

Ruelle, D. (1987). A mathematical reformulation of Derrida’s REM and GREM. Commun. Math. Phys., 108, 225–239. Ruelle, D. (1999). Statistical Mechanics: Rigorous Results. World Scientific, Singapore. Rujan, P. (1993). Finite temperature error-correcting codes. Phys. Rev. Lett., 70, 2968–2971. Schmidt-Pruzan, J. and Shamir, E. (1985). Component structure in the evolution of random hypergraphs. Combinatorica, 5, 81–94. Sch¨ oning, U. (1999). A probabilistic algorithm for k-SAT and constraint satisfaction problems. In Proc. 40th IEEE Symposium on Foundations of Computer Science, FOCS, New York, pp. 410–414. Sch¨ oning, U. (2002). A probabilistic algorithm for k-SAT based on limited local search and restart. Algorithmica, 32, 615–623. Selman, B. and Kautz, H. A. (1993). Domain independent extensions to gsat: Solving large structural satisfiability problems. In Proc. IJCAI-93, Chambery, France. Selman, B. and Kirkpatrick, S. (1996). Critical behavior in the computational cost of satisfiability testing. Artif. Intell., 81, 273–295. Selman, B., Kautz, H. A., and Cohen, B. (1994). Noise strategies for improving local search. In Proc. AAAI-94, Seattle, WA. Selman, B., Mitchell, D., and Levesque, H. (1996). Generating hard satisfiability problems. Artif. Intell., 81, 17–29. Shannon, C. E. (1948). A mathematical theory of communication. Bell Syst. Tech. J., 27, 379–423, 623–655. Available online at http://cm.bell-labs.com /cm/ms/what/shannonday/paper.html. Sherrington, D. and Kirkpatrick, S. (1975). Solvable model of a spin glass. Phys. Rev. Lett., 35, 1792–1796. Shohat, J. A. and Tamarkin, J. D. (1943). The Problem of Moments, Mathematical Surveys No. 1. American Mathematical Society, New York. Sinclair, A. (1997). Convergence rates for Monte Carlo experiments. In Numerical Methods for Polymeric Systems, IMA Volumes in Mathematics and Its Applications, Vol. 102,pp. 1–18. Springer, Berlin. Sipser, M. and Spielman, D. A. (1996). Expander codes. IEEE Trans. Inf. Theory, 42, 1710–1722. Sokal, A. D. (1996). Monte Carlo methods in statistical mechanics: foundations and new algorithms. In Lectures at the Cargèse Summer School ‘Functional Integration: Basics and Applications’. Available online at http://www.math.nyu.edu/faculty/goodman/teaching/Monte Carlo/ monte carlo.html. Sourlas, N. (1989). Spin-glass models as error-correcting codes. Nature, 339, 693. Spielman, D. A. (1997). The complexity of error-correcting codes. In Fundamentals of Computation Theory (eds Chlebus, B. S. and Craja, L.), Lecture Notes in Computer Science, Vol. 1279, pp. 67–84. Springer, Berlin. Sportiello, A. (2004). Personal communication. Stepanov, M. G., Chernyak, V. Y., Chertkov, M., and Vasic, B. (2005). Diagnosis of weaknesses in modern error correction codes: A physics approach. Phys.

References

Rev. Lett., 95, 228701–228704. Svenson, P. and Nordahl, M. G. (1999). Relaxation in graph coloring and satisfiability problems. Phys. Rev. E , 59, 3983–3999. Talagrand, M. (2000). Rigorous low temperature results for the mean field p-spin interaction model. Probab. Theor. Relat. Fields, 117, 303–360. Talagrand, M. (2003). Spin Glasses: A Challenge for Mathematicians. Springer, Berlin. Tanaka, T. (2002). A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors. IEEE Trans. Inf. Theory, 48, 2888–2910. Tanner, R.M. (1981). A recursive approach to low complexity codes. IEEE Trans. Inf. Theory, 27, 533–547. Thouless, D. J. (1986). Spin-glass on a Bethe lattice. Phys. Rev. Lett., 6, 1082–1085. Thouless, D. J., Anderson, P. W., and Palmer, R. G. (1977). Solution of ‘Solvable model of a spin glass’. Phil. Mag., 35, 593–601. Toulouse, G. (1977). Theory of the frustration effect in spin glasses: I. Comm. Phys., 2, 115–119. Viana, L. and Bray, A. J. (1985). Phase diagrams for dilute spin glasses. J. Phys. C , 18, 3037–3051. Wainwright, M. J. and Maneva, E. (2005). Lossy source encoding via message-passing and decimation over generalized codewords of LDGM codes. In IEEE International Symposium on Information Theory, Adelaide, pp. 1493–1497. Wainwright, M. J., Jaakkola, T. S., and Willsky, A. S. (2005a). A new class of upper bounds on the log partition function. IEEE Trans. Inf. Theory, 51(7), 2313–2335. Wainwright, M. J., Jaakkola, T. S., and Willsky, A. S. (2005b). MAP estimation via agreement on trees: Message-passing and linear programming. IEEE Trans. Inf. Theory, 51(11), 3697–3717. Walkup, D. W. (1979). On the expected value of a random assignment problem. SIAM J. Comput., 8, 440–442. Wang, C. C., Kulkarni, S. R., and Poor, H. V. (2006). Exhausting errorprone patterns in ldpc codes. IEEE Trans. Inf. Theory, submitted. Eprint http://arxiv.org/abs/cs/0609046 W¨ astlund, J. (2008). An easy proof of the ζ(2) limit in the random assignment problem. Electron. Commmun. Probab., 13, 258–265. Weigt, M. and Hartmann, A. K. (2000). Number of guards needed by a museum: A phase transition in vertex covering of random graphs. Phys. Rev. Lett, 84, 6118– 6121. Wong, K. Y. M. and Sherrington, D. (1988). Intensively connected spin glasses: Towards a replica symmetry breaking solution of the ground state. J. Phys. A, 21, L459–L466. Wormald, N. C. (1999). Models of random regular graphs. In Surveys in Combinatorics, 1999 (eds. J. D. Lamb and D. A. Preece), London Mathematical Society Lecture Note Series, pp. 239–298. Cambridge University Press, Cambridge. Yakir, B. (1996). The differencing algorithm LDM for partitioning: A proof of a conjecture of Karmarkar and Karp. Math. Oper. Res., 21, 85. Yedidia, J. S., Freeman, W. T., and Weiss, Y. (2001). Generalized belief propagation.

References

In Advances in Neural Information Processing Systems, NIPS, pp. 689–695. MIT Press, Cambridge, MA. Yedidia, J. S., Freeman, W. T., and Weiss, Y. (2005). Constructing free energy approximations and generalized belief propagation algorithms. IEEE Trans. Inf. Theory, 51, 2282–2313. Yuille, A. L. (2002). CCCP algorithms to minimize the Bethe and Kikuchi free energies: Convergent alternatives to belief propagation. Neural Comput., 14, 691– 1722. Zhou, H. (2003). Vertex cover problem studied by cavity method: Analytics and population dynamics. Eur. Phys. J. B , 32, 265–270.

Index absorbing set, 514 annealed average, 102 annealing, 86 antiferromagnet, 242 antiferromagnetic, 44, 44 , 46, 315 aperiodicity, see Markov chain Arrhenius law, 282 assignment problem, 53 , 355–379 ζ(2), 362 BP equations, 358 min-sum algorithm, 360 multi-index, 376 asymptotic equipartition property, 72 AT line, 168 autocorrelation, 84 auxiliary graphical model, 434 average, 3

bit error rate (BER), 110 , 117, 273, 331, 336–338 bit-flipping algorithm, 236 block error, 19 , 62, 110, 118, 120, 126, 231, 233, 237, 495 block length, 19 Boltzmann average, 25 , 118 Boltzmann distribution, 25 , 28 , 73, 78, 82, 86, 94, 127, 179 boundary conditions, periodic, 80 , 276, 310 BP, see belief propagation BSC, 17 , 120 decoding threshold, 235, 236, 338, 351 dynamical phase transition, 509, 510, 512 finite-temperature decoding, 118–120 LDPC code, 231–239, 337, 339, 350 MAP decoding, 113–117 sphere packing, 125–126

backbone, 413 balanced minimum cut problem, 286 Bayes rule, 10 , 108 Bayesian network, 269 , 287 BEC, 17 , 344, 351 bit error probability, 513 BP decoding, 342–347 decoding thresholds, 351 metastable states, 500–506 belief propagation (BP), 297 , 291–326, 536 equations, 297 pairwise models, 304 exact on trees, 299 threshold, 338–352 beliefs, 311 Bernoulli process, 5, 543 Bethe equations, see cavity method Bethe free energy, see Bethe free entropy Bethe free entropy, 303 , 312 1RSB, 439, 455 function of BP messages, 302 MAP decoding, 347 pairwise models, 305 stationary points, 314 Bethe measure, 430 , 432, 433, 448, 461 extremal, 433, 459 Bethe state, see Bethe measure Bhattacharya parameter, 232 binary memoryless symmetric channel (BMS), 327 bit, 5

capacity, see channel, capacity capacity-achieving ensembles, 342 Carleman’s theorem, 152 cavity method, 189, 358, 422, 429–466 1RSB, 424, 439 , 429–466 energetic, 307 , 323, 453, 455, 459, 474, 485 equations, 322 , 376, 390 RS, 321, 384 Cayley formula, 191 certificate, 57 chain rule, 8, 10 , 11, 12 channel, 16 binary erasure, see BEC binary symmetric, see BSC capacity, 18 memoryless, 17 Z, 18 channel coding, 16 , 19, 20 check nodes, 220 Chernoff bound, 232 chromatic number, 491 clause, 53 clause density, 207 clique, see graph, complete, 177, 282 cloning method, 260 clustering transition, see dynamical glass transition clusters, 104, 163, 254 , 255, 257, 259, 413, 421, 422, 474, 485 code ensemble, 108

Index error-correcting, 19 instantaneous, 13 rate, see rate repetition, 20 source, 12 , 13, 14 uniquely decodable, 13 codebook, 19 , 107 codeword, 12 , 19 colouring, 54 , 180, 488 combinatorial optimization, 51 compatibility function, 175 complete algorithm, 201 complexity (statistical physics), 258 , 259, 433, 434, 436, 446 1RSB expression, 439 complexity class, 57 NP, 57 NP-complete, 58 NP-hard, 59 polynomial, 55 , 57 compression, 12, 14, 16 computation tree, 368 computational complexity, 51 concentration, 94 condensation, 100 , 101, 105, 153, 471, 473, 487 configuration, 51 configurational entropy, see complexity (statistical physics) conjunctive normal form, 58 , 198 connected, 48 Cook’s theorem, 58 , 62 Coppersmith–Sorkin formula, 372 core (2-core), 343 , 413–415 correlation decay, 520 correlation function, 39 , 520 connected, 32 point-to-set, 522 correlation length, 40 cost function, 51 crossover, 17 Curie–Weiss model, 40 , 75, 77, 85, 264 cycle, 48 data-processing inequality, see inequality decimation BP-guided, 472 SP-guided, 481 decision problem, 51 decoder, 19 decoding finite-temperature, 118 symbol MAP, see MAP typical-pairs, 494 word MAP, see MAP degree, 48 in multigraphs, 52 degree profile, 181 , 182, 220, 224 edge-perspective, 192 ∆-stable configurations, 497

density evolution, 321 , 323 1RSB, 440, 442 2RSB, 529 assignment, 361 BP decoding, 329, 343, 503 XORSAT, 409 detailed balance, 27 , 81, 82, 283, 460, 520 distance in a graph, 192 minimum, 123 distance enumerator, 110 , 111, 115, 123, 222 DPLL algorithm, 201 , 202, 203 resolution, 204 dynamical glass transition, 258 , 460, 473, 487, 509, 535 Edwards–Anderson model, 44 , 46, 179, 244 Edwards–Anderson order parameter, 251 , 265 encoding, 19 energy, 24 density, 33 gap, 29 internal, 29 level, 93 entropy, 5 canonical, 29 conditional, 10, 10 , 11, 12 density, 33

-coupling method, 256 , 524 equilibration, 84 erasure, see BEC error floor, 339 , 513 Eulerian circuit, 52 evaluation problem, 51 event, 3 expander, 238 expectation value, 3 expurgation, 124 , 230 extremality, 433 , 523 extreme value statistics, 163 Fano’s inequality, see inequality ferromagnet, 35 , 43, 242 , 246, 250, 271, 381–400 ferromagnetic, 44, 44 , 46, 61, 80, 168, 251, 275, 286, 292, 316 finite-size scaling, 138 , 218, 416, 427 first-moment method, 96 fluctuation–dissipation theorem, 33 free entropy Bethe, see Bethe free entropy free energy, 28 Bethe, see Bethe free energy density, 33 Gibbs, 78, 78 , 79, 80 free entropy, 28 , 29, 34, 297 density, 33, 87 Friedgut’s theorem, 210 FRSB, see replica, full symmetry breaking

Index frustration, 45 , 46, 245, 245 fully connected model, 155 , 244, 465 function node, 175 G¨ artner–Ellis theorem, 76 gauge transformation, 246 generator matrix, 126 , 220 giant component, see graph Gibbs variational principle, 78 , 79, 80 Gilbert–Varshamov bound, 123 , 124 Gilbert–Varshamov distance, 111 glass, see phase -glass glass transition, 253, 253 , 258, 493, 538 Glauber dynamics, see heat bath algorithm graph, 48 complete, 48 component, 186 cyclic number of a, 189 degree-constrained factor, 182 factor, 175 , 173–195 giant component, 187–190 partitioning, 286 random k-factor, 181 random ensembles, 180–194 random regular, 182 unicyclic, 187 weighted, 48 graphical model, 175 greedy algorithm, 86 GREM, 527 Griffiths’ inequality, see inequality ground state, 28 , 30, 37, 60, 61, 103, 164 Gumbel distribution, 164 Hamiltonian cycle, 52 Hammersley–Clifford theorem, 177 , 194 Hamming bound, 123 Hamming code, 178, 220 Hamming distance, 110 , 163, 205, 216 Hamming space, 110 , 123 Hamming sphere, 123 , 125 hard decision, 328 heat bath algorithm, 82 , 250, 259, 273, 275, 280, 284, 287, 460, 461, 474, 498, 500, 536 hyperedge, 181 , 195, 376, 377 hypergraph, 180 , 194 hyperloop, 406 ideal gas, 243 inclusion principle (in assignment), 361 incomplete search, 204 independent identically distributed (i.i.d.), 8 indicator function, 4 induced subgraph, 430 inequality data-processing, 11 Fano, 11 , 347 Griffiths, 391 Jensen, 4 , 7, 11

Kraft, 14 , 15 Markov, 96 , 489 information, mutual, 10, 10 , 11, 18 insertion, 17 instance, 51, 93 irreducibility, see Markov chain Ising model, 35, 35 , 36, 46, 74, 80, 241, 243, 275, 276, 284, 286, 292, 310, 386, 389 Ising spin, 25 , 75, 155, 241 Jensen’s inequality, see inequality k-body interaction, 24 KL divergence, see Kullback–Leibler divergence Kraft’s inequality, see inequality Kronecker symbol, 88 Kullback–Leibler (KL) divergence, 7 , 78 large deviations, 65–89 principle, 73 rate function, 73 LDPC codes, 219–240, 327–353, 493 BSC thresholds, 235, 338, 350, 351 rate, 230 weight enumerator, 224, 226 leading exponential order, 34 leaf removal, see peeling algorithm likelihood, maximum, 109 , 231, 239, 495 linear codes, 220 random, 126 , 219 local stability, 167 , 252, 339, 378, 386, 391, 473, 522, 529, 534 in coding, 340 of 1RSB, 531 locally consistent marginals, 311 log-likelihood, 121, 232 , 293 , 328, 330, 332, 333, 348, 358, 382, 494, 501 loop, 193 magnetization, 25 average, 37 instantaneous, 41 spontaneous, 37 MAP, 108 , 109, 113, 118, 127 symbol, 109 , 114 threshold, 234 , 236, 348, 351, 505 word, 109 , 113 marginal, 8 Markov chain, 9, 9 , 10, 11, 74, 81 aperiodicity, 81 , 82, 83, 500 irreducibility, 81 , 82, 83, 86, 87, 500 stationarity, 81, 81 , 82–84, 86, 282, 283 with memory, 177 Markov inequality, see inequality Markov property, global, 175 , 177 max-marginal, 305 max-product algorithm, 306, 306 , 308 MAX-SAT problem, 198, 279

Index

maximum cut problem, 61 MCMC, see Monte Carlo method mean-field approximation, 80, 80 , 313 mean-field model, 155 , 241, 244, 262, 381, 522 message passing, 316 , 317, 453, 536 auxiliary model, 436 messages (set of), 430 Metropolis algorithm, 82 min-sum algorithm, 307 , 309, 323, 360, 366, 368, 369, 453, 456 minimum distance, 229, 273 minimum spanning tree, 48 mixing time, 84 mode (of a distribution), 305 moment-generating function, 73 Monte Carlo method, 81 , 82, 83, 86, 87, 89, 257, 272–287, 500, 505, 535 Monte Carlo sweep, 274 multigraph, 52 nat, 5 near-codeword, see trapping set neighbourhood (in a graph), 192 directed, 319 Nishimori temperature, 247 non-interacting, 24 , 156 non-reversing path, 299 NP, see complexity class NP-complete, see complexity class NP-hard, see complexity class number partitioning, 54 , 131–144 unconstrained, 132 observable, 24 , 25, 28 optimal, 51 optimization problem, 51 overlap, 162 , 252–255, 257, 258, 263, 462 distribution, 162 , 252, 254, 256, 258, 523 matrix, 147 , 157 p-spin model, see spin glass paramagnetic, see phase, paramagnetic Parisi 1RSB parameter, 434 Parisi formula (assignment), 372 Parisi hierarchical RSB scheme, 161 parity check matrix, 220 participation ratio, 100 partition function, 25 path, 48 peeling algorithm decoding, 343 XORSAT, 412 percolation, 189 perfect partition, 131 phase ferromagnetic, 37 glass, 100, 163, 241, 250 , 252, 253, 257, 394 paramagnetic, 37 , 385

transition, 33 first-order, 33 second-order, 33 physical degradation, 334 Poisson point process, 165 Poisson random variable, 544 Poisson–Dirichlet process, 165 , 166, 462, 524 polynomially reducible, 56 population dynamics, 323 , 394, 442, 534 1RSB, 442 , 464, 512 potential method, 257 Potts spin, 26 probability density function (pdf), 4 probability distribution, 3 pure states, 518 , 519 quench, 86 quenched average, 102 random code ensemble, see RCE random cost model (RCM), 136 random energy model, see REM random graphical model, 318 random variable continuous, 3 discrete, 3 independent, 7 sequence, 8 rate code rate, 19 , 20, 111, 115, 120, 220, 221, 230, 339 design, 221 , 230 growth, 111 , 223 rate function, see large deviations rate of an exponential random variable, 371 RCE, 107 , 108, 110, 495 reconstructibility, see extremality recursive distributional equations, see density evolution relaxation time, 259, 277, 279, 283 , 520 REM, 93–106, 163, 262 replica solution, 145 replica full symmetry breaking, 161 , 163, 253, 399, 463, 535 method, 146 , 145–169, 253, 261, 262 symmetry, see RS symmetry breaking, see RSB replicon, 168 resolution proof, 203 reversibility, see detailed balance RS, 148 , 158, 321, 322, 381, 396, 432, 523 RSB, 162, 253, 396 1RSB, 150, 160, 429–466 cavity equations, 434 discontinuous, 253 dynamical, 432 , 459, 461, 487 static, 432, 450, 459, 462, 487 higher levels, 524 sample, 93

Index Sanov’s theorem, 67 , 87 SAT Easy-SAT, 421 Hard-SAT, 421 satisfiability, 53 , 58, 197–218, 303, 525 K-satisfiability, 54 , 198 random, 207 , 467–492 threshold K-SAT conjecture, 209 bounds, 217 table, 481 search tree, 133 self-averaging, 94 sequential importance sampling, 304 Shannon, 14, 16, 21, 128, 231, 236, 338, 342, 351 channel coding theorem, 20, 120, 121, 126, 236 code, 15, 15 , 16 entropy, 29 random code ensemble, see RCE source coding theorem, 14 source–channel separation, 19 Sherrington–Kirkpatrick model, 156 , 167, 168, 262, 395 soft decision, 328 source coding, 12 , 15, 16, 21 SP, see survey propagation spectral gap, see relaxation time sphere packing, 123–126 spin glass, 44 , 179, 241–266, 381, 391–403 p-spin glass, 155, 155 , 167, 404, 407 diluted, 244 maximum cut, 61 SPY, see survey propagation stationarity, see Markov chain stopping set, see core subcube, 103 sum–product algorithm, see belief propagation surface tension, 285 survey propagation, 425, 453 , 455, 474, 536 decimation, 482 marginal, 482 SPY, 453, 455 susceptibility, 39 ferromagnetic, 251 non-linear, see susceptibility, spin glass spin glass, 252 , 392

symmetric random variable (in coding), 331 syndrome, 235 temperature, 25 critical, 37 high, 28 , 29 low, 28 , 29 thermodynamic limit, 33 threshold 2-SAT, 209 BP, see belief propagation condensation, 487 energy, 485 LDPC MAP decoding, see MAP random graph, 187 SAT–UNSAT, 479 satisfiability (bounds), 217 satisfiability (conjecture), 210 XORSAT, 415 transfer matrix, 37 transition probability, 9 trapping set, 514 travelling salesman problem, 52 tree, 48 , 186, 187, 190, 191 binary, 14 ensemble, 192 rooted, 190 tree-graphical model, 296 truth assignment, 198 type (of a sequence), 67 , 71, 87, 122, 128 typical sequence, 72 typical-pairs, see decoding, typical-pairs ultrametricity, 163 , 166, 527 unfrustrated, 245 unit clause propagation, 199, 199 , 214, 309 UNSAT certificate, 202 variable node, 175 variation distance, 84 warning propagation, 309 , 456, 457, 478 waterfall region, 514 weight enumerator, 222 , 222–231 XOR satisfiability, 404 , 403–427 ensembles, 407 threshold, 417

Information, Physics, and Computation (Oxford Graduate Texts)

Information, Physics, and Computation (Oxford Graduate Texts)

Supersymmetry (Oxford Graduate Texts)

Supersymmetry (Oxford Graduate Texts)