Advances in Imaging and Electron Physics, Volume 150 (Advances in Imaging and Electron Physics)

ADVANCES IN IMAGING AND ELECTRON PHYSICS VOLUME 150 EDITOR-IN-CHIEF PETER W. HAWKES CEMES-CNRS Toulouse, France HON...

Author: Peter W. Hawkes

285 downloads 1182 Views 4MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

ADVANCES IN IMAGING AND ELECTRON PHYSICS VOLUME 150

EDITOR-IN-CHIEF

PETER W. HAWKES CEMES-CNRS Toulouse, France

HONORARY ASSOCIATE EDITORS

TOM MULVEY BENJAMIN KAZAN

Advances in

Imaging and Electron Physics

E DITED BY

PETER W. HAWKES CEMES-CNRS Toulouse, France

VOLUME 150

AMSTERDAM • BOSTON • HEIDELBERG • LONDON NEW YORK • OXFORD • PARIS • SAN DIEGO SAN FRANCISCO • SINGAPORE • SYDNEY • TOKYO Academic Press is an imprint of Elsevier

Academic Press is an imprint of Elsevier 525 B Street, Suite 1900, San Diego, California 92101-4495, USA 84 Theobald’s Road, London WC1X 8RR, UK ∞ This book is printed on acid-free paper.

Copyright © 2008, Elsevier Inc. All Rights Reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the Publisher. The appearance of the code at the bottom of the first page of a chapter in this book indicates the Publisher’s consent that copies of the chapter may be made for personal or internal use of specific clients. This consent is given on the condition, however, that the copier pay the stated per copy fee through the Copyright Clearance Center, Inc. (www.copyright.com), for copying beyond that permitted by Sections 107 or 108 of the U.S. Copyright Law. This consent does not extend to other kinds of copying, such as copying for general distribution, for advertising or promotional purposes, for creating new collective works, or for resale. Copy fees for pre-2008 chapters are as shown on the title pages. If no fee code appears on the title page, the copy fee is the same as for current chapters. 1076-5670/2008 $35.00 Permissions may be sought directly from Elsevier’s Science & Technology Rights Department in Oxford, UK: phone: (+44) 1865 843830, fax: (+44) 1865 853333, E-mail: [email protected]. You may also complete your request on-line via the Elsevier homepage (http://elsevier.com), by selecting “Support & Contact” then “Copyright and Permission” and then “Obtaining Permissions.” For information on all Elsevier Academic Press publications visit our Web site at www.books.elsevier.com ISBN-13: 978-0-12-374217-9

PRINTED IN THE UNITED STATES OF AMERICA 08 09 10 11 9 8 7 6 5 4 3 2 1

CONTENTS

C ONTRIBUTORS . . . . . . . . . . . . . . . . . . . . . . . . . . . vii P REFACE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix F UTURE C ONTRIBUTIONS . . . . . . . . . . . . . . . . . . . . . . xiii

On Some Iterative Concepts for Image Restoration I NGRID DAUBECHIES , G ERD T ESCHKE AND L UMINITA V ESE I. Introduction . . . . . . . . . . . . . . . . . . . . . . . . II. Simultaneous Decomposition, Deblurring, and Denoising of Images by Means of Wavelets . . . . . . . . . . . . . . . . III. Vector-Valued Regimes and Mixed Constraints . . . . . . . IV. Image Restoration with General Convex Constraints . . . . V. Hybrid Wavelet–PDE Image Restoration Schemes . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . .

. . . . .

2 3 21 30 42 48

Significant Advances in Scanning Electron Microscopes (1965–2007) R.F.W. P EASE I. II. III. IV. V. VI. VII. VIII. IX.

Introduction . . . . . . . . . . Brief Review of SEM Operation Contrast . . . . . . . . . . . . Improvements since 1965 . . . . Tabletop SEM . . . . . . . . . Aberration Correction . . . . . Digital Image Processing . . . . Ultrahigh Vacuum . . . . . . . Future Developments . . . . . . References . . . . . . . . . . . v

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

53 55 58 61 75 75 79 81 81 83

vi

CONTENTS

Ptychography and Related Diffractive Imaging Methods J.M. RODENBURG I. II. III. IV. V.

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . Ptychography in Context . . . . . . . . . . . . . . . . . . . Coordinates, Nomenclature, and Scattering Approximations . . The Variants: Data, Data Processing, and Experimental Results Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . .

87 88 119 138 178 180

. . . . . . . . . .

185 188 191 196 204 206 209 212 215 215

Advances in Mathematical Morphology: Segmentation J EAN S ERRA I. II. III. IV. V. VI. VII. VIII. IX.

Introduction . . . . . . . . . . . . . . . . . Criteria, Partitions, and Segmentation . . . . . Connective Segmentation . . . . . . . . . . . Examples of Connective Segmentations . . . . Partial Connections and Mixed Segmentations Iterated Jumps and Color Image Segmentation Connected Operators . . . . . . . . . . . . . Hierarchies and Connected Operators . . . . . Conclusion . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

C UMULATIVE AUTHOR I NDEX . . . . . . . . . . . . . . . . . . . . 221 C ONTENTS OF VOLUMES 100–149 . . . . . . . . . . . . . . . . . 241 S UBJECT I NDEX . . . . . . . . . . . . . . . . . . . . . . . . . . . 257

CONTRIBUTORS

Numbers in parentheses indicate the pages on which the authors’ contributions begin.

I NGRID DAUBECHIES (1), Princeton University, PACM, Washington Road, Princeton, NJ 08544-1000, USA R.F.W. P EASE (53), Department of Electrical Engineering, Stanford University, Stanford, CA 94305, USA J.M. RODENBURG (87), Department of Electronic and Electrical Engineering, University of Sheffield, Mappin Street, S1 3JD, UK J EAN S ERRA (185), Laboratoire A2SI, ESIEE, University of Paris-est, BP 99 93162 Noisy-le-Grand cedex, France G ERD T ESCHKE (1), Konrad-Zuse Institute Berlin, Takustr. 7, D-14195 Berlin-Dahlem, Germany L UMINITA V ESE (1), Department of Mathematics, UCLA, 405 Hilgard Avenue, Los Angeles, CA 90095-1555, USA

vii

This page intentionally left blank

PREFACE

The first volume of this series appeared in 1948 as Advances in Electronics, soon to be renamed Advances in Electronics and Electron Physics from volume 6 to volume 90. The founder editor, Ladislaus (Bill) Marton, remained in charge until his death in 1979, after which his wife Claire continued until she too died, in 1981; my first volume was 59 (1982). Marton is sure of a permanent place in the history of science for in the very early days of electron microscopy, he read the pioneering papers of Ernst Ruska and his colleagues in Berlin and decided to build such an instrument himself. He was in the Free University of Brussels at the time and his first attempts are recorded in the pages of the Bulletin de l’Académie Royale de Belgique, as we might expect, but also in Nature and Physical Review; his early micrograph of a leaf of the sundew Drosera intermedia is widely reproduced and recognized as the first electron micrograph of a stained biological specimen, though admittedly only a proof-of-principle. A biographical article on him can be found in Supplement 16 of these Advances (The Beginnings of Electron Microscopy) and his Early History of the Electron Microscope (San Francisco Press, 1968 and 1994) revives those first efforts vividly. Over the years he attracted many distinguished contributors. The first volume included articles by A. Rose, to whom we owe a much-used criterion for visibility; R.G.E. Hutter on electron deflection, with a very full account of the theory, the first in English; and M. Stanley Livingston on particle accelerators. In Volume 2 we meet Hilary Moss (who later wrote a Supplement) on progress in cathoderay tubes; Pierre Grivet on electron lenses – his memoirs are to be found in Supplement 16; and G. Liebmann on numerical methods in electron optics, the standard reference for many years until the finite-element method arrived on the scene. Volume three contains articles on magnetrons by L. Brillouin and on network synthesis by G.A. Guillemin while Volume 4 (1952) not only includes H.S.W. Massey on electron scattering in solids but also an article by C.V.L. Smith on electronic digital computers! R.Q. Twiss writes about magnetrons in Volume 5, long before being indissolubly linked with HanburyBrown; this volume also has an article on color television by C.J. Hirsch. Two other series of Advances published by Academic Press, Advances in Image Pickup and Display and Advances in Optical and Electron Microscopy were amalgamated with AEEP, which in 1996 became Advances in Imaging and Electron Physics, thus marking the increasing number of articles ix

x

PREFACE

on image processing; the editors of those series appear here as honorary associate editors. In the past, the series published both regular volumes and supplements, of which 25 appeared. Several are mentioned elsewhere in this preface; among the most successful were Supplement 22 by C.A. Brau on Free-electron Lasers, of which a paperback was produced and Coulomb Interactions in Particle Beams by G.H. Jansen (Supplement 21, 1990). Much of the controversial work of Henning Harmuth has appeared in supplements or regular volumes. The three-volume Supplement 13 edited by Albert Septier on Applied Charged Particle Optics was also heavily used. My own favorite is Supplement 16, already cited, for which Ernst Ruska wrote the Foreword (and of course Supplement 7). Publication of this 150th volume of the series is thus an event to be celebrated and, to mark the occasion, I have brought together leaders of some of the main themes of past and – I hope – future volumes: electron microscopy of course, since Ladislaus Marton was one of the pioneers; mathematical morphology, which has often appeared in our page and also fills a supplement, so often cited that it usually appears just as “Academic Press, 1994” (H.J.A.M. Heijmans, Morphological Image Operators, Supplement 25, 1994) with no mention of the Advances; ptychography, a highly original approach to the phase problem, the latter also the subject of a much cited Supplement (W.O. Saxton, ‘Computer Techniques for Image Processing in Electron Microscopy,’ Supplement 10, 1978); and wavelets, which have become a subject in their own right, not just a tool in image processing. Wavelets are discussed by Ingrid Daubechies, with Gerd Tesche and Luminita Vese as co-authors. Doyenne of the wavelet community, Ingrid Daubechies lists 18 entries in her homepage under ‘Awards and Honors’ including four doctorates honoris causa. The topic treated here is image restoration, a form of inverse problem to which wavelets offer a powerful iterative solution. Fabian Pease, no less distinguished in a very different domain, is particularly welcome here as a link with the past. The scanning electron microscope, successfully developed in the Cambridge University Engineering Department under the direction of C.W. (later Sir Charles) Oatley, was eventually commercialized by the Cambridge Instrument Company in 1965; and also in 1965, the first full account of the instrument appeared in our pages (vol. 21, 1965, pp. 181–247), written by C.W. Oatley, W.C. Nixon and R.F.W. Pease, of whom the latter (the only survivor of the trio) provides an account of the transformation of scanning electron microscopy since those early days. Oatley appears with two other former students of his, Dennis McMullan and Kenneth C.A. Smith, in Supplement 16 and a whole volume is devoted to his role in the development of this type of microscope (volume 133, 2004, the centenary of his birth).

PREFACE

xi

The word ‘ptychography’ was coined by the late Walter Hoppe (Professor in Munich), but his ideas were ahead of their time and it was only with the work of John Rodenburg, catalyzed by the late Richard Bates, that the immense potential of this approach became apparent. This account of the subject, which blends history with the latest developments, is the first truly objective study of the subject. Finally, mathematical morphology is studied by Jean Serra. If Ingrid Daubechies is the doyenne of wavelets, then Jean Serra is the doyen of mathematical morphology. Founder member of the Centre de Morphologie Mathématique in the Ecole des Mines at Fontainebleau, his books and numerous other publications are the essential sources of much ‘morphological’ thinking. Here, he concentrates on image segmentation, an aspect of image processing with a huge literature, in which mathematical morphology is playing an important role. I invariably end my prefaces to AIEP by thanking the authors for contributing to the series and for their efforts to make their specialized knowledge accessible to a wider public. On this occasion, let me thank not only those who have written for volume 150 but also the huge list of past authors, whose names are recorded in the cumulative author index included here. This work will soon be available on-line from the Elsevier ScienceDirect database. All volumes of Advances in Electronics and Electron Physics and Advances in Imaging and Electron Physics will be included, back to Volume I. Peter Hawkes


FUTURE CONTRIBUTIONS

S. Ando Gradient operators and edge and corner detection P. Batson (special volume on aberration-corrected electron microscopy) Some applications of aberration-corrected electron microscopy C. Beeli Structure and microscopy of quasicrystals A.B. Bleloch (special volume on aberration-corrected electron microscopy) Aberration correction and the SuperSTEM project C. Bobisch and R. Möller Ballistic electron microscopy C. Bontus and T. Köhler (vol. 151) Reconstruction algorithms for computed tomography G. Borgefors Distance transforms Z. Bouchal Non-diffracting optical beams A. Buchau Boundary element or integral equation methods for static and time-dependent problems B. Buchberger Gröbner bases L. Busin, N. Vandenbroucke and L. Macaire (vol. 151) Color spaces and image segmentation xiii

xiv


T. Cremer Neutron microscopy N. de Jonge and E.C. Heeres Electron emission from carbon nanotubes G.R. Easley and F. Colonna (vol. 151) Generalized discrete Radon transforms and applications to image processing A.X. Falcão The image foresting transform R.G. Forbes Liquid metal ion sources C. Fredembach Eigenregions for image classification A. Gölzhäuser Recent advances in electron holography with point sources D. Greenfield and M. Monastyrskii Selected problems of computational charged particle optics M. Haider (special volume on aberration-corrected electron microscopy) Aberration correction in electron microscopy M.I. Herrera The development of electron microscopy in Spain N.S.T. Hirata (vol. 152) Stack filter design M. Hÿtch, E. Snoeck and F. Houdellier (special volume on aberrationcorrected electron microscopy) Aberration correction in practice J. Isenberg Imaging IR-techniques for the characterization of solar cells K. Ishizuka Contrast transfer and crystal images


xv

A. Jacobo Intracavity type II second-harmonic generation for image processing B. Kabius (special volume on aberration-corrected electron microscopy) Aberration-corrected electron microscopes and the TEAM project S.A. Khan (vol. 152) The Foldy–Wouthuysen transformation in optics L. Kipp Photon sieves A. Kirkland and P.D. Nellist (special volume on aberration-corrected electron microscopy) Aberration-corrected electron micrsocpy G. Kögel Positron microscopy T. Kohashi Spin-polarized scanning electron microscopy O.L. Krivanek (special volume on aberration-corrected electron microscopy) Aberration correction and STEM R. Leitgeb Fourier domain and time domain optical coherence tomography B. Lencová Modern developments in electron optical calculations H. Lichte New developments in electron holography M. Matsuya Calculation of aberration coefficients using Lie algebra S. McVitie Microscopy of magnetic specimens S. Morfu, P. Marquié, B. Nofiélé and D. Ginhac (vol. 152) Nonlinear systems for image processing

xvi


T. Nitta (vol. 152) Back-propagation and complex-valued neurons M.A. O’Keefe Electron image simulation D. Oulton and H. Owens Colorimetric imaging N. Papamarkos and A. Kesidis The inverse Hough transform K.S. Pedersen, A. Lee and M. Nielsen The scale-space properties of natural images S.J. Pennycook (special volume on aberration-corrected electron microscopy) Some applications of aberration-corrected electron microscopy T. Radliˇcka (vol. 151) Lie algebraic methods in charged particle optics V. Randle (vol. 151) Electron back-scatter diffraction E. Rau Energy analysers for electron microscopes E. Recami and M. Zamboni-Rached Localized waves: a brief review H. Rose (special volume on aberration-corrected electron microscopy) The history of aberration correction in electron microscopy G. Schmahl X-ray microscopy R. Shimizu, T. Ikuta and Y. Takai Defocus image modulation processing in real time S. Shirai CRT gun design methods


xvii

T. Soma Focus-deflection systems and their applications J.-L. Starck (vol. 152) Independent component analysis: the sparsity revolution I. Talmon Study of complex fluids by transmission electron microscopy N. Tanaka (special volume on aberration-corrected electron microscopy) Aberration-corrected microscopy in Japan M.E. Testorf and M. Fiddy Imaging from scattered electromagnetic fields, investigations into an unsolved problem N.M. Towghi Ip norm optimal filters E. Twerdowski Defocused acoustic transmission microscopy Y. Uchikawa Electron gun optics K. Urban and J. Mayer (special volume on aberration-corrected electron microscopy) Aberration correction in practice K. Vaeth and G. Rajeswaran Organic light-emitting arrays M. van Droogenbroeck and M. Buckley Anchors in mathematical morphology R. Withers (vol. 152) Disorder, structured diffuse scattering and local crystal chemistry M. Yavor Optics of charged particle analysers Y. Zhu (special volume on aberration-corrected electron microscopy) Some applications of aberration-corrected electron microscopy


ADVANCES IN IMAGING AND ELECTRON PHYSICS, VOL. 150

On Some Iterative Concepts for Image Restoration INGRID DAUBECHIES1 , GERD TESCHKE2 , LUMINITA VESE3 1 Princeton University, PACM, Washington Road, Princeton, NJ 08544-1000, USA 2 Konrad-Zuse Institute Berlin, Takustr. 7, D-14195 Berlin-Dahlem, Germany 3 Department of Mathematics, UCLA, 405 Hilgard Avenue, Los Angeles, CA 90095-1555, USA

I. Introduction . . . . . . . . . . . . . . . . . . . . . . . II. Simultaneous Decomposition, Deblurring, and Denoising of Images by Means of Wavelets A. Wavelet-Based Reformulation of the Variational Problem . . . . . . . . B. Wavelet Preliminaries . . . . . . . . . . . . . . . . . . . C. Iterative Strategy for Image Decomposition . . . . . . . . . . . . D. Redundancy and Adaptivity to Reduce Artifacts . . . . . . . . . . . 1. Translation Invariance by Cycle Spinning . . . . . . . . . . . . 2. Directional Sensitivity by Frequency Projections . . . . . . . . . . 3. Weighted Penalty Functions . . . . . . . . . . . . . . . . E. Image Examples . . . . . . . . . . . . . . . . . . . . III. Vector-Valued Regimes and Mixed Constraints . . . . . . . . . . . . A. Some Remarks on Frame Dictionaries and Sparsity . . . . . . . . . . 1. Frames, Sparsity, and Inverse Problems . . . . . . . . . . . . . 2. Extension to Frame Dictionaries . . . . . . . . . . . . . . . B. Iterative Approach by Surrogate Functionals . . . . . . . . . . . . C. Audio Coding Example . . . . . . . . . . . . . . . . . . 1. A Synthetic Example . . . . . . . . . . . . . . . . . . 2. Real Data: Glockenspiel . . . . . . . . . . . . . . . . . IV. Image Restoration with General Convex Constraints . . . . . . . . . . . A. General Convex Constraint Preliminaries . . . . . . . . . . . . . 1. Reformulation of the Problem . . . . . . . . . . . . . . . B. Solving the Inverse Problem for Convex Penalization . . . . . . . . . . C. Numerical Illustrations . . . . . . . . . . . . . . . . . . 1. Iterative Algorithm for PDE-Based Deblurring and Denoising . . . . . . 2. Comparison of the Iteration Schemes with Besov and BV Constraints . . . . V. Hybrid Wavelet-PDE Image Restoration Schemes . . . . . . . . . . . . A. Combined Wavelet-PDE Scheme in the Presence of Blur . . . . . . . . . 1. Characterization of Minimizers . . . . . . . . . . . . . . . B. Numerical Illustration . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . .

2 3 3 5 7 11 11 13 14 15 21 21 22 23 24 26 27 27 30 30 32 33 37 37 38 42 43 43 46 48

1 ISSN 1076-5670 DOI: 10.1016/S1076-5670(07)00001-8

Copyright 2008, Elsevier Inc. All rights reserved.

2

DAUBECHIES ET AL .

I. I NTRODUCTION This chapter discusses several iterative strategies for solving inverse problems in the context of signal and image processing. We essentially focus on problems for which it is reasonable to assume that the solution has a sparse expansion with respect to a wavelet basis or frame. In each case, we consider a variational formulation of the problem and construct an iteration scheme for which the iterates approximate the solution. To this end, we apply surrogate functionals; the corresponding strategy was shown to converge in norm and to regularize the problem (see Daubechies et al., 2004). We discuss special cases and generalizations. The surrogate functional method in its initial setup as described in Daubechies et al. (2004) amounts to a combination of Landweber’s method and a shrinkage operation, applied in each iteration step. The shrinkage is due to the presence of the 1 -penalization term in the functional. Recent developments in the field of signal and image processing have shown the importance of sparse representations for various tasks in inverse problems (such as compression, denoising, deblurring, decomposition, texture analysis); 1 -constraints select for such sparsity. Here we limit ourselves to illustrating a small number of concrete inverse problems for which we show in detail the variational formulations and the resulting expressions for the iteration. For all cases we discuss convergence and give detailed numerical illustrations. In addition to these case studies, we also present strategies for more general constraints. We begin with the concrete problem of simultaneously denoising, decomposing, and deblurring a given image. The associated variational formulation of the problem contains terms that promote sparsity and smoothness. We show how to transform the problem such that the basic method of Daubechies et al. (2004) applies. In a second example, we discuss a natural extension to vectorvalued inverse problems. Potential applications include seismic or astrophysical data decomposition/reconstruction and color image reconstruction. The illustration presented here contains audio data coding. After these two case studies, we turn to more general formulations. We allow the constraint to be some other positive, homogeneous, and convex functional than the 1 -norm. In the linear case, and under fairly general assumptions on the constraint, we prove that weak convergence of the iterative scheme always holds. In certain cases (i.e., for special families of convex constraints) this weak convergence implies norm convergence. The presented technique covers a wide range of problems. Here we discuss in greater detail image restoration problems in which Besov- or bounded variation (BV) constraints are involved. We close with sketching the design of hybrid wavelet-partial differential equation

ON SOME ITERATIVE CONCEPTS FOR IMAGE RESTORATION

3

(PDE) image restoration schemes (i.e., with variational problems that contain wavelet and BV constraints).

II. S IMULTANEOUS D ECOMPOSITION , D EBLURRING , AND D ENOISING OF I MAGES BY M EANS OF WAVELETS This section is devoted to wavelet-based treatments of variational problems in the field of image processing. In particular, we follow approaches presented in Meyer (2002), Vese and Osher (2003, 2004), and Osher et al. (2003) and discuss a special class of variational functionals that induce a decomposition of images into oscillating and cartoon components and possibly an appropriate “noise” component. In the setting of Vese and Osher (2003) and Osher et al. (2003), the cartoon component of an image is modeled by a BV function; the corresponding incorporation of BV penalty terms in the variational functional leads to PDE schemes that are numerically intensive. By replacing the BV penalty term by a B11 (L1 ) term (which amounts to a slightly stronger constraint on the minimizer) and writing the problem in a wavelet framework, elegant and numerically efficient schemes are obtained with results very similar to those obtained in Osher et al. (2003) and superior to those from Rudin et al. (1992). This approach allows incorporating bounded linear blur operators into the problem so that the minimization leads to a simultaneous decomposition, deblurring, and denoising. A. Wavelet-Based Reformulation of the Variational Problem As mentioned previously, we focus on a special class of variational problems that induce a decomposition of images into “texture” and “cartoon” components. Ideally, the cartoon part is piecewise smooth with possibly abrupt edges and contours; the texture part fills in the smooth regions in the cartoon with typically oscillating features. Inspired by Meyer (2002), Vese and Osher (2003) and Osher et al. (2003) propose to model the cartoon component by the space BV; this induces a penalty term that allows edges and contours in the reconstructed cartoon images, leading to a numerically intensive PDE-based scheme. Our main goal is to provide a computationally thriftier algorithm by using a wavelet-based scheme that solves not the same but a very similar variational problem, in which the BV constraint, which cannot easily be expressed in the wavelet domain, is replaced by a B11 (L1 )-term (i.e., a slightly stricter constraint, since B11 (L1 ) ⊂ BV in two dimensions). Moreover, we can easily incorporate the action of linear bounded blur operators; we also show convergence of the proposed scheme.

4

DAUBECHIES ET AL .

To provide a brief description of the underlying variational problems, recall the methods proposed in Vese and Osher (2003) and Osher et al. (2003). They follow the idea of Meyer (2002), proposed as an improvement on the total variation framework of Rudin et al. (1992). In principle, the models can be understood as a decomposition of an image f into f = u + v, where u represents the cartoon part and v the texture part. In the model by Vese and Osher (2003, 2004), the decomposition is induced by solving inf Gp (u, g1 , g2 ), where 2 Gp (u, g1 , g2 ) = |∇u| + λf − (u + div g)L

u,g1 ,g2

2 (Ω)

+ μ|g|L

p (Ω)

,

Ω

(1) R2 ,

and v = div g = div(g1 , g2 ). The first term is with f ∈ L2 (Ω), Ω ⊂ the total variation of u. If u ∈ L1 and |∇u| is a finite measure on Ω, then u ∈ BV(Ω). This space allows discontinuities; therefore edges and contours generally appear in u. The second term represents the restoration discrepancy; to penalize v, the third term approximates (by taking p finite) the norm of the space G of oscillating functions introduced by Meyer (with p = ∞), which is in some sense dual to BV(Ω). (For details, refer to Meyer, 2002.) Setting p = 2 and g = ∇P + Q, where P is a single-valued function and Q is a divergence-free vector field, it is shown in Osher et al. (2003) that the vpenalty term can be expressed by 1/2 −1 2 |g| ∇(Δ) v = = vH −1 (Ω) . L (Ω) 2

Ω

(The H −1

calculus is allowed as long as we deal with oscillatory texture/noise components that have zero mean.) With these assumptions, the variational problem in Eq. (1) simplifies to solving inf G2 (u, v), where 2 G2 (u, v) = |∇u| + λf − (u + v)L

u,g1 ,g2

2 (Ω)

+ μvH −1 (Ω) .

(2)

Ω

In general, one drawback is that the minimization of Eq. (1) or (2) leads to numerically intensive schemes. Instead of solving Eq. (2) by means of nonlinear partial differential equations and finite difference schemes, we propose a wavelet-based treatment. We are encouraged by the fact that elementary methods based on wavelet shrinkage solve similar extremal problems where BV(Ω) is replaced by


5

the Besov space B11 (L1 (Ω)). Since BV(Ω) can not be simply described in terms of wavelet coefficients, it is not clear that BV(Ω) minimizers can be obtained in this way. Yet, it is shown in Cohen et al. (1999), exploiting B11 (L1 (Ω)) ⊂ BV(Ω) ⊂ B11 (L1 (Ω))-weak, that methods using Haar systems provide near BV(Ω) minimizers. So far no similar result exists for general (in particular, smoother) wavelet systems. We shall nevertheless use wavelets that have more smoothness/vanishing moments than Haar wavelets, because we expect them to be better suited to the modeling of the smooth parts in the cartoon image, though we may not obtain provable “near-best BV minimizers,” we hope to nevertheless be not far off. Limiting ourselves to the case p = 2, replacing BV(Ω) by B11 (L1 (Ω)), and, moreover, extending the range of applicability by incorporating a bounded linear operator K, we derive the following variational problem: inf Ff (v, u), u,v

where

2 Ff (v, u) = f − K(u + v)L2 (Ω) + γ v2H −1 (Ω) + 2α|u|B 1 (L1 (Ω)) . 1

(3) In order to now establish a wavelet-based scheme that solves the latter problem, we first need to recall some basic facts on wavelets. B. Wavelet Preliminaries Let us briefly recall some facts on wavelets that are needed later. Especially important for our approach are the smoothness characterization properties of wavelets. The membership of a function in many different smoothness functional spaces can be determined by examining the decay properties of its wavelets coefficients. For a comprehensive introduction and overview on this topic, refer reader to the abundant literature (e.g., Daubechies, 1992, 1993; Cohen et al., 1992; Dahmen, 1996; DeVore et al., 1992, 1988; Frazier and Jawerth, 1990; and Triebel, 1978). For readers interested more in the gist of the theory than in a more elaborate, mathematically precise description, it suffices to know the following points: • Wavelet expansions provide successive approximations at increasingly finer scales. If a function f is given, and fJ is its approximation at scale 2−J , then the next finer approximation fJ +1 can be written as i i fJ +1 = fJ + f, ψ˜ J,k ψJ,k , i,k

6

DAUBECHIES ET AL . i (x) = 2j ψ i (2j x1 − k1 , 2j x2 − k2 ) are the wavelets used in the where ψj,k i a corresponding dual family. The index i indicates that expansion, and ψj,k in dimensions larger than 1, several wavelet templates are typically used. In two dimensions, there are usually three different wavelets, and i takes the values 1, 2, 3. (Note that the details of the approximation scheme that computes fJ from f depend on the wavelet family under consideration.) If ψ ∈ C s (i.e., ψ has “differentiability” of order s, where s need not to be integer), then f has differentiability of order r < s if and only if f, ψ˜ i ≤ C2−j (r+s) . (4) j,k

For the sake of convenience, we often “bundle” i, j, k into one index λ, and write f, ψ˜ λ simply as fλ . In this case, |λ| stands for j . In this notation, the requirement (4) becomes |fλ | ≤ C2−|λ|(r+s) . • The smoothness of f can be characterized in detail by using several parameters to describe it, such as in Besov spaces. For smoothness r < 1, for instance, we define

1/p f (x + h) − f (x)p dx ω(f ; t)p = sup |h|≤t

(this is an Lp -measured modulus of continuity for f ), and |f |Bqr (Lp (Ω)) =

∞

t −r ω(f ; t)p

q

1/q dt/t

.

0

(Basically, this measures, in a fine q-gained scale, whether ω(f ; t)p decays at least as fast as t r when t → 0.) For instance, if we consider on Ω = (0, 1]2 the function f (x) = x1 + x2 − x1 + x2 , where x = max{n ∈ Z; n ≤ x}, which has a discontinuity along the diagonal x1 + x2 = 1 in the square, then ω(f ; t)1 ∼ C|t| as |t| → 0 and we easily check |f |B 1−ε (L1 (Ω)) < ∞ for all ε > 0. In fact, for this 1

f ∈ B11 (L1 (Ω)) (i.e., with r = 1) as well, but to verify this we need a more sophisticated L1 -measured modulus of continuity. One important link of wavelets to these detailed smoothness spaces is that they provide a good estimate of Besov norms. In particular, in two dimensions,

f ∈ B1s+1 L1 (Ω) ⇐⇒ 2|λ|s |fλ | < ∞; λ


7

for s = 0 this shows that f ∈ B11 (L1 (Ω)) if and only if its coefficients are in 1 . Another special case, again in two dimensions, is p = q = 2; the Besov spaces then reduce to Sobolev spaces: B2s (L2 (Ω)) = W2s , which can on Ω also be easily characterized in terms of Fourier coefficients: 2s 2 f ∈ W2s ⇐⇒ |k1 | + |k2 | fˆk < ∞. k

For these spaces,

f ∈ W2s = B22 L2 (Ω)

⇐⇒

22|λ|s |fλ |2 < ∞.

λ

This holds even for s < 0; in that case, functions with modest W2s norm have large-amplitude, high-frequency oscillations. C. Iterative Strategy for Image Decomposition We aim to find the minimizer of the functional 2 Ff (v, u) = f − K(u + v)L (Ω) + γ v2H −1 (Ω) + 2α|u|B 1 (L1 (Ω)) . (5) 2

1

At first, we may observe the following Lemma II.1. If the null space N (K) of the operator K is trivial, then the variational problem [Eq. (5)] has a unique minimizer. This can be seen as follows:

Ff μ(v, u) + (1 − μ)(v , u ) − μFf (v, u) − (1 − μ)Ff (v , u ) 2

= −μ(1 − μ) K(u − u + v − v )L (Ω) + γ v − v 2H −1 (Ω) 2

+ 2α μu + (1 − μ)u B 1 (L (Ω)) − μ|u|B 1 (L1 (Ω)) 1 1 1 − (1 − μ)|u |B 1 (L1 (Ω)) (6) 1

with 0 < μ < 1. Since the Banach norm is convex, the right-hand side of Eq. (6) is nonpositive (i.e., Ff is convex). Since N (K) = {0}, the term K(u − u + v − v ) can be 0 only if u − u + v − v = 0; moreover, v − v is 0 only if v − v = 0. Hence, Eq. (6) is strictly less than 0 if (v, u) = (v , u ) (i.e., Ff is strictly convex). Conversely, because Ff (v, u) → ∞ as v, u → ∞, Ff must have a minimizer. Solving this problem by means of wavelets requires switching to the sequence space formulation. When K is the identity operator, the problem

8

DAUBECHIES ET AL .

simplifies to 2

−2|λ| 2 fλ − (uλ + vλ ) + γ 2 |vλ | + 2α|uλ | , inf u,v

(7)

λ∈J

where J = {λ = (i, j, k): k ∈ Jj , j ∈ Z, i = 1, 2, 3} is the index set used in the separable setting. The minimization of Eq. (7) is straightforward, since it decouples into easy one-dimensional (1D) minimizations. This results in an explicit shrinkage scheme (presented also in Daubechies and Teschke, 2004): Proposition II.1. Let f be a given function. The functional Eq. (7) is minimized by the parameterized class of functions v˜γ ,α and u˜ γ ,α given by the following nonlinear filtering of the wavelet series of f : −1 v˜γ ,α = 1 + γ 2−2|λ| fλ − Sα(22|λ| +γ )/γ (fλ ) ψλ λ∈Jj0

and u˜ γ ,α = fj0 +

Sα(22|λ| +γ )/γ (fλ )ψλ ,

λ∈Jj0

where St denotes the soft shrinkage operator, Jj0 all indices λ for scales larger than j0 , and fj0 is the approximation at the coarsest scale j0 . When K is not the identity operator, the minimization process results in a coupled system of nonlinear equations for the wavelet coefficients uλ and vλ , which is not as straightforward to solve. To overcome this problem, we adapt an iterative approach. As in Daubechies et al. (2004), we derive the iterative algorithm from a sequence of so-called surrogate functionals that are each easy to minimize, and for which hopefully the successive minimizers have the minimizing element of Eq. (5) as a limit. However, contrary to Daubechies et al. (2004), our variational problem has mixed quadratic and nonquadratic penalties. This requires a slightly different use of surrogate functionals. In Defrise and DeMol (2004a, 2004b) a similar u + v problem is solved by an approach that combines u and v into one vector-valued function (u, v). We follow a different approach in which we first solve the quadratic problem for v and then construct an iteration scheme for u. To this end, we introduce the differential operator T := (−Δ)1/2 . Setting v = T h the variational problem [Eq. (5)] reads as inf Ff (h, u),

(u,h)

(8)

9


with

2 Ff (h, u) = f − K(u + T h)L2 (Ω) + γ h2L2 (Ω) + 2α|u|B 1 (L1 (Ω)) . 1

Minimizing Eq. (8) with respect to w results in h˜ γ (f, u) = (T ∗ K ∗ KT + γ )−1 T ∗ K ∗ (f − Ku), or equivalently v˜γ (f, u) = T (T ∗ K ∗ KT + γ )−1 T ∗ K ∗ (f − Ku). Inserting this explicit expression for h˜ γ (f, u) in Eq. (8) and defining Tγ2 := I − KT (T ∗ K ∗ KT + γ )−1 T ∗ K ∗ ,

(9)

Ff h˜ γ (f, u), u = fγ − Tγ Ku2L2 (Ω) + 2α|u|B 1 (L1 (Ω)) .

(10)

fγ := Tγ f, yields

1

Thus, the remaining task is to solve

inf Ff h˜ γ (f, u), u , where u

Ff h˜ γ (f, u), u = fγ − Tγ Ku2L2 (Ω) + 2α|u|B 1 (L1 (Ω)) . 1

(11)

The corresponding variational equations in the sequence space representation are

∀λ: K ∗ Tγ2 Ku λ − (K ∗ fγ )λ + α sign(uλ ) = 0. This gives a coupled system of nonlinear equations for uλ . For this reason we construct surrogate functionals that remove the influence of K ∗ Tγ2 Ku. First, we choose a constant C such that K ∗ Tγ2 K < C. Since Tγ 1, it suffices to require that K ∗ K < C. Then we define the functional 2 Φ(u; a) := Cu − a2L2 (Ω) − Tγ K(u − a)L (Ω) , 2

which depends on an auxiliary element a ∈ L2 (Ω). We observe that Φ(u; a) is strictly convex in u for any a. Since K can be rescaled, our analysis is limited without loss of generality to the case C = 1. Finally, we add Φ(u; a) to Ff (h˜ γ (f, u), u) and obtain the following surrogate functional:

Ffsur h˜ γ (f, a), u; a = Ff h˜ γ (f, u), u + Φ(u; a)

= u2λ − 2uλ a + K ∗ Tγ2 (f − Ka) λ + 2α|uλ | λ

+ fγ 2L2 (Ω) + a2L2 (Ω) − Tγ Ka2L2 (Ω) .

(12)

10

DAUBECHIES ET AL .

The advantage of minimizing Eq. (12) is that the variational equations for uλ decouple. The summands of Eq. (12) are differentiable in uλ except at the point of nondifferentiability. The variational equations for each λ are now given by

uλ + α sign(uλ ) = a + K ∗ Tγ2 (f − Ka) λ . This results in an explicit soft shrinkage operation for uλ

uλ = Sα a + K ∗ Tγ2 (f − Ka) λ . The next proposition summarizes our findings; it is the specialization to our particular case of a more general theorem in Daubechies et al. (2004). Proposition II.2. Suppose K is a linear bounded operator modeling the blur, with K maps L2 (Ω) to L2 (Ω) and K ∗ K < 1. Moreover, assume Tγ ˜ u; a) is given by is defined as in Eq. (9) and the functional Ffsur (h,

Ffsur h˜ γ (f, u), u; a = Ff h˜ γ (f, u), u + Φ(u; a). Then, for arbitrarily chosen a ∈ L2 (Ω), the functional Ffsur (h˜ γ (f, u), u; a) has a unique minimizer in L2 (Ω). The minimizing element is given by

u˜ γ ,α = Sα a + K ∗ Tγ2 (f − Ka) , where the operator Sα is defined component-wise by Sα (x) = Sα (xλ )ψλ . λ

The proof follows from Daubechies et al. (2004). An iterative algorithm can now be defined by repeated minimization of Ffsur :

u0 arbitrary; un = arg min Ffsur h˜ γ (f, u), u; un−1 n = 1, 2, . . . . u

(13) The convergence result of Daubechies et al. (2004) can again be applied directly: Theorem II.1. Suppose K is a linear bounded operator, with K ∗ K < 1, and that Tγ is defined as in Eq. (9). Then the sequence of iterates

∗ 2 n−1 n = 1, 2, . . . , unγ ,α = Sα un−1 γ ,α + K Tγ f − Kuγ ,α , with arbitrarily chosen u0 ∈ L2 (Ω), converges in norm to a minimizer u˜ γ ,α of the functional 2

Ff h˜ γ (f, u), u = Tγ (f − Ku)L (Ω) + 2α|u|B 1 (L1 (Ω)) . 2

1


11

If N (Tγ K) = {0}, then the minimizer u˜ γ ,α is unique, and every sequence of iterates converges to u˜ γ ,α in norm. Combining the result of Theorem II.1 and the representation for v˜ summarizes how the image can finally be decomposed in cartoon and oscillating components. Corollary II.1. Assume that K is a linear bounded operator modeling the blur, with K ∗ K < 1. Moreover, if Tγ is defined as in Eq. (9) and if u˜ γ ,α is the minimizing element of Eq. (11), obtained as a limit of unγ ,α (see Theorem II.1), then the variational problem inf Ff (h, u),

(u,h)

with

2 Ff (h, u) = f − K(u + T h)L2 (Ω) + γ h2L2 (Ω) + 2α|u|B 1 (L1 (Ω)) 1

is minimized by the class

u˜ γ ,α , (T ∗ K ∗ KT + γ )−1 T ∗ K ∗ (f − K u˜ γ ,α ) , where u˜ γ ,α is the unique limit of the sequence

∗ 2 n−1 unγ ,α = Sα un−1 γ ,α + K Tγ f − Kuγ ,α ,

n = 1, 2, . . . .

D. Redundancy and Adaptivity to Reduce Artifacts The nonlinear filtering rule of Proposition II.1 provides explicit descriptions of v˜ and u˜ that are computed by fast discrete wavelet schemes. However, nonredundant filtering often creates artifacts in terms of undesirable oscillations, which manifest themselves as ringing and edge blurring (Fig. 1). Poor directional selectivity of traditional tensor product wavelet bases likewise cause artifacts. This section discusses various refinements on the basic algorithm that address this problem. In particular, we use redundant translation invariant schemes, complex wavelets, and additional edge dependent penalty weights. We describe these generalizations here and provides examples in the next section. 1. Translation Invariance by Cycle Spinning Assume that an image has 2M rows of 2M pixels, where the gray value of each pixel gives an average of f on a square 2−M × 2−M , which we denote by fkM , with k a double index running through all the elements of {0, 1, . . . , 2M − 1} × {0, 1, . . . , 2M − 1}. A traditional wavelet transform then computes

12

DAUBECHIES ET AL .

F IGURE 1. An initial geometric image f (left) and two versions of f (the middle decomposed with the Haar wavelet basis and the right with the Db3 basis) where the soft shrinkage operator with shrinkage parameter α = 0.5 was applied.

fj0 and fj,l,i with j0 ≤ j ≤ M, i = 1, 2, 3, and l ∈ {0, 1, . . . , 2j − 1} × {0, 1, . . . , 2j − 1} for each j , where the fj,l,i represents the different species of wavelet coefficients (in two dimensions, there are three), mostly localized on (and indexed by) the squares [l1 2−j , (l1 + 1)2−j ] × [l2 2−j , (l2 + 1)2−j ]. Because the corresponding wavelet basis is not translation invariant (as can be seen from the localization of the wavelet coefficients; fj0 has a similar translation noninvariance that we did not denote explicitly), Coifman and Donoho (1995) proposed to recover translation invariance by averaging over the 22(M+1−j0 ) translates of the wavelet basis. Since many wavelets occur i (x − 2M n) in more than one of these translated bases (in fact, each ψj,k in exactly 22(j +1−j0 ) different bases), the average over all these bases uses only (M + 1 − j0 )22M different basis functions (and not 24(M+1−j0 ) = number of bases × number of elements in each basis). This approach is called i cycle spinning. With a slight abuse of notation, writing ψj,k+2 j −M n for the i (x − 2M n), this average can then be written as translation ψj,k

f

M

=

cycled fj0

+2

−2(M+1)

M −1 M−1 2

l1 ,l2 =0 j =j0

22j

3

i i fj,l2 −M+j ψj,l2−M+j .

i=1

Performing nonlinear filtering in each of the bases and averaging the result then corresponds to applying the corresponding nonlinear filtering on the (much smaller number of) coefficients in the last expression. This is the standard method to implement thresholding on cycle-spinned representations. The resulting sequence space representation of the variational functional in Eq. (7) must be adapted to the redundant representation of f . Therefore, note


13

that the Besov penalty term takes the form i p 1/p 2(j (s−1+2/p)+2(j −M))p fj,l2 . |f |Bps (Lp ) ∼ −M+j |λ|j0

and · 2H −1 change similarly. Consequently, the same The norms · minimization rule is obtained but with respect to a richer class of wavelet coefficients. 2L2

2. Directional Sensitivity by Frequency Projections Several authors (Kinsbury, 1999; Selesnick, 2001; Fernandes et al., 2000) have shown that if positive and negative frequencies are treated separately in the 1D wavelet transform (resulting in complex wavelets), the directional selectivity of the corresponding 2D multiresolution analysis is improved. This can be done by applying the following orthogonal projections: P + : L2 → L2,+ = f ∈ L2 : supp fˆ ⊆ [0, ∞) , P − : L2 → L2,− = f ∈ L2 : supp fˆ ⊆ (−∞, 0] . ˜ and The projectors P + and P − may be either applied to f or to {φ, φ} ˜ {ψ, ψ}. In a discrete framework, these projections must be approximated. This has been done in different ways in the literature. Hilbert transform pairs of wavelets are used in Kinsbury (1999) and Selesnick (2001). In Fernandes et al. (2000), f is projected (approximately) by multiplying with shifted generator symbols in the frequency domain. We follow the second approach, that is, (P + f )∧ (ω) := fˆ(ω)H (ω − π/2) and (P − f )∧ (ω) := fˆ(ω)H (ω + π/2), where f denotes the function to be analyzed and H is the low-pass filter for a conjugate quadrature mirror filter pair. Then fˆ(ω) = (B + P + f )∧ (ω) + (B − P − f )∧ (ω),

(14)

where the backprojections are given by (B + f )∧ = fˆH (· − π/2) and

(B − f )∧ = fˆH (· + π/2),

respectively. This technique provides a simple multiplication scheme in Fourier, or equivalently, a convolution scheme in time domain. In a separable 2D framework the projections must be carried out in each of the two frequency variables, resulting in four approximate projection operators P ++ , P +− , P −+ , and P −− . Because f is real, we have (P ++ f )∧ (−ω) = (P −− f )∧ (ω) and

(P +− f )∧ (−ω) = (P −+ f )∧ (ω),

14

DAUBECHIES ET AL .

so that the computation of P −+ f and P −− f can be omitted. Consequently, the modified variational functional takes the form

2

2

Ff (u, v) = 2 P ++ f − (u + v) L2 + P +− f − (u + v) L2

+ 2λ P ++ v2H −1 + P +− v2H −1 + 2α|u|B 1 1(L1 ) +−

++ 2 2 ≤2 P f − (u + v) L + P f − (u + v) L 2 2

++ 2 +− 2 + 2λ P vH −1 + P vH −1

, + 4α |P ++ u|B 1 + |P +− u|B 1 1(L1 )

1(L1 )

{P ++ v, P ++ u}

and {P +− v, P +− u} which can be minimized with respect to separately. The projections are complex valued so the thresholding operator needs to be adapted. Parameterizing the wavelet coefficients by modulus and angle and minimizing yields the following filtering rules for the projections of v˜γ ,α and u˜ γ ,α (where ·· represents any combination of +, −) −1 ··

·· P ·· v˜γ ,α = 1 + γ 2−2|λ| P fλ − Sα(22|λ| +γ )/γ |P ·· fλ | eiω(P f ) ψλ |λ|j0

and P ·· u˜ γ ,α = (P ·· f )j0 −1

·· + 1 + γ 2−2|λ| Sα(22|λ| +γ )/γ |P ·· fλ | eiω(P f ) ψλ . |λ|j0

Finally, the backprojections must be applied to obtain the minimizing functions ++ ++ v˜γBP P v˜γ ,α + B −− P ++ v˜γ ,α + B +− P +− v˜γ ,α + B −+ P +− v˜γ ,α ,α = B

and ++ ++ u˜ BP P u˜ γ ,α + B −− P ++ u˜ γ ,α + B +− P +− u˜ γ ,α + B −+ P +− u˜ γ ,α . γ ,α = B

3. Weighted Penalty Functions To improve edge-preserving capability we additionally introduce a positive weight sequence wλ in the H −1 penalty term. Consequently, we seek to minimize a slightly modified sequence space functional fλ − (uλ + vλ )2 + γ 2−2|λ| wλ |vλ |2 + 2α|uλ | . (15) λ


15

The resulting texture and cartoon components take the form −1 1 + γ wλ 2−2|λ| fλ − Sα(22|λ| +γ wλ )/γ wλ (fλ ) ψλ v˜γw,α = |λ|j0

and u˜ w γ ,α = fj0 +

Sα(22|λ| +γ wλ )/γ wλ (fλ )ψλ .

|λ|j0

The main goal is to introduce a control parameter that depends on the local structure of f . The local penalty weight wλ should be large in the presence of an edge and small otherwise; the result of this weighting is enhancement of the sensitivity of u near edges. To achieve this, the edges must first be localized, which is done by a procedure similar to an edge detection algorithm in Mallat and Zhong (1992). This scheme rests on the analysis of the cycle-spinned wavelet coefficients fλ at or near the same location but at different scales. It is expected that the fλ belonging to fine decomposition scales contain information on edges (well localized) as well as oscillating components. Oscillating texture components typically appear in fine scales only; edges leave a signature of larger wavelet coefficients through a wider range of scales. We thus apply the following less sophisticated edge detector. Suppose that f ∈ VM and je denotes some critical scale; then for a certain range of scales |λ| = |(i, j, k)| = j ∈ {j0 , . . . , j1 − je − 2, j1 − je − 1}, we mark all positions k where |fλ | is larger than a level-dependent threshold parameter tj . Here the value tj is chosen proportional to the mean value of all wavelet coefficients of level j . We indicate that |fλ | represents an edge if k was marked for all j ∈ {j0 , . . . , j1 − je − 2, j1 − je − 1}. Finally, we adaptively choose the penalty sequence by setting Θλ if j ∈ {M − 1, . . . , j1 − je } and k was marked as an edge, wλ = ϑλ otherwise, where ϑλ is close to 1 and Θλ is much larger in order to penalize the corresponding vλ s. E. Image Examples This section presents examples of decomposed, deblurred, and denoised images. We begin with the case where K is the identity operator. To show how the nonlinear (redundant) wavelet scheme acts on piecewise constant functions, we decompose a geometric image (representing cartoon components only) with sharp contours (Fig. 2). We observe that u˜ represents the cartoon part

16

DAUBECHIES ET AL .

F IGURE 2. From left to right: initial geometric image f , u, ˜ v˜ + 150, computed with Db3 in the translation invariant setting, α = 0.5, γ = 0.01.

F IGURE 3. Left: noisy segment of a woman image; middle and right: first two scales of S(f ) inducing the weight function w.

very well. The texture component v˜ (plus a constant for illustration purposes) contains only some very weak contour structures. Next, we demonstrate the performance of the Haar shrinkage algorithm successively incorporating redundancy and local penalty weights. The redundancy is implemented by cycle spinning as described in Section II.D.1. The local penalty weights are computed in the following manner. First, the shrinkage operator S is applied to f with a level-dependent threshold (the threshold per scale is equal to twice the mean value of all the wavelet coefficients of the scale under consideration). Second, for those λ according to the nonzero values of Sthreshold (fλ ), we set wλ to Θλ = 1 + C (here C = 10; moreover, we set wλ equal to ϑλ = 1 elsewhere). The coefficients Sthreshold (fλ ) for the first two scales of a segment of the image Barbara are shown in Fig. 3. Figure 4 presents our numerical results. The upper row shows the original and the noisy image. The next row visualizes the results for nonredundant Haar shrinkage (method A). The third row shows the same but incorporating cycle spinning (method B), and the last row shows the incorporation of cycle spinning and local penalty weights. Each


17

F IGURE 4. Top: initial and noisy image. Second row: nonredundant Haar shrinkage (method A). Third row: translation invariant Haar shrinkage (method B). Bottom: translation invariant Haar shrinkage with edge enhancement (method C). Second through fourth rows from left to right: u, ˜ v˜ + 150 and u˜ + v, ˜ α = 0.5, γ = 0.0001, computed with Haar wavelets and critical scale je = −3.

18

DAUBECHIES ET AL .

TABLE 1 S IGNAL - TO - NOISE RATIOS (SNR S ) OF THE SEVERAL DECOMPOSITION METHODS (H AAR SHRINKAGE , TRANSLATION INVARIANT H AAR SHRINKAGE , TRANSLATION INVARIANT H AAR SHRINKAGE WITH EDGE ENHANCEMENT ) Haar shrinkage

SNR (f, fε )

SNR (f, u + v)

SNR (f, u)

Method A Method B Method C

20, 7203 20, 7203 20, 7203

18, 3319 21, 6672 23, 8334

16, 0680 16, 5886 17, 5070

F IGURE 5. From left to right: initial fabric image f , u, ˜ v˜ + 150, computed with Db4 incorporating frequency projections, α = 0.8, γ = 0.002.

extension of the shrinkage method improves the results. This is also confirmed by comparing the signal-to-noise ratios (which is here defined as follows: SNR(f, g) = 10 log10 (f 2 /f − g2 )) (Table 1). The next experiment is done on a fabric image (Fig. 5). In contrast to the previous examples, here we present the use of frequency projection as introduced in Section II.D.2. The numerical result shows convincingly that the texture component can be also well separated from the cartoon component. To compare the performance with the BV − L2 model (Rudin et al., 1992) and with the BV − H −1 model (Osher et al., 2003), we apply our scheme to a woman image (the same used in Vese and Osher, 2003; Osher et al., 2003) (Fig. 6). We obtain very similar results as obtained with the model proposed in Osher et al. (2003). Compared with the results obtained with the BV − L2 model (Rudin et al., 1992) we observe that our reconstruction of the texture component contains much less cartoon information. In terms of computational cost we have observed that even with applying cycle spinning and edge enhancement our proposed wavelet shrinkage scheme is less time consuming than the BV − H −1 restoration scheme (Table 2), even when the wavelet method is implemented in MATLAB, which is slower than the compiled version for the scheme of Osher et al. (2003).


19

F IGURE 6. Top (left to right): initial woman image f , u˜ and v˜ + 150, computed with Db10 (method C), α = 0.5, γ = 0.002. Bottom (left to right): u and v obtained by the BV − H −1 model (Osher et al., 2003) and the v component obtained by the classical Total Variation model (Rudin et al., 1992). TABLE 2 C OMPARISON OF COMPUTATIONAL COST OF THE PDE- AND WAVELET- BASED METHODS Data basis

Barbara image (512 × 512 pixels)

Hardware architecture Operating system OS distribution Model Memory size (MB) Processor speed (MHz) Number of CPUs

PC Linux Redhat7.3 PC, AMD Athlon-XP 1024 1333 1

Computational cost

(Average over 10 runs)

PDE scheme in Fortran (compiler f77) Wavelet shrinkage (method A) (MATLAB) Wavelet shrinkage (method B) (MATLAB) Wavelet shrinkage (method C) (MATLAB)

56, 67 sec 4, 20 sec 24, 78 sec 26, 56 sec

We end this section with an experiment where K is not the identity operator. In our particular case, K is a convolution operator with Gaussian kernel. The implementation is done simply in Fourier space. The upper row in Fig. 7

20

DAUBECHIES ET AL .

F IGURE 7. Top (left to right): initial image f , blurred image Kf . Middle (left to right): deblurred u, ˜ deblurred v˜ + 150. Bottom: deblurred u˜ + v, ˜ computed with Db3 using the iterative approach, α = 0.2, γ = 0.001.

shows the original f and the blurred image Kf . The lower row visualizes the results: the cartoon component u, ˜ the texture component v, ˜ and the sum of both u˜ + v. ˜ The deblurred image u˜ + v˜ contains (after a small number of iterations) contains more small-scale details than Kf . This definitely shows the capabilities of the proposed iterative deblurring scheme [Eq. (13)].


21

III. V ECTOR -VALUED R EGIMES AND M IXED C ONSTRAINTS Section II considered a concrete image decomposition problem where the solution was assumed to be a vector of functions, namely (v, u). Since the constraint on v was a quadratic one, we could derive for an explicit expression the component v. Therefore, Eq. (5) could be transformed into the much simpler form of Eq. (11), for which we could directly apply the basic algorithm of Daubechies et al. (2004). It is now natural to generalize the iterative approach proposed in Daubechies et al. (2004) to the vector-valued situation. We now assume we have an m-dimensional data vector (f1 , . . . , fm ) available from which we wish to reconstruct an n-dimensional object (v1 , . . . , vn ) and where, moreover, the constraints on the object might be a mixture of smoothness and sparsity measures. Similar problems were discussed in Defrise and DeMol (2004a), Anthoine (2005), Fornasier and Rauhut (2006), Elad et al. (2005), and Starck et al. (2005). We limit ourselves to the special case m = 1, with the extra assumption that (v1 , . . . , vn ) has a sparse expansion (or satisfies some other constraint) with respect to several bases or frames. The main difference from the preceding section is that we provide a rich dictionary of bases/frames that serves as a reservoir of building blocks for (v1 , . . . , vn ). Our primary motivation for this work was an approach in audio data coding by Torrésani et al. (Molla and Torrésani, 2005; Jaillet and Torrésani, 2005; Daudet and Torrésani, 2002), who represented audio signals by means of wavelets for transients and local cosine functions for tonal components. Their approach produces sparse representations of audio signals that are very efficient in audio coding. At the end of this section we demonstrate how the scheme developed here works for such audio coding. A. Some Remarks on Frame Dictionaries and Sparsity Sparsity can be achieved by using a suitable basis in the underlying function space. In the preceding section, we introduced redundant systems to reduce artifacts. However, recent studies indicate that redundant systems, such as frames, or dictionaries of waveform systems may also lead to better (i.e., sparser) representations. With such dictionaries of waveform systems, there exist several methods (e.g., best orthogonal basis, matching pursuit, basis pursuit; see Chen et al., 1999) that allow a decomposition of a signal into an optimal superposition of dictionary elements, where optimal means having the smallest 1 norm of coefficients among all such decompositions. Numerical schemes to implement these iterative pursuit schemes in highly overcomplete dictionaries often lead to very large-scale optimization problems.

22

DAUBECHIES ET AL .

As an alternative to these methods, we discuss a method for finding the p -optimal decomposition (1 ≤ p ≤ 2) for which the skeletal structure is taken from the iterative strategy proposed in Daubechies et al. (2004). The advantage of this method is that, we need not make strong assumptions on the preselected family of frames to achieve convergence of the iteration process. 1. Frames, Sparsity, and Inverse Problems A frame {φλ : λ ∈ Λ} in a Hilbert space H is a set of vectors for which there exist constants 0 < A ≤ B < ∞ such that, for all v ∈ H, v, φλ H 2 ≤ Bv2 . Av2H ≤ H λ∈Λ

Frames are typically overcomplete – that is, for a given vector v ∈ H, many different sequences g ∈ 2 of coefficients can be found so that v= gλ φλ . (16) λ∈Λ

Some of these sequences have special properties; for instance, the sequence with minimal 2 norm may be preferred. The problem of finding sequences g can be considered as an inverse problem. To this end, we introduce the operator F (often called the frame operator) that maps a function v ∈ H to the element F v of 2 by F v = {v, φλ H }λ∈Λ . The adjoint F ∗ maps a sequence ∗ ∗ g ∈ 2 to the element F g of H via F g = λ∈Λ gλ φλ (i.e., solving Eq. (16) amounts to solving F ∗ g = v). The sequence g with minimal 2 norm is obtained by standard least-squares methods for these equations. It is often of interest to find sequences that are sparser than the minimum 2 norm solution. For instance, if the object v is known to be a noisy version of a sparse linear combination of the φλ , it is reasonable to seek a coefficient sequence with small p norm (e.g., p = 1) (see Daubechies et al., 2004). It then makes sense to compute the sequence g that minimizes v − F ∗ g2H + αgp . p

(17)

In many applications, the features or signals of interest cannot be observed directly but must to be inferred from other, observable quantities. Very often there is a linear relationship K : H → H between the feature, modeled by a function v, and the derived quantities, modeled by another function z, which often has additional noise; the relation between v and z can then be written as f = z + e = Kv + e.


23

To find an estimate for v from observed f , one can minimize the discrepancy f − Kv2H .

(18)

Combining Eqs. (17) and (18) yields Φ(g) = f − AF ∗ g2H + αgp , p

(19)

where we allow 1 ≤ p ≤ 2. As seen in Section II, for this variational problem an iterative method to approximate the minimizer was suggested (Daubechies et al., 2004). 2. Extension to Frame Dictionaries Instead of using only one single frame, we now aim to represent the function we are searching for by several different frames. This is reasonable since there are certain classes of signals where one particular frame (or basis) is not optimally suited (in the sense of locally best sparse approximation). Since a finite union of frames is again a frame, variational formulation [Eq. (19)] applies here as well. However when imposing mixed or different constraints on the different frames, a setup where each frame is treated individually is better suited. An extensive discussion on this subject can be found in Teschke (2007). We denote with {φλi : λ ∈ Λi , i = 1, 2, . . . , n} the finite family (or dictionary) of frames where each individual collection {φλi : λ ∈ Λi } is assumed to be a frame for H. For each frame, we may consider the associated frame operator Fi : H → 2 , which is defined by the map v → v i := {v, φλi }λ∈Λi . A natural composition of all frame operators is given by the sum of its adjoints, n Fi∗ v i . v1, . . . , vn →

i=1

With regard to our linear relationship K : H → H , we may define the operator K n : (2 )n → H by n

K n : (2 )n : v 1 , . . . , v n → KFi∗ v i , i=1

where the adjoint

(K n )∗ : H

→ (2

)n

is given by

g → KA∗ g = (F1 A∗ g, . . . , Fn A∗ g). With this specific operator K n we may define the following variational functional 2 (20) Φ(g) := f − K n g H + α · |g|,

24

DAUBECHIES ET AL . p

p

where g = (g 1 , . . . , g n ) ∈ (2 )n , |g| := (g 1 p11 , . . . , g n pnn ) and α = (α1 , . . . , αn ) represents n positive regularization parameters. As before, we restrict ourselves to 1 ≤ pi ≤ 2, but not necessarily requiring pi = pj . For n = 2, several concepts to minimize Eq. (20) are suggested by Daubechies and Teschke (2004, 2005) and Defrise and DeMol (2004a). We adapt the strategy proposed by Daubechies et al. (2004) in the following text. B. Iterative Approach by Surrogate Functionals At first, it can easily be verified that Φ as defined in Eq. (20) is convex. In order to apply the √ technique of Gaussian surrogate functionals, we define a ˜ constant C := C B1 + · · · + Bn , where C˜ is an upper bound for K and Bi is the upper-frame bound with respect to Fi . Then, for some auxiliary element a ∈ (2 )n , the Gaussian surrogate extension for the data misfit term takes the form 2 2 sur (g; a) = f − K n g H + C 2 g − a2(2 )n − K n g − K n a H . This functional is again convex and it holds sur (g; a) − f − KA g2H 0. Therefore, it is reasonable to consider instead of Φ the surrogate functional Φ sur (g; a) := sur (g, a) + α · |g|, (21) satisfying for all a ∈ (2 )n , Φ sur (g; g) = Φ(g) and Φ sur (g; a) Φ(g). To approach the minimizer g of Eq. (20), we consider the following iteration: g0 arbitrary;

gm+1 = arg min Φ sur (g; gm ) g

m = 0, 1, . . . .

(22)

To execute the iteration of Eq. (22), we must evaluate the necessary conditions for a minimum of Eq. (21). For some generic a ∈ (2 )n we have Φ sur (g; a) =

n

2 C 2 gλi − 2gλi Fi A∗ f + C 2 a i − Fi K ∗ K n a λ

i=1 λ∈Λi

p + αi gλi i

2 + f 2H + C 2 a2(2 )n − K n a H ,

(23)

where gλi represents for the coefficients of g i . We observe that through the Gaussian surrogate extension the variational equations for the individual gλi decouple, which allows – as we shall see – an explicit computation of the minimizer.


25

Each summand in Eq. (23) is differentiable in gλi except for pi = 1 at = 0. For pi > 1, the minimization reduces to solving

i i pi −1 αi −2 ∗ 2 i ∗ n g F p sign g = C K f + C a − F K K a . gλi + i i i λ λ λ 2C 2 (24) As it can be retraced in Daubechies et al. (2004), the map Fτ,p (x) = x + τp sign(x)|x|p−1 is for any p > 1 a one-to-one map from R to itself, we thus find that for all i = 1, . . . , n and λ ∈ Λi ,

gλi = Sαi /2C 2 ,pi C −2 Fi K ∗ f + C 2 a i − Fi K ∗ K n a λ , gλi

where Sαi /2C 2 ,pi is defined by Sαi /2C 2 ,pi := (Fαi /2C 2 ,pi )−1 . For pi = 1, let the sign function be set valued (because of the nondifferentiability of | · | at 0); that is sign(t) = ±1 for t ≷ 0 and sign(t) ∈ [−1, 1] for t = 0, leading to

αi sign gλi C −2 Fi K ∗ f + C 2 a i − Fi K ∗ K n a λ . (25) gλi + 2 2C In this case, the associated operator Sαi /2C 2 ,1 is simply the well-known soft shrinkage operator with threshold αi /2C 2 . Introducing for some h ∈ 2 the sequence-wise acting operator St,pi (h) = {St,pi (hλ )}λ∈Λi , we may define the following generalized shrinkage operator for a vector of sequences (g 1 , . . . , g n ) ∈ (2 )n and parameter vectors t = (t1 , . . . , tn ) and p = (p1 , . . . , pn ),

St,p (g) = St1 ,p1 g 1 , . . . , Stn ,pn g n . With the latter shorthand notation the minimizer g of Eq. (21) can be written in the more clearly arranged form

∗

∗ g = Sα/2C 2 ,p C −2 K n f + C 2 a − K n K n a . (26) The following proposition can be found in Teschke (2007), or it can be retraced with the help of Daubechies et al. (2004). Proposition III.1. Suppose the operator K maps a Hilbert space H to ˜ Furthermore, suppose we another Hilbert space H and is bounded by C. are given n frames where the respective frame operators Fi map H to 2 with upper frame bounds Bi . Assume, moreover, that f is an element of H and a ∈ (2 )n . If Φ sur (g; a) is defined as in Eq. (21) on (2 )n , then Φ sur (g; a) has a unique minimizer in (2 )n . This minimizer is given by

∗

∗ (27) g = S α 2 C −2 K n f + C 2 a − K n K n a . 2C

For all h ∈ (2

)n ,

one has

Φ sur (g + h; a) Φ sur (g; a) + C 2 h2(2 )n .

26

DAUBECHIES ET AL .

This result directly carries over to the iteration in Eq. (22). Corollary III.1. Make the same assumptions as in Proposition III.1. Pick g0 ∈ (2 )n arbitrarily. Then the iterates of the algorithm Eq. (22) have the following explicit form:

gm+1 = Sα/2C 2 ,p C −2 KA∗ f + C 2 gm − KA∗ KA gm . (28) As the final result of this section, it can be verified that the proposed iteration in Eq. (22) converges in the norm of (2 )n . Theorem III.1. Suppose the operator K maps a Hilbert space H to another ˜ Furthermore, suppose we are given n Hilbert space H and is bounded by C. frames where the respective frame operators Fi map H to 2 with upper frame bounds Bi . Assume, moreover, that f is an element of H and a ∈ (2 )n . Then the sequence of iterates

∗

∗ gm+1 = Sα/2C 2 C −2 K n f + C 2 gm − K n K n gm , m = 0, 1, 2, . . . , with g0 arbitrarily chosen in (2 )n , converges in norm to a minimizer of the functional 2 Φ(g) = f − K n g H + α · |g|. The complete proof of this theorem is quite lengthy and technical; for convenience, refer to Daubechies et al. (2004) or Teschke (2007). Essentially the proof consists of two steps. At first, based on Opial’s theorem (see Opial, 1967), the weak convergence is shown. A second step shows that the convergence holds also in norm. C. Audio Coding Example This section shows the usefulness of the proposed multiframe approach. We present two numerical experiments from different perspectives: convergence rates, sparsity achievement, and approximation quality. The overall configuration of our algorithm is as follows; for the sake of simplicity, we choose as our underlying frames a wavelet basis (Haar system) and a (nonlocal) Fourier basis only. Hence, B1 = B2 = 1. In the examples, we restrict ourselves to K = I . Consequently, the constant √ C in our Gaussian surrogate is not allowed to be equal or smaller than 2. We aim to achieve sparsity in both representations (i.e., we set p1 = p2 = 1). The variational problem is thus simply given by

2 Φ g 1 , g 2 = f − F1∗ g 1 + F2∗ g 2 + α1 g 1 + α2 g 2 , 1

1


27

and the minimization by Gaussian surrogates yields the following iteration 1 (g )m+1 (g 2 )m+1 Sα1 /2C 2 ,1 (C −2 {F1 f + C 2 (g 1 )m − F1 F1∗ (g 1 )m − F1 F2∗ (g 2 )m }) . = Sα2 /2C 2 ,1 (C −2 {F2 f + C 2 (g 2 )m − F2 F1∗ (g 1 )m − F2 F2∗ (g 2 )m }) Because we deal with bases only, the application of F1 F1∗ , F1 F2∗ , F2 F1∗ , and F2 F2∗ simplifies the discrete decomposition and reconstruction schemes. If going beyond bases (i.e., using frames), it is necessary to compute (approximate) all the (mixed) gram matrices. This may be costly but can be optimized by picking localized and reasonably incoherent frames. As an experimental observation, in case the frame generating analyzing atoms is not reasonably distinct, the scheme is not able to separate the signal components adequately – all the sequences g i contain very similar informations. 1. A Synthetic Example In this example we have simulated a signal f that is a composition of two different components: a harmonic wave and noisy perturbation within the interval [350, 400]. As a sampled discrete vector it has a total number of 631 coefficients in the time-domain representation. This discrete vector is used as input for our algorithm. The results for α1 = α2 = 0.2 are shown in Fig. 8. Involving the Haar wavelet basis and the Fourier basis splits the signal into very sparse and well-separated components. The sparseness evolution of the two individual components can be seen in the sparsity plot in Fig. 8, indicating that the chosen frames fulfill the signal structure adequately. 2. Real Data: Glockenspiel This data set represents a real audio signal consisting of tonal components and a sequence of (bell) attacks. We again apply Haar wavelet and Fourier splitting. Figure 9 shows the results for α1 = 0.02 and α2 = 0.01. As expected, the Haar system captures all the bell attacks very well, and, moreover, the Fourier system the tonal components. The sparsity evolution graph shows the rapid decay of the number of wavelet coefficients, which can be explained by a fast “bell attack” localization process through the iteration. In summary, whenever the dictionary consists of complementary frames, the proposed algorithm produces a sparse representation in which the individual components overlap inconsiderably. However, a different choice of penalty weights would imply a different splitting of the signal; if α1 α2 , then almost all the signals would be captured by the wavelet system and vice versa. The audio results can be downloaded from http://www.zib.de/ AG_InverseProblems/wav/.

28

DAUBECHIES ET AL .

F IGURE 8. From top left to bottom right: synthetic data, Haar wavelet component (g 2 ) (in time domain) after 100 iterations, SNR evolution through the iteration process, Fourier component (g 1 ) (in time domain) after 100 iterations, reconstruction and error after 100 iterations, and sparsity evolution through the iteration process.


29

F IGURE 9. From top left to bottom right: Glockenspiel data, Haar wavelet component (g 2 ) (in time domain) after 30 iterations, SNR evolution through the iteration process, Fourier component (g 1 ) (in time domain) after 30 iterations, reconstruction after 30 iterations, and sparsity evolution through the iteration process.

30

DAUBECHIES ET AL .

IV. I MAGE R ESTORATION WITH G ENERAL C ONVEX C ONSTRAINTS Sections II and III considered image restoration problems in which the constraints on the signal/image to be reconstructed could be directly expressed (or could be adequately replaced) by means of basis or frame coefficients. However, for certain applications it might not be desirable to formulate the constraints in this manner. Often a more general description of the constraint is much better suited. Section II (in the context of image decomposition) showed that in the setting of Vese and Osher (2003) and Osher et al. (2003), the cartoon component of an image was modeled by a BV function. The BV penalty term was then replaced by a B11 (L1 ) term (amounting to a slightly stronger constraint) in order formulate the problem in the elegant wavelet framework in which the proposed iteration scheme could be easily applied. It might now be interesting to determine whether a similar iteration scheme can be executed when waiving the comfort of a wavelet framework and allowing the solution to be a BV function (or fulfilling some other general homogeneous convex constraint). A. General Convex Constraint Preliminaries As before, we consider a functional of the form f − Kv2H + 2αJ (v),

(29)

where J (v) < ∞, or even J (v) < 1 is the mathematical translation of the a priori knowledge (at times, we will use · for · H ). In what follows, we consider two different choices of J (v), both adapted to the case where the inverse problem consists in deblurring and denoising a 2D image (as in Daubechies and Teschke, 2005 which was, in turn, inspired by Daubechies et al., 2004 and Vese and Osher, 2003). Both approaches are natural sequels to Daubechies and Teschke (2005). In the first approach, we consider J (v) of the same type as in Daubechies and Teschke (2005), but in a more general framework, where J (v) can be any positive, convex, one-homogeneous functional. An extensive discussion of such functionals, in much greater generality than presented here, is provided in Combettes and Wajs (2005). To be self-contained and to avoid introducing the full complexity of Combettes and Wajs (2005), we present a sketch of a simpler version that suffices for our case (for a detailed discussion on the proof, refer to Daubechies et al., 2007). In the second approach, J (v) is the same as in Rudin and Osher (1994) and Osher et al. (2003), but the numerical solution in Osher et al. (2003) of a fourth-order nonlinear PDE is replaced by an iterative


31

approach similar to Daubechies and Teschke (2004, 2005) (see to the related prior work of Bect et al., 2004). We assume that the functional to minimize takes the form of Eq. (29), where J is a positive, convex, and one-homogeneous functional. In this case, the variational problem can be recast as follows: Consider J ∗ , the Fenchel transform or so-called dual functional of J (see Ekeland and Témam, 1999, and Rockafellar and Wets, 1998). Since J is positive and one-homogeneous, there exists a convex set C such that J ∗ is equal to the indicator function χC over C. In Hilbert space, we have total duality between convex sets and positive and one-homogeneous functionals, i.e., J = (χC )∗ , or (χC )∗ (v) = sup v, h = J (v) h∈C

(see, for example, Ekeland and Témam, 1999; Aubert and Aujol, 2005; Chambolle, 2004; Combettes and Wajs, 2005). (Note: Combettes and Wajs, 2005, provides a much more general and complete discussion; we restrict ourselves here to a simple situation and only sketch the arguments. For a complete, detailed discussion, see Combettes and Wajs, 2005.) This implification yields the following reformulation of our problem: given some closed convex set C ⊂ H (on which we may still impose extra conditions, below), we wish to minimize

FC (v) = f − Kv2H + 2α sup v, h ,

(30)

h∈C

where we assume K to be a bounded operator from H to itself, with K < 1. We consider two particular cases in more detail. Example 1. As in Daubechies and Teschke (2004), a particular orthonormal basis {φλ }λ∈Λ in H is preselected, and the prior is defined as v, φλ . J (v) = λ∈Λ

This can be viewed as a special case of Eq. (30), since in this case

C = h ∈ L2,per [0, 1]2 ; h, φλ ≤ 1, ∀λ . Similarly, the case with the prior |v|w = wλ f, φλ , λ∈Λ

with inf wλ > 0, λ∈Λ

also fits into the framework of Eq. (30), with C now defined by

C = h ∈ L2,per [0, 1]2 ; h, φλ ≤ wλ−1 , ∀λ .

32

DAUBECHIES ET AL .

When K = I and the problem is ill-posed, the resulting minimization scheme amounts to Landweber iteration with thresholding applied in each step. Example 2. In the BV regularization framework (Rudin et al., 1992; Rudin and Osher, 1994), one considers functionals of the form 2 (31) f − KvL2 (Ω) + 2α |∇v| Ω

and minimizes over all possible v ∈ L2 (Ω). Expressing this functional by means of a convex set C, C is the L2 closure of

C = h ∈ L2,per [0, 1]2 ; h = div g, where g is a second field in Cc1 (Ω)2

g1 (x, y)2 + g2 (x, y)2 1/2 ≤ 1 ; that satisfies g∞ = sup (x,y)∈[0,1]2

that is, we may again write

sup v, h = h∈C

|∇v| = |v|BV . Ω

For details on the structure of C, see Evans and Gariepy (1991) and Ambrosio et al. (2000). Results on iterative strategies developed for Example 1 carry over to the BV case, and much of the analysis elaborated in Daubechies and Teschke (2004) can be generalized to the minimization of Eq. (31). 1. Reformulation of the Problem We assume that C is a closed convex set in H, C is symmetric (i.e., h ∈ C ⇒ −h ∈ C), and there exists finitely many vectors a1 , . . . , aN ∈ H, and r > 0 so that Br (0) ∩ {a1 , . . . , aN }⊥ ⊂ C (i.e., {h: h, ai = 0; i = 1, . . . , N and h < r} ⊂ C). Note: we introduce the finite-dimensional subspace to which C is orthogonal for two reasons. First, there are cases of interest in which C consists of functions that have zero mean in [0, 1]2 (e.g., if C contains only divergences of smooth periodic fields). Second, it is easier to restrict ourselves to only fine-scale functions (see below). Defining the functionals L(v, h) := f − Kv2H + 2αv, h , we can rewrite infv∈H FC (v) as inf sup L(v, h).

v∈H h∈C

(32)


33

Note that L(v, h) is continuous in both arguments; it is also convex with respect to v and concave with respect to h. This means that (provided some technical conditions are fulfilled; see Boyd and Vandenberghe, 2005; HiriartUrruty and Lemaréchal, 1993; Ekeland and Témam, 1999; or Daubechies et al., 2007), we can apply the minimax theorem, which allows us to interchange inf and sup in Eq. (32). In this case, the minimax theorem moreover asserts that inf and sup are achieved – the inf is a minimum, the sup is a maximum. B. Solving the Inverse Problem for Convex Penalization Although the case where K ∗ K does not have a bounded inverse (i.e., where the inverse problem is ill-posed) is of most interest to us, we start by sketching the approach in the easier well-posed case. Theorem IV.1. Suppose that all assumptions made above hold true, and K ∗ K has bounded inverse in its range. If we define A := (K ∗ K)−1/2 and, for an arbitrary closed convex set K ⊂ H, SK := I d − PK , where PK is the (nonlinear) projection on K (i.e., PK ϕ = arg minh∈K h − ϕ), then the minimizing v is given by v = ASαAC {AK ∗ g}. An obvious example is the case of simply denoising an image, without deblurring: Example 3. Consider the denoising problem with an 1 -constraint in the basis {φλ }λ∈Λ . In this case K = I d, so that A = I d as well, and C = v; supv, φλ ≤ 1 . λ

Moreover, in the real case

PC v, φλ =

v, φλ signv, φλ

if |v, φλ | ≤ 1, if |v, φλ | > 1.

This implies that SαAC ◦ AK ∗ is exactly the soft thresholding operator

SαAC (AK ∗ f ), φλ = Sα f, φλ . In the complex case, PC v, φλ =

v, φλ v,φλ |v,φλ |

if |v, φλ | ≤ 1, if |v, φλ | > 1,

34

DAUBECHIES ET AL .

and the SαAC ◦ AK ∗ reduces to the complex soft thresholding operator; that is,

SαAC (AT ∗ f ), φλ = Scα f, φλ f,φλ f, φλ − α |f,φ if |f, φλ | > α, λ | = 0 if |f, φλ | ≤ α. In the most interesting problems, the operator K ∗ K does not have a bounded inverse. Then the surrogate functionals introduced in Daubechies and Teschke (2004) can be used. Replacing Eq. (29) by a family of surrogate functionals yields 2 Gn,C (v) = FC (v) + vn − v2 − K(vn − v)H = −2v, K ∗ f + 2α sup v, h − 2v, vn + 2v, K ∗ Kvn h∈C

+ v + f + vn 2 − Kvn 2H , 2

2

and we have Proposition IV.1. Let C be as assumed in Section IV.A.1. Then the minimizer of Gn,C is given by vn+1 := (I d − PαC )(vn + K ∗ f − K ∗ Kvn ).

(33)

As mentioned above, Combettes and Wajs (2005) contains an extensive discussion, including (not easily verifiable) conditions that ensure strong convergence for an iteration of the type in Eq. (33). However, the full generality of Combettes and Wajs (2005) makes it less easy to read for the special case discussed here. Since the iteration is very similar to the one in Daubechies and Teschke (2004), a very similar strategy for the proof of convergence holds as well. It can be retraced in Daubechies et al. (2007) that up to strong convergence the techniques apply in almost the same way (weak convergence is achieved by applying Opial’s theorem; see Opial, 1967). Achieving norm convergence requires more attention to the structure of C, however. It may be argued that weak convergence suffices for practical purposes, because every numerical computation is always finite-dimensional so that weak and strong (i.e., norm) convergence of the vn are equivalent. However, it is often useful to establish norm convergence for the infinite dimensional Hilbert space as well, since this then implies that the rate of convergence, and the other constants involved, do not “blow up” as the dimensionality of the discretization increases.

35


Obtaining norm convergence requires more work. The following facts can be verified in Daubechies et al. (2007): • • • •

weak −−→ v, ¯ for n → ∞, vn − ¯ v¯ = v¯ + K ∗ f − K ∗ K v¯ − PαC (v¯ + K ∗ f − K ∗ K v), vn+1 = vn + K ∗ f − K ∗ Kvn − PαC (vn + K ∗ f − K ∗ Kvn ), vn+1 − vn → 0, for n → ∞.

Defining un := vn − v¯

and w := v¯ + K ∗ f − K ∗ K v, ¯

(34)

we can recast these facts as follows: weak un − −−→ 0, for n → ∞, PαC (v) − PαC (w + un − K ∗ Kun ) − K ∗ Kun → 0,

for n → ∞.

Then, without any change, Lemmas 3.15 and 3.17 of Daubechies et al. (2004) can be applied, leading to K ∗ Kun → 0,

for n → ∞,

yielding the equivalent formulation weak un − −−→ 0, for n → ∞, PαC (w) − PαC (w + un ) → 0,

for n → ∞.

(35)

To obtain norm convergence of the vn , we must establish un → 0. For general convex sets C the conditions Eq. (35), where α > 0 and w ∈ H are arbitrary (but fixed), actually do not imply norm convergence of the un to 0. Abstract sufficient and necessary conditions for norm convergence are given in Combettes and Wajs (2005); the following theorem (for a proof, see Daubechies et al., 2007) gives a more concrete restriction on C under which we can establish norm convergence. weak Theorem IV.2. Suppose un − −−→ 0 and PαC (w) − PαC (w + un ) → 0. Moreover, assume that un is orthogonal to w, PC (w). If for some sequence γn (with γn → ∞) the convex set C satisfies γn un ∈ C, then un → 0.

Unfortunately, this theorem is not sufficiently strong to be applied to the BV functional of Example 2 above. Without full detail, we show here how it (just) falls short. The set C in Example 2 is (loosely speaking) the set of all divergences of 2D fields that are uniformly bounded by 1. In particular, it contains the

36

DAUBECHIES ET AL .

functions √ 2π |n1 | + |n2 | e2πi(n1 x+n2 y) 1 1 = −i div √ sign(n1 )e2πi(n1 x+n2 y) , √ sign(n2 )e2πi(n1 x+n2 y) , 2 2

hn (x, y) :=

where |n1 | + |n2 | = 0. Because C is closed and convex, it also contains all the αn h n , n∈Z2

with n∈Z2 αn = 1. Suppose now (just for the sake of simplifying the argument, which can also be made, a bit more lengthily, without this assumption) that PC (un ) → 0 as n → ∞, that is, that the condition PαC (w) − PαC (w + un ) → 0

as n → ∞

holds true for w = 0. That would mean that, for all g ∈ C lim un − PC (un ), g − PC (un ) n→∞ 2 = lim un , g + PC (un ) − un , PC (un ) − PC (un ), g n→∞

= lim un , g , n→∞

which implies that limn→∞ un , g is nonpositive. Since the same is true for −g ∈ C, it follows that limn→∞ un , g = 0 for all g ∈ C. Consequently, u n , hk → 0 as n → ∞, or even, for all sequences (αk )k∈Z2 with k∈Z2 αk = 1, αk ∈ [0, 1] ∀k, αk |k1 | + |k2 | un , ek → 0 as n → ∞, k

where ek (x, y) = e2πi(k1 x+k2 y) . This just misses ensuring that un , ek 2 → 0 as n → ∞. k

This concludes the theoretical analysis of our first case described in the Introduction (where J (f ) in Eq. (29) is convex).


37

C. Numerical Illustrations 1. Iterative Algorithm for PDE-Based Deblurring and Denoising In the framework of Rudin and Osher (1994), the edge-preserving energy functional is of the form 2 F (v) = f − KvL2 (Ω) + 2α φ(∇f ) dx, (36) Ω

→ R is typically a positive continuous funcwhere the potential tion, with at most linear growth at infinity. Convex examples include (note that, only for illustration’s sake, we also give examples beyond the onehomogeneous case): φ : R2

• φ(ξ ) = |ξ | (the total variation minimization; (see Rudin et al., 1992; Rudin and Osher, 1994), • φ(ξ ) = |ξ1 | + |ξ2 |, • φ(ξ ) = 1 + |ξ |2 (the function of minimal surfaces; (see Aubert and Vese, 1997; Vese, 2001), • φ(ξ ) = log cosh(1 + |ξ |2 ), or 1 |ξ |2 if |ξ | 1, • φ(ξ ) = 2 |ξ | − 12 if |ξ | 1 (used in Demengel and Temam, 1984; Chambolle and Lions, 1997). In the nonconvex case, examples of the potential φ are |ξ | • φ(ξ ) = 1+|ξ |p , or • φ(ξ ) = log(1 + |ξ |p ), with p = 1 or p = 2 for instance, see Geman and Geman (1984), Geman and Reynolds (1992), Perona and Malik (1990), and more recently, Aubert and Vese (1997). p

Let us now restrict again to the one-homogeneous case and assume in addition that φ is differentiable. Then the Euler–Lagrange equation associated with the minimization problem Eq. (36), which must be satisfied by a minimizer v, if such a minimizer exists, is given by

(37) K ∗ Kv − K ∗ f = α div ∇ξ φ(∇v) , in Ω, where ∇φξ = (φξ1 , φξ2 ), and with the boundary conditions ∇ξ φ(∇v) · n = 0 on ∂Ω, where n is the unit exterior normal to the boundary. In the case α > 0, the partial differential equation (37) is nonlinear for the examples of potential φ given above. Moreover, the presence of the term K ∗ Kf makes it computationally expensive and numerically nontrivial.

38

DAUBECHIES ET AL .

To overcome these problems, we propose here to not directly solve Eq. (37) numerically but to apply the surrogate functional algorithm (see Daubechies et al., 2004, or the previous sections); that is, we construct a sequence of iterates vn that approximate v, without needing to invert K ∗ K at every iteration. Conversely, the direct implementation of the projection PαC associated with our minimization is rather complicated. In this case we prefer to avoid it by switching to an expression based on the Euler–Lagrange equation. The total iteration is thus as follows: start with an initial v0 ; find vn , n > 0, as a minimizer of the surrogate functionals

Gn−1 (vn ) = f − Kvn 2L2 (Ω) − Kvn − Kvn−1 2L2 (Ω) + vn − vn−1 2L2 (Ω) + 2α φ(∇vn ), (38) Ω

where we have assumed that K ∗ K < 1. The associated Euler–Lagrange equation in vn , now easily solved in practice, is:

vn = vn−1 + K ∗ v − K ∗ Kvn−1 + α div ∇ξ φ(∇vn ) , (39) together with the same boundary conditions. One then simply carries out this iterative algorithm to find (an approximation to) the desired minimizer. 2. Comparison of the Iteration Schemes with Besov and BV Constraints To illustrate the capabilities and differences with respect to reconstruction quality and computational cost, we present some numerical results of the Besov (wavelet framework) and the BV approach. We assume that the linear degradation model f = Kv + e, where f is the given data, as a square integrable function in L2 (Ω), v is the unknown true image; and e is additive noise of zero mean. The operator K : L2 (Ω) → L2 (Ω) models a linear and continuous degradation operator by a convolution with a Gaussian kernel. In the first approach, we chose a wavelet frame that is simply given by a translation invariant wavelet system and applied the iterative deconvolution scheme of Section II (see also Daubechies et al., 2004; Daubechies and Teschke, 2004, 2005). As the example image we consider a fingerprint and its blurred version (Fig. 10). The blur operator T used in the experiments has the discrete spatial representation given in Table 3. The results obtained with iteration from the previous section are shown in Fig. 11 and the convergence rates are given in Table 4. The blur convolution is easily implemented as a multiplication in Fourier domain, which means that we switch between the wavelet and Fourier representation at every step of the iteration process.


F IGURE 10.

39

Left: original image; right: blurred version.

TABLE 3 S PATIAL DISCRETIZATION OF THE BLUR OPERATOR K 0 0 0 0 0 1 0.25 60 0 0 0 0 0

0 0 0.25 0.5 0.5 0.5 0.5 0.5 0.25 0 0

0 0.25 0.5 1 1 1 1 1 0.5 0.25 0

0 0.5 1 1 1 1 1 1 1 0.5 0

0 0.5 1 1 1 1 1 1 1 0.5 0

0.25 0.5 1 1 1 1 1 1 1 0.5 0.25

0 0.5 1 1 1 1 1 1 1 0.5 0

0 0.5 1 1 1 1 1 1 1 0.5 0

0 0.25 0.5 1 1 1 1 1 0.5 0.25 0

0 0 0.25 0.5 0.5 0.5 0.5 0.5 0.25 0 0

0 0 0 0 0 0.25 0 0 0 0 0

F IGURE 11. Top left to bottom right: blurred image, several iterates using the wavelets scheme: 1st, 100th, 500th, 1000th, 2000th, 3000th, 4000th.

40

DAUBECHIES ET AL .

TABLE 4 C ONVERGENCE RATES FOR THE WAVELET FRAME - BASED SHRINKAGE ALGORITHM APPLIED TO THE BLURRY FINGERPRINT IMAGE

Iteration

CPU time

RMSE vn − vorig /vorig

1 100 200 350 500 1000 2000 3000 4000 5000

0.28600841 11.0832407 21.3800888 36.9959693 52.2572860 104.031806 207.406313 312.880765 419.051499 524.362921

0.17322202591876 0.16619377802673 0.16053661204032 0.15341521024097 0.14754897184399 0.13476786059559 0.12458884053496 0.11962522471006 0.11633033283685 0.11388107705039

F IGURE 12. Top left to bottom right:blurred image and several iterates using the surrogate functional iteration of Eq. (39) with φ(ξ ) = + |ξ |2 (total variation minimization with regularization): 1st, 100th, 500th, 1000th, 2000th, 3000th, 4000th.

Next, we present numerical results for the second (PDE) approach. Figure 12 shows the results of the iterative algorithm [Eq. (39)] on the same blurred and noisy image. For comparison with the purely PDEbased method (without the iterative approach corresponding to surrogate functionals), Fig. 13 shows the results of methods for Eqs. (39) and (37), which appear very similar. Table 5 lists the CPU time and the relative root mean squared error (RMSE) for the first 5000 iterations of both methods, illustrating that the surrogate functional method produces a better error decay


41

F IGURE 13. Deblurring results obtained using the models in Eq. (39) (left) and Eq. (37) (right) with φ(ξ ) = + |ξ |2 (total variation minimization with regularization).

TABLE 5 C OMPARISON OF THE CONVERGENCE RATES FOR PDE- BASED APPROACH Iteration

CPU time

RMSEvn − vorig /vorig

1 100 200 350 500 1000 2000 3000 4000 5000

With surrogate functionals 0 53.7299995 107.440002 188.470001 268.699982 542.880005 1085.82996 1625.08997 2151.29004 2684.62988

0.256365806 0.22643654 0.213100523 0.19897379 0.188747868 0.168787047 0.153525516 0.148305818 0.146324202 0.145639017

1 100 200 350 500 1000 2000 3000 4000 5000

Without surrogate functionals 0.400000036 55.1000023 109.139999 191.959991 276.119995 551.720032 1105.32996 1666.43994 2234.70996 2790.92993

0.257728785 0.236117765 0.226667866 0.216331303 0.208168939 0.189164072 0.169159457 0.159271181 0.15375042 0.150457606

Set as for both algorithms (the classical PDE iteration and the Gaussian surrogate functional iteration), we list the number of iterations, the CPU time, and the corresponding relative RMSE applied to the blurry fingerprint image, using the total variation minimization. The new method using the surrogate functionals converges faster to the restored image: the relative RMSE vn − vorig /vorig hits the value 0.15 at 2500 iterations instead of 5000, and uses a CPU time of ∼1300 instead of 2790; there thus seems to be a speed-up factor of 2.

42

DAUBECHIES ET AL .

for the same amount of CPU time. (These two computations were performed out on the same machine; note that the numerical results in Fig. 11 were obtained on a different computer and should thus not be compared with this table.)

V. H YBRID WAVELET-PDE I MAGE R ESTORATION S CHEMES Section II showed a constructed wavelet-based scheme that solves the variational problem in Eq. (3), inf Ff (v, u), u,v

where

2 Ff (v, u) = f − K(u + v)L2 (Ω) + γ v2H −1 (Ω) + 2α|u|B 1 (L1 (Ω)) , 1

in a numerically very efficient way. As discussed in Section II.D, nonredundant wavelet filtering often creates artifacts that manifest as ringing and edge blurring. The suggested method to reduce these artifacts while maintaining a very low computational cost level was by introducing redundant wavelet systems. Another way to present sharp edges, as mentioned in Section II.A, is to model the cartoon part of the image as a function of bounded variation. Up to the linear operator K, this coincides with Eq. (2), 2 inf Ef (v, u) = f − K(u + v)L (Ω) + γ v2H −1 (Ω) + 2α|u|BV(Ω) (40) 2

(v,u)

and was considered in Vese and Osher (2003) and Osher et al. (2003). Instead of solving Eq. (40) in a pure PDE fashion (as done in Vese and Osher, 2003, and Osher et al., 2003), we propose a combined wavelet-PDE scheme, which maintains the advantage of wavelets to well represent oscillatory patterns and simple minimization, with the added advantage of nonlinear PDE formulation, which preserves sharp edges and representation of functions of bounded variation. In the proposed alternating scheme, the minimization in u is solved in a PDE function, by finite differences, while the minimization in v is solved in a wavelet function. The data function f is known in the spatial domain f (x1 , x2 ) ≈ fi,j , but also in the frequency domain, by its wavelet coefficients (fλ )λ∈J . Let us assume K = I for the moment. Keeping v = v n fixed, n 0, we compute u = un minimizing (f − v) − u2 dx + 2α |∇u| dx, Ω

Ω


43

solving the Euler–Lagrange equation by finite differences ∇u . u = (f − v) + α div |∇u| Now keeping u = un fixed, compute its wavelet coefficients (uλ )λ∈J , and compute v = v n+1 minimizing with respect to v (fλ − uλ ) − vλ 2 + γ 2−2|λ| |vλ |2 . λ∈J

This leads to decoupled 1D minimizations and yields the desired v n+1 = v˜γ ,α as before, from its wavelet coefficients vλ . Then the steps are repeated. A. Combined Wavelet-PDE Scheme in the Presence of Blur Now consider the problem 2 inf Gf (v, u) = f − K(u + v)L (v,u)

2 (Ω)

+ γ v2H −1 (Ω) + 2α|u|BV(Ω) ,

(41) in the combined wavelet-PDE scheme; again, we would like to apply the surrogate functionals for the minimization. One way to minimize this and to avoid inverting operators involving K ∗ K at every iteration is to consider the unknown pair (v, u) and to directly apply the surrogate functionals approximation; knowing (vn−1 , un−1 ), find (vn , un ) minimizer of 2 Gf,n (vn , un ) = f − K(un + vn )L2 (Ω) + γ vn 2H −1 (Ω) + 2α|un |BV(Ω) 2 − K(un + vn ) − K(un−1 + vn−1 )L (Ω) 2 2 + μ(un + vn ) − (un−1 + vn−1 )L (Ω) , 2

where μ is such that K ∗ K μ. 1. Characterization of Minimizers Theoretically, we consider slightly modified Ff (v, u) and Ef (v, u), as 2 Gf (v, u) = f − K(u + v)L (Ω) + γ vH −1 (Ω) + 2α|u|B(Ω) , (42) 2

where |u|B(Ω) is one of the semi-norms |u|BV(Ω) or |u|B 1 (L1 (Ω)) . In this 1 slightly modified case, we have the following characterization of minimizers (inspired from Meyer, 2002; Le and Vese, 2005): let f ∈ L2 (Ω), u ∈ B(Ω) ⊂ L2 (Ω), v ∈ L2 (Ω) ∩ H −1 (Ω). We introduce the texture norm.

44

DAUBECHIES ET AL .

Definition V.1. Given w ∈ L2 (Ω), γ , α > 0, define (w, g + h) w∗ = sup , 2α|g|B(Ω) 2α|g|B(Ω) + γ hH −1 (Ω) g∈B(Ω), h∈H −1 (Ω)∩L2 (Ω)

+ γ hH −1 (Ω) = 0 ,

where (·,·) is the L2 (Ω) inner product. Then we can show the following characterization of minimizers for the two models: Theorem V.1. Let w = f − K(u + v). Then 1. K ∗ f ∗ ≤ 12 if and only if (v, u) = (0, 0) is a minimizer of Eq. (42). 2. Suppose K ∗ f ∗ > 12 . Then (v, u) is a minimizer of Eq. (42) if and only if K ∗ w∗ =

1 2

and (K ∗ w, u + v) =

1 2α|u|B(Ω) + γ vH −1 (Ω) . 2 (43)

Proof. If the model in Eq. (43) yields minimizers u = 0 and v = 0, then for any g ∈ B(Ω), h ∈ H −1 (Ω) ∩ L2 (Ω), 2 f 2L2 (Ω) ≤ 2α|g|B(Ω) + f − K(g + h)L (Ω) + γ hH −1 (Ω) . (44) 2

Substituting in Eq. (44) g by g and h by h, and taking → 0+ , we obtain ∗

(K f, g + h) ≤ 1 2α|g|B(Ω) + γ h −1 (45) H (Ω) . 2 By the definition of .∗ , therefore K ∗ f ∗ ≤ 12 . For the converse property in Step 1, assume that K ∗ f ∗ 12 . Then, for any g ∈ B(Ω) and h ∈ H −1 (Ω) ∩ L2 (Ω), with 2α|g|B(Ω) + γ hH −1 (Ω) = 0, we have

(K ∗ f, g + h) 2α|g|B(Ω) + γ hH −1 (Ω) f ∗ 1 2α|g|B(Ω) + γ hH −1 (Ω) . 2 We also have 2 2α|g|B(Ω) + f − K(g + h)L (Ω) + γ hH −1 (Ω) 2 2 2 = 2α|g|B(Ω) + f − 2(K ∗ f, g + h) + K(g + h) L2 (Ω)

L2 (Ω)

45


+ γ hH −1 (Ω)

2α|g|B(Ω) + f 2L2 (Ω) − 2α|g|B(Ω) + γ hH −1 (Ω) 2 + K(g + h)L (Ω) + γ hH −1 (Ω) 2 2 2 = f L2 (Ω) + K(g + h)L (Ω) f 2L2 (Ω) = Gf (0, 0). 2

Therefore, u = 0 and v = 0 gives the optimal decomposition in this case. Now suppose K ∗ f ∗ > 12 . Let (v, u) be an optimal decomposition given by Eq. (43). We have u ≡ 0 or v ≡ 0. For g ∈ B(Ω), h ∈ H −1 (Ω) ∩ L2 (Ω), and ∈ R, 2 Gf (v + h, u + g) = 2α|u + g|B(Ω) + w − K(g + h)L (Ω) 2

+ γ v + hH −1 (Ω)

≥ 2α|u|B(Ω) + w2L2 (Ω) + γ vH −1 (Ω) ,

(46)

and thus, by triangle inequality, 2α|u|B(Ω) + w2L2 (Ω) + γ vH −1 (Ω) 2 ≤ 2α|u|B(Ω) + 2α|||g|B(Ω) + w − K(g + h)L (Ω) 2

+ γ vH −1 (Ω) + ||hH −1 (Ω) . Therefore,

2 w2L2 (Ω) ≤ 2α|||g|B(Ω) + w − K(g + h)L

2 (Ω)

+ γ ||hH −1 (Ω) ,

and thus, w2L2 (Ω) ≤ 2α|||g|B(Ω) 2

+ w2L2 (Ω) − 2(K ∗ w, g + h) + 2 K(g + h)L + γ ||hH −1 (Ω) ,

2 (Ω)

yielding 2 2α|||g|B(Ω) − 2(K ∗ w, g + h) + 2 K(g + h)L ≥ 0.

2 (Ω)

+ γ ||hH −1 (Ω)

Dividing both sides of the latter equation by > 0, we obtain 2 −2(K ∗ w, g + h) + K(g + h)L (Ω) + 2α|g|B(Ω) + γ hH −1 (Ω) ≥ 0. 2

Taking → 0, yields 2(K ∗ w, g + h) ≤ 2α|g|B(Ω) + γ hH −1 (Ω) ,

46

DAUBECHIES ET AL .

for all g ∈ B(Ω), h ∈ H −1 (Ω) ∩ L2 (Ω) and thus, 1 . (47) 2 If we take −1 < < 0, and replace (h, g) with (v, u) in Eq. (46), then it implies 2

2(K ∗ w, u + v) ≤ 2α|u|B(Ω) + γ vH −1 (Ω) + 2 K(u + v)L (Ω) . K ∗ w∗ ≤

2

(48) Dividing again by < 0, we obtain 2(K ∗ w, u + v) ≥ (2α|u|B(Ω) + γ vH −1 (Ω) ). Therefore equality holds, (K ∗ w, u + v) =

1 2α|u|B(Ω) + γ vH −1 (Ω) , 2

(49)

and Eq. (49) together with Eq. (47) implies K ∗ w∗ = 12 . Conversely, if Eq. (43) holds for some (v, u) and K ∗ w∗ = 12 , then for any g ∈ B(Ω), h ∈ H −1 (Ω) ∩ L2 (Ω), 2 2α|u + g|B(Ω) + w − K(g + h)L + γ v + hH −1 (Ω) 2

∗

≥ 2(K w, u + g + v + h) + w2L2 (Ω) − 2(K ∗ w, g + h) 2 + 2 K(g + h)L (Ω) 2 2 ∗ = 2(K w, u + v) + w2L2 (Ω) + 2 K(g + h)L (Ω) 2 2 2 2 = 2α|u|B(Ω) + γ vH −1 (Ω) + wL2 (Ω) + K(g + h)L (Ω) 2

≥

2α|u|B(Ω) + γ vH −1 (Ω) + w2L2 (Ω)

= Gf (v, u).

Therefore, (v, u) is an optimal decomposition and minimizer of Eq. (42). B. Numerical Illustration Figure 14 shows a numerical result obtained with the proposed hybrid approach for the case K = I . The proposed scheme is numerically stable and faster than the method proposed in Osher et al. (2003), where the minimization was solved by a fourth-order nonlinear PDE with restrictive CFL condition. The method is also simpler than the method from Vese and Osher (2003) with p = 2. As expected, the proposed method gives better cartoon and texture separation than by the ROF method (Rudin et al., 1992 corresponding to γ = 0, v = 0).


47

F IGURE 14. Image decomposition into cartoon and texture by the combined wavelet-PDE scheme, with parameters J = 6, γ = 0.01, α = 3, 20 iterations. Top: original image f (left) and plot of un+1 − un , v n+1 − v n . Middle: u and f − u obtained by the PDE scheme only (γ = 0, v = 0). Bottom: u and v obtained by the combined scheme.

48

DAUBECHIES ET AL .

R EFERENCES Ambrosio, L., Fusco, N., Pallara, D. (2000). Functions of Bounded Variation and Free Discontinuity Problems. Oxford Univ. Press, Oxford. Anthoine, S. (2005). Different wavelet-based approaches for the separation of noisy and blurred mixtures of components. Application to astrophysical data. PhD thesis. Aubert, G., Aujol, J.-F. (2005). Modeling very oscillating signals. Applications to image processing. Applied Mathematics and Optimization 51 (2), 163. Aubert, G., Vese, L. (1997). A variational method in image recovery. SINUM 34 (5), 1948–1979. Bect, J., Blanc-Féraud, L., Aubert, G., Chambolle, A. (2004). A l(1)-unified variational framework for image restoration. Lecture Notes in Computer Science 2034, 1–13. Boyd, S., Vandenberghe, L. (2005). Convex Optimization. Cambridge Univ. Press, Cambridge. Chambolle, A. (2004). An algorithm for total variation minimization and applications. Journal of Mathematical Imaging and Vision 20 (1–2), 89. Chambolle, A., Lions, P.-L. (1997). Image recovery via total variation minimization and related problems. Numerische Mathematik 76 (2), 167. Chen, S., Donoho, D., Saunders, M. (1999). Atomic decomposition by basis pursuit. SIAM Journal on Scientific Computing 20 (1), 33–61. Cohen, A., Daubechies, I., Feauveau, J.-C. (1992). Bi-orthogonal bases of compactly supported wavelets. Communications on Pure and Applied Mathematics 45, 485–560. Cohen, A., DeVore, R., Petrushev, P., Xu, H. (1999). Nonlinear approximation and the space BV(R2 ). American Journal of Mathematics 121, 587–628. Coifman, R., Donoho, D. (1995). Translation-invariant de-noising. In: Antoniadis, A., Oppenheim, G. (Eds.), Wavelets and Statistics. Springer-Verlag, New York, pp. 125–150. Combettes, P.L., Wajs, V.R. (2005). Signal recovery by proximal forward– backward splitting. Multiscale Modeling and Simulation 4 (4), 1168. Dahmen, W. (1996). Stability of multiscale transformations. Journal of Fourier Analysis and Applications 2, 341–361. Daubechies, I. (1992). Ten Lectures on Wavelets. SIAM, Philadelphia. Daubechies, I. (1993). Wavelet transforms and orthonormal wavelet bases. Proceedings of Symposia in Applied Mathematics 47. Daubechies, I., Teschke, G. (2004). Wavelet-based image decomposition by variational functionals. In: Truchetet, F. (Ed.), Wavelet Applications in Industrial Processing. In: Proceedings of SPIE, vol. 5266, pp. 94–105.


49

Daubechies, I., Teschke, G. (2005). Variational image restoration by means of wavelets: Simultaneous decomposition, deblurring and denoising. Applied and Computational Harmonic Analysis 19 (1), 1–16. Daubechies, I., Defrise, M., DeMol, C. (2004). An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics 57, 1413–1541. Daubechies, I., Teschke, G., Vese, L. (2007). Iteratively solving linear inverse problems with general convex constraints. Inverse Problems and Imaging 1 (1), 29–46. Daudet, L., Torrésani, B. (2002). Hybrid representations for audiophonic signal encoding. In: Special Issue on Image and Video Coding Beyond Standards. Signal Processing 82 (11), 1595–1617. Defrise, M., DeMol, C. (2004a). Inverse imaging with mixed penalties. Conference proceedings. Defrise, M., DeMol, C. (2004b). Linear inverse problems with mixed smoothness and sparsity constraints. Preprint. Demengel, F., Temam, R. (1984). Convex functions of a measure and applications. Indiana University Mathematics Journal 33 (5), 673. DeVore, R., Jawerth, B., Popov, V. (1988). Interpolation of Besov spaces. Transactions of the American Mathematical Society 305, 397–414. DeVore, R., Jawerth, B., Popov, V. (1992). Compression of wavelet decompositions. American Journal of Mathematics 114, 737–785. Ekeland, I., Témam, R. (1999). Convex Analysis and Variational Problems. In: Classics in Applied Mathematics, vol. 28. Society for Industrial and Applied Mathematics, Philadelphia. Elad, M., Starck, J.-L., Querre, P., Donoho, D. (2005). Simultaneous cartoon and texture image in painting using morphological component analysis (MCA). Applied and Computational Harmonic Analysis 19, 340–358. Evans, L., Gariepy, R. (1991). Measure Theory and Fine Properties of Functions. CRC Press, Boca Raton, FL. Fernandes, F.C.A., von Spaendonck, R., Coates, M.J., Burrus, S. (2000). Directional complex-wavelet processing. In: Proceedings of Society of Photo-Optical Instrumental Engineers – SPIE2000, Wavelet Applications in Signal Processing VIII. San Diego, California. Fornasier, M., Rauhut, H. (2006). Recovery algorithms for vector valued data with joint sparsity constraint. Preprint. Frazier, M., Jawerth, B. (1990). A discrete transform and decompositions of distribution spaces. Journal of Functional Analysis 93, 34–170. Geman, D., Reynolds, G. (1992). Constrained restoration and the recovery of discontinuities. IEEE Transactions on PAMI 14 (3), 367–383. Geman, S., Geman, D. (1984). Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on PAMI 6 (6), 721–741.

50

DAUBECHIES ET AL .

Hiriart-Urruty, J.-B., Lemaréchal, C. (1993). Convex Analysis and Minimization Algorithms I: Fundamentals. Springer, Berlin. Jaillet, F., Torrésani, B. (2005). Time-frequency jigsaw puzzle: Adaptive multiwindow and multilayered Gabor expansions. Preprint. Kinsbury, N. (1999). Image processing with complex wavelets. In: Walvets: The Key to Intermittent Information? No. 1760. In: Philosophical Transactions: Mathematical, Physical and Engineering Sciences, vol. 357. The Royal Society, pp. 2543–2560. Sep. 15. Le, T., Vese, L. (2005). Image decomposition using total variation and div(BMO). SIAM Multiscale Modeling and Simulation 4 (2), 390–423. Mallat, S., Zhong, S. (1992). Characterization of signals from multiscale edges. IEEE Transactions on Pattern Analysis and Machine Intelligence 14 (7), 710–732. Meyer, Y. (2002). Oscillating Patterns in Image Processing and Nonlinear Evolution Equations. University Lecture Series, vol. 22. AMS. Molla, S., Torrésani, B. (2005). A hybrid scheme for encoding audio signal using hidden Markov models of waveforms. Applied and Computational Harmonic Analysis 18 (2), 137–166. Opial, Z. (1967). Weak convergence of the sequence of successive approximations for nonexpansive mappings. Bulletin of the American Mathematical Society 73 (4), 591–597. Osher, S., Solé, A., Vese, L. (2003). Image decomposition and restoration using total variation minimization and the H −1 norm. SIAM Multiscale Modeling and Simulation 1 (3), 349. Perona, P., Malik, J. (1990). Scale space and edge detection using anisotropic diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence 12, 629–639. Rockafellar, R.T., Wets, R.J.-B. (1998). Variational Analysis. Springer, Berlin. Rudin, L., Osher, S. (1994). Total variation based image restoration with free local constraints. Proceedings ICIP, 31–35. Rudin, L., Osher, S., Fatemi, E. (1992). Nonlinear total variations based noise removal algorithms. Physica D 60, 259–268. Selesnick, I.W. (2001). Hilbert transform pairs of wavelet bases. IEEE Signal Processing Letters 8 (6), 170–173. Starck, J.-L., Elad, M., Donoho, D. (2005). Image decomposition via the combination of sparse representation and a variational approach. IEEE Transactions on Image Processing 14 (10), 1570–1582. Teschke, G. (2007). Multi-frame representations in linear inverse problems with mixed multi-constraints. Applied and Computational Harmonic Analysis 22, 43–60. Triebel, H. (1978). Interpolation Theory, Function Spaces, Differential Operators. Verlag der Wissenschaften, Berlin.


51

Vese, L. (2001). A study in the BV space of a denoising-deblurring variational problem. Applied Mathematics and Optimization 44 (2), 131–161. Vese, L., Osher, S. (2003). Modeling textures with total variation minimization and oscillating patterns in image processing. Journal of Scientific Computing 19 (1–3), 553–572. Vese, L., Osher, S. (2004). Image denoising and decomposition with total variation minimization and oscillatory functions. Journal of Mathematical Imaging and Vision 20 (1–2), 7–18.



Significant Advances in Scanning Electron Microscopes (1965–2007) R.F.W. PEASE Department of Electrical Engineering, Stanford University, Stanford, California 94305, USA

I. II. III. IV.

V. VI. VII. VIII. IX.

Introduction . . . . . . . . . . . . . . . . Brief Review of SEM Operation . . . . . . . . . . Contrast . . . . . . . . . . . . . . . . . Improvements since 1965 . . . . . . . . . . . . A. Sources . . . . . . . . . . . . . . . . B. Immersion Lens Detector . . . . . . . . . . . C. Retarding Field Optics, Low-Voltage Operation . . . . . D. Low-Loss Imaging . . . . . . . . . . . . . 1. Techniques for Studying Semiconductor Devices and Circuits E. Electron-Channeling Patterns . . . . . . . . . . F. Photon Collection-Cathodoluminescence . . . . . . . G. Phonon Collection . . . . . . . . . . . . . H. Environmental SEM . . . . . . . . . . . . . Tabletop SEM . . . . . . . . . . . . . . . Aberration Correction . . . . . . . . . . . . . Digital Image Processing . . . . . . . . . . . . Ultrahigh Vacuum . . . . . . . . . . . . . . Future Developments . . . . . . . . . . . . . References . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

53 55 58 61 61 63 64 68 69 70 72 73 74 75 75 79 81 81 83

I. I NTRODUCTION Today’s published scanning electron micrograph captions do not need to point out that a scanning (reflection) electron microscope (SEM) was used to record the image; the SEM is so widely used that it is usually unnecessary. This development is far removed from the early 1960s when the SEM was a laboratory curiosity that was more complicated than a conventional (projection) electron microscope (now known as a transmission electron microscope [TEM]) yet had resolution more than an order of magnitude worse. However, by 1965 it was recognized that resolution was not the only metric by which to judge a microscope. Useful information was the real metric and the SEM shone in this respect. The surface of almost any solid sample could be imaged in a way that matched our macroscopic viewing experience so that even at magnifications typical of a light microscope (< 1000×), the 53 ISSN 1076-5670 DOI: 10.1016/S1076-5670(07)00002-X


54

PEASE

depth of focus and the contrast formation when using secondary electrons as the output signal yielded far more information than a light microscope. One very mundane example was counting the hairs on the body of two variants of the Tribolium flour beetle. Although the hairs were visible in a light microscope, specular reflections from the body also were counted. No such problems were present with the SEM, and the entomologists’ reaction was the SEM pictures were “just like a textbook diagram” (Pease et al., 1966). Two competing techniques existed; one was to operate a conventional electron microscope in reflection. But the required angle of incidence was so large (i.e., a grazing angle) to give enough elastically scattered electrons to form an image that surfaces with rough topography did not yield useful information. The other technique was to form a replica of the surface and then view the replica in transmission in a conventional electron microscope. Replica technology was very successful but required much skill, and many surfaces (e.g., the body hairs of a beetle) were impossible to replicate. The SEM offered a solution to these difficulties. Thus, thanks partly to the very successful pioneering use of a custom-built SEM by the Pulp and Paper Research Institute of Canada (Smith, 2004), the Cambridge Instrument Company and JEOL companies both undertook to develop the SEM as commercial products. At the same time the laboratory of Charles Oatley at Cambridge University continued to research various aspects of the SEM, including a demonstration of better than 10-nm resolving power to establish a clear order of magnitude improvement over the resolution of the light microscope (Pease and Nixon, 1965; Oatley et al., 1965). Both the Cambridge Instrument Company and JEOL introduced commercial SEMs in 1965 and 1966, respectively. Numerous significant improvements, such as new sources, detectors and lens arrangements, and new imaging configurations, have been invented and incorporated since then. The following text provides a subjective selection of these developments. Only innovations that have directly affected the SEM per se are covered. Thus, one example of an omitted innovation is the use of a computercontrolled SEM as a pattern generator for nanostructure fabrication and photomask manufacture (Herriott et al., 1975).1 In addition the development of the scanning transmission electron microscope (STEM) is mentioned only briefly in terms of its impact on the SEM. Applications are covered only insofar as they have impacted the instrument itself. 1 For more up-to-date descriptions, review November–December issues of Journal of Vacuum Science and Technology.

ADVANCES IN SCANNING ELECTRON MICROSCOPES

(1965–2007) 55

F IGURE 1. Schematic view of SEM. The magnification control need be only an attenuator for the scan signals in the column. More recent models may use a computer monitor for display and update the brightness of each pixel as the scan progresses in the column.

II. B RIEF R EVIEW OF SEM O PERATION Much fuller descriptions of SEM operation can be found in Oatley et al. (1965), Oatley (1972), Reimer (1998), and Reichelt (1998, 2007). An image of the electron source is projected as a finely focused beam at the surface of the sample by one or more electron lenses (Fig. 1). The beam is scanned in a twodimensional (2D) raster across the sample surface, and the video signal can be derived from any measurable effect of the beam on the sample. Secondary electron emission is the usual effect, so by detecting the secondary electrons the image is built point-by-point on the display. Thus the magnification is simply the ratio of the size of the display scan divided by the size of the scan on the sample; increasing the magnification is accomplished by attenuating the scan signal to the deflection coils in the SEM column. At low magnifications (e.g., < 10,000×) the resolution observed in a micrograph is usually determined by the number of pixels in the image. At high magnifications the observed resolution is set by the volume of interaction of the beam with the sample and by the size of the beam at the sample

56

PEASE

surface. Thus the resolving power of the microscope is given by the diameter, d, of the beam at the sample. Several factors contribute to the value of d. Because the source has non-zero diameter and finite current density, there is the Gaussian diameter d0 set by the projected source diameter; making this smaller by shortening the focal length of lens 1 reduces the beam current to an unacceptable level as described below. Diffraction, spherical aberration, and chromatic aberration also contribute to the total beam diameter. How each of these contributions vary with the beam convergence angle, α, at the sample is illustrated in Fig. 2. How the contributions combine to give the total beam diameter has been the subject of some debate (Crewe et al., 1970) and the heavy line is an approximation. What is clear is that there is an optimum value of α that is usually between 1 and 10 mrad. The contributions, ds and dc , from spherical and chromatic aberration increase as

F IGURE 2. Schematic graphs showing variance in the different contributions to overall focused beam diameter with the semi-angle of convergence at the sample α. The diffraction disk dλ and the Gaussian disk (due to the non-zero source size) d0 both vary as α −1 . The chromatic disk dc varies as α, and the spherical aberration disk varies as α 3 . How these combine is discussed in Oatley et al. (1965), Oatley (1972), Reimer (1998), and Reichelt (2007). The curve of dtotal is merely a subjective curve. However, it does illustrate how the optimum value of a depends on the different contributions at different values of α. Obviously the positions of the different curves depends the values of the different parameters such as beam current at the sample, wavelength, and spherical and chromatic aberration coefficients.


(1965–2007) 57

α 3 and α, respectively, and are also proportional to the coefficients of spherical and chromatic aberration Cs and Cc . Chromatic aberration is also proportional to V /V , where eV is the landing energy of the electrons on the sample (e is the electronic charge). Axial astigmatism, due to rotational asymmetry in the focusing field, and beam wobble due to stray magnetic fields also degrade the resolution but can be effectively eliminated by the use of a stigmator and well-designed shielding, respectively. Electrostatic charging of the sample or of contamination in the column can cause the beam to wander erratically and give rise to timevarying axial astigmatism; this is a frequent cause of poor performance. With most sources the lenses form a demagnifying projection system so that it is only the aberrations of the final lens that are significant. In a welldesigned lens the values of Cc and Cs are approximately (within a factor of 2) the same as the focal length. In Fig. 3 the sample is outside the magnetic field

F IGURE 3. Contrast formation in the SEM (secondary electron emission mode). The top picture shows the setup in the SEM. Collection from areas marked B is efficient and hence appear bright. Regions marked I are not struck by the beam because they are hidden from the beam by other parts of the specimen. Regions marked D are struck by the beam but collection of secondary electrons from these areas is hampered by the specimen geometry and hence these areas appear dark. The lower picture shows the familiar macroscopic light optical analog and hence micrographs obtained using the arrangement at the top are easily interpreted and provide a vivid rendition of surface topography.

58

PEASE

of the lens to allow efficient collection of the secondary electrons. This means that the focal length of the final lens must be longer than that of the objective lens in a TEM where the sample is immersed in the magnetic field of the lens. Thus the aberrations of the SEM tend to be worse than in the TEM, leading to poorer resolution. As discussed in the following text, this need not always be the case. A further limitation of the SEM compared to the TEM is the serial nature of the point-by-point buildup of the picture. A high-quality image (that is, one in which a 5% change in intensity between resolvable picture elements (pixel) is discernible by the human eye) requires at least 10,000 quanta per pixel; these quanta might be grains in a photograph or dots on a display. If each primary beam electron generates one such quantum, then the beam must deliver 16e–16 C/pixel; to record a 4-M pixel picture in 100 s, the beam current must be 64 pA. This is the most favorable case. If each beam electron delivers, on the average, less than one picture quantum, then more current is required to maintain adequate picture quality. It is the minimum flux of quanta in the video chain that governs the shot noise in the image, so even if each primary electron generates more than one picture quantum then the primary beam current (or electron flux) governs the noise as in the first case above. As shown in Fig. 2, the finest resolution is obtained when the beam current is smallest (the limit being d0 = I 1/2 = 0) so there is a trade-off between signal-to-noise ratio (which is governed by beam current) and resolution. Fortunately, the SEM usually uses secondary electrons for the video signal and, as described below, the efficiency of generation of secondary electrons is high (∼1) and the electron detectors can detect single electrons, which allows approaching the best case above.

III. C ONTRAST One advantage of the scanning approach to building up the image is that the information used as the video signal need not be focused. Thus any measurable effect can be used. Examples include emission of secondary electrons (characterized as having electron energies < 50 eV), emission of backscattered electrons (characterized as having energies comparable with that of primary electrons), X-ray emission, induced current, and visible photon emission (cathodoluminescence). As mentioned, the SEM usually uses secondary electron emission because, as described in Figs. 3 and 4, this yields a very lifelike rendition of the surface topography, which is the most frequently sought information. The caption in


(1965–2007) 59

F IGURE 4. Comparison of TEM and SEM views. Comparison views of a clump of latex spheres are shown on the left. The TEM image has better resolution but, unlike the SEM image, cannot show how the spheres are situated. The views on the right are of a woven grid. The TEM image is essentially a shadow and deducing that the grid is woven would be a major undertaking, whereas the woven nature is obvious in the SEM image. In the SEM image notice also the carbon film stretched across the grid and appearing transparent because the SEM beam penetrates the film without significant spreading and can image the underlying grid structure. See Hayes and Pease (1968).

60

PEASE

F IGURE 5. Two configurations of secondary electron detection. On the left is the version described by Everhart and Thornley (1960). The sample is outside the field of the final lens allowing for non-flat samples and yielding strikingly vivid pictures of the surface topography as described in Figs. 3 and 4. However, the price is that the aberration coefficients of the final lens are larger than those of an immersion lens. On the right is the in-lens detector described by Kimura and Tamura (1967). This allowed detection of secondary electrons from a sample immersed in the magnetic field of the final lens, thus allowing the use of a lens with low aberrations but the sample is usually restricted in size. Some SEMs offer both configurations.

Fig. 3 contains a brief explanation as to why secondary electron collection provides such a lifelike image of the sample. Moreover, the generation of secondary electrons can be very efficient; coefficients of 1 can be achieved at low (e.g., 1 keV) primary beam energy. The secondary electron detector (Fig. 5, left) is usually a biased (to 10 kV) scintillator optically coupled to a photomultiplier (Everhart and Thornley, 1960). A low-energy (4 eV) secondary electron is attracted by the 300 V on the grid and is then accelerated into the biased scintillator to generate approximately 50 photons, which are then detected by the photomultiplier. Individual secondary electrons can be detected in this manner. The collection efficiency of secondary electrons depends also on the local fields at the surface of the sample so that not only topography but local voltage gradients (voltage contrast) can be determined. This is useful for examining integrated circuits but can also be deleterious if the sample surface is charging. When X-rays are detected and analyzed, an image that represents a map of the chemical elements present can be obtained; this practice predates the introduction of the commercial SEM using secondary electron detection. Unfortunately, the efficiency of production of X-rays is much lower than that


(1965–2007) 61

of secondary electrons, so large currents are required and the resolution of the instrument is degraded to about a micrometer or more. Such scanning X-ray microanalysis (as it is known in the United Kingdom) or electron microprobe analysis (as it is known in the United States) was already a well-developed discipline in itself by 1965. Goldstein et al. (2005) provide an excellent review article that covers both SEM and microanalysis. When the scanning beam lands near or on a p–n junction, the excited holeelectron pairs (HEPs) induce a junction current and this effect, electron beaminduced current (EBIC), was already discovered in 1958 but was not seriously exploited until after 1965. A good reference for this subject is Thong (1993). In summary, by 1965 the SEM using secondary electron detection was demonstrating a resolving power of 10 nm, although the resolution obtained also depended on the sample properties. The recording time was about one minute. The accelerating voltage was usually between 10 and 25 kV. EBIC, cathodoluminescence, and voltage contrast had been demonstrated but not yet seriously exploited. X-ray microanalysis was well established. The introduction of commercial SEMs in 1965–1966 marked the coming of age of the technology. The following text describes some of the more significant advances in SEM technology since that time.

IV. I MPROVEMENTS SINCE 1965 A. Sources The tungsten hairpin source had been the cathode of choice primarily because it could withstand vacua of 10−4 Torr that were typical for easily demountable vacuum systems using neoprene gaskets, yet still generate sufficient current density (> 1 A/cm2 ) for about 40 hours before burning out. Improved vacuum technology, including the introduction of Viton gaskets and turbomolecular pumps, allowed the introduction of better emitters. The first was the lanthanum hexaboride emitter LaB6 . The intrinsic advantages, high electron emission, and low evaporation rate of LaB6 had been known since 1951. But a major practical difficulty had been that hot LaB6 is very reactive and finding a suitable mount was not solved until Broers invented the gun structure in which the LaB6 is a sintered rod heated only at one end with the other, cool, end used for the mounting (Fig. 6) (Broers, 1967). With this structure current densities exceeding 20 A/cm2 were obtained for lifetimes exceeding 100 hours. A further refinement was the Vogel mount in which a single crystal of LaB6 was mounted on a graphite hairpin (graphite being one of the few materials that does not react with hot LaB6 ). The engineering of the Vogel mount was a serious challenge that was successfully met.

62

PEASE

F IGURE 6. The first practical electron gun using LaB6 as the emitter. The LaB6 is in the form of a rod heated at only one end and mounted at the cool end to avoid the LaB6 reacting with the mount (Broers, 1967).

The advantages of field emission had been known since the late 1930s, but attempts to operate field emitters in demountable vacua had been unsuccessful until Crewe et al. (1970) solved the problem using both ultrahigh-vacuum technology (e.g., copper gaskets) and carefully grown tungsten of the required crystal orientation. Activating the source involved a particular sequence of heating steps. The brightness (current density per unit solid angle) was orders of magnitude higher than with thermionic emitters (tungsten or LaB6 ). This was particularly attractive for the scanning transmission electron microscope (STEM) that Crewe was building since there was sufficient signal to use only a small fraction of the transmitted electrons for the video signal. By using energy analysis of the transmitted electrons he was able to locate individual atoms. A commercial version of this instrument built by Vacuum Generators proved very successful. Also, following Crewe’s success, commercial SEMs using cold field emitters were introduced by Hitachi and by Coates and Welter. The idea of combining field and thermionic electron emission had also existed for some time (notably at the General Electric Company), but the first successful implementation was the built-up zirconiated structure pioneered


(1965–2007) 63

by the electron beam lithography group at Hewlett Packard Laboratories and further developed by groups at Bell Labs and IBM. It has since been offered commercially by the FEI Company. This structure is more stable than cold field emitters and is more suitable for electron beam lithography where the required currents are higher than those in high-resolution SEMs. Its performance is somewhere between that of LaB6 and cold field emitters (Tuggle and Swanson, 1985; Swanson and Swanson, 1975). One overdue development in the late 1960s was that most people started to believe in the Boersch effect in which electron currents of high perveance (IV−3/2 ) lead to anomalously high energy spread, about an order of magnitude higher than that expected from the Maxwell–Boltzmann expression for tungsten at 3000 K (0.26 eV) (Boersch, 1954). In a thermionic gun the electrons are slowed after emission by the Wehnelt electrode to restrict the area of emission, and so there is a region of high perveance that leads to significant electron–electron interactions, and hence momentum transfer, to give rise to the anomalously high-energy spreads predicted by Boersch and even measured by his team. For reasons that are unclear there was widespread skepticism that the Boersch effect was significant; even as late as 1972 doubts were still expressed as to whether the effect was serious (Pfeiffer, 1972). No such slowing occurs in the field emission guns, and hence the energy spread is less than 0.5 eV as opposed to 3 to 5 eV characteristic for thermionic guns; therefore the improved resolution of the field emission SEMs is due to the lower energy spread as well as the high brightness. B. Immersion Lens Detector The previous review of the state of the art before 1965 pointed out that the resolution of the SEM was inferior to that of the TEM because the longer focal lengths of the final lens of the SEM compared with the objective of the TEM led to larger aberration coefficients. As also noted, the reason for the longer focal lengths was the desire to put the sample outside the lens field to facilitate the collection of secondary electrons. Kimura and Tamura (1967) recognized that even if the low-energy secondary electrons are trapped by the lens field it is still possible to accelerate them onto a scintillator for efficient detection. Hence the sample can be immersed in the lens field and lenses with aberration coefficients similar to those of TEM objectives can be used. Their arrangement is shown in Fig. 5. The conventional arrangement, also shown in Fig. 5, is still convenient as large samples cannot usually be inserted into the lens field. Thus several commercial SEMs have featured both detectors to allow the operator to choose according to the sample size. Usually the SEM’s final lens inner polepiece diameter is still larger than that of the objective of a

64

PEASE

TEM so that even with the upper detector the SEM’s resolving power cannot usually match that of a TEM. However, there has been a recent demonstration of resolving lattice images with an SEM operating at 30 kV using an in-lens detector (Ngo et al., 2007). In addition, some TEMs offer a scanning option with detection of secondary electrons when the sample is at the maximum magnetic field in the middle of the condenser-objective so the resolving power can approach that of a TEM, but the actual resolution observed is often lower because the secondary electrons emerge from an area a few nanometers in diameter; that is somewhat larger than the resolution of TEM (now down to 0.1 nm or even less). C. Retarding Field Optics, Low-Voltage Operation As described in the 1965 article the design of the final lens is crucial to achieve high resolution and the aberration coefficients were governed primarily by the geometry of the lens. Thus operation at low voltage was regarded as unattractive because chromatic aberration would increase because V /V would increase. But this assumed either an Einzel lens for an electrostatic lens instrument or a constant electron energy for a magnetic lens. This author wondered how Zworykin et al. (1942) had managed to achieve a claimed 50-nm resolution at 800-eV landing energy in 1942, but the answer did not become clear until 1966 when Engel (1966) showed photoelectron emission micrographs with 20-nm resolution. The answer was to run a cathode lens backwards – that is, retard the electrons before they land on the sample. In one version the retarding region is after the magnetic field and in the second the regions overlap. A straightforward analysis of the first version showed that the chromatic aberration coefficient is about the length of the retarding region times the ratio of landing energy/initial energy (Pease, 1967). In both cases the chromatic aberration coefficient is proportional to this ratio when the ratio is small (e.g., < 0.1). Thus lowering the landing energy increases V /V and proportionately reduces Cc and so, for the same value of α, leaves the value of dc unchanged. One disadvantage of these first retarding-field configurations was that the sample was immersed in a strong electric field and so needed to be flat and normal to the optical axis to avoid distorting the field. Much later it was found that the use of a guard ring electrode at the same potential as the sample allowed the sample to be in a field-free region with little loss of resolution (Cass et al., 1996). One example of such an arrangement is the Gemini column in which the sample can be at ground potential and a liner tube (“booster”) at a high positive potential (Fig. 8). The second polepiece of the final magnetic lens acts as the guard ring. With this arrangement sub-10-nm resolution at


(1965–2007) 65

F IGURE 7. The use of a retarding field just in front of the sample results in most of the focusing action coming from the parabolic paths of the electrons. This results in both Cs and Cc being proportional to the ratio of landing energy over accelerating potential and hence is very attractive for low landing energies. In the configuration shown here (Pease, 1967) the sample is immersed in the electrostatic field. But in later versions (Cass et al., 1996) it was possible to have the sample in a field-free region while retaining the optical advantages of the retarding field.

66

PEASE

F IGURE 8. Diagram of the optics of a commercial SEM using a retarding field lens for low-voltage operation (see Carl Zeiss website at http://www.zeiss.com).

landing energies of less than 1000 eV are possible (Fig. 9). This is important because the high secondary electron emission coefficients (> 1) at such a low landing energy facilitate the examination of insulating surfaces because any tendency to charge positively is eliminated when the low-energy (∼1– 10 eV) secondary electrons are attracted back to the surface. Pawley (1992), Joy (2006), and Reimer (1993) provide reviews of low-voltage SEM. A scanning tunneling microscope (STM) can be used to achieve the very finest beams at very low landing energies (Fig. 10). With the STM an asperity on the surface of a metallic tip is kept within a few nanometers of the sample surface by using the tunneling or field-emitted current to control the gap. Atomic resolution is possible at very low biases (e.g., 100 mV). As the bias increases, the tip is moved further from the sample surface to maintain a manageable current; this leads to a broadening of the beam at the sample. Practical operation of this configuration proved tricky and, as far as the author is aware, the STM is now used only at very low biases in the vacuum tunneling


(1965–2007) 67

F IGURE 9. Low-voltage micrograph taken with the SEM of Fig. 8 showing the high-resolution achievable (Carl Zeiss website at http://www.zeiss.com).

F IGURE 10. Schematic view of scanning tunneling microscope. Electrons tunnel either directly between tip and sample or from tip to vacuum and then to sample. Atomic resolution on bulk samples is regularly achieved. Scanning is achieved with x- and y-piezo drives and the all-important gap is controlled using feedback from the tunneling current to the z-piezo drive. In contrast to the SEM the finest resolution is obtained at the lowest electron energy because that is when the tip is closest to the sample and hence spreading is minimized.

68

PEASE

mode and under ultrahigh vacuum conditions. However, the invention of STM led to the whole field of scanning (mechanical) probe microscopy, which is beyond the scope of this chapter. It also led to the awarding of the 1986 Nobel Prize in Physics to Ernst Ruska, Gerd Binnig, and Heinrich Rohrer for the STM and the TEM. For further information, see Meyer et al. (2004). D. Low-Loss Imaging Another innovation to achieving high resolution was to use as a video signal only elastically backscattered electrons (Fig. 11). These emerge from the sample mostly from a region within a few nanometers from the point of impact of the beam. One advantage of this technique is that any thin film contamination on the surface of the sample, which can significantly affect secondary electron emission, has little effect on elastic backscattering. This technique led to the demonstration of about 2.5-nm resolution, which represented a significant advance. In an ingenious arrangement the electron detector was configured to use the magnetic field of the final lens as the energy filter (Wells et al., 1973).

F IGURE 11. Arrangements for building up an image from only elastically scattered electrons. Filtering is accomplished using the magnetic field of the final lens. The addition of the curved grids improves the collection of the elastically scattered electrons (Wells et al., 1973).


(1965–2007) 69

1. Techniques for Studying Semiconductor Devices and Circuits One of the biggest applications for the SEM is the study of semiconductor structures, devices, and circuits. Much of this work involves secondary electrons as the video signal because determining surface topography of structures less than 32 nm wide but more than 100 nm high is critical to achieving a manufacturing capability in state-of-the-art integrated circuits. However, a number of techniques have exploited other measurable effects of the beam on the sample. One, already mentioned, is voltage contrast, which was first described in 1958 (Everhart et al., 1959). An embellishment of this technique was to apply the voltage as an alternating voltage and phase-lock the video amplifier to the applied voltage. This allowed detection of voltages as low as a few millivolts (Fig. 12) (Saparin, 1966). In addition, the beam could be used to inject a signal by generating a local source of HEPs and, again by phase-locking the output to the frequency of the modulated beam, it was possible to enhance the sensitivity of detection of the EBIC. Variations in the value of the induced current could be used

F IGURE 12. Schematic view of voltage contrast obtained by phase-locking the video amplifier to the signal applied to the sample. It is also possible to blank the primary beam with the signal variably delayed and achieve a stroboscopic image (Tuggle and Swanson, 1985; Thong, 1993).

70

PEASE

F IGURE 13. Display of a commercial, stroboscopic, electron beam testing system from Schlumberger showing the voltage contrast images (bottom) and the voltage traces (top left) and the input conditions (top right). These systems are a form of sampling oscilloscope. Picture courtesy of Neil Richardson.

to detect local defects such as clusters of recombination-generation centers. Several companies developed a range of electron beam testing systems for analyzing integrated circuits; one example is shown in Fig. 13. However, these techniques are no longer in widespread use. One reason is that most of the circuit is now buried under several levels of metalization. Another suggested reason is that voltage contrast is less detectable as the circuit nodes have shrunk to below 65 nm. E. Electron-Channeling Patterns With single-crystal samples the backscatter coefficient over a limited collection angle depends slightly on the angle of incidence of the primary beam. Hence as the beam scans over the surface of a sample, the beam lands on different parts of the sample at different angles and the resulting variations in the collected backscattered electrons show up as arc-shaped lines. These were observed and reported first by Coates (1969) of the Royal


(1965–2007) 71

(a)

(b) F IGURE 14. (a) Low-magnification SEM images of a silicon sample whose surface has first been amorphized by nonselective ion bombardment and then selectively annealed in the rectangular area by a high-current electron beam. (b) The control sample, unimplanted unannealed (100) Si (c) corner pf the annealed area. The channeling patterns are clearly visible only in the annealed regions (Ratnakumar et al., 1979).

Radar Establishment. The appearance of these lines, as well as the detailed mechanism of their formation, are similar to Kikuchi lines, already well known in electron diffraction (Venables and Harland, 1973). They can be used to determine the crystallinity of samples (Joy et al., 1982). An example of their use is determining the area of annealed ion implant damage as shown in Fig. 14. The lines are visible only within the rectangular area previously annealed by a high-current (> 10 µA) scanned beam (Ratnakumar et al., 1979). Another application is in determining the crystal orientation of very

72

PEASE

(c) F IGURE 14.

(continued)

small areas; for this the scanning arrangement must be modified so that the beam rocks about one spot on the sample. Selected areas as small as 1 µm across can be analyzed in this manner. F. Photon Collection-Cathodoluminescence The use of emitted X-ray photons has been mentioned previously. Visible photons can also form a signal. Indeed a very early SEM was used in this manner to examine phosphors (Davoine et al., 1960). Most applications using cathodoluminescence have been for examining direct bandgap semiconductors; this application was reported as early as 1964 (Wittry and Kyser, 1964) but did not come readily available until the 1970s. By exciting HEPs and observing the recombination radiation it is possible to derive information about the local lifetimes of carriers in much the same way as with EBIC contrast but without the need for a nearby p–n junction. One common pitfall in modifying the SEM for cathodoluminescence contrast is that so many materials luminesce under electron bombardment that often it is spurious light from scattered electrons striking, for example, the light pipe that forms most of the signal. This author used an arrangement in which light from the sample


(1965–2007) 73

F IGURE 15. SEM picture of spinach leaf whose cell walls are selectively stained with thioflavin-T, a fluorescent stain that luminesces under electron bombardment. (240× original magnification on 100-mm display; Pease and Hayes, 1966).

was focused by metal mirrors onto a light pipe made of quartz rather than the usual plastic. At one point there was interest in using cathodoluminescence contrast to examine biological samples by suitably staining the samples with a fluorescent dye. This offered the promise of deriving biochemical information with electron optical resolution and the first results were encouraging (Fig. 15) (Pease and Hayes, 1966). However, at higher magnification the higher density of electrons quickly poisoned the dyes used. It would be interesting to determine whether the new dyes based on nanoparticles of semiconductors are more robust. G. Phonon Collection The use of phonons to build up the image was reduced to practice in the technique known as thermal wave microscopy in the 1980s. A periodically pulsed e-beam was used to locally heat the sample and the thermal stress

74

PEASE

wave was detected by a piezoelectric sensor. The technique was capable of resolving not only grain structure but also defects very deep (millimeter scale) beneath the surface – for example, at a bonding interface. The technique lost favor as pulsed lasers became more readily available. Recently, however, there has been a resurgence of interest (Trevor Page, University of Newcastle, UK, private communication; also see the Therma-Wave website at http://www.thermawave.com). H. Environmental SEM The advantages of viewing hydrated or even living samples in the SEM are obvious. In one early experiment it was found that certain creatures such as the Tribolium flour beetle secreted enough moisture to stay alive for about 30 minutes (Pease et al., 1966) (Fig. 16). After that time the beetle began to charge and did not survive. It appeared that the moisture not only kept the beetle alive but also inhibited charging. Much later the “environmental SEM” was

F IGURE 16. SEM image of the head of a (living) Tribolium beetle illustrating the striking rendition of surface topography in the SEM. Even though the magnification is only 200× (or a field of view ∼500 µm across), a corresponding light microscope image would contain far less information because of its limited depth of focus (Hayes and Pease, 1968; Everhart and Thornley, 1960).


(1965–2007) 75

introduced commercially. Such instruments used differential pumping since the gun needed to be maintained at high vacuum (< 10−4 Torr) but the sample could be at much higher pressure. The limit was set by electron scattering in the ambient atmosphere. Such SEMs can view remarkably wet samples. With a cooled stage 100% humidity is possible. An example in Fig. 17 shows a salt crystal first dissolving and then regrowing as the ambient pressure was decreased. Also shown is an example of X-ray contrast in which an energydispersive detector is used to build up the picture with selected energy X-rays.

V. TABLETOP SEM At various times attempts have been made to introduce an inexpensive SEM. For example, ISI marketed a tabletop SEM in 1971 at a selling price of $10,000 – ten times less than other SEMs. The instrument featured an aircooled diffusion pump so no house water was required, but this device did not succeed commercially. One reason was that the sample chamber was too small. Another was that the display had disappointing resolution, and a third reason was that insulating samples needed to be vacuum-coated with metal to prevent charging. Quite recently the Hitachi Company has introduced a tabletop SEM, the TM-1000 model. This also sells for ten times less than a top-of-the-line SEM. The display and control system is provided by the associated computer, and the vacuum system allows for environmental operation so no vacuum-coating station is needed. The TM-1000 can achieve only 30-nm resolution (i.e., sharp pictures as 10,000× maximum), but this is fine for a large range of samples; this is ten times better than most optical microscopes and is particularly attractive for educational purposes (Fig. 18). Other manufacturers are now following suit.

VI. A BERRATION C ORRECTION Since 1947 it has been realized that it is possible in principle to correct for spherical and chromatic aberrations in electron lenses by introducing optical elements lacking rotational symmetry. The principle is well described in the classic text by Hawkes and Kasper (1996) and by Zach and Haider (1995).2 However, efforts to reduce principle to practice were ineffective for many years largely because of the practical impossibility of achieving simultaneously the necessary adjustments to the myriad electrodes and magnetic fields that made for a correcting system. However, with computer control of a large array (e.g., 48) of power supplies, such correction has now 2 See also Hawkes and Kasper (1996).

76 PEASE F IGURE 17. Example of SEM operation with the sample in a wet atmosphere. The salt crystal is dissolving in the wet atmosphere and then recrystallizing. Such an “environmental” SEM can view samples without them drying out; this technique also can be used to reduce charging of insulating samples (Dusevich website). Image courtesy of V.M. Dusevich.


(1965–2007) 77

F IGURE 18. A tabletop SEM, the Hitachi TM-1000 model, which sells for 10% of the price of a good-quality conventional SEM and can view insulating and hydrated samples at useful magnifications up to 10,000×.

been demonstrated and successfully applied to the TEM (Weißbäcker and Rose, 2001), STEM (see the NION, CEOS, and Hitachi Company websites at http://www.nion.com/, http://www.ceos-gmbh.de/, and http://www.hitachihitec.com/global/em/tem/tem_index.html), and more recently to the SEM (see the JEOL Company website at http://www.jeol.com). The JEOL Company has marketed an aberration-corrected SEM since 2005 and has shown striking improvement especially at low voltages (Figs. 19 and 20). Curiously, as far as the author is aware, only one aberration-corrected

78 PEASE

(a) Schematic and photograph of a commercially available SEM incorporating correction for spherical and chromatic aberration (see JEOL Company website). The corrector is composed of four sets of 12-pole correctors. F IGURE 19.


(1965–2007) 79

(b) Graphs of resolution versus landing energy for two commercially available SEMs – one with aberration correction and one without. Note the improvement at low landing energies brought about by the correction (see JEOL Company website). F IGURE 19.

(continued)

competitor has yet been announced (Maas et al., in press). This suggests that resolution per se may not be the dominant selling point for the SEM. Moreover, the improvement in resolution achieved by aberration correction incurs a corresponding decrease in depth of field. This is less serious in the TEM and STEM where the samples must be thinned anyway but it is a definite detraction for the SEM for most applications.

VII. D IGITAL I MAGE P ROCESSING Digital image processing is an obviously beneficial development. One straightforward impact was the replacement of the long-persistence cathode ray tube for viewing with a computer display whose digital memory is updated as the SEM beam scans the sample. In addition, other operations are now easily incorporated including the use of commercial image processing systems such as Adobe Photoshop. Many of these operations are not effective when the original image quality is limited by shot noise as mentioned previously; it is often instructive to compare what can be achieved with simple controls on brightness and contrast with the more exotic digital techniques. However, because of the prevalence and effectiveness of digital enhancement

80 PEASE F IGURE 20. Images obtained with a commercial SEM (JSM 7700F) showing the improved image quality at 1 kV when using the aberration corrector (see JEOL Company website).


(1965–2007) 81

it is now necessary to specify in published micrographs the extent of any such enhancement. This issue is not restricted to the SEM.

VIII. U LTRAHIGH VACUUM The availability of ultrahigh vacuum technology is also an obvious development but one that is expensive and awkward to incorporate. In principle, secondary electron emission is strongly affected by the work function of the sample and so might be expected to yield equivalent contrast variations in the SEM; such a feature is clearly attractive. However, in regular high-vacuum systems (those using organic seals and pressures ∼10−5 Torr) the surfaces are contaminated by the beam cross-linking organic materials adsorbed on the surface of the sample, which masks the effect of the underlying work function. Operating in ultrahigh vacuum requires the use of copper seals and reduces the rate of contamination but results in very slow turnaround of samples. Moreover, reducing the contamination to a negligible level requires that the pressure in the sample chamber be less than about 5 × 10−11 Torr; such operation requires extreme care in cleaning and that the sample chamber be baked. Thus, as with the STM, this mode makes for only very specialized applications.

IX. F UTURE D EVELOPMENTS Ideally, better resolution can still be sought with the SEM. However, the scanning helium ion microscope recently developed by the ALIS Corporation and now marketed by Zeiss appears to offer a better chance of improved resolution for most samples because of the smaller volume of interaction between the ion beam and the solid sample (Morgan et al., 2006). Thus the SEM as we know may have come to the practical “end of the line” for improved resolution; note that the samples used to demonstrate high resolution in Fig. 20 are small, dense, and strongly emitting particles on a low-density, poorly emitting, background. Few samples in real life are this attractive. Opportunities do exist for making the SEM faster through the use of multiple-beam techniques developed for scanning electron beam lithography. One approach is to use multiple miniature columns (each a couple of inches long) such as those originally developed by Chang at IBM, then at ETEC (Chang et al., 1996), and now at NovelX (see the NovelX Company website at http://www.novelx.com). Another is to use the distributed-axis approach originally described by Groves and Kendall (1998) and later by Pickard et al.

82

PEASE

(2003). The semiconductor industry certainly has a need for fast nanometerscale inspection of wafers and photomasks and they may well invest in such a technology. It is not clear that a market for faster SEMs exists outside the semiconductor integrated circuit industry. The miniature columns mentioned above might find application as in situ SEM monitors of etching and deposition processes. One early (1962) SEM was indeed used for in situ monitoring of ion-etching processes and produced some striking images of eroding surfaces (Stewart, 1962), but since then the SEM has been used as a stand-alone tool for such process monitoring. But since the 1962 SEM was a full-size SEM with the chamber set up for ion beam etching, the miniature column could be inserted in the process chamber and use the vacuum already present there. Another, almost certain, development will be to exploit further the computational techniques now becoming available. As mentioned, computerenhanced imaging is already being used, but the scale to date is small compared with the computational effort currently applied to enhancing resolution in photolithography (Wong, 2001) and in optical equipment for inspecting semiconductor wafers and photomasks. This development will probably take place even without further development of SEM hardware. A more likely scenario is that hardware will be developed (e.g., more extensive multichannel and multifunctional signal detection) to maximize the benefits of computational techniques. In assessing its history, it is instructive to recount what has made the SEM so successful: 1. 2. 3. 4. 5.

Ease of use and virtually no sample preparation. Huge range of magnifications (5× to 500,000×). Huge depth of focus (from 100× to 1000× the image resolution). Easy interpretation of images. Little restriction on sample size (e.g., 300-mm silicon wafers).

At one time, in the mid-1980s, the STM and then the atomic force microscope (AFM; also known as the scanning force microscope, SFM) appeared poised to capture the market. They had superior resolution, could operate in air, and were simpler to operate. But the STM requires ultrahigh vacuum to be of significant use (its use in air is restricted primarily to examining graphite) and indeed continues to be used most effectively under ultrahigh-vacuum conditions as a research tool. The SFM has found much more widespread use than has the STM, particularly for determining surface topography of almost flat samples, such as semiconductor wafers, magnetic disks, and polymer surfaces. However, it requires skill (particularly in forming and preserving a well-shaped tip) and is restricted to fairly flat samples and a


(1965–2007) 83

small range of magnification. Thus the use of the SEM continued to increase despite the appearance of the STM and SFM. A likely development is that begun by the Hitachi TM-1000. That is, a wider, less affluent clientele could be sought. Analogies can be misleading, but the development of the computer may be a useful comparison. Until about 1980 the focus was on larger and faster computers, but personal computers began to appear in the 1970s. In the 1980s personal computers became a serious market even for the established computer manufacturers; today the supercomputer is only a small niche. The TM-1000 model is still too expensive for high school use, so perhaps a grand challenge would be the introduction of an SEM with similar capabilities but selling for less than $10,000, thus bringing it within range of a high school science laboratory. I look forward to seeing this progress realized.

ACKNOWLEDGEMENTS I would like to acknowledge valuable input from many people, notably Professor David Joy, Professor Timothy Groves, and especially Dr. Peter Hawkes, who provided help and input well beyond the normal call of duty for an editor.

R EFERENCES Boersch, H. (1954). Experimentelle Bestimmung der Energieverteilung in thermisch ausgelösten Elektronenstrahlen. Z. Phys. 139, 115–146. Broers, A.N. (1967). Electron gun using long-life lanthanum hexaboride cathode. In: Pease, R.F.W. (Ed.), Proceedings of the 9th Annual IEEE Symposium Electron, Ion and Laser Beams, Berkeley. San Francisco Press, San Francisco, CA, pp. 1–6. Cass, T.R., Jau, J., Dohse, H.J., Brodie, A.D., Meisburger, W.D. (1996). Application of the SEMSpec electron-beam inspection system to inprocess defect detection on semiconductor wafers. Microelectron. Eng. 30, 567–570. Chang, T.H.P., Thomson, M.G.R., Kratschmer, E., Kim, H.S., Yu, M.L., Lee, K.Y., Rishton, S.A., Hussey, B.W., Zolgharnain, S. (1996). Electron-beam microcolumns for lithography and related applications. J. Vac. Sci. Techol. B 14, 3774–3781. Coates, D.G. (1969). Pseudo-Kikuchi orientation analysis in the scanning electron microscope. Scan. Electron. Microsc. 9, 29–40. Crewe, A.V., Wall, J., Langmore, J. (1970). Visibility of single atoms. Science 168, 1338–1340.

84

PEASE

Davoine, F., Bernard, P., Pinard, P. (1960). La fluorescence des halogenures alcalins observée au microscope à balayage. Proc. European Regional Conf. Delft 1, 165–168. Dusevich, V.K. XL30 ESEM-FEG images. http://dentistry.umkc.edu/ microscopy/Esemimg.htm. Engel, W. (1966). Emission microscopy with different kinds of electron emission. In: Proceedings of the 6th International Congress on Electron Microscopy, vol. 1, Kyoto. Maruzen, Tokyo, pp. 217–218. Everhart, T.E., Thornley, R.F.M. (1960). Wide-band detector for micro-microampere low energy electron current. J. Sci. Instrum. 37, 246–248. Everhart, T.E., Wells, O.C., Oatley, C.W. (1959). Factors affecting contrast and resolution in the scanning electron microscope. J. Electron. Cont. 7, 97–111. Goldstein, J., Newbury, D.E., Joy, D.C., Lyman, C.E., Echlin, P., Lifshin, E., Sawyer, L., Michael, J. (2005). Scanning Electron Microscopy and X-ray Microanalysis, third ed. Springer, New York. Groves, T.R., Kendall, R.A. (1998). Distributed, multiple variable shaped electron beam column for high throughput maskless lithography. J. Vac. Sci. Techol. B 16, 3168–3173. Herriott, D.R., Collier, R.J., Alles, D.S., Stafford, J.W. (1975). A practical electron lithography system. IEEE Transactions on Electron Devices 22, 385–392. Hawkes, P.W., Kasper, E. (1996). Principles of Electron Optics. Academic Press. Hayes, T.L., Pease, R.F.W. (1968). The scanning electron microscope: Principles and applications in biology and medicine. Adv. Biol. Med. Phys. 12, 85–137. JEOL Company websites. Products: http://www.jeol.com/PRODUCTS and JEOL News Magazine: http://www.jeol.com/tabid/86/Default.aspx. Joy, D.C. (2006). Low voltage scanning electron microscopy – Current status, present problems and future solutions. e-J. Surf. Sci. Nanotechnol. 4, 369– 375. Joy, D.C., Newbury, D.E., Davidson, R.L. (1982). Electron channeling patterns in the scanning electron microscope. J. Appl. Phys. 53, R81. Kimura, H., Tamura, H. (1967). New method of detecting secondary electrons on scanning electron microscope. In: Pease, R.F.W. (Ed.), Proceedings of the 9th Annual IEEE Symposium Electron, Ion and Laser Beams, Berkeley. San Francisco Press, San Francisco, CA, pp. 198–205. Maas, D., Bosch, E., Horikx, J., Mentink, S., Henstra, S. (in press). Electrostatic aberration correction in low-voltage SEM. In: Proceedings of the 7th International Conference on Charged Particle Optics. Nuclear Instruments and Methods in Physics Research.


(1965–2007) 85

Meyer, E., Hug, H.J. (2004). Scanning Probe Miscroscopy: The Lab on a Tip. Springer, ISBN 3540431802. Morgan, J., Notte, J., Hill, R., Ward, W. (2006). An introduction to the helium ion microscope. Microscopy Today 14. July. Also available at ALIS Corporation website: http://www.aliscorporation.com/downloads/ Microscopy_Today.pdf. Ngo, V.V., Hernandez, M., Roth, W., Joy, D.C. (2007). STEM imaging of lattice fringes in a UHR in-lens FESEM. Microscopy Today 16, 12–16. NovelX Company website: http://www.novelx.com/products. Oatley, C.W. (1972). The Scanning Electron Microscope. Part 1: The Instrument. Cambridge Univ. Press, Cambridge, MA. Oatley, C.W., Nixon, W.C., Pease, R.F.W. (1965). Scanning electron microscopy. Adv. Electron. El. Phys. 21, 181–247. Pawley, J.B. (1992). LVSEM for high resolution topographic and density contrast imaging. Adv. Electron. Electron Phys. 83, 203–274. Pease, R.F.W. (1967). Low-voltage scanning electron microscopy. In: Pease, R.F.W. (Ed.), Proceedings of the 9th Annual IEEE Symposium Electron, Ion and Laser Beams, Berkeley. San Francisco Press, San Francisco, CA, pp. 198–205. Pease, R.F.W., Hayes, T.L. (1966). Scanning electron microscopy of biological material. Nature 210, 1049. Pease, R.F.W., Nixon, W.C. (1965). High resolution scanning electron microscopy. J. Sci. Instrum. 42, 81–85. Pease, R.F.W., Hayes, T.L., Camp, A.S. (1966). Electron microscopy of living insects. Science 154, 1185–1186. Pfeiffer, H.C. (1972). Basic limitations of probe forming systems due to electron–electron interaction. In: Johari, O. (Ed.), Proceedings of the 5th Annual Scanning Electron Microscopy Symposium. Illinois Institute of Technology Research Institute, Chicago, p. 113. Pickard, D.S., Groves, T.R., Meisburger, W.D., Crane, T., Pease, R.F. (2003). Distributed axis electron beam technology for maskless lithography and defect inspection. J. Vac. Sci. Techol. B 21, 2834–2838. Ratnakumar, K.N., Pease, R.F.W., Bartelink, D.J., Johnson, N.M. (1979). Scanning electron beam annealing with a modified SEM. J. Vac. Sci. Techol. 16, 1843–1846. Reichelt, R. (2007). Scanning electron microscopy. In: Hawkes, P.W., Spence, J.C.H. (Eds.), Science of Microscopy. Springer, New York, pp. 133–272. Reimer, L. (1993). In: Image Formation in Low-Voltage Scanning Electron Microscopy. In: SPIE Tutorial Text, vol. TT12. SPIE Press, Bellingham, WA. Reimer, L. (1998). Scanning Electron Microscopy. Springer, New York.

86

PEASE

Saparin, G.V. (1966). Observation of the p–n junction at small constant and alternating bias with the scanning electron microscope. In: Proceedings of the 6th International Congress on Electron Microscopy, vol. 1, Kyoto. Maruzen, Tokyo, pp. 609–610. Smith, K.C.A. (2004). Exploring the potential of the scanning electron microscope. Adv. Imag. Electr. Phys. 133, 93–109. Stewart, A.D.G. (1962). Investigations of the topography of surfaces with a scanning electron microscope. In: Breeze Jr., S.S. (Ed.), Proceedings of the 5th International Congress on Electron Microscopy. Academic Press, New York. Swanson, L.W., Swanson, N.A. (1975). Field electron cathode stability studies: Zirconium/tungsten thermal-field cathode. J. Appl. Phys. 46, 2029– 2050. Therma-Wave website: http://www.thermawave.com/. Thong, J.T.L. (1993). Electron Beam Testing Technology. Plenum Press, New York. Tuggle, D.W., Swanson, L.W. (1985). Emission characteristics of the ZrO/W thermal field emission source. J. Vac. Sci. Techol. B 3, 220–223. Venables, J.A., Harland, C.J. (1973). Electron backscattered patterns – A new technique for obtaining crystallographic information in the scanning electron microscope. Phil. Mag. 27, 1193–1200. Weißbäcker, C., Rose, H. (2001). Electrostatic correction of the chromatic and of the spherical aberration of charged-particle lenses. J. Electron Microsc. 50, 383–390. Wells, O.C., Broers, A.N., Bremer, C.G. (1973). Method for examining solid specimens with improved resolution in the scanning electron microscope (SEM). Appl. Phys. Lett. 23, 353–355. Wittry, D.B., Kyser, D.F. (1964). Use of electron probes in the study of recombination radiation. J. Appl. Phys. 25, 2439–2442. Wong, A. (2001). Resolution Enhancement Techniques in Optical Lithography. SPIE Press, Bellingham, WA. Zach, J., Haider, M. (1995). Aberration correction in a low voltage scanning microscope. Nucl. Instrum. Meth. Phys. Res. A 363, 316–325. Zworykin, V.K., Hillier, J., Snyder, R.L. (1942). A scanning electron microscope. ASTM Bull. 117, 15–23.


Ptychography and Related Diffractive Imaging Methods J.M. RODENBURG Department of Electronic and Electrical Engineering, University of Sheffield, Mappin Street, S1 3JD, United Kingdom

I. Introduction . . . . . . . . . . . . . . . . . II. Ptychography in Context . . . . . . . . . . . . . A. History . . . . . . . . . . . . . . . . . B. The Relative Strength of Diffractive and Imaging Methods . . C. The Definition of Ptychography . . . . . . . . . . D. A Qualitative Description of Ptychography . . . . . . . E. Mathematical Formulation of Crystalline STEM Ptychography . F. Illumination a “Top Hat” or Objects with Finite Support . . . G. Two-Beam Ptychography . . . . . . . . . . . . H. Some Comments on Nyquist Sampling . . . . . . . . I. Concluding Remarks on “Classical” Ptychography . . . . III. Coordinates, Nomenclature, and Scattering Approximations . . . A. Geometry of the Two-Dimensional, Small-Angle Approximation B. Three-Dimensional Geometry . . . . . . . . . . C. Dynamical Scattering . . . . . . . . . . . . . IV. The Variants: Data, Data Processing, and Experimental Results . . A. The Line Scan Subset . . . . . . . . . . . . . B. The Bragg–Brentano Subset: Projection Achromatic Imaging . C. Ptychographical Iterative (pi) Phase-Retrieval Reconstruction . D. The Wigner Distribution Deconvolution Method . . . . . V. Conclusions . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . .

87 88 89 93 94 97 103 109 113 114 117 119 119 123 132 138 140 149 163 172 178 180

I. I NTRODUCTION Ptychography is a nonholographic solution of the phase problem. It is a method of calculating the phase relationships between different parts of a scattered wave disturbance in a situation where only the magnitude (intensity or flux) of the wave can be physically measured. Its usefulness lies in its ability (like holography) to obtain images without the use of lenses, and hence to lead to resolution improvements and access to properties of the scattering medium (such as the phase changes introduced by the object) that cannot be easily obtained from conventional imaging methods. Unlike holography, it does not

87 ISSN 1076-5670 DOI: 10.1016/S1076-5670(07)00003-1


88

RODENBURG

need a stable reference wave: the only interferometer used in ptychography is diffraction interference occurring in the object itself. The concept of ptychography was originated by the late Walter Hoppe between about 1968 and 1973 (Hoppe, 1969a, 1969b; Hoppe and Strube, 1969; Hegerl and Hoppe, 1970, 1972). In its original formulation it was difficult to contemplate how to implement an actual experimental proof of the idea at short (atomic scale) wavelengths, other than for a crystalline object in a scanning transmission electron microscope (STEM). Hoppe and Strube (1969) demonstrated the idea experimentally using visible light optics, but at the time that Hoppe was thinking about these ideas, STEMs were simply not sufficiently developed (particularly the detectors available) to undertake the necessary measurements, and anyway, the benefits of the technique were ill defined and offered no clear improvements over existing imaging, diffraction, and holographic methods. In fact, when a proof of principle was eventually established by Nellist et al. (1995), it was discovered that ptychography did have one uniquely and profoundly important advantage over all other phaseretrieval or imaging techniques: it is not subject to the limitations of the coherence envelope (the “information limit”), which today is still regarded as the last hurdle in optimal electron imaging. In view of the recent advances in iterative solution of the ptychographic data set (see Section IV.C) and their experimental implementation in visible light optics and hard X-ray imaging, it is timely to rehearse the neglected strengths of the ptychographical principle. Recent years have seen a growing interest in iterative solutions of the phase problem. In most of this modern work, Hoppe’s pioneering thoughts have been entirely neglected. This is unfortunate for two reasons. On the one hand, there is the scholarly misrepresentation of the precedence of certain key ideas in the history of the field of phase problems. On the other hand, a true appreciation of what ptychography is all about, especially with respect to “conventional” diffractive imaging methods, means that a great opportunity has yet to be fully exploited. It is our intention in this Chapter to try to point out what these advantages are and why this Cinderella of the phase-retrieval story may be on the brink of revolutionizing short-wavelength imaging science.

II. P TYCHOGRAPHY IN C ONTEXT In this section, we place ptychography in the context of other phase-retrieval methods in both historical and conceptual terms.

PTYCHOGRAPHY AND DIFFRACTIVE IMAGING

89

A. History Recovery of the lost phase (or relative phase) by some calculation method is still considered a problem because when these issues were first addressed (before World War II), it appeared that in certain scattering geometries (especially those most easily accessible experimentally) a solution could not exist, and if it did exist, it would be very ill-conditioned mathematically. X-ray crystallography was the first experimental discipline to have to face up to the consequences of lost phase. In fact, given that large quantities of stereochemical a priori information are often available, the crystalline diffraction phase problem is much more tractable than the generalized phase problem. Indeed, it could be argued that Watson and Crick’s discovery of the structure of DNA (Watson and Crick, 1953) was itself an ad hoc solution to a phase problem – they had access to the intensity of the scattered X-ray diffraction pattern from a crystal composed of DNA molecules and essentially found a “solution” (a physical distribution of atoms) that was consistent with that intensity and with the known stereochemistry constraints of the constituent amino acids. This use of a priori information is an important characteristic of all solutions to the phase problem. The key advantage of phase-retrieval methods in the context of imaging is that image formation can be unshackled from the difficulties associated with the manufacture of high-quality lenses of large numerical aperture. In some fields, such as stellar interferometry (Michelson, 1890), it is only possible to record a property related to the intensity of the Fourier transform of the object. The creation of an image from these data is then a classic phase problem (in this case, rather easily solvable given the a priori information that the object in question is finite, of positive intensity, and is situated on an empty background). In the field of visible wavelength optical microscopy, there is little need to solve the phase problem because very good lenses of very large numerical aperture can be relatively easily manufactured (although a phase problem still exists in the sense that the image itself is recorded only in intensity). For short-wavelength microscopies (using X-rays or high-energy electrons), good-quality lenses are not at all easy to manufacture. For example, in the case of electron imaging, there has recently been enormous progress in the manufacture of non-round lens correctors (for which spherical aberration need not be positive, as with a round magnetic lens), a scheme first proposed by Scherzer (1947) but which has only recently become feasible with the advent of powerful, cheap computers that can be interfaced to very highstability power supplies. However, the cost of such systems is high (typically of the order of a million dollars per corrector), and the gains in resolution over existing round magnetic lenses are only of an order of a factor of 2 or so. In the case of X-ray microscopy, the zone plate remains the predominant lens

90

RODENBURG

technology, but this technology requires manufacturing precision of at least the final requisite resolution of the lens itself; making a sufficiently thick zone plate with very high lateral definition (< 50 nm), is difficult, especially for hard X-rays. For example, compare the best resolution obtainable from such a lens (also ∼50 nm) with the “resolution” of conventional X-ray diffraction using the same wavelength; notwithstanding the fact that the object of interest (the unit cell) must be amenable to crystallization, the effective resolution of the “image” (an estimate of that unit cell) is roughly the same as the wavelength: ∼ 0.1 nm. In the case of aperiodic objects (unique structures), solution of the phase problem has evolved via several diverse routes in communities that have historically had little interaction. The most elegant “direct solution” – indeed, so direct that in its original form it does not require any mathematical computation at all – is holography. Gabor (1948) first formulated this concept in an amazingly concise and insightful short letter with the title “A New Microscopical Principle.” His interest was in overcoming the rather profound theoretical limitations of round electron lenses, a fact that had been realized by Scherzer (1936) before the war and eloquently quantified by him shortly after the war (Scherzer, 1949). It was clear that aberrations intrinsic in the electron lens could not be overcome (although, as already discussed, later work on nonround lenses has circumvented these difficulties). We will touch on Gabor’s idea below but observe here that the use of a strong and known reference can record an analog of the phase of a wave field. Reconstruction of the actual electron image was envisaged as a two-step process, the second step performed using visible light optics that mimicked the aberrations present in the electron experiment. This is an example of a phase problem being solved in conjunction with the use of a “poor” lens so as to improve the performance of that lens. In fact, the current popular perception of holography is as a method of generating the illusion of three-dimensional (3D) images; the 3D information in ptychography will be an element of our discussion in Section IV.B. The pure diffraction intensity phase problem is defined as occurring in a situation where only one diffraction pattern can be measured in the Fraunhofer diffraction plane and is recorded without the presence of any lenses, physical apertures, or holographic reference wave. Even today many physicists unfamiliar with the field would declare that the pure diffraction intensity problem must be absolutely insoluble, given that the phase of a wave is known to encode most of the structural information within it. A common game is to take two pictures of different people, Fourier transform both pictures, and then exchange the modulus of those transforms while preserving their phase. If these are then both transformed back into images, the two faces appear with reasonable clarity in the same image in which the Fourier domain phase was preserved. In other words, the modulus (or intensity) information in the


91

Fourier domain is far less important in defining the structure of an image than its phase. If the phase is completely lost, it would stand to reason that the chances of being able to regain it in some way are negligible. However, in situations where the object is isolated and is not too large relative to the wavelength and the scattering angles being processed, this problem is now known to be almost always tractable. The provenance of the accumulated knowledge that has led to this rather astonishing conclusion is complicated and merits its own lengthy review. In the context of X-ray crystallography, Bernal and his many eminent research students laid much of the groundwork for constructing 2D and 3D maps of the unit cell from diffraction patterns taken through many different angles. Patterson (1934) noted that the Fourier transform of the intensity of the diffraction pattern is the autocorrelation function of the unit cell. Absence of phase is therefore equivalent to having an estimate of the object correlated with the complex conjugate of itself. Coupled with further a priori information about related crystalline structures and stereochemistry, this and similar conceptual tools led to the systematic solution of a wide range of complicated organic molecules, exemplified by the work of Hodgkin, who determined the structure of penicillin and later vitamin B12 (Hodgkin et al., 1956) using similar methods. In the context of crystallography, the term direct methods (pioneered by workers such as Hauptman, Karle, Sayre, Wilson, Woolfson, Main, Sheldrick, and Giacovazzo; see for example the textbook by Giacovazzo, 1999) refers to computational methods that do not necessarily rely on the extremely detailed knowledge of stereochemistry necessary to propose candidate solutions that fit the diffraction data (or, equivalently, the autocorrelation (Patterson) function). Instead, the likelihood of the phase of certain reflections is weighted in view of how the corresponding Bragg planes must, in practice, fill space. Here the essential a priori knowledge is that the object is composed of atoms that cannot be arbitrarily close to one another. Direct methods can be used to solve routinely the X-ray diffraction problem for unit cells containing several hundreds of atoms (not counting hydrogen atoms), beyond which the probabilistic phase relationships become weak. A variety of other techniques are available for larger molecules, such as substituting a very heavy atom at a known site within the unit cell (this can be thought of acting as a sort of holographic reference for the scattered radiation) or changing the phase of the reflection of one or more atoms by obtaining two diffraction patterns at different incident energies, above and below an absorption edge. Independent of the crystallographic problem, an entirely different approach to the recovery of phase was derived in the context of the mathematical structure of the Fourier transform itself, especially as it relates to analytic functions. In any one experiment, an X-ray diffraction pattern explores the

92

RODENBURG

surface of a sphere (the Ewald sphere) in a 3D reciprocal space, which is the Fourier transform of the electron density in real space. However, periodicity in real space confines the relevant Fourier data to exist on the reciprocal lattice. Classical X-ray diffraction is about Fourier series, not explicitly Fourier transforms. In contrast, there are a variety of physical problems in which the continuous Fourier transform is the crucial element. One such class is the so-called half-plane problem, wherein it is known, for one reason or another, that the Fourier transform of a function only has finite magnitude at positive (or negative) frequencies, while negative (or positive) frequencies all have zero magnitude. An example would be a reactive electrical system with a complicated frequency response subjected to a step change in the input voltage (or current). Here the half-plane is in the time domain, but this places constraints on the phase of the allowable frequency components. Depending on the form of the underlying differential equation that determines a response of a system, the solution could sometimes be expressed in terms of Laplace transforms (identical to the Fourier transform except for the lack of the imaginary number in the exponential). More generally, a solution could be composed of a Fourier integral with the “frequency” component in the exponential taking on any complex value (thus embracing both the Fourier and Laplace transforms). The theory of the resulting analytic function of the complex variable that arises in this class of situation has been extensively investigated from a purely mathematical standpoint (see, for example, Whittaker and Watson, 1950). This analytic approach leads to the Kramers– Kronig relations in dielectric theory (Kramers, 1927; Kronig, 1926) and a host of other results in a wide range of fields, including half-plane methods as a means of improving resolution in the electron microscope (Misell, 1978; Hohenstein, 1992). In the latter, the half-plane is a physical aperture covering half of the back focal plane of a conventional imaging lens. It is certainly true that Bates, who went on to contribute to variants of the ptychographical principle with Rodenburg (Section IV.D) came from this same “analytical solution” school of thought. In an original and oblique approach to the phase problem, it was Hoppe who proposed the first version of the particular solution to the phase problem that we explore in this Chapter. Following some earlier work (Hoppe, 1969a, 1969b; Hoppe and Strube, 1969), Hegerl and Hoppe (1970) introduced the word ptychography to suggest a solution to the phase problem using the convolution theorem, or rather the “folding” of diffraction orders into one another via the convolution of the Fourier transform of a localized aperture or illumination function in the object plane. Ptycho comes from the Greek word “πτυξ” (Latin transliteration “ptux”) meaning “to fold” (as in a fold in a garment; the German word for convolution is Faltung). We presume the authors were following the example set by Gabor (1949), who constructed the


93

term holography from the Greek “ολο” (Latin transliteration “holo”), meaning “whole”: the whole wave disturbance being both its modulus and phase over a surface in space. Like holography, ptychography aspires to reconstruct the entire wave field scattered from an object. Unlike holography, it does not require a reference beam; it derives its phase knowledge by recording intensity only in the diffraction plane and then, via properties of the convolution theorem, solving directly for the structure of the object. Exactly how this can be achieved and put into practice is the main content of this Chapter. B. The Relative Strength of Diffractive and Imaging Methods Although not widely disseminated, it is nowadays recognized that diffraction phase-retrieval microscopy has certain distinct advantages over conventional forms of imaging and holography. The key requirement is the coherent interference of different wave components. In the case of conventional transmission imaging, a lens is used to form an image. Beams that pass along different paths through the lens are required to arrive at a point in the image plane in such a way that their relative phase is well determined by a particular path difference. In the case of the short-wavelength microscopies (electron and X-ray), the stability and reproducibility of this path difference is difficult to achieve. Vibration, lens instability, energy spread in the illuminating beam, and other sources of laboratory-scale interference can easily sabotage the constancy of the pertinent path difference, leading to the incoherent superposition of waves. The establishment of the appropriate path length is itself problematic if good-quality lenses are unavailable. It should noted that even in the context of so-called incoherent imaging, where the object of interest is self-luminous or illuminated by an incoherent source of radiation, then it is still a requirement that the waves emanating or being scattered from any point in the object interfere with themselves coherently when they arrive in the image plane. In the case of holography, where a reference wave is added to a wave field of interest (again, one that has been scattered from the object of interest), the requirements for mechanical stability and the absence of other sources of experimental error (arising, say, from ripple on the lens supplies in the case of electron holography) are extremely demanding. In contrast to these conventional imaging methods, so-called “diffractive imaging,” wherein an estimate of the object is made indirectly by solving the phase problem in the diffraction pattern scattered by it, is nowadays generally recognized as having certain experimental advantages (even though there are also a number of potentially grave disadvantages). In relation to the question of path length discussed above, the great strength of diffraction is that the interferometer used to effect the encoding of structural information onto the wave field is (at least in the case of atomic-scale wavelength

94

RODENBURG

radiations) of the same dimension as the atoms themselves. The requirement to re-interfere nonparallel wave components that have traversed substantial distances (millimeters or more) through the laboratory environment is removed. This is the great experimental strength of diffraction – after all, it is well known that rather crude instrumentation can undertake reasonably accurate diffraction measurements. As an undergraduate, I was required to measure the spacing of the atomic planes in NaCl using a benchtop Xray diffractometer during the course of a single afternoon. In contrast to conventional imaging or holography at atomic-scale wavelengths, diffraction is experimentally relatively trivial. In the case of high-energy electrons, the experimental robustness of diffraction relative to imaging is observed simply by comparing the breadth of reciprocal space (scattering angle) in the selected area diffraction pattern that has significant intensity with the corresponding width of the bright-field diffractogram (that is, the Fourier transform of the bright-field image). If a microscope was perfectly stable, then the diffractogram would be as wide in kspace (scattering angle) as the diffraction pattern. In fact, the former is quickly attenuated by the inevitable lens and high-tension instabilities. When these errors are coupled with strong aberrations in the imaging lens, the effective numerical aperture of an electron lens is severely attenuated by an envelope function (Frank, 1973). Wave components scattered through large angles (in fact, no more than a few degrees) are incapable of contributing usefully to the image. Since image resolution is proportional to the effective numerical aperture of the lens, then resolution is severely compromised. We see, then, that if it is possible to dispense with the lens or any other form of macroscopic interferometry and instead rely on interference arising from waves scattered directly from the atoms themselves, we should be able to obtain much more information about the structure of a specimen, without all the expense and complication of a short-wavelength imaging lens. Certain experimental challenges are removed, albeit at the price of introducing a difficult computational problem. Ptychography is essentially a method of blending these strengths of diffraction with a further experimental variation (the collection of more than one diffraction pattern), but which thereby greatly reduces the complexity and difficulty of solving the phase problem while at the same time greatly improving the applicability of the diffractive imaging method, most importantly to imaging objects of infinite size even in the presence of partial coherence in the illuminating beam. C. The Definition of Ptychography If we refer back directly to Hoppe’s series of three papers in 1969, where these ideas were first suggested, then we might conclude that all “finite object”


95

solutions to the phase problem (including the so-called oversampling method of the PDI problem, Miao et al., 1999) fall under the scope of ptychography. But, such a broad definition would not be usefully discriminating. Although Hoppe clearly understood that it should be possible to solve the phase problem from a single diffraction pattern (assuming it is known that the object is finite), he pointed out that this would be exceedingly difficult, at least using the computational techniques available at that time. The main thrust of his argument in the 1969 articles was that the ambiguities that occur when only a single diffraction pattern is recorded can immediately be resolved if it is possible to either move the specimen relative to a defining aperture, or relative to some sort of illumination function, and record a second diffraction pattern. In a later article (Hegerl and Hoppe, 1972), the illumination function was not moved, but it was changed in its functional form. For the purposes of this Chapter, the term ptychography is reserved to apply to a method that fulfills at least all of the following three characteristics: 1. The experimental arrangement comprises a transmission object that is illuminated by a localized field of radiation or is isolated in some way by an aperture mounted upstream of the object. Scattered radiation from this arrangement provides an interference pattern (usually, but not necessarily, a Fraunhofer diffraction pattern) at a plane where only intensity can be recorded. 2. At least two such interference patterns are recorded with the illumination function (or aperture function) changed or shifted with respect to the object function by a known amount. 3. A calculation is performed using at least two of these patterns in order to construct an estimate of the phase of all diffraction plane reflections, or, equivalently, of the phase and amplitude changes that have been impressed on the incident wave in real space by the presence of the object. As it stands, this definition certainly differentiates ptychography from many competing solutions to the phase problem. We will reserve the term pure diffractive imaging (PDI) to apply to a solution of the Fraunhofer phase problem in which only one diffraction pattern is recorded. This is quite distinct from ptychography, although in fact the solution of the PDI problem can be thought of as relying on the convolution theorem (see Section II.F), and hence being due to the ptycho element of ptychography. I would argue, however, that once the phase problem is formulated in terms of the convolution theorem, then the direct solution that follows via the Fourier shift theorem is so compelling that it would seem absurd to rely on only one diffraction pattern, provided of course that there is some way of achieving the shift of the object relative to the illumination function or aperture in some experimentally feasible way. For this reason, I contend that the recording of at

96

RODENBURG

least two interference patterns is a defining component of ptychography, quite independent of whether the Greek meaning of the word actually embraces this specification. Our definition thus also renders ptychography distinct from the throughfocal-series (Misell, 1973) or transport of intensity (see for example Gureyev et al., 1995) solutions to the phase problem. Although in both of these cases multiple interference patterns (in this case, Fresnel diffraction patterns) are collected, the illumination (usually a plane wave) always remains fixed and unchanged with respect to the object of interest. The same applies in the case of holography. It could be argued that any method that alters the illumination condition, or the way in which the object is presented to the field of radiation, and subsequently records more than two intensity distribution as a means of solving for phase (e.g., McBride et al., 2005) falls within the present definition. In fact, as will be shown in the following text, the main practical implementations of ptychography use only a lateral shift of the object or illumination function. Imposing this restriction would, ironically, discount one of the few articles published by Hegerl and Hoppe (1972) demonstrating computationally what they had themselves called ptychography, although there is really no doubt that Hoppe had originally conceived of the scheme in terms of lateral shifts alone (Hoppe, 1982). A further issue in its exact definition is whether ptychography relates only to crystalline objects. At one level this point is irrelevant in that any image, regardless of the size of its field of view, can always be represented computationally as one very big unit cell. Indeed, all PDI methods do exactly this, as does any image processing algorithm that relies on equally spaced samples in the reciprocal space; this includes any method that uses a fast Fourier transform. However, there is a common perception, fueled if only by Hoppe himself in a late review (Hoppe, 1982), that ptychography essentially relates only to physically periodic objects. That is to say, as the object, aperture or illumination field is shifted laterally, the next part of the object to be illuminated by radiation is identical to that part of it that has shifted out of the field of illumination. It is certainly true that the simplest computational implementation of ptychography does only relate to this very specific crystalline case. However, more recent work has shown that the crystalline restriction simply need not apply. For the purposes of this Chapter, we will therefore extend the three characteristics enumerated above by adding a fourth: 4. When the change of illumination condition is a simple lateral shift (or shift of the object), then ptychography allows a large number of interference patterns (as many as required) to be processed in order to obtain an image of a nonperiodic structure of unlimited size.


97

D. A Qualitative Description of Ptychography We start by describing qualitatively the principle of ptychography (as defined above) using the very simplest manifestation of the method, as first described by Hoppe (1969a, 1969b) and Hoppe and Strube (1969). Detailed mathematical treatments, and a discussion of the principal approximations that are implicitly assumed in this section, are presented in later sections. Consider a lens focusing a source of radiation onto a crystalline object of interest as shown in Fig. 1. (In this Chapter, optical components are drawn on a horizontal optic axis with radiation incident from the left.) Assume for the time being that the lens is aberration free, it has a sharp-edged circular aperture in its back focal plane, and the beam crossover at the object plane is perfectly focused. In other words, the illumination function in the plane of the entrance surface of the object is of the form of an Airy disk function (see, for example, Born and Wolf, 1999). Let us also assume that this illumination function interacts multiplicatively with the object function; the object function will be modeled as a (possibly strong) phase transmission function that

F IGURE 1. Elementary crystalline ptychographical experimental setup in the scanning transmission electron microscope (STEM) configuration. Radiation incident from the left impinges on a focusing lens, which has a sharply defined diaphragm. The crystalline specimen lies in or near the plane of the beam crossover (this produces the illumination function incident on the specimen, often called a probe in the STEM literature). Downstream of the specimen a diffraction pattern evolves. Each diffraction order is spread into a disk corresponding to the range of incident illumination angles in the beam. We will assume that the distances between the lens and the object and the object and the detector plane are large: the detector plane lies in the Fraunhofer regime.

98

RODENBURG

may also attenuate the wave as it passes through the object. Note that the transmission function is not the same as the atomic (or optical) potential function; we are not assuming the Born approximation (see Section III.C), but we are assuming that the object is quasi-2D (Cowley, 1995). In the Fraunhofer diffraction plane, a very large distance downstream of the object, the far-field diffraction pattern is observed. If this periodic object had been illuminated by a plane wave, the diffraction pattern would, as usual, consist of a series of spikes of intensity. In fact, because there is a range of incident vectors subtending from the pupil of the lens, each diffraction peak is spread out into the shape of a disk. In the absence of a specimen, there is only one disk – simply a shadow image projection of the aperture in the back focal plane of the lens. When the crystalline object is in place, it is possible to arrange for the aperture size to be such that the diffracted disks overlap one another as shown in Fig. 2(a). For what follows in this section, we require that there is a point between any two diffracted discs where only those two disks overlap with one another (i.e., there are no multiple overlaps). If the source of radiation is sufficiently small and monochromatic, then illumination over

(a)

(b)

(c)

(d)

F IGURE 2. Schematic representation of the diffraction orders and phase relationships in the STEM ptychograph. (a) Two diffracted disks lying in the Fraunhofer diffraction plane (the right-hand side of Fig. 1). (b) Phase relationship of the underlying amplitudes of these two disks. The square roots of the measured intensities give the lengths of the arrows, but not their phase. However, the triangle of complex numbers must be closed, although there are two indistinguishable solutions. (c) For three linearly positioned interfering disks. (d) For 2D functions, ambiguity in this phase problem (as in all Fourier phase problems) is reduced because the ratio of measurements to unknowns increases.


99

the back focal plane of the lens will be effectively coherent. In the Fraunhofer plane, this means that the diffracted beams will interfere coherently within these regions of overlap (provided other sources of experimental instability, such as the lens, are negligible). In the classic diffraction phase problem (where the illumination is a plane wave), the intensity (and only the intensity) of each of the diffracted beams can be measured: the phase is lost. In Fig. 2(a), we also measure intensity only, but now we have the opportunity of measuring the intensity of the sum of two diffracted amplitudes. Let us label these amplitudes by the complex numbers Z1 and Z2 , each representing the real and imaginary components (or modulus and phase) of the wave filling each disk. We can therefore measure |Z1 |2 , |Z2 |2 , and, within their region of intersection, |Z1 + Z2 |2 . Let the phase difference between Z1 and Z2 be an angle φ. In the complex plane, we see a relationship between amplitudes Z1 and Z2 and their sum Z1 + Z2 (shown in Fig. 2(b)). Because the moduli Z1 , Z2 and Z1 +Z2 can be measured by taking the square root of the intensity measured in the two disks and their region of overlap, respectively, then there are only two values of φ (φ and −φ) that will allow the triangle of amplitudes Z1 , Z2 and Z1 + Z2 to connect with one another. In other words, by measuring these three intensities, and assuming they interfere with one another coherently, we can derive an estimate of the phase difference between Z1 and Z2 , but not the sense (positive or negative) of this phase difference. Clearly this simple experiment has greatly improved our chances of solving the phase problem: instead of total uncertainty in the phase of either Z1 or Z2 , there are now only two candidate solutions for the phase difference between Z1 and Z2 . In fact, without loss of generality, we will always assign one of these values (say, Z1 ) as having an absolute phase of zero. This is because in any diffraction or holographic experiment we can only ever possibly measure the relative phase between wave components. Another way of saying this is that we are concerned with time-independent solutions to the wave equation, and so the absolute phase of the underlying waves (which are time dependent) is lost and is of no significance – in the case of elastic scattering, the relative phase of the scattered wave distribution has had all the available structural information about the object impressed upon it. If the disk of amplitude Z1 is the straight-through (or zero order) beam, then in what follows we will always assign this as having zero phase: this particular two-beam phase problem (i.e., the relative phase of Z2 ) has been completely solved apart from the ambiguity in Z2 or Z2∗ , where Z2∗ is the complex conjugate of Z2 . Now consider the interference of three beams lying in a row (Fig. 2(c)). The underlying amplitudes of these diffracted beams are labeled as Z1 , Z2 , and Z3 . We now measure five intensities: three in the disks and two in the overlaps. These five numbers will not quite give us the six variables we

100

RODENBURG

need – the modulus and phase of Z1 , Z2 , and Z3 – but they will give us |Z1 |, |Z2 |, |Z3 | and two phase differences (each of which suffer from a positive/negative ambiguity). Assigning zero phase to Z1 , we can solve for the phase of all the diffracted beams (Z2 and Z3 ) to within four possible solutions, depending on the four combinations of the ambiguous phases. It is clear that as the number, N, of diffracted beams increases, then there will be 2N possible solutions for the one diffraction pattern. In fact, this 2N ambiguity is a completely general feature of the phase problem in one dimension (Burge et al., 1976) when the object space is known to be finite (in the current case, the object and illumination function is of a particular form that gives a very clear manifestation of the phase ambiguity). Of course, if the three beams are not collinear, but form a triangle in a 2D diffraction pattern so that each beam overlaps with the other two (Fig. 2(d)), then the ratio of intensity measurements to unknowns can be increased, thereby limiting the number of ambiguous solutions to the entire pattern. Again, even though we are discussing a very specialized scattering geometry, the same argument applies in all diffractive imaging methods – as the dimension of the problem is increased, the likelihood of ambiguous solutions existing reduces (see for example the discussion by Hawkes and Kasper, 1996). The key benefit of ptychography is that a complete solution of the phase, resolving the ambiguities discussed above, can be achieved by shifting either the object function or the illumination relative to one another by a small amount, and then recording a second set of intensity measurements. We can think of this via the Fourier shift theorem, but to begin we restrict ourselves to a qualitative description. In the scattering geometry of Fig. 1, a shift in the illumination function can be achieved by introducing a phase ramp across the illumination pupil. One way of thinking of this is via Fig. 3(a) (which is rather exaggerated). To produce a tilt in the wave coming out of the lens (which will consequently lead to a shift in the illumination function at the object plane), we must rotate slightly the hemisphere of constant phase that emerges from the lens and which results in the wavefronts being focused at the crossover at the object plane. If the path length is increased at the top of the hemisphere and decreased at bottom (as shown in Fig. 3(a)), then the effect is to move the position of the illumination downward relative to the object. The path difference changes linearly across the hemisphere. A linear phase ramp, where the phase is equal to 2π times the path difference divided by the wavelength, has the equivalent influence. In what follows, we ignore the fact that this tilt in the beam will not only shift the illumination, but also shift the diffraction pattern in the far field; in the case of most situations of interest this latter shift is negligible for typical lens working distances. In the Fraunhofer plane, one side of the zero order (unscattered) beam has therefore been advanced in phase (marked as point P on Fig. 3(b)), say by a factor of eiγ , and a point


101

near the opposite side (P ) has been retarded in phase, say by a factor of e−iγ . If we label the underlying amplitude of the zero-order beam as Z1 , then the complex value of the wave at P is now Z1 eiγ . Suppose now that there is a diffracted beam of underlying amplitude Z2 (shown separately on the top right of Fig. 3(b)). At an equivalent point P the actual complex value of the wave

(a) In the STEM configuration, a shift in the probe (indicated by the large pointer) can be achieved by introducing a phase ramp in the plane waves (dotted lines) over the back focal plane of the lens (incident beams illuminating the lens are drawn parallel). This is equivalent to tilting slightly the hemisphere of wavefronts focusing the probe onto the object, consequently introducing the shift. This manifests itself as the phase change over A(u, v).

(b) As the probe is shifted in real (object) space (top), a phase ramp is introduced over the undiffracted beam in reciprocal space (lying in the Fraunhofer diffraction plane). F IGURE 3.

102

RODENBURG

(c) At the top we draw the undiffracted beam (heavy circle) and a diffracted beam. The probe movement has introduced phase ramps across both beams. When the spacing of the diffracted beams is smaller (bottom circles), the equivalent points P and P meet up and interfere at P . F IGURE 3.

(continued)

disturbance is now Z2 e−iγ . If these two disks overlap, then the equivalent points P and P can meet up and interfere with one another, and the total amplitude where these points meet (P in Fig. 3(c)) will be Zs , where Zs = Z1 eiγ + Z2 e−iγ .

(1)

Before we moved the illumination by introducing these phase changes, we measured Z1 , Z2 and Z1 + Z2 . With reference to the complex sum shown in Fig. 4, let Z0 be equal to Z1 +Z2 . In view of the complex conjugate ambiguity, Z1 + Z2∗ = Zc was also a candidate solution. However, after shifting the illumination and recording Zs , we can now discount this (wrong) solution because if the same phase changes had been applied to Z1 + Z2∗ , the final sum would have been given by Zw (see Fig. 4). This rather naïve analysis actually summarizes the entire principle of ptychography. In later sections, we will show that the same underlying principle can be applied to extended noncrystalline objects and the illumination function does not have to be of any particular form, as long as it is reasonably localized. We will also have to consider the geometric and scattering theory approximations we have made in this section.


103

F IGURE 4. How ptychography unlocks the complex conjugate phase ambiguity. In this diagram, the lengths of filled pointers represent the square root of the intensity of our four measurements (two from the centers of the diffracted disks, two from the overlap region at different probe positions). Before moving the illumination (probe) the three initial measurements (top left) have two possible solutions (top and middle right). When the phase shifts (Fig. 3) are applied (bottom left), the wrong (−φ) phase can be discounted: the hatched pointer (Zw ) is not the length of the measured modulus Zs . All phases are plotted with respect to Z1 ; in practice, Z1 and Z2 rotate equal angles in opposite directions, but the effect on the measured moduli is the same.

E. Mathematical Formulation of Crystalline STEM Ptychography For the sake of simplicity, an image processing-type of nomenclature and not an optical or electron scattering nomenclature is adopted in this section. In Section III, we will connect the relevant wave k-vectors with the implicit geometric approximations made here. In this section we simply define a math-

104

RODENBURG

ematical form of the 2D Fourier transform as $f (x, y) = f (x, y)ei2π(ux+vy) dx dy = F (u, v), and its inverse as $

−1

F (u, v) =

F (u, v)e−i2π(ux+vy) dx dy = f (x, y),

(2a)

(2b)

where f (x, y) is a 2D image or transmission function that may be complex valued, described by Cartesian coordinates x and y; F (u, v) is the Fourier transform of f (x, y); u and v are coordinates in reciprocal space (that is, the Fourier transform space); and for the compactness of later equations we use the symbol $ to represent the whole Fourier integral. This definition of the Fourier transform differs from that used in some fields (including electron microscopy, although not crystallography) by having a positive exponential. The inclusion of the factor of 2π in the exponential renders the definition of the inverse much simpler by not requiring a scaling factor dependent on the dimensionality of the functions involved. This greatly simplifies some arithmetic manipulations that must be used in later sections. As before, let us assume that the object can be represented by a thin complex transmission function. In other words, the presence of the object has the effect of changing the phase and intensity (or, rather, modulus) of the wave incident upon it. The exit wave, ψe (x, y), immediately behind the object, will have the form ψe (x, y) = a(x, y) · q(x, y),

(3)

where x and y are coordinates lying in the plane of the (2D) object function, a(x, y) is the complex illumination function falling on the object, and q(x, y) is the complex transmission function of the object. As emphasized previously, q(x, y) is not the potential or optical potential of the object; it is equal to the exit wave that would emerge from a thin (but possibly very strongly scattering) object when illuminated with a monochromatic plane wave. For a thick specimen, we might hope that q(x, y) = ei

σ V (x,y,z) dz

,

(4)

where V (x, y, z) is the optical potential (or, in the case of high-energy electrons, the atomic potential) of the object, and σ is a constant describing the strength of the scattering for the particular radiation involved. In this case the ptychographical image derived below can be related directly to the projection of the potential (by taking the logarithm of q(x, y) – at least in the absence of phase wrapping), despite strong (dynamical) scattering. In fact, this is not the case for thick specimens because the integral with respect to z does not


105

account for propagation effects as the wave traverses and diffracts through the thickness of the object. This distinction is emphasized because it has led to many misunderstandings in the literature. In particular, because q(x, y) can introduce very large phase changes, including more than 2π , it is quite wrong to suggest that the multiplicative approximation is a “weak phase” or kinematical approximation; it is really a “thin object” approximation. Assuming this approximation, we can write via the convolution theorem that the amplitude in the far field, M(u, v, ), is given by M(u, v) = $q(x, y) ⊕ $a(x, y) , (5) where we note that in a real experiment u and v are must be scaled appropriately by the wavelength and the camera length (the distance between the object and the detector plane; see Section III.A). The convolution operator, ⊕, is defined for any two 2D functions, g(x, y) and h(x, y), as g(x, y) ⊕ h(x, y) = g(X, Y )h(x − X, y − Y ) dX dY. (6) Now let us relate this to our qualitative description of ptychography in the previous section. In the case of electrons, the optical setup shown in Fig. 1 is achieved in the STEM configuration. In this case, a(x, y) is referred to as the “probe” because the very intense beam crossover at the specimen plane can be used to probe the specimen to obtain electron energy loss spectra and other analytical signals from the small volume of matter illuminated. a(x, y) is itself the back Fourier transform of the aperture function, which we call A(u, v), lying in the back focal plane of the lens. It should be noted that u and v are angular coordinates of the Fourier components that make up the incident radiation. Because beams cross over in the vicinity of the object, a diaphragm lying in the back focal plane appears reversed in the diffraction plane. A(u, v) is therefore the complex amplitude of the illumination as it arrives in the diffraction plane. A(u, v) can include phase components corresponding to the presence of aberrations; its modulus is usually of the form of a “top hat” function (although this is considerably modified when the illumination is partially coherent). The diffracted amplitude is now: (7a) M(u, v) = $ $−1 A(u, v) ⊕ $q(x, y), or M(u, v) = A(u, v) ⊕ Q(u, v),

(7b)

where Q(u, v) is the Fourier transform of the object transmission function. It is this convolution (folding) in the diffraction plane that leads to the term ptychography. Everything in this Chapter depends on the existence of

106

RODENBURG

this convolution. The STEM configuration is particularly straightforward to understand when we assume a very simple form of A(u, v) – a sharply defined circular aperture. In our simplest manifestation of ptychography, we make two measurements, intensities I1 (u, v) and I2 (u, v), each from different positions of the probe, where 2 (8a) I1 (u, v) = A(u, v) ⊕ Q(u, v) and

2 I2 (u, v) = A(u, v)eiux ⊕ Q(u, v)

(8b)

and where we have assumed the second measurement is made with the illumination function shifted with respect to the origin in real space by a distance x in the x-direction to give a(x − x, y): The exponential derives from the Fourier shift theorem, namely, that i2πux dx = g(X)ei2πu(X−x) dX $g(x + x) = g(x + x)e = G(u)e−iu2πx

(9)

for any function g(x) whose Fourier transform is G(u). In the simplest version of ptychography, we are interested in the case of q(x, y) being a periodic 2D crystal. Q(u, v) then only has values on an equivalent 2D reciprocal lattice. Let us suppose for simplicity that this lattice is rectilinear with a periodicity of u and v, in the u and v directions, respectively. We can say that Qn,m δ(u − nu, v − mv), (10) Q(u, v) = n,m

where Qm,n is a complex number associated with the amplitude of a reciprocal lattice point indexed by m and n, the sum is over the positive and negative integers for all m and n, and δ(u, v) is a Dirac delta function defined as 0 for u = u0 or v = v0 , δ(u − u0 , v − v0 ) = (11a) ∞ for u = u0 and v = v0 and where ∞ δ(u − u0 , v − v0 ) du dv = 1.

(11b)

−∞

To achieve the overlapping disks we described in the previous section (Fig. 2(a)), the aperture function A(u, v) must have a diameter of at least v or u, whichever is the larger. Suppose that u > α > u/2, where α is radius of the aperture, as shown in Fig. 5. As before, we consider just two of


107

F IGURE 5. In the simplest STEM implementation of crystalline ptychography, intensity is measured at the centers of the zero-order and first-order diffraction disks and at the midpoint of their overlap region.

the disks in the far field: the zero-order beam (n = m = 0) and one diffracted beam (n = 1, m = 0), then Q(u, v) = Q0,0 δ(u, v) + Q1,0 δ(u − u, v)

(12)

M(u, v) = Q0,0 A(u, v) + Q1,0 A(u − u, v).

(13)

and so

With reference to Fig. 5, let us measure the intensity at a point midway between the centers of the two interfering aperture functions, namely, at u = u/2 and v = 0. The wave amplitude at this point is u u u , 0 = Q0,0 A , 0 + Q1,0 A − ,0 (14) M 2 2 2 and hence the intensity at this point in the diffraction plane with the illumination field (the probe) on axis, is 2 u u u , 0 = Q0,0 A , 0 + Q1,0 A − , 0 . (15) I1 2 2 2 We now move the probe a distance x in the x-direction (parallel with the reciprocal coordinate u). From Eq. (8b) we obtain a second intensity measurement 2 2πu 2πu u α α , 0 = Q0,0 A , 0 ei 2 x + Q1,0 A − , 0 e−i 2 x . I2 2 2 2 (16)

108

RODENBURG

The amplitudes of these two equations are identical in form to the qualitative description derived in Section II.C [Eq. (1)], with the phase term γ replaced by π ux and with Z1 and Z2 replaced by the underlying phases of the lattice reflections Q0,0 and Q1,0 . In the previous section we assumed that the aperture function A(u, v) did not have any aberrations (i.e., phase components) across it; this means the value of A(u, v) is unity, and so this term does not appear in Eq. (1). In general, the aperture function will have phase changes across it, corresponding to defocus, spherical aberration, and so on, and so these must be accounted for in some way – indeed the probe movement (which leads to a linear phase ramp multiplying the aperture function, as in Eq. (8b) is itself a sort of aberration term. It should be pointed out that in general it may not be experimentally convenient to use an aperture function that is small enough to guarantee that there exist points in the diffraction pattern where only two adjacent diffracted disks overlap. Clearly, as the unit cell increases in size, the diffraction disks become more and more closely spaced in the diffraction plane. In the limit of a nonperiodic object, it may appear as if the ptychographical principle is no longer applicable, because the pairs of interfering beams can no longer be cleanly separated. In fact, we will see in Section IV that this is not the case, provided we adopt more elaborate processing strategies. In this very simplest arrangement with just two diffracted beams, we still need to measure a total of four intensities to derive their relative phase. As well as measurements made at the disk overlaps, we need the intensities of the diffracted beams themselves (lying outside the region of overlap, but within a single disk, as in Fig. 5). Our task is to solve for the phase difference between Q0,0 and Q1,0 given that |Q0,0 |2 = I1 (0, 0) = I2 (0, 0), |Q1,0 |2 = I1 (u, 0) = I2 (u, 0), u ,0 , |Q0,0 + Q1,0 |2 = I1 2 Q0,0 eiγ + Q1,0 e−iγ 2 = I2 u , 0 . 2

(17)

In conventional crystallography, we are usually interested only in the structure of the object, not its absolute position in terms of x and y. A displacement ambiguity always occurs in the classic PDI phase problem: the intensity of the Fraunhofer plane is unaffected by lateral displacements of the object. In other words, in normal diffraction, we are never sensitive to a linear-phase ramp applied across the diffraction plane. The usual crystallographic convention assumes that a point of symmetry is implicitly at the origin of real space. In contrast, in ptychography there is an explicit definition of absolute position


109

in real space – the known position of the confined illumination – but there is no reason to suppose that the crystal itself just happens to be aligned centrosymmetrically with this point in space. If the “crystallographic phase” of Q(u, v) (arising from adopting a centrosymmetric convention for the phase of the diffracted beams) is Q0 (u, v), then Q(u, v) = Q0 (u, v)−i(ux+vy) ,

(18)

where x and y is the displacement of the crystal relative to the optic axis. As a result, the particular phases of Q0,0 and Q1,0 that are consistent with Eq. (17) relate to the exact geometry of the lens, which defines the exact position of the illumination function a(x, y). In practice, the value of γ , proportional to the probe movement distance, can be chosen at will. In almost all conceivable experimental implementations of crystalline STEM (electron) ptychography, the exact position of the probe relative to the object is not known accurately; only relative displacements can be measured (say by applying an offset to the probe scan coils in STEM). In this case, we could choose to scan the probe position until I1 ( v 2 , 0) is at a maximum and simply assert the convention that the probe at this position is at x = y = 0. Relative to this definition, then Q0,0 and Q1,0 have zero phase difference between them (because in this condition they add to give maximum modulus). However, the phase of all other reflections will be altered accordingly via Eq. (18). Equations (17) can be solved either geometrically (via the complex plane construction in Fig. 2(b)) or by direct substitution. Ill-defined solutions can still arise if we choose the value of γ unfavorably; for example, we find that I1 = I2 if either x = nπ/u, where n is an integer, or when γ /2 happens to be the same as the phase difference between Q0,0 and Q1,0 . We note, however, that Eqs. (17) represent a tiny fraction of the number of measurements at our disposal. Obviously, we need to track the intensity at the midpoints of all pairs of reflections to solve for all of the relative phases of all pairs of adjacent beams. We can also record many diffraction patterns, from many probe positions, instead of just the two required to unlock the complex conjugate ambiguity. How to do this in the most computationally favorable way has been the subject of much work, which we will discuss in Section IV. F. Illumination a “Top Hat” or Objects with Finite Support The crystalline STEM implementation of ptychography is easy to understand intuitively because the illumination function (probe) in real space is of a particular form that conveniently provides a far-field diffraction pattern in which the interferences of the various pairs of beams are tidily separated. In general, the illumination function can be of any functional form. In the

110

RODENBURG

previous section, the exact shape of a(x, y) did not need to be known because it was itself a Fourier transform of a neat shape, which subsequently was convolved over the diffraction pattern. In his original papers outlining the key ideas in ptychography, Hoppe (1969a, 1969b) first described the STEM implementation and then went on to consider the effect of sharply defined rectilinear illumination patches, achieved by apertures of various dimensions mounted right up against the specimen. He observed that the same broad concepts for solving the phase problem applied in the Fourier domain. We note that this physical implementation is identical to the concept of a finite support that arises in much of the theory of phase retrieval, although Hoppe’s thoughts on this matter seem to exist in isolation, at least as far as the cited literature is concerned. It is easiest to understand why a finite support leads to a ptychographical solution of the phase problem by thinking of a 1D object function, q(x), illuminated by a top-hat function a(x), of the forms shown in Fig. 6. Let us first discuss this qualitatively. If q(x) is periodic, then its Fourier transform, Q(u) consists of a series of spikes (crystalline reflections), the modulus and phase of which determine the Fourier series components of the structure lying within the unit cell. When the crystal is perfect and of infinite extent, then there is no scattered amplitude lying between these discrete reflections. When we isolate only a small region of the specimen by the function a(x), then the diffracted amplitude M(u) is, via Eq. (7), the convolution of the Fourier transform of a(x) with Q(u). Provided a(x) has small physical extent (so that A(u) has a width that is a significant proportion of the reciprocal lattice spacing), then the effect of the convolution will be to introduce substantial amplitude lying at points between the conventional lattice reflections; it is here that adjacent diffraction orders can interfere with one another, just as in the STEM configuration. Once again, by moving the aperture function laterally in the object plane, a phase ramp can be introduced across A(u), altering the relative phase of the interfering crystalline reflections, and hence obtaining a direct estimate of their phase. Unfortunately, A(u) is now of the form of a sinc function (see Eq. (20) below) and has infinite extent, so that the tails of many diffraction orders interfere with one another at points midway between the diffraction orders (illustrated in Fig. 6). This is in stark contrast to the STEM configuration where we can arrange for two and only two pairs of beams to interfere at the reciprocal lattice midpoints. To state this argument mathematically, we write b < −x (19) 2 and a(x) = 0 elsewhere. The Fourier transform of a(x) can be derived from direct evaluation of the 1D from of Eq. (2a), yielding the definition of the sinc a(x) = 1

for x
2α/3 the third term is zero. In other words, for the high spatial frequency region 2α > |U | > 2α/3, we can write simply that

B(U ) = |A|2 δ(U ) − icF (U ) , (87) and the back Fourier transform is defined as the ptychograph

Πp (r) ≈ |A|2 1 − icϑ(r) = |A|2 q(r),

(88)

the subscript p designating that Πp (r) is a projection image of the object (see below). Amazingly, the transmission function of the object has been reconstructed, scaled by the intensity of the illumination. If fact, we have made two approximations; we have ignored the second-order F 2 term and the lower third of the frequency spectrum of the image. However, both of these issues arise in conventional imaging, but the conventional image is additionally subject to all the usual constraints of the form of the particular transfer function, including its zeros, whereas Πp (r) has unity transfer at all spatial frequencies as emphasized by Cowley (2001). Furthermore, its transfer function is twice the width of the imaging aperture, and in theory this aperture

154

RODENBURG

can be as large as desired because aberrations and defocus instabilities do not appear to express themselves in Πp (r). That is half the good news. Before burdening the reader with the (very) bad news, I first explain perhaps an even more interesting property of Πp (r). It is called a projection image because Πp (r) only reconstructs information from a set of perfect Bragg reflections that explore a plane of reciprocal space (not the curved surface of the Ewald sphere). Consider Fig. 22. By reciprocity, we illuminate an object in a conventional TEM by radiation incident at a particular angle β0 , where u0 = sin β0 /λ. If the object is reasonably weak, then the image will have an intensity component arising from the interference of this unscattered wave with waves scattered through the objective aperture. It is well known (especially by electron microscopes salesmen) that the highest-resolution interference fringes in any TEM are obtained by illuminating a crystal tilted off the zone axis into a strongly excited two-beam condition with a g-vector of, say, g. If the beam tilt is now put at −g/2, then the two beams (transmitted and diffracted) will travel through the objective lens at equal and opposite angles to the optic axis as shown in Fig. 22. The interference fringes obtained in the image plane will have a spatial frequency of U = g. However, as far as demonstrating the best-possible fringe resolution of a microscope, this condition is very

F IGURE 22. The top diagram shows the STEM configuration; the bottom diagram shows the equivalent TEM experiment via reciprocity (compare Fig. 15). The Bragg–Brentano data set interferes an undiffracted beam (labeled UD) with a beam that set off from within the lens in the direction of beam DIFF. The latter is diffracted through twice the Bragg angle and hence interferes with UD at the detector pixel (marked with a black box). By reciprocity, this is the same as illuminating the object with a beam tilted by θB from the optic axis. The diffracted beam now lies at an equal but opposite point in the back focal plane. In the image plane, we have lattice fringes. Πp takes a Fourier transform of these fringes (i.e., a Fourier transform with respect to the probe position coordinate R), and hence extracts only the information relating to these symmetric Bragg beams.


155

convenient because the two beams lie on the same achromatic ring of the object lens. In other words, variations in lens excitation or energy spread (chromatic spread) in the beam induce identical phase changes in the two beams. The fringe resolution is therefore only limited by the effective source size in the illuminating beam, which leads to a substantially better resolution (in the fringes) than can be obtained when the illumination is parallel with the optic axis, in which case all the image-forming beams are not on the same achromatic ring as the transmitted beam. Now in the case of B(U ), all such pairs of achromatic beams are processed simultaneously. This is possible because in the STEM configuration each detector pixel is equivalent (by reciprocity) to each possible illumination beam angle in TEM. The sets of interference fringes that would be measured in TEM are separated out from one another by the Fourier transform of I (u, R) we undertake with respect to R. (Remember that R is the TEM image pixel coordinate.) Now consider how these symmetric beams correspond to one another in 3D reciprocal space. As the angle of illumination (in the TEM view of this experiment) is increased to u, we pick out the periodicity 2U in the image plane: that is to say, the periodicity that arises from the interference of the incident beam with the beam that is scattered to −u. We see, therefore, that for all values of u we are in a symmetric Bragg scattering condition, with u = sin θB /λ. This is exactly the same as in the X-ray Bragg–Brentano condition, except there the goniometer of the specimen is rotated at half the angular speed of the detector. With an X-ray diffractometer the principal advantage is that a large specimen diffracts the same Bragg beam into the detector provided both are equidistant from the object, thus optimizing counting statistics. In terms of the description in Section III.C, we see that the negative curvature of the 3D surface S compensates for the curvature of the Ewald sphere, thus solving for a plane in 3D diffraction space (Fig. 23). A plane in reciprocal space corresponds to projection in real space. This is simply a restatement of the fact that Πp (r) is independent of defocus. The projection property of Πp (r) was demonstrated by Plamann and Rodenburg (1994) using visible light optics. Two TEM copper grids were mounted separated by a distance of 14 mm (Fig. 24). The two square grids were rotated at an angle to each other and illuminated by a convergent beam of laser light emulating a STEM probe. The lens diameter and working distance were chosen such that the intrinsic resolution of a conventional image was ∼120 µm. This was just sufficient to resolve the grid bars (spacing 250 µm, each of width 50 µm) but was large enough so that the illuminating probe spread significantly through the depth of the entire object. In other words, by adjusting the z-displacement of the object, either grid could be brought noticeably into better focus than the other. Data were collected on a CCD camera as a function of all specimen shifts, −R, equivalent to shifting the

156

RODENBURG

F IGURE 23. In the Bragg–Brentano data set, the incident beam is symmetrically tilted relative to the optic axis with respect to the scattered beam. In this way, we solve for a plane in reciprocal space, not the amplitude lying over the Ewald sphere as in the conventional image (compare Figs. 12 and 14).

F IGURE 24.

Schematic of the specimen used to obtain the reconstructions in Fig. 25.


157

F IGURE 25. Images a and c are the conventional visible light analog STEM images of the object shown in Fig. 24. Image a is with the beam focused on the diagonally mounted grid, and image c is on the squarely-mounted grid. Images b and d are their respective Πp ptychographs (all these images are in modulus only). The Πp ptychographs produce a good representation of the projection of the object at double resolution. Reproduced from Optik, courtesy of Elsevier.

illumination through R. In the conventional bright-field images obtained with the beam crossover focused on each grid, we can see poor images, each showing the respective grids at different angles to one another (Fig. 25, a and c). In the corresponding Πp (r) constructed from the same experimental data (Fig. 25, b and d), but using all u and the respective Fourier components of U , we obtain a double-resolution image of the two grids superimposed on each another – a projection of the object. The reason the image has higher resolution can be thought of as arising from the fact that an incident beam lying at the edge of the aperture function is scattered through twice the angle of the aperture radius to meet up with the beam that set off from the opposite

158

RODENBURG

side of the aperture. The conventional image interferes the on-axis beam with beams lying up to only the radius of the aperture function, not its diameter. Plamann and Rodenburg (1998) undertook some electron dynamical scattering calculations for the same scattering geometry as the Bragg–Brentano subset for the silicon [110] zone-axis dumbbell Πp (called reconstruction method II in the cited paper). The results are shown in Fig. 26. The lower image in each image pair is the phase of Πp , which approximately preserves the overall structure of the silicon dumbbells up to a total specimen thickness of almost 30 nm (these calculations were performed for 300 keV electrons). If we look at the phase and amplitude of the Bragg beams of which Πp is composed (Fig. 27), then what we see is that all the beams accumulate phase as an approximately linear function of thickness. By multiplying Πp by an

F IGURE 26. Modulus (top rows) and phase (bottom rows) of Πp , simulated by multislice calculations, for various specimen thicknesses of silicon (see text). Thickness ranges from 2.9 nm (top left) to 28.8 nm (bottom right). The phase image is a reasonable estimate of the structure of the dumbbells. A better representation can be obtained by subtracting the phase ramp (shown in Fig. 27) and plotting the imaginary part of Πp (Fig. 28). Reproduced from Acta Crystallographica with permission of the International Union of Crystallography.


159

appropriate phase offset, the imaginary part of the result is almost a perfect projection of the potential up to a thickness of ∼20 nm (Fig. 28). After enumerating the very beneficial properties of Πp , we now have to face up to some moderately bad news. Something that we have so far ignored in the theory above is that we assume that pixels in u and R are infinitesimally small and that the positions and angles spanned by R and u are infinite. If our Fourier transforms are not undertaken over infinite limits, the convenience of the Dirac delta function no longer applies. This is not intractable, but it must be accounted for by using a softly varying filter function applied at the experimental limits of the data (McCallum and Rodenburg, 1992). A related

(a) F IGURE 27. The modulus (a) and phase (b) of various calculated diffracted beams (see text) as a function of specimen thickness: these are the beams that compose Πp . Solid line: (111). Lowest amplitude line is the forbidden reflection (002); in this configuration it remains virtually unexcited until a thickness of 20 nm. Other beams shown are (220), (004), and (113). The main inference is that all the beams acquire a (roughly) linear phase change as a function of thickness. This is why the phase-offset ptychograph (Fig. 28) is more closely representative of the atomic structure. Reproduced from Acta Crystallographica with permission of the International Union of Crystallography.

160

RODENBURG

(b) F IGURE 27.

(continued)

problem that severely affects the experimental feasibility of the method is that these integrals must indeed be performed over reasonably large fields of view. In other words, a great deal of data must be collected before any processing can occur – unlike the PIE method (see next section), there is no possibility of constructing a real-time image of the object no matter how much of how much computing power is available. If the detector pixels are finite in size – which of course they have to be to get any counts at all – then the aberration insensitivity is lost, at least to first order. This is because when the aberrations are large, the intensities fringes in the Ronchigram (the central disk, the shadow image of the probe-forming aperture) are on a very small scale. When a finite pixel integrates over an area larger than a fringe spacing, the required signal is obliterated. We can picture this via reciprocity as the illuminating beam in TEM subtending a significant range of incoherent beams at the specimen plane – an important factor which affects the usable


161

F IGURE 28. Πp calculated for crystalline silicon on the [110] zone axis, but with a phase offset (corresponding to the inner potential) subtracted from the ptychograph. The bottom row of images shows the imaginary part of this phase-offset image (the top row is the real part of the same image). The phase very closely represents the projection of the atomic potential in the z-direction, even up to rather large thicknesses (17.3 nm on extreme right). Refer to Plamann and Rodenburg (1998) for further details. Reproduced from Acta Crystallographica with permission of the International Union of Crystallography.

information limit. The form of the effective transfer function given these parameters was examined by Nellist and Rodenburg (1994). Now for the very bad news. It should be noted here that there are some substantial experimental complications with actually collecting and processing these data, even if the mathematics is so neat. This issue is rather close to my own personal experience. After the initial rush of optimism on realizing the potential benefits of processing I (u, R) in various ways, and after much fanfare at conferences declaring that this must be the way ahead for TEM, experimental realities began to bite. Yes, the data seem to be robust to so many of the usual constraints of the transfer function and partial coherence, but there is a correspondingly large complement of different experimental difficulties. To construct Πp (r), we may only need a 2D plane in I (u, R), but we still have to collect the whole of I (u, R) in order to extract it. Between 1991 and 1996, our small Cambridge group struggled, with increasingly marginal returns, to put this (and the WDDC method described in Section IV.D) onto a practical experimental footing with high-energy electrons. Reconstructions of Πp (r) were obtained, but their quality and scientific relevance were exceedingly low. An example, the best obtained after several years’ work, is shown in Fig. 29. It is on a grid of 32 × 32 pixels in R. That meant that more than 900 diffraction patterns needed to be collected. Even if captured at TV rates (40 ms/frame), this implies that we needed a total data capture time of 36 s. To confirm that the reconstruction was at least reproducible, the data were collected from the same region of specimen twice to check that the reconstructions from independent data but from the same area of

162

RODENBURG

F IGURE 29. The phase of two Πp double-resolution electron ptychographs of amorphous carbon. The two images are of the same area of the specimen (field of view 2.5 nm), but using entirely different data collected one after the other. The accelerating voltage was 100 keV. The objective aperture (in STEM mode) was stopped down so that the intrinsic resolution of the imaging lens was 0.8 nm. The reconstruction doubles this resolution and is ∼0.4 nm. Certain features (e.g., the bright V shape (top right) and the dark V shape (top left), are clearly reproduced. However, the reconstructions are not very impressive given the effort undertaken to obtain the necessary data (see text and Rodenburg et al., 1993 for further details). Reproduced from Ultramicroscopy, courtesy of Elsevier.

object were the same. For a number of irritating experimental reasons related to simultaneously controlling the microscope and synchronizing the data capture, TV rates could not in fact be achieved in practice. The data for this tiny image therefore took about 80 s to collect. This may not sound long until it is remembered that all the foregoing theory assumes that the data are collected from a set of equally spaced points in R, each separated at the ångstrom level, with no distortion in the scan. Lateral wobble in the probe position (the source of the ptychographic phase) is fatal. During the course of his doctoral work, Peter Nellist slowly removed more and more of the wires connecting the console of the microscope (a VG HB501 dedicated STEM) to the column in an attempt to remove sources of electronic instability in the probe position. This, combined with the other usual difficulties of TEM (such as specimen contamination, specimen damage, and those phenomena that many microscopists secretly recognize but do not very often talk about openly – like the occasional proclivity for the specimen to jump laterally every few tens of seconds, presumably because of charging – further hampered our efforts. Other irritations included charging of the objective aperture, inducing non-round aberrations (in a dedicated STEM, the objective aperture is the probe-forming aperture); hysteresis in the scan coils; earthing loops; imperfections and blooming in the detector (we used an


163

image intensifier); the general paucity of electron counts that we could obtain when the source was sufficiently demagnified to achieve an appropriate (even if rather low) degree of coherence; vibration in the field-emission tip (the VG design had the tip mounted on a thin bent wire, which had several vibrational modes with which we became all too familiar); and, perhaps worst of all, the need to reduce thermal drift to a usable level of about 1 or 2 Å/min. Even quite simple things, like focusing the probe at the specimen plane, were difficult tasks when the machine was switched to collecting diffraction patterns. In short, this was very much a living nightmare, made bearable only by the persistent sense of humor of the group. Cowley and Winterton (2001) did eventually implement the scheme successfully with electrons for a 1D scan of probe positions across the edge of a nanotube. Perhaps now that detector and data transfer/storage technology is so much more advanced, it may be worth revisiting the technique for 2D objects. C. Ptychographical Iterative (pi) Phase-Retrieval Reconstruction Conventional iterative phase-retrieval methods collect a set of data, such as a single diffraction pattern from an isolated object, and then process this offline, applying object size constraints in real space. However, as a microscopic imaging method this leaves much to be desired. Microscope users need to be able to scan their specimens over large fields of view in order to identify areas of interest. Once identified, the user must be able to zoom in to a feature of interest, observing it at higher and higher resolution. Isolated object PDI methods do not allow for this sort of essential microscope functionality. In this section, we are going to discuss a recent development in the iterative phase-retrieval method that actually owes much more to the ptychographical direct solution: multiple diffraction patterns are recorded as an illumination field is shifted with respect to the object of interest (or vice versa). A modified iterative method is used to construct the resulting image – a ptychograph that we will call Ππ , or a “pi-type” reconstruction. Here we examine in detail a particular variant of pi-type reconstruction called the PIE algorithm. Iterative methods of phase retrieval were first proposed by Gerchberg and Saxton (1972) and later developed significantly by Fienup (1978, 1982). Early work was performed on 1D objects, if only because of computational constraints at that time. John Chapman (as differentiated from Henry Chapman, who has also contributed significantly to this field) was the first to consider the practical issues surrounding an experimental implementation of the technique in the electron microscope for the 1D image – diffraction plane iterative phase-retrieval (Gerchberg–Saxton) method (Chapman, 1975). With the development of the personal computer, it became much easier to test the

164

RODENBURG

efficacy of iterative methods in two dimensions and see the results of different feedback parameters immediately. In two or more dimensions, the method is much more robust because of the usual phase problem constraints (discussed in Section II.D). Rather than requiring both image and diffraction intensity data (and hence a good-quality lens, as in the Gerchberg–Saxton algorithm), it was found (Fienup, 1978) that simply knowing that the object is isolated, effectively embedded in an empty unit cell that has roughly as many (known) zero-valued pixels as pixels in the object image enables us to solve the 2D phase problem very effectively without significant ambiguity. There are still stagnation issues resulting from the competition of the inherently ambiguous solutions (such as q(r) = q ∗ (−r), these having the same diffraction pattern intensity). Fully complex images are more difficult to solve for (Fienup, 1987), but with some refinements, this can sometimes be achieved. Following a rather stimulating conference organized by Spence et al. (2001) on solution of the nonperiodic phase problem, I was encouraged to revisit the ptychographical data set but to try to incorporate into it all the advantages of the iterative technique. The possibility of doing the PDI experiment in the electron microscope had been discussed as part of a review of ptychography (Rodenburg, 1989), but in view of the detector technology available at that time had been dismissed out of hand. Using image-plate technology, Zuo et al. (2003) obtained a very impressive reconstruction of a carbon nanotube from a single electron diffraction pattern via a single PDI data set. Given that it was possible to achieve an iteratively reconstructed image from a single diffraction pattern, and given that it is known that it is possible to solve the phase problem directly from two or more diffraction patterns via ptychography, it seemed logical to suppose that a hybrid method existed that could exploit an optimized combination of these methods. Landauer (1996) had briefly considered processing a number of images collected under different illumination conditions in the electron microscope to reconstruct the modulus and phase of the Fraunhofer diffraction pattern using the known constraint of the size and shape of the objective aperture lying in the back focal plane of the object lens. This was far from diffractive imaging; the work was purely theoretical and had to assume the presence of a good lens. However, it did indicate that multiple applications of the Fienup hybrid input–output method could be “sewn” together to extend the plane of interest (in this case, not the image but the diffraction pattern). Faulkner and Rodenburg (2004) used an aperture constraint alone, presumed to be coincident with the object plane, to show in model calculations that the iterative method could indeed be applied to extended real-space objects. Preliminary investigations into the possibility of manufacturing an aperture via focused ion beam lithography that was small enough to perform this type of reconstruction in the electron microscope were fraught with difficulties.


165

The aspect ratio of the hole required in order to block off the unwanted radiation was large (say a diameter of 10 nm with a depth of 200 nm). Such apertures contaminate quickly in the microscope and also introduce channeling and propagation artifacts. It became obvious that in any practical implementation of iterative phaseretrieval ptychography, at least in the electron microscope, it would be necessary to use a focused spot of illumination, as in the STEM configuration. Unfortunately, such patches of illumination are necessarily “soft” in the sense that there are always low-amplitude ringing effects, which extend to large distances in the object plane. Because the lens has a limited extent in reciprocal space, its real-space impulse-response function is always smooth. There are therefore no analogous support-type constraints in real space. Furthermore, any aberration or defocus in the lens introduces phase changes into the illuminating beam. A method was required that would feed into the phase-retrieval loop the essence of ptychography: when the illumination function is moved, its shape remains constant; equivalently, the diffraction pattern is a convolution, but a convolution that can be recorded for a series of phase ramps applied across the Fourier transform of the illumination function, according to the fundamental ptychographical equation (41). The breakthrough came with the development of the so-called PIE update function (Rodenburg and Faulkner, 2004a, 2004b). The use of the measured modulus constraint in the Fraunhofer diffraction plane is identical in PIE to that used in all conventional iterative phase-retrieval methods (Marchesini, 2007, provides an excellent review). The difference with PIE is that there is no simple support constraint in real-space. Instead, a current estimate of the object function is changed incrementally according to the intensity information that has been fed in from the diffraction plane. The iteration proceeds as follows. We seed the algorithm with an initial estimate of Ππ , which we will call Πe,1 (r), this being our first (n = 1) such estimate, labeled (A) in Fig. 30. It is usual to put Πe,1 (r) = 1; that is, we start by supposing that Πe,1 (r) is of the form of a transmission grating function of unity transmission (free space) at all points in r. Suppose we have a current nth estimate of the ptychograph Πe,n (r). Then in the next (n + 1)th iteration, a modified estimate, will be given by Πe,n+1 (r) according to the equation

Πe,n+1 (r) = Πe,n (r) + U (r) ψc,n (r, R j ) − a(r − R j ) · Πe,n (r) , (89) where U (r) is an update function, given by U (r) =

a ∗ (r − R j ) |a(r − R j )| . |amax (r − R j )| (|a(r − R j )|2 + ε)

(90)

166

RODENBURG

F IGURE 30. The PIE flow diagram. At ‘A’ we start with an initial guess of the object function, Πe,n (r), usually that it is perfectly transparent (i.e., of unity modulus everywhere and zero phase). This estimate is then continuously modified at each iteration via the update function. An iteration consists of choosing a particular illumination position, R j , and then forming ψe = a(r−R j )·Πe,n (r). We then propagate this to the far field (in fact, any propagator can be used), preserve the phase of the resulting diffraction pattern amplitude but substitute the modulus calculated from the square root of the intensity of the diffraction pattern obtained from that probe position R j to give an estimate of the complex value of the diffraction pattern. The difference between ψe and the corrected back-propagated exit wave, ψe , is used to update Πe,n (r). Any number of diffracted patterns can be used (three are shown, one dotted). Hollow pointers represent the transfer of data, line arrows show the flow of control.

In Eq. (89) we multiply our current estimate of the ptychograph by our illumination function (which we presume is known accurately) when it is at a particular position R j with respect to the object function. Assuming the multiplicative approximation (Eq. (3)), we now form what would be an estimate of the exit wave if the object was truly a 2D grating, represented in Eq. (89) by the last term in the square brackets, that is, a(r − R j ) · Πe,n (r). This wave is then propagated to the detector plane where, as usual in iterative


167

methods, we preserve the phase of the resulting calculated diffraction pattern but replace the modulus with I (u, R j ). We then transform back to the object plane to find a “corrected” estimate of the same exit wave function (corrected in the sense that measured intensity data in the far-field have now modified our original estimate of the exit wave), appearing as the first term in the square brackets of Eq. (89), which we call ψc,n (r, R j ). We subtract (point-by-point) the estimated exit wave from the corrected exit wave. The resulting difference function is then fed back into our running estimate of the ptychograph. The amount we change the current estimate Πe,n (r) is different at different parts of the image, weighted point by point as a function of the probe amplitude when it is positioned at R j . In Eq. (90) amax is the maximum modulus value of the illumination function, and ε is a constant determined by the total number of counts in a typical diffraction pattern. This term is like a Wiener filter as used in conventional deconvolution methods in order to suppress the effects of noise in the data, especially in the regions where a(r) has low or zero amplitude. It is essentially a division by a(r), effecting the deconvolution of the ptychographic folding (Eq. (41)) of the far-field intensity. Although the PIE can be run with some success using only one diffraction pattern (one value of R j ), its real advantages arise when data from different probe positions are used concurrently. If the area of the illumination function has an approximate diameter d, convergence is rapidly increased if data are collected from a grid of points separated in R by about d/2. Thus, for example, we could collect four diffraction patterns from R 1 = (0, 0), R 2 = (d/2, 0), R 3 = (0, d/2), and R 4 = (d/2, d/2). The first iteration could use experimental data I (u, R 1 ), the second I (u, R 2 ), and so on, returning at n = 5 to reuse data I (u, R 1 ) and so on, reusing each diffraction pattern in a cyclically repeating order. However, many other permutations are possible. The j th pattern could be used for the first hundred iterations before moving on to the (j + 1)th pattern. Clearly, this latter strategy results in rather slower convergence because the ptychographic phase data are only being exploited once in every hundred iterations. In general, processing a different pattern for each subsequent iteration is the most effective strategy. Indeed, this can produce a reasonable quality image after iterating with each diffraction pattern just once (Rodenburg et al., 2007a) In conventional iterative methods, it routinely takes thousands of iterations before a reasonable image can be obtained. The method is also fantastically immune to noise. Faulkner and Rodenburg (2005) modeled a variety of experimental sources of noise in a STEM configuration and found that recognizable images could be obtained when Poisson noise corresponding to very low beam currents was added. Up to 50% of random noise added to each detector pixel reading also produced interpretable images. Conventional

168

RODENBURG

iterative methods often cease to converge sensibly at noise levels of more a than few percent. It should be remembered that the data collection process is completely decoupled from the iterative calculation itself. The reconstruction can be processed off-line, after all the data have been collected, or in real time, concurrently as further data are being collected. Because the PIE is completely indifferent to the position of the next R j , the field of view can be extended continuously, in real time, according to whatever part of the specimen the user wishes to image. This is one of the defining properties of PIE. The field of view is independent of the size of the detector. Clearly, the sampling condition must still be satisfied, but this depends only on the width of the illumination function – it is nothing to do with the size of the final field of view. The forward and back propagations necessary for each iteration can be conducted on a reciprocal space sampling of say, us , which defines a value of d ≈ 2/us . However, any particular value of R j is unconstrained by d. We could use data from R 1 = (100d, 50d) and R 2 = ((100 · 5)d, (50 · 5)d) and use the update function to alter only the area of ptychograph spanned by the illumination function when it is centered on either of these coordinates. There is nothing to stop Ππ spanning a total image area of, say, 1000×1000d 2 . If the detector has 1000 ×1000 pixels, then in this situation the entire ptychograph could contain 106 ×106 pixels: unlike conventional iterative methods, the size of the reconstruction can be as large as desired. Figure 31 shows a visible light experimental demonstration of the PIE method (from Rodenburg et al., 2007a). The illumination function was generated simply by an aperture somewhat upstream of the object (a 3-mm long ant sectioned and mounted between a microscope slide and cover slip). The aperture was 0.8 mm in diameter, with λ = 630 nm. The object was moved over a grid of 10×10 illumination positions, each separated by slightly less than half the aperture width. An example of a typical diffraction pattern is shown in Fig. 32, as is the recorded intensity of the illumination function at the plane where it impinges on the object. As can be seen, the disc of the realspace aperture has been blurred out by propagation, causing Fresnel fringelike features to develop around the edges of it. This function is therefore “soft” and does not present us with the sort of sharp support function required for conventional iterative phase retrieval. Figure 31 shows intensity and phase reconstructions of Ππ . The resolution of the ptychograph is ∼17 µm; that is, about 50 times smaller than the diameter of the aperture. Note that in the phase of the reconstruction we can discern a phase ramp across the entire image. This is due to the cover slip not being perfectly parallel to the underlying slide. More recent unpublished work has shown slowly changing phase distributions over very wide fields of view can be accurately reconstructed. The same sort of information can be extracted from


169

F IGURE 31. Visible light Ππ reconstruction (i.e., Πe,n , where in this case n = 25). Residual structure in the modulus (left) arises from slight detector misalignment and misestimate of the illumination function: the 10 × 10 scan of probe positions is hence visible. Note that in the phase image (right), a slowly varying phase ramp is discernible – such large-scale phase variations (in this case due to a curved cover slip) are lost in conventional Zernike and electron microscopic images. The ant is ∼3 mm in length, the scale bar is 1 mm. For more details, see Rodenburg et al. (2007a). Reproduced from Ultramicroscopy, courtesy of Elsevier.

F IGURE 32. Raw data of ant reconstruction shown in Fig. 31. The intensity of the illumination function (left) is measured at the plane of the object. Note that propagation between the aperture and the object has led to the development of Fresnel fringes around its edges. A typical diffraction pattern is shown on the right. Reproduced from Ultramicroscopy, courtesy of Elsevier.

holograms, but here we do not have any interferometric optical component or lens, except the diffraction itself occurring at the object plane. This is a potentially important imaging capability of pi-type ptychography.

170

RODENBURG

F IGURE 33. Hard X-ray PIE reconstructions from Rodenburg et al. (2007b). Image A: High-resolution SEM image of the object. B: Examples of nine of the diffraction patterns recorded from the illumination at the positions circled on A. C: Modulus and D phase of the Ππ ptychograph. Aliasing effects in the reconstructions are due to the sampling in the image file; these features are not visible in the raw data – see the plot across the outer rings of the zone plate in D. Reprinted with permission from Phys. Rev. Lett 98 (2007) 034801, Copyright 2007 American Physical Society.

A much more scientifically significant result is shown in Fig. 33 (from Rodenburg et al., 2007b) where an exactly analogous experimental configuration was used with hard (8 keV) X-rays. In visible light optics there is very little need to resort to ptychography to obtain good-quality images. However, any sort of imaging with X-rays is difficult, and especially difficult with hard X-rays because of the thickness required in the zone plate to introduce the requisite phase change in the incident beam. The round object in the center of Fig. 33 is in fact such a zone plate, although here it is being used as a resolution test object. The experiment was performed at the Swiss synchrotron


171

at the Paul Scherrer Institute in collaboration with the group of Franz Pfeiffer. The only difference in the experimental setup is the source of radiation (a substantially coherent third-generation synchrotron) and the scale of the aperture, which in this case had a diameter 5 µm. The aperture to object distance was a few millimeters. It was not possible to image the intensity of the incident illumination field, and so this had to be estimated by assuming the aperture was a perfect disk illuminated evenly by a perfectly coherent plane wave and propagating this computationally to the object plane. The object was stepped through a square array of 17 × 17 positions, each separated from one another by 2 µm. The (mechanical) stepper motors used were not of a particularly accurate type: piezoelectric actuators could certainly improve the result. The specimen includes a random distribution of gold particles scattered around the zone plate. The reconstruction (especially the intensity component) is far from perfect, but this was the first attempt at this experiment; the data were collected over a single weekend on a beamline that was scheduled for maintenance. Given that the outer zones of the zone plate are resolved, the ptychograph has clearly achieved resolution that is as good as that which could be obtained from this (reasonably state-of-the art) zone plate when used as an imaging lens. With further development, it would seem that pi imaging might provide routine hard X-ray micrographs at potentially much higher resolution than currently available technology. Pi is certainly the most compact and elegant solution to the calculation of Π (r) from the fundamental ptychographic equation (41). The deconvolution of the diffraction plane amplitude (Fig. 14) is achieved in real-space by the division implicit in Eq. (41) of a(r). The method allows for immediate realtime output of ptychographs as data are concurrently being collected. The solution so obtained can be refined in one part of the field of view while further data are collected from new areas of the object. There is no limit to the size of the field of view. The computational method has been called an “engine” (Faulkner and Rodenburg, 2005) – a “ptychographic iterative engine” (PIE) – because, unlike all the preceding iterative phase algorithms, the measured data arrays can be continuously expanded while the algorithm progresses through the data. In principle, the user can define in real time which parts of the objects should be reconstructed. The pi data set is of dimension Nu × Nv × N1−∞ × N1−∞ . In this sense, it really is a diffractive method, most of the information being captured in the u coordinate, only a few sparse points in the R plane being required. It is both extremely flexible in that only as many data as are needed for a particular object size need to be collected (the number of illumination positions is variable between 1 and infinity). Since each area of the object is generally illuminated about twice, it is exceedingly efficient relative to

172

RODENBURG

the Bragg–Brentano or Wigner distribution deconvolution (WDDC) methods (Sections IV.B and IV.D). Indeed, the method will work even if most of the object is only illuminated once, although increased redundancy helps with the convergence of the algorithm. D. The Wigner Distribution Deconvolution Method In Section IV.B we formed a function G(u, U ), calculated by taking the Fourier transform of I (u, R) with respect to the R coordinate. From this we could extract B(U ) and hence the projection pytochograph. However, B(U ) is just a 2D subset of an intrinsically 4D data set. By picking only one particular Fourier component in U for each location in u, a huge amount of data are wasted. In fact, the Bragg–Brentano subset was only developed much later as a result of a different line of reasoning that had been sparked in the autumn of 1988 when Owen Saxton very helpfully introduced me to Professor Richard Bates, who was passing through Cambridge on his way from New Zealand to a conference in Europe. Our meeting only lasted 20 minutes and was mostly spent brewing and drinking tea. Richard was severely jet lagged. His long white hair was characteristically disheveled; he struck me as the archetypal mad professor. I had an imminent teaching commitment that afternoon and so had to bring the social niceties to an abrupt end. In the last two minutes before racing off, I scribbled down Eq. (41) and said that I would be very grateful to receive any suggestions on how to obtain a comprehensive solution for the object function q(r), given only the intensity I (u, R) and a knowledge of a(r). I did not hear from him at all until I received a telex 6 months later: it said, “I have solved your problem. Please send fax number.” The stream of equations that subsequently came out of the fax machine at first completely confused me. A day or two later I was first enlightened and then delighted. Two weeks later I was frantically writing a research grant proposal. Without that telex, this Chapter, and most of what is in it, would not exist. The brief collaboration that followed was truly fruitful. I was a cynical experimentalist; Richard was an inspired theoretician who passionately loved the phase problem and had worked on it for many years from a wide range of perspectives. I could relate many stories at this point: sometimes sparks would fly; Richard was a true original but also, at times, incredibly stubborn. He died of cancer 2 years after our first meeting, just as a vast horizon of future work was expanding before us. I was impressed that he remained absolutely focused on productive work right until the very end, never at any point being aggrieved that fate had determined that he would die much younger than today’s average span.


173

What he wrote in that fax (disguised somewhat in his inimitable nomenclature and penchant for massive equations) was that if (91) H (U , r) = I (u, R)ei2π(R·r−u·U ) du dR, then H (U , r) = χq (U , r) · χa (U , −r), where for some general function, f (r), χf (U , r) = f (r + r a )f ∗ (r a )ei2πU ·r a dr a .

(92)

(93)

Similar equations occur in the radar literature, where Eq. (93) defines the “ambiguity function,” although in that field the measured data are phase sensitive insofar as the time of arrival of a reflection is known. This class of function is also called a Wigner distribution function (Johnston, 1989) – it arises as an expression of the uncertainty principle in quantum mechanics, reciprocal and real-space coordinates. Why is this important? First, it served as a very useful mathematical tool to analyze the whole of the “entire” data set I (u, R). Second, it showed that I (u, R) could be decomposed in this mixed real and reciprocal space into a product of two functions, one of which depends only on the specimen transmission function q(r), the other depending only on the illumination function a(r). It follows that I (u, R) can in principle be separated via deconvolution into a 4D function that depends only on the object function and has all the properties of the probe forming optics, whatever they may be, removed from it. Third, once this separation has been accomplished, one further Fourier transform (over two of the four coordinates) of χq (U , r), can yield one of two representations of all the relative phase relationships in the scattered wavefield. In real space we have L(U , R) = q(R + U )q ∗ (R),

(94)

and in reciprocal space we have D(u, r) = Q∗ (u − r)Q(u).

(95)

It is perhaps not immediately obvious that either one of the two equations above gives the phase of the entire scattered wavefield (or at least, the phase and amplitude of the resulting ptychograph). In fact, it does so with massive overredundancy. Of course, in any phase problem we are never going to able to solve the absolute phase of the scattered waves because this changes at very high frequency as a function of time and so is meaningless in the context of the time-independent wave equation. This is expressed in, say, Eq. (94), by

174

RODENBURG

the fact that only the difference in the complex value of q between a point at R and R + U is expressed in L(U , R). We therefore start by agreeing to assign zero phase to one pixel in q(R), for example, by putting (96) q(0) = L(0, 0). The WDDC ptychograph is then given by L(U , 0) . (97) ΠW (U ) = √ L(0, 0) However, we can choose to assign any phase to any pixel in q(R), and form a different ptychograph by dividing L(U , R) by that pixel. In other words, we can form a whole family of ptychographs L(U , R u ) . (98) ΠW (U , R u ) = √ | L(0, R u )| It was quickly discovered (Rodenburg, 1989) that though the WDDC could be made relatively immune to noise by performing the deconvolution step using a Wiener filter, fatal errors arose when the illumination function was “soft” in being bandwidth limited in the sense that there is a sharp aperture in the back focal plane (e.g., as in the case when a(r) is of the form of a STEM probe generated by a lens with a diaphragm). This is akin to the difficulties that arise in iterative phase retrieval in the same configuration when there is no definite and sharp support function in the object plane. The calculation presented in Bates and Rodenburg (1989) had to be performed using a truncated probe (i.e., a finite probe cut off with a sharp edge) in order to make the method work at all. Although this was alluded to in the last paragraph in Section 3 of that paper, it was of great concern to the author. Clearly, if a sharp-edged feature is a requirement of the method, then it is also true that a lens of very wide numerical aperture is required, defeating the whole purpose of ptychography. The reciprocal version of the method (Eq. (95)) was found much later to overcome this latter difficulty (Rodenburg and Bates, 1992). In reciprocal space, the aperture function is sharp, and so its Fourier transform has values out to very large values of u, meaning that the deconvolution step does not encounter divide-by-zeros except at a few points; these can be avoided by using other volumes of the data set (which is extremely redundant). Once again, we can construct a whole family of ptychographs. The fact that the aperture is finite may seem to imply that one such ptychograph, for example

D ∗ (0, r) $ ΠW (−r) = √ (99) D(0, 0) would be limited in r, and hence there would be no gain in resolution. But this is not the case. We use the term stepping out (see Section IV.A and


175

Fig. 17(a)) to suggest that even if D(u, r) is of limited extent in r, we can use the phase reconstructed from one plane, say from D(0, r), to obtain an estimate of $(ΠW (r)) up to the aperture diameter 2α. Once we have the phase at $(ΠW (2α)), we now use a new plane of D(u, r), namely, D(2α, r), to increase the extent of $(ΠW (r)), via

$ ΠW (−2α − r) =

D ∗ (2α, r) . $(ΠW (−2α))

(100)

The procedure can be repeated many times. There is also an equivalent but much richer set of possible phase-closure paths (as in the crystalline case; see Fig. 20). D(u, r) can be considered as a continuous set of all relative phase differences between every pair of Fourier components of ΠW . This leads us on to an important – perhaps the most important – theoretical result we can derive from the WDDC analysis. In practice, no electron or X-ray experiment is perfectly coherent. Even the Bragg–Brentano subset is subject to the coherence envelope if, as must be the case, the detector pixels are of finite size. Consider the question of spatial coherence deriving from the fact that any practical source is also of finite size. In the STEM configuration this means the back focal plane is not illuminated perfectly coherently. In the diffraction plane we can no longer write Eq. (41), but we must moderate the strength of the interference between every pair of beams that interfere with one another by a coherence function. The entire data set becomes I (u, R) = Γ (u − ua )A(ua )Q(u − ua )A∗ (ub ) × Q∗ (u − ub )ei2πR·(ua −ub ) dua dub ,

(101)

where Γ (u − ua ) is called the complex degree of coherence: assuming that the propagator between the source and the back focal plane of the probeforming lens satisfies the Fraunhofer approximation, then, via the van Cittert– Zernike theorem (see, for example, Born and Wolf, 1999), Γ (u − ua ) is given by the Fourier transform of the intensity profile of the source function. We simply state the result (derived in Rodenburg and Bates, 1992) that under these circumstances D(u, r) = Γ (r)Q∗ (u − r)Q(u).

(102)

In other words, if the transfer function is limited by partial coherence effectively reducing the usable aperture size, then this appears just as an attenuation factor in D(u, r), but only in the r-directions. The stepping out process is not affected by finite aperture size, requiring instead only adjacent beams to interfere with one another; the quality and resolution of the reconstruction obtained via WDDC are not limited by partial coherence.

176

RODENBURG

This is equivalent to the observations made in Section IV.A in the context of crystalline ptychography. Indeed, the whole point of the silicon line-data reconstruction was to demonstrate that ptychography could extract image data at a resolution not limited by the coherence width of the lens. Historically, the realization of the independence of the ptychographical data set on partial coherence came earlier (by about 4 years) than the very limited proof of this fact achieved using crystalline line-scan data set (described in Section IV.A). It was the magnitude of the experimental difficulties we faced in collecting the entire data set (see Section IV.B) that meant this (perhaps the most crucial) property of the WDDC data set could not be demonstrated on nonperiodic objects. Sadly, the order of publication has implied that the only ptychographical data set that is immune to partial coherence is the crystalline data set.

F IGURE 34. First visible light optical demonstration of the WDDC method. The top left image is the intensity of a “good” resolution conventional image, taken with a lens of relatively large numerical aperture. The bottom left image is the conventional image obtained with the aperture stopped down so that all the image features are no longer resolvable. All the other images are reconstruction from ptychographical data collected using this latter (small numerical aperture) lens configuration. The top middle and right images are the modulus and phase of the ΠW ptychograph reconstructed at double resolution. The corresponding lower pair of images are at four times improved resolution (compare bottom left), thus demonstrating the so-called stepping-out procedure for nonperiodic objects. See McCallum and Rodenburg (1992) for experimental details. Reproduced from Ultramicroscopy, courtesy of Elsevier.


177

F IGURE 35. First demonstration of the WDDC method using X-rays by Chapman (1996). Top left: the incoherent bright-field image of the object (a group of latex particles). Top right: the conventional coherent bright-field image taken from an offset pixel in the diffraction plane. Lower images are the modulus (left) and phase (right) of the ΠW ptychograph. The scale bar is of length 0.5 µm.

A 1D visible light demonstration of the WDDC principle was first achieved by Friedman (Friedman and Rodenburg, 1992) as his MPhil project. McCallum and Rodenburg (1992) demonstrated a 2D imaging version of the technique, also using visible light. Pictures of the reconstructions are shown in Fig. 34. Again, these are not particularly impressive, but it must be remembered that even the largest laboratory-scale computer workstations at that time only had about 64 Mbyte of RAM. Given that we performed the deconvolution in double-precision arrays, it becomes clear why images on a scale larger than 32 × 32 pixels (i.e., 900 diffraction patterns) could not be processed. In an astonishing solo effort, Henry Chapman demonstrated the technique using soft X-rays (Chapman, 1996), as shown in Fig. 35. These results were much more impressive than anything obtained in the electron microscope, but the gains in resolution relative to the incoherent image were nevertheless not entirely compelling given the extreme quantities of data and beam time required for the technique. Nellist and Rodenburg

178

RODENBURG

(1994) explored how other aspects of partial coherence in the electron microscope – for example, lens instability, detector pixel size (which can be thought of via reciprocity as form of incoherence function), and source voltage instability – would impact on the WDDC method. McCallum and Rodenburg (1993a) showed that the huge redundancy in the entire WDDC set means that it is possible to solve for both the object function and the illumination function simultaneously, provided it is at least known that the illumination function is well localized. For those who prefer not to deal with the mathematics, Rodenburg (2001) has written a rather simplified review using pictorial representations of how WDDC relates to Fourier holography, tilt-series reconstruction, and other methods. It is possible that it may be appropriate to revisit the WDDC method now Moore’s law has made computers so much larger, faster, and more powerful. Certainly there are many unanswered questions about how best to optimize the technique, especially in the context of the recent success of the PIE method. The latter is far more practical and much easier to implement. However, melding or extending PIE with WDDC or the Bragg–Brentano data in some way should, I think, be the subject of further work.

V. C ONCLUSIONS The scope of this Chapter has been limited to a particular definition of the rarely used term ptychography. I believe the definition chosen is in the spirit of that intended by its inventor, Walter Hoppe. However, there is no doubt that I have stretched this definition to embrace some techniques that are more comprehensive than the original concepts outlined in Hoppe’s 1969 papers. Indeed, my chosen formal definition excludes one of the techniques (the two-beam technique) published under the designation “ptychography” by Hoppe and his then research student, Hegerl (Hegerl and Hoppe, 1972). Some years ago I had dinner with Hegerl (who is now retired); I got the distinct impression that he did not very much enjoy working on his doctorate. Even in the Ultramicroscopy issue celebrating Hoppe’s retirement, Hoppe (1982) himself wrote about the subject in such a way as to suggest that this was an obscure but interesting idea that was never properly pursued and anyway would probably be exceedingly difficult to put into practice. Hegerl certainly seemed to be genuinely surprised that anyone should be bothered to want to know more details about the work that he had undertaken during a period of his life that he was grateful to have put behind him. Indeed, his initial reaction seemed to be that anyone pursuing this line of reasoning must themselves be heading towards disillusion and despair. (During the 1990s, I did temporarily reach a similar state of mind for the reasons discussed in Section IV.B.)


179

Since I am so close to the subject, it is possible that I have rather overinterpreted Hoppe’s insight into the phase problem, unconsciously dressing it up with the wisdom of hindsight. His original thoughts certainly appear to hang in isolation from the rest of the existing phase-problem literature. I suppose the concept was simply too far ahead of its time. In 1968 there were huge barriers to putting the idea into practice at atomic wavelengths. The first was that any sort of computing was extraordinarily more difficult than it is today. I imagine that Hegerl spent all of his research coding the necessary equations to solve for the ptychographical phases. The same work can nowadays be done in an afternoon using MATLAB. Indeed, solution of the phase problem in any field progressed so much faster once computers were powerful enough to provide real-time graphical output of images and functions. Such facilities only became widely available in the 1980s, and even then they were not entirely easy to use. Most of the progress we made in the early 1990s with the WDDC and Bragg–Brentano methods derived from the fact that we had the access to a dedicated network of high-powered workstations (which were anyway entirely puny relative to a modern laptop), all of which were equipped with graphical interfaces. The 4D data sets we used could be processed by the image processing program IMPROC, originally developed by Owen Saxton. The version we used had been greatly changed first by Richard Bates’ New Zealand group and then by Bruce McCallum, who had been a doctoral student with Bates and who rewrote it into a 4D version. Once we had immediate access to graphical output from slices taken through the entire 4D set and its Fourier transform, many new ideas came to mind and could be put into practice. Aside from computers, the two most important experimental issues that affect ptychography relate to the degree of coherence in the illuminating beam and the detector efficiency and dynamic range. In the case of X-rays, thirdgeneration sources with high coherence have only relatively recently become available. Although a suitable degree of coherence could be achieved in the STEM about the time of Hoppe’s work (Crewe was at that time obtaining the first atomic-resolution STEM results; Crewe et al., 1968), this technique was far from routine and vacuum constraints in STEM meant that photographic film (the highest-efficiency 2D detector technology at that time) could not be used because it would outgas into the ultrahigh vacuum. The work undertaken in the early 1990s on the method was therefore really the first time it could ever have been undertaken. Even then the detectors and computing facilities were only marginally capable of doing justice to the technique. Now that these technological difficulties are tractable, perhaps the age of ptychography has arrived at last. Various flavors of ptychography are discussed in Section IV. We have seen that, unlike holography or through-focal reconstruction, the method does

180

RODENBURG

not recover the exit wave unless the object is thin. However, the image it provides can, if processed appropriately, give a good representation of the object structure, even in the presence of dynamical scattering. Furthermore, the scattering geometry of ptychography (Fig. 14) gives access to a rather larger volume of reciprocal space than that of the conventional image. In Section IV.B, we saw why one subset of these data can provide a perfect projection of the object, suggesting that ptychography may be extended to 3D imaging via conventional tomographic techniques. A further key advantage of the technique is that it is not subject to the usual limitations of the coherence envelope and hence there is, in theory, no limit to the resolution of ptychography other than the wavelength. What we have not discussed here are the fundamental limitations imposed by thermal diffuse scattering and the fall-off of the elastically scattered signal at high angles, at least in the case of electrons. This will certainly define an ultimate physical limitation to the resolution of electron ptychographic reconstructions (which, short of using atoms or protons, will provide the highest imaging resolution of the solid state). However, these limits can in principle be reached via ptychography without expensive lenses or very high coherence. Perhaps the biggest potential benefit of ptychography will therefore be to dispose of the main costs of the electron microscope (the lenses), substituting them for a good detector and a large (but now inexpensive) computer. No other phase-retrieval technique can offer this prospect while at the same time delivering the usual imaging functionality of a microscope.

R EFERENCES Bates, R.H.T., Rodenburg, J.M. (1989). Sub-Angstrom transmission microscopy: A Fourier transform algorithm for microdiffraction plane intensity information. Ultramicroscopy 31, 303–308. Berndt, H., Doll, R. (1976). Method for direct phase determination in electron microscopic diffraction patterns. Optik 46, 309–332. Berndt, H., Doll, R. (1978). Electron interferences for evaluation of phases in electron-diffraction patterns. Optik 51, 93–96. Berndt, H., Doll, R. (1983). An electron interferometer for phase determination in diffraction patterns of natural crystals. Optik 64, 349–366. Born, M., Wolf, E. (1999). Principles of Optics, seventh ed. Cambridge Univ. Press, Cambridge. Burge, R.E., Fiddy, M.A., Greenaway, A.H. (1976). Phase problem. Proc. Roy. Soc. (London) Series A 350, 191–212. Chapman, J.N. (1975). Application of iterative techniques to investigation of strong phase objects in the electron-microscope. 1. Test calculations. Phil. Mag. 32, 527–552.


181

Chapman, H.N. (1996). Phase-retrieval X-ray microscopy by Wignerdistribution deconvolution. Ultramicroscopy 66, 153–172. Coene, W., Janssen, G., deBeeck, M.O., van Dyck, D. (1992). Phase retrieval through focus variation for ultraresolution in field-emission transmission electron microscopy. Phys. Rev. Lett. 69, 3743–3746. Cowley, J.M. (1995). Diffraction Physics. North-Holland, Amsterdam. Cowley, J.M. (2001). Comments on ultra-high-resolution STEM. Ultramicroscopy 87, 1–4. Cowley, J.M., Winterton, J. (2001). Ultra-high-resolution electron microscopy of carbon nanotube walls. Phys. Rev. Lett. 87. art. No. 016101. Crewe, A.V., Wall, J., Welter, L.M. (1968). A high-resolution scanning transmission electron microscope. J. Appl. Phys. 39, 5861–5868. Faulkner, H.M.L., Rodenburg, J.M. (2004). Moveable aperture lensless transmission microscopy: A novel phase retrieval algorithm. Phys. Rev. Lett. 93. Article No. 023903. Faulkner, H.M.L., Rodenburg, J.M. (2005). Error tolerance of an iterative phase retrieval algorithm for moveable illumination microscopy. Ultramicroscopy 103, 153–164. Fienup, J.R. (1978). Reconstruction of an object from the modulus of its Fourier transform. Opt. Lett. 3, 27–29. Fienup, J.R. (1982). Phase retrieval algorithms – A comparison. Appl. Opt. 21, 2758–2769. Fienup, J.R. (1987). Reconstruction of a complex-valued object from the modulus of its Fourier-transform using a support constraint. J. Opt. Soc. Am. A 4, 118–123. Frank, J. (1973). Envelope of electron-microscopic transfer functions for partially coherent illumination. Optik 38, 519–536. Friedman, S.L., Rodenburg, J.M. (1992). Optical demonstration of a new principle of far-field microscopy. J. Phys. D. Appl. Phys. 25, 147–154. Gabor, D. (1948). A new microscopic principle. Nature 161, 777–778. Gabor, D. (1949). Microscopy by reconstructed wave-fronts. Proc. Roy. Soc. (London) A 197, 454–487. Gerchberg, R.W., Saxton, W.O. (1972). Practical algorithm for determination of phase from image and diffraction plane pictures. Optik 35, 237–246. Giacovazzo, C. (1999). Direct phasing in crystallography: Fundamentals and applications. In: International Union of Crystallography Monographs on Crystallography. In: Oxford Science Publications, vol. 8. Oxford. Gureyev, T.E., Roberts, A., Nugent, K.A. (1995). Partially coherent fields, the transport-of-intensity equation, and phase uniqueness. J. Opt. Soc. Am. A 12, 1942–1946. Hawkes, P.W., Kasper, E. (1996). Principles of Electron Optics. Academic Press, London [see pp. 1641–1649].

182

RODENBURG

Hegerl, R., Hoppe, W. (1970). Dynamische Theorie der Kristallstrukturanalyse durch Elektronenbeugung im inhomogen Primärstrahlwellenfeld. Ber. Bunsenges. Physik. Chemie 74, 1148–1154. Hegerl, R., Hoppe, W. (1972). Phase evaluation in generalized diffraction (ptychography). In: Proceeding of the 5th European Congress on Electron Microscopy, pp. 628–629. Hodgkin, D.C., Kamper, J., Mackay, M., Pickworth, J., Trueblood, K.N., White, J.G. (1956). The structure of vitamin-B12 . Nature 178, 64–66. Hohenstein, M. (1992). Single side-band imaging in high-resolution electronmicroscopy. Appl. Phys. A 54, 485–492. Hoppe, W. (1969a). Diffraction in inhomogeneous primary wave fields. 1. Principle of phase determination from electron diffraction interference. Acta Crystallogr. A 25, 495–501. Hoppe, W. (1969b). Diffraction in inhomogeneous primary wave fields. 3. Amplitude and phase determination for nonperiodic objects. Acta Crystallogr. A 25, 508–515. Hoppe, W. (1982). Trace structure analysis, ptychography, phase tomography. Ultramicroscopy 10, 187–198. Hoppe, W., Strube, G. (1969). Diffraction in inhomogeneous primary wave fields. 2. Optical experiments for phase determination of lattice interferences. Acta Crystallogr. A 25, 502–507. Johnston, J.A. (1989). Wigner distribution and FM radar signal design. IEE Proc. F 136, 81–87. Kramers, H.A. (1927). La diffusion de la lumière par les atomes. Atti Cong. Internaz. Fisica 2, 545–557. Kronig, R. de L. (1926). On the theory of dispersion of X-rays. J. Opt. Soc. Amer. 12, 547–557. Landauer, M.N. (1996). Indirect modes of coherent imaging in highresolution transmission electron microscopy. PhD thesis, University of Cambridge. Lichte, H. (1986). Electron holography approaching atomic resolution. Ultramicroscopy 20, 293–304. Lin, J.A., Cowley, J.M. (1986). Reconstruction from in-line electron holograms by digital processing. Ultramicroscopy 19, 179–190. Marchesini, S. (2007). A unified evaluation of iterative projection algorithms for phase retrieval. Rev. Sci. Instrum. 78. Article No. 011301. McBride, W., O’Leary, N.L., Nugent, K.A., Allen, L.J. (2005). Astigmatic electron diffraction imaging: A novel mode for structure determination. Acta Crystallogr. A 61, 321–324. McCallum, B.C., Rodenburg, J.M. (1992). 2-Dimensional optical demonstration of Wigner phase retrieval microscopy in the STEM configuration. Ultramicroscopy 45, 371–380.


183

McCallum, B.C., Rodenburg, J.M. (1993a). Simultaneous reconstruction of object and aperture functions from multiple far-field intensity measurements. J. Opt. Soc. Am. A 93, 231–239. McCallum, B.C., Rodenburg, J.M. (1993b). An error analysis of crystalline ptychography in the STEM mode. Ultramicroscopy 52, 85–99. Miao, J., Charalambous, P., Kirz, J., Sayre, D. (1999). Extending the methodology of X-ray crystallography to allow imaging of micrometer-sized noncrystalline specimens. Nature 400, 342–344. Michelson, A.A. (1890). On the application of interference methods to astronomical measurements. Phil. Mag. 30, 1–21. Misell, D.L. (1973). A method for the solution of the phase problem in electron microscopy. J. Phys. D 6, L6–L9. Misell, D.L. (1978). The phase problem in electron microscopy. Adv. Opt. Electron Microsc. 7, 185–279. Mott, N.F., Massey, E.C. (1965). The Theory of Atomic Collisions, third ed. Oxford Univ. Press, Oxford. Nellist, P.D., Rodenburg, J.M. (1994). Beyond the conventional information limit: The relevant coherence function. Ultramicroscopy 54, 61–74. Nellist, P.D., Rodenburg, J.M. (1998). Electron ptychography I: Experimental demonstration beyond the conventional resolution limits. Acta Crystrallogr. A 54, 49. Nellist, P.D., McCallum, B.C., Rodenburg, J.M. (1995). Resolution beyond the “information limit” in transmission electron microscopy. Nature 374, 630. Patterson, A.L. (1934). A Fourier series method for the determination of the components of interatomic distances in crystals. Phys. Rev. 46, 372–376. Plamann, T., Rodenburg, J.M. (1994). Double resolution imaging with infinite depth of focus in single lens scanning microscopy. Optik 96, 31–36. Plamann, T., Rodenburg, J.M. (1998). Electron ptychography II: Theory of three-dimensional scattering effects. Acta. Crystallogr. A 54, 61–73. Rodenburg, J.M. (1988). Properties of electron microdiffraction patterns from amorphous materials. Ultramicroscopy 25, 329–343. Rodenburg, J.M. (1989). The phase problem, microdiffraction and wavelength-limited resolution – A discussion. Ultramicroscopy 27, 413– 422. Rodenburg, J.M. (2001). A simple model of holography and some enhanced resolution methods in electron microscopy. Ultramicroscopy 87, 105–121. Rodenburg, J.M., Bates, R.H.T. (1992). The theory of super-resolution electron microscopy via Wigner distribution deconvolution. Phil. Trans. A 339, 521–553. Rodenburg, J.M., Faulkner, H.M.L. (2004a). High resolution imaging. UK patent application PCT/GB2005/001464.

184

RODENBURG

Rodenburg, J.M., Faulkner, H.M.L. (2004b). A phase retrieval algorithm for shifting illumination. Appl. Phys. Lett. 85, 4795–4797. Rodenburg, J.M., McCallum, B.C., Nellist, P.D. (1993). Experimental tests on double resolution coherent imaging via STEM. Ultramicroscopy 48, 304– 314. Rodenburg, J.M., Hurst, A.C., Cullis, A.G. (2007a). Transmission microscopy without lenses for objects of unlimited size. Ultramicroscopy 107, 227– 231. Rodenburg, J.M., Hurst, A.C., Cullis, A.G., Dobson, B.R., Pfeiffer, F., Bunk, O., David, C., Jefimovs, K., Johnson, I. (2007b). Hard X-ray lensless imaging of extended objects. Phys. Rev. Lett. 93. Article No. 034801. Sayre, D. (1952). Some implications of a theorem due to Shannon. Acta. Crystallogr. 5, 843. Scherzer, O. (1936). Some defects of electron lenses. Z. Phys. 101, 593–603. Scherzer, O. (1947). Sphärische und chromatische Korrektur von ElektronenLinsen. Optik 2, 114–132. Scherzer, O. (1949). The theoretical resolution limit of the electron microscope. J. Appl. Phys. 20, 20–29. Spence, J.C.H. (1978). Practical phase determination of inner dynamical reflections in STEM. In: Scan. Electron Microsc. Pt. 1, pp. 61–68. Spence, J.C.H., Cowley, J.M. (1978). Lattice imaging in STEM. Optik 50, 129–142. Spence, J.C.H., Howells, M., Marks, L.D., Miao, J. (2001). Lensless imaging: A workshop on new approaches to the phase problem for non-periodic objects. Ultramicroscopy 90, 1–6. Watson, J.D., Crick, F.H.C. (1953). Molecular structure of nucleic acids – A structure for deoxyribose nucleic acid. Nature 171, 737–738. Whittaker, E.T., Watson, G.N. (1950). A Course of Modern Analysis. Cambridge Univ. Press, London and New York. Zuo, J.M., Vartanyants, I., Gao, M., Zhang, R., Nagahara, L.A. (2003). Atomic resolution imaging of a carbon nanotube from diffraction intensities. Science 300, 1419–1421.


Advances in Mathematical Morphology: Segmentation JEAN SERRA Laboratoire A2SI, ESIEE, University of Paris-est, BP 99 93162 Noisy-le-Grand cedex, France

I. Introduction . . . . . . . . . . . II. Criteria, Partitions, and Segmentation . . . . III. Connective Segmentation . . . . . . . A. Reminder on Set Connection . . . . . B. Connective Criteria and Segmentation Theorem IV. Examples of Connective Segmentations . . . A. Seedless Segmentations . . . . . . . 1. Smooth Connection . . . . . . . 2. Quasi-Flat Zones . . . . . . . . 3. A Feature-Space Based Segmentation . . B. Seed-Based Segmentations . . . . . . 1. The Seeds Theorem . . . . . . . 2. Single-Jump Connection . . . . . . V. Partial Connections and Mixed Segmentations . A. Two Levels of Segmentations . . . . . 1. Watershed Contours . . . . . . . VI. Iterated Jumps and Color Image Segmentation . VII. Connected Operators . . . . . . . . A. Examples . . . . . . . . . . . 1. Reconstruction Opening . . . . . . 2. Alernating Filter by Reconstruction . . . 3. Leveling . . . . . . . . . . . 4. Extension to Numerical Functions . . . VIII. Hierarchies and Connected Operators . . . . IX. Conclusion . . . . . . . . . . . References . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

185 188 191 191 193 196 196 196 197 197 199 199 201 204 204 205 206 209 211 211 211 211 211 212 215 215

I. I NTRODUCTION When Doctor Hawkes proposed that I write a chapter on advances in mathematical morphology, I thought it was a rather easy request. Since 1992, the date of the creation of the International Society for Mathematical Morphology (ISMM), eight congresses on the subject have taken place and consulting their proceedings suffices to grasp the recent history (Serra and Salembier, 1993; Serra and Soille, 1994; Maragos et al., 1996; Heijmans and Roedink, 1998; Goutsias et al., 2000; Talbot and Beare, 2002; Ronse et al., 2005; 185 ISSN 1076-5670 DOI: 10.1016/S1076-5670(07)00004-3


186

SERRA

Banon et al., 2007). It only remained for me to synthesize 300 papers and to reduce this documentation to a few trends, two or three good ideas, and new domains of application. Assuming that this could be done, should I present the synthesis by listing all major items one after the other and run the risk of describing a patchwork where the main threads vanish (but the actual thing is a patchwork)? Or, should I do the opposite and concentrate on what appears to me as the main theme over the past 15 years – the segmentation – and run the risk of ignoring some lateral branches of lesser interest? I chose the second approach for the following reason. When we look back at the past 15 years, a number of studies, algorithms, and concepts that previously seemed independent of each other when they were published organize themselves as the elements of a puzzle that makes sense only in retrospect. It appears that the morphologists turned around the notion of connective segmentation, closer and closer but by hesitating, as wild warriors who dance around their totem. The crystal seed for a theory of morphological segmentation appears in 1988 (Serra, 1988, Chapter 2) with the notion of a connection (Definition 4 below), but no one, including its author, noticed by this time that connection and segmentation were the two faces of a unique concept. During the 1990s, a series of developments on the so-called connected filters (Salembier and Serra, 1995), which simplify images without smoothing their contours (Crespo et al., 1995), arose in parallel with remarkable improvements of the watershed techniques (Meyer and Beucher, 1990). The discovery of the leveling filter (Meyer, 1998; Serra, 1999), which is indeed the most powerful connected filter with the opening by reconstruction, improved the segmentations based on partitioning. Both filters provide a precise control of the scale variations (Serra, 1999; Pesaresi and Benediktsson, 2001; BragaNeto and Goutsias, 2003) (see Section VIII). The convergence of these lines, already sketched in Garrido et al. (1998) and Meyer (2001) for video imagery, led in 2005 to the theory of connective segmentation (Serra, 2005, 2006), which in turn led back to new segmentation tools (e.g., both smooth and jump connections), and to new developments, based on attribute filters (Wilkinson, 2005) and on partial segmentation (Ronse and Serra, 2007). A second theoretical topic emerged progressively during the recent past, namely, the discrete approach (Bertrand, 1999; Najman et al., 2005). Of course, an implementation is always discrete in image processing, but the current trend, deeper, often rests on the refusal of any continuous background. The space is replace by a graph on which several discrete topologies are defined (Cousty et al., 2007a). A series of significant properties can be established in this framework (Borgefors et al., 2001). For example, it is proved that in finite graphs all the ways to view the notion of a watershed reduce to a unique one, a result inaccessible in the Euclidean context. On the

ADVANCES IN MATHEMATICAL MORPHOLOGY: SEGMENTATION

187

contrary, the notion of convexity, uniquely defined in vector spaces, blows up into a number of discrete approaches with different properties (Kiselman, 2004; Serra, 2007a; Melin, 2003). When dealing with stochastic models, the discrete point of view comes down to emphasize all types of simulations (Lantuejoul, 2002; Serra and Chermant, 1997; Angulo and Jeulin, 2007). The following text adopts a slightly different angle and avoid the conflict: all notions presented apply as well in continuous and discrete spaces. Finally, the third theme that can be derived from current morphology is that of differential formalism. A number of operators (dilation, opening, leveling, geodesy), can be interpreted as iterations of differential forms as soon as they involve convex structuring elements (Alvarez et al., 1992; van den Boomgaard and Smeulders, 1994; Maragos, 1996), because then self-dilation and magnification are equivalent (Matheron, 1975, Chapter 1). But a covariance, for example, cannot be decomposed in this manner. This differential approach leads to a better handling of digital interpolations (Osher and Sethian, 1988) and allows formalization of the segmentation in terms of optimizations of functionals (Morel and Solimi, 1995). It also proved to be well adapted to scale-space parameters (Maragos and Meyer, 1999). As for the applications, during the 1990s, they were oriented primarily toward video and multimedia analysis, such as compression and scene description, image indexing, motion modeling, and so forth, where in all cases a segmentation phase lies in the core of the work (Salembier and Marqués, 1999). An example below illustrates the point (Section V). In video-phonie, the foreground is usually a face, which deserves to be coded more finely than the background. In addition, this segmentation is needed to recognize the speaker, and finally, it tracks the silhouette through the time sequence of images (Gomila, 2001). Another feature of the applications is the increasing multiplicity of sensors: a same field of observation is increasingly investigated by several images in the sense of several numerical functions. They can come from time succession but may also be stacks of images in confocal, polarized, microscopy, or the three bands for the color information, or the n bands of the satellites images, or again complex data sets in geographical information systems, where the distributions of populations, diseases, fortunes, and so on, generate graphs in superimposition with the geographical maps. This last domain is currently expanding rapidly. Classically, multivariate analysis tends to assign similar roles to all variables in the process of data reduction, unlike mathematical morphology, which controls some variations by means of others. This technique typically is used in the example on color image segmentation, where saturation balances luminance against hue (Section VI). The presentation that follows is not exhaustive. The theory of connective segmentation serves here to reorganize the various aspects just outlined in a

188

SERRA

consistent perspective (the proofs of the propositions are given in Serra, 2006). Therefore, the order of the presentation is the inverse of the chronological one. In complement to the proceedings of the ISMM congresses, and to the review (Serra, 2007b) of 1997, readers may consult (Heijmans, 1994 and Soille, 1999). The first book, more mathematical, gives a good formalism for morphological operations, and the second, more recent, provides more algorithms and case studies. Two other books, associated with Georges Matheron’s jubilee (Jeulin, 1997), and memory (Bilodeau et al., 2005), also merit montion. The first describes the state of the art in random sets in 1996 and can be completed, for the simulations part, by Lantuejoul (2002). The second dates from 2005 and covers the entire scope of the field of mathematical morphology. Finally, readers may review the online publication of the “morphological digest” ([email protected]). In addition, a number of presentations, courses, and beta versions of papers are available in the websites of the various departments working in mathematical morphology.

II. C RITERIA , PARTITIONS , AND S EGMENTATION In image processing, the expression image segmentation means that, given a function f : E −→ T , the space E is partitioned into zones in which the function f is homogeneous, in some optimal sense. There are essentially two ways for considering this optimization problem. One of them associates numbers with the partitions and makes the optimization apply to these numbers. This first method, termed variational (Morel and Solimi, 1995), describes the image under study by a numerical function, often called energy, whose variation represents the amount of information left in each smooth version of the image. The most famous example of this approach was proposed in Mumford and Shah (1988). The second method acts directly on the set of all partitions and expresses optimization in this partition space. It appeared in image processing with the split-and-merge procedures (Pavlidis, 1977); its most popular representatives are the watershed operators (Beucher and Lantuejoul, 1979; Meyer and Beucher, 1990). Even though it is not explicitly formulated, one of the two methods usually underlies each particular algorithm, and sometimes both of them. This Chapter is devoted to the second approach only. It begins by defining and commenting the three notions of criterion, partition, and segmentation. (Note: the three letters iff mean if and only if.) Definition 1 (Criterion). Let E and T be two arbitrary sets and let F be a family of functions from E into T . A criterion σ on class F is a binary function from F ⊗ P (E) into {0, 1} such that, for each function f ∈ F , and


189

for each set A ∈ P (E), we have either σ [f, A] = 1 (criterion is said satisfied on A) or σ [f, A] = 0 (criterion is said refuted on A). Moreover, we decide conventionally that, for all functions, all criteria are satisfied on the empty set, that is, σ [f, ∅] = 1 ∀f ∈ F . For instance, the criterion that is satisfied if x ∈ A,

t f (x)

(1)

with t fixed, defines the threshold criterion applied to function f . Likewise, when space E is metric with distance d, and equipped with a connection C , then the connected zones A such that x, y ∈ A ⇒ f (x) − f (y) kd(x, y), yield the k-Lipschitz criterion. A zone A is said to be homogeneous according to f and σ when σ (f, A) = 1. Definition 2 (Partition). Let E be an arbitrary set. A partition D of E is a mapping x → D(x) from E into P (E) such that (i) for all x ∈ E: x ∈ D(x), (ii) for all x, y ∈ E: D(x) = D(y) or Da (x) ∩ Da (y) = ∅. D(x) is called the class of the partition at point x. A partition A is said to be finer (respectively coarser) than a partition B when each class of A is included in a class of B. This ordering relation leads to a complete lattice structure where the coarsest element has one class only, namely, set E itself; and the finest partition has all the points of E as classes. Given a criterion σ , when σ [f, D(x)] = 1 for all the classes D(x), x ∈ E of a partition D, then D is said to satisfy, or to fulfill criterion σ . The infimum and supremum of a family {Di , i ∈ I } of partitions are obtained via their classes Di (x). The mapping D of E into P (E) D(x) = Di (x), i ∈ I generates obviously a partition D, where for all x ∈ E, D(x) is the largest element of P (E) that is contained in each Di (x). Therefore, D is the infimum of the Di , in the sense of the partition lattice. In the case of Fig. 1(a) it is obtained by taking both continuous and dotted border lines. The expression

190

SERRA

(a)

(b)

F IGURE 1. (a) Infimum and supremum of two partitions. (b) Segmentation of R 1 by means of the flat zones of function f .

D = Di of the supremum is more complex; it means that for all x and all i, class Di (x) is the smallest set that is a union of classes Di (y), y ∈ E (see Fig. 1(a)). Ronse (1998) shows that x and y belong to the same class of the supremum if and only if there is a chain of classes from {Di (z), i ∈ I, z ∈ E} beginning at x, ending at y, and linking them. Fig. 1(b) depicts a basic example of segmentation. The image under study is represented by function f : E → R, where E is a set equipped with a connection. Function f is segmented into flat and connected zones when for any x ∈ E, the class D(x) is the largest connected component of E, including point x and on which function f is constant and equal to f (x). When there is no flat neighborhood around point x, then the segmentation class reduces to x itself. All criteria do not lend themselves to such maximum partitioning. Here are two examples of lack of supremum. Suppose, first, that we wish to partition space E into various zones, connected or not, where function f is Lipschitz with parameter k = 1. Two nondisjoint zones A and B may very well be found, such that the criterion is satisfied on A and on B, but not on A ∪ B (see Fig. 2(a)). In this case, there is no largest zone containing the points of A and where the criterion be satisfied. The Lipschitz criterion does not yield a segmentation. The second counterexample is the concern of criterion σ [f, A] = 1 ⇔ sup f (x), x ∈ A − inf f (x), x ∈ A k, (2) with k > 0. Fig. 2(b) shows that a function cannot be segmented by using it (except for k = 0, where the flat zones criterion again applies). In other words, the partitions referred to with a segmentation concept are coarsest ones (i.e., those having the largest classes). Besides, we can always construct a smaller partition, namely the one that reduces space P (E) on all


(a)

191

(b)

F IGURE 2. (a) This function is Lipschitz inside sets A and B, but not inside their union. (b) The variation of function f is smaller than k on the two segments C and D but not on their union.

its singletons {x}. These remarks suggest replacing the initial definition of image segmentation by the more precise following one. Definition 3 (Segmentation). Given a function f ∈ F , an arbitrary set A ⊆ E and a criterion σ , let {Di (A), i ∈ I } be the family of all partitions of set A into homogeneous zones of f according to σ . The criterion σ is said to segment the function f when (1) σ [f, {x}] = 1 for all {x} ∈ P (E), (2) family {Di (A)} is closed under supremum. Then the supremum partition Di (A) defines the segmentation of f over set A according to criterion σ . In particular, when set A coincides with the space E of definition of f , then Di (E) is the maximum partition of the whole space. In the above k-Lipschitz criterion it is precisely this maximum partition that is missing. Axiom σ [f, {x}] = 1 ensures that whatever criterion is considered, there is always at least one way to partition E into zones (the singletons) that satisfy it. This is a compulsory condition for dealing with a largest partition. Note that the above segmentation does not hold on E only but on all subsets of E.

III. C ONNECTIVE S EGMENTATION A. Reminder on Set Connection In mathematics, the concept of arcwise connectivity is formalized in the framework of topological spaces. According to this notion, a set A is con-

192

SERRA

nected when, for every a, b ∈ A, there exists a continuous mapping ψ from [0, 1] into A such that ψ(0) = a and ψ(1) = b. A basic result governs the meaning of arcwise connectivity, according to which the union of connected sets whose intersection is not empty is still connected: {Ai connected} and Ai = ∅ ⇒ Ai connected . (3) Here, we adopt the nontopological approach to connectivity developed in Serra (1988), where Eq. (3) serves as a starting point and not as a consequence, and the word connection is introduced to mark the difference form the classical notion of connectivity. Definition 4 (Connection). Let E be an arbitrary non-empty space. We call connected class or connection C any family in P (E) such that (0) ∅ ∈ C , (i) for all x ∈ E, {x} ∈ C , (ii) for each family {Ci , i ∈ I } in C , Ci = ∅ implies

Ci ∈ C .

Any set C of a connected class C is said to be connected, and the empty set, as well as the singletons {x}, x ∈ E, are always connected. Definition 4 covers the various arcwise connectivities for continuous and digital spaces and for graphs. However, numbers of other connections can be defined on set E (Serra, 2000; Ronse, 1998; Heijmans, 1999). Two of them are depicted in Fig. 3, namely, • Given a partition D of the space, all the subsets of each class D(x), x ∈ E, of the partition generate a family closed under union (Serra, 1988, Chapter 2). Hence, we have the connection C = A ∩ D(x), x ∈ E, A ∈ P (E) . The connected component at point x equals the intersection A ∩ D(x) between A and the class of the partition in x. (This technique will be constantly used in the next three sections.) • Given an extensive dilation and a first connection, the clusters of dilated particles generate a second-level connection. When a point x ∈ A is fixed, axiom (ii) of Definition 4 orients us toward the (connected) union γx (A) of all connected components containing x and included in A γx (A) = {C; C ∈ C , x ∈ C ⊆ A}. (4)


193

F IGURE 3. Left: connection generated by a partition (lines). The connected component of set A at point x is the union of the two disjoint blobs of A ∩ Dx . Right: the particles of each cluster belong to a second-level connection.

The mapping γx : A −→ γx (A) from P (E) into itself is obviously increasing (A ⊆ B ⇒ γx (A) ⊆ γx (B)), anti-extensive (γx (A) ⊆ A), and idempotent (γx [γx (A)] = γx (A)). In algebra, such an operator is called an opening. The link between connection and opening is given by the following theorem (Serra, 1988, Chapter 2): Theorem 1. The datum of a connected class C on P (E) is equivalent to that of a family {γx , x ∈ E} of openings such that (iii) for all x ∈ E, we have γx (x) = {x}, (iv) for all A ⊆ E, x, y ∈ E, γx (A) and γy (A) are equal or disjoint, (v) for all A ⊆ E, and all x ∈ E, we have x ∈ / A ⇒ γx (A) = ∅. The γx ’s are called the point-connected openings of connection C . Historically speaking, the number of applications or of theoretical developments suggested (and permitted) by Theorem 1 during the 1990s is considerable. Salembier and Serra (1995), Vincent (1993), Crespo et al. (1995), among many others, show that it has opened the way to an objectoriented approach to segmentation, compression, and understanding of still and moving images. B. Connective Criteria and Segmentation Theorem Which conditions a criterion σ must satisfy to be a segmentation tool? The need for largest partitions orients us toward a connection-based approach via the concept of a connective criterion.

194

SERRA

Definition 5 (Connective criterion). A criterion σ : F ⊗ P (E) → {0, 1} is connective when for each f ∈ F the sets A such that σ (f, A) = 1 form a connection; that is, when (1) σ is satisfied on the class S of the singletons and by the empty set,

∀f ∈ F , {x} ∈ S ,⇒ σ f, {x} = 1, (2) and when for any function f ∈ F and for all families {Ai } of P (E) for which the criterion σ is satisfied, we have " # ! σ (f, Ai ) = 1 ⇒ σ f, Ai = 1. (5) Ai = ∅ and In other words, when a connective criterion σ is satisfied by a function f on a family {Ai } of regions of the space, and if all these regions have one common point, then it is also satisfied on the union Ai . The following key theorem characterizes the connective segmentation by identifying it with classes of connections (Serra, 2006). Theorem 2. Consider two arbitrary sets E and T , and a family F of functions f : E → T and let σ be a criterion on class F . Then the three following statements are equivalent: (1) Criterion σ is connective. (2) For each function f ∈ F , those sets on which criterion σ is satisfied generate a connection C (6) C = A | A ∈ P (E) and σ [f, A] = 1 . (3) Criterion σ segments all functions of the family F . The resulting segmentation is said to be connective. In particular, the segmentation of function f over the whole set E, and according to the connective criterion σ , is the partition Df whose class at point x is given by A: A ∈ P (E), x ∈ A, σ (f, A) = 1 x ∈ E. Df (x) = (7) Df (x) is obtained as the union of all sets containing x and where criterion σ is satisfied. Theorem 2 expresses that the two notions of segmentation and connection, which could seem far from each other, are indeed a unique concept. This theorem could not be stated by means of the classical arcwise connectivity only, segmentation criteria that do not involve paths are easily found. As a matter of fact, the more general concept of a connection is exactly right for the theorem to work. However, this theorem becomes false if we


195

say that “C is the set of all classes of a family of partitions closed under the supremum,” because then all possible partitions with classes in C are no longer taken into account. That incorrect reading may cause errors. A brief comparison between variational and connective methods, for segmentation, is instructive. 1. Unlike variational approaches, the connective one does not involve differential operators; both spaces E and T are totally arbitrary and may be continuous, discrete, irregular graphs, and so on. Therefore Theorem 2 opens the way to all applications where various heterogeneous variables are defined on E. Such a circumstance arises in geography, for example, where radiometric data (satellite images) live together with physical ones (altitude, slope of the ground, sunshine, distance to the sea) and with statistical data (demography, fortunes, diseases). 2. In variational methods, the segmentation holds on the whole starting space E, and only on it. In the connective case, this is not the case, as Theorem 2 provides any subspace A ⊆ E with a largest partition. Here is a consequence of this property. In the traveling of a TV camera, a same object S to segment often appears inside successive masks, Y and Z, for example. If the segmentation is performed in each mask separately, it must result in the same contour for S as long as this object lies in the masks. Take a pointer x inside S, and let Dx (Y ) and Dx (Z) be the two associated segmentation classes for some connective criterion. If both Dx (Y ) and Dx (Z) belong to the two masks Y and Z, then the two segmentations Dx (Y ) and Dx (Z) of S are identical (see Fig. 4(b)), a result inaccessible by variational methods.

F IGURE 4. (a) A bicolored function has been thresholded. The ellipse (respectively crescent) is the connected component of the red (respectively green) threshold at point x. (b) Permancency of a segmented region as the mask is moved.

196

SERRA

3. The connective criteria form a complete lattice, which allows accounting for the criteria in a parallel way. This lattice is not isomorphic to that of the partitions. Suppose that a color image, in red-green-blue (RGB) representation, is used. A first criterion extracts the arcwise-connected components of the plane where the red band is above threshold r0 (plus the singletons). This gives the ellipse of Fig. 4(a). A second criterion acts similarly for the green band, which results in the crescent. The infimum of the two criteria consists in all arcwise-connected components where both red and green bands are above their respective thresholds; the class D ∗ (x) of this infimum at point x is the set B in light gray, whereas the union A∪B gives the class at point x of the infimum of the red and green partitions. In conclusion, we can say that Theorem 2 belongs to the category of gobetweens from an abstract definition, hard to implement (must we calculate all partitions for checking whether there is a larger one?) and a more “local” notion that better lends itself to easy validations. However, it does not provide by itself any “recipe” for deriving nice connective criteria.

IV. E XAMPLES OF C ONNECTIVE S EGMENTATIONS We now illustrate the theory with a few examples. For the sake of simplicity, we omit repeating the axioms of connective criteria for the singletons and the empty set. Also, the starting space E is supposed to be provided with an initial connection C0 , which may intervene (or not) in the definition of the connective criterion under study, and the arrival space T is R, Z, or one of their closed subsets. The examples are classified in two categories accordingly as the criteria involve or not some seeds. A. Seedless Segmentations 1. Smooth Connection In the counterexample of Fig. 2(a), the distance used to reject the Lipschitz function was based on an overall metric on space E. What happens if restrict the distance values are restricted or if the overall metric is replaced by a new one whose geodesics should lie inside the sets A and B of Fig. 2? The criterion σ of the so-called smoothed connection, solves these problems by putting σ [f, A] = 1 iff each point x of A is the center of an open ball inside which f is k-Lipschitz. Criterion σ turns out to be connective and generates segmented zones A that are both smooth and C0 connected. When C0 is an arcwise connection, then function f is k-continuous along all paths included ◦

in the interior A of A.


(a)

(b)

197

(c)

F IGURE 5. (a) Electron micrograph of concrete. (b) and (c): Segmentation of image (a) by smooth connections of parameters 6 and 7, respectively.

For the 2D digital space Z2 in particular, the smallest value of the range a is 1 (unit square or hexagon). It suffices, to implement it, to erode functions f and −f by the cone H (k, 1), whose base is the unit square or hexagon and whose height is k, the origin being placed at the top, and then to take the intersection of the two sets where f equals its erosion and its dilation, respectively. The smooth connection is well adapted to the separation of smooth zones from more granular ones with similar grays, as often appears in electron microscopy. Figure 5 depicts two segmentations of a concrete micrograph, performed by using smooth connections. 2. Quasi-Flat Zones Instead of requiring that f be ω-continuous along all paths included in A, we can also only require the ω-continuity for at least one path. This more comprehensive new criterion is still connective and leads to the connection according to the quasi-flat zones (Meyer, 1998). This time, the digital implementation involves geodesic reconstructions. 3. A Feature-Space Based Segmentation Here is an example of segmentation for an already connected set, hence where a connective criterion can only hold on some feature space (Serra, 2002). The material under study is a stack of 100 thin sections of the shinbone of a chicken embryo. The biologists want to know whether the growth of the bone proceeds by nested cylinders. Figure 6 shows two bone sections. They are perpendicular to the long axis of the shinbone and exhibit sorts of circular “crowns.” If the assumption of the biologists is true, the successive cylinders

198

SERRA

(a) F IGURE 6.

(b) Two cross sections of the shinbone orthogonally to their axes.

should be linked by a few 3D bridges to ensure the robustness of the whole bone structure. How can the truth of their hypothesis be tested? And if need be, how can the pile of cross sections be segmented into the various nested cylindric crowns? The proposed procedure is based on the concept of using the central hole of the bone (the medullary region) as a cylindric marker, and, starting from it, to invade the solid bone by 3D geodesic dilations. Figure 7(a) shows geodesic dilation in two dimensions. However the current case is 3D, and for its implementation we need to choose a unit sphere; we will take the unit cube-octahedron preferably to the unit cube. It is more isotropic (14 face directions instead of 6), more compact (12 voxels versus 26) and closer to an Euclidean sphere (the 12 vertices are at the same distance from the center). The geodesic dilations create a 3D wavefront that propagates through the bone. The surface area of this wavefront is measured at each step of its discrete propagation. Figure 7(b) shows the plot of this surface area measurement versus the step number. Several minima are clearly visible. Indeed, starting from the medullary cylinder the bone the wave invades the solid zones: • It meets a bone cylinder first, where the front spreads out largely. • Then the front must cross some narrow bridges to reach the second cylindric crown, hence, its surface area reduces. • The front then again propagates over the second cylindric crown, and its surface area extends. The plot of Fig. 7(b), with its well-pronounced minima, corroborates the biological assumption of nested cylindric crowns, and the abscissae of the


(a)

199

(b)

F IGURE 7. (a) Two-dimensional geodesic wavefronts from point z inside mask X. (b) Surface area of the 3D wavefront at each step of the geodesic propagation.

successive minima indicate the geodesic distance from the axial marker to the bottlenecks of the bridges. If the 3D bone is split into the zones that the minima separate, do we perform a segmentation? Yes, indeed. The set can be forgotten for one instant and only the plot considered. The following criterion apply: $ σ [f, {x}] = 1, σ [f, A] = 1 when segment A ∈ P (Z1 ) contains no minimum of f, σ [f, A] = 0 when not where f is a 1 − D numerical digital function. This obviously connective criterion divides the x-axis into segments with no minimum and assigns point classes to all abscissae of minima. Projecting back, this division onto the 3D bone set yields the segmentation of the shinbone into its nested cylinders (Fig. 8). The technique in this example consists of replacing a 3D set (which is hardly tractable), by a 1D significant function (the wavefront plot) and then defining a convenient connective criterion in this feature space. B. Seed-Based Segmentations 1. The Seeds Theorem Many segmentation processes work by aggregating the points of the space around a family G0 of seeds located in space E. The clustering process

200

SERRA

(a) F IGURE 8.

(b) Segmentation of the two above sections.

may comprise (or not) several iterations; the seeds locations may be fixed or depend on iterations; the seeds themselves may be endogenous (e.g., the minima of f ) or carry external information. In all these variants the clustering algorithm results in the assignment of each point x ∈ E to one, or more, final seed. Then every such process generates a connective criterion. We have that Theorem 3. Given a function f ∈ F , an initial seed family G0 , and a clustering process that leads to a final space distribution G of the seeds, the criterion σ [f, A] given by $ 1 when all points of A are assigned to a unique σ [f, A] = final seed g(A) ∈ G, 0 when not, is connective. In the final maximum partition, the regions whose points are aggregated to more than one seed reduce to singletons (readers may note that the criterion “the points of A are assigned to one or two seeds” is not connective, and that the criterion “the points of A are assigned to one or more seed” comes back to take set E as the unique class). As an example, we can use the popular Chassery–Garbay algorithm for color image segmentation (Chassery and Garbay, 1984) to build regions from a series of seeds. Each pixel is compared to the seeds according to an aggregation criterion that involves color distances plus other possible attributes, such as texture or convexity constraints. These comparisons result first in segmentation of the space. Then a new seed is redefined in each


201

class according to various motivations, and the segmentation procedure is iterated, and so on for a finite number of times. By applying Theorem 3 to the last iteration, the whole procedure is connective. The same comment applies to a number of classification algorithms for clouds of points that work in multidimensional spaces. 2. Single-Jump Connection Human vision easily accepts small variations when they originate from the minima or the maxima of the function under study. The jump criteria now presented work this way according to a few variants (Serra, 2000). They generalize notions such as thresholds or level sets. Consider a function f : E → T that has the family {mj } of minima, of supports {Mj } in E, and take for C0 an arcwise connection. Given a path p = {x1 , x2 , . . . , xn }, one classically calls variation of f along path p the quantity % ! v(p) = f (xi ) − f (xi ), 1 i n . Similarly, we say that two points x, y ∈ E are k-close to each other when the infimum of the variation for all possible paths from x to y equals k. Unlike that of the smooth connection, the parameter k, here, applies to possibly noncontinuous functions, as it may correspond to a certain amount of jumps of f . The single-jump criterion σ is defined by the following requirement: σ [f, A] = 1 if A is connected, and if each point x ∈ A is k-close of a point y belonging to the support M ⊆ A of a minimum of f that lies inside A. The positive number k is called the jump height of σ . Criterion σ is obviously connective; when it is satisfied by all the Ai of a family {Ai , i ∈ I } and if x ∈ i Ai , then i Ai is connected because each Ai ∈ C0 ; in addition at each point z ∈ i Ai the value f (z) is less than k above a minimum. A 1D example is shown in Fig. 9 where the segmented regions may not be flat and may comprise several minimum zones M (unlike the watersheds presented below). The one-jump criterion σ is illustrated in the alumina micrograph f of Fig. 10(a), for a jump k = 12. Conventionally in Fig. 11(a), (b), and (c) the white particles are the nonsingleton classes of the partitions, and the sets in black regroup all singletons classes. In Fig. 11(a) the bottoms of the valleys appear as white stripes (flat bottoms) inside black tubes (sharp edges). As the single-jump criterion σ is not self-dual, unlike the smooth connection, it may be instructive to look to the dual form. The new criterion, σ say, derives from σ by replacing “minimum” by “maximum” in the definition of σ . The bottoms of the valleys now become dark but less continuous (Fig. 11(b)).

202

SERRA

F IGURE 9.

(a)

Example of segmentation by jump connection.

(b)

(c)

F IGURE 10. (a) Initial alumina micrograph; (b) superimposition of (a) and (b) of the Skiz of set Fig. 11(c); (c) same procedure for a jump k = 8.

These valleys, due to the grains boundaries, can be better extracted if both criteria are combined into a symmetrical version, by taking the infimum, and still connective, criterion σ ∧ σ . Owing to the display convention of Fig. 11, it suffices to take the intersection of the two sets Fig. 11(a) and (b). The result is shown in Fig. 11(c). This intersection may seem rough, but its skeleton by influence zones yields the rather satisfactory boundary extraction shown in Fig. 10(b). Finally, Fig. 10(c) shows the same procedure when the jump height k equals 8.


(a)

(b)

203

(c)

F IGURE 11. Jump connections generated by the alumina micrograph of Fig. 10(a), for a jump k = 12. (a) From below, (b) from above, (c) infimum of below and above. In black, the set of all singletons.

(a)

(b)

(c)

(d)

F IGURE 12. (a) Electron micrograph of concrete; (b) segmentation of image (a) by jump connection of range 12; (c) segmentation of image (a) by smooth connection of parameter 7; (d) intersection of connections (b) and (c).

This section concludes with a comparison of the smooth and jump segmentations on the concrete electron micrograph of Fig. 12. Both are noisy, but as they extract very different pieces of information, their infimum, which is still a connective segmentation, results in the rather nice Fig. 12(d), where the noise is practically removed.

204

SERRA

V. PARTIAL C ONNECTIONS

AND

M IXED S EGMENTATIONS

Union of criteria can be approached in another manner if priorities are introduced between the various criteria. Then optimal multipartitions of the space are obtained by means of partial connections (Serra, 2005; Ronse and Serra, 2007). Definition 6 (Partial connection). Let E be an arbitrary space. Any class C1 ⊆ P (E) such that (i) ∅ ∈ C1 , (ii) for each family {Ci , i ∈ I } in C1 , Ci = ∅, implies a partial connection on E.

Ci ∈ C1 , defines

Unlike a connection, a partial connection does not necessarily contain the set S of all singletons of P (E). What are the consequences of this missing axiom? A (quasi-)connected opening γ1,x can still associate with each point x ∈ E by putting for any A ⊆ E ∅ when the family {C1 ∈ C1 , x ∈ C1 ⊆ A} is empty, (8) γ1,x (A) = {C1 ∈ C1 , x ∈ C1 ⊆ A} when not. The only change with the connection case is that now γ1,x (A) may equal ∅, even when x ∈ A. As a consequence, the supremum γ1 = {γ1,x , x ∈ E} generates an opening on P (E) whose residual ρ1 (A) = A \ γ1 (A), for a set A ⊆ E is not necessarily empty. A. Two Levels of Segmentations Let C1 be a partial connection on P (E) of point openings {γ1,x , x ∈ E}, and C2 be a connection on P (E) of point-connected openings {γ2,x , x ∈ E}. A hierarchy is introduced between them by restricting the classes according to C2 to the zones that are not reached by C1 . This can be done via the operator γ2,x [ρ1 (A)] when x ∈ ρ1 (A), χ2,x (A) = (9) ∅ when not. Operator χ2,x is not increasing, as it acts on set A ⊆ E via the residual ρ1 (A) of γ1 (A). Nevertheless, by combining it with the partial opening γ1,x , an optimum partition is reached in the following sense. Proposition 1. Let C1 be a partial connection and C2 be a connection, both on P (E), where set E is arbitrary. Then,


(a)

(b)

(c)

(d)

205

F IGURE 13. (a) Initial view of the silhouette to segment; (b) segmentation of the face by a color-based criterion; (c) marker of the chest; (d) resulting segmented silhouette.

(1) The union of the two families {γ1,x } and {χ2,x }, x ∈ E, of operators partition every set A ⊆ E into two families of classes {A1,i } and {A2,j }, (2) this partition is the greatest one with classes of C1 on γ1 (A) and classes of C2 on A \ γ1 (A). In Fig. 13, drawn from Gomila (2001), one seeks to segment the silhouette of the person in the foreground (Fig. 13(a)). A first partial segmentation, based on the colors of the hair and of the skin, leads to the white set of Fig. 13(b). Then a second segmentation, based on shape, extracts the shoulders by means of the three-rectangle marker in Fig. 13(c) (minus the already segmented pixels). This marker holds uniquely on the complement of set Fig. 13(b). The union of the two segmentations results in the silhouette shown in Fig. 13(d). 1. Watershed Contours When the numerical function f is interpreted as a relief in E, then all points of the relief whose path of swiftest slope meet a same minimum M ⊆ E define the watershed of M (Meyer and Beucher, 1990; Bertrand, 2005; Najman et al., 2005). Alternatively, the watershed of M can be introduced as the set of all points of E flooded from M. Regardless of the exact definition, Theorem 3 shows that the associated criterion is connective. The watersheds partition the defined space E into arcwise-connected catchment basins, plus into all the points of the watershed contours, considered as singletons in the segmentation

206

SERRA

partition. Moreover, the points of the flat zones turn out also to be singletons in the “pure” flooding algorithm, as they have a zero gradient. Therefore, an additional function is introduced by taking the distance to the downstream edge. This second function results in a second watershed segmentation, which can be combined to the first one due to Proposition 1, and this double process can be iterated. Indeed, the watershed algorithms always involve this mixed segmentation, even when it is not explicit. On the other hand, as a gradient module is often noisy, the derived watershed partitions are oversegmented. The drawback is avoided by replacing the minima gradient g by the given seeds. Theorem 3 ensures that the obtained partition is still optimal. As the oldest valuable segmentation technique in mathematical morphology (it dates from 1979: Beucher and Lantuejoul, 1979), watersheds are the subject of an abundant literature. This Chapter highlights just three of its features. First, the contours extracted by watershed are never a priori smooth; this feature endows watershed with the advantage of working even when only two or three gray levels are defined (Beucher, 1990). Note that the actual word also is not smooth: dendrites, trees exhibit a real roughness. Other operators, such as the snakes, comprise a smoothness action intrinsic to their own formalism. One can appreciate (or not) the cosmetic. Second, the role of the seeds, or markers, primordial in watersheding, make the procedure well adapted to time sequences; the segmentation of image n provides, after a small cleaning, the markers for image n + 1. Finally, as watersheds lend themselves to various decomposition (by gray levels, by queuing files, and so on), they extend to four and five dimensions in a correct way. This is particularly true for topological watersheds (Bertrand, 2005; Najman et al., 2005). The best example I know of this generalization concerns the automatic and accurate detection of the heart muscle from 3D radiographic sequences (Cousty et al., 2007b).

VI. I TERATED J UMPS AND C OLOR I MAGE S EGMENTATION Proposition 1 extends to n phases by replacing C2 by a partial connection and by adding a third connection C3 , and so on. In particular, when all partial connections Ci (1 i < ∞) are identical, then the method yields a series of similar mixed segmentations. For example, the segmentation by unique jumps seen above generates, according the value of k, a large number of zones of singletons. Then a second jump segmentation may be performed in these singletons zones (as depicted in Fig. 9), and so on. This iterated jump segmentation has shown its application to segment textures (e.g., chromatin of lymphocytes, Angulo, 2003). Here, we illustrate it on an example of color image segmentation drawn from Angulo and Serra (2007).


207

An abundant literature exists on the subject, for which a detailed state of the art can be found in Cheng et al. (2001). In addition to the usual difficulties of segmentation, color imagery sets two specific problems: 1. Which representation must we chose: red-green-blue, hue-luminancesaturation, or something else? 2. Sooner or later, the processing of multidimensional data goes through a scalar reduction, which in turns yields the final partition. How should the reduction be performed, and when? Polar representation with one variable for light intensity and two variables defined on the chromatic circle may allow better handling of the color parameters for quantitative purposes (Carron and Lambert, 1994; Gevers and Smeulders, 1999). Indeed, the changes of representation in TV receivers is revealing. The input Hertzian signal is coded as one gray image plus two chrominances associated with green-red and blue-yellow contrasts. The output signal requires the three bands of green, red, and blue for the photoreceivers of the monitor. However, the manufacturers take none of these representations for the user’s interface and prefer human adjustments based on light (luminance), contrast (saturation), and, in case of computers, hue. Hence, this last triplet is the most adapted to human vision. Unfortunately, the usual polar representations, such as hue, luminance, saturation (HLS) or hue, saturatic and value (HSV), which serve as basic standards in the computers, are deeply inadequate for quantitative image processing. In the HLS system, the mean of two luminances is sometimes higher than both input values (!), the saturation of the mean of two complementary colors may increase, which is in total contradiction with Newton’s disk theory (Newton, 1671) (see Angulo and Serra, 2007 for a few such other instances). That is why a complete renewal of the polar color representation, based on distances and norms, was introduced in Hanbury and Serra (2003). The representation used here is based on norm L1 (the new formalism that derives HLS from red, green, and blue is given in Angulo and Serra, 2007). Concerning the second point – data reduction – the most popular procedures consist of replacing, from the beginning, the bunch of images by the sole gradient module. This scalar image is then the unique piece of information used to find the segmentation contours. In fact, as shown in Angulo and Serra (2007), such a reduction arrives too early; and the relevant information is better preserved by the following: 1. Segmenting separately the luminance, the saturation, and the hue, and 2. Combining the obtained partitions of the luminance and of the hue by means of that of the saturation: the latter is taken as a criterion for choosing at each place either the luminance class or the hue one.

208

SERRA

The three bands of the “parrots” image, in L1 norm representation, are depicted in Fig. 14(a)–(c). Each band is segmented by iterated-jump connection. As the jump parameter k increases, the oversegmentations reduce, but heterogeneous regions appear in compensation. A satisfactory balance seems to be reached for k = 20 (for 8-bits images). However, some small areas to be removed still remain. One filters out the classes of surface area smaller than 50, and reconstitutes the partition using the skeleton by influence zones. This algorithm is identically applied to the luminance and to the circular hue by taking the origin in the red, which results in the two segmentations depicted in Fig. 15. How to merge the two partitions of Fig. 15(a) and 15(b) into a unique one? The solution is to split the saturation image into two sets, Xs and Xsc , of high and low saturations, respectively, and assign the hue partition to the first set and the luminance to the second. A class of the synthetic partition is either the intersection of a luminance class with the low-saturation zone Xsc or the intersection of a hue class with the high-saturation zone Xs . If the classes of the luminance, the hue, and the synthetic partition at point x are denoted by

(a) F IGURE 14. (c) hue.

(b)

Representation of the “parrots” image in L1 norm. (a) Luminance, (b) saturation,

(a) F IGURE 15. false color.

(c)

(b)

Gray segmentations of the luminance (a) and the hue (b). Both are depicted in


(a)

(b)

209

(c)

F IGURE 16. (a) Segmentation of the saturation (presented in gray tones); (b) optimal threshold of (a); (c) final synthetic partition, superimposed to the initial image.

Al (x), Ah (x), and A(x), respectively, we have A(x) = Al (x) ∩ Xsc

when x ∈ Xsc ,

A(x) = Ah (x) ∩ Xs

when x ∈ Xs .

The simplest way to generate the set Xs consists of thresholding the saturation image, but this risks a resulting irregular set Xs , with holes, small particles, and so on. It is preferable to start from the mosaic image of the saturation provided by the same segmentation algorithm as for the the hue and the luminance (Fig. 16(a)). An optimal threshold on the saturation histogram determines the value for the achromatic/chromatic separation (Fig. 16(b)). The final composite partition is depicted in Fig. 16(c), which is excellent. Is a segmentation technique that depends on two parameters (jump height k = 20 and surface area = 50) a general procedure or an ad hoc trick? The above algorithm, without changes, and with the same parameters was tested on a selection of images from a reference database Berkeley Segmentation Dataset and Benchmark (BSDB) in comparison with manual segmentations and also by watersheds. The results, which can be seen in Angulo and Serra (2007), are good.

VII. C ONNECTED O PERATORS We have just seen how partial connections segmentation in a sequential manner. Should it be also possible to insert some filtering between the initial image and the final partition? Such a combination will make sense only if the filtering step fits correctly with criterion σ and prepares its segmentation by regrouping, hence simplifying the σ classes. Consider an arbitrary operator ψ on F and a criterion σ . The pair {σ, ψ}, viewed as a whole, turns out to be a criterion based on the quantity σ [ψ(f ), A], since this quantity equals either 0 or 1. Moreover, criterion

210

SERRA

{σ, ψ} inherits some basic properties that σ may satisfy. For example, if criterion σ is connective, then criterion {σ, ψ} is connective as well. To prove it, we observe that the connectiveness axiom (Definition 5) applies for all functions f ∈ F . Since ψ(F ) ⊆ F , it suffices to replace f by ψ(f ) in Definition 5. The new property to be added now to criterion {σ, ψ} no longer holds on σ but on operator ψ. In image processing, one the adjective connected is usually used to signify that an operation ψ on T E enlarges the segmentation partition (Salembier and Serra, 1995; Crespo et al., 1995). We maintain this term but redefine it within the background of connective criteria. Definition 7. An operator ψ on T E is said to be connected with respect to a connective criterion σ when for all f ∈ T E and for all A ⊆ E it satisfies the relationship σ [f, A] = 1

⇒

σ [ψ(f ), A] = 1.

(10)

In other words, the connection generated by σ and ψ(f ) is larger than that due to σ and f . In particular, when a connection C is defined over P (E) and when σ (f, A) = 1 for A connected and f constant, then Proposition 1 reduces to the classical definition of the connected operators by flat zones (i.e., f constant over the connected set A implies ψ(f ) constant over A (Salembier and Serra, 1995; Crespo et al., 1995). Consider now the segmentation class Df (x) at point x, that is, Df (x) =

A: x ∈ A, σ [f, A] = 1 .

Since σ is connective, Proposition 1 shows that at every point x, σ [ψ(f ), Df (x)] = 1, hence Df (x) ⊆ Dψ(f ) (x) so that Df Dψ(f ) (i.e., the operator ψ enlarges the segmentation partitions). Since such an enlargement can only be reached by merging classes, the overall boundary set of the classes reduces. The most popular connected operators are also morphological filters, i.e., are increasing and idempotent. The fact that they increase allows them to be defined for sets, and to straightforwardly extend them to functions. Their idempotence means that they go to the end of their potentialities and do not act more under iteration (when we speak of a “yellow filter” in optics or of a “low pass-filter” in electronics, we intuitively formulate the same idea).


211

A. Examples Here are three examples of set connected filters on P (E), or equivalently on the binary functions F : E → {0, 1}. The set E is provided with an initial connection C0 . 1. Reconstruction Opening The opening γx of the flat zone criterion, when applied to set A, is nothing but the C0 -connected component, or grain, of A that contains point x. It is a connected opening, just as more generally the supremum γM = {γx | x ∈ M}, where γM (A) is the union of all grains of A that hit M. The operation γM is called the reconstruction opening of marker M (sometimes written as γ rec (A, M)). 2. Alernating Filter by Reconstruction By duality, the reconstruction opening induces a connected closing, also by reconstruction. It consists of suppressing all pores that hit marker M and is denoted by ϕM . Then the composition products ϕM γN and γN ϕM , where markers M and N are independent, turn out to be connected filters. 3. Leveling The marker M is said to touch A if it hits A or if one of the grains of M is adjacent or one of those of A (i.e., these two grains are disjoint but their union is connected). The union γ M (A) of all grains of A touching M still defines a connected opening, γ M for example. It admits for dual operator the closing ϕ M . When applied to set A, ϕ M (A) provides the complement of the the union of all pores of Ac that touch M c . The product of the opening γ M by the closing ϕ M c (and not ϕ M ) defines the levelling νM of marker M νM = γ M ϕ M c = ϕ M c γ M = γ M ∪ (co ∩ ϕ M c )

(11)

where co stands for the complement operator A → Ac . The leveling νM is a connected filter whose two factors commute. Therefore, if M = μ(A) is itself a function of A for a self-dual operator μ, then the leveling νM becomes in turn self-dual for the complement. 4. Extension to Numerical Functions If a numerical function is interpreted as a stack of decreasing cross sections, then a new stack of decreasing sets can be generated by transforming each cross section according to one of the above filters. Consequently, this filter transforms a function into another one (of course, the implementations do not

212

SERRA

F IGURE 17.

(a)

Opening by reconstruction of function f according to marker g.

(b)

(c)

F IGURE 18. (a) Initial image, f (for example) of a mosaic; (b) marker image g obtained by eroding the initial image by a disk of size 4; (c) reconstruction of f from g (i.e., supremum γg (f ) of all pulse openings for the pulses initial image (a)).

work that way, but by direct supremums and infimums). Figure 17 depicts the behavior of a reconstruction opening in one dimension. In the cross sections of f , the connected components are preserved when they hit those of marker g and are removed when not. The same operator, but in 2D, is depicted in Fig. 18, where marker g is obtained by eroding the initial image f . The cleaning effect of the opening operates here by suppressing (or not) the contour lines but never by smoothing them.

VIII. H IERARCHIES AND C ONNECTED O PERATORS Most of the operations ψ on sets or functions depend on a size, or scale parameter λ, and for the sake of consistency, the family {ψλ , λ > 0} must


213

satisfy the internal structure of a semigroup, which states that the composition product ψμ ψλ lies still in the family. If in addition, we want the ψλ to be increasing and idempotent operators, then the convenient structure is that of a Matheron semigroup, defined by the law ψμ ψλ = ψsup(λ,μ)

λ, μ 0,

(12)

where the value λ = 0 is associated with the identity operator. When the filters {ψλ , λ > 0} are connected for a criterion σ , then the segmentations of any function f ∈ F according to the (σ, ψλ )s increase with λ, and their boundary sets decrease with λ (Braga-Neto and Goutsias, 2003). This property may be considered an answer to the epistemological question “by composing a (nonoptimal) operator with an optimization procedure, do we still obtain something that can be called optimal?” The answer is positive, in the connective framework, and gives the sense of variation of the product optimum as a function of λ. In the functional approach, a similar parameter λ appears, for example, when balancing the term “λ× perimeter of the partition” against a space integral that increases with the amending effect of ψ. But the resulting partitions are not ordered. Here the reduction holds on the boundary set itself, and not on its measure, which can be infinite. Practitioners often take advantage of such a pyramidal structure to create synthetic partitions that borrow their contours to different levels of the hierarchy (Salembier and Marqués, 1999) (for example, a small detail is kept at a low level and propagated to a higher one; this can work well because the boundaries of the small detail miss those of the larger partition). How should the {ψλ } families be chosen so that they satisfy Matheron semigroups? This question is answered for the three types of filters introduced in the previous section. 1. All sequences {γλ , λ > 0} of (connected or not) decreasing openings, and by duality the sequences of increasing closings {ϕλ , λ > 0}, do satisfy the semigroup in Eq. (12); the {γλ } (respectively {ϕλ }) families are called granulometries (respectively antigranulometries). 2. Consider the composition product Mi = ϕi γi . . . ϕ2 γ2 ϕ1 γ1 , where {γi } is a connected granulometry, and {ϕi } a connected antigranulometry. The Mi are filters, called alternating sequential filters (ASFs), and the {Mi , i a positive integer} families satisfy the Matheron semigroup equation (12) (Sternberg, 1986; Serra, 1988). 3. Any leveling (11) depends on a marker parameter M. Order the markers according to their activity, by stating that M2 is more active than M1 when, for all sets A, marker M2 hits both more grains and more pores of A than M1 does (written as M1 M2 ). The families of levelings that are

214

SERRA

(a)

(b)

(c)

F IGURE 19. Leveling by flat zones. (a) Spanish village of Altea; (b) and (c) levelings of (a) by the markers M80 and M230 , respectively, where Mm is the function of all maxima and minima of dynamic > m.

parametrized by activity satisfy the Matheron semigroup λμ0

⇒

Mμ Mλ

⇒

νMμ νMλ = νMλ νMμ = νMλ , (13)

which allows precise control of their amending effect (Serra, 2000). A photograph of the Spanish village of Altea (Fig. 19), illustrates the leveling semigroup (13). The self-dual marker m80 is obtained by replacing f by 0 out of the maxima and minima of f with a dynamics 80 (over 256 gray levels) and by leaving f unchanged on these extrema (the maxima of f with a dynamics k are those that emerge at more than k above the first saddle point when going down. They are obtained by the reconstruction opening γ rec (f ) of f by marker f − k, where k is a positive constant). The leveling hierarchy may be illustrated as follows. Take the transform image Fig. 19(b) as a new initial image, and compute its flat leveling for the marker m230 made of all extrema with a dynamics 230. It results in Fig. 19(c). As levelings satisfy Eq. (13), the image depicted in Fig. 19(c) would have been the same as if we had directly calculated the leveling of the initial image f for marker m230 . Leveling(f/m230 ) = Leveling Leveling(f/m80 )/m230 = Leveling Leveling(f/m230 )/m80 . The progressive leveling action appears clearly when confronting Fig. 19(b) and (c). Notice the relatively correct preservation of some fine details and border lines, which are kept because of their high dynamics, unlike other elements (textures, or a house near the sea, for example) that completely vanish.


215

IX. C ONCLUSION As a method for artificial vision, mathematical morphology is trapped by its inability to grasp the universe of meanings and the symbolic representations of the human mind. If it sometimes is more efficient than other approaches in pattern recognition, this is due to a better hold on the geometry of the scenes under study, but nothing more. However, physics or biology offer situations where the method may be fruitful, because a number of natural phenomena are managed by supremum/infimum-type relationships (possibly combined with other laws) more often than one can imagine today. As far as applications are concerned, the 1960s were the time of the first wails of the newborn method, with regard to petrography. With the 1970s the world of stereology and of optical quantitative microscopy (material sciences, biology) was in the foreground, replaced during the 1980s by industrial control and medical radiology. The 1990s brought a new setting again with the multimedia environment. The current decade is more oriented toward large databases. On the one hand, the recent earth satellites generate a profusion of extraordinarily precise images of cities, coasts, forests, and so on, which are ideal for assessing developments and risks of disasters. On the other hand, data mining from large bases, such as the Internet, for various identification purposes and also for security controls, requires morphological processing. Each new wave of applications superimposes itself on the previous, still-active ones. The segmentation of a space where a physical variable spreads out was set as the largest partition of this space into homogeneous regions. We showed that this problem admits a largest solution when the underlying criterion is connective – when it generates connections over all the subsets where it is satisfied, and uniquely in that case (Theorem 2). We could derive from this identification various combinations for segmentations, by infima, and by iterations (partial segmentations). In addition, by introducing connected operators, we are able to manage composition products between segmentations and filters. By so doing, we did not set out to discover a general segmentation procedure, which probably does not exist. Rather, we focused on the conditions criteria must fulfil for decomposing a complex situation into simpler ones while maintaining optimality.

R EFERENCES Alvarez, L., Guichard, F., Lions, P.-L., Morel, J.M. (1992). Axiomatisation et nouveaux opérateurs de la morphologie mathématique. C. R. Acad. Sci. Paris 315, 265–268.

216

SERRA

Angulo, J. (2003). Morphologie mathématique et indexation d’images couleur. Application à la microscopie en biomédecine. Doctoral thesis, Centre de Morphologie Mathématique, Ecole des Mines, Paris. Angulo, J., Jeulin, D. (2007). Stochastic watershed segmentation. In: Barrera, J., Braga-Neto, U.M. (Eds.), Mathematical Morphology and its Applications to Image and Signal Processing. Ministério da Ciência e Tecnologia, Brasil, pp. 265–276. Angulo, J., Serra, J. (2007). Modeling and segmentation of color images in polar representations. Image and Vision Computing 25, 475–495. Banon, G., Barrera, J., Braga-Neto, U.M. (Eds.) (2007). Mathematical Morphology and its Applications to Image and Signal Processing. Ministério da Ciência e Tecnologia, Brazil. Bertrand, G. (1999). New notions of discrete topology. In: Bertrand, G., Couprie, M., Perroton, L. (Eds.), Lecture Notes in Computer Science, vol. 1568. Springer, New York, pp. 218–227. Bertrand, G. (2005). On topological watersheds. Journal of Mathematical Imaging and Vision 22, 217–230. Beucher, S. (1990). Segmentation d’images et morphologie mathématique. PhD thesis, Ecole des Mines de Paris. Beucher, S., Lantuejoul, C. (1979). Use of watersheds in contour detection. In: Proceedings of the International Workshop on Image Processing, RealTime Edge and Motion Detection/Estimation. Rennes, France, September. Bilodeau, M., Meyer, F., Schmitt, M. (2005). Space, Structure, and Randomness. Springer, New York. Borgefors, G., Ramella, G., Sanniti di Baja, G. (2001). Hierarchical decomposition of multi-scale skeletons. IEEE Trans. on Pattern Analysis and Machine Intelligence 23, 1296–1312. Braga-Neto, U.M., Goutsias, J. (2003). A multiscale approach to connectivity. Computer Vision and Image Understanding 89, 70–107. Carron, T., Lambert, P. (1994). Color edege detector using jointly hue, saturation and intensity. Proc. of IEEE International Conference on Image Processing (ICIP’94), 977–981. Chassery, J.M., Garbay, C. (1984). An iterative segmentation method based on a contextual color and shape criterion. IEEE Transactions on Pattern Analysis, Machine Intelligence 6, 794–800. Cheng, H.D., Jiang, X.H., Sun, Y., Wang, J. (2001). Color image segmentation: Advances and prospects. Pattern Recognition 34, 2259–2281. Cousty, J., Bertrand, G., Najman, L., Couperie, M. (2007a). Watersheds, minimum spanning forests, and the drop water principle. Submitted to IEEE. Cousty, J., Najman, L., Couprie, M., Clément-Guinaudeau, S., Goissen, T., Garot, J. (2007b). Automated, accurate and fast segmentation of 4D cardiac


217

MR images. In: Sachse, F.B., Seemann, G. (Eds.), Procs. of Functional Imaging and Modeling of the Heart 2007. In: LNCS, vol. 4466, pp. 474– 483. Crespo, J., Serra, J., Schafer, R.W. (1995). Theoretical aspects of morphological filters by reconstruction. Signal Processing 47, 201–225. Garrido, L., Salembier, Ph., Garcia, D. (1998). Extensive operators in partition lattices for image sequence analysis. Signal Processing 66, 157–180. Gevers, Th., Smeulders, A.W.M. (1999). Color-based object recognition. Pattern Recognition 32, 453–464. Gomila C. (2001). Mise en correspondance de partitions en vue du suivi d’objets. Doctoral thesis, Ecole des Mines, Paris. Goutsias, J., Vincent, L., Bloomberg, D.S. (Eds.) (2000). Mathematical Morphology and its Applications to Image and Signal Processing. Kluwer, Amsterdam. Hanbury, A., Serra, J. (2003). Colour image analysis in 3D-polar coordinates. Proc. of DAGM Symposium, Vienna. Heijmans, H.J.A.M. (1994). Morphological operators. In: Hawkes, P. (Ed.), Advances in Electronics and Electron Physics, 24, vol. 50. Academic Press, Boston. Heijmans, H.J.A.M. (1999). Connected morphological operators for binary images. Computer Vision and Image Understanding 73, 99–120. Heijmans, H.J.A.M., Roedink, J.B.T.M. (Eds.) (1998). Mathematical Morphology and its Applications to Image and Signal Processing. Kluwer, Amsterdam. Jeulin,, D. (Ed.) (1997). Advances in Theory and Applications of Random Sets. World Scientific. Kiselman, C. (2004). Convex functions on discrete sets. In: Klette, R., Zunic, J. (Eds.), In: Lecture Notes in Computer Science, vol. 3322. IWCIA, pp. 443–457. Lantuejoul, C. (2002). Geostatistical Simulation, Models and Algorithms. Springer, Berlin. Maragos, P. (1996). Differential morphology and image processing. IEEE Trans. Image Processing 78, 922–937. June. Maragos, P., Meyer, F. (1999). Nonlinear PDEs and numerical algorithms for modeling levellings and reconstructions filters. In: Mads, N., et al. (Eds.), Scale-Space Theories in Computer Vision. In: Lecture Notes in Computer Science, vol. 1682, pp. 363–374. Maragos, P., Schafer, R.W., Butt, M.A. (Eds.) (1996). Mathematical Morphology and its Applications to Image and Signal Processing. Kluwer, Amsterdam. Matheron, G. (1975). Random Sets and Integral Geometry. Wiley, New-York. Melin, E. (2003). Digital straight lines in Khalimsky plane. Accepted for publication in Mathematica Scandinavia.

218

SERRA

Meyer, F. (1998). The levelings. In: Mathematical Morphology and its Applications to Image and Signal Processing. Kluwer, Amsterdam. Meyer, F. (2001). An overview of morphological segmentation. International Journal of Pattern Recognition and Artificial Intelligence 15, 1089–1118. Meyer, F., Beucher, S. (1990). Morphological segmentation. JVCIR 1, 21–46. Morel, J.-M., Solimi, S. (1995). Variational Methods in Image Segmentation. Birkhäuser, Boston. Mumford, D., Shah, J. (1988). Boundary detection by minimizing functionals. In: Ulmann, S., Richards, W. (Eds.), Image Understanding. Najman, L., Couperie, M., Bertrand, G. (2005). Mosaics and watersheds. In: Ronse, C., Najman, L., Decencière, E. (Eds.), Mathematical Morphlology: 40 Years On. Springer, pp. 187–196. Newton, I. (1671). A theory concerning light and colors. Philosophical Transactions of the Royal Society 80, 3075–3087. Osher, S., Sethian, J.A. (1988). Front propagating with curvature-dependant speed: Algorithms based on Hamilton–Jacobi formulations. J. Comput. Physics 79, 12–49. Pavlidis, T. (1977). Structural Pattern Recognition. Springer, New York. Pesaresi, M., Benediktsson, J.A. (2001). A new approach for the morphological segmentation of high-resolution satellite imagery. IEEE Transactions on Geoscience and Remote 39, 309–320. Ronse, C. (1998). Set theoretical algebraic approaches to connectivity in continuous or digital spaces. Journal of Mathematical Imaging and Vision 8, 41–58. Ronse, C., Serra, J. (2007). Les fondements algébriques de la morphologie. In: Najman, L., Talbot, H. (Eds.), Morphologie Mathématique: État de l’Art. Hermès. Ronse, C., Najman, L., Decencière, E. (Eds.) (2005). Mathematical Morphlology: 40 Years On. Springer, Berlin. Salembier, P., Marqués, F. (1999). Region-based representations of image and video: Segmentation tools for multimedia services. IEEE Transactions on Circuits and Systems for Video Technology 9, 1147–1167. Salembier, P., Serra, J. (1995). Flat zones filtering, connected operators, and filters by reconstruction. IEEE Transactions on Image Processing 4, 1153– 1160. Serra, J. (Ed.) (1988). Image Analysis and Mathematical Morphology, Part II: Theoretical Advances. Academic Press, London. Serra, J. (1999). Set connections and discrete filtering (Proc. DGCI 1999). In: Bertrand, G., Couprie, M., Perroton, L. (Eds.), In: Lecture Notes in Computer Science, vol. 1568. Springer, pp. 191–206. Serra, J. (2000). Connectivity for sets and functions. Fundamenta Informaticae 41, 147–186.


219

Serra, J. (2002). Morphological descriptions using three-dimensional wavefronts. Image Anal. Stereol. 21, 1–9. Serra, J. (2005). Morphological segmentation of colour images. In: Ronse, C., Najman, L., Decencière, E. (Eds.), Mathematical Morphlology: 40 Years On. Springer, pp. 151–176. Serra, J. (2006). A lattice approach to image segmentation. Journal of Mathematical Imaging and Vision 24, 83–130. Serra, J. (2007a). Digital Steiner sets and Matheron Semli-groups. In: Banon, G., Braga-Neto, U.M. (Eds.), ISMM 2007. Serra, J. (2007b). The random spread model. In: Banon, G., Braga-Neto, U.M. (Eds.), ISMM 2007. Serra, J., Chermant, J.L. (1997). Mathematical morphology: A historical overview. Acta Stereologica 16, 156–165. Serra, J., Salembier, P. (Eds.) (1993). Mathematical Morphology and its Applications to Signal Processing. Univ. Politèchnica de Catalunya. Serra, J., Soille, P. (Eds.) (1994). Mathematical Morphology and its Applications to Image Processing. Kluwer, Amsterdam. Soille, P. (1999). Morphological Image Analysis. Springer-Verlag, Berlin/Heidelberg. Sternberg, S.R. (1986). Morphology for gray-tone functions. Computer Vision, Graphics and Image Processing 35, 333–355. Talbot, H., Beare, R. (Eds.) (2002). Mathematical Morphology. CSIRO, Sydney. van den Boomgaard, R., Smeulders, A. (1994). The morphological structure of the images: The differential equations of morphological scale-space. IEEE Trans. Pattern. Anal. Mach. Intellig. 16, 1101–1113. Vincent, L. (1993). Morphological grayscale reconstruction in image analysis: Applications and efficient algorithms. IEEE Transactions in Image Processing 2, 176–201. Wilkinson, M. (2005). Attribute-space connected filters. In: Ronse, C., Najman, L., Decencière, E. (Eds.), Mathematical Morphlology: 40 Years On. Springer, pp. 85–94.


Cumulative Author Index

This index lists the authors of all articles that have appeared in Advances in Electronics and Electron Physics, vols. 1–89 and Advances in Imaging and Electron Physics, vols. 90–149, labeled E; authors of supplements to Advances in Electronics and Electron Physics are labeled ES; authors of contributions to two serials that have merged with these Advances, namely Advances in Image Pickup and Display (labeled D or DS for the sole supplement) and Advances in Optical and Electron Microscopy (labelled M) are also included.

A

Amboss, K., E:26, 1 Amores, J., E:144, 291 An, M., E:80, 1; E:93, 1 Anderson, A. E., E:12, 59; E:16, 299 Anderson, D. G., E:16, 547 Anderton, H., E:22 B, 919 Andreadis, I., E:110, 63; E:119, 1 Anupindi, N., E:93, 1 Arce, G. R., E:117, 173 Arneodo, A., E:126, 1 Arnoldus, H. F., E:132, 1 Arps, R. B., ES:12, 219 Arsac, J., E:48, 202 Asakura, K., E:96, 251 Asano, M., E:22 A, 643; E:22 A, 651; E:28 B, 309; E:28 A, 381 Ashworth, F., E:3, 1 Asif, A., E:145, 1 Aslam, M., E:22 A, 113; E:22 A, 407; E:22 A, 571 Åström, A., E:105, 2 Aubert, J., ES:13A, 159 Aukerman, L. W., E:34, 95 Auld, B. A., E:71, 251

Aach, T., E:128, 1 Abbiss, J. B., E:87, 1 Abdelatif, Y., E:93, 1 Abies, H. D., E:28 A, 1 Abraham, G., E:35, 270 Abraham, J. M., E:22 B, 671 Abrahams, E., E:6, 47 Adachi, K., E:96, 251 Adams, J., E:22 A, 139 Adams, K. M., E:37, 80 Ade, G., E:89, 1 Adler, I., ES:6, 313 Adriaanse, J.-P., M:14, 2 Afzelius, B. A., E:96, 301, 385 Agaian, S. S., E:130, 165 Agar, A. W., E:96, 415; E:133, 317 Aggarwal, J. K., E:87, 259 Ahmad, N., E:28, 999 Ahmed, H., E:102, 87; E:133, 179 Aikens, R., E:16, 409 Airey, R. W., E:16, 61; E:22 A, 113; E:22 A, 407; E:22 A, 571; E:28 A, 89; E:40 A; E:40 B Akis, R., E:107, 2 Alam, M. F., E:107, 73 Alam, M. S., E:89, 53 Albanese, R., E:102, 2 Alexander, J. W. F., E:16, 247 Allan, F. V., E:16, 329 Allen, J. D., E:22 B, 835; E:22 B, 849 Allen, R. D., M:1, 77 Alpern, M., E:22 A, 5 Alvarez, L., E:111, 168

B Bacik, H., E:28 A, 61 Bacsa, W. S., E:110, 1 Baier, P. W., E:53, 209 Bakken, G. S., E:28 B, 907 Bakos, G., E:22 B, 713 Bakos, J. S., E:36, 58 Baldinger, E., E:8, 255

221

222

CUMULATIVE AUTHOR INDEX

Balk, L. J., E:71, 1 Ball, J., E:22 B, 927 Ballu, Y., ES:13B, 257 Bandyopadhyay, S., E:89, 93 Baranova, L. A., E:76, 3 Barbe, D. F., D:3, 171 Bargellini, P. L., E:31, 119 Barlow, G. E., E:11, 185 Barnabei, M., E:125, 1 Barnes, A., E:36, 1 Barnes, C. F., E:84, 1 Barnett, M. E., E:28 A, 545; M:4, 249 Barton, G., E:16, 409 Barybin, A. A., E:44, 99; E:45, 1 Baskett, J. R., E:28 B, 1021 Bastard, G., E:72, 1 Basu, S., M:8, 51 Bates, C. W., E:28 A, 451; E:28 A, 545 Bates, D. J., E:44, 221 Bates, R. H. T., E:67, 1 Batey, P. H., E:22 A, 63 Baud, C., E:48, 1 Baum, P. J., E:54, 1 Baum, W. A., E:12, 1; E:12, 21; E:16, 383; E:16, 391; E:16, 403; E:22 A, 617; E:28 B, 753 Baumgartner, W., E:28 A, 151 Baun, W. L., ES:6, 155 Beadle, C., M:4, 361 Beckman, J. E., E:22 A, 369; E:28 B, 801 Bedini, L., E:97, 85, E:120, 193 Beesley, J., E:22 A, 551 Beizina, L. G., E:89, 391 Bell, A. E., E:32, 194 Bellier, M. M., E:16, 371 Béné, G. J., E:27, 19; E:49, 86 Benford, J. R., M:3, 1 Berényi, D., E:42, 55; E:56, 411 Berg, A. D., E:22 B, 969 Berg, H. C., M:7, 1 Berger, H., E:22 B, 781 Bergeson, H. E., E:45, 253 Bernet, S., E:146, 1 Berry, R. S., E:51, 137; E:57, 1 Bertero, M., E:75, 2; ES:19, 225 Berz, M., E:108, 1 Bescos, J., E:80, 309 Bethge, H., M:4, 237 Beurle, R. L., E:12, 247; E:16, 333; E:28 B, 635 E:28 B, 1043

Beyer, R. R., E:22 A, 229; E:22 A, 241; E:22 A, 251; E:28 A, 105; E:28 A, 229 Bianchini, M., E:140, 1 Bied-Charreton, P., E:28 A, 27 Bijaoui, A., E:22 A, 5; E:28 A, 27 Billig, E., E:10, 71 Billingsley, F. C., M:4, 127 Binh, V. T., E:95, 63; E:148, 1 Binnie, D. M., E:16, 501 Biondi, M. A., E:18, 67 Bird, J. P., E:107, 2 Birnbaum, G., ES:2 Biro, O., E:82, 1 Black, N. D., E:112, 1 Blake, J., E:16, 213 Blamoutier, M., E:27 A, 273 Blanc, A., E:141, 1 Blewett, J. P., E:29, 223 Bloch, F., E:3, 15 Bloch, I., E:128, 51 Blum, E.-J., E:56, 98 Boato, G., E:60, 95 Boccacci, P., ES:19, 225 Boerio, A. H., E:22 A, 229; E:27 A, 159 Bogdanov, E. V., E:21, 287 Boischot, A., E:20, 147 Boiti, M., ES:19, 359 Bok, A. B., M:4, 161 Boksenberg, A., E:28 A, 297 Bonetto, R. D., E:120, 135 Bonjour, P., ES:13A, 1 Bonnet, N., E:114, 1 Bostanjoglo, O., E:76, 209; E:110, 21; E:121, 1 Bostrom, R. C., M:2, 77 Bottino, A., E:139, 1 Bouix, S., E:135, 1 Boulmer, J., E:39, 121 Boussakta, S., E:111, 3 Boussuge, C., E:16, 357 Boutot, J. P., E:60, 223 Bouwers, A., E:16, 85 Bowen, J. S., E:28 B, 767 Bowers, M. T., E:34, 223 Bowers, R., E:38, 148 Bowhill, S. A., E:19, 55 Bowles, K. L., E:19, 55 Boyce, J. F., E:77, 210 Boyde, A., E:133, 165 Boyer, L.A., E:22 B, 885

CUMULATIVE AUTHOR INDEX Bradley, D. J., E:22 B, 985 Brand, P. W. J. L., E:28 B, 737 Brand, P. W., E:22 B, 741; E:28 B, 783 Brandinger, J. J., D:2, 169 Brandon, D. G., M:2, 343 Branscomb, L. M., E:9, 43 Bratenahl, A., E:54, 1 Brau, C. A., ES:22 Brauer, W., E:11, 413 Brault, J. W., M:1, 77 Braun, P., E:57, 231 Bray, J., E:91, 189 Bremmer, H., E:61, 300 Breton, B. C., E:133, 449 Bretto, A., E:131, 1 Brianzi, P., ES:19, 225 Brillouin, L., E:3, 15; E:3, 85 Brittain, J. E., E:50, 412 Brodersen, R. W., E:51, 265 Brodie, I., E:83, 1 Broers, A. N., E:133, 207 Broerse, P. H., E:22 A, 305 Brooks, F. P., E:18, 45 Brooks, H., E:7, 85 Broussaud, G., E:19, 255 Brown, J. D., ES:6, 45 Brown, J., E:10, 107 Bruin, F., E:15, 327 Bube, R. H., E:56, 163 Bugnolo, D. S., E:61, 300 Burns, J., E:12, 97 Burstein, E., E:7, 1 Burtt, R. B., E:16, 213; E:16, 247 Buzzi, J. M., ES:13C, 351 Byatt, D., E:16, 265

C Cahay, M., E:89, 93 Cailler, M., E:61, 162 Calderwood, J. H., E:22 B, 1003 Caldwell, D. O., E:16, 469; E:16, 475 Calestani, G., E:123, 29, 311 Calogero, F., ES:19, 463 Camarcat, N., ES:13C, 351 Campagna, M., E:41, 113 Campana, S. B., D:3, 171 Canadas, G., ES:19, 533 Cantini, P., E:60, 95 Capitelli, F., E:123, 291

223

Cappellini, V., E:66, 141 Carinena, J. F., E:72, 182 Carini, A., E:124, 1 Carlemalm, E., E:63, 270 Casazza, P. G., E:115, 1 Cassiday, G. L., E:45, 253 Castaing, R., E:13, 317 Castenholz, A., M:7, 73 Catchpole, C. E., E:16 A, 567 Catchpole, C. E., E:22 A, 113; E:22 A, 425 Caulfield, H. J., E:142, 1 Celotta, R. J., E:56, 219 Cervellera, C., E:140, 61 Cha, B. H., E:134, 1 Chadan, G., ES:19, 175 Challinor, A., E:95, 271 Cham, W. K., E:88, 1 Champion, R. L., E:58, 143 Champness, P. E., E:101, 1; E:121, 53 Chan, F. T., E:49, 134 Chang, J., ES:13C, 295 Chapel, F., ES:19, 533 Charles, D. R., E:22 A, 315; E:22 A, 323 Charman, W. H., E:23 B, 705 Charman, W. N., E:22 A, 101 Charrier, M. S., E:16, 5 Chatterton, P. A., E:28 B, 1041 Chavent, G., ES:19, 509 Chen, W., E:102, 87 Chen, Y., E:117, 1 Chen, Z. W., E:83, 107 Chenett, E. R., E:46, 314; E:23, 303 Chernov, Z. S., E:21, 287 Chirico, G., E:126, 195 Chizeck, H. J., E:79, 1 Cho, Y., E:127, 1 Chodorow, M., E:3, 43 Chouffani, K., E:142, 1 Christophorou, L. G., E:46, 56 Churchill, J. L. W., E:8, 317 Churchill, J. N., E:47, 267; E:58, 1 Ciddor, P. E., M:8, 25 Clayton, R. H., E:22 A, 507 Cleaver, J. R. A., E:133, 485 Cohen, M. L., E:51, 1 Cohen, M., E:28 B, 725 Cola, R., D:3, 83 Colby, J. W., ES:6, 177 Cole, F. T., E:71, 75

224


Coles, D. K., E:2, 300 Colliex, C., E:63, 270; M:9, 65 Collings, P. R., E:28 A, 105 Collins, T. W., E:47, 267; E:58, 1 Combes, M., E:28 A, 39 Combettes, P. L., E:95, 155 Condon, P. E., E:12, 123 Conrad, A. C., E:28 B, 907 Cooper, J. A., E:86, 1 Cooper, A. W., E:24, 155 Cooper, R., E:22 B, 995 Cope, A. D., E:22 A, 175 Corney, A., E:29, 115 Cornille, H., ES:19, 481 Corps, R. J., E:28 B, 827 Cosslett, V. E., E:96, 3; ES:16, 23; M:10, 215, E:133, 237, 285 Cowley, J. M., E:46, 1; E:98, 323 Cozens, J. R., E:20, 99 Cram, L. E., E:54, 141 Cranstoun, G. K. L., E:28 B, 875 Crewe, D. A., E:102, 187; E:121, 91 Cristobal, G., E:80, 309 Cristóbal, G., E:97, 1; E:131, 65 Crompton, R. W., E:27, 1 Crosta, G., ES:19, 523 Csanády, A., E:96, 181 Cuer, M., ES:19, 533 Cufi, X., E:120, 1 Culshaw, W., E:15, 197 Cumming, G. L., ES:19, 179 Cuninghame-Green, R. A., E:90, 1 Cuocci, C., E:123, 291 Cuperman, V., E:82, 97 Curran, S. C., E:8, 317 Curzon, A. E., E:24, 109 Cusick, D. R., E:68, 1 Czekalowski, G. W. A., E:28 B, 653

D da Silva, E. A. B., E:97, 191 Danforth, W. E., E:5, 169 Daniels, M. V., E:28 B, 635 Danilatos, G. D., E:71, 110; E:78, 2 D’Ariano, G. M., E:128, 205 David, D.-J., E:57, 411 Davidson, J. L., E:84, 61 Davies, A. R., ES:19, 553 Davies, A. J., E:47, 51

Davies, E. R., E:126, 93 Davies, J. G., E:9, 95 Davis, G. P., E:16, 119 Davis, R. J., E:22 B, 875 Dawson, P. H., E:27, 59; E:31, 1; E:53, 153; ES:13B, 173 Day, J. E., E:10, 239 de Baere, W., E:68, 235 de Baets, B., E:89, 255 de Haan, E. F., E:12, 291 de Hosson, J. T. M., E:125, 119 de Lang, H., M:4, 233 de Man, H. J., E:55, 77 de Mey, G., E:61, 2 de Raedt, H., E:125, 119 de Ridder, D., E:126, 351 de Santis, A., E:99, 291; E:119, 267 de Santis, P., ES:19, 281 de Witt, J. H., E:16, 419 Dean, R. J., E:22 A, 441 Decker, R. W., E:28 A, 19; E:28 A, 357 Declerck, G. J., E:47, 197 DeCorpo, J. J., E:42, 2 Decoster, N., E:126, 1 Deen, M. J., E:99, 65; E:106, 2; E:109, 2 Defrise, M., ES:19, 261 Degasperis, A., ES:19, 323 Delalande, C., E:72, 1 Delcroix, J.-L., E:35, 88 Delong, A., ES:16, 63 Delpech, J.-F., E:39, 121; E:46, 131 Deltrap, J. H. M., E:28 A, 443 DeMol, Ch., ES:19, 261 den Dekker, A. J., E:117, 241; E:130, 1 Denisse, J. F., E:20, 147 Dennison, E. W., E:12, 307; E:16, 447; E:22 A, 435; E:28 B, 767 Deprettere, E. F. A., E:37, 80 Deriche, R., E:145, 149 Deutscher, K., E:28 A, 419 Deval, L., E:12, 195 Diaspro, A., E:126, 195 Doane, F. W., E:96, 79 Dobie, M. R., E:88, 63 Doe, L. A., E:22 B, 705 Dolan, W. W., E:8, 89 Dolizy, P., E:28 A, 367 Dolon, P. J., E:22 B, 781; E:22 B, 927 Donal, J. S., E:4, 188 Donati, S., E:26, 251

CUMULATIVE AUTHOR INDEX Donelli, G., E:43, 1 Donjon, J., D:1, 225 Donoho, D., E:132, 287 Doolittle II, R. F., E:16, 535; E:22 B, 823 Doran, C., E:95, 271 Dorset, D. L., E:88, 111 Dorsey, J. R., ES:6, 291 Doucet, H. J., ES:13C, 351 Dougherty, E. R., E:117, 1 Doughty, D. D., E:12, 59; E:16, 235; E:22 A, 261 Dow, W. G., E:10, 1; E:17, 1 Dowden, S., E:12, 31 Dracass, J., E:16, 547 Dragoun, O., E:60, 1 Dragt, A. J., E:67, 65 Driard, B., E:22 B, 949; E:28 B, 931 Druckmann, I., E:94, 315 Drummond, D. G., ES:16, 81 Dubbeldam, L., M:12, 140 Dubochet, J., M:8, 107 Duchene, B., ES:19, 35 Duchesne, M., E:12, 5; E:16, 19; E:16, 27; E:22 A, 5; E:22 A, 609; E:28 A, 27 Duchet, M., E:22 A, 323; E:22 A, 499 Ducumb, P., E:133, 251, 269 Duin, R. P. W., E:126, 351; E:133, 251, 269, 285 Dunham, T., E:22 B, 729 Dunlap, J., E:22 B, 713 Dunning, F. B., E:59, 79 Dupouy, G., ES:16, 103; M:2, 167 Dupre, M. M., E:16, 371 Dvoˇrak, M., E:28 A, 347 Dyke, W. P., E:8, 89

E Eberly, D., E:94, 1 Echlin, P., E:133, 469 Edgecumbe, J., E:22 A, 635 Egan, D. W., E:28 B, 801 Egli, P. H., E:7, 1 Egmont-Petersen, M., E:126, 351 Eichmeier, J., E:19, 178 Einstein, P. A., E:12, 317 Eisenstein, A. S., E:1, 1 El-Kareh, A. B., ES:4 Elad, M., E:132, 287 Elliott, C. T., E:22 B, 995; E:28 B, 1041

225

Elmers, H. J., E:142, 159 Elvey, C. T., E:9, 1 Emberson, C. J., E:22 A, 51 Emberson, D. L., E:16, 127; E:22 A, 129; E:28 A, 119 Emeleus, K. G., E:20, 59 England, L., E:28 A, 545 Ennos, A. E., E:12, 317 Eom, K.-B., E:70, 79 Erbudak, M., E:62, 2 Erickson, H. P., M:5, 163 Erickson, W. C., E:32, 1 Errington, R. J., E:113, 249 Eschard, G., E:28 A, 499; E:28 B, 989 Essig, S. E., E:12, 73 Evangelista, G., E:117, 73 Evans, H. D., E:12, 87 Evans, J. H., M:5, 1 Everhart, T. E., E:133, 137, 147

F Faber, J. S., E:111, 92 Facchi, P., E:142, 53 Fahrenbruch, A. L., E:56, 163 Farago, P. S., E:21, 1 Faruqi, A. R., E:145, 55 Faughnan, B. W., D:4, 87 Fawcett, J. M., E:28 A, 289 Fay, C. E., ES:5 Fäy, T. D., E:22 B, 723 Feibelman, W. A., E:16, 235 Feinerman, A. D., E:102, 187; E:121, 91 Feinstein, D. L., E:32, 312 Felenbok, P., E:28 A, 39 Ferguson, E. E., E:24, 1 Fernández-Morán, H., ES:16, 167 Ferraro, M., E:84, 131 Ferrier, R. P., M:3, 155 Ferry, D. K., E:58, 312; E:107, 2 Ferwerda, H. A., E:66, 202 Fiddy, M. A., E:87, 1; ES:19, 61 Fiermans, L., E:43, 139 Filby, R. S., E:22 A, 273 Fink, J., E:75, 122 Finn, B. S., E:50, 176 Fisher, A. D., E:69, 115 Fisher, D. G., D:1, 71 Fisher, R. M., E:96, 347 Flanagan, T. P., E:16, 547 Fleming, W. J., E:31, 162

226


Flinn, E. A., E:12, 87; E:16, 155 Flory, L. E., E:22 B, 885 Fokas, A. S., ES:19, 409 Folkes, J. R., E:16, 325; E:28 A, 375 Fonseca, S. B. A., E:65, 2 Foote, D. P., E:28 B, 1059 Forchheimer, R., E:105, 2 Ford, W. K., E:12, 21; E:16, 403; E:22 B, 697 Foreman, P. H., E:16, 163 Forest, E., E:67, 65 Foster, G., E:49, 134 Fourie, J. T., E:93, 57 Fowler, R. G., E:20, 1; E:35, 1; E:41, 1 Fowweather, F., E:12, 311 Frank, K., E:38, 55 Frank, L., E:123, 327; E:128, 309 Franks, J., E:47, 1 Fransen, M. J., E:111, 92 Franzen, W., E:8, 255; E:39, 73 Fraser, D. B., D:2, 65 Fredendall, G. L., D:2, 169 Frederick, L. W., E:16, 403; E:22 B, 723 Freeman, A. J., E:65, 358 Freeman, J. R., ES:13C, 295 Freeman, K. G., E:28 B, 837 Freeman, M. R., E:125, 195 Freixenet, J., E:120, 1 Frey, J., E:38, 148 Frieden, B. R., E:90, 123; E:97, 409 Fritz, E. F., E:49, 299 Fröhlich, H., E:53, 85 Frolich, H., E:2, 185 Fromm, W. E., E:4, 258 Frost, R. L., E:84, 1 Fujioka, H., E:73, 234 Fujita, H., E:96, 749 Fukami, A., E:96, 251 Funke, H., ES:19, 99 Fürhapter, S., E:146, 1

G Gabler, F., M:4, 385 Gader, P., E:144, 165 Gagnepain, J.-J., E:77, 84 Gahm, J., M:5, 115 Galan-Malaga, H., ES:19, 51 Gale, B. C., E:28, 999 Galtier, Ch., E:48, 202

Ganachaud, J. P., E:61, 162 Ganson, A., E:22 A, 11; E:22 A, 31 García, A. G., E:124, 63 Garcia, N., E:95, 63 Gardiol, F. E., E:59, 139; E:63, 140 Garfield, B. R. C., E:16, 329; E:22 A, 459; E:28 A, 375 Garlick, G. F. J., E:2, 152; E:16, 607 Garrett, C. G. B., E:14, 1 Garthwaite, E., E:12, 379 Garwin, E. L., E:22 A, 635 Gasteratos, A., E:110, 63; E:119, 1 Gatti, E., E:26, 251 Gaur, J., D:3, 83 Gauthier, J.-C., E:46, 131 Ge, B. D., ES:19, 71 Gebel, R. K. H., E:12, 195; E:16, 451; E:22 A, 189; E:28 B, 685 Gemmi, M., E:123, 311 Geneux, E., E:27, 19 Georges, A. T., E:54, 191 Gerace, I., E:97, 85 Germani, A., E:99, 291; E:119, 267 Gerthsen, D., E:107, 121 Gertner, I., E:80, 1 Geuens, P., E:136, 111 Geurts, A., E:28 B, 615 Geus, J. W., E:96, 287 Ghose, D., E:79, 73 Giacovazzo, C., E:123, 291 Giardina, C. R., E:67, 121 Giarola, A. J., E:65, 2 Gibson, J. D., E:72, 259 Giese, R., E:16, 113; E:28 B, 919 Gilboa, G., E:136, 1 Gildemeister, O., E:16, 113; E:28 B, 919 Ginzton, E. L., E:3, 43 Glickman, L. G., E:89, 391 Gnädinger, A. P., E:51, 183 Godehardt, R., E:94, 81 Goetze, G. W., E:16, 145; E:16, 557; E:22 A, 219; E:22 A, 229; E:22 A, 241; E:22 A, 251; E:27 A, 159; E:28 A, 105 Goldberg, S., E:14, 207 Goldstein, D. J., M:6, 135 Goldstein, J. I., ES:6, 245 Goldstein, L., E:7, 399 Goloskokov, Y. V., E:89, 391 Gonzalo, C., E:80, 309 Gopinath, A., E:69, 1

CUMULATIVE AUTHOR INDEX Gordon, A. W., E:28 A, 433 Gorenflo, R., ES:19, 243 Gori, F., ES:19, 269, 281 Görlich, P., E:11, 1; E:14, 37 Gorog, I., D:4, 87 Goto, S., E:16, 621 Goulard, B., E:109, 163 Gousseau, Y., E:111, 168 Graf, J., E:28 A, 499 Granatstein, V. L., E:32, 312 Graves, C. D., E:16, 535; E:22 B, 823 Greatorex, C. A., E:12, 317; E:16, 593 Green, M., E:22 A, 251; E:28 B, 807 Greenspan, D., E:63, 189; E:98, 1 Grella, R., ES:19, 269 Griffith, O. H., M:10, 269 Grigoryan, A. M., E:130, 165 Grivet, P., E:2, 48; E:23, 39; E:50, 89; ES:16, 225 Groom, M., M:8, 107 Grosch, G. A., E:28 B, 603 Grosse, A., E:22 A, 465 Grossmann, A., ES:19, 289 Grover, D., E:91, 231 Groves, P. R., E:28 B, 827 Grubin, H. L., E:51, 310 Grümm, H., E:96, 59 Grunbaum, F. A., ES:19, 307 Grzymala-Busse, J. W., E:94, 151 Guenard, P. R., E:3, 43 Guérin, J., E:28 A, 39 Guest, A., E:28 A, 471 Guillard, C., E:22 A, 315 Guillemin, E. A., E:3, 261 Guldner, Y., E:72, 1 Gull, S., E:95, 271 Gumnick, J. L., E:22 A, 507 Guyot, L. F., E:16, 91; E:22 B, 949

H Haar, F. M., E:106, 293 Hachenberg, O., E:11, 413 Hagen, C. W., E:143, 1 Haguenau, F., E:96, 93 Hahn, E., E:75, 233 Haine, M. E., E:12, 317; E:6, 295 Hall, C. E., ES:16, 275 Hall, J. S., E:12, 21; E:16, 403 Hallerman, G., ES:6, 197

227

Hambrecht, F. T., E:38, 55 Hanbury, A., E:128, 123 Handel, P. H., E:69, 55 Hanna, A. H., E:28 A, 443 Hansen, J. R., E:28 B, 807 Hanson, D. C., E:57, 145 Hanszen, K.-J., E:59, 2; M:4, 1 Harbour, J., E:12, 311 Hardy, D. F., M:5, 201 Hardy, J., E:142, 1 Hardy, T. D., E:106, 2 Harmuth, H. F., E:36, 195; E:41, 168; E:50, 261; E:129, 1, E:137, 1; ES:9; ES:11; ES:14; ES:15; ES:18; ES:23 Hartel, P., E:120, 41 Harth, W., E:34, 281; E:38 A, 535 Hartikainen, J., M:12, 313 Hartmann, P., E:22 A, 519; E:28 A, 409 Hartmann, U., E:87, 49 Hase, T., E:79, 271 Hasegawa, S., E:28 B, 553 Hashimoto, H., E:96, 597 Haskell, B. G., ES:12, 189 Hassan, M. F., ES:19, 553 Hasted, J. B., E:13, 1 Haus, H. A., E:38, 195 Hawkes, P. W., E:96, xiii, 405, 849; E:115, 355; E:123, 1, 207; ES:13A, 45; ES:16, 589; ES:7; M:7, 101 Hay, G. A., E:12, 363; E:16, 581; E:28 B, 653 Hayashi, K., E:140, 119 Hayward, R. W., E:5, 97 Hazan, J.-P., D:1, 225 Hébert, C., E:123, 413 Heimann, W., E:12, 235; E:16, 217; E:22 A, 601; E:28 B, 677 Heinrich, H., E:22 A, 355 Heinrich, K. F. J., M:6, 275 Heinzl, J., E:65, 91 Henkin, G. M., ES:19, 191 Henneberger, W. C., E:112, 55 Herbatreit, J. W., E:1, 347; E:20, 199 Herold, E. W., DS:1 Herrmann, K. H., M:9, 1 Herrmann, M., E:28 B, 955 Hersey, J. B., E:9, 239 Herstel, W., E:16, 610; E:22 A, 363; E:28 B, 647

228


Hertz, C. H., E:65, 91 Hess, K., E:59, 239 Hesthaven, J. S., E:127, 59 Hewish, A., E:91, 285 Hewitt, A. V., E:22 A, 101; E:28 A, 1 Heydenreich, J., E:96, 171; M:4, 237 Heyman, E., E:103, 1 Heyman, P. M., D:4, 87 Hézard, C., E:22 A, 609 Hibi, T., ES:16, 297 Hiddink, M. G. H., E:131, 147 Higatsberger, M. J., E:56, 291 Hill, D. A., E:16, 475; E:16, 531 Hills, B. L., E:28 B, 635 Hilsum, C., E:91, 171 Hiltner, W. A., E:12, 17; E:16, 37 Hinder, G. W., E:22 B, 801 Hines, R. L., M:9, 180 Hiraga, K., E:101, 37; E:122, 1 Hirano, T., E:96, 735 Hirashima, M., E:22 A, 643; E:22 A, 651; E:28 B, 309; E:22 A, 661; E:28 A, 381 Hirayama, T., E:28 A, 189 Hirschberg, J. G., M:14, 121 Hirschberg, K., E:28 A, 419 Hirsh, C. J., E:5, 291 Hitz, D., E:144, 1 Hobson, J. P., E:17, 323 Hoene, E. L., E:28 B, 677 Hofmann, I., ES:13C, 49 Hok, G., E:2, 220 Holcomb, W. G., M:2, 77 Holden, A. J., E:91, 213 Holloway, P. H., E:54, 241 Holmes, P. J., E:10, 71 Holmshaw, R. T., E:28 A, 471 Holmstrom, F. E., E:47, 267; E:58, 1 Holscher, H., E:135, 41 Holschneider, M., ES:19, 289 Holt, A. G. J., E:111, 3 Holz, G., D:3, 83 Hooper Jr., E. B., E:27, 295 Hopmann, W., E:22 A, 591; E:22 B, 1011 Hora, H., E:69, 55 Hori, H., E:28 A, 253 Hörl, E. M., E:96, 55 Hormigo, J., E:131, 65 Horne, R. W., M:6, 227 Hosaka, Y., E:96, 735 Hotop, H.-J., E:85, 1

Hounslow, M. W., E:93, 219 Houston, J. M., E:17, 125; E:43, 205 Hovmoller, S., E:123, 257 Hradil, Z., E:142, 53 Hron, F., ES:19, 179 Huang, L. Y., E:96, 805 Hubbard, E. L., E:25, 1 Huebener, R. P., E:70, 1 Huggett, J. M., E:77, 140 Huijsmans, D. P., E:85, 77 Hulstaert, C. E., E:96, 287 Humphries, D. W., M:3, 33 Humphries, S., ES:13C, 389 Hunt, B. R., E:60, 161 Huston, A. E., E:16, 249; E:22 B, 957 Hutchison, J. L., E:96, 393 Hutson, F. L., ES:19, 89 Hutter, E. C., E:7, 363 Hutter, R. G. E., D:1, 163; E:1, 167; E:6, 371 Hynek, J. A., E:16, 409; E:22 B, 713

I Ianigro, M., E:123, 291 Ichikawa, A., E:96, 723 Idier, J., E:141, 1 Iftekharuddin, K. M., E:102, 235 Il’in, V. P., E:78, 156 Imiya, A., E:101, 99 Inghram, M. G., E:1, 219 Inskeep, C. N., E:22 B, 671 Ioannides, A. A., ES:19, 205 Ioanoviciu, D., E:73, 1 Iredale, P., E:22 B, 801; E:28 B, 589; E:28 B, 965 Ito, K., E:96, 659 Ivey, H. F., E:6, 137; ES:1 Izzo, L., E:135, 103

J Jaarinen, J., M:12, 313 Jackson, F. W., E:28 A, 247 Jackson, R. N., E:35, 191 Jackway, P. T., E:99, 1; E:119, 123 Jagannathan, R., E:97, 257 Jain, S. C., E:67, 329; E:78, 104; E:82, 197; ES:24 Jane, M. R., E:16, 501 Jansen, G. H., ES:21

CUMULATIVE AUTHOR INDEX Jareš, V., E:28 A, 523 Jaulent, M., ES:19, 429 Jaumot, F. E., E:17, 207 Jedliˇcka, M., E:22 A, 449; E:28 A, 323 Jeffers, S., E:22 A, 41 Jenkinson, G. W., E:28 B, 1043 Jennings, A. E., E:22 A, 441 Jense, G. J., E:85, 77 Jensen, A. S., E:22 A, 155; E:28 A, 289 Jensen, K. L., E:149, 4 Jervis, P., E:133, 547 Jesacher, A., E:146, 1 Jetto, L., E:99, 291; E:119, 267 Johnson, H. R., E:22 B, 723 Johnson, J. M., E:28 A, 487; E:28 A, 507 Johnson, K. E., E:35, 191 Jones, B. K., E:87, 201 Jones, I. P., E:125, 63 Jones, L. W., E:12, 153; E:16, 513; E:22 B, 813 Jones, R. C., E:5, 1; E:11, 87 Jones, R., E:89, 325 Jordan, J. A., E:28 B, 907 Jordan, A. K., ES:19, 79 Jory, H. R., E:55, 2 Jouffrey, B., E:96, 101; E:123, 413 Jourlin, M., E:115, 129 Joy, R. T., M:5, 297 Judice, C. N., D:4, 157 Jullien, G. A., E:80, 69

K Kabashima, Y., E:125, 231 Kahan, E., E:22 A; E:22 B; E:28 A; E:28 B; E:28 B, 725 E:33 A; E:33 B Kahl, F., E:94, 197 Kaiser, M., E:75, 329 Kajiyama, Y., E:28 A, 189 Kak, S. C., E:94, 259 Kalafut, J. S., E:28 A, 105 Kamogawa, H., E:96, 673 Kan, S. K., E:34, 1 Kanaya, K., E:96, 257; ES:16, 317 Kaneko, E., D:4, 1 Kano, T., E:79, 271 Kao, K. C., E:22 B, 1003 Karady, G., E:41, 311 Karetskaya, S. P., E:89, 391 Karim, M. A., E:89, 53; E:107, 73

229

Karmohapatro, S. B., E:42, 113; E:79, 73 Kashyap, R. L., E:70, 79 Kasper, E., M:8, 207; E:116, 1 Kateshov, V. A., E:78, 156 Kaufman, H. R., E:36, 266 Kaufmann, U., E:58, 81 Kaup, D. J., ES:19, 423 Kaw, P. K., E:27, 187 Kawahara, T., E:28 A, 189 Kawakami, H., E:28 A, 81 Kawata, S., M:14, 213 Kay, E., E:17, 245 Kazan, B., E:8, 447; E:28 B, 1059 Kechriotis, G., E:93, 1 Keen, R. S., E:30, 79 Kellenberger, E., E:63, 270 Kelly, J., E:43, 43 Kelly, R. J., E:57, 311; E:68, 1 Kennedy, D. P., E:18, 167 Kennedy, S. W., E:5, 213 Kerr, F. J., E:32, 1 Kerre, E., E:89, 255 Kerwin, L., E:8, 187 Kerwin, W. S., E:124, 139 Kessler, B., E:124, 195 Kestener, P., E:126, 1 Key, M. H., E:28, 999 Keyes, R. W., E:70, 159 Khan, S. A., E:97, 257 Khogali, A., E:22 A, 11; E:22 A, 31 Khursheed, A., E:115, 197; E:122, 87 Kidger, M. J., E:28 B, 759 Kimbell, G. H., E:31, 1 King, J. G., E:8, 1 Kingston, A., E:139, 75 Kislov, V. Y., E:21, 287 Kistemaker, J., E:21, 67 Kiuchi, Y., E:28 A, 253 Klein, N., E:26, 309 Knight, R. I., E:44, 221 Knoll, M., E:8, 447; E:19, 178 Koenderink, J. J., E:103, 65 Kohashi, T., E:22 B, 683; E:28 B, 1073 Kohen, C., M:14, 121 Kohen, E., M:14, 121 Kohl, H., E:65, 173 Kolev, P. V., E:109, 2 Komoda, T., E:96, 653, 685 Komrska, J., E:30, 139 Konigsberg, R. L., E:11, 225

230


Konrad, G. T., E:29, 1 Kornelson, E. V., E:17, 323 Kossel, D., E:28 A, 419 Kovalevsky, V. A., E:84, 197 Krahl, D., M:9, 1 Krakauer, H., E:65, 358 Krappe, H. J., ES:19, 129 Krieser, J. K., E:28 B, 603 Kron, G. E., E:16, 25; E:16, 35; E:22 A, 59; E:28 A, 1 Kronland-Martinet, R., ES:19, 289 Kropp, K., M:4, 385 Krowne, C. M., E:92, 79; E:98, 77; E:103, 151; E:106, 97 Krüger, N., E:131, 81 Kruit, P., E:96, 287; E:111, 92; M:12, 93 Kühl, W., E:28 B, 615 Kulikov, Yu. V., E:78, 156 Kulkarni, A. D., E:66, 310 Kunt, M., E:112, 1 Kunze, C., E:16, 217; E:28 B, 955 Kunze, W., E:28 B, 629 Kuo, K. H., E:96, xxviii Kusak, L., E:44, 283 Kuswa, G. W., ES:13C, 295 Kutay, M. A., E:106, 239 Kyser, D. F., E:69, 176

L Labeyrie, A., E:28 B, 899 Ladaga, J. L., E:120, 135 Ladouceur, H. D., ES:19, 89 Lahme, B., E:132, 69 Lainé, D. C., E:39, 183 Lakshmanasamy, S., ES:19, 79 Lallemand, A., E:12, 5; E:16, 1; E:22 A, 1 Lambert, F. J., ES:19, 509 Lambert, L., E:95, 3 Lambropoulos, P., E:54, 191 Lamport, D. L., E:28 B, 567 Landolt, M., E:62, 2 Lannes, A., E:126, 287 Lansiart, A., E:22 B, 941 Laques, P., E:22 B, 755 Lasenby, A., E:95, 271 Lashinsky, H., E:14, 265 Latecki, L. J., E:112, 95 Lauer, R., M:8, 137 Laurentini, A., E:139, 1

Lavine, R. B., ES:19, 169 Law, H. B., DS:1 Lawes, R. A., E:91, 139 Lawless, W.L., E:10, 153 Lawson, J. D., ES:13C, 1 Le Carvennec, F., E:28 A, 265 Le Contel, J. M., E:28 A, 27 Le Poole, J. B., ES:16, 387; M:4, 161 Leach, S., E:57, 1 Leder, L. B., E:7, 183 Lee, J. N., E:69, 115 Legoux, R., E:28 A, 367 Lehmann, J., E:7, 363 Lehmann, M., E:123, 225 Leifer, M., E:3, 306 Lejeune, C., ES:13A, 159; ES:13C, 207 Lenz, F., E:18, 251; E:96, 791 Lenz, R., E:138, 1 Leon, J. J.-P., ES:19, 369 Lessellier, D., ES19, 35, 51 Levi, D., ES:19, 445 Levi-Setti, R., ES:13A, 261 Levine, E., E:82, 277 Lewis, P. H., E:88, 63 Lewis, P. R., M:6, 171 Li, M. C., E:53, 269 Li, Y., E:85, 231 Lichte, H., E:123, 225; M:12, 25 Liddy, B. T., E:28 A, 375 Lieber, M., E:49, 134 Liebl, H., M:11, 101 Liebmann, G., E:2, 102 Lightbody, M. T. M., ES:19, 61 Limansky, I., E:22 A, 155 Lin, Y., E:141, 77 Lina, J.-M., E:109, 163 Linden, B. R., E:16, 311 Lindgren, A. G., E:56, 360 Lindsay, P. A., E:13, 181 Lisgarten, N. D., E:24, 109 Liu, I. D., E:28 B, 1021 Liu, S., E:141, 77 Liu, X., E:138, 147 Liu, Z., E:145, 95 Livingston, M. S., E:1, 269; E:50, 2 Livingston, W. C., E:16, 431; E:22 B, 705; E:23, 347 Lockner, T. R., ES:13C, 389 Lodwick, W. A., E:148, 75 Long, B. E., E:28 A, 119

CUMULATIVE AUTHOR INDEX Long, J. V. P., E:133, 259 Loo, B. W., E:22 B, 813 Low, W., E:24, 51 Lowrance, J. L., E:28 B, 851 Lu, C., E:93, 1 Luedicke, E., E:22 A, 175 Lukac, R., E:140, 187 Luppa, H., E:96, 171 Luukkala, M., M:12, 313 Lynds, C. R., E:22 B, 705 Lynds, R., E:28 B, 745 Lynton, E. A., E:23, 1

M Ma, C. L. F., E:99, 65 Maccari, A., ES:19, 463 Macres, V. G., ES:6, 73 Maeda, H., E:22 A, 331 Maeda, H., E:22 B, 683; E:28 A, 81 Maggini, M., E:140, 1 Mailing, L. R., E:22 B, 835 Majumdar, S., E:22 B, 985 Maldonado, J. R., D:2, 65 Malherbe, A., E:22 A, 493 Malnar, L., E:23, 39 Mandel, J., E:82, 327 Mandel, L., E:16 Mangclaviraj, V., E:96, xxiii Mankos, M., E:98, 323 Manley, B. W., E:16, 287; E:28 A, 471 Manna, M., ES:19, 429 Manson, S. T., E:41, 73; E:44, 1 Maragos, P., E:88, 199 Marchand-Maillet, S., E:106, 186 Marie, G., D:1, 225 Marinozzi, V., M:2, 251 Markushevich, V. M., ES:19, 191 Marshall, F. B., E:22 A, 291 Marshall, T. C., E:53, 48 Marti, J., E:120, 1 Martin, A., E:67, 183 Martin, B., ES:13A, 321 Martin, R., E:28 B, 981 Martinelli, R. U., D:1, 71 Martinez, G., E:81, 1 Marton, C., E:50, 449 Marton, L., E:7, 183; E:50, 449; ES:4; ES:6, 1 Massey, H. S. W., E:4, 2

231

Massmann, H., ES:19, 129 Masuda, K., E:37, 264 Mataré, H. F., E:42, 179; E:45, 40 Matson, C. L., E:124, 253 Matsuyama, T., E:86, 81 Matteucci, G., E:99, 171; E:122, 173 Mattiussi, C., E:113, 1; E:121, 143 Matzner, H., E:82, 277 Maunsbach, A. B., E:96, 21, 301 Maurer, C., E:146, 1 May, K. E., ES:19, 117 Mayall, B. H., M:2, 77 Mayer, H. F., E:3, 221 Mayer, J., E:123, 399 McCombe, B. D., E:37, 1; E:38, 1 McCurdy, W. J., E:142, 1 McDermott, D. B., E:53, 48 McGee, J. D., E:12, 87; E:16; E:16 A, 47; E:16, 61; E:22 A, 11; E:22 A, 31; E:22 A, 41; E:22 A, 113; E:22 A, 407; E:22 A, 571; E:22 B, 1003; E:22 A; E:22 B; E:33 A; E:33 B; E:28 A, 61; E:28 A, 89; E:28 A; E:28 B McGowan, J. C., E:118, 1 McKay, K. G., E:1, 66 McKee, J. S. C., E:73, 93 McLean, W. L., E:23, 1 McMullan, D., E:22 A; E:22 B; E:33 A; E:33 B; E:28 A, 61; E:28 A; E:28 B; E:28 A, 173; E:40 A; E:40 B; E:52 E:133, 3, 37, 59, 227, 523; ES:16, 443 McNamara, D. J., E:139, 179 McNish, A. G., E:1, 317 Mead, C. W., ES:6, 227 Medved, D. B., E:21, 101 Meeks, S. W., E:71, 251 Meffert, B., E:129, 1, E:137, 1 Meier, F., E:41, 113; E:62, 2 Meitzler, A. H., D:2, 65 Melamed, T., E:103, 1 Melford, D. A., E:133, 289 Mellini, M., E:76, 282 Melton, B. S., E:9, 297 Mende, S. B., E:22 A, 273 Mendelsohn, M. L., M:2, 77 Mendlovic, D., E:106, 239 Mendlowitz, H., E:7, 183 Merli, P. G., E:123, 375 Mertens, R. P., E:55, 77; E:82, 197 Mestwerdt, H., E:28 A, 19

232


Metherell, A. J. F., M:4, 263 Metson, G. H., E:8, 403 Meyer, F., E:148, 193 Meyer-Arendt, J. R., M:8, 1 Meyerhoff, K., E:28 B, 629 Michelsen, K., E:125, 119 Midgley, D., E:36, 153 Migliori, A., E:123, 311 Milla, F. E., E:71, 75 Millar, R. J., E:112, 1 Miller, D. E., E:28 A, 513 Miller, T. M., E:55, 119 Millonig, G., M:2, 251 Milnes, A. G., E:61, 64 Misell, D. L., E:32, 64; M:7, 185 Missiroli, G. F., E:99, 171, E:122, 173 Miyaji, K., E:22 A, 331; E:22 B, 683; E:28 B, 1073 Miyashiro, S., E:16, 171; E:16, 195; E:28 A, 191 Miyazaki, E., E:22 A, 331; E:28 A, 81 Mladjenović, M. S., E:30, 43 Mnyama, D., E:67, 1 Mockler, R. C., E:15, 1 Moenke-Blankenburg, L., M:9, 243 Möllenstedt, G., E:18, 251; E:96, 585; M:12, 1 Molski, M., E:101, 144 Monastyrsky, M. A., E:78, 156 Monro, P. A. G., M:1, 1 Montefusco, L. B., E:125, 1 Moore, D. S., M:6, 135 Morandi, V., E:123, 375 Morel, J.-M., E:111, 168 Moreno, T., E:14, 299 Morgan, B. L., E:28 A; E:28 B; E:28 B, 1051; E:40 A; E:40 B; E:52; E:64 A; E:64 B; E:74 Morgan, J. M., E:22 B, 885 Morgera, S. D., E:84, 261 Mori, N., E:99, 241, E:121, 281 Morlet, J., ES:19, 289 Morozumi, S., E:77, 2 Morrell, A. M., DS:1 Morton, G. A., E:4, 69; E:12, 183 Moschwitzer, A., E:47, 267 Mosig, J. R., E:59, 139 Moss, H., E:2, 2; ES:3 Motz, H., E:23, 153 Mueller-Neuteboom, S., M:8, 107

Mugnier, L. M., E:141, 1 Müller, E. W., E:13, 83 Müller, H., E:120, 41 Müller, J., E:55, 189 Müllerová, I., E:128, 309 Muls, P. A., E:47, 197 Mulvey, T., E:91, 259; E:95, 3; E:96, xix; ES:16, 417; E:115, 287 Mumolo, E., E:124, 1 Muñoz, X., E:120, 1 Munro, E., ES:13B, 73, E:133, 437 Murata, K., E:69, 176 Murowinski, R., E:106, 2 Murray, L. R., E:77, 210 Muselli, M., E:140, 61 Musmann, B. G., ES:12, 73

N Nachman, A. I., ES:19, 169 Nagai, K., E:70, 215 Nagayama, K., E:138, 69 Nakajima, N., E:93, 109 Nakamura, S., E:28 B, 1073 Nakamura, T., E:22 B, 683; E:16, 621; E:28 B, 1073 Nakayama, Y., E:16, 171; E:16, 195 Nakazawa, E., E:79, 271 Namba, S., E:37, 264 Napolitano, A., E:135, 103 Narcisi, R. S., E:29, 79 Nassenstein, H., E:16, 633 Nathan, R., M:4, 85 Nation, J. A., ES:13C, 171 Natterer, F., ES:19, 21 Navarro, R., E:97, 1 Needham, M. J., E:28 A, 129 Neil, V. K., ES:13C, 141 Nellist, P. D., E:113, 147 Nelson, P. D., E:28 A, 209 Nepijko, S. A., E:102, 274; E:113, 205; E:136, 227; E:142, 159 Neumann, M. J., E:12, 97 Newbury, D. E., E:62, 162 Newth, J. A., E:16, 501 Newton, A. C., E:28 A, 297 Nicolo Amati, L., ES:19, 269 Niedrig, H., E:96, 131 Niklas, W. F., E:16, 37; E:22 B, 781; E:22 B, 927 Ninomiya, T., E:28 A, 337

CUMULATIVE AUTHOR INDEX Niquet, G., E:28 A, 409 Nixon, W. C., E:21, 181; E:133, 195, 253 Nobiling, R., ES:13A, 321 Noe, E. H., E:16, 547 Norman, D. J., E:22 A, 551 Norton, K. A., E:1, 381 Novice, M., E:28 B, 1087 Novotný, B., E:28 A, 523 Nozawa, Y., E:22 B, 865; E:28 B, 891 Nudelman, S., E:28 B, 577 Nugent, K. A., E:118, 85 Nussli, J., E:60, 223

O Oatley, C. W., E:21, 181; E:133, 7, 111, 415, 419; ES:16, 443 Oesterschulze, E., E:118, 129 Ogle, J., D:3, 83 Oho, E., E:105, 78; E:122, 251 Oikawa, T., E:99, 241; E:121, 281 O’Keefe, T. W., E:28 A, 47 Okress, E. C., E:8, 503 Oleson, N. L., E:24, 155 Oliver, M., E:28 A, 61 Olson, C. L., ES:13C, 445 Olson, S. L., E:71, 357 Oman, R. M., E:26, 217 Omote, M., E:110, 102 Ong, P. S., ES:6, 137 Orloff, J., E:138, 147 Overstone, J. A., E:11, 185 Overwijk, M. H. F., E:111, 92 Owen, G., E:133, 445 Oyama, A., E:96, 679 Ozaktas, H. M., E:106, 239

P Padovani, C., ES:19, 269 Paganin, D., E:118, 85 Paindavoine, M., E:127, 125 Pal, S. K., E:88, 247 Palma, C., ES:19, 281 Pandit, M., E:53, 209 Paoletti, L., E:43, 1 Papiashvili, I. I., E:22 A, 59 Papliński, A. P., E:109, 200; E:119, 55 Paredes, J. L., E:117, 173 Parham, A. G., E:22 B, 801 Paris, M. G. A., E:128, 205

233

Parish, C. M., E:147, 1 Park, R.-H., E:134, 1 Paro, L., E:80, 165 Pascazio, S., E:142, 53 Paul, A. C., ES:13C, 141 Pawley, J. B., E:83, 203 Pawley, M. G., E:4, 301 Pease, R. F. W., E:21, 181; E:133, 187, 195 Pelc, S. R., M:2, 151 Pempinelli, F., ES:19, 377 Peng, L.-M., E:90, 205 Penman, J., E:70, 315 Pennycook, S. J., E:113, 147; E:123, 173 Perfilieva, I., E:147, 137 Peˇrina, J., E:142, 53 Perl, M. L., E:12, 153; E:16, 513 Perrenoud, J., E:27, 19 Perrott, R. H., E:85, 259 Pesch, P., E:12, 17 Petley, C. H., E:28 B, 837 Petrou, M., E:88, 297; E:130, 243; E:132, 109 Picat, J. P., E:28 A, 39 Pickering, H. W., ES:20 Picklesimer, M. L., ES:6, 197 Pierce, D. T., E:41, 113; E:56, 219 Pierce, J. A., E:1, 425 Pike, E. R., ES:19, 225 Pike, W. S., E:22 B, 885 Piller, H., M:5, 95 Pincus, H. J., M:7, 17 Pinoli, J. C., E:115, 129 Pinsker, Z. G., E:11, 355 Pippard, A. B., E:6, 1 Piroddi, R., E:132, 109 Plataniotis, K. N., E:140, 187 Plies, E., M:13, 123 Pluta, M., M:6, 49; M:10, 100 Pohl, D. W., M:12, 243 Polaert, R., E:28 B, 989 Poon, T.-C., E:126, 329 Porter, J. H., E:39, 73 Porter, N. A., E:16, 531 Potter, D. C., E:16, 501 Poultney, S. K., E:31, 39 Powell, J. R., E:22 A, 113; E:28 B, 745 Powers, W., E:16, 409; E:22 B, 713 Pozzi, G., E:93, 173; E:99, 171; E:122, 173; E:123, 207

234


Pratt, W. K., ES:12, 1 Preikszas, D., E:120, 41 Preston, K., M:5, 43 Prewitt, J. M. S., M:2, 77 Prince, J. L., E:124, 139 Pritchard, D. H., D:2, 169 Prosser, R. D., E:22 B, 969 Ptitsin, V. E., E:112, 165 Pucel, R. A., E:38, 195 Pulfrey, D. L., E:28 B, 1041 Purcell, S. T., E:95, 63

Q Qian, L. Z., E:96, xxviii Qian, L., E:142, 1 Quintenz, J. P., ES:13C, 295

R Rado, G. T., E:2, 251 Raffan, W. P., E:28 A, 433 Raffy, M., ES:19, 217 Rainforth, W. M., E:132, 167 Ramberg, E. G., DS:1 Ramm, A. G., ES:19, 153 Ramsdale, P. A., E:47, 123 Randall, R. P., E:12, 219; E:22 A, 87; E:28 B, 713 Rapperport, E. J., ES:6, 117 Rattey, P. A., E:56, 360 Recknagel, A., E:96, 171 Redhead, P. A., E:17, 323 Redlien, H. W., E:57, 311 Reed, R., ES:16, 483 Rehaˇcek, J., E:142, 53 Reid, M., E:112, 1 Reimer, L., E:81, 43 Reininger, W. G., E:22 A, 155 Reisner, J. H., E:73, 134 Rempfer, G. F., M:10, 269 Retzlaff, G., E:28 B, 629 Reynolds, G. T., E:22 A, 71; E:22 A, 381; E:28 B, 939; M:2, 1 Reynolds, T. T., E:16, 487 Riblet, H. B., E:11, 287 Rice, P. L., E:20, 199 Richards, E. A., E:28 B, 661 Richards, E. W. T., E:28 B, 981 Richter, K. R., E:82, 1 Rindfleisch, T., E:22 A, 341

Ritsch-Marte, M., E:146, 1 Ritter, G. X., E:80, 243; E:90, 353; E:144, 165 Rittner, E. S., E:31, 119; E:42, 41 Rivest, J.-F., E:144, 243 Roach, F. E., E:18, 1 Roane, G. D., E:22 A, 291 Roberts, A., E:12, 135 Robinson, L. C., E:26, 171 Roese, J. A., ES:12, 157 Rofheart, M., E:80, 1 Rohwer, C. H., E:146, 57 Ronchi, L., E:51, 64 Roos, J., M:4, 161 Roptin, D., E:61, 162 Rösch J., E:12, 113; E:16, 357; E:16, 371 Rose, A., E:1, 131 Rose, D. C., E:9, 129 Rose, H., E:65, 173; E:94, 197; ES:13C, 475; E:120, 41; E:132, 247 Rosen, D., M:9, 323 Rosenauer, A., E:107, 121 Rosenberger, H. E., M:3, 1 Rosencwaig, A., E:46, 208 Rosensweig, R. E., E:48, 103 Rosink, J. J. W. M., E:131, 147 Rossner, H. H., ES:19, 129 Roth, W., E:29, 79 Rothstein, J., E:14, 207 Rougeot, H., E:48, 1 Rouse, J. A., M:13, 2 Roux, G., E:22 B, 941 Roux, S.G., E:126, 1 Rowe, E. G., E:10, 185 Rowe, J. E., E:29, 1; E:31, 162 Rowlands, R. O., E:31, 267 Rubinacci, G., E:102, 2 Rüdenauer, F., E:57, 231 Ruedy, J. E., E:12, 183 Ruggiu, G., E:48, 202 Ruska, E., M:1, 115 Russell, L. A., E:21, 249 Russell, P. E., E:147, 1 Ryden, D. J., E:22 B, 801; E:28 B, 589 Ryssel, H., E:58, 191 Rzhanov, A. V., E:49, 1

S Saad, D., E:125, 231 Saalfeld, F. E., E:42, 2

CUMULATIVE AUTHOR INDEX Sabatier, P. C., ES:19, 1 Sacchi, M. F., E:128, 205 Sackinger, W. M., E:28 A, 487; E:28 A, 507 Sakai, A., ES:20 Sakoda, S., E:110, 102 Sakrison, D. J., ES:12, 21 Sakurai, T., ES:20 Salerno, E., E:97, 85 Sampson, D. G., E:97, 191 Sancho, M., E:81, 1 Santander, M., E:72, 182 Santini, P. M., ES:19, 389 Sarti, L., E:140, 1 Sarty, G. E., E:111, 244 Sasaki, T., E:16, 621 Sattlier, K., E:41, 113 Sauzade, M. D., E:34, 1 Saxton, W. O., ES:10 Saylor, C. P., M:1, 41 Sayood, K., E:72, 259 Scad, D. B., E:16, 487 Scarl, F., E:38 A, 535 Schaffner, J., E:5, 367 Schagen, P., D:1, 1; E:16, 75; E:16, 105; E:16, 287; E:28 A, 393 Schapink, F. W., E:96, 287 Scharfe, M. E., E:38, 83 Schattschneider, P., E:123, 413 Scheggi, A. M., E:51, 64 Scheinfein, M. R., E:98, 323 Scherzer, O., E:128, 445 Schied, W., ES:19, 117 Schiek, B., E:55, 309 Schilz, W., E:55, 309 Schimmel, G., E:96, 149 Schirmeisen, A., E:135, 41 Schiske, P., E:96, 59 Schlesinger, S. P., E:53, 48 Schluter, R. A., E:16, 475 Schmerling, E. R., E:19, 55 Schmidlin, F. W., E:38, 83 Schmidt, M., E:28 B, 767 Schnable, G. L., E:30, 79 Schneeberger, R. J., E:16, 299; E:16, 235 Schneider, C. M., E:142, 159 Schneider, J., E:58, 81 Schon, R. W., E:19, 178 Schönhense, G., E:113, 205; E:136, 227; E:142, 159

235

Schooley, A. H., E:19, 1 Schreiber, W. F., E:3, 306 Schumacher, B. W., E:65, 229 Schuster, G., E:28 B, 919 Sebe, N., E:144, 291 Sedov, N. N., E:102, 274; E:113, 205; E:136, 227 Seib, D. H., E:34, 95 Sekunova, L. M., E:68, 337 Semet, V., E:148, 1 Septier, A., E:14, 85; ES:13, M:1, 204 Séquin, C. H., ES:8 Shahriar, M., ES:19, 179 Shao, Z., E:81, 177 Shapiro, G., E:3, 195 Shaw, M. P., E:51, 310; E:60, 307 Shen, H. C., E:95, 387 Shen, Q., E:134, 69 Sheppard, C., M:10, 1 Shi, H., E:90, 353 Shidlovsky, I., D:4, 87 Shih, F. Y., E:140, 265 Shimadzu, S.-I., E:96, 665 Shimizu, Y., M:14, 249 Shinoda, G., ES:6, 15 Shirouzo, S., E:28 A, 191 Shnaider, M., E:109, 200; E:119, 55 Shrager, P. G., E:20, 261 Shtrikman, S., E:82, 277 Sicuranza, G. L., E:124, 1 Siddiqi, K., E:135, 1 Siegel, J., D:3, 83 Siegmann, H. C., E:41, 113; E:62, 2 Silvis-Cividjian, N., E:143, 1 Silzars, A., E:44, 221 Simon, G. T., E:96, 79 Simon, J. C., E:19, 255 Simpson, J. H., E:2, 185 Singer, B., D:3, 1 Singer, J. R., E:15, 73 Sirou, F., E:22 B, 949 Skorinko, G., E:16, 235 Slark, N. A., E:12, 247; E:16, 141; E:22 A, 63 Slodzian, G., ES:13B, 1 Slump, C. H., E:66, 202 Slusky, R. D., D:4, 157 Smiley, V. N., E:56, 2 Smit, J., E:6, 69 Smith G. R., E:73, 93

236


Smith, C. V. L., E:4, 157 Smith, C. W., E:12, 345; E:22 B, 1003 Smith, D. J., M:11, 2 Smith, F. H., M:6, 135; M:9, 223 Smith, K. C. A., E:133, 3, 93, 111, 311, 467, 501; ES:16, 443 Smith, R. W., E:22 B, 969; E:28 B, 1011; E:28 B, 1041; E:28 B, 1051 Smith-Rose, R. L., E:9, 187 Smyth, M. J., E:22 B, 741; E:28 B, 737 Snelling, M. A., E:133, 321, 335 Snoek, C., E:21, 67 Snoussi, H., E:146, 163 Sochen, N., E:136, 1 Sodha, M. S., E:27, 187 Soethout, L. L., E:79, 155 Solomon, P. R., E:51, 310 Somaroo, S., E:95, 271 Somlyody, A., D:3, 83 Southon, M. J., E:22 B, 903 Spehr, R., E:120, 41; ES:13C, 475 Spindt, C. A., E:83, 1 Spinella, S., E:44, 221 Spreadbury, P. J., E:133, 153 Srivastava, D., E:95, 387 Stahnke, I., E:22 A, 355 Starck, J.-L., E:132, 287 Stark, A. M., E:28 B, 567 Statz, H., E:38, 195 Staunton, R. C., E:107, 232; E:119, 191 Stebbings, R. F., E:59, 79 Steeds, J. W., E:123, 71 Steeves, G. M., E:125, 195 Stelzer, E. H. K., E:106, 293 Steriti, R., E:87, 1 Sternheimer, R. M., E:11, 31 Stevefelt, J., E:39, 121 Stewart, A. D. G., E:133, 175, 335 Stiller, D., E:96, 171 Stone, H. D., E:22 A, 565 Stopa, M., E:107, 2 Stoudenheimer, R. G., E:12, 41 Strausser, Y. E., E:21, 101 Stricker, S., E:25, 97 Stürimer, W., E:16, 613 Sturrock, J. M., E:133, 387 Su, T., E:34, 223 Sugon, Q. M., E:139, 179 Süsskind, C., E:8, 363; E:20, 261; E:50, 241; ES:16, 501

Suzuki, H., E:105, 267 Svalbe, I. D., E:89, 325; E:139, 75 Svelto, V., E:26, 251 Svitashev, K. K., E:49, 1 Swank, R. K., E:43, 205 Swanson, L. W., E:32, 194 Swanson, R. A., E:16, 487 Symons, R. S., E:55, 2 Syms, C. H. A., E:28 A, 399 Szepesi, Z., E:28 B, 1087 Szmaja, W., E:141, 175

T Tabbara, W., ES:19, 35, 51 Tabernero, A., E:97, 1 Tachiya, H., E:28 A, 337 Tadano, B, E:96, 227 Takenaka, H., M:14, 249 Taketoshi, L., E:28 A, 337 Taneda, T., D:2, 1 Taneja, I. J., E:76, 328; E:80, 165; E:91, 37; E:138, 177 Tari, S., E:111, 327 Tarof, L., E:99, 65 Tashiro, Y., E:96, 679 Taylor, A., E:16, 557 Taylor, D. G., E:16, 105; E:22 A, 395; E:28 B, 837 Taylor, S., E:12, 263 te Winkel, J., E:39, 253 Tepinier, M., E:28 A, 409 Ter-Pogossian, M., E:22 B, 927 Terol-Villalobos, I. R., E:118, 207 Tescher, A. G., ES:12, 113 Tessier, M., E:22 A, 493 Teszner, J. L., E:39, 291; E:44, 141 Teszner, S., E:39, 291 Tewary, V. K., E:67, 329 Thalhammer, R., E:135, 225 Theile, R., E:12, 277 Theodorou, D. G., E:12 A, 477 Thijssen, J. M., E:84, 317 Thomas, G., E:96, 21 Thomas, R. S., M:3, 99 Thonemann, F. F., E:11, 185 Thornley, R. F. M., E:133, 147, 159 Thornton, P. R., E:48, 272; E:54, 69 Thumwood, R. F., E:16, 163; E:22 A, 459; E:28 A, 129

CUMULATIVE AUTHOR INDEX Tian, Q., E:144, 291 Tiemeijer, P. C., E:111, 92 Timan, H., E:63, 73 Todkill, A., E:16, 127 Toepfer, A. J., E:53, 1 Tolimieri, R., E:80, 1; E:93, 1 Tomoviˇc, R., E:30, 273 Tompsett, M. F., ES:8 Tonazzini, A., E:97, 85; E:120, 193 Tovey, N. K., E:93, 219 Towler, G. O., E:28 A, 173 Triest, W. E., E:4, 301 Trindade, A. R., E:35, 88 Trolander, H. W., E:30, 235 Trombka, J. I., ES:6, 313 Trott, M. D., E:79, 1 Trunk, G. V., E:45, 203 Tschumperlé, D., E:145, 149 Tsuji, S., E:28 A, 253 Turcotte, P., E:109, 163 Turnbull, A. A., E:28 A, 393 Turner, P. S., E:96, 39 Twiddy, N. D., E:22 A, 273 Twiss, R. Q., E:5, 247

U Unger, H.-G., E:34, 281 Uno, Y., E:28 A, 81 Unwin, D. J., E:133, 339 Ura, K., E:73, 234; E:96, 263

V Vachier, C., E:148, 193 Vainshtein, B. K., M:7, 1 Valdés, J., E:138, 251 Valdrè, U., E:96, 193 Valentine, R. C., M:1, 180 Vallat, D., E:60, 223 van Aert, S., E:130, 1 van de Laak-Tijssen, D. J. J., E:115, 287 van de Walle, G. F. A., E:79, 155 van den Bos, A., E:117, 241; E:130, 1 van den Handel, J., E:6, 463 van der Polder, L. J., E:28 A, 237 van der Vaart, N. C., E:131, 147 van der Ziel, A., E:4, 110; E:46, 314; E:49, 225; E:50, 351 van Doorn, A. J., E:103, 65

237

van Dyck, D., E:65, 296; E:96, 67; E:123, 105; E:130, 1; E:136, 111 van Iterson, W., E:96, 271 van Kempen, H., E:79, 155 Van Khai, T., E:48, 202 van Overhagen, J., E:28 B, 615 van Overstraeten, R. J., E:47, 197; E:55, 77; E:82, 197 van Overstraeten, R., E:78, 104 van Roosmalen, J. H. T., E:28 A, 281 van Rooy, T. L., E:111, 92 van Vliet, L. J., E:126, 351 Vance, A. W., E:7, 363 Vardavoulia, M. I., E:119, 1 Varma, B. P., E:28 A, 89 Vasseur, J. P., E:48, 202 Veghte, J. H., E:30, 235 Vennik, J., E:43, 139 Verbeek, P. W., E:126, 351 Verkleij, A. J., E:96, 287 Vernier, P., E:22 A, 519; E:28 A, 409 Veron, S., E:22 A, 493; E:28 A, 461 Verwey, J. F., E:41, 249 Vesk, M., E:96, 39 Vicente, R., E:125, 231 Viehböck, F. P., E:57, 231 Vikram, C. S., E:142, 1 Vilim, P., E:22 A, 449 Vine, J., E:28 A, 47; E:28 A, 537 Virágh, S., E:96, 181 Vitulano, D., E:134, 113 Vodovnik, L., E:30, 283 Vogl, P., E:62, 101 Voisin, P., E:72, 1 von Ardenne, M., E:96, 635; ES:16, 1 von Aulock, W. H., ES:5 von Borries, H., E:81, 127 von Engel, A., E:20, 99 Voorman, J. O., E:37, 80 Vosburgh, K. G., E:43, 205 Vuković, J. B., E:96, xxv

W Wachtei, M. M., E:12, 59 Wade, R. H., M:5, 239 Wadlin, M. L., E:7, 363 Wagner, K. H., E:28 B, 1033 Wagner, R. J., E:37, 1; E:38, 1 Wait, J. R., E:25, 145 Walker, J. S., E:124, 343

238


Walker, M. F., E:16, 241; E:22 B, 761; E:28 B, 773 Wallman, B. A., E:133, 359 Walsh, D. C. I., E:138, 321 Walsh, J. E., E:58, 271 Walter, G. G., E:139, 225 Walters, F. W., E:16, 249 Walters, J., E:16, 501 Wang, J., E:93, 219 Wang, Y. L., E:81, 177 Wardley, J., E:16, 227; E:22 A, 211; E:28 A, 247 Warnecke, R. R., E:3, 43 Watanabe, K., E:86, 173 Watanabe, Y., E:96, 723 Waters, J. R., E:16, 487 Watson, C. J., E:23, 153 Wayland, H., M:6, 1 Webster, H. F., E:17, 125 Webster, W. M., E:6, 257 Wechsler, H., E:69, 262 Wehner, G. K., E:7, 239 Weichan, C., ES:16, 525 Weimer, P. K., E:13, 387; E:37, 182 Weingartner, H. C., E:5, 213 Weirich, T.E., E:123, 257 Welford, W. T., M:2, 41 Wells, O. C., E:133, 127 Welton, M. G. E., M:2, 151 Welton, T. A., E:48, 37 Wendt, G., E:28 A, 137 Wenzel, D., E:94, 1 Westphal, J., E:142, 1 Wheeler, B. E., E:16 A, 47; E:16, 61; E:22 A, 51 Whelan, M. J., E:39, 1 Whetten, N. R., E:27, 59 White, J. E., E:3, 183 White, N. S., E:113, 249 White, R. M., E:51, 265 Whitmell, D. S., E:22 B, 903 Wickramasinghe, H. K., M:11, 153 Wijn, H. P. J., E:6, 69 Wilburn, B., E:112, 233 Wilcock, W. L., E:16, 127; E:16, 383; E:22 A, 535; E:22 A, 629; E:28 A, 513 Wild, J. P., E:7, 299 Wild, M., E:146, 57 Williams, D. B., E:62, 162 Williams, F. E., E:5, 137

Williams, G. F., E:75, 389 Williams, M. A., M:3, 219 Williamson, W., E:49, 134 Willingham, D., E:22 A, 341 Wilson, B. L. H., E:91, 141 Wilson, G. A., E:28 B, 1051 Wilson, J. N., E:90, 353 Wilson, R. G., ES:13B, 45 Wimmer, E., E:65, 358 Windridge, D., E:134, 181 Winkler, G. M. R., E:44, 34 Winternitz, P., ES:19, 457 Winters, K. H., E:78, 104 Winters, R., E:131, 147 Wise, H. S., E:28 B, 981 Witcomb, M. J., E:96, 323 Wlérick, G., E:12, 5; E:16, 5, 357, 371; E:22 A, 465; E:28 B, 787 Wolf, R. C., ES:6, 73 Wolfe-Coote, S. A., E:96, 323 Wolff, I. O., ES:16, 557 Wolfgang, L. T., E:22 B, 671 Wollnik, H., ES:13B, 133 Wolpers, C., E:81, 211 Wolstencroft, R. D., E:28 B, 783 Wood, J., E:107, 310 Woodhead, A. W., E:16, 105; E:28 B, 567 Woolgar, A. J., E:16, 141 Woonton, G. A., E:15, 163 Wörgötter, F., E:131, 81 Wreathall, W. M., E:16, 333; E:22 A, 583 Wyatt, J. R., E:42, 2 Wyckoff, R. W. G., ES:16, 583 Wynne, C. G., E:28 B, 759

X Ximen, J., E:81, 231; E:91, 1; E:97, 359; ES:17; M:13, 244

Y Yada, K., E:96, 217, 245, 773 Yagi, K., M:11, 57 Yakowitz, H., ES:6, 361 Yakushev, E. M., E:68, 337 Yalamanchili, S., E:87, 259 Yamamoto, H., E:79, 271 Yamamoto, M., D:2, 1 Yamazaki, E., E:105, 142 Yang, E. S., E:31, 247

CUMULATIVE AUTHOR INDEX Yang, F., E:127, 125 Yang, T., E:109, 266; E:114, 79 Yarman, C. E., E:141, 257 Yaroslavskii, L. P., E:66, 1 Yavor, M. I., E:103, 277; E:86, 225 Yavor, S. Ya., E:76, 3 Yazici, B., E:141, 257 Yildirim, N., E:60, 307 Yu, F. T. S., E:63, 1 Yu, J., E:144, 291

Z Zacharias, J. R., E:8, 1 Zacharov, B., E:12, 31; E:16, 67; E:16, 99 Zaefferer, S., E:125, 355

Zakhariev, B. N., ES:19, 99, 141 Zalm, P., E:25, 211 Zamperoni, P., E:92, 1 Zavalin, A., E:142, 1 Zawisky, M., E:142, 53 Zayezdny, A. M., E:94, 315 Zdanis, R. A., E:16, 487 Zeevi, Y. Y., E:136, 1 Zeh, R. M., M:1, 77 Zeitler, E., E:25, 277 Ziliani, F., E:112, 1 Zimmerman, B., E:29, 257 Zolesio, J. P., ES:19, 533 Zou, X., E:123, 257 Zucchino, P. M., E:28 B, 851

239


CONTENTS OF VOLUMES 100–149

VOLUME 100 (1997) Partial Cumulative Index

VOLUME 101 (1997) 1. Applications of transmission electron microscopy in mineralogy, P.E. Champness 2. High-resolution electron microscopy of quasicrystals, K. Hiraga 3. Formal polynomials for image processing, A. Imiya 4. The dual de Broglie wave, M. Molski

1–36 37–98 99–142 143–238

VOLUME 102 (1997) 1. Finite element methods for the solution of 3D eddy current problems, R. Albanese and G. Rubinacci 2. Nanofabrication for electronics, W. Chen and H. Ahmed 3. Miniature electron optics, A.D. Feinerman and D.A. Crewe 4. Optical interconnection networks, K.M. Iftekharuddin and M.A. Karim 5. Aspects of mirror electron microscopy, S.A. Nepijko and N.N. Sedov

1–86 87–186 187–234 235–272 273–324

VOLUME 103 (1998) 1. Space-time representation of ultra wideband signals, E. Heyman and T. Melamed 2. The structure of relief, J.J. Koenderink and A.J. van Doorn 3. Dyadic Green’s function microstrip circulator theory for inhomogeneous ferrite with and without penetrable walls, C.M. Krowne 241

1–64 65–150

151–276

242

CONTENTS OF VOLUMES

100–149

4. Charged particle optics of systems with narrow gaps: A perturbation theory approach, M.I. Yavor

277–388

VOLUME 104 (1999) Cumulative Index

VOLUME 105 (1999) 1. Near-sensor image processing, A. Åström and R. Forchheimer 2. Digital image processing technology for scanning electron microscopy, E. Oho 3. Design and performance of shadow-mask color cathode ray lubes, E. Yamazaki 4. Electron gun systems for color cathode ray tubes, H. Suzuki

1–76 77–140 141–266 267–404

VOLUME 106 (1999) 1. Effects of radiation damage on scientific charge coupled devices, T.D. Hardy, M.J. Deen, and R. Murowinski 2. CAD using Green’s functions and finite elements and comparison to experimental structures for inhomogeneous microstrip circulators, C.M. Krowne 3. Discrete geometry to image processing, S. Marchand-Maillet 4. Introduction to the fractional Fourier transform and its applications, H.M. Ozaktas, M.A. Kutay, and D. Mendlovic 5. Confocal microscopy, E.H.K. Stelzer and F.M. Haar

1–96

97–184 185–238 239–292 293–346

VOLUME 107 (1999) 1. Magneto-transport as a probe of electron dynamics in open quantum dots, J.P. Bird, R. Akis, D.K. Ferry, and M. Stopa 2. External optical feedback effects in distributed feedback semiconductor lasers, M.F. Alam and M.A. Karim 3. Atomic scale strain and composition evaluation from high-resolution transmission electron microscopy images, A. Rosenauer and D. Gerthsen 4. Hexagonal sampling in image processing, R.C. Staunton 5. The group representation network: A general approach to invariant pattern classification, J. Wood

1–72 73–120

121–230 231–308 309–408

CONTENTS OF VOLUMES

100–149

243

VOLUME 108 (1999) Modern Map Methods in Particle Beam Physics by MARTIN BERZ

VOLUME 109 (1999) 1. Development and applications of a new deep level transient spectroscopy method and new averaging techniques, P.V. Kolev and M.J. Deen 2. Complex dyadic multiresolution analyses, J.-M. Lina, P. Turcotte and B. Goulard 3. Lattice vector quantization for wavelet-based image coding, M. Shnaider and A.P. Papliński 4. Fuzzy cellular neural networks and their applications to image processing, T. Yang

1–162 163–198 199–264 265–446

VOLUME 110 (1999) 1. Interference scanning optical probe microscopy: Principles and applications, W.S. Bacsa 2. High-speed electron microscopy, O. Bostanjoglo 3. Soft mathematical morphology: Extensions, algorithms, and implementations, A. Gasteratos and I. Andreadis 4. Difference in the Aharonov–Bohm effect on scattering states and bound states, S. Sakoda and M. Omote

1–20 21–62 63–100 101–172

VOLUME 111 (1999) 1. Number theoretic transforms and their applications in image processing, S. Boussakta and A.G.J. Holt 2. On the electron-optical properties of the ZrO/W Schottky electron emitter, M.J. Fransen, Th.L. Van Rooy, P.C. Tiemeijer, M.H.F. Overwijk, J.S. Faber, and P. Kruit 3. The size of objects in natural and artificial images, L. Alvarez, Y. Gousseau, and J.-M. Morel 4. Reconstruction of nuclear magnetic resonance imaging data from non-Cartesian grids, G.E. Sarty 5. An integrated approach to computational vision, S. Tari

1–90

91–166 167–242 243–326 327–366

244

CONTENTS OF VOLUMES

100–149

VOLUME 112 (2000) 1. Second-generation image coding, N.D. Black, R.J. Millar, M. Kunt, M. Reid, and F. Ziliani 2. The Aharonov–Bohm effect – A second opinion, W.C. Henneberger 3. Well-composed sets, L. Jan Latecki 4. Non-stationary thermal field emission, V.E. Ptitsin 5. Theory of ranked-order filters with applications to feature extraction and interpretive transforms, B. Wilburn

1–54 55–94 95–163 165–231 233–332

VOLUME 113 (2000) 1. The finite volume, finite element, and finite difference methods as numerical methods for physical field problems, C. Mattiussi 2. The principles and interpretation of annular dark-field Z-contrast imaging, P.D. Nellist and S.J. Pennycook 3. Measurement of magnetic fields and domain structures using a photoemission electron microscope, S.A. Nepijko, N.N. Sedov and G. Schönhense 4. Improved laser scanning fluorescence microscopy by multiphoton excitation, N.S. White and R.J. Errington

1–146 147–203

205–248 249–277

VOLUME 114 (2000) 1. Artificial intelligence and pattern recognition techniques in microscope image processing and analysis, N. Bonnet 2. Continuous-time and discrete-time cellular neural networks, T. Yang

1–77 79–324

VOLUME 115 (2001) 1. Modern tools for Weyl–Heisenberg (Gabor) frame theory, P.G. Casazza 2. Logarithmic image processing: The mathematical and physical framework for the representation and processing of transmitted images, M. Jourlin and J.C. Pinoli 3. Recent developments in scanning electron microscope design, A. Khursheed

1–127

129–196 197–285

CONTENTS OF VOLUMES

100–149

4. Jan Bart Le Poole (1917–1993) pioneer of the electron microscope and particle optics, T. Mulvey and D.J.J. van de Laak-Tijssen 5. A souvenir of Philips electron microscopes, P.W. Hawkes

245

287–354 355–361

VOLUME 116 (2001) Numerical Field Calculation for Charged Particle Optics by ERWIN KASPER

VOLUME 117 (2001) 1. Optimal and adaptive design of logical granulometric filters, E.R. Dougherty and Y. Chen 2. Dyadic warped wavelets, G. Evangelista 3. Recent developments in stack filtering and smoothing, J.L. Paredes and G.R. Arce 4. Resolution reconsidered – Conventional approaches and an alternative, A. van den Bos and A.J. den Dekker

1–71 73–171 173–239 241–360

VOLUME 118 (2001) 1. Magnetic resonance imaging and magnetization transfer, J.C. McGowan 2. Noninterferometric phase determination, D. Paganin and K.A. Nugent 3. Recent developments of probes for scanning probe microscopy, E. Oesterschulze 4. Morphological image enhancement and segmentation, I.R. Terol-Villalobos

1–83 85–127 129–206 207–273

VOLUME 119 (2001) 1. Binary, gray-scale, and vector soft mathematical morphology: Extensions, algorithms, and implementations, M.I. Vardavoulia, A. Gasteratos and I. Andreadis 2. Still image compression with lattice quantization in wavelet domain, M. Shnaider and A.P. Paplinski 3. Morphological scale-spaces, P.T. Jackway 4. The processing of hexagonally sampled images, R.C. Staunton

1–53 55–121 123–189 191–265

246

CONTENTS OF VOLUMES

100–149

5. Space-variant two-dimensional filtering of noisy images, A. De Santis, A. Germani and L. Jetto

267–318

VOLUME 120 (2002) 1. A review of image segmentation techniques integrating region and boundary information, X. Cufi, X. Muñoz, J. Freixenet and J. Marti 2. Mirror corrector for low-voltage electron microscopes, P. Hartel, D. Preikszas, R. Spehr, H. Müller and H. Rose 3. Characterization of texture in scanning electron microscope images, J.L. Ladaga and R.D. Bonetto 4. Degradation identification and model parameter estimation in discontinuity-adaptive visual reconstruction, A. Tonazzini and L. Bedini

1–39 41–133 135–191

193–284

VOLUME 121 (2002) 1. High-speed electron microscopy, O. Bostanjoglo 2. Applications of transmission electron microscopy in mineralogy, P.E. Champness 3. Three-dimensional fabrication of miniature electron optics, A.D. Feinerman and D.A. Crewe 4. A reference discretization strategy for the numerical solution of physical field problems, C. Mattiussi 5. The imaging plate and its applications, N. Mori and T. Oikawa

1–51 53–90 91–142 143–279 281–332

VOLUME 122 (2002) 1. The structure of quasicrystals studied by atomic-scale observations of transmission electron microscopy, K. Hiraga 2. Add-on lens attachments for the scanning electron microscope, A. Khursheed 3. Electron holography of long-range electrostatic fields, G. Matteucci, G.F. Missiroli and G. Pozzi 4. Digital image-processing technology useful for scanning electron microscopy and its practical applications, E. Oho

1–86 87–172 173–249 251–327

CONTENTS OF VOLUMES

100–149

247

VOLUME 123 (2002) Microscopy, Spectroscopy, Holography and Crystallography with Electrons Guest Editors PIER GEORGIO MERLI, GIANLUCA CALESTANI, MARCO VITTORI-ANTISARI 1. 2. 3. 4. 5.

Signposts in electron optics, P.W. Hawkes Introduction to crystallography, G. Calestani Convergent beam electron diffraction, J.W. Steeds High-resolution electron microscopy, D. van Dyck Structure determination through Z-contrast microscopy, S.J. Pennycook 6. Electron holography of long-range electromagnetic fields: A tutorial, G. Pozzi 7. Electron holography: A powerful tool for the analysis of nanostructures, H. Lichte and M. Lehmann 8. Crystal structure determination from EM images and electron diffraction patterns, S. Hovmoller, X. Zou and T.E. Weirich 9. Direct methods and applications to electron crystallography, C. Giacovazzo, F. Capitelli, C. Cuocci and M. Ianigro 10. Strategies in electron diffraction data collection, M. Gemmi, G. Calestani and A. Migliori 11. Advances in scanning electron microscopy, L. Frank 12. On the spatial resolution and nanoscale feature visibility in scanning electron microscopy, P.G. Merli and V. Morandi 13. Nanoscale analysis by energy-filtering TEM, J. Mayer 14. Ionization edges: Some underlying physics and their use in electron microscopy, B. Jouffrey, P. Schattschneider and C. Hébert

1–28 29–70 71–103 105–171 173–206 207–223 225–255 257–289 291–310 311–325 327–373 375–398 399–411

413–450

VOLUME 124 (2003) 1. V-Vector algebra and Volterra filters, A. Carini, E. Mumolo and G.L. Sicuranza 2. A brief walk through sampling theory, A.G. García 3. Kriging filters for space–time interpolation, W.S. Kerwin and J.L. Prince

1–61 63–137 139–193

248

CONTENTS OF VOLUMES

100–149

4. Constructions of orthogonal and biorthogonal scaling functions and multiwavelets using fractal interpolation surfaces, B. Kessler 5. Diffraction tomography for turbid media, C.L. Matson 6. Tree-adapted wavelet shrinkage, J.S. Walker

195–251 253–342 343–394

VOLUME 125 (2003) 1. An algebraic approach to subband signal processing, M. Barnabei and L.B. Montefusco 2. Determining the locations of chemical species in ordered compounds: ALCHEMI, I.P. Jones 3. Aspects of mathematical morphology, K. Michelsen, H. De Raedt and J.Th.M. De Hosson 4. Ultrafast scanning tunneling microscopy, G.M. Steeves and M.R. Freeman 5. Low-density parity-check codes – A statistical physics perspective, R. Vicente, D. Saad and Y. Kabashima 6. Computer-aided crystallographic analysis in the TEM, S. Zaefferer

1–62 63–117 119–194 195–229 231–353 355–415

VOLUME 126 (2003) 1. A wavelet-based method for multifractal image analysis: From theoretical concepts to experimental applications, A. Arneodo, N. Decoster, P. Kestener and S.G. Roux 2. An analysis of the geometric distortions produced by median and related image processing filters, E.R. Davies 3. Two-photon excitation microscopy, A. Diaspro and G. Chirico 4. Phase closure imaging, A. Lannes 5. Three-dimensional image processing and optical scanning holography, T.-C. Poon 6. Nonlinear image processing using artificial neural networks, D. De Ridder, R.P.W. Duin, M. Egmont-Petersen, L.J. Van Vliet and P.W. Verbeek

1–92 93–193 195–286 287–327 329–350

351–450

VOLUME 127 (2003) 1. Scanning nonlinear dielectric microscopy, Y. Cho 2. High-order accurate methods in time-domain computational electromagnetics: A review, J.S. Hesthaven

1–57 59–123

CONTENTS OF VOLUMES

100–149

3. Prefiltering for pattern recognition using wavelet transform and neural networks, F. Yang and M. Paindavoine 4. Electron optics and electron microscopy: Conference proceedings and abstracts as source material, P.W. Hawkes

249

125–206 207–379

VOLUME 128 (2003) 1. 2. 3. 4.

Fourier, block, and lapped transforms, T. Aach On fuzzy spatial distances, I. Bloch Mathematical morphology applied to circular data, A. Hanbury Quantum tomography, G.M. D’Ariano, M.G.A. Paris and M.F. Sacchi 5. Scanning low-energy electron microscopy, I. Müllerová and L. Frank 6. Scale-space methods and regularization for denoising and inverse problems, O. Scherzer

1–50 51–122 123–204 205–308 309–443 445–530

VOLUME 129 (2003) Calculus of Finite Differences in Quantum Electrodynamics by HENNING F. HARMUTH AND BEATE MEFFERT

VOLUME 130 (2004) 1. Statistical experimental design for quantitative atomic resolution transmission electron microscopy, S. Van Aert, A.J. Den Dekker, A. Van Den Bos, and D. Van Dyck 2. Transform-based image enhancement algorithms with performance measure, A.M. Grigoryan and S.S. Agaian 3. Image registration: an overview, M. Petrou

1–164 165–242 243–291

VOLUME 131 (2004) 1. Introduction to hypergraph theory and its use in engineering and image processing, A. Bretto 2. Image segmentation using the Wigner–Ville distribution, J. Hormigo and G. Cristóbal 3. Statistical and deterministic regularities: Utilization of motion and grouping in biological and artificial visual systems, N. Krüger and F. Wörgötter

1–64 65–80

81–146

250

CONTENTS OF VOLUMES

100–149

4. The hopping electron cathode for cathode ray tubes, J.J.W.M. Rosink, M.G.H. Hiddink, R. Winters, and N.C. Van Der Vaart

147–237

VOLUME 132 (2004) 1. Evanescent waves in the near and the far field, H.F. Arnoldus 2. Symmetry and the Karhunen–Loève decomposition, B. Lahme 3. Analysis of irregularly sampled data: A review, R. Piroddi and M. Petrou 4. Recent developments in the microscopy of ceramics, W.M. Rainforth 5. Five-dimensional Hamilton–Jacobi approach to relativistic quantum mechanics, H. Rose 6. Redundant multiscale transforms and their application for morphological component separation, J.-L. Starck, M. Elad, and D. Donoho

1–67 69–107 109–165 167–246 247–285

287–348

VOLUME 133 (2004) Sir Charles Oatley and the Scanning Electron Microscope Guest editors BERNARD C. BRETON, DENNIS McMULLAN, KENNETH C.A. SMITH 1. Charles Oatley: Father of modern scanning electron microscopy, K.C.A. Smith and D. McMullan 2. The early history of the scanning electron microscope, C.W. Oatley 3. The development of the first Cambridge scanning electron microscope, 1948–1953, D. McMullan 4. An improved scanning electron microscope for opaque specimens, D. McMullan 5. Exploring the potential of the scanning electron microscope, K.C.A. Smith 6. The scanning electron microscope and its fields of application, K.C.A. Smith and C.W. Oatley 7. Building a scanning electron microscope, O.C. Wells 8. Contrast formation in the scanning electron microscope, T.E. Everhart

3–6 7–34 37–57 59–91 93–109 111–125 127–136 137–145

CONTENTS OF VOLUMES

100–149

9. Wide-band detector for micro-microampere low-energy electron currents, T.E. Everhart and R.F.M. Thornley 10. A simple scanning electron microscope, P.J. Spreadbury 11. New applications of the scanning electron microscope, R.F.M. Thornley 12. A.D.G. Stewart and an early biological application of the scanning electron microscope, A. Boyde 13. Investigation of the topography of ion bombarded surfaces with a scanning electron microscope, A.D.G. Stewart 14. The scanning electron microscopy of hot and electron-emitting specimens, H. Ahmed 15. Towards higher-resolution scanning electron microscopy, R.F.W. Pease 16. High resolution scanning electron microscopy, R.F.W. Pease and W.C. Nixon 17. The application of the scanning electron microscope to microfabrication and nanofabrication, A.N. Broers 18. Scanning electron diffraction: A survey of the work of C. W. B. Grigson, D. McMullan 19. The development of the X-ray projection microscope and the X-ray microprobe analyser at the Cavendish Laboratory, Cambridge, 1946–60, V. E. Cosslett 20. The contributions of W. C. Nixon and J. V. P. Long to X-ray microscopy and microanalysis: Introduction, P. Duncumb 21. X-ray projection microscopy, W.C. Nixon 22. Microanalysis, J.V.P. Long 23. Development of the scanning electron probe microanalyser, 1953–1965, P. Duncumb 24. Micro-analysis by a flying-spot X-ray method, V.E. Cosslett and P. Duncumb 25. Tube investments research laboratories and the scanning electron probe microanalyser, D.A. Melford 26. Commercial exploitation of research initiated by Sir Charles Oatley, K.C.A. Smith 27. AEI electron microscopes – Background to the development of a commercial scanning electron microscope, A.W. Agar 28. M ICROSCAN to S TEREOSCAN at the Cambridge Instrument Company, M.A. Snelling 29. A new scanning electron microscope, A.D.G. Stewart and M.A. Snelling

251

147–152 153–158 159–164 165–173 175–178 179–185 187–193 195–205 207–226 227–233

237–250 251–252 253–257 259–267 269–283 285–287 289–308 311–315 317–320 321–334 335–337

252

CONTENTS OF VOLUMES

100–149

30. Memories of the scanning electron microscope at the Cambridge Instrument Company, D.J. Unwin 31. From microscopy to lithography, B.A. Wallman 32. Commercial electron beam lithography in Cambridge, 1973–1999: A view from the drawing board, J.M. Sturrock 33. Charles Oatley: The later years, The Editors 34. The detective quantum efficiency of the scintillator/ photomultiplier in the scanning electron microscope, C.W. Oatley 35. Professor Oatley remembered, E. Munro 36. Recollections of Professor Oatley’s reincarnation as a research student, G. Owen 37. My life with the S TEREOSCAN, B.C. Breton 38. Research at the Cambridge University Engineering Department post-S TEREOSCAN, K.C.A. Smith 39. The development of biological scanning electron microscopy and X-ray microanalysis, P. Echlin 40. From the scanning electron microscope to nanolithography, J.R.A. Cleaver Appendix I. Sir Charles William Oatley, O.B.E., F.R.S. (Royal Society Biographical Memoir), K.C.A. Smith Appendix II. A History of the scanning electron microscope, 1928–1965, D. Mcmullan Appendix III. The Cambridge instrument company and electron-optical innovation, P. Jervis

339–358 359–386 387–412 415–418

419–436 437–443 445–448 449–465 467–468 469–484 485–498 501–521 523–545 547–555

VOLUME 134 (2005) 1. Circulant matrix representation of feature masks and its applications, R.-H. Park and B. Ho Cha 2. Phase problem and reference-beam diffraction, Q. Shen 3. Fractal encoding, D. Vitulano 4. Morphologically debiased classifier fusion: A tomography-theoretic approach, D. Windridge

1–68 69–112 113–179 181–266

VOLUME 135 (2005) 1. Optics, mechanics, and Hamilton–Jacobi skeletons, S. Bouix and K. Siddiqi 2. Dynamic force microscopy and spectroscopy, H. Holscher and A. Schirmeisen

1–39 41–101

CONTENTS OF VOLUMES

100–149

3. Generalized almost-cyclostationary signals, L. Izzo and A. Napolitano 4. Virtual optical experiments, R. Thalhammer

253

103–223 225–333

VOLUME 136 (2005) 1. Real and complex PDE-based schemes for image sharpening and enhancement, G. Gilboa, N. Sochen and Y.Y. Zeevi 2. The S-state model for electron channeling in high-resolution electron microscopy, P. Geuens and D. Van Dyck 3. Measurement of electric fields on object surface in an emission electron microscope, S.A. Nepijko, N.N. Sedov and G. Schönhense

1–109 111–226 227–316

VOLUME 137 (2005) Dogma of the Continuum and the Calculus of Finite Differences in Quantum Physics by HENNING F. HARMUTH AND BEATE MEFFERT

VOLUME 138 (2005) 1. Spectral color spaces: Their structure and transformations, R. Lenz 2. Phase contrast enhancement with phase plates in electron microscopy, K. Nagayama 3. A study of optical properties of gas phase field ionization sources, X. Liu and J. Orloff 4. On symmetric and nonsymmetric divergence measures and their generalizations, I.J. Taneja 5. Features and future of the International System of Units (SI), J. Valdés 6. The importance sampling Hough transform, D.C.I. Walsh

1–67 69–146 147–175 177–250 251–320 321–359

VOLUME 139 (2006) 1. Retrieval of shape from silhouette, A. Bottino and A. Laurentini 1–73 2. Projective transforms on periodic discrete image arrays, A. Kingston and I. Svalbe 75–177 3. Ray tracing in spherical interfaces using geometric algebra, Q.M. Sugon, Jr. and D.J. McNamara 179–224

254

CONTENTS OF VOLUMES

100–149

4. Prolate spheroidal wave functions and wavelets, G.G. Walter

225–295

VOLUME 140 (2006) 1. Recursive neural networks and their applications to image processing, M. Bianchini, M. Maggini, and L. Sarti 2. Deterministic learning and an application in optimal control, C. Cervellera and M. Muselli 3. X-ray fluorescence holography, K. Hayashi 4. A taxonomy of color image filtering and enhancement solutions, R. Lukac and K.N. Plataniotis 5. General sweep mathematical morphology, F.Y. Shih

1–60 61–118 119–185 187–264 265–306

VOLUME 141 (2006) 1. Phase diversity: A technique for wave-front sensing and for diffraction-limited imaging, L.M. Mugnier, A. Blanc, and J. Idier 2. Solving problems with incomplete information: A gray systems approach, Y. Lin and S. Liu 3. Recent developments in the imaging of magnetic domains, W. Szmaja 4. Deconvolution over groups in image reconstruction, B. Yazıcı and C.E. Yarman

1–76 77–174 175–256 257–300

VOLUME 142 (2006) 1. Conservative optical logic devices: COLD, H.J. Caulfield, L. Qian, C.S. Vikram, A. Zavalin, K. Chouffani, J. Hardy, W.J. McCurdy, and J. Westphal ˇ 2. Advanced neutron imaging and sensing, J. Reháˇ cek, Z. Hradil, J. Peˇrina, S. Pascazio, P. Facchi, and M. Zawisky 3. Time-resolved photoemission electron microscopy, G. Schönhense, H.J. Elmers, S.A. Nepijko, and C.M. Schneider

1–52 53–157 159–323

VOLUME 143 (2006) 1. Electron-beam-induced nanometer-scale deposition, N. Silvis-Cividjian and C.W. Hagen

1–235

CONTENTS OF VOLUMES

100–149

255

VOLUME 144 (2006) 1. Recent progress in high frequency electron cyclotron resonance ion sources, D. Hitz 2. Fixed points of lattice transforms and lattice associative memories, G. Ritter and P. Gader 3. An extension of mathematical morphology to complex signals, J.-F. Rivest 4. Ranking metrics and evaluation measures, J. Yu, J. Amores, N. Sebe, and Q. Tian

1–164 165–242 243–289 291–316

VOLUME 145 (2007) 1. Applications of noncausal Gauss–Markov random field models in image and video processing, A. Asif 2. Direct electron detectors for electron microscopy, A.R. Faruqi 3. Exploring third-order chromatic aberrations of electron lenses with computer algebra, Z. Liu 4. Anisotropic diffusion partial differential equations for multichannel image regularization: Framework and applications, D. Tschumperlé and R. Deriche

1–53 55–93 95–148

149–209

VOLUME 146 (2007) 1. Spiral phase microscopy, S. Fürhapter, A. Jesacher, C. Maurer, S. Bernet, and M. Ritsch-Marte 2. LULU theory, idempotent stack filters, and the mathematics of vision of Marr, C.H. Rohwer and M. Wild 3. Bayesian information geometry: Application to prior selection on statistical manifolds, H. Snoussi

1–56 57–162 163–207

VOLUME 147 (2007) 1. Scanning cathodoluminescence microscopy, C.M. Parish and P.E. Russell 2. Fuzzy transforms: A challenge to conventional transforms, I. Perfilieva

1–135 137–196

VOLUME 148 (2007) 1. Planar cold cathodes, V.T. Binh and V. Semet

1–73

256

CONTENTS OF VOLUMES

100–149

2. Interval and fuzzy analysis: A unified approach, W.A. Lodwick 3. On the regularization of the watershed transform, F. Meyer and C. Vachier

VOLUME 149 (2007) Electron Emission Physics by KEVIN L. JENSEN

75–192 193–249

Index

A

Bragg–Brentano data set 113, 147, 150, 154, 156 method 172, 179 subset 118, 172, 175

Aberration 57, 58, 90, 97, 105, 108, 130, 154, 160, 165 chromatic 56, 57, 64, 75 coefficients 60, 63, 64 correction 53, 79 spherical 56, 57, 75 Algorithms, iterative 8, 10, 38, 40 Ambiguities 95, 98–100 Angle of incidence 54, 70, 125 Aperture function 95, 105, 106, 108, 110, 113, 146, 153, 157, 158, 174 Approximations 5, 8, 38, 56, 97, 105, 120, 121, 126, 132–134, 153 kinematical 105, 137, 138 weak phase 133, 135, 137, 138 Arcwise connectivities 191, 192 Artifacts 1, 11, 21, 42, 165 Astigmatism 147, 148 Atomic force microscopy (AFM) 82 Average phase change 137, 138

C Cartoon 3, 11, 46, 47 components 3, 15, 18, 20, 30 part 3, 4, 15, 42 Coherence, partial 94, 142, 147, 161, 175, 176, 178 Color image reconstruction 2 segmentation 185, 187, 200, 206 Compression 2, 187, 193 Computational cost 18, 19, 38 Connected components 192, 193, 195, 211, 212 operators 185, 210, 215 Connection 186, 189, 190, 192–194, 197, 203, 204, 210, 215 jump 186, 202, 203 partial 204, 206 smooth 185, 196, 197, 201, 203 Connective criteria 193–197, 199, 200, 210 segmentation 185, 186, 194, 203 Connectivity 192 Constraints 2, 3, 21, 23, 30, 33, 92, 137, 147, 153, 161, 164 Convergence 2, 3, 22, 26, 34, 56, 167, 172, 186 rates 26, 38, 40, 41 Convex 2, 7, 9, 24, 30, 31, 33, 36 constraints 1, 2 penalization 1, 33 sets 31, 32, 35 closed 31–33 Convolution 38, 92, 105, 106, 110–112, 128, 165 theorem 92, 93, 95, 105, 134, 138

B Beams illuminating 93, 94, 113, 131, 155, 160, 165, 179 pairs of 109, 110, 136 primary 58, 69, 70 scattered 137, 156 undiffracted 101, 102, 115, 137, 142, 154 unscattered 132, 135, 137 Besov constraints 1, 2, 38 Bounded variation constraints 1–3, 38 function 3, 30 penalty terms 3, 30 Bragg angle 113, 118, 154 beams 154, 155, 158 reflections, perfect 154

257

258 Crystal 89, 106, 109, 110, 114, 115, 117, 154, 186 Crystalline silicon 137, 140, 141, 161 Crystallography 91, 104, 108, 145, 148, 149, 158, 159, 161 Cycle spinning 1, 11, 12, 16, 18

D Data cube 140, 142, 150 set 27, 118, 138–140, 147, 150, 172, 174, 179 ptychographical 119, 122, 151, 164, 176 Deblurring of images 1, 2, 3, 30, 33, 41 Decomposition of images 2, 4, 21, 206 optimal 22, 45, 46 Delta functions 111, 112, 115, 127, 132, 152 Denoising 1–3, 30, 33, 37 Detector 54, 63, 88, 119–121, 125, 129, 138, 139, 141, 155, 162, 163, 168, 179, 180 plane 97, 105, 120, 121, 127, 130, 150, 166 Diffracted beams 99–102, 107–109, 115, 118, 144, 147, 154 disks 98, 103, 140, 146 Diffraction 56, 88, 93, 94, 99, 111, 137, 141, 169 orders 92, 97, 98, 110, 112, 114–116, 118, 139, 140, 142, 143, 147 adjacent 110, 112 patterns 90, 91, 93–95, 97, 98, 100, 108–110, 112–114, 116, 118, 122, 132, 134, 138–140, 142, 143, 150, 151, 161, 163–170 single 95, 163, 164 plane 93, 95, 105, 107, 108, 114, 116–118, 128, 137, 150, 163, 165, 175, 177 spots 116–118 Digital image processing 53, 79 Dilation 187, 192, 197, 198 Directional sensitivity 1, 13 Discrete wavelet schemes 11 Dynamical scattering 87, 123, 132, 134, 143, 147, 150, 180

INDEX

E Edge enhancement 17, 18 Edges 3, 4, 11, 15, 157, 163, 168, 169 Electron lenses 55, 75, 90, 94 microscopy 104, 119, 123, 132, 147, 197 Electron-channeling patterns 53, 70 Emitters, coldfield 62, 63 Environmental SEM 53, 74 Ewald sphere 92, 125, 127, 129–132, 134, 136, 145, 154–156 Exit wave 104, 111, 120, 123, 124, 129, 133, 134, 137, 141, 144, 145, 166, 167, 180

F Feature-space based segmentation 185, 197 Field emission 62 Filtering rules 14 Filters 186, 208, 211, 213, 215 connected 186, 211 Finite support 87, 109, 110 Focal length 56–58, 63 plane 92, 97–99, 101, 105, 113, 133, 142, 145, 153, 154, 164, 174, 175 Fourier coefficients 7, 112 domain 38, 90, 91, 110 shift theorem 95, 100, 106, 131, 138 transform 89–92, 94, 104–106, 110, 112, 117, 118, 124, 126, 132–134, 136, 150, 151, 153–155, 165, 172–175, 179 Frame 1, 2, 21–23, 25–27 dictionaries 1, 21, 23 Fraunhofer condition 125, 127 diffraction plane 90, 98, 101, 114, 124, 165 Frequency projections 1, 13, 18 Functionals 30, 32, 187

G General convex constraint preliminaries 1, 30 Geodesic dilations 198 Geographical information systems 187 Grazing angle 54

259

INDEX

H Hemisphere 100, 101, 124, 125 High-energy electrons 94, 138, 161 Hilbert space 22, 25, 26, 31 Holography 87, 90, 93, 94, 96, 133, 134, 140, 141, 179 Huygens’ principle 121

I Illumination function 92, 95–97, 100, 102, 106, 109, 111, 113–115, 117–119, 121, 122, 126, 130, 138–140, 165–169, 173, 174, 178 Image blurred 39, 40 conventional 134, 137, 140, 142, 143, 145, 150, 153, 155, 156, 158, 176, 180 decomposition 1, 3, 7, 30, 47 initial 20, 209, 212, 214 noisy 16, 17, 40 plane 93, 139, 154, 155 processing 2, 3, 186, 188, 210 ptychographical 104, 119 resolution 82, 94 restoration 3, 1–3, 30 saturation 208, 209 segmentation 191, 197, 203, 206 Imaging methods 87, 93 Immersion lens detector 53, 63 Incident beams 101, 113, 114, 126, 155–157, 170 plane waves 113, 127, 128, 130, 131 Intensity diffracted 116, 117, 143 measurements 100, 107, 118 sampling 116 Interference patterns 95, 96 Inverse problems 1, 2, 22 Iterated jumps 185 Iteration schemes 1, 2, 8, 30, 38 Iterative strategy 1, 2, 7, 22, 32

K Kinematical scattering 130, 137

L Lagrange equation 37, 38, 43

Landing energy 57, 64, 66, 79 Lanthanum hexaboride (LaB6 ) 61–63 Lattice points, reciprocal 106, 111, 113, 116 Lens 56, 58, 60, 63, 64, 87, 89, 90, 93, 94, 97–101, 130, 133, 141–143, 145, 153, 154, 164, 165, 174, 176 probe-forming 141, 143, 147, 148, 153 Levelings 185, 187, 211, 213, 214 Light microscope 53, 54 Lipschitz criterion 189–191 Low-loss imaging 53, 68 Luminance 207-209

M Magnetic field 58, 60, 64, 68, 75 lenses 64, 89 Magnifications 53, 55, 73, 74, 77, 82, 83, 187 Matheron semigroups 213, 214 Measurements 88, 98, 103, 106, 108, 109, 117, 138, 139 Method, iterative 23, 163, 164, 167, 168 Minimization 3, 8, 25, 27, 38, 42, 43, 46 total variation 37, 40, 41 Minimizers 1, 3, 5, 7, 8, 10, 11, 23–26, 34, 37, 38, 43, 44 Mixed segmentations 185, 206 Models 3, 4, 18, 19, 38, 41, 42, 44, 55

N Nonlinear filtering 8, 11, 12 Norm convergence 2, 34, 35 Numerical functions 185, 187, 188, 205, 211 Nyquist sampling 87, 114, 116, 127

O Object crystalline 88, 96–98, 118, 123, 147 function 95, 97, 100, 104, 110, 117, 119, 123, 127, 150, 165, 166, 172, 173, 178 nonperiodic 108, 176 periodic 96, 98, 146 plane 92, 97, 100, 110, 117–120, 125, 139, 164, 165, 167, 169, 171, 174 space 100, 117, 130 structure 147, 180 thin 105, 133

260 transmission function 105 weak phase 132, 137 Objective lens 58, 154 Optic axis 109, 127–129, 131, 132, 139, 154–156 Optimizations 187, 188 Oscillating components 11, 15 Oversampling 116

P Partial connections 185, 204, 209 Partitions 185, 188–193, 195, 196, 201, 205–208, 213 synthetic 208, 209, 213 Pattern recognition 215 Penalty weights, local 16 Periodicity 92, 106, 113, 114, 116, 145, 150, 155 Phase absolute 99, 121, 126, 173 change 101, 118, 120, 127, 131–133, 135 components 105, 108 difference 99, 100, 108, 109, 112, 117, 137 ptychographical 113, 143 image 158, 169 lost 89 offset 159, 161 problem 87–90, 92–96, 98–100, 110, 116, 121, 125, 164, 172, 173, 179 ptychographic 111, 144, 162 ramp 100–102, 110, 117, 158, 165, 168, 169 relative 89, 93, 99, 108–110, 140, 147 retrieval 110, 163 Phase-closure errors 147 Phase-locking 69 Phase-retrieval methods 88, 89 iterative 163, 165 Phase, zero 99, 100, 135, 166, 174 Phonon collection 53, 73 Piezoelectric sensor 74 Plane complex 99, 133, 135, 142 wave 96, 98, 99, 101, 117, 119, 133 Probe positions 103, 109, 136, 139, 140, 142, 143, 151, 162, 167 Projections 13, 14, 33, 53, 104, 133, 137, 145, 147, 155, 157, 161

INDEX Ptychograph 121, 133, 134, 138, 145, 147, 149, 161, 163, 166–168, 171, 173, 174 Ptychographical convolution 116, 122 data set 119, 122, 164, 176 phases 111, 143, 179 reconstructions 134, 136 Ptychography 87, 88, 90, 92–97, 102, 103, 105–110, 113, 114, 117–119, 133, 134, 136–138, 140–143, 147, 149, 164, 165, 169, 170, 176, 178–180 classical 118, 138 crystalline 107, 117, 118, 136, 176 electron 147 two-beam 87, 113, 114

R Radiation 93–98, 104, 113, 119, 126, 139, 153, 165, 171 Real space 92, 95, 106, 109, 111, 114, 118, 122, 128, 130, 133, 136, 137, 155, 163, 165, 173 Reciprocal space 92, 94, 96, 101, 104, 111, 113, 115, 117, 118, 122, 125, 126, 128, 130, 131, 137, 138, 154–156, 173, 174 Reciprocity 139, 142, 154, 155, 160, 178 Reconstruction 18, 28, 29, 90, 129, 140, 141, 156, 161–164, 168, 170, 171, 175–177, 185, 186, 211, 212 opening 185, 211, 212, 214 Redundant systems 21 Refinements 11, 164 Replica technology 54 Resolution atomic 66, 67 improved 63, 81, 176

S Sample chamber 75, 81 surface 55, 60, 66 Sampling 114, 115, 138, 150, 170 Saturation 207–209 Scanning commercial SEM 54, 60–63, 66 force microscopy (SFM) 82, 83 (mechanical) probe microscopy 68

261

INDEX (reflection) electron microscope (SEM) 43, 53–58, 60–64, 66, 67, 69, 71–75, 77, 79, 81–83, 93, 170, 213, 214 tabletop SEM 75, 77 transmission electron microscope (STEM) 54, 62, 77, 79, 88, 97, 101, 105, 106, 109–113, 117, 118, 123, 127, 138, 139, 142, 154, 155, 179 tunneling microscopy (STM) 66–68, 81–83 Scattering 104, 122, 123, 126, 127, 129, 133, 134, 137, 147, 150 angles 91, 94, 119, 126, 127, 145 objects 123 Secondary electron emission 57, 58, 66, 68, 81 electrons 54, 55, 57, 58, 60, 61, 63, 64, 66, 69 Seed-based segmentations 185, 199 Seedless segmentations 185, 196 Seeds 196, 199, 200, 206 theorem 185, 199 Segmentation 185–188, 190, 191, 193–195, 197, 199, 200, 202, 204–209, 213, 215 morphological 186 partitions 210 theorem 185, 193 Segments 16, 191, 194, 195, 199, 205 Shift 95, 96, 100, 101, 113, 131, 132, 140 lateral 96, 140 Shifting 100, 102, 112, 117, 118, 155 Signal-to-noise ratio 18, 58 Signals, video 55, 58, 62, 68, 69 Silhouette 187, 205 Silicon 136, 142, 143, 145–147, 149, 158 Simulations 187 Single-jump connection 185, 201 Singletons 191, 192, 194, 196, 200, 203–206 Sparsity 1, 2, 21, 22, 26–29 Stepping route 142, 149 Stochastic models 187 Surface area 198, 199, 208, 209 topography 57, 58, 60, 69, 74, 82 Surrogate functional method 2, 40 functionals 1, 8, 9, 24, 34, 38, 41, 43

T Texture analysis 2 component 16, 18, 20 part 3, 4 Thermal wave microscopy 73 Thermionic emitters 62 Thickness 105, 136, 137, 147, 148, 158, 159, 170 Time domain 13, 28, 29, 92 Tomography 137 Transfer function 153, 161, 175 Translation invariance 1, 11, 12 Transmission electron microscope (TEM) 53, 54, 58, 62, 63, 64, 68, 77, 79, 88, 97, 154, 155, 161, 162 function 98, 104, 120, 123, 153 complex 104 Tungsten 62, 63 Two-beam ptychography 87, 113, 114

U Ultrahigh vacuum 53, 68, 81, 179 Unit cell 90, 91, 96, 108, 110–115, 117, 123, 138, 140, 145 Unscattered beam 132, 135, 137

V Variational equations 9, 10, 24 functionals 3, 12, 14 problems 1, 3–5, 7, 8, 23, 26, 31, 42 Variations 69, 70, 155, 187, 188, 191, 201, 213 total 4 Voltage contrast 60, 61, 69, 70

W Watershed 201, 206, 209 algorithms 206 contours 185, 205 Wave components 93, 94, 99, 134 disturbance 93, 119, 121 Wavefield 90, 93, 117 Waveform systems 21 Wavefronts 100, 101, 198, 199

262 Wavelength 56, 88–91, 100, 105, 119, 122, 135, 180 Wavelet 1, 3, 5–7, 12, 13, 21, 38, 42, 123 coefficients 5, 8, 12–16, 27, 42, 43 complex 11, 13 domain 3 preliminaries 1, 5 shrinkage 4, 19 transform 11, 13 Wavelet-based scheme 3, 5, 42 treatment 3, 4 Waves

INDEX incident 95, 120, 126, 127, 129 scattered 114, 127, 131, 133, 135, 136, 173 spherical 121, 125, 126 Weighted penalty functions 1, 14 Wigner distribution deconvolution 137, 161, 172, 174–179

X X-ray 60, 89, 93, 120, 138, 170, 177, 179 crystallography 89, 91, 114 microanalysis 61

Advances in Imaging and Electron Physics, Volume 150 (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 148 (Advances in Imaging and Electron Physics) (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 127 (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 132 (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 143 (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 120 (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 121 (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 111 (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 102 (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 113 (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 144 (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 128 (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 125 (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics (Volume 112) (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 113 (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, volume 136 (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 111 (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 101 (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 135 (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 130 (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 99 (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 141 (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 146 (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 123: Advances in Electron Microscopy and Diffraction (Advances in Imaging and Electron Physics)

Advances in Imaging and Electron Physics, Volume 134 (Advances in Imaging & Electron Physics)

Advances in Imaging & Electron Physics - Volume 100 Cumulative Index (Advances in Imaging and Electron Physics)

Advances in Imaging & Electron Physics, Volume 122 (Advances in Imaging and Electron Physics)